After 17 years of existence, the Apache HTTP web server is one of the most successful and dominant open source projects of all time. Today Apache is looking to extend that streak of success with the new Apache 2.4 web server release, marking the first major Apache web server update since the 2.2 release in 2005.

A key focus in the 2.4 release is improved performance which is delivered by way of multiple innovations.

“What we have done is checked 2.4 against itself and other web-servers; in general, we find 2.4 to be the fastest version of Apache by far,” Jim Jagielski, ASF President and Apache HTTP Server Project Management Committee, told InternetNews.com.

Jagielski added that with the different Multi-Processing Modules (MPMs) available in Apache 2.4, administrators can fine-tune Apache to be faster where it makes sense. For example, he noted that if there is a need for a super fast request/response, then the Prefork MPM might make sense. If there is a need for a super fast concurrency-tagged server, then Event is the ticket.

“We also show that as far as true performance is based – real-world performance as seen by the end-user- 2.4 is as fast, and even faster than some of the servers who may be “better” known as being “fast”, like nginx,” Jagielski said.

Nginx has emerged in recent years to be a rival of sorts to Apache, offering up a web server that has the promise of being faster in some situations. While Apache has long dominated the market as being the most widely deployed web server technology, Nginx is according to at least one metric, now in the number two spot.

Jagielski said that all web-servers and web-caches have their uses and sweet spots.

“With Apache 2.4,it is generic enough, powerful enough and now performant enough that it can be used in places where the multiple-box approach may have been used,” Jagielski said.

In a multiple box approach, there is the potential for additional latency, which is not the case in a single box Apache approach

“Of course, you could also use Apache as a replacement for either nginx or varnish or even replace both with Apache; that’s the power and flexibility that Apache 2.4 provides,” Jagielski said. “One reason to do it would be a single, consistent learning curve for both the web front-end as well as the caching layer.”

Apache Software Foundation committer Graham Legget told InternetNews.com that there were significant improvements made to Apache httpd’s mod_cache mechanism in order to support high traffic environments that are subjected to extreme traffic in a broadcasting and media environment.

“These improvements include support for serving stale data during the revalidation of expired content, as well as the serving of stale data as an alternative to an application server error,” Legget said. “The ability to place the cache precisely within the filter chain allows the option to cache application generated content, while personalising the same content further down the filter chain, without being forced to deploy separate caching layers.”

Additionally Legget noted that there have also been improvements to the cache reporting mechanisms. He explained that the cache can now report in detail why a particular response was or wasn’t cached, which can help developers and operations debug issues with potentially uncacheable or inefficient content delivery. Apache has also updated its httpd caching guide to better explain the operation of three-state RFC2616 HTTP caches, and how they can improve server performance: http://httpd.apache.org/docs/2.4/caching.html#http-caching

mod_proxy
With Apache 2.4, proxy capabilities have also been significantly improved with the integration of mod_proxy and mod_proxy balancer.

“For the cloud, you want and need dynamic configuration: you need to easily and quickly add proxies, adjust load-balancing capabilities, turn-off backends, etc,” Jagielski said. “With 2.4, Apache allows you to do all that, also, in addition to the dynamic capability that cloud infrastructures demand, we also have the performance as well.”

mod_session
Another new innovation in Apache 2.4 is the mod_session module for implementing sessions.

“We want to instead provide a service to module developers as well as sys-admins who have always needed to worry about implementing sessions, in one way or another,” Jagielski said. “Mod_session provides a universal framework that people can use, which has been lacking for a long time.”

Leggett added that a key reason for the development of mod_session along with mod_auth_form was to allow Apache httpd to handle the form based single sign on scenarios that are typically handled by application servers directly, usually in various incompatible ways.

“The idea is that httpd can be configured to worry about the sign on process, optionally passing the user credentials to applications running behind an Apache httpd reverse proxy using simple basic authentication,” Leggett explained. “This can help provide a common sign on mechanism in an environment where legacy or incompatible heterogeneous systems are present.”

The Future of Apache
For existing Apache 2.2.x users, migrating to 2.4 is quite easy, according to Jagielski. He noted that there are minimal configuration changes. He added that the vast majority of modules do not require any changes to run under 2.4; they just need to be re-compiled and then the work is done.

While Apache 2.4 is a significant release that add a lot, Jagielski said that there’s always more work do be done.

“We’re looking at really streamlining the internal filter chain, which provides flexibility but can slow things down,” Jagielski said. “We’re also looking at a new MPM called Simple which is as elegant, efficient and “simple” as possible, in order to get the best performance with the absolute minimum of resource utilization.”

Problem:

I need to set up some significant usage reporting for a web application. I’m looking for a reporting application for Apache style logs that’s robust and customizable. Not looking just for some little report of top URLs for the day, but something that goes into detail on user agents, page load times, IP addresses, etc. Something with a pluggable report system would be ideal because I need to do some app-specific custom reports.

Javascript-based logging like Mint and server-hosted things like Google Analytics won’t work.

I need something that works primarily off of logs. Storing data in MySQL is ok, but I’d prefer just to work straight from log files.

What’s good, flexible, and not ugly? I know about Webalizer, Analog, and AWStats, but in the past when I’ve used them they’ve all seemed a bit awkward and limited. It’s been awhile, maybe they’ve improved? Or maybe there’s an alternative?

Solution:

Webalizer, Analog, or AWStats

Analog shows you the usage patterns on your web server. It’s

* Ultra-fast
* Scalable
* Highly configurable

Compiling and using Analog

Downloading and unpacking

Download analog from http://www.analog.cx/download.html

Unpack the Analog source code via the following commands:-

gunzip analog-6.0.tar.gz
tar -xvf analog-6.0.tar

Then move the whole analog directory to where you would like it located, maybe /usr/local/ so enter something like mv analog-6.0 /usr/local/

Then, cd into the relevant directory, ie. cd /usr/local/analog-6.0 and edit the anlghead.h to edit some default values. Most operational parameters can be put in a config file, so you only really need to set the hostname etc.

Compile analog by typing make

If all is well, an analog binary should be placed in the /usr/local/analog-6.0 directory. You now need to edit the relevant configuration files before you can run analog to analyse your apache logs.

Trying to compile sysbench on a 64bit RedHat Enterprise Linux.

drv_mysql.c:35:19: error: mysql.h: No such file or directory
drv_mysql.c:36:26: error: mysqld_error.h: No such file or directory
drv_mysql.c: In function âmysql_drv_describeâ:
drv_mysql.c:284: error: âMYSQL_RESâ undeclared (first use in this function)
drv_mysql.c:284: error: (Each undeclared identifier is reported only once
drv_mysql.c:284: error: for each function it appears in.)
drv_mysql.c:284: error: âresâ undeclared (first use in this function)
drv_mysql.c:285: error: âMYSQL_ROWâ undeclared (first use in this function)
drv_mysql.c:285: error: expected â;â before ârowâ
drv_mysql.c:286: error: âMYSQL_FIELDâ undeclared (first use in this function)
drv_mysql.c:286: error: âfieldsâ undeclared (first use in this function)
drv_mysql.c:328: error: ârowâ undeclared (first use in this function)
drv_mysql.c: In function âmysql_drv_connectâ:
drv_mysql.c:360: error: âMYSQLâ undeclared (first use in this function)
drv_mysql.c:360: error: âconâ undeclared (first use in this function)
drv_mysql.c:366: error: expected expression before â)â token
drv_mysql.c:401: error: âMYSQL_VERSION_IDâ undeclared (first use in this function)
drv_mysql.c: In function âmysql_drv_disconnectâ:
drv_mysql.c:441: error: âMYSQLâ undeclared (first use in this function)
drv_mysql.c:441: error: âconâ undeclared (first use in this function)
drv_mysql.c: In function âmysql_drv_bind_paramâ:
drv_mysql.c:520: error: âMYSQLâ undeclared (first use in this function)
drv_mysql.c:520: error: âconâ undeclared (first use in this function)
drv_mysql.c:520: error: expected expression before â)â token
drv_mysql.c: In function âmysql_drv_queryâ:
drv_mysql.c:729: error: âMYSQLâ undeclared (first use in this function)
drv_mysql.c:729: error: âconâ undeclared (first use in this function)
drv_mysql.c:739: error: âER_LOCK_DEADLOCKâ undeclared (first use in this function)
drv_mysql.c:739: error: âER_LOCK_WAIT_TIMEOUTâ undeclared (first use in this function)
drv_mysql.c:739: error: âER_CHECKREADâ undeclared (first use in this function)
drv_mysql.c: In function âmysql_drv_store_resultsâ:
drv_mysql.c:789: error: âMYSQLâ undeclared (first use in this function)
drv_mysql.c:789: error: âconâ undeclared (first use in this function)
drv_mysql.c:790: error: âMYSQL_RESâ undeclared (first use in this function)
drv_mysql.c:790: error: âresâ undeclared (first use in this function)
drv_mysql.c:791: error: âMYSQL_ROWâ undeclared (first use in this function)
drv_mysql.c:791: error: expected â;â before ârowâ
drv_mysql.c:839: error: âER_LOCK_DEADLOCKâ undeclared (first use in this function)
drv_mysql.c:839: error: âER_LOCK_WAIT_TIMEOUTâ undeclared (first use in this function)
drv_mysql.c:855: error: ârowâ undeclared (first use in this function)
drv_mysql.c: In function âmysql_drv_free_resultsâ:
drv_mysql.c:879: error: âMYSQL_RESâ undeclared (first use in this function)
drv_mysql.c:879: error: expected expression before â)â token
make[3]: *** [libsbmysql_a-drv_mysql.o] Error 1
make[3]: Leaving directory `/tmp/sysbench-0.4.12/sysbench/drivers/mysql'
make[2]: *** [all-recursive] Error 1
make[2]: Leaving directory `/tmp/sysbench-0.4.12/sysbench/drivers'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/tmp/sysbench-0.4.12/sysbench'
make: *** [all-recursive] Error 1

I will describe here very quickly how to compile sysbench under CentOS. Probably it will work for other distributions if you have the same problem.

Here’s the error:

[... some lines removed ...]
/bin/sh ../libtool --tag=CC   --mode=link gcc -pthread -g -O2      -o sysbench sysbench.o sb_timer.o sb_options.o sb_logger.o db_driver.o tests/fileio/libsbfileio.a tests/threads/libsbthreads.a tests/memory/libsbmemory.a tests/cpu/libsbcpu.a tests/oltp/libsboltp.a tests/mutex/libsbmutex.a drivers/mysql/libsbmysql.a -L/usr/local/mysql/lib/ -lmysqlclient_r   -lrt -lm
../libtool: line 838: X--tag=CC: command not found
../libtool: line 871: libtool: ignoring unknown tag : command not found
../libtool: line 838: X--mode=link: command not found
../libtool: line 1004: *** Warning: inferring the mode of operation is deprecated.: command not found
../libtool: line 1005: *** Future versions of Libtool will require --mode=MODE be specified.: command not found
../libtool: line 2231: X-g: command not found
../libtool: line 2231: X-O2: command not found
../libtool: line 1951: X-L/usr/local/mysql/lib/: No such file or directory
../libtool: line 2400: Xsysbench: command not found
../libtool: line 2405: X: command not found
../libtool: line 2412: Xsysbench: command not found
../libtool: line 2420: mkdir /.libs: No such file or directory
../libtool: line 2547: X-lmysqlclient_r: command not found
../libtool: line 2547: X-lrt: command not found
../libtool: line 2547: X-lm: command not found
../libtool: line 2629: X-L/root/sysbench-0.4.12/sysbench: No such file or directory
../libtool: line 2547: X-lmysqlclient_r: command not found
../libtool: line 2547: X-lrt: command not found
../libtool: line 2547: X-lm: command not found
../libtool: line 2629: X-L/root/sysbench-0.4.12/sysbench: No such file or directory
../libtool: line 2547: X-lmysqlclient_r: command not found
../libtool: line 2547: X-lrt: command not found
../libtool: line 2547: X-lm: command not found
../libtool: line 5162: Xgcc -pthread "" "" -o @OUTPUT@ sysbench.o sb_timer.o sb_options.o sb_logger.o db_driver.o  tests/fileio/libsbfileio.a tests/threads/libsbthreads.a tests/memory/libsbmemory.a tests/cpu/libsbcpu.a tests/oltp/libsboltp.a tests/mutex/libsbmutex.a drivers/mysql/libsbmysql.a -L/root/sysbench-0.4.12/sysbench -lmysqlclient_r -lrt -lm: No such file or directory
../libtool: line 5163: Xgcc -pthread "" "" -o @OUTPUT@ sysbench.o sb_timer.o sb_options.o sb_logger.o db_driver.o  tests/fileio/libsbfileio.a tests/threads/libsbthreads.a tests/memory/libsbmemory.a tests/cpu/libsbcpu.a tests/oltp/libsboltp.a tests/mutex/libsbmutex.a drivers/mysql/libsbmysql.a -L/root/sysbench-0.4.12/sysbench -lmysqlclient_r -lrt -lm: No such file or directory
../libtool: line 5168: X: command not found
../libtool: line 5172: : command not found

There are some libtool errors, we will recreate all automake utilities with autogen:

$ ./autogen.sh

If you don’t get any error, run:

./configure
make
make install

All set, test it by running:

 sysbench --test=cpu --cpu-max-prime=20000 run

This is how to setup public key authentication. Here are the steps:

1. Make sure the target SSH server accept public key authentication
As root, open sshd_config on the target server, usually located in /etc/ssh/sshd_config

vim /etc/ssh/sshd_config

Make sure these lines are available and not commented

PubkeyAuthentication yes
AuthorizedKeysFile .ssh/authorized_keys

2. Make sure the ssh client know where the private key is located
As root, open ssh_config on the client machine, usually located on /etc/ssh/ssh_config

vim /etc/ssh/ssh_config

Make sure this line is available:

IdentityFile ~/.ssh/identity
IdentityFile ~/.ssh/id_rsa
IdentityFile ~/.ssh/id_dsa

3. Create a pair of private and public keys
To do public key authentication, you need to have a pair of keys. Private key is stored on ssh client machine as your identification, and public key is stored on the target ssh server, used by sshd to authenticate you on ssh server.

Log into the ssh client machine with the account that you’ll be using to connect to target server, and issue this command:

ssh-keygen -t dsa

The output should be similiar to this:

#:~/.ssh$ ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/root/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_dsa.
Your public key has been saved in /root/.ssh/id_dsa.pub.
The key fingerprint is:
be:4d:36:7c:29:a9:e0:8a:c0:c5:d5:65:40:fe:97:4b root@192.168.1.1

Since my requirement was passwordless authentication, I didn’t enter any passphrase. Note that the private (id_dsa) and public (id_dsa.pub) keys have been generated, and stored .ssh directory

#:~$ cd .ssh
#:~/.ssh$ ls
authorized_keys  id_dsa  id_dsa.pub  id_rsa  id_rsa.pub

4. Install the public key into the target ssh server
Copy the public key into target ssh server

#:~$ cd .ssh
#:~/.ssh$ scp id_dsa.pub surfer@M5base:/home/surfer/.ssh
root@192.168.1.2's password:
id_dsa.pub                                    100%  605     0.6KB/s   00:00

log into the ssh server as the user that will use the passwordless authentication, in my case, it’s “surfer”. Install the public key into the server by issuing this command:

#:~$ cd .ssh
#:~/.ssh$ cat id_dsa.pub >> authorized_keys

The above commands will add the public key into authorized_keys file. It is important to do add the key by using the above command so that any existing keys in the authorized_keys file are not overwritten.

Sometime, the .ssh folder is not available, and you are required to create them manually. This may cause some problem regarding the access restriction of that folder. To create the folder, do the following:

#:~$ mkdir .ssh
#:~$ chmod -R 700 .ssh

5. Test your connection
Try to log to the target ssh server, the output should be close to this:

#:~/.ssh$ ssh root@192.168.1.2
--------------
Last login: Wed Oct 31 21:57:19 2008 from localhost
#:~$

Now you can test your setup by copying file from ssh client machine to the ssh server:

#:~$ scp test.tar.gz  user@192.168.1.2:/home/
test.tar.gz                        100%   33MB  11.2MB/s   00:03

… and you’re done

In Linux machines, the “top” command is one of the most frequently used commands to get information about your machine performance. It gives you information about your CPU usage, RAM usage and the load average of the machine. The load average represents the average system load over a period of time and in normal cases it should be under 1.0. You can get the load average by typing “top” or “uptime” in your shell and it will give you 3 numbers which represent the load in the last 1, 5 and 15 minutes. An IDLE CPU will result in 0 load, while a load of 2.5 represents that the CPU was overloaded by 150% in the last minute.

How do I diagnose the load average?

You can know whether your computer is dealing with a heavy load simply by noticing that it is slow and the response time is more usual. To find out what is causing the problem, you have to type the “top” command in your shell and focus on the first 3 lines of the output. The important thing for us in the 1st line is the load average; notice the 3 numbers. If they are between 0 and 5, then your computer is fine. If they are between 5 and 10, this is a high load, but if they are more than 10, then you have some problems that need more investigation. If your load is more than 10, you should now take a look at the 3rd line “cpu(s):” : this is an ideal view of the 3rd line in the “top” command Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st

1- check the the first 2 numbers as they represent the percentage of the total time the CPU is spending to process stuff. If they are always 99-100%, you have to upgrade your CPU.

2- check the 5th number as it represents the percentage of time a CPU takes in waiting for an I/O. If the number is above 90%, this means that the CPU is spending a lot of time waiting for I/O. This could be due to an HDD problem or a Network problem or an application trying to access data at a rate higher than what the network or HDD are designed for. To know what process is causing this problem, write this command in your shell: ” ps faux”. It will list all the running processes in your system. Have a look at the stat column and you should see letters like these: R = running S = sleeping D = waiting for something, so look for any process that has D and you can check it and see what is the problem with this process.

Sessions are a standard feature of web application servers, however the interoperability of sessions between different web application servers is generally limited. This talk introduces the mod_session modules collection in httpd v2.4 that attempts to create a unified session for httpd and web application servers. It is aimed at people interested in practical single sign-on, as well as for people trying to bring sanity to mixed architecture environments. We will start by introducing some of the history behind mod_session, why it was developed, and what problems mod_session is trying to solve. We will then continue by introducing what a session is within mod_session, and cover some of the ways in which a session might be created and where a session might be stored. We will then move on to cover the options available for securing sessions using encryption, and handling sessions on highly loaded or distributed environments. The focus will then shift to web applications and web application servers, and how they might read from and write to a session without having to care how the session is implemented or configured. We will then show some examples of how sessions might be used in practice, with examples in secure and in highly loaded environments. The talk will conclude by looking at some of the future development planned for mod_session, and how people can get involved in further development before httpd v2.4 is finally released.

ApacheCon Europe 2009

Reducing file sizes and removing unnecessary HTTP requests made to your server can not only reduce the tole on your bandwidth but also improve the user experience when using your site. Sites laden with with large high resolution images and complex flash animations are becoming common place on the world wide web as connection speeds continue to increase. Whilst this bring many benefits it also takes any attention away from developers optimizing their sites to provide the best possible service.

Web professionals have a tendency to assume that their users are using a 10Mb connection and will be able to download their 6MB flash intro video in mere seconds. Sadly this is not the case. Whether its because the user is running downloads, is in an area with bad signal quality or just has not yet upgraded to a fast modern connection there are various reasons why a person may be browsing the internet on a slow connection. The last thing as developers we want to do to that person is exclude them just because their connection is not as fast as it could be. That is why I have written these few quick easy tips on how to streamline your site using Apaches built in features to ensure that users access your site as quickly and easily as possible. These tips are all based on guidelines present by yahoo and implementing them all will give you a Y!Slow score of A, making it apparent that you care about users with slower connections.

Optimizing with the .htaccess file

There are a few simple lines which you can add to your .htaccess file on your site which will set Apache up to use these bandwidth saving features. If you do not have a .htaccess file at the moment or you are unsure of what it is simply create a text file called “.htaccess” (note no file extension) in the root of your site and Apache should detect it automatically. If this is not the case contact your system administrator for further details.

Header unset ETag
FileETag None

The above 2 lines when added to your .htaccess file will stop Apache sending ETags out with assets from your site. The ETag is used to track changes to a file on the server so they can be compared to the cache of the users web browser. Removing the ETags forces the browser to compare date changed values instead which is a much more accurate method of tracking changes to content.

Header set Expires "Thu, 15 Apr 2012 20:00:00 GMT"

The above lines dictate to the server to send out expiry headers with any files with the extensions dictated in the first line. By setting those expiry headers far in the future it stops the users browser from request these assets again if they are in the cache and therefore stops unnecessary HTTP requests. As a rule of thumb I would recommend not applying this rule to js and css files as these are most likely to change and could damage the user experience if they are changed but not re-downloaded. If you make any big changes to your site I would recommend disabling these few lines (by putting a # at the start of each line) so that your changes can be properly circulated until browser caches expire.

SetOutPutFilter DEFLATE

This last .htaccess option uses the Apache module DEFLATE to compress all text based output. Using this option means all files with a text mime-type are compressed using gzip and can be reduced in size by up to 80%. This option is vitally important, especially for very text heavy sites as it massively decreases the amount which a user has to download when accessing your site.

All the options stated above are all entirely server side changes and will not effect the user in any way other than quicker download times. All internet browsers are compatible with these modifications.

Minify your JavaScript on the Fly

Adding functionality to your site using JavaScript can be great but when you have finished you often end up with massive JavaScript files which are full of needless comments and white space. To combat this a man named Douglas Crockford wrote a small app called jsmin which strips out those unnecessary elements without damaging the file. Crockfords app is now available in many forms such as a windows .exe programs or Python, Ruby, Perl scripts which allow you to minify your code prior to run time. However, we are going to use the PHP version of jsmin to minify our JavaScript on the fly meaning we can keep our scripts in their true form without the user seeing them.

First you will need to download the PHP version of jsmin from Douglas Crockfords website (http://www.crockford.com/javascript/jsmin.html). Then you will need to create a small PHP script which will be referenced by the page as a JavaScript file when infact its a PHP script masquerading as a JavaScript file. In that file you will need to first send a header indicating the file is JavaScript, then include the jsmin library before finally echoing out the minified JavaScript. Here is an example:

Now simply point your page(s) to use this new script instead of page_script_full.js and you will make a massive saving on transfer times.