Context
I'm involved in the deployment of a new website for a local nonprofit, which was developed by local university students as part of their course work. They chose to use the C5 CMS system to build the website (http://www.concrete5.org/)
I was responsible for migrating the development website onto a production system, and tuning performance of the website. In the course of this process, I got a lot of help from the C5 community - in particular, this thread in the forums - http://www.concrete5.org/community/forums/chat/concrete-on-steroids/ )and also a big thanks to Shaun (AKA Phallanx, as per his handle in the thread/forums) who helped me out with final tuning, and getting the performance into great shape. (Please note also that his php-based website optimizer app, MISER, is available on SourceForge at the URL, http://sourceforge.net/projects/miser/ )
A few reference notes about the site I was debugging,
- The production server is a XenServer based VM hosted with Vexxhost in Montreal (http://vexxhost.com/cloud_hosting ) - I like their business, so I wanted to give them credit here :-). The VM in question is running CentOS 5.6, has stock YUM repos enabled by default, and has system resources of one vCPU (E5620 @ 2.40GHz); 1 gig RAM dedicated; 5 gigs disk (it is a modest sized website, after all) and an appropriate amount of monthly bandwidth - approx cost is $30/mo I think.
- Out of the box, CentOS is running PHP 5.1.6, which is below rev for 'ideal' operation of C5. I followed the steps outlined here: http://wiki.centos.org/HowTos/PHP_5.1_To_5.2 in order to painlessly get a newer version of PHP installed (5.2.10 - still not the latest and greatest, but a good step up).
- Otherwise, it is stock install for apache, mysql, gd, pear, etc.
Basic High-Level notes on tuning:
- Initial setup gave dreadful performance - initial page load times were horrible (10 seconds or longer). After the first page visit, subsequent pages performed better, but still it was not painless. Additionally, the admin pages (once logged in) all performed quite sluggishly.
- MySQL was tuned to reflect settings more consistent with "my.medium" template; and also with some adjustments as per the "SQLTuner" script. However, performance remained relatively unchanged. (see below for copy of the mysql config file)
- APC PHP cache was installed, and this helped performance, but things were still not great. APC install was pretty typical (via PECL) - for notes also see below.
- I experimented with C5 "cache" settings internally (via the dashboard) but had inconsistent results - ie - things didn't get much better, and sometimes it wasn't clear they were better; and after some posts in the forums suggestsing cache hindered performance - I ended up turning off internal cache settings entirely in C5.
- The biggest single performance boost came with turning on "KeepAlives" in the apache config. By default they were turned off. I subsequently adjusted the "KeepAlive Timeout" down from 15seconds to a more reasonable 5 seconds, to help alleviate what I understand to be one of the main concern with KeepAlives - allowing individual clients to monopolize too large a slice of your apache thread pool for tool long.
- I also installed the 3rd party add-on, miser, and it gave additional boosts to performance - and no negative impact - so an easy win.
- I experimented with "Google PageSpeed" (http://code.google.com/speed/page-speed/) instealled into apache as a module; this did appear to boost performance somewhat but had a few 'wrinkles' - in particular, some internal C5 editing behaviour stopped working properly (adding new block to a page - the spinner would time out // and in the httpd error log there was a line reading, "add_block_popup.php not found". Disabling pagespeed module made this issue disappear, so I have left pagespeed off.
The end-game config I'm using presently is:
- Keep-Alives enabled, 5 second timeout, other settings defaults
- APC installed
- C5 internal cache not enabled at all
- Miser installed
The result of all this? First-visit page loads have dropped from >10seconds to ~2 seconds or less. Repeat page loads have dropped from 0.85 to 0.5 seconds. Additionally, the admin interface (ie, once logged in to the C5 admin interface) has gone from "arrgh" slow to "perfectly usable and snappy".
Page speed tests for 'benchmarking' shown below were all performed using the website, http://www.webpagetest.org/ using default test site (Dulles, VA, DSL / IE7 client) (Thanks again to Shaun for pointing this one out to me - a great free resource for testing your website and getting solid metrics and good advice on where tuning is needed).
also note, metrics captured below are basic - as per the output from the testing site above; with
- (A) = doc complete score in seconds
- (B) = fully loaded score in seconds
Note that the bench scores below progress thus:
- Fully tuned / fastest
- gradually turning off tuning features, getting progressively slower
- then turning them back on, different order, to get better feel for 'relative value' of tune features
- Endpoint is same as start point, in theory
Note that since scores are slightly different for start,end it also illustrates that .. running the same test 2 times in a row can yield slightly different results .. and this is normal :-)
FASTEST:
Tuned, with miser and APC
-------------------------
(A) (B)
First view: 1.67s/2.23s
ReloadView: 0.55s/0.55s
without miser, with APC
-------------------------
(A) (B)
First view: 2.31s/2.31s
ReloadView: 0.65s/0.65s
without miser or apc
-------------------------
(A) (B)
First view: 3.30/3.30s
ReloadView: 0.85s/0.85s
ABSOLUTE WORST:
without miser,APC and with keepalives turned OFF
-------------------------
(A) (B)
First view: 17.85s/17.85s
ReloadView: 0.84s/0.84s
without keepalives or APC, but with Miser turned back on:
-------------------------
(A) (B)
First view: 17.09/20.43s
ReloadView: 0.87s/0.87s
without keepalives, but with APC and miser turned back on:
-------------------------
(A) (B)
First view: 19.52s/19.52s
ReloadView: 0.62s/0.62s
BACK WHERE WE STARTED:
with keepalives, APC and miser turned back on:
-------------------------
(A) (B)
First view: 1.38s/1.72s
ReloadView: 0.56s/0.56s
OTHER MISC REF INFO POSSIBLY OF INTEREST:
MYSQL: Config used for MySQL is as follows:
[root@VM etc]# cat my.cnf # # [mysqld] datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock user=mysql # # TDC added config as per my-medium stock my.cnf # april-18-11 skip-locking key_buffer = 16M max_allowed_packet = 1M table_cache = 512 sort_buffer_size = 512K net_buffer_length = 8K read_buffer_size = 256K read_rnd_buffer_size = 512K myisam_sort_buffer_size = 8M skip-bdb # ## SQL TUNER TWEAKS APR-26-11 TDC max_connections = 25 query_cache_size = 16M join_buffer_size = 256K tmp_table_size = 128M max_heap_table_size = 128M thread_cache_size = 8 innodb_buffer_pool_size =16M # ## END OF TDC PASTE CONFIG # old_passwords=1 # [mysqld_safe] log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid # #EOF
In this configuration, MySQL is gobbling up decent amount of ram, but nothing crazy, as illustrated by this TOP capture:
top - 16:20:42 up 7 days, 18:03, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 81 total, 2 running, 79 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1048752k total, 897932k used, 150820k free, 135712k buffers Swap: 1048568k total, 0k used, 1048568k free, 586232k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1356 mysql 15 0 155m 39m 4896 S 0.0 3.8 3:42.05 mysqld
APACHE Apache is using most of the ram in the VM, but we've still got a decent chunk left unallocated, so we seem to be running smoothly. Rough capture from top is shown below:
top - 16:20:42 up 7 days, 18:03, 2 users, load average: 0.00, 0.00, 0.00 Tasks: 81 total, 2 running, 79 sleeping, 0 stopped, 0 zombie Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem: 1048752k total, 897932k used, 150820k free, 135712k buffers Swap: 1048568k total, 0k used, 1048568k free, 586232k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1356 mysql 15 0 155m 39m 4896 S 0.0 3.8 3:42.05 mysqld 18918 apache 15 0 99.6m 24m 18m S 0.0 2.4 0:01.44 httpd 18917 apache 15 0 103m 23m 13m S 0.0 2.3 0:00.87 httpd 18921 apache 15 0 99.9m 22m 16m S 0.0 2.2 0:00.84 httpd 18920 apache 15 0 99.8m 22m 16m S 0.0 2.2 0:00.84 httpd 18924 apache 23 0 99.6m 21m 16m S 0.0 2.1 0:00.97 httpd 18919 apache 15 0 99.6m 21m 16m S 0.0 2.1 0:00.36 httpd 19006 apache 15 0 97.9m 20m 16m S 0.0 2.0 0:00.95 httpd 18922 apache 15 0 99.8m 16m 10m S 0.0 1.6 0:00.35 httpd 18923 apache 15 0 99.8m 16m 10m S 0.0 1.6 0:00.38 httpd 20166 apache 15 0 99.8m 16m 10m S 0.0 1.6 0:00.32 httpd 20167 apache 15 0 99.9m 15m 9.9m S 0.0 1.5 0:00.10 httpd 20165 apache 15 0 98.4m 12m 8200 S 0.0 1.2 0:00.02 httpd 18915 root 18 0 89296 7236 4304 S 0.0 0.7 0:00.01 httpd
Note the following adjustments that were made which impact apache config:
in /etc/http/conf/httpd.conf: KeepAlive On MaxKeepAliveRequests 100 KeepAliveTimeout 5 Other tweaks: in dir, /etc/httpd/conf.d/ [root@store conf.d]# more sitename.conf <Directory "/var/www/html"> # -- Expires Headers -- <ifModule mod_expires.c> ExpiresActive On ExpiresDefault "access plus 1 seconds" ExpiresByType text/html "access plus 1 seconds" ExpiresByType image/gif "access plus 2592000 seconds" ExpiresByType image/jpeg "access plus 2592000 seconds" ExpiresByType image/png "access plus 2592000 seconds" ExpiresByType text/css "access plus 604800 seconds" ExpiresByType text/javascript "access plus 216000 seconds" ExpiresByType application/x-javascript "access plus 216000 seconds" </ifModule> </Directory> MOD DEFLATE CONFIG: [root@store conf.d]# more mod_deflate.conf # # TDC May-5-11 # # Insert filter SetOutputFilter DEFLATE # Netscape 4.x has some problems... BrowserMatch ^Mozilla/4 gzip-only-text/html # Netscape 4.06-4.08 have some more problems BrowserMatch ^Mozilla/4\.0[678] no-gzip # MSIE masquerades as Netscape, but it is fine # BrowserMatch \bMSIE !no-gzip !gzip-only-text/html # NOTE: Due to a bug in mod_setenvif up to Apache 2.0.48 # the above regex won't work. You can use the following # workaround to get the desired effect: BrowserMatch \bMSI[E] !no-gzip !gzip-only-text/html # Don't compress images SetEnvIfNoCase Request_URI \ \.(?:gif|jpe?g|png)$ no-gzip dont-vary SetEnvIf Request_URI \.js no-gzip # Make sure proxies don't deliver the wrong content <IfModule mod_headers.c> Header append Vary User-Agent env=!dont-vary </IfModule>
Please note, these tweaks - for expires and compress for httpd - are directly copied from other resources found via google; I don't take any credit for them; but alas I'm having trouble finding the exact URLs right now as I wham this little document together; hence I am not putting it here - sorry.
APC NOTES:
Config for APC is tweaked thus:
[root@store conf.d]# cd /etc/php.d [root@store php.d]# more apc.ini extension=apc.so #apc.shm_size = 64 apc.shm_size=64M [root@store php.d]#
Note that install of APC is more or less stock, as per these basic steps:
yum install php-pear httpd-devel php-devel pcre pcre-devel then pecl install apc once done, create a file apc.ini approx similar to the sample provided above. reload apache if you want to observe APC cache hit performance, put a copy of apc.php into a 'secret dir' only known to you somewhere on your web server; this file is available as part of the APC distribution; for example you could grab it as follows: 492 cd /opt/src; mkdir apc 493 cd apc 494 wget http://pecl.php.net/get/APC 497 mv APC apc.tgz 498 gzip -d apc.tgz 499 tar xvf apc.tar 500 ls -la 501 cd APC-3.1.7/ 502 ls -la 503 cp apc.php /var/www/html/secret-dir-name-only-known-to-you-the-webmaster/ Then point your browser to the URL, http://host.name.your.site/secret-dir-name-only-known-to-you-the-webmaster/apc.php and you will see some happy stats back from APC if it is installed and working. On the website I've setup, I seem to see >95% cache hit rate with APC after a few visits on the site.
Finally a small footnote and disclaimer: This is here for my reference only; if you do this and your site blows up, or other various and sundry nasty things happen - you have nobody but yourself to blame for working on a production host, live, without backups, and for listening to advice that anyone like me is willing and able to give away for free :-)
Anyhow. I do hope this info helps someone out there, sometime.