The WordPress CMS service which has been growing rapidly and has seen a huge spike in traffic thanks to some 50+ campus sites coming on board (and one very popular site http://elearning.ubc.ca). Last year during the month of September the service had (79,821) page views fast forward a year and the service averaged (2,009,301) page views 25x increase!

WordPress CMS Service page views Sept 2009 – 2010

UBC Blogs which has moved out of the “pilot” mode is also increasing not as rapidly as the CMS service which is kind of a surprise we though it would be the opposite… In September 2009 UBC Blogs saw (158,654) page views in September 2010 (434,203) more than double.

UBC Blogs page views September 2009 – 2010

Even wiki.ubc.ca has seen a double in traffic as well. Thanks to quite a few courses taking a stab at developing their course in the wide open wiki.

Exciting times.

Why the Web is not going to die
That standard is HTML5, which Facebook plans to push very hard. If HTML5 can deliever a wide range of functions with a good user experience, then Facebook can develop one HTML5 version for every new product it launches and “that would be awesome,” Zuckerberg said. And HTML5 could combat the problem of unfair access raised by the great App Migration.

Already Facebook has launched a special mobile site for people in developing countries (and some developed ones). 0.facebook lets users access a speedy, data charge-free version of Facebook with a mobile browser from any basic phone even in parts of the world with poor infrastructure.

Until the devices that people use to access the internet are completely standardized, the Web will never die.

While at Google i/O this was obvious Google is really pushing HTML5 and I heard a few comments like wouldn’t it be great if you only had to write your mobile app once… It will happen and Apple will not like it.

10x fold? This is related to the CMS WordPress install we help maintain/develop with Public Affairs one site on that service is the reason for the traffic spike basically all logins/logouts to the campus LMS is directed through that site otherwise it would have been more or less business as usual for that service.

We did plan for this and load tested and we thought it could handle it but when we were hit with a huge flood of traffic because of a Vista outage we were crushed CPU was maxed out. *This is the reason why I love UBC IT VMs imagine being in this scenario with a locked in piece of hardware you would be hooped big time. We were able to contact UBC IT Infrastructure and they added a CPU literally on the fly! Do you know how cool that is? It’s very cool. Chris dropped the CPU in and the load came down the site was still sluggish but functional. VMs rock!

BUT… this was still very surprising to me to see it crumble on Tuesday. I have had a site hit by front page Digg and Reddit traffic which received considerably more traffic on a 7 dollar per month Webhost account back in 2006 and did not crash! I was simply using wp-cache 2 (before Super-Cache) a Cutline theme and one plugin! No APC no Super-Cache no MySQL Query Cache! It held up on shared host with hundreds of other users. I was for lack of a better word PO’d at our fail whale. To remedy we looked at all our code we disabled all plugins that were not being used cleaned up some code, made some tweaks to PHP and it seemed to help. The following night during the next big spike (which had even more users) it responded OK but still not as fast as I would like it to be.

Why did this fail? We did load testing right? I believe it is because our CLF theme/plugins are considerably more heavy than when we tested (there were a lot of code changes over the month). During the post crash testing we tested disabling and enabling widgets and showed you can 2x the number of page requests your page can handle which is significant (I am not sure what widgets were enabled during the initial test the site was still being actively developed). Another culprit is Domain Mapping which is used on most of the websites on the service. Domain Mapping runs each time a page is hit even when the page is cached via Super-Cache (check this file to see the query log of a hit to the cached page on elearning.ubc.ca) . Luckily we use MySQL Query cache so most of these would have been served from memory so it doesn’t change page requests/second by that much. But this really shows how much of a hog for resources WordPress can be if you let it and why you should do all the little things:

mysql> SHOW STATUS LIKE 'Qcache%';
+-------------------------+----------+
| Variable_name           | Value    |
+-------------------------+----------+
| Qcache_free_blocks      | 11524    |
| Qcache_free_memory      | 36261544 |
| Qcache_hits             | 50840927 |
| Qcache_inserts          | 2040382  |
| Qcache_lowmem_prunes    | 113918   |
| Qcache_not_cached       | 1857166  |
| Qcache_queries_in_cache | 24673    |
| Qcache_total_blocks     | 64344    |
+-------------------------+----------+
8 rows in set (0.00 sec)

I hope we think more critically about features we add to higher traffic pages in the future and think more new school in web development. New school as in performance matters and little changes on the app level front end / back end code can help more than old school adding a whack of hardware (ironic because we are doing/did this… but I argue that is so we can do upgrades with less downtime not as much performance).

*I think FB, Google, Yahoo! really promote the new school style well I would say they started it. FB basically re-tooled PHP and contributed to APC this probably saves them tons of $cash by not adding a boatload of new servers as well as make their users happy with faster response times. Yahoo! does a great job promoting front end performance the same with Google’s Let’s make the Web Faster. I like the way they think.

Before a few weeks ago we had one server and things were ok’ish but we planned on adding the BP for September along with a shiny CWL login button so it meant a lot more users might jump on board… so we added a couple more  servers a database Master/Slave setup with a few more to come (another web and a cache).

But… before the Master/Slave we did database “partitioning”, what is this besides something that made me more than nervous for a fair amount? Well because WPMU creates around 8 database tables per blog overtime this will grow to a very large number of tables things start to get slow during backups (very slow actually would take 10-30 minutes to do a backup, this is because it would have to lock up a large number of tables 21k+) performance also begins to lag reportedly once you move past the 1000+ blog range, so we needed to move to a “Mulit-database” setup to be exact which basically involves blowing apart your database and enabling a Multi-DB plugin. *There are a few Multi-DB plugins: Hyper-DB used by WordPress.com which is quite complicated and provides little to now help documentation or support. SharDB developed by Ron Rennick which is based on HyperDB and then there is Multi-DB which is developed by WPMU Premium Dev and used by Edublogs… I decided to go with the Multi-DB *for now* first choice would have been SharDB because of the fact it’s based on HyperDB and it is a community plugin vs the “premium”  but there were issues once the database moved to it’s own server (connection/timeout errors) it worked beautifully on the same box but on it’s own a no go. Ron mentioned this will be fixed in the future at which time I think we would switch.

Should be noted doing this step although nerve racking is relatively straight forward because of the great setup scripts that accompany both SharDB and Multi-DB and some great how to posts by Jim Groom I was able to figure this out and test during a Sakai presentation… implementation on the live took a fair amount of time approx 30 minutes (breaking 1 db into 256 dbs is slow). Once this was completed I have to say it felt faster (no empirical evidence) but doing things that were not so snappy prior like as an admin listing all users reported results much faster. I implemented the Master/Slave setup the following AM which is probably the coolest part because we have zero downtime with backups now and essentially a spare server… (if the master died we could flip to this quite easily). Here is how that was done *I am not a DBA like I said in the past so there maybe better ways but this seemed to work well so far*.

On the Master:
the my.cnf contains these settings related to master slave:

# for master/slave
server-id=1
relay-log=/mnt/mysql_log/logs/mysql-relay-bin
relay-log-index=/mnt/mysql_log/logs/mysql-relay-bin.index
master-info-file=/mnt/mysql_log/logs/mysql-master.info
relay-log-info-file=/mnt/mysql_log/logs/mysql-relay-log.info

# Enable binary logging
log-bin=/mnt/mysql_log/binary_logs/blogsdb1-bin
#Expire the logs every three days
expire_logs_days=3

Create a slave user on the master:

mysql> GRANT REPLICATION SLAVE ON
 *.* to slave_user@'SLAVE_IP_ADDRESS'
 IDENTIFIED BY 'slave_pw';

Dump the master db and import into the slave.

On the Slave include this in your my.cnf:

# for master/slave
# changes made to do slave
server-id=2
relay-log=/mnt/mysql_log/logs/mysql-relay-bin
relay-log-index=/mnt/mysql_log/logs/mysql-relay-bin.index
master-info-file=/mnt/mysql_log/logs/mysql-master.info
relay-log-info-file=/mnt/mysql_log/logs/mysql-relay-log.info

mysql> CHANGE MASTER TO
MASTER_HOST='MASTER_IP_ADDRESS',
MASTER_USER='slave_user',
MASTER_PASSWORD='slave_pw';

Start the slave:

mysql> start slave;

Check the status of the slave:

mysql> show slave status\G

If things worked out you should see:

Slave_IO_Running: Yes
Slave_SQL_Running: Yes

Try to run SHOW MASTER STATUS and SHOW SLAVE STATUS (quickly master first) you will see the Relay_Log_Pos will be the pretty much the same:

On master:

mysql> show master status\G ;
*************************** 1. row ***************************
File: blogsdb1-bin.000009
Position: 579567003
Binlog_Do_DB:
Binlog_Ignore_DB:
1 row in set (0.00 sec)

Exec_Master_Log_Pos should be close to the Position on master.

Now for backups just have your backup script run stop slave then dump and restart which gives no downtime for backups on production! Bloggers can post now at 3:00 am and not make passive aggressive tweets about blog DT.

From the YSlow documentation:

Web pages are becoming increasingly complex with more scripts, style sheets, images, and Flash on them. A first-time visit to a page may require several HTTP requests to load all the components. By using Expires headers these components become cacheable, which avoids unnecessary HTTP requests on subsequent page views. Expires headers are most often associated with images, but they can and should be used on all page components including scripts, style sheets, and Flash.

Tested with LEAP and with these 6 lines in your .htaccess (WP was having issues with the code reason why this is screened)

The site went from a

to a honor roll A score in YSlow.

Phase one has involved minor scaling at the application / server level. We implemented PHP opcode caching with APC, content caching using Donncha O Caoimh’s kick ass WP-Super-Cache and implemented some MySQL tuning techniques (what this post will be about).

Our Server

We have used a relatively modest server spec for the first two years 2 GB of RAM, 1 Intel(R) Xeon(R) CPU E5440 @ 2.83GHz with a 48GB SAN mount. This has seemed to work fine for the most part with almost 2000 users to date (these 2000 users are rarely writing to the DB 95% reads the main reason why this has been working) and modest traffic according to Webalizer averaging around 1.8 million page view/month. About a month ago the IT VM gods granted us an extra 2GB of ram bumping the total to 4GB which meant I could a little more MySQL tuning (you can’t really tune with minimal memory, with 2GB we were doing only query caching).

Warning

*Before I start, this tuning worked for the UBC Blogs setup you should not implement this unless you take the time to understand MySQL and each of the settings your are changing and also keep an eye on your app after implementing (CPU usage, response time etc). I am by no means a MySQL DBA (I worked with real whip smart DBA’s at the Bioinformatics Centre in the past I know what they are like and I frankly do not have the DB passion to memorize every error code, command, and the skills to model DB’s to handle multiple genomes etc.(real DBA stuff) but I have been using maintaining MySQL and building web apps based of the LAMP stack since 1999 (I am old) so I think I know enough to be get the work done (or enough to be dangerous) *.

So here are the tweaks that I found to work remember mileage may vary.

Before

Before you you begin tuning use some sort of tuning script to give you an idea of what’s going on under the hood. I used the mysqltuner.pl which is kind of old school so some of the recommends are not relevant but it gives you a good picture of what is going on. There are probably better ones out there but this has been in my toolbox for awhile.

Here is what it told me:

-------- Performance Metrics -------------------------------------------------
[--] Up for: 24d 17h 31m 13s (58M q [27.337 qps], 1M conn, TX: 2B, RX: 499M)
[--] Reads / Writes: 93% / 7%
[--] Total buffers: 59.0M global + 2.7M per thread (100 max threads)
[OK] Maximum possible memory usage: 327.7M (16% of installed RAM)
[OK] Slow queries: 0% (2K/58M)
[!!] Highest connection usage: 100% (101/100)
[OK] Key buffer size / total MyISAM indexes: 8.0M/80.5M
[OK] Key buffer hit rate: 95.6% (384M cached / 16M reads)
[OK] Query cache efficiency: 54.1% (26M cached / 49M selects)
[!!] Query cache prunes per day: 379131
[OK] Sorts requiring temporary tables: 0% (1K temp sorts / 4M sorts)
[!!] Temporary tables created on disk: 40% (3M on disk / 9M total)
[!!] Thread cache is disabled
[!!] Table cache hit rate: 0% (64 open / 3M opened)
[OK] Open file limit used: 0% (128/24K)
[OK] Table locks acquired immediately: 99% (30M immediate / 30M locks)


-------- Recommendations -----------------------------------------------------
General recommendations:
Add skip-innodb to MySQL configuration to disable InnoDB
Add skip-bdb to MySQL configuration to disable BDB
Run OPTIMIZE TABLE to defragment tables for better performance
Enable the slow query log to troubleshoot bad queries
Reduce or eliminate persistent connections to reduce connection usage
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries without LIMIT clauses
Set thread_cache_size to 4 as a starting value
Increase table_cache gradually to avoid file descriptor limits
Variables to adjust:
max_connections (> 100)
wait_timeout (< 28800)
interactive_timeout (< 28800)
query_cache_size (> 25M)
tmp_table_size (> 32M)
max_heap_table_size (> 16M)
thread_cache_size (start at 4)
table_cache (> 64)

So some of the stuff is not useful as I said some is. I will go thought what I changed based on the recommends.

Optimize Tables

No brainer you have to do this every now and then. Pretty much the same as defragging your disk file system, it’s also important to do the same thing with MySQL data tables. If you don’t you may end up with slow or corrupted tables over time.

*I read on a blog post once someone wrote a php script to loop through all their databases and optimize the tables, that is a perfectly good waste of coffee drinkin’ time! Let MySQL do the work for you!

run this command:

mysqlcheck -u root -p --auto-repair --check --optimize --all-databases


*Note run this in off hours it does slow things down because it locks tables and if you have a large WPMU there will be a lot of MySQL tables to lock so it will take some time.

Query Caching

Like I said we were doing this in the past now with a little more RAM we could bump it up.
* Too much here is not a good thing the max recommended is 512MB .

From MySQL: “A Practical Look at the MySQL Query Cache” –

The MySQL query cache provides substantial benefits over other database engine caching strategies because not only is the overhead of hard parsing for identical queries avoided, but the sometimes very high overhead of recreating complex result sets from either disk or memory caches is averted as well, greatly lessening both physical and logical I/O. The end results are extremely fast response times for business applications (data warehouses, BI applications, Web applications, OLTP systems, etc.) where end user clients are repetitively executing the same MySQL queries.

I bumped up the query_cache_size up to 128M which has captured a lot of queries that would normally go to disk this is represented by the Qcache_hits 31758409 and is storing a fair amount of queries in memory as well with Qcache_queries_in_cache at 26371 .


mysql> SHOW GLOBAL STATUS LIKE '%qcache%';
+-------------------------+----------+
| Variable_name | Value |
+-------------------------+----------+
| Qcache_free_blocks | 9566 |
| Qcache_free_memory | 28278048 |
| Qcache_hits | 31758409 |
| Qcache_inserts | 10434728 |
| Qcache_lowmem_prunes | 3865108 |
| Qcache_not_cached | 2203713 |
| Qcache_queries_in_cache | 26371 |
| Qcache_total_blocks | 68193 |
+-------------------------+----------+
8 rows in set (0.00 sec)

Thread Cache Size

This was at the default which was 0 which is bad because this affects the thread cache rate.

Thread_cache_size determines how many threads MySQL will hold open in memory to handle new connections. The hit rate should be as close to 100% as possible. You can calculate your hit ratio by dividing the ‘threads_created’ status variable by the ‘connections’ status variable:

100 - ((Threads_created / Connections) * 100)

For UBC Blogs the thread cache rate was:

100 – ((1629684 / 1629684 ) * 100) = 0% == very bad. This is because the thread_cache_size was set to the default 0. MySQL was basically holding 0 threads in cache so it was forced to create new threads all the time.

I set the thread_cache_size to 16 to see if it helps (* You should start at a low value and move up).

Checking after a ~ couple weeks of running

mysql> SHOW GLOBAL STATUS LIKE 'Connections';
+---------------+---------+
| Variable_name | Value |
+---------------+---------+
| Connections | 1315861 |
+---------------+---------+
1 row in set (0.00 sec)



mysql> SHOW GLOBAL STATUS LIKE 'Threads_created';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| Threads_created | 3057 |
+-----------------+-------+
1 row in set (0.00 sec)


Now doing the calculation again:

100 – ((3057 / 1315861 ) * 100 ) = 99.76% which is a huge improvement.

Table cache

Every time MySQL accesses a table, it places it in the cache. If the application accesses many tables like WordPress MU, it is faster to have more of these in the cache.

We had it set to the default 64.

mysql> SHOW GLOBAL STATUS LIKE 'Opened_tables';
+---------------+---------+
| Variable_name | Value |
+---------------+---------+
| Opened_tables | 3948355 |
+---------------+---------+
1 row in set (0.00 sec)

30 minutes later:

mysql> SHOW GLOBAL STATUS LIKE 'Opened_tables';
+---------------+---------+
| Variable_name | Value |
+---------------+---------+
| Opened_tables | 3955277 |
+---------------+---------

You can see the number go up and up. Increasing the table cache size will help a lot I doubled this to 128 to start will see how it works out.

Key buffer size

From MySQL:

You can increase the value to get better index handling for all reads and multiple writes; on a system whose primary function is to run MySQL using the MyISAM storage engine, 25% of the machine’s total memory is an acceptable value for this variable. However, you should be aware that, if you make the value too large (for example, more than 50% of the machine’s total memory), your system might start to page and become extremely slow.
You can increase the value to get better index handling for all reads and multiple writes; on a system whose primary function is to run MySQL using the MyISAM storage engine, 25% of the machine’s total memory is an acceptable value for this variable. However, you should be aware that, if you make the value too large (for example, more than 50% of the machine’s total memory), your system might start to page and become extremely slow.

The Key_reads/Key_read_requests ratio should normally be less than 0.01. The Key_writes/Key_write_requests ratio is usually near 1 if you are using mostly updates and deletes, but might be much smaller if you tend to do updates that affect many rows at the same time or if you are using the DELAY_KEY_WRITE table option.

Here is what we were getting:

mysql> SHOW STATUS LIKE 'Key_%';
+------------------------+-----------+
| Variable_name | Value |
+------------------------+-----------+
| Key_blocks_not_flushed | 0 |
| Key_blocks_unused | 6405 |
| Key_blocks_used | 3166 |
| Key_read_requests | 386078455 |
| Key_reads | 16844790 |
| Key_write_requests | 3127706 |
| Key_writes | 1148023 |
+------------------------+-----------+
7 rows in set (0.00 sec)

key_read / key_reads_requests
Ours is over 4.36 which is not so good.

key_write / key_writes_requests = 0.36

The default key_buffer is 64Mb moved this up to 128MB to start.

A month later checking:


mysql> SHOW STATUS LIKE 'Key_%';
+------------------------+-----------+
| Variable_name | Value |
+------------------------+-----------+
| Key_blocks_not_flushed | 0 |
| Key_blocks_unused | 5994 |
| Key_blocks_used | 4059 |
| Key_read_requests | 311890660 |
| Key_reads | 15189451 |
| Key_write_requests | 3943721 |
| Key_writes | 1743544 |
+------------------------+-----------+
7 rows in set (0.00 sec)


key_read / key_reads_requests
Ours is over 4.87 which is not good even after moving up to 128MB, until we move to a dedicated MySQL server 128MB is the max we can set it at to be on the safe side. I think the fact this is off maybe due to the fact that there is a huge number of tables in a WordPress MU install with a significant number of users.

key_write / key_writes_requests = 0.44 up a bit.

That is it for now.

Benchmarking

Doing a Apache Bench web load test before and after with a search on the UBC Blogs homepage we get:

AB Testing a dynamic page **before** upgrade


[scott@rab ~]$ ab -n 1000 -c 100 http://blogs.ubc.ca/?s=outage


Concurrency Level: 100
Time taken for tests: 4.350686 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 16953480 bytes
HTML transferred: 16544000 bytes
Requests per second: 229.85 [#/sec] (mean)
Time per request: 435.069 [ms] (mean)
Time per request: 4.351 [ms] (mean, across all concurrent requests)
Transfer rate: 3805.38 [Kbytes/sec] received


same page after adding the RAM and tuning


[scott@rab ~]$ ab -n 1000 -c 100 http://blogs.ubc.ca/?s=outage


Concurrency Level: 100
Time taken for tests: 2.144289 seconds
Complete requests: 1000
Failed requests: 0
Write errors: 0
Total transferred: 16953000 bytes
HTML transferred: 16544000 bytes
Requests per second: 466.36 [#/sec] (mean)
Time per request: 214.429 [ms] (mean)
Time per request: 2.144 [ms] (mean, across all concurrent requests)
Transfer rate: 7720.51 [Kbytes/sec] received


You can see the Time per request was cut in half shows what some extra RAM and a little MySQL tuning can speed things up mostly the RAM helped in this case…

Still more to do but on the right track. MySQL and databases in general are not really out the box get up and go ready, you have to do some tuning to get the most out of the install and tweak for your app, all part of the fun I guess…

I recommend skimming through: “High Performance MySQL: Optimization, Backups, Replication, and More” contains some useful stuff.

This is an interesting graphic from Pingdom our response times have dropped in half since Jan 27, 2010 . There could be many contributing factors network activity, server load etc for this.

I think the real reason was we dropped some mu plugins on that date:

WPMU-Topposts was a total hog. I noticed a drop in CPU use immediately upon removing. It was basically writing to the db every time a page was opened and sorting several million records to determine which blog was popular, a cool feature on the homepage to show the most popular blogs but who cares if it is making the server sluggish…

Or it could have been a magical network change, who knows as long as the curve is not going up that quickly I am happy.

Now with the upcoming WordPress 3.0 blogs will be referred to as sites! Although just a word I really like the change, WordPress is much more than a blog platform it can be used as a CMS, portfolio, lightweight LMS (from what I hear…), news magazine etc. The name change will answer the “I thought WordPress was for blogging”…

I guess In the future I will not be a “Site Admin” but a “Network Admin” just don’t ask me to debug your Cisco router

WordPress 3.0

**We better have a meeting about this to discuss how to best inform all users because it will be very confusing to them and many will think their blogs disappeared.** (I am joking but in reality it seems as if academic users are not the most adaptive users out there).

Also new in 3.0 is the default theme Twenty Ten, much more flexible and clean, with most users using the default theme this is a good thing. Things I wish it also included:

• Font Choice
• Color Selection
• Column Layout Choices
• **Custom CSS upload**

With those options that would satisfy 95% of user needs on our system.

Here is a screenshot of the default theme Twenty Ten with a custom background and header image. Super patriotic Eh!

Here is the video in the screen because it is cool.

Googles description of the tool:

Speed Tracer is a tool to help you identify and fix performance problems in your web applications. It visualizes metrics that are taken from low level instrumentation points inside of the browser and analyzes them as your application runs. Speed Tracer is available as a Chrome extension and works on all platforms where extensions are currently supported (Windows and Linux).

Using Speed Tracer you are able to get a better picture of where time is being spent in your application. This includes problems caused by JavaScript parsing and execution, layout, CSS style recalculation and selector matching, DOM event handling, network resource loading, timer fires, XMLHttpRequest callbacks, painting, and more.

**It is supported for OSX as well they need to update the description**

Example usage:

Using LEAP as an example again because it is a total pig of a site, we can see where Speed Tracer picks up the sluggishness.

Sluggishness is designated by events taking longer than 100ms which is generally when latency is noticed. Starting at 3.13s we had one of these events on the LEAP homepage.

Looking at the Network activity for that range you can see we have some calls to Facebook and Google Analytics which are slowing things down.

I love tools like this because they are great for answering the “Why is my blog so slow?” If the user is blaming the server (and you know it is not the server) and know it’s their shoddy web design send them some screen shots from Speed Tracer showing them their bottle necks.

Google Speed Tracer Intro:

About WP-Minify:

WP Minify intercepts scripts and style printing at the ‘wp_print_scripts’ and ‘wp_print_styles’ hook. WP Minify grabs these files in the proper order (minding dependencies) and passes that list to the Minify engine. The Minify engine then returns a consolidated, minified, and compressed script or style. WP Minify then references this compressed script or style in the WordPress header instead of each individual scripts/styles.

The way most themes and plugins are developed has always bugged me so I was happy to see this plugin released. Most WordPress sites have way to many HTTP requests, if you are going to be hosting a large number blogs/sites you have to start to think about details like this. For example say you have 500 sites on your system and the sites use the same set of plugins and each site is averaging 200 unique visitors/day with an average of 20 HTTP requests to JS and CSS files that is 1+ million HTTP requests (I am using the assumption that all users have a browser cache enabled or else it could be more). This could be greatly reduced in many cases using a plugin like WP-Minify .

I decided to take a poke at checking HTTP requests to LEAP on verf, LEAP was developed with the HyBrid theme and uses around a dozen plugins. I used Yslow and Web Inspector to check HTTP requests and load times.

Before Minify

HTTP Requests

Load time before Minify

After Minify with empty cache

HTTP Requests

*still high (using the default setting) with WP-Minify you can go in and add files that are not being picked up. You can see a big reduction in JS HTTP request calls. **LEAP may not be the best site to use as an example because of the calls to 3rd parties.

Load time after Minify

Almost cut in half! Over half as many HTTP requests for JS files is the key.

From the Yahoo! Developer Performance rules:

Reducing the number of HTTP requests in your page is the place to start. This is the most important guideline for improving performance for first time visitors. As described in Tenni Theurer’s blog post Browser Cache Usage – Exposed! , 40-60% of daily visitors to your site come in with an empty cache. Making your page fast for these first time visitors is key to a better user experience.

It would be great is something like WP-Minify was included in the WordPress core but until then this plugin is a great start. Would also help if plugin and theme developers did not include redundant JS in their plugins that already exists in the core!

We will be taking a closer look at WP-Minify and hopefully roll it out to all the sites on blogs@ubc and the CMS hosting service.

Time to step it up and help make the web a faster place!

After Michael tortured himself trying to write mod_rewrite rules to match the conditions along with writing a custom 404 redirect I decided to do a Google “WordPress Redirection” and low and behold Redirection a little rockstar of a plugin popped up first hit. This plugin has pretty much every feature you would need 301, pass through abilities, logging (great to see who is linking to the old pages still) and CSV bulk import (which was super useful because Michael had all the links in a spreadsheet), .htaccess rule export…

**This is a good lesson to all developers, somebody has probably done what you are going to start work on so do a search first especially when working with WordPress!

I give this plugin 5 stars not a sexy plugin but very useful for  those moving an old website to WP. Probably worth adding some bills into the developers PayPal account.

Please note that all contributions to UBC Wiki may be edited, altered, or removed by other contributors. If you do not want your writing to be edited mercilessly, then do not submit it here. You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource (see UBC Wiki:Copyrights for details). Do not submit copyrighted work without permission!

To change this text go to MediaWiki:Copyrightwarning2 and edit the text. We removed the “mercilessly” better would be adding a Copyright notice like Wikipedia which we do not use…

Content that violates any copyrights will be deleted. Encyclopedic content must be verifiable. You irrevocably agree to release your contributions under the CC-BY-SA 3.0 License and the GFDL. You agree that a hyperlink or URL is sufficient attribution under the Creative Commons license. See the Terms of Use for details.

We have have been experimenting with ways to get content into YouTube from a single app with a trusted user base. So that meant either MediaWiki or WordPress (only two apps we have with CWL right now). WordPress was the first choice but there was nothing out there, eventually found this great little extension from Travis Derouin of http://www.wikihow.com called YouTubeAuthSub which takes advantage of the YouTube AuthSub/ClientLogin features.

After quite a few email exchanges with Travis I managed to get it working (discovered someone mangled the SVN on MediaWiki.org hosting his extension without his knowledge) ended up finding a working copy from the Wikia SVN in the mean time…

Recommends

You probably do not want all users in your wiki to access this extension so I recommend creating a YouTube group on your wiki install and assigning users to that group through the Special:UserRights . To add a group to MediaWiki add something like this to your LocalSettings.php:


## YouTube Namespace
define("NS_YOUTUBE", 116);
define("NS_YOUTUBE_TALK", 117);

$wgContentNamespaces[] = NS_YOUTUBE;


You will also need to add this to the extension YouTubeAuthSub/YouTubeAuthSub_body.php after the $this->setHeaders(); in the function execute()


# Check permissions
/// OLT BEGIN ///
// Make sure the user is in the youtube OR sysop group
$isYouTubeEduGroup = ( in_array('youtube', $wgUser->getGroups()) || in_array('sysop', $wgUser->getGroups()) );
if(!$isYouTubeEduGroup ) {

Possible Bugs

We recently had issues with some accounts because of a change in the YouTube API which they did not warn users against! They basically decided to add “=” into some of the tokens. The original code used a split() to extract tokens so this broke. I recommend doing something really ugly like this to avoid failed tokens if you are having issues (for some reason not all accounts have issues).

Look for the foreach ($lines as $line) { in the YouTubeAuthSub_body.php and add and comment the following:


/// OLT BEGIN ///
// Check if string contains Auth
$authPat = '/^Auth/';
preg_match($authPat,$line,$authMatch);
if(!empty($authMatch)){
$token = substr($line, 5);
var_dump($token);
}

Other change YouTube change the client login URL look for the:

$result = wfSpecialYouTubePost("https://www.google.com/youtube/accounts/ClientLogin?"

and change to:

$result = wfSpecialYouTubePost("https://www.google.com/accounts/ClientLogin?"


To Do
Add ability to add to playlists within the YouTube account, not sure when I will get to this but it is a “to do” maybe someone will to do it first

That’s it.

Now with the new Add User site admins can bulk add users either by email or Student/Staff/Faculity ID. By email users receive an invite similiar to the default WordPress Add User if they are in the system they will be added, if not they will be directed to sign up. If the admin uses ID’s they will have to tell the user ahead of time to sign up for an account (I expect some problems with this) once the user logs in for the first time they will be added.

I know there are bulk add user plugins out there already but they would not work with the CWL and to be honest I think this Add User functionality is much slicker.

Also we are not connected to the SIS (probably never will be) so this is the next best thing at the moment. Special thanks to OLT super programmers Pan Luo and Godfrey Chan for working on this. This was Pan’s first time working with WordPress and he did a great job implementing the new CWL features, Godfrey showed him the ropes in WP plugin development which he is pretty much a master of. The Selenium QA tests they developed for this also make me very happy

UBC Blogs in it’s current state (using WordPress MU) has been online since September 2008 the OLT has been hosting blogs for many years prior on a Moveable Type platform. We are currently officially a pilot which means (at least the way I interpret it) we cannot really advertise the service and we hide the account setup/login page… That being said we still have a decent number of users and blogs probably 10% of the user base was migrated over from the old Moveable Type platform the rest are newbies. We have a another server for more VIP sites (basically means mapped domains and custom themes) which sees about the same traffic with only a couple dozen sites.

User and Blogs Growth
This chart is interesting shows the user account growth chart in the system overtime 1438 users and 795 blogs.

Blogs growth (click for large)

Content Growth.
We ran this a few weeks ago on verf(too much of a hog to run on production) to check the total posts and comments in the system. These numbers go way back to 2004 because some of the old MT blogs were moved into the new system. Still pretty impressive numbers with over 24,000 posts. I am pretty sure this has to be one of the larger websites on campus in terms of number of hosted pages.

Traffic
We haven’t been using Google Analytics since the start but here are some numbers from this term so far.

Sept-Nov 2009 stats

Bounce rate is not the greatest 60% of people leave after visiting one page.

How are people arriving?

Top 10 URL’s this term

Interesting that four of the blogs are courses: digitalliteracy2009, etec522sept09, etec540sept09 and micb405 (private).

Where are people coming from? Top 10 visiting countries.

From our web traffic and content growth it seems that this campus “publishing platform” is alive and well and growing at a healthy rate despite being on the down low.

On further investigation, looking at Novak’s blog he had reCapthca off for some reason (not so Super Novak)… after he turned it on spam went down a little but it was still up big time. Next step was looking at the server logs at first I thought we could use some .htaccess trickery to block the bots from hitting the wp-comments-post.php


RewriteEngine On
RewriteCond %{REQUEST_METHOD} POST
RewriteCond %{REQUEST_URI} .wp-comments-post\.php*
RewriteCond %{HTTP_REFERER} !.*blogs.ubc.ca.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^$
RewriteRule (.*) http://%{REMOTE_ADDR}/$ [R=301,L]


But the bot was smart enough to send proper referrals so this would not work. What to do next? I started grepping through the server logs to see what the bots were hitting on their blogs both were being hit by old posts (although both do not post that often so almost everything is old) but I mean old like over 3 months old. Both had a lot of spam being directed at post on a “Forum like display plugin” post as well as a couple others. So on both blogs I set turn off comments after 90 days (Novak) and 120 days (Brian) and I disabled trackbacks. This pretty much had an immediate impact spam was virtually stopped on both blogs. In conclusion this is a temporary fix I recommend everyone having spam issues right now should do this (I always recommend disabling trackbacks).

This definitely is a bigger issue if bots can spam and bypass both reCaptcha and Akismet (in some cases) WordPress has a fairly serious security issue hopefully WordPress addresses this soon.

A simple, open, server-to-server web-hook-based pubsub (publish/subscribe) protocol as an extension to Atom (and RSS).

Parties (servers) speaking the PubSubHubbub protocol can get near-instant notifications (via webhook callbacks) when a topic (feed URL) they’re interested in is updated.

The thing I like about this is in a “Feed WordPress” scenario. Instead of polling the same system a Hub Server could be polled taking some of the strain of the publishing WPMU server and also allowing faster content updates… I think Feed WordPress would need some changes but seems doable.

There is already a WordPress Plugin that supports the protocol…

PubSubHubbub Demo – Real-Time CrunchUp

Timber
http://www.blurbia.com/plugins/timber/

Timber is intended primarily as a developer’s tool, to be enabled in a sandbox environment while debugging plugins or themes in-progress. However, it also has potential in a production setting, as a way to hide errors from the end-user while still logging them for administrator review. The configuration settings allow for precise control of what type of errors and what error information is logged, so it can be deployed in varying scenarios without runaway logs filling the database.

Synected
http://www.blurbia.com/plugins/synected/

Synected enables shortened URL creation on your own blog. Rather than relying on services such as tinyurl.com or bit.ly, Synected lets you easily create and use short links based off your own domain. This releases you from dependence on a third party — a server error on tinyurl no longer prevents visitors from reaching your site. In addition, it strengthens your brand, keeping your domain name in view of your audience even on Twitter and other micro-communication platforms.

These guys develop some pretty tight themes also:
http://demo.blurbia.com/melora/ Fully customizable. With sites like this, plus the other framework themes we are getting very close to not having to design themes anymore… which is great because you will be able to role out a site in a day vs weeks.