Holy HTTP requests! Our WordPress themes and plugins are HTTP hogs.

Posted on February 3, 2010 by Scott McMillan

This was inspired by todays post on WPMU.org with regards to WP-Minify .

About WP-Minify:

WP Minify intercepts scripts and style printing at the ‘wp_print_scripts’ and ‘wp_print_styles’ hook. WP Minify grabs these files in the proper order (minding dependencies) and passes that list to the Minify engine. The Minify engine then returns a consolidated, minified, and compressed script or style. WP Minify then references this compressed script or style in the WordPress header instead of each individual scripts/styles.

The way most themes and plugins are developed has always bugged me so I was happy to see this plugin released. Most WordPress sites have way to many HTTP requests, if you are going to be hosting a large number blogs/sites you have to start to think about details like this. For example say you have 500 sites on your system and the sites use the same set of plugins and each site is averaging 200 unique visitors/day with an average of 20 HTTP requests to JS and CSS files that is 1+ million HTTP requests (I am using the assumption that all users have a browser cache enabled or else it could be more). This could be greatly reduced in many cases using a plugin like WP-Minify .

I decided to take a poke at checking HTTP requests to LEAP on verf, LEAP was developed with the HyBrid theme and uses around a dozen plugins. I used Yslow and Web Inspector to check HTTP requests and load times.

Before Minify

HTTP Requests

Load time before Minify

After Minify with empty cache

HTTP Requests

*still high (using the default setting) with WP-Minify you can go in and add files that are not being picked up. You can see a big reduction in JS HTTP request calls. **LEAP may not be the best site to use as an example because of the calls to 3rd parties.

Load time after Minify

Almost cut in half! Over half as many HTTP requests for JS files is the key.

From the Yahoo! Developer Performance rules:

Reducing the number of HTTP requests in your page is the place to start. This is the most important guideline for improving performance for first time visitors. As described in Tenni Theurer’s blog post Browser Cache Usage – Exposed! , 40-60% of daily visitors to your site come in with an empty cache. Making your page fast for these first time visitors is key to a better user experience.

It would be great is something like WP-Minify was included in the WordPress core but until then this plugin is a great start. Would also help if plugin and theme developers did not include redundant JS in their plugins that already exists in the core!

We will be taking a closer look at WP-Minify and hopefully roll it out to all the sites on blogs@ubc and the CMS hosting service.

Time to step it up and help make the web a faster place!

Redirecting pages in WordPress

Posted on December 15, 2009 by Scott McMillan

Have you ever tried to redirect over a 150 URLs to their new locations? That was a challenge faced with the launch of the new OLT website.

After Michael tortured himself trying to write mod_rewrite rules to match the conditions along with writing a custom 404 redirect I decided to do a Google “WordPress Redirection” and low and behold Redirection a little rockstar of a plugin popped up first hit. This plugin has pretty much every feature you would need 301, pass through abilities, logging (great to see who is linking to the old pages still) and CSV bulk import (which was super useful because Michael had all the links in a spreadsheet), .htaccess rule export…

**This is a good lesson to all developers, somebody has probably done what you are going to start work on so do a search first especially when working with WordPress!

I give this plugin 5 stars not a sexy plugin but very useful for those moving an old website to WP. Probably worth adding some bills into the developers PayPal account.

Changing the MediaWiki edit warning

Posted on December 11, 2009 by Scott McMillan

This was brought to my attention by Jeff Miller. MediaWiki uses some harsh default Terms of Service warning text on the Edit page which to be honest I never noticed but I do understand how it might sketch out a user taking the plunge into the open wiki world:

Please note that all contributions to UBC Wiki may be edited, altered, or removed by other contributors. If you do not want your writing to be edited mercilessly, then do not submit it here. You are also promising us that you wrote this yourself, or copied it from a public domain or similar free resource (see UBC Wiki:Copyrights for details). Do not submit copyrighted work without permission!

To change this text go to MediaWiki:Copyrightwarning2 and edit the text. We removed the “mercilessly” better would be adding a Copyright notice like Wikipedia which we do not use…

Content that violates any copyrights will be deleted. Encyclopedic content must be verifiable. You irrevocably agree to release your contributions under the CC-BY-SA 3.0 License and the GFDL. You agree that a hyperlink or URL is sufficient attribution under the Creative Commons license. See the Terms of Use for details.

Delete old revisions in MediaWiki

Posted on December 11, 2009 by Scott McMillan

A course instructor just asked if he could delete revision histories so he could reuse group project pages in his wiki without exposing the previous history to the next group of users. Generally deleting anything in a MediaWiki is quite difficult, I knew this may have been possible via a command line maintenance script DeleteOldRevisions.php but this was not the greatest solution (who wants to be bothered each term to run this) after a Google found this great extension SpecialDeleteOldRevisions2 which did the trick and then some . Allows wild card revision deletes along with date range revision deletes. Very handy for any stand alone course wiki installs.

Direct upload to YouTube through MediaWiki

Posted on December 8, 2009 by Scott McMillan

Might be of interest for those who want to get more content into their YouTube EDU account.

We have have been experimenting with ways to get content into YouTube from a single app with a trusted user base. So that meant either MediaWiki or WordPress (only two apps we have with CWL right now). WordPress was the first choice but there was nothing out there, eventually found this great little extension from Travis Derouin of http://www.wikihow.com called YouTubeAuthSub which takes advantage of the YouTube AuthSub/ClientLogin features.

After quite a few email exchanges with Travis I managed to get it working (discovered someone mangled the SVN on MediaWiki.org hosting his extension without his knowledge) ended up finding a working copy from the Wikia SVN in the mean time…

Recommends

You probably do not want all users in your wiki to access this extension so I recommend creating a YouTube group on your wiki install and assigning users to that group through the Special:UserRights . To add a group to MediaWiki add something like this to your LocalSettings.php:

## YouTube Namespace define("NS_YOUTUBE", 116); define("NS_YOUTUBE_TALK", 117);


$wgExtraNamespaces[NS_YOUTUBE] = "YouTube";

$wgExtraNamespaces[NS_YOUTUBE_TALK] = "YouTube_talk";
## Subpages has to be enabled explictly.

$wgNamespacesWithSubpages[NS_YOUTUBE] = false;

$wgNamespacesWithSubpages[NS_YOUTUBE_TALK] = false;

$wgContentNamespaces[] = NS_YOUTUBE;

You will also need to add this to the extension YouTubeAuthSub/YouTubeAuthSub_body.php after the $this->setHeaders(); in the function execute()

# Check permissions /// OLT BEGIN /// // Make sure the user is in the youtube OR sysop group $isYouTubeEduGroup = ( in_array('youtube', $wgUser->getGroups()) || in_array('sysop', $wgUser->getGroups()) ); if(!$isYouTubeEduGroup ) {


                        if( !$wgUser->isLoggedIn() ) {

                                $wgOut->showErrorPage( 'uploadnologin', 'uploadnologintext' );

                        }else{

                                if(!$isYouTubeEduGroup){

                                        $wgOut->showErrorPage( 'badaccess', 'badaccess-groups', $result = array('YouTube') );

                                }

                        }

                        return;

                }

                /// OLT END ///

Possible Bugs

We recently had issues with some accounts because of a change in the YouTube API which they did not warn users against! They basically decided to add “=” into some of the tokens. The original code used a split() to extract tokens so this broke. I recommend doing something really ugly like this to avoid failed tokens if you are having issues (for some reason not all accounts have issues).

Look for the foreach ($lines as $line) { in the YouTubeAuthSub_body.php and add and comment the following:

/// OLT BEGIN /// // Check if string contains Auth $authPat = '/^Auth/'; preg_match($authPat,$line,$authMatch); if(!empty($authMatch)){ $token = substr($line, 5); var_dump($token); }


                                        // Check if string contains YouTubeUser

                                        $userPat = '/^YouTubeUser/';

                                        preg_match($userPat,$line,$userMatch);

                                        if(!empty($userMatch)) {

                                          $YouTubeUser = substr($line, 12);

                                        }
                                        /// OLT END ///
                                        /**

                                         ** This does not work. YouTube API decided to add = into token strings so split breaks.
                                        $params = split("=", $line);
                                        switch ($params[0]) {

                                                case "Auth":

                                                $token = $params[1];

                                                var_dump($token);

                                                break;

                                                case "YouTubeUser":

                                                $YouTubeUser = $params[1];

                                                break;

                                        }
                                        **/

Other change YouTube change the client login URL look for the:
$result = wfSpecialYouTubePost("https://www.google.com/youtube/accounts/ClientLogin?"
and change to:
$result = wfSpecialYouTubePost("https://www.google.com/accounts/ClientLogin?"

To Do
Add ability to add to playlists within the YouTube account, not sure when I will get to this but it is a “to do” maybe someone will to do it first

That’s it.

Improved add user feature on UBC Blogs

Posted on November 27, 2009 by Scott McMillan

Today we implemented probably my favorite new feature on UBC Blogs. It’s not a sexy plugin like the Section Widget which is very cool and useful for someone trying trying to make a WordPress/CMS , but is is a VERY useful and VERY MUCH needed Add User feature on blogs.ubc.ca. If anyone has tried to add more than one user to a blog on our setup they know what I am talking about, for those that haven’t here was the problem. There was a huge bug in our system if someone tried to add a user to a blog that was not already in the system the user would most often not end up in the blog, the problem resulted from the CWL layer and a username/email mismatch. They could add a user to the blog IF they read the instructions on how to add a user to a blog properly on our system (basically ensure the user is in the system then add the exact username and email they signed up with) sounds easy? Wrong 95% of people do not read instructions doesn’t matter how many PhD’s they have nobody RTFM anymore they just fill out forms and click. So there has been ALOT of support issues especially at the beginning of the term and quite a few unhappy users. We told users about the “Add User Sidebar Widget” to bypass these issues which worked for some but was still too many steps for most to comprehend (amazes me really).

Now with the new Add User site admins can bulk add users either by email or Student/Staff/Faculity ID. By email users receive an invite similiar to the default WordPress Add User if they are in the system they will be added, if not they will be directed to sign up. If the admin uses ID’s they will have to tell the user ahead of time to sign up for an account (I expect some problems with this) once the user logs in for the first time they will be added.

I know there are bulk add user plugins out there already but they would not work with the CWL and to be honest I think this Add User functionality is much slicker.

Also we are not connected to the SIS (probably never will be) so this is the next best thing at the moment. Special thanks to OLT super programmers Pan Luo and Godfrey Chan for working on this. This was Pan’s first time working with WordPress and he did a great job implementing the new CWL features, Godfrey showed him the ropes in WP plugin development which he is pretty much a master of. The Selenium QA tests they developed for this also make me very happy

UBC Blogs Stats and Growth

Posted on November 23, 2009 by Scott McMillan

This is inspired by Jim Groom’s recent post. I used to be a little obsessed with stats I did a brief stint doing the “Internet Marketing” thing back in the heyday so the numbers were very important $$ but now they take on a little different meaning in academia (justification of existence?? which == $$ I guess)…

UBC Blogs in it’s current state (using WordPress MU) has been online since September 2008 the OLT has been hosting blogs for many years prior on a Moveable Type platform. We are currently officially a pilot which means (at least the way I interpret it) we cannot really advertise the service and we hide the account setup/login page… That being said we still have a decent number of users and blogs probably 10% of the user base was migrated over from the old Moveable Type platform the rest are newbies. We have a another server for more VIP sites (basically means mapped domains and custom themes) which sees about the same traffic with only a couple dozen sites.

User and Blogs Growth
This chart is interesting shows the user account growth chart in the system overtime 1438 users and 795 blogs.

Blogs growth (click for large)

Content Growth.
We ran this a few weeks ago on verf(too much of a hog to run on production) to check the total posts and comments in the system. These numbers go way back to 2004 because some of the old MT blogs were moved into the new system. Still pretty impressive numbers with over 24,000 posts. I am pretty sure this has to be one of the larger websites on campus in terms of number of hosted pages.

Traffic
We haven’t been using Google Analytics since the start but here are some numbers from this term so far.

Sept-Nov 2009 stats

Bounce rate is not the greatest 60% of people leave after visiting one page.

How are people arriving?

Top 10 URL’s this term

Interesting that four of the blogs are courses: digitalliteracy2009, etec522sept09, etec540sept09 and micb405 (private).

Where are people coming from? Top 10 visiting countries.

From our web traffic and content growth it seems that this campus “publishing platform” is alive and well and growing at a healthy rate despite being on the down low.

A Word on WordPress Spam.

Posted on September 25, 2009 by Scott McMillan

A couple weeks ago Brian and Novak mentioned they seemed to see more spam after the upgrade to WordPress MU 2.8.4 interesting I thought so I checked my blog the main blog which had virtually no spam, looking at their blogs they had quite a bit of spam but Akismet was catching 95% of it so I though no big deal, I also chalked it up to the fact people read their blogs and in the case of Brian’s alot of people read his blog (still not the most popular on blog on blogs.ubc.ca which receives virtually no spam odd but not that odd). The last three days things changed and it was basically out of control, the two blogs combined were receiving over 4000 spam messages a day, most were being caught by Akismet but many were getting through. I checked the other top 10 blogs and they had nothing in comparison. I was starting to get paranoid if this happened to other blogs on our site this would for sure end up being a DoS.

On further investigation, looking at Novak’s blog he had reCapthca off for some reason (not so Super Novak)… after he turned it on spam went down a little but it was still up big time. Next step was looking at the server logs at first I thought we could use some .htaccess trickery to block the bots from hitting the wp-comments-post.php

RewriteEngine On RewriteCond %{REQUEST_METHOD} POST RewriteCond %{REQUEST_URI} .wp-comments-post\.php* RewriteCond %{HTTP_REFERER} !.*blogs.ubc.ca.* [OR] RewriteCond %{HTTP_USER_AGENT} ^$ RewriteRule (.*) http://%{REMOTE_ADDR}/$ [R=301,L]

But the bot was smart enough to send proper referrals so this would not work. What to do next? I started grepping through the server logs to see what the bots were hitting on their blogs both were being hit by old posts (although both do not post that often so almost everything is old) but I mean old like over 3 months old. Both had a lot of spam being directed at post on a “Forum like display plugin” post as well as a couple others. So on both blogs I set turn off comments after 90 days (Novak) and 120 days (Brian) and I disabled trackbacks. This pretty much had an immediate impact spam was virtually stopped on both blogs. In conclusion this is a temporary fix I recommend everyone having spam issues right now should do this (I always recommend disabling trackbacks).

This definitely is a bigger issue if bots can spam and bypass both reCaptcha and Akismet (in some cases) WordPress has a fairly serious security issue hopefully WordPress addresses this soon.

Realtime Syndication with the PubSubHubbub protocol

Posted on August 6, 2009 by Scott McMillan

Seems like whenever you come back from a vacation there is something new being discussed that rocks your world. This time for it’s the PubSubHubbub protocol developed by Google Engineers.

A simple, open, server-to-server web-hook-based pubsub (publish/subscribe) protocol as an extension to Atom (and RSS).

Parties (servers) speaking the PubSubHubbub protocol can get near-instant notifications (via webhook callbacks) when a topic (feed URL) they’re interested in is updated.

The thing I like about this is in a “Feed WordPress” scenario. Instead of polling the same system a Hub Server could be polled taking some of the strain of the publishing WPMU server and also allowing faster content updates… I think Feed WordPress would need some changes but seems doable.

There is already a WordPress Plugin that supports the protocol…

PubSubHubbub Demo – Real-Time CrunchUp

Timber PHP debugging in WordPress

Posted on June 24, 2009 by Scott McMillan

Stumbled upon Blurbia and interesting WordPress development company that is doing some pretty solid work.

Timber
http://www.blurbia.com/plugins/timber/

Timber is intended primarily as a developer’s tool, to be enabled in a sandbox environment while debugging plugins or themes in-progress. However, it also has potential in a production setting, as a way to hide errors from the end-user while still logging them for administrator review. The configuration settings allow for precise control of what type of errors and what error information is logged, so it can be deployed in varying scenarios without runaway logs filling the database.

Synected
http://www.blurbia.com/plugins/synected/

Synected enables shortened URL creation on your own blog. Rather than relying on services such as tinyurl.com or bit.ly, Synected lets you easily create and use short links based off your own domain. This releases you from dependence on a third party — a server error on tinyurl no longer prevents visitors from reaching your site. In addition, it strengthens your brand, keeping your domain name in view of your audience even on Twitter and other micro-communication platforms.

These guys develop some pretty tight themes also:
http://demo.blurbia.com/melora/ Fully customizable. With sites like this, plus the other framework themes we are getting very close to not having to design themes anymore… which is great because you will be able to role out a site in a day vs weeks.