Generalists vs. the Army of Waste

In many larger organizations, there is often a push to put things in a certain “order” that administration (non-educators and non-developers) is comfortable with. In the case of higher-ed, especially with its large, cubicle-seated, vertically organized groups, there is a strong trend to divide technology into The student learning management system (LMS) that covers “all” students’ teaching and learning needs, and into The content management system (CMS) that, managed by marketing people, is meant for “administrative” websites. And, of course, research is under a separate VP and thus needs to have its own systems and staff. And so on, for all the lovely silos. The more broken down, the merrier; this system creates more bureaucracy, more reporting and analysis. It is disconnected both from within itself and from its users, so it requires more staff to keep it going, more managers and more analysts.

This approach is a very typical and also very wrong one, as the expertise and resources needed to run and support, let’s say CMS (seen as “administrative” or “marketing” tool, though some of the best research or teaching and learning websites are built using UBC  CMS), are identical to those of an open publishing framework for teaching and learning (such as blog or wiki platforms) and shouldn’t be separated – one general, flexible and robust platform is capable of responding to most web needs. To further complicate things, in the administration’s desire to do things “properly” (like mid-’70s-IT properly) we are introducing too many flavours of unnecessary overhead or waste (Lean terminology), known as learning or web strategists (my title actually, though I wear few other hats), project managers, business analysts, faculty liaisons and many other fancy names.

We work (teach, design, code…) in the environment of rapid and constant change (perpetual beta) and by building so many layers between the faculty member who wants to experiment and embrace change, and the developer who wants to give, hack and create change, we are actually killing the experiment and killing the change itself.  We kill creativity and innovation, while we are supposed to support it. By stripping down web developers to the role of only coding whatever project manager or business analyst requires (and because managers and analysts are not creators and producers, they are not in business of inventing, hacking, dreaming and designing), we kill the much needed seeds of agility and entrepreneurship and we leave very little room for linchpins in us, with hard and practical skills.

That’s the shame, because the best of Web2.0 is telling us that we need to simplify things, not over-plan or over-analyse. Hear the idea, build, measure, learn. That’s why instead of many unnecessary roles, we need developers to be more of the generalists (Scott, thanks for this find) – for typical web company that’s the person that does information architecture, UX, coding front-end, back-end, db, scaling, bit of design; at the university, it is all of that plus bit of pedagogy and learning design. Not a biggy, few months of the right exposure for a reasonably intelligent individual.

a glance back

Enej, Will and I have been presenting at the Vancouver WordCamp and then during UBC’s Open Access week and somehow in both presentations we were going back through the history of UBC’s Open Learning Platform (the pilot and non-approved name for UBC Blogs, Wiki, CMS and related apps).

I’ve realized that I don’t really have a documented overview on how we got here, and even worse – what is the current state of the platform and plans for the future.

So, I have quickly put together basic wiki page (see it embeded below) on some (definitely not all) milestones of historic development of our platform. Hopefully, we will work on this page and also add more granular pages on specific and important points and decisions.

2004-2008

  • MovableType based blogs, number of websites running on various technologies

2008

  • WordPress based, Campus Wide Login (CWL) enabled blogs.ubc.ca launched, over 700 users/blogs migrated from MovableType
  • Common Look and Feel (CLF) introduced by UBC Public Affairs
  • MediaWiki gets CWL and CLF treatment and becomes UBC Wiki - wiki.ubc.ca

2009

  • CWL enabled sites.olt.ubc.ca for hosting websites
  • Around 20 websites on the server by the end of 2009

2010

  • UBC blogs gets new social homepage and social (buddypress) network
  • CTLT and PA partner and sites.olt.ubc.ca becomes UBC CMS and inherits PA’s cms.ubc.ca
  • Minimal WordPress Hybrid theme becomes the parent theme for most of CMS websites

2010-2012

  • over 10,000 blogs and over 500 active, domain-mapped websites. Over 15,000 users of which over 12,000 students.
  • Continual improvement of our platform - the main developments:
    • Solid back-end: partitioned mySQL to 256 databases; Both blogs and CMS have load balancer in front of 4 app servers and master/slave db servers.
    • Extending our UBC CLF theme - almost all sites run on this theme - see manual
    • Writing plugins to support UBC community, under CC license, over 200k downloads from Wordpress.org only!
    • Established great user community through monthly larger clients' meetings, end users' support through open web clinics twice a week, community based UBC wiki documentation
source: http://wiki.ubc.ca/Documentation:The_History_of_Open_Learning_Platform_at_UBC

 

A Year Later

Actually, over a year since I last posted here. I guess I am not much of a blogger.
Anyways, just checking in, I am not sure why, perhaps to lubricate those few rows in one of the thousands of mySql tables that store much of UBC’s publishing content.

Aside from my dormant blog, things look great, we’ve never been busier and I mean healthy busy: working with great people, getting smart requests and supporting development of many new and existing websites.
I’ve never been happier with how our little team is performing and how much we deliver.

about running large WP install

Just during last week, two well-known edublog spaces, Jim Groom’s UMW blogs and Alan Levine’s CodDogBlog (both WP-based) experienced serious hiccups and in case of UMW blogs serious downtime. There are many to blame, from mindless spammers to the mighty Google, but in the nutshell, problem seems to be the nature of how we use WP in general: our small pieces became one giant does-it-all piece with many loose holes that all the bad guys (viagra/casino/porn) and good guys (google and similar bots) so vigorously exploit.

this is why it is leaning
Can we hold it together?

In the next couple of posts, I will try to go through our experiences with running WordPress in larger institution; here is the preview:

  • scope (what we offer):
    Delete and forget about WP themes and base your offer on templates: blog, course, website, community portal, project management space, resource repository, rss aggregate, event calendar, research space, photo gallery…
  • signups and user management:
    Implement campus’ single-sign-on authentication, and write plugins to connect to campus student information system to download students to course blogs. One of the biggest time-consuming tasks with large user base systems is the user management and account creation and you want to automate this as much as possible.
  • development  framework and system maintenance:
    VMs, SVNs and Backups – your university IT department is your friend. Hang out with these guys and revisit your statements on cheap and dirty if you want to last. Development, Verification and Production server – you gotta have all three. In sync. Often looks like mission impossible but it is actually doable.
  • typical development tasks
    Based on our experience, these are the typical tasks in WP development that require framework mentioned in the previous bullet – you don’t want to try any of these on your production server:

    • Major Content/Architectural changes – your client will ask you to shift few hundred pages around. This is the case primarily with WP-based websites mostly
    • Look and Feel work – you have to change all your typography, headers and footers and rewrite all your CSS files because campus decided to update their look and feel.
    • Plug-ins development  – Make sure that your development environment is exactly the same as your production one or you are up for nasty surprises.
    • Back-end upgrades – Your LAMP, your WP core needs to stay upto date.
    • Migrations – We did migrate few hundred MovableType blogs to WP and survived.
  • buy-in, support and training
    Don’t count on your edupunk coolness. You are here to provide good service and support.
  • searching and reporting
    It is often pain in the butt and slows down the servers but it also gives a lot in return. Any WP coding that stores content in custom fields will not be indexed by WP itself so just use Google search instead. And after using Google analytics you can’t use any other tool. Also make sure to be able to report on number of blogs (and their types) and users at any time.
  • new requests
    Say no. They are typically unreasonable, otherwise plugins would be already written. If you hear it 1000 times than you might want to consider it. Clients are usually wrong.

one, to bind them all…

the constant gardenerConstant Gardener, from Flickr.

Being the most flexible and open of all web spaces, wiki can quickly become a messy hotchpotch of various pages written in variety of tones, driven by different needs and intended for miscellaneous audiences. And that’s not almost ok.  It is ok as long as it belongs to Namespace Notebook (or test or perhaps crap).

Growth of the wiki space, its usability and maintenance is often compared to gardening where wiki gardener or wiki gnome plays an important role in wiki’s overall health.

So what’s the deal with UBC wiki, who is it for and how is it going to be used?

New, MediaWiki-based, CWL-enabled (CWL is UBC’s single-sign-on implementation) UBC wiki has been up and running for almost a month, available to everyone with a valid CWL account (yes, we proudly display the CWL login link, meaning that this is NOT a pilot project). We are hoping that, as it is case with other successful large organization’s wikis (stretching this to a very larger organization called Earth, we could argue that its best information source is Wikipedia), this wiki will grow and become useful lexicon of UBC and a few things beyond that.

So far, we have recognized four major means of using UBC wiki (all four to be represented by corresponding Namespaces):

  • UBC dictionary (lexicon, glossary)
    This is the original, simple wiki idea – flat Wikipedia-like approach, for anything UBC related; it lives in the default MediaWiki NameSpace (no subpages allowed, here is why).
    For example, Genome page should inventory UBC resources about Genome – topics like people, groups and departments that research genome; papers, posters and thesis published about genome etc. In the ideal scenario, UBC faculty, students and staff would update topics of their professional (and wider) interests and so make resources more presentable and easier to find. Another example: By slightly modifying great google maps extension, that is already running on our server, we could build multi-layered maps of UBC where  community could contribute to a spatial representation of UBC; every conference visitor would appreciate a decent coffee at UBC Google Map layer.
    (here is a new Blenz coffeee, we need others; Google also has to update their maps as this is quite a new big neighborhood now.)

    View Wesbrook Place in a larger map

  • Course repository
    With UBC wiki course space, our main goal is to enable space for public and open-content licensed wikis.
    In previous years we have used both blogs and wikis to develop, host and deliver courses. Lately, we experimented with hosting course-content in wikis and republishing in other web spaces, such as blogs or LMS, using in-house-developed MediaWiki extensions to support embedding part of the page, the whole page and in the future, collections of pages (the whole course content).
    Before we decide to go with the Courses Namespace, we hoped that it will be possible to maintain the flat structure of the wiki while somehow keep things simple for instructor to build linear and hierarchical course content. It turned out to be quite a challenge – we opted for creating standalone Courses Namespace instead, space that allows subpages and that will have its own search and set of extensions to export, print or republish course material.
  • Documentation
    Documentation space is very similar in architecture to the course space. It will support subpages and allow users to build linear and hierarchical documentation, support and training materials.
  • Yet-to-be-named space (Notebook, Test or whatever?…)
    To keep things from going completely wide and in lack of proper gardener or policeman, we decided to create a Namepace where people could do pretty much anything without worrying that their page will be deleted because it doesn’t comply with wiki etiquette. Planning a party, developing a thesis, testing extensions or just showing what wiki can do – please use Notebook.

So where from here?

While it seems that we do have some valid ideas, technology seems pretty straight-forward, and it can only get better and easier to work with. On the other hand, we have yet to come with right strategies to get people interested in this, show potential and get the buy-in.
Only then we could really say that we have actually  accomplished something.

section widget

It is not often that I feel completely pleased about the work that team I work with or myself have finished. There is always a bit of “well, this could be done better”, or “we should have…”.

But after our final testing and publishing of Section Widget to WordPress Plugin Directory, my heart is full.

Yes, some of “well, this could be done better”, and “we should have…” still exist, but I am absolutely pleased and I see enormous potential on how this widget can continue to grow and change the way we think of the WordPress as the Content Management System (hint: next version will enable custom fields management). Special credit goes to our fantastic co-op student Godfrey Chan, for making things happen in hours rather than weeks, while maintaining out-of-this-world quality of code.

Please learn about it, download it and enjoy it!

gimmie wiki!

Enej Bajgorić created the plugin for republishing MediaWiki (or any HTML) pages within WP.

For those who liked the idea, here are some good news: we are continuing development of this plugin for our culturepedia project.

Here is the functionality in the nutshell;

  • seamlessly integrate wiki pages, categories or searches into WP blogs.
  • works as both shortcode (more here and here) and widget.
  • content can be constantly refreshed (as wiki pages changes) or one-time brought in.
  • the wiki content will be searchable by google (not the case with WikiInc).
  • the wiki pages could be further shared by using embed functionality.

For example, if I want to publish all the pages from ubc wiki that are stored in both culturepedia AND idioms category, I should shortcode something like this:

[gimmiewiki -category URL:http://wiki.ubc.ca category:culturepedia,idioms refresh:yes popups:yes|embed|link]

Or if we end up developing a visual component, it could look like this:

Will get back with more!

migrating a kiloblog: ubc goes from movabletype to wordpress

We have been running Movable Type for over three years and for number of reasons (far better platform, CWL integration, the support overhead and availability of plugins, to name a few) decided to move to WordPress.

Since last summer we have pretty smooth, CWL-enabled, pilot WPMU installation available to selected UBC faculty, staff and students; so far, things look pretty good!

Now that we are at ease with our ability to run the new system, we are planning to migrate more than a thousand MT users  and blogs to WP.  We are expecting that only the small number of people will actually want to move their blog – for the rest… well, we are thinking about it as the great autumn-cleaning opportunity. Anyways, EVERY blog will be backed up and saved in case users come in the future to restate their blogs.
Here is the process with its requirements and challenges along with the ideas on how to go around them:

Setup: There are three environments in which the migration takes place:

  • Original MT (Source)
  • Verification WP (Check-point)
  • Production WP (Target)

The process:

1.       Notifying the Users and Project Tracking.
We will send the email to users describing the migration process and inviting them to create an account and a blog on the Production WP, which is the new location for their blog. We have classified all the users on the scale of 1 to 5, based on the importance of the blog (we have tons of abandoned blogs) and we also have special VIP list – those blogs get extra attention!
Here are the fields of our migration tracking database:
blog_id, blog_name, entries_count, blog_site_url, author_id, author_name, author_email, other_blog_owners, blog_description(tags), priority_rating, migration_notes,  vip, new_author_name, other_authors_translation, new_url.
You can sense the complexity of this operation by just looking at these fields!

2.       User reports back
Upon creating the account, blog user gets back to us with the account and blog info (details instructions on how to do this were sent in an email). We then proceed with the next step of migration:

3.       The Import to Verification WP
MT blogs and users’ info are exported from MT to Verification WP by using WeblogsMigrator plugin that Michael Ha and Seth Tee have specifically developed for this purpose. This WP plugin actually queries the MT database and retrieves all the information on the given blog and user and then stores them in the Verification WP. So, we end up with blogs, authors and comments, all properly imported within WP Verification. Users from MT keep the same username in WP Verification. It is worth mentioning that in this process, all the links that point to the other posts within same blog are being re-written to point to the new URLs. It is even cooler that we are grabbing all the media files (images, movies) and storing them automatically in WP Verification.

4.       Finalizing the Look and Feel and dealing with blogs’ extras.
Here, we apply theme that look close to one in MT, for look and feel consistency. For VIPs, this may involve additional tweaking or creation of the new theme. If needed, we consult with users on how to proceed with extra content (if there is any) of their MT blogs. Usually, this means dealing with side bar content and advising on how to recreate these in WP (through using widgets).

5.       From WP Verification to Production
After examining the blog and round of QA, we do another round of export/import; this time using slightly changed WeblogMigrator. Things get hairy with blogs that have multiple authors; for this reason, we made sure that this plugin, which looks very much like standard WP to WP import, also provides username matching. With this, we make sure that the original users keep ownership of their posts after the blog is migrated. It is expected that many users will have different usernames from those on MT, that’s why we cannot automate this process. This is only important for multi user blogs.

6.       Redirecting old URLs
One of the major requirements of this process is to make sure that every post will automatically redirect to the new post’s URL.  In the process of importing (step 3), we record the old and the new URL for each post and store it in the database. This table is now queried to produce the  .htaccess file that will, for every post, redirect to the new post URL.

7.       User takes control
At this point user has the complete control over their blog. We are expecting that owners of more complex blogs will require additional support that we can provide through pointing to various WP support websites and examples online, or via direct support.

resolving “comment-goes-to-admin-not-author” wp issue

We are planning the migration of our Movable-Type based websites to WordPress. While offering way nicer management interface and wealth of advanced controls, default WP installation still misses a few things that we took for granted in MT.

Here is the one: On our big websites that run on MT, such as Leap or OLT, we have multiple authors maintain number of posts and pages. Whenever there is a comment on any of these posts or pages, it should be directed to the author of the post for approval.

MT does this well, all the comments indeed go to the authors of the posts. WP however, redirects all the comments to the site admin (?!) with authors left in the dark about the new comments. Admin is usually the technical person who created the blog, without the clue what comments to approve. This makes management of the site very difficult as authors need to be able to respond to the comments in timely fashion.

Luckily we‘ve found Mark Ghosh’s extension that does exactly what we need: emails the comment to authors for approval. The extension is installed and working great at blogs.ubc.ca.