Our group was tasked with creating a rubric to evaluate LMSs for BCcampus. Our group spanned 2 provinces and one territory. We completed our task through a shared document in Google. A link to our completed work is here.
The specifics of the scenario we were given were these:
BCcampus is a publicly-funded agency that offers “teaching, learning, and educational technology support“ to the 25 post-secondary institutions in British Columbia and one in the Yukon. As part of its shared services, BCcampus has been running two LMS platforms; one open-sourced (Moodle) and the other vendor based (D2L) whose contract will expire shortly. With current restructuring, BCcampus will be losing half of its tech support team in three months, and therefore the possibility of running two separate LMS platforms is no longer an option. BCcampus will need to decide which LMS they wish to proceed with across the board. However, as BCcampus is known for its leadership in innovation, they are open to the idea of selecting an entirely new platform. As employees at BCcampus, we have been asked to create an evaluation rubric to help the BCcampus leadership team in the decision making process. We have designed the rubric to help select a LMS that will fit with the needs, the vision and the mandate of BCcampus: “to connect, collaborate and innovate“. The selection process must also recognize the LMS’ ability to conform and adapt to the geographically and culturally diverse needs of BCcampus’ partner institutions and the thousands of post-secondary students across the country.
The detailed rubric we developed looks like this:
| Criteria |
Fair |
Good |
Excellent |
| 1. Access |
| LMS system requirements (compatible with the current system and adaptive to future changes). |
The LMS requires considerable system upgrades to integrate with the current BCcampus system. |
The LMS requires minor system upgrades to be compatible with the current BCcampus system. |
The LMS is fully compatible with the current system used by the BCcampus. |
| Devices. |
The LMS can be accessed with limited functionality on mobile phones and tablets. |
Certain features of the LMS may not be accessed on mobiles phones and tablets. |
The LMS can be fully accessed using a full range of devices including: mobile phones, tablets, and desktop computers. |
| System requirements. |
The LMS is designed to be used with a particular browser, or requires plugins, or special browser configurations. |
The LMS supports the use of more than one browser. May require a downloadable app to run LMS on mobile and other devices. |
The LMS runs optimally on Windows Internet Explorer, Google Chrome, and Mozilla Firefox and requires no special set-up requirements. |
| 2. Support |
| Support available for the interconnectedness of all partner institutions. |
Training/ support is only available for a time limited period after the LMS is initially set up. Online resources are limited. |
Training/ support is available by phone only during specific hours of the week. There are some online resources. |
Training/ support is available whenever the user has need. Online support is accessible 24/7. |
| Training and maintenance support for educators. |
IT support is available during regular office hours by email or phone. |
IT support is available by email or phone. |
IT support is available 24/7 by email, phone, or live chat. |
| IT support for students. |
Student can access IT support by email only; response will be only during office hours. |
Students can access IT support by email or live chat during office hours. |
Student can access IT support 24/7 by email or live chat. |
| 3. Functionality |
| Provides a well thought out interface for all users of the LMS. |
The LMS is relatively easy to use. |
The LMS is easy to use and possesses some intuitive features. |
The LMS has superior scope and sequence and exceptional insight into the needs of the users. |
| Layers of privacy and security that accommodate various applications (medical, legal, personal). |
LMS provides adequate security for many of the applications. |
LMS provides adequate security for most of the applications. |
LMS provides excellent security for all the applications. |
| LMS innovations adapt and keep pace with future learners. |
Limited design options. May require customized work by vendor, provider or open source LMS specialist. |
Some “themes” available to change look and structure of user interface. |
LMS interface look and feel can be customized by local administrators. The structure and organization is intuitive, and adaptable to fit needs. |
| 4. Cost |
| Initiation/migration costs (set-up, implementation, staff training). |
Implementation costs are above budget. |
Costs are in-line with last year’s budget forecasts. |
Costs are well under budget. |
| LMS maintenance (server back-ups, updates, course back-ups, course edits, archiving, staffing). |
Regular maintenance will require more financial resources. |
Regular maintenance costs are the same. |
Regular maintenance costs are reduced. |
| Potential future costs (next 4-5 years). |
No or limited stability in cost. Possibility of increases within 4-5 years. |
Cost is stable for the stated time frame. |
No foreseeable cost increases in the future. |
| 5. Customization |
| LMS is adaptable to changes in technology and user driven initiatives. |
The LMS is adequate in its current services. |
The LMS has features that are progressive, but many of those changes were the result of user suggestions. |
The LMS is structurally agile and responsive to changes in user needs and developments in technology. |
| Communication modalities. |
The LMS has limited integration of new technology into their services; tools such as social media are not capitalized on. |
The LMS has integrated new tools and technology, such as social media, yet it does not enhance the user’s experience. |
The LMS has successfully and seamlessly integrated new tools and technology, such as social media, that enhances the user’s experience. |
| Ability to customize learning paths (documentation, archiving, formative assessment). |
The LMS has some features, but is limited in scope. |
The LMS has features that blend what they can offer with what can be inserted or included by other providers. |
The LMS reflects in ways that are superior and extensive how information is actually interacted with by the user. |
| 6. Organizational Requirements |
A/synchronous communication mediums that facilitate interactions between:
1) instructor – student
2) student – student
3) student – resources |
The LMS has basic offerings that facilitate simple communication avenues. |
The LMS has a variety of formats for facilitating student interactions. |
LMS design enables easy and effective communication options all interaction types. |
| Cognitive design. |
The spatial and segmented cues of the LMS are adequate. |
The LMS has features that pre-load the user prior to more detailed training. |
The design elements of the LMS are effective so that learners are focused on the learning content rather than the medium. |
| Systemic alignment of LMS with BCcampus and its affiliates. |
The LMS fits well with the current mandate of BCcampus. The LMS requires some minor adjustments. |
The LMS fits very well with the current mandate of BCcampus. The LMS requires few minor adjustments. |
The LMS fits perfectly with the current mandate of the BCcampus. No additional adjustments are required. |
The reasoned articulation of our choices are in this researched-supported rationale:
Our LMS evaluation rubric is based on elements of the Bates SECTIONS model as well as Chickering and Ehrmann’s seven principles of good practice. As Coates points out, “decisions about university teaching and learning should not be restricted to checklist evaluations of technical and organizational factors” (2005). With that in mind, we did not limit our rubric to standard evaluation criteria, rather we designed the rubric around the specific needs of BCcampus; developing six high-level criteria encompassing the primary concerns of the agency: Access, Support, Functionality, Cost, Customization, and Organizational Requirements.
With 25 affiliates, the importance of versatility without restrictions is paramount. A system that can be accessed by every institution, device and end user, without compromise to functionality, will keep BCcampus at the forefront of innovation. Further, the BCcampus IT support will be reduced in three months making system compatibility an important requirement.
Secondly, it is important that the selected LMS provides support to all users. IT support must be available for BCcampus, the institutions, and the individual users, as these stakeholders might require different assistance at various times. If teachers and learners are not well supported then there is a high risk that the LMS may not be used at all (Bates, 2014).
The functionality of the LMS is an integral criteria for BCcampus to maintain their leadership in innovation by empowering the current and future learners through intuitive user interfaces. Furthermore, the functionality should not compromise the security and privacy of its users in order for open discussions to occur without consequences (Bates, 2014).
BCcampus uses an innovative collaborative model that allows various post-secondary institutions to share resources and costs (BCcampus, 2013). Participating institutions can access the resources and technologies they would not be able to individually. BCcampus must have information about the initial implementation, and future LMS maintenance costs in order to budget effectively. This in turn, will affect the participating institutions and their budgeting.
As Spiro mentioned, students are increasingly taking charge of their own learning; gone are the days of one-size-fits-all courses (2014). This mentality calls for a LMS platform that is customizable – allowing students and teachers alike, the opportunity to curate their own learning path. Further, BCcampus is looking for a long term LMS, meaning it has to be adaptable and able to integrate with new technologies in order to meet the needs of its diverse student population.
Lastly, BCcampus provides a leading-edge collaborative interface. As such, the LMS it uses must have strong cognitive design in its structural components, align with the mandates of its partners, and provide exceptional modalities for communication. We understand this will not be the only tool an evaluation team uses, but believe that the six areas covered will provide a well-rounded summary to the committee.
We used this relevant literature to support our decisions:
Bates, T. (2014). Teaching in digital age http://opentextbc.ca/teachinginadigitalage/ (Chapter 8 on SECTIONS framework)
BCcampus. (September 2013). Annual Report 2012-2013, Strategic Plan 2013-2016, Retrieved from: http://bccampus.ca/files/2013/10/2013-ar-stratplan.pdf
Chickering, A. W., & Ehrmann, S., C. (1996). Implementing the seven principles: Technology as lever. American Association for Higher Education Bulletin, 49(2), 3-6. Retrieved fromhttp://www.aahea.org/articles/sevenprinciples.htm
Coates, H., James, R., & Baldwin, G. (2005). A critical examination of the effects of Learning Management Systems on university teaching and learning. Tertiary Education and Management, 11,(1), 19-36. http://link.springer.com/article/10.1007/s11233-004-3567-9
Spiro, K. (2014). 5 elearning trends leading to the end of the Learning Management Systems. Retrieved from http://elearningindustry.com/5-elearning-trends-leading-to-the-end-of-the-learning-management-system
My reflection upon your experience completing this assignment
The learning objectives for this module were centred on understanding the role of delivery platforms, and to think about the challenges and opportunities that come with those platforms. Our group work, toward these ends, afforded me a unique learning experience.
My first realization was that evaluative work, such as creating a rubric to select an LMS interface for BCcampus, cannot be a one-person task. The sheer enormity of facets to be aware of needs to be tackled by a team. That team needs be one that have members who know their facts, the implication of those facts, and be thorough.
My second realization was two-part. I did not know/understand what LMS are/were. And further to that, it was a stretch to comprehend the ubiquitous role they play. To add to what I wrote in the first point, many and varied are the things that the LMS interface must provide and perform, and many more are the things that we still wished they could do. The best ones seem to be “invisibly there.’
My third and final realization for this purpose, is that the work that goes on “behind the scenes” often goes under-appreciated by those that use these systems. We know when we don’t like something (how it looks, how it functions, getting help), but sadly it is rare to acknowledge all of the thinking and planning and coding and envisioning and testing and modifying and . . . that several layers of hard work that have resulted in what we see before us.
Keri