by Noan Fesnoux,  Yik-Wah Penner, Scott Valentine

1.0 – Introduction

The implied goal of this assignment is to encourage group members to consider the procedures and strategies for developing an evaluation rubric to guide BCcampus leadership in the selection of a learning management system (LMS) that will adequately:

1. serve the needs of all B.C. postsecondary institutions and
2. provide sufficient flexibility to adequately respond to future needs, while
3. adhering to current and anticipated future resource constraints.

 

However, there is a great deal of information that is not provided in the case which, in actual practice, would have to be gathered prior to the development of such a rubric.

First and foremost, in order to effectively carry out such an exercise, a needs analysis would be the first step in defining organizational requirements. In order to conduct a comprehensive needs analysis, decision-makers should ensure that the views of all relevant stakeholders should be solicited. All too often, the needs of teachers and students are underrepresented in such a process. [1]

From this enumeration of stakeholder needs, a comprehensive list of organizational requirements can be developed. This list can serve as the criteria upon which to evaluate potential products.[2] In this case, the findings of a needs analysis are not explicated; therefore, assumptions must be made in order to enable our group to create a rubric. We would not advise making such assumptions in real practice – the stakes are too high.

There is one other aspect of this case that merits closer scrutiny. The scale and scope of this investment underpins the need for careful due diligence. The sheer cost and strategic importance of selecting a system(s), suggests that not only should a needs analysis be undertaken, the field of possibilities should be expanded beyond the two LMS’s that the organization currently uses (one is an open-source system and the other is a vendor-based system). Moreover, just because the IT department budget will be cut in half in three months, this does not automatically dictate that the organization should transition from two systems to one. The decision to adopt one, two or more systems should be based on user need (present and future), organizational capacity and comparative costs.

It is with these caveats in mind that the development of an evaluation rubric will now be explained.

2.0 – Identifying prospective contenders

According to Craig Weiss, there are more than 500 companies/developers that offer LMS software.[3] Given the cost and strategic importance of LMS selection to BCcampus, the first step in identifying contending LMS offerings should be to specify a list of needs that must absolutely be met by the LMS. Failure to meet these “essential needs” should disqualify contending technologies from further consideration, thereby narrowing the list.

We also recommend that all non-essential needs be further broken down into two other groups – “wants” and “wishes”. The “wants” are affordances that are important to have, but not essential. The “wishes” are nice to have, but not necessarily important. By creating these two groups of categories, wants can be assigned higher priorities (weighting) than wishes.[4]

In summary, we suggest that any rubric should be split into three sections: essential needs, wants and wishes. Failure to meet all essential needs disqualifies any competing product. Meanwhile, the ability to meet wants should be weighted as more important than wishes.

In order to narrow the list of prospective systems, we have decided that we would first screen the available LMS by customer volume – selecting the top 10 as options. The logic behind this is two-fold. First, the adoption of an LMS is a costly, time consuming endeavour. It would be a waste of resources to implement a change and then discover that the LMS provider has gone out of business, leaving the offering unsupported. Since a higher probability of commercial success is partly predicated on market share, we have decided to vet the systems based on volume. Second, by adopting one of the larger systems, we are investing in a product from a firm that has the capital to support continued innovation.

3.0 – Rubric Development Foundation

Once needs have been classified into essential needs, wants and wishes, the specific criteria can be then included in a column entitled “criteria for evaluation” on the evaluation template. Next to this column, a series of products can be listed and evaluated based on each criterion. An example of what one row might look like on an evaluation template is included in Table 1.

Table 1: A Sample Row of an Evaluation Template

Criteria	LMS 1	LMS 2	LMS 3
Assessment Criterion 1 Descriptive about this criterion.

We recommend that the LMS product evaluation process be based on a five-point Likert scale. The Likert scale rubric can be defined at the top of the evaluation template by employing a legend. An example of a scheme for guiding analysis is provided in Table 2. Obviously, it is possible to use various other weighting scales – i.e. ranking of 10 or 100 – or even adopt a different approach such as comparatively ranking each of the competitive products for each criteria. However, we like the premise of using a five-point Likert scale because it allows us to tabulate the results and to graphically see where certain products excel or fall short.

Table 2: Rubric Evaluation Scheme Example

Rubric Legend:

★ = fails to adequately meet minimum requirements

★★= falls just short of meeting minimum requirements

★★★= Meets minimum requirements

★★★★= Slightly exceeds minimum requirements

★★★★★= Significantly exceeds minimum requirements

A rubric scheme such as the one above could be used to evaluate each criterion. We would recommend that the evaluation committee includes students, learners, IT technicians and administrators to ensure a balanced analysis. An example of how the rubric scheme might be applied to the criterion presented in Table 1 is depicted in Table 3. In this example, if “Assessment criterion 1” is an essential need, LMS1 would be immediately eliminated from the analysis. Once all of the criteria have been evaluated in this way, the stars can be added up within the three sections of essential needs, wants and wishes; and the totals can be compared across product offerings.

Table 3: Applying Rubric Evaluation Scheme to Criteria

Criterion	LMS 1	LMS 2	LMS 3
Assessment Criterion 1 Descriptive about this criterion.	★	★★★★	★★★

 

Having now defined our strategy for constructing a rubric for evaluating LMS affordances, we can now turn to the presentation of a complete list of criterion that we assume would be highlighted in a needs analysis. Once again, without an appropriate needs analysis, a rubric that effectively evaluates LMS according to the needs of BCcampus cannot be constructed. However, we are willing to speculate on what such a rubric might look like based on a cursory examination of the BCcampus web-site. Our criteria have been extracted from a more comprehensive LMS checklist created by the University of Central Florida.[5]

4.0 – The BCcampus LMS System Evaluation Rubric

Recall that we have decided to evaluate the top 10 LMS on the market, based on user volume because we feel that the top offerings will stand the best chance at market survival, allowing us to use the system in a sustainable manner.

Therefore, the first step would be to eliminate any of the 10 best used systems that did not address the “essential needs” of BCcampus. To illustrate, in regard to the first essential need related to “Access”, three of the LMS under consideration would be eliminated from the selection process in the assessment depicted in Table 4 (Note that the ✔or ✘ have been assigned for indicative purposes only. They do not reflect the actual features of the systems.) For this first stage of the analysis, the descriptive questions are dichotomous yes/no questions because failure to meet the criteria disqualifies the offering from further consideration.

Although it might be possible to cobble together various web-based solutions to resolve any serious deficiencies (i.e. failure to address an essential need) with a given LMS, taking a patchwork strategy to LMS design is risky, time consuming and possibly fraught with technological incompatibilities. It is for this reason that we feel that an LMS must at a minimum address all “essential needs”.

 

Table 4: Comparing Organization Needs to what many LMS provide

Criterion Rubric Legend T	1	2	3	4	5	6	7	8	9
Access Is the LMS accessible via different platforms? Does the UI adapt for handhelds, but also provide a good layout for laptop and desktop users (ie is it responsive?)	✔	✔	✔	✔	✔	✘	✔	✘	✔	✘
Long Term Support Has the LMS provider established ongoing support, and does it regularly roll out updates to keep itself current? Furthermore, does it do this for installed plugins as well?
Integration with Outside Apps Can the LMS import users from the existing user database (eg. Excel) or application (eg. Google Apps)?
Support Existing Content Can the LMS import course materials packaged in our current LMSs?
Flexibility/Scalability Can the LMS support these needs as the organization grows and requires greater resources and functions?
Cost Ceiling Can the LMS meet the essential needs of the organization within the allotted financial budget?
Reliability Can the LMS perform consistently at a high level with regular maintenance?

Once the list of 10 vendors has been narrowed down to only those LMS that are capable of satisfying all essential needs, the organization could then begin to evaluate the ability of the remaining contenders to satisfy the “wants” and “wishes” of BCcampus. An example of what this might look like is provided in Table 5. In this analysis, if costs were the same, LMS2 would emerge as the preferred choice. If costs differed significantly then the organization would have to factor cost into the decision process at this stage. The criterion found in Table 5 are loosely adapted from The Matrix Transformed: Achieving Better Focus and Insight in Learning Management System Selection, by Dr. Laura Winer of McGill University.[6]

Table 5: Comparing Wants and Wishes for 2-3 LMSs

Criterion Rubric Legend: ★ = fails to adequately meet minimum requirements ★★= falls just short of meeting minimum requirements ★★★= Meets minimum requirements ★★★★= Slightly exceeds minimum requirements ★★★★★= Significantly exceeds minimum requirements	LMS 1	LMS 2	LMS 3
Pedagogical Diversity To what degree does the LMS support a diverse range of pedagogies, including online courses, blended learning, constructivism, and flipped classroom models?	★	★★★★	★★★
Social Media Integration To what degree does the LMS include tools so that users can link accounts on other social media platforms to their LMS account. This will allow for greater usability and more frequent discussions within the LMS.	★★	★★★	★★★★
Appearance to Match BCCampus To what degree can we skin or modify the LMS to the point that it has the same look and feel as the BCCampus website? How easy is this to do?	★★★★	★★★	★★
Communication To what degree can the LMS provide both synchronous (instant messaging, chat, and discussion) and asynchronous (e-mail) communication?	★	★★★★	★★★
Open Source To what degree does the LMS abide by a FOSS (Free and Open Source Software) policy, and how strongly does the community support that project. (nb. BCCampus takes value in Open Source Software and Creative Commons based on their website)	★★	★★★★	★★★
WANTS SUBTOTAL (# of stars)	10	18	15
Learning Analytics To what extent is the LMS capable of  gathering and reporting on student learning outcomes?	★★★	★★★★	★★★
Training Materials To what extent are training and learning resources available to users?	★★★★	★★	★★
Calendar Functionality To what level does the LMS offer the collaborative calendar functions (which include but are not limited to sharing invitations, sharing calendar events, creating joint calendars)?	★★★	★★★	★★
Courseware Sharing To what degree does the LMS provide tools to easily share courseware with other institutions, and are there tools to  search for open courseware built into the LMS?	★★★	★★★	★★
WISHES SUBTOTAL (# of stars)	13	12	10

*Plugins Available
References

[1] Clayton R. Wright, “Selecting a Learning Management System: Advice from an Academic Perspective”, EDUCAUSE Review, April 21, 2014. Accessed May 29, 2015 at: http://www.educause.edu/ero/article/selecting-learning-management-system-advice-academic-perspective

[2] Foreman, Steve. “Five Steps to Evaluate and Select an LMS: Proven Practices”, learningsolutionsmag.com. June 5, 2013. Accessed May 29, 2015 at: http://www.learningsolutionsmag.com/articles/1181/five-steps-to-evaluate-and-select-an-lms-proven-practices

[3] Craig Weiss, “Top Ten Learning Management Systems—July Rankings,” E-Learning 24/7 Blog, 16 July 2013. Accessed May 29, 2015 at: http://www.educause.edu/ero/article/selecting-learning-management-system-advice-academic-perspective

[4] Clayton R. Wright, “Selecting a Learning Management System: Advice from an Academic Perspective”, EDUCAUSE Review, April 21, 2014. Accessed May 29, 2015 at: http://www.educause.edu/ero/article/selecting-learning-management-system-advice-academic-perspective

[5] The link to this checklist is: https://online.ucf.edu/about/history-of-cdl/lms-migration/lms-evaluation-checklist/

[6] Winer, L. R., Finkelstein, A. B., Deutsch, M. D., & Masi, A. C. (2005). The Matrix Transformed: Achieving Better Focus and Insight in Learning Management System Selection. EDUCAUSE 2005. Retrieved June 7, 2015, from http://net.educause.edu/ir/library/pdf/EDU05197.pdf