Delivery Platform Evaluation Rubric
Jeremy Inscho, Chelsea Woods and Simon Forst
University of British Columbia
ETEC 565A
September 28, 2012
Précis
In response to growing competition in North America to provide distance education courses, Athabasca University, a world leader in distance education, is increasing its focus on the English speaking, South Asian market. This ad-hoc committee was convened to produce an evaluative tool to aid the Provost in determining the most suitable Learning Management System (LMS), or systems for the target market. While many rubrics for comparing LMSs will have similarities, the tool for this task must also consider the specific needs of Athabasca University to attract and retain students and address the challenges of providing distance education in a region with inconsistent internet connectivity. It is possible that a variety of systems satisfy the university’s requirements to varying degree. These range from highly interactive and visual systems such as Elluminate!, more traditional systems such as WebCT/Vista, Moodle and Desire2Learn (D2L), and less flexible systems of CD-ROMs, USB memory and print-based materials. The following rubric is the assessment tool developed by the committee for the purposes of informing the process of selecting a system or systems.
Rubric:
| Category | Criteria |
| (1) Strongly Disagree (2) Disagree (3) Neutral (4) Agree (5) Strongly Agree | |
| Students | Students have near-ubiquitous access to mobile technologies
(1) (2) (3) (4) (5) |
| LMS is setup to take advantage of mobile technologies
(1) (2) (3) (4) (5) |
|
| Community has reliable cellular service
(1) (2) (3) (4) (5) |
|
| Community has access to reliable wired and/or wireless internet service
(1) (2) (3) (4) (5) |
|
| LMS provides support for a variety of media
(1) (2) (3) (4) (5) |
|
| LMS provides support for a variety of learning activities
(1) (2) (3) (4) (5) |
|
| (1) Strongly Disagree (2) Disagree (3) Neutral (4) Agree (5) Strongly Agree | |
| Ease of Use and Reliability | LMS primarily relies on common computer literacy skills
(1) (2) (3) (4) (5) |
| LMS is not overly reliant on literacy skills
(1) (2) (3) (4) (5) |
|
| Unfamiliar tools are easy to learn and become familiar with
(1) (2) (3) (4) (5) |
|
| LMS is well structured
(1) (2) (3) (4) (5) |
|
| LMS is intuitive to use
(1) (2) (3) (4) (5) |
|
| LMS is easy to navigate
(1) (2) (3) (4) (5) |
|
| LMS is able to support emerging interface devices (such as mobile platforms)
(1) (2) (3) (4) (5) |
|
| LMS is proven software
(1) (2) (3) (4) (5) |
|
| LMS can be navigated effectively with low bandwidth and cellular data interruptions
(1) (2) (3) (4) (5) |
|
| Servers are able to run by Athabasca University
(1) (2) (3) (4) (5) |
|
| Sufficient support is available to address server software maintenance and repair needs
(1) (2) (3) (4) (5) |
| (1) Strongly Disagree (2) Disagree (3) Neutral (4) Agree (5) Strongly Agree | |
| Costs | LMS software and licensing are available at an acceptable cost
(1) (2) (3) (4) (5) |
| LMS integrates with library systems for access to copyrighted materials
(1) (2) (3) (4) (5) |
|
| Course development can be done at an acceptable tradeoff of research time, public service and administration otherwise performed by course developers
(1) (2) (3) (4) (5) |
|
| Interactivities can be added easily
(1) (2) (3) (4) (5) |
|
| Interactivities are reusable in subsequent course iterations
(1) (2) (3) (4) (5) |
|
| Reasonable student-teacher ratios can be maintained
(1) (2) (3) (4) (5) |
|
| Enrolment growth is projected to occur slowly
(1) (2) (3) (4) (5) |
|
| (1) Strongly Disagree (2) Disagree (3) Neutral (4) Agree (5) Strongly Agree | |
| Teaching and Learning | Medium for teaching will require significant video
(1) (2) (3) (4) (5) |
| Medium for teaching will require significant audio
(1) (2) (3) (4) (5) |
|
| Medium for teaching will require significant text
(1) (2) (3) (4) (5) |
|
| Skills required to be developed will primarily be demonstrated through video
(1) (2) (3) (4) (5) |
|
| Skills required to be developed will primarily be demonstrated through audio
(1) (2) (3) (4) (5) |
|
| S kills required to be developed will primarily be demonstrated through text
(1) (2) (3) (4) (5) |
|
| (1) Strongly Disagree (2) Disagree (3) Neutral (4) Agree (5) Strongly Agree | |
| Interaction | Learning activities will require significant synchronous interactivities
(1) (2) (3) (4) (5) |
| Learning activities will require significant asynchronous interactivities
(1) (2) (3) (4) (5) |
|
| Learning activities will require significant independent study
(1) (2) (3) (4) (5) |
|
| LMS allows for student-teacher e-mail and/or chat interaction
(1) (2) (3) (4) (5) |
| (1) Strongly Disagree (2) Disagree (3) Neutral (4) Agree (5) Strongly Agree | |
| Organizational Issues | Technical support is available through the university
(1) (2) (3) (4) (5) |
| Technical support is available through a third-party
(1) (2) (3) (4) (5) |
|
| Technical support is available through the LMS provider
(1) (2) (3) (4) (5) |
|
| Technical support is available in the primary language(s) spoken in the region
(1) (2) (3) (4) (5) |
|
| LMS and online materials are hosted through the university
(1) (2) (3) (4) (5) |
|
| LMS and online Materials are hosted through a third-party
(1) (2) (3) (4) (5) |
|
| (1) Strongly Disagree (2) Disagree (3) Neutral (4) Agree (5) Strongly Agree | |
| Novelty | LMS is novel enough to be a positive factor in student retention
(1) (2) (3) (4) (5) |
| LMS is proven reliable
(1) (2) (3) (4) (5) |
|
| LMS is new to the target student audience
(1) (2) (3) (4) (5) |
|
| (1) Strongly Disagree (2) Disagree (3) Neutral (4) Agree (5) Strongly Agree | |
| Speed | Courses are able to be designed for the LMS in an acceptable timeframe
(1) (2) (3) (4) (5) |
| Courses are able to be transferred from current (North American) LMS with ease
(1) (2) (3) (4) (5) |
|
| Course updates and changes are able to be incorporated quickly and easily
(1) (2) (3) (4) (5) |
|
| Changes can reasonably be made by instructors
(1) (2) (3) (4) (5) |
|
Rationale/Articulation
In setting-out to create a rubric for the selection of an LMS, our committee began with the foundations of the SECTIONS Framework (Bates & Poole, 2003) and looked to add or remove criteria as necessary to tailor it to the needs described and implied in the scenario. While some criteria were removed, as they did not well-inform the decision, there were multiple factors affecting the choice to include additional criteria.
The first factor considered was internet connectivity and the delivery of course materials as Athabasca University reaches out to the South Asian market. In addition to inconsistent internet connectivity (Smyth, Kumar, Medhi, Toyama, 2010), there are issues associated with shipping due to inconsistencies in the mail and customs service, as well as with CD ROMs as increasingly, technology devices do not come with a CD drive. Our solution, therefore, was to look for variety in the options for communicating and receiving information. For example, course modules could be packaged and zipped for easy download, so that students can download all the readings and module descriptions in chunks when they have a good connection, rather than spending time searching for reading materials online. Furthermore, information and marketing giant Nielsen (Harun, 2011; Nielsen 2011) suggests a dramatically increasing penetration of smartphones and other mobile technologies in South Asia and Asia in general. With this ubiquitousness of mobile technologies another consideration for working around internet connectivities and delivery issues is to use an LMS that has mobile technology integration allowing for delivery via cellular service.
The second factor considered was the attraction and retention of students. Chickering and Gamson (1987) suggest Seven Principles of Good Practice that an effective online course should employ to keep the interest and motivation of its participants. Some of the points that an LMS could help with are allowing various forms of interaction between students and their instructors and classmates.
The final factor affecting the rubric was the level of support available. As it is likely that many students would be taking courses in their second or third languages and in a time-zone nearly opposite to that of their instructors, we included items in our rubric to help assess the levels of support available to students. As Schaffhauser (2010) suggests, support is critical to keeping an LMS from becoming unusable.
The nature of the courses intended to be offered also plays a significant role in the selection of an LMS. For this reason, it may be that Athabasca University chooses a primary LMS, with the option for departments to use a secondary LMS or two based on their needs as indicated by the rubric.
References
Bates, A.W. & Poole, G. (2003). Chapter 4: a Framework for Selecting and Using Technology. In Effective Teaching with Technology in Higher Education: Foundations for Success. (pp. 77-105). San Francisco: Jossey Bass Publishers
Chickering, A.W. & Gamson, Z.F. (1987). Seven Principles for Good Practice in Undergraduate Education. American Association for Higher Education Bulletin, 39 (7), 3-7.
Harun, N., (2011, July 12) Smartphone penetration in Asia set to boom. NielsenWire. Retrieved from http://blog.nielsen.com/
Nielsen (2011, November 10) Surging internet usage in South East Asia reshaping the media landscape. NielsenWire. Retrieved from http://blog.nielsen.com/
Panettieri, J. (2007). Addition by subtraction. University Business, August, 58-62.
Schaffhauser, D. (2010). Mission Critical: Selecting the Right LMS. Campus Technology, 23, 11 p34-43. Retrieved from http://campustechnology.com/
Smyth, T., Kumar, S., Medhi, I., & Toyama, K. (2010). Where there’s a will there’s a way: mobile media sharing in urban india. Proceedings of the 28th international conference on Human factors in computing systems, 753–762. Retrieved from http://dl.acm.org/