Delivery Platform Evaluation | ETEC565A ePortfolio

Precis

BCCampus is a publicly-funded agency that offers teaching, learning and educational technology support to the 25 post-secondary institutions in British Columbia and 1 in the Yukon. As part of its shared services, BCcampus has been running two LMS platforms; one open-sourced (Moodle) and the other vendor based (D2L) whose contract will expire shortly. With current restructuring, BCCampus will be losing half of its tech support team in 3 months, and therefore the possibility of running 2 separate LMS platforms is no longer an option. BCCampus will need to decide which LMS they wish to proceed with across the board. However, as BCCampus is known for its leadership in innovation, they are open to the idea of selecting an entirely new platform. As employees at BCCampus, we have been asked to create an evaluation rubric to help in the decision making process. We have designed the rubric to help select an LMS that will fit with the needs, the vision and mandate of BCCampus: to connect, collaborate and innovate. The selection process must also recognize the LMS’ ability to conform and to adapt to the geographically and culturally diverse needs of BCcampus’ partner institutions.

Rubric

Criteria	Fair	Good	Excellent
1. Access
LMS System requirements	The LMS requires considerable system upgrade to integrate with the current BCCampus system	The LMS requires minor system upgrade to be compatible with the current BCCampus system	The LMS is fully compatible with the current system used by the BCCampus
Devices	The LMS can be accessed with limited functionality on mobile phones and tablets	Certain features of the LMS may not be accessed on mobiles phones and tablets	The LMS can be accessed using a full range of devices with include mobile phones, tablets, desktop computers
System requirements for end users	The LMS is designed to be used with a particular browser, or requires plugins, or special browser configurations.	The LMS supports the use of more than one browser. May require downloadable app so LMS can run on mobile and other devices.	The LMS runs optimally on Windows Internet Explorer, Google Chrome and Mozilla Firefox and requires no special set-up requirements.
2. Support
Support available for the interconnectedness of all partner institutions especially as they all need to be at the front of their game to attract students (to exist)	Training support is only available for a time limited period after the LMS is initially set up. Online resources are limited.	Training support is available by phone only during specific hours of the week. There are some online resources.	Training/ support is available whenever the user has need. Online support are accessible 24/7.
Training and maintenance support for educators	IT support is available during regular office hours by email and phone.	IT support is available by email or phone	IT support is available 24/7 by email, phone or live chat
IT support for students	Student can access IT support by email only; response will be only during office hours	Students can access IT support by email or live chat during office hours.	Student can access IT support 24/7 by email or live chat
3. Functionality
Provide a well thought out interface for all users of the LMS	LMS is relatively easy to use	LMS is easy to use and possesses some intuitive features	LMS has superior scope and sequence, and exceptional insight into the needs of the users
Layers and applications of privacy and security that fit necessary applications (medical, legal, personal)	LMS provides adequate security for many of the applications	LMS provides adequate security for most of the applications	LMS provides excellent security for all the applications
LMS innovations adapt and keep pace with future learners	Limited design options. Or may require customized work by vendor provider or open source LMS specialist.	Some “themes” available to change, look and structure of user interface.	LMS interface look and feel can be customized by local administrators. The structure and organization is intuitive, and adaptable to fit needs.
4. Cost
Are the one-time costs associated with the set-up, implementation, staff training if required, or migration of content to LMS feasible for BC Campus?	Implementation costs are above budget.	Costs are in-line with last years budget forecasts.	Costs are well under budget.
LMS maintenance (server back-ups, updates, course back-ups, course edits, archiving, staffing, etc) and upkeep costs within budget?	Regular maintenance will require more financial resources	Regular maintenance costs are the same.	Regular maintenance costs are reduced.
Are future (next 4-5 years) costs likely to remain constant or are they fairly unpredictable and will depend on other parties (i.e. vendor).	No or limited stability in cost for stated time frame. Possibility of increases within 4-5 years.	Cost is stable for the stated time frame.	No foreseeable cost increases in the future.
5. Customization
LMS adaptable to changes in technology and user driven initiatives	WYSIWYG LMS is adequate in its current services	LMS has features that are progressive, but many of those changes were the result of user suggestion	LMS is structurally agile and real-time responsive to changes in user needs and changes in technology
Communication modalities adjust/tailored to suit institutions/educators	The LMS has integrated new technology (e.g. social media) into their services, yet these were integrated poorly and with great technical difficulties	The LMS has successfully integrated new technology (e.g. social media) into their services, yet the integration was not perfectly seamless	The LMS has successfully and seamlessly integrated new technology (e.g. social media) into their services
Adaptable to customized learner paths (self-directed, adaptive, personally curative in the documenting and archiving kind of way, providing avenues of formative assessment)	LMS has some features, but is limited in scope	LMS has features that blend what they can offer with what can be inserted or included by other providers	LMS reflects in ways that are superior and extensive; how information is actually interacted with by the user.
6. Organizational Requirements
A/synchronous communication mediums that facilitate instructor-student, student-student, and student-material/resource interactions	Basic offerings that facilitate simple communication avenues	LMS has a variety of formats for facilitating student interactions	LMS design is so effective that communication is easily and readily navigated by all users.
Is the design of the LMS effective in its cognitive design?	Spatial and segmented cues are adequate	LMS has features that pre-load the user prior to more detailed training	LMS design elements are so effective that learners are caught up by learning content rather than medium.
Systemic alignment of LMS with BCcampus and its affiliates	The LMS fits well with the current mandate of BCCampus, requires some minor adjustments	The LMS fits very well with the current mandate of BCCampus, requires few minor adjustments	The LMS fits perfectly with the current mandate of the BCCampus, no additional adjustments

Rationale

Our team has based its rubric on elements of Bates work as well as Chickering and Ehrmann’s seven principles of good practice. We believe the structure and specifics of Bates ‘SECTIONS’ together with practical guidance of the principles, provided the foundation needed for a well grounded evaluative tool. As Coates points out however, “Decisions about university teaching and learning should not be restricted to checklist evaluations of technical and organizational factors” (2005). With that in mind we also focused the language of our rubric with forward thinking. What is crucial to keeping BCcampus relevant to the geographically and culturally diverse population it serves, and innovatively necessary to its partner institutions? What will help the institution continue its mandate to connect and collaborate by not only keeping pace with learning communities, but expanding with leading-edge vision? We understand this will not be the only tool an evaluation team uses, but believe that the six areas covered will provide a well-rounded summary to the committee.

References

Bates, T. (2014). Teaching in digital age http://opentextbc.ca/teachinginadigitalage/ (Chapter 8 on SECTIONS framework)

Chickering, A. W., & Ehrmann, S., C. (1996). Implementing the seven principles: Technology as lever. American Association for Higher Education Bulletin, 49(2), 3-6. Retrieved from http://www.aahea.org/articles/sevenprinciples.htm

Coates, H., James, R., & Baldwin, G. (2005). A critical examination of the effects of Learning Management Systems on university teaching and learning. Tertiary Education and Management, 11,(1), 19-36. http://link.springer.com/article/10.1007/s11233-004-3567-9
Spiro, K. (2014). 5 elearning trends leading to the end of the Learning Management Systems. Retrieved from http://elearningindustry.com/5-elearning-trends-leading-to-the-end-of-the-learning-management-system

Or Version 2.0

ETEC 565 Group 1 – BCcampus

Group members:

Danielle Couture

Keri Fleming

Edwin Fong

Parmdip Gill

Colleen Huck

Précis

BCcampus is a publicly-funded agency that offers “teaching, learning, and educational technology support“ to the 25 post-secondary institutions in British Columbia and one in the Yukon. As part of its shared services, BCcampus has been running two LMS platforms; one open-sourced (Moodle) and the other vendor based (D2L) whose contract will expire shortly. With current restructuring, BCcampus will be losing half of its tech support team in three months, and therefore the possibility of running two separate LMS platforms is no longer an option. BCcampus will need to decide which LMS they wish to proceed with across the board. However, as BCcampus is known for its leadership in innovation, they are open to the idea of selecting an entirely new platform. As employees at BCcampus, we have been asked to create an evaluation rubric to help the BCcampus leadership team in the decision making process. We have designed the rubric to help select a LMS that will fit with the needs, the vision and the mandate of BCcampus: “to connect, collaborate and innovate“. The selection process must also recognize the LMS’ ability to conform and adapt to the geographically and culturally diverse needs of BCcampus’ partner institutions and the thousands of post-secondary students across the country.

Criteria	Fair	Good	Excellent
1. Access
LMS system requirements (compatible with the current system and adaptive to future changes).	The LMS requires considerable system upgrades to integrate with the current BCcampus system.	The LMS requires minor system upgrades to be compatible with the current BCcampus system.	The LMS is fully compatible with the current system used by the BCcampus.
Devices.	The LMS can be accessed with limited functionality on mobile phones and tablets.	Certain features of the LMS may not be accessed on mobiles phones and tablets.	The LMS can be fully accessed using a full range of devices including: mobile phones, tablets, and desktop computers.
System requirements.	The LMS is designed to be used with a particular browser, or requires plugins, or special browser configurations.	The LMS supports the use of more than one browser. May require a downloadable app to run LMS on mobile and other devices.	The LMS runs optimally on Windows Internet Explorer, Google Chrome, and Mozilla Firefox and requires no special set-up requirements.
2. Support
Support available for the interconnectedness of all partner institutions.	Training/ support is only available for a time limited period after the LMS is initially set up. Online resources are limited.	Training/ support is available by phone only during specific hours of the week. There are some online resources.	Training/ support is available whenever the user has need. Online support is accessible 24/7.
Training and maintenance support for educators.	IT support is available during regular office hours by email or phone.	IT support is available by email or phone.	IT support is available 24/7 by email, phone, or live chat.
IT support for students.	Student can access IT support by email only; response will be only during office hours.	Students can access IT support by email or live chat during office hours.	Student can access IT support 24/7 by email or live chat.
3. Functionality
Provides a well thought out interface for all users of the LMS.	The LMS is relatively easy to use.	The LMS is easy to use and possesses some intuitive features.	The LMS has superior scope and sequence and exceptional insight into the needs of the users.
Layers of privacy and security that accommodate various applications (medical, legal, personal).	LMS provides adequate security for many of the applications.	LMS provides adequate security for most of the applications.	LMS provides excellent security for all the applications.
LMS innovations adapt and keep pace with future learners.	Limited design options. May require customized work by vendor, provider or open source LMS specialist.	Some “themes” available to change look and structure of user interface.	LMS interface look and feel can be customized by local administrators. The structure and organization is intuitive, and adaptable to fit needs.
4. Cost
Initiation/migration costs (set-up, implementation, staff training).	Implementation costs are above budget.	Costs are in-line with last year’s budget forecasts.	Costs are well under budget.
LMS maintenance (server back-ups, updates, course back-ups, course edits, archiving, staffing).	Regular maintenance will require more financial resources.	Regular maintenance costs are the same.	Regular maintenance costs are reduced.
Potential future costs (next 4-5 years).	No or limited stability in cost. Possibility of increases within 4-5 years.	Cost is stable for the stated time frame.	No foreseeable cost increases in the future.
5. Customization
LMS is adaptable to changes in technology and user driven initiatives.	The LMS is adequate in its current services.	The LMS has features that are progressive, but many of those changes were the result of user suggestions.	The LMS is structurally agile and responsive to changes in user needs and developments in technology.
Communication modalities.	The LMS has limited integration of new technology into their services; tools such as social media are not capitalized on.	The LMS has integrated new tools and technology, such as social media, yet it does not enhance the user’s experience.	The LMS has successfully and seamlessly integrated new tools and technology, such as social media, that enhances the user’s experience.
Ability to customize learning paths (documentation, archiving, formative assessment).	The LMS has some features, but is limited in scope.	The LMS has features that blend what they can offer with what can be inserted or included by other providers.	The LMS reflects in ways that are superior and extensive how information is actually interacted with by the user.
6. Organizational Requirements
A/synchronous communication mediums that facilitate interactions between: 1) instructor – student 2) student – student 3) student – resources	The LMS has basic offerings that facilitate simple communication avenues.	The LMS has a variety of formats for facilitating student interactions.	LMS design enables easy and effective communication options all interaction types.
Cognitive design.	The spatial and segmented cues of the LMS are adequate.	The LMS has features that pre-load the user prior to more detailed training.	The design elements of the LMS are effective so that learners are focused on the learning content rather than the medium.
Systemic alignment of LMS with BCcampus and its affiliates.	The LMS fits well with the current mandate of BCcampus. The LMS requires some minor adjustments.	The LMS fits very well with the current mandate of BCcampus. The LMS requires few minor adjustments.	The LMS fits perfectly with the current mandate of the BCcampus. No additional adjustments are required.

Rationale

Our LMS evaluation rubric is based on elements of the Bates SECTIONS model as well as Chickering and Ehrmann’s seven principles of good practice. As Coates points out, “decisions about university teaching and learning should not be restricted to checklist evaluations of technical and organizational factors” (2005). With that in mind, we did not limit our rubric to standard evaluation criteria, rather we designed the rubric around the specific needs of BCcampus; developing six high-level criteria encompassing the primary concerns of the agency: Access, Support, Functionality, Cost, Customization, and Organizational Requirements.

With 25 affiliates, the importance of versatility without restrictions is paramount. A system that can be accessed by every institution, device and end user, without compromise to functionality, will keep BCcampus at the forefront of innovation. Further, the BCcampus IT support will be reduced in three months making system compatibility an important requirement.

Secondly, it is important that the selected LMS provides support to all users. IT support must be available for BCcampus, the institutions, and the individual users, as these stakeholders might require different assistance at various times. If teachers and learners are not well supported then there is a high risk that the LMS may not be used at all (Bates, 2014).

The functionality of the LMS is an integral criteria for BCcampus to maintain their leadership in innovation by empowering the current and future learners through intuitive user interfaces. Furthermore, the functionality should not compromise the security and privacy of its users in order for open discussions to occur without consequences (Bates, 2014).

BCcampus uses an innovative collaborative model that allows various post-secondary institutions to share resources and costs (BCcampus, 2013). Participating institutions can access the resources and technologies they would not be able to individually. BCcampus must have information about the initial implementation, and future LMS maintenance costs in order to budget effectively. This in turn, will affect the participating institutions and their budgeting.

As Spiro mentioned, students are increasingly taking charge of their own learning; gone are the days of one-size-fits-all courses (2014). This mentality calls for a LMS platform that is customizable – allowing students and teachers alike, the opportunity to curate their own learning path. Further, BCcampus is looking for a long term LMS, meaning it has to be adaptable and able to integrate with new technologies in order to meet the needs of its diverse student population.

Lastly, BCcampus provides a leading-edge collaborative interface. As such, the LMS it uses must have strong cognitive design in its structural components, align with the mandates of its partners, and provide exceptional modalities for communication. We understand this will not be the only tool an evaluation team uses, but believe that the six areas covered will provide a well-rounded summary to the committee.

References

Bates, T. (2014). Teaching in digital age http://opentextbc.ca/teachinginadigitalage/ (Chapter 8 on SECTIONS framework)

BCcampus. (September 2013). Annual Report 2012-2013, Strategic Plan 2013-2016, Retrieved from: http://bccampus.ca/files/2013/10/2013-ar-stratplan.pdf