Platform Evaluation for a Community Organization:
Big Brothers Big Sisters Organization of Canada
Laurie Carswell
Donna-Marie Hamilton
Caitlin Langford
Grant Naylor
Janet Rothney
University of British Columbia
Platform Evaluation for a Community Organization:
Big Brothers Big Sisters Organization of Canada
Big Brothers Big Sisters Organization of Canada (BBBSC) is a non-profit organization that has been providing mentoring programs for youth in Canada for one hundred years (Big Brothers Big Sisters Canada, 2015). Mentoring programs are done through one to one matching of a child with an adult, in school mentoring and community programs. BBBSC establishes local agencies in Canadian cities and towns to meet the mentoring needs in the lives of youth in their local communities. A Board of Directors governs the organization at each local agency, while an Executive Director manages program staff and volunteers carry out the day-to-day programming. Board members and staff of the agency engage in strategic planning, training, and development to ensure that goals of the agency are met and program quality is maintained.
The organization would benefit from a Learning Management System (LMS) to provide training, development and orientation for BBBSC’s staff, volunteers and board members. BBBSC operates on a stringent budget with funds obtained from community fundraising activities and sponsorship. BBBSC handles sensitive information and therefore places a high value on confidentiality. The organization values the time and resources of its sponsors, volunteers, staff, and board members and is seeking an affordable and secure platform for providing ongoing training and development. BBBSC is seeking to utilise the affordances of a LMS to keep its staff, board members, and volunteers abreast with current strategic planning strategies and new fund development matrices for not-for profit organizations. An LMS will help all BBBSC organizations across the country to engage with each agency, while developing innovative strategies to cope with the increased demands from sponsors to plan for the future. A LMS will help BBBSC to develop and demonstrate practices that are sustainable for program growth and funding in the future.
Selection of LMS based upon S.E.C.T.I.O.N.S.
Learning Management Systems (LMS) have become very prominent within organizations and educational fields. According to Coates, James, & Baldwin (2005), LMS have grown through the wide range of multimedia and internet developments that arose in the 1990s. Characteristics that define LMS provide affordances that transcend the fields of academia into the world of corporate organizations, including non-profit organizations such as Big Brother Big Sisters of Canada. As such, they provide organizations opportunities for asynchronous and synchronous communication, delivery modules for content related to the training and development of volunteers, and methods for both formative and summative assessment.
With vested interests that span costs and functionalities, it is of vital importance to these organizations that they properly assess and evaluate the success and effectiveness of such LMS. Bates and Poole (2003) suggest that the ever-changing landscapes that are technology integration require decision makers to have a ¨framework or set of criteria to guide them¨ (p. 75). Bates and Poole (2003) developed a successful framework for assessing the integration of technological affordance that fits the model required by organizations to assess the integration of LMS. Similar to the model developed by Nel, Dreyer & Carstens, (2010), S.E.C.T.I.O.N.S. enables organizations to select a LMS based on their specific needs. A rubric following the S.E.C.T.I.O.N.S. model for technology integration would assist in the assessment of various LMS, including Moodle, WordPress, and Eliademy. Accordingly, models for selecting and applying technology need have the following characteristics:
- It will work in a variety of learning contexts.
- It allows decisions to be taken at a strategic level.
- It gives equal attention to educational and operational issues.
- It will identify critical differences between different media and technologies.
- It is easily understood, pragmatic, and cost effective.
- It will accommodate new developments in technology (Bates & Poole, 2003, p. 77).
While the Bates & Poole S.E.C.T.I.O.N.S. criteria model was developed with educational institutions in mind, many of the characteristics described apply to the development of learning modules that support the training of those working within the BBBS infrastructure. S.E.C.T.I.O.N.S. refers to:
(S) Student Demographics– staff, board members, and volunteers taking training courses
(E) Ease of Use and Reliability
(C) Costs – which are of integral importance in a non-profit organization
(T) Teaching and Learning – which we will refer to as Facilitation and Learning including instructional approaches and best technologies
(I) Interactivity – including successful, authentic learning opportunities, which in turn generates enthusiasm for the tasks at hand
(O) Organizational Issues – including privacy and confidentiality
(N) Novelty – integral in the ever-changing world of technology affordance
(S) Speed – including the ability for the LMS to adapt to best meet the needs of all parties involved (Bates & Poole, 2003, p. 79-80)
Rubric Rationale
Technology is in a constant state of flux. As non-profit organizations look to find more cost effective, reliable, and accessible methods of training their staff, alternate learning environment management is necessary. In order to assess the affordances that LMS’s have in providing authentic opportunities within the BBBS organization, developing a rubric provides the most holistic form of assessment.
LMS Assessment Rubric
| Rating | 3 – Strong | 2 – Moderate | 1 – Poor |
| 1) Demographics
Under BBBS: **Staff and Board Members** · Access · Differences Around Learning with Technology |
– access features beyond basic features
– accessible in multiple languages |
– access basic features
– some ability to adapt to cultural needs of clients |
– limited features
– challenging for ESL clients as content only offered in English language |
| 2) Ease of Use
· Orientation · Reliability |
– detailed, and specific set of instructions
– support systems in place – user is quickly able to demonstrate proficiency |
– some technology support will be needed to build and manage course site | – IT support necessary in all aspects of delivering a course |
| 3) Cost
· Development · Maintenance |
– free; open source
– minimal maintenance |
– fees are required
– moderate maintenance |
– costly membership
– constant maintenance |
| 4) Facilitation & Learning
· Synchronous or Asynchronous · Content Coverage |
– built-in assessment tools
– multimedia files can be embedded – collaboration and social media tools available – asynchronous and synchronous learning tools |
– asynchronous and synchronous features
– collaborative tools are difficult to access and use – some multimedia can be uploaded |
– isolated learning in asynchronous format
– no assessment features – text heavy content |
| 5) Interactivity & Implementation
· Collaborative Features · Feedback Opportunity · Ability to sync with other technology |
– collaborative features available
– feedback opportunities exist so clients can easily connect with BBBSC organization – ease in ability to sync with other technology |
– some collaborative features available
– feedback opportunities exist so clients can easily connect with BBBSC organization – some ability to sync with other technology |
– lack of collaborative features available
– difficult for volunteers to connect with BBBSC organization – no ability to sync with other technology |
| 6) Organization
· Affordance for Flow from Module to Module · Privacy and Confidentiality are Addressed |
– tabs are included
– privacy and confidentiality are addressed |
– content hyperlinked
– privacy and confidentiality are somewhat addressed |
– content housed on one document
– privacy and confidentiality are not addressed |
| 7) Novelty
· Authentic Engagement of those Training |
– site can be personalized
– content continuously updated |
– some differences from course to course in look and design format | – rigid format from course to course
– site cannot be personalized |
| 8) Speed
· Availability for Quick Access · Work Through at Self-Regulated Pace |
– reliable and quick access tech support infrequently required | – inconsistent site reliability
– tech support often necessary |
– poor access; site often down
– rarely runs independent of tech support |
The following assessment of the three LMS platforms Moodle, WordPress and Eliademy have been rated using the rubric above:
| 1) Staff and Board Member Demographics | Moodle | WordPress | Eliademy |
| Affordances | – access/resources available in multiple languages
– supported by a team of International developers (Moodle.org), which provides tutorial and IT services |
– available to users without institutional affiliation
– web-based platform |
– supports various languages |
| Limitations | – clients may not have strong technical background: some proficiency with technology is required | – multilingual sites can only be accommodated with additional plugins
– some proficiency with web authoring and site management are required |
– some proficiency with technology is required |
| Rating: | 3 | 2 | 3 |
| 2) Ease of Use and Reliability | Moodle | WordPress | Eliademy |
| Affordances | – web-based
– familiar textbook format; learners scroll through their material from top to bottom and access digital pages via the table of content. -mobile friendly with app implementation |
– rated very highly for usability by users and instructors
– plugins, themes and widgets provide sophisticated interfaces without large development costs – mobile friendly |
– web-based
– users can create their account by signing up with a social network account like Facebook, or through Email – each course can be assigned to a variety of categories for better sorting and usage – users can also create new topics within a course, where they can add pictures, files, YouTube videos – starting a discussion on a course or topic is easy – users can also add participants by email or by sharing a link – allows for Moodle backup files to be restored to their system -mobile friendly |
| Limitations | – lacks tabs that allow students to move freely through material | – if multiple plugins and widgets are implemented, IT support and maintenance is more likely to be needed | – learning how to use tools may require IT support |
| Rating: | 3 | 3 | 3 |
| 3) Costs | Moodle | WordPress | Eliademy |
| Affordances | – free to download; open source
– no licensing fees |
– much of WP is free and open source; including themes, plugins and widgets (password protecting, CMS file configuration, comment threading and moderation) | – free to download; open source
– no licensing fees |
| Limitations | – in-house IT support necessary | -web hosting, occasional IT support, some plugins/widgets would all charge separately; more difficult to manage | – premium Rate – 1Euro per users, per month |
| Rating: | 3 | 2 | 3 |
| 4) Facilitation and Learning | Moodle | WordPress | Eliademy |
| Affordances | – videos can be uploaded to the site; videos are large files and difficult to share otherwise
– clients can upload assignments and answer questions in a quiz – assessment reporting and data collecting features |
– can be configured to direct learners through content or allow discovery at learner’s pace
– variety of resources can be embedded or linked |
-videos can be uploaded
– interactive discussion boards – task management and grading tools – customizable course completion certificates Premium Package: – Real time videos, instant messaging, presentation sharing – ability to annotate any document in real time – ability to make webinars, conferences up to 50 participants – learners are visible at a glance, who can be assigned to groups to manage enrolments – access to analytics for each team member |
| Limitations | – challenging for clients to collaborate
– instructors unable to dialogue with clients in live discussion |
– assessment tools require additional plugins | – some features require fees |
| Rating: | 2 | 2 | 3 |
| 5) Interactivity | Moodle | WordPress | Eliademy |
| Affordances | – multimedia features such as Internet links, audio files and video files can be uploaded
– Google features can be integrated into Moodle via plugins |
– plugins can improve communication functions
– comments can be rated, threaded and moderated |
– multimedia features such as Internet links, audio files and video files can be uploaded
– allows users to sync with their Evernote account so that they can use their notes in the courses – can integrate Google files |
| Limitations | – IT support would be necessary to support uploading multimedia features
– lacks social media features |
-synchronous/asynchronous communication is not the main function of WP
– no discussion boards assignment dropboxes, embedded email functionality or teleconferencing abilities |
– some proficiency with technology is required |
| Rating | 2 | 1 | 3 |
| 6)Organizational Issues
– privacy and confidentiality is necessary for the BBBSC |
Moodle | WordPress | Eliademy |
| Affordances | – can be downloaded to BBBSC server
– password protected by organization |
– plugins are available to password protect
– can be hosted locally to ensure privacy |
private – accessed by invitation |
| Limitations | – IT staff member needed to support server issues that arise | – user perception of privacy may be lacking
– using plugins to protect privacy may be less robust |
-possible cost
– tech support required to monitor and create course(s) |
| Rating: | 3 | 2 | 3 |
| 7) Novelty | Moodle | WordPress | Eliademy |
| Affordances | – multimedia files can be uploaded to the site | – frequent updates keeps sites looking fresh
– can be configured in many ways; sites can be customized to suit the needs of users rather than work around generic LMS formatting |
– multimedia files can be uploaded to the site (ex. Google Drive files)
– invites and certificates can be customized |
| Limitations | – clients are unable to personalize pages | – users may expect a more standardized tool | – pages do not afford personalization |
| Rating: | 2 | 3 | 2 |
| 8) 8) Ability to customize to BBBSC needs | Moodle | WordPress | Eliademy |
| a Affordances | – consistency in format from course to course | – many free plugins, widgets and themes available
– vibrant user community willing to help and innovate |
-30 Day free trial
– you only pay per user so you can start small |
| Limitations | – rigid format of Moodle discourages creative design | – clients may be less familiar with formatting styles
– no user-level customization |
– a lot of initial work setting up an online training program |
| Rating: | 2 | 3 | 2 |
| 9) 9) Speed | Moodle | WordPress | Eliademy |
| Affordances | -content can be added relatively quickly | – very easy to make changes to look and feel
– plugins, themes and widgets updated regularly |
– allows for clients to access features in a timely manner; downloads are quick
– timely integration of features from other platforms |
| Limitations | – content can only be added quickly if user has technical proficiency; less intuitive | – requires IT maintenance to maintain updates | – requires some IT maintenance to maintain updates |
| /// Rating: | 2 | 3 | 3 |
| Overall Rating Score: | 22 | 21 | 25 |
Rubric information gathered from the following sources; Moodle: Moodle.org (2015), SoftwareInsider (2015) WordPress: Braender, Kapp, & Yeras, (2009), Mitchell, (2013), Weiler, (2014), WordPress (2015), Eliademy: Eliademy (2015), EdTechReview (2015), moodle news (2015).
Selection, Synthesis and Rationale
Eliademy appears to be the best fit for BBBSC, as their chief concerns are communication and privacy, both of which are addressed well with Eliademy. On April 23 2015, the e-learning platform Eliademy.com won the Edison Gold Award 2015 for the category Living, Working and Learning. Eliademy’s mission is to democratize education with technology (Open Education Europa, 2015). Eliademy scored 25 points on our rubric, demonstrating appeal to all demographics within BBBSC, scores well on ease of use and reliability, facilitation and learning, organizational needs, and cost (of importance by necessity for a non-profit organization). Eliademy is currently used by non-profit and for-profit companies, as well as educational organizations (see Eliademy’s showcase at: https://eliademy.com/catalog/featured.html), and as such successfully straddles the needs of BBBSC.
Overall, Eliademy has the highest rating based on our S.E.C.T.I.O.N.S. rubric. Eliademy and Moodle had very similar strengths in terms of cost, ease of use and privacy issues; however, Eliademy was rated higher in the teaching and learning category over Moodle. WordPress was quite distinct in strengths from Moodle and Eliademy in the areas of novelty and speed, but rated poorly on privacy and communication concerns. While novelty and speed are essential for social media sites, it is not as important as cost and privacy issues related to the BBBSC organization in terms of training volunteers.
Further to Bates and Poole, Forman (2013) provides insight into the evaluation and selection process that should be taken into consideration if the direction the organization seeks is to implement use of an LMS, or to transform and change their LMS in current use. As such, Forman (2013) describes a five-point approach to the evaluation and selection process.
Forman (2013) suggests organizations pay particular attention to the analysis of needs, requirements, product vetting, evaluation of products, and the selection of the product. Forman (2013) argues, ¨that the process an organization uses to evaluate and select an LMS can have a significant impact on the perceived success of the LMS implementation and the organization’s ultimate satisfaction with the solution¨. Of note to our organization, is a thorough examination and evaluation of the products (#4) conducted through our S.E.C.T.I.O.N.S. assessment prior to the selection of the best fit LMS to meet the goals of the BBBSC organization. Eliademy should be sustainable for the foreseeable future in that it is currently a thriving, growing community, and fits in with trends in LMS development away from large enterprise solutions like Connect and BlackBoard, as per Porto, 2015 and Spiro, 2015.
Conclusion
Big Brothers and Big Sisters of Canada has maintained its status as a very prominent non-profit organization within Canada, providing mentorship services and opportunities for volunteers. Given the ever changing world of technology and with the affordances that LMS’s offer including their abilities to provide training for its board members, staff and volunteers at opportune times across great distances, BBBSC would benefit greatly from the tools that LMS’s bring to an organization. Given the opportunity to assess Moodle, WordPress, and Eliademy through S.E.C.T.I.O.N.S., Eliademy is the strongest LMS that best meets the needs of a secure platform for the successful training and development of those involved in the organization. Using the Eliademy platform is an integral step in BBBSC being sustainable in their training and development program. Engaging in this technology will help BBBSC to maintain their prominence as a leading organization in the not for profit sector.
References
Bates and Poole. (2003) A Framework for Selecting and Using Technology. In Effective Teaching with Technology. San Francisco: Jossey-Bass. Pages 75-105.
Bates, T. (2014). Chapter 4: The ADDIE model. In Teaching in a Digital Age: Guidelines for designing teaching and learning for a digital age. Retrieved from: http://opentextbc.ca/teachinginadigitalage/
Big Brothers Big Sisters Canada (2015) Retrieved from: http://www.bigbrothersbigsisters.ca/en/home/mentoringprograms/default.aspx
Braender, LM., Kapp, C.M., Yeras, J. (2009). Using Web technology to teach students about their digital world. Journal of Information Systems Education, 20 (2).
Coates, H., James, R., & Baldwin, G. (2005). A critical examination of the effects of Learning Management Systems on university teaching and learning.Tertiary Education and Management, 11, (1), 19-36.
EdTechReview. “Eliademy – Free Online Courses Creation”. (2015). Retrieved from
http://edtechreview.in/reviews/282-eliademy-free-online-courses-creation
Eliademy. (2015). Retrieved from https://eliademy.com/en#sthash.locDv8Px.dpbs
Eliademy. “People in Need, welcome to Eliademy”. (2015). Retrieved from
https://eliademy.com/blog/tag/nonprofit/
Forman, S. (2013). Five Steps to Evaluate and Select an LMS: Proven Practices. Retrieved from: http://www.learningsolutionsmag.com/articles/1181/five-steps-to-evaluate-and-select-an-lms-proven-practices
Mitchell, R.L. Choosing an open-source CMS, part 3: Why we use WordPress. Computerworld. (Feb 2013). Retrieved from: http://www.computerworld.com/article/2495111/open-source-tools/choosing-an-open-source-cms–part-3–why-we-use-wordpress.html
Moodle: LMS software – ratings, reviews, and reatures. Retrieved from
http://lms.softwareinsider.com/l/226/Moodle
Moodle: open source learning platform. (2015). Retrieved from https://moodle.org/
Moodle news. “Another Look at Eliademy, a cloud-based alternative to Moodle”. (2013). Retrieved from http://www.moodlenews.com/2013/another-look-at-eliademy-a-free-alternative-to-moodle/
Nel, C., Dreyer, C., & Carstens, W. A. M. (2010). Educational technologies: A classification and evaluation. Tydskrif vir letterkunde, 35(4), 238-258.
Open Education Europa. “Eliademy won the Edison Gold Award 2015 for the category
Living, Working and Learning”. (2015). Retrieved from
http://www.openeducationeuropa.eu/en/news/
eliademy-won-edison-gold-award-2015-category-living-working-and-learning
Porto, S. (2015). The uncertain future of Learning Management Systems. The Evolllution: Illuminating the Lifelong Learning Movement. Retrieved from
http://www.evolllution.com/opinions/uncertain-future-learning-management-systems/
Spiro, K. (2014). 5 elearning trends leading to the end of the Learning Management Systems. Retrieved from http://elearningindustry.com/5-elearning-trends-leading-to-the-end-of-the-learning-management-system
Weiler, N.B.. How to use WordPress as a Learning Management System. (April 2014) Retrieved from: http://www.elegantthemes.com/blog/tips-tricks/how-to-use-wordpress-as-a-learning-management-system