Welcome to Do It Yourself Learning Analytics
In this section we will be taking you on a journey to explore how one can run a simple analysis on ones own classrooms grades. We will be using Google Forms/Spreadsheets to help us gather data and a combination of a script called Flubaroo and excel to do our analysis.
Below is a tutorial showing you how to setup a quiz and answer sheet. In addition, the finished products, both the quiz and results are available to you.
Tutorial:
Finished products:
Results (there are 2 tabs on the bottom of the chart, click on the one entitled Grades to see Flubaroo’s results)
Running Your Own Data Analysis
Below are a series of steps you can follow to complete a data analysis on a post-assessment quiz that you would give your class before starting a unit.
Laying the groundwork:
- Examine curricular target expectations
- Create a post-assessment
- Administer the post-assessment BEFORE teaching
- Pick a proficiency level
Collect and chart data:
- Use the excel sheet below, input your data and examine student results. Please note there are 3 tabs in the excel sheet that allow you to input data for different situations
- Individual Teacher
- Multiple Teachers teaching the same class
- Multiple Teachers with multiple items
- proficiency tabulators
Analyze strengths and obstacles:
Now that you have your data, you can use the questions below to help you reflect on what your data is telling you.
- On what areas of the assessment did the students perform well? (most frequently correct?)
- On what areas of the assessment did the students perform poorly? (most frequently missed?)
- To what do you attribute their lack of success in these areas? (type of instruction, time spent, feedback, previous formative assessment?)
- In what areas does your partner or other periods have similar results?
- In what areas does your partner or periods have different results?
- Are there any implications or changes that immediately come to the forefront with regards to future instruction? What will you do differently based on the results obtained?
Comments
- Have you done your own data analysis? If yes, please share your experience
Looking forwards to everyones experience in this section
I love Google Forms and have used it for a number of reasons. Most recently I organized a PD day based on information collected from survey. I surveyed all participants about their use of technology, comfort level and their experiences that I based the days activities on. You feel so much more confident going forward having information based on sound data. You can see the results here: https://docs.google.com/a/ecoleparc.ca/forms/d/1kr51NzFK4Ke6Z_xRzslWfmGWq_p6xqiw2alopgEeF6U/viewanalytics
We do exactly the same thing for our PD and have found the data lets us pinpoint certain areas we need to work on. Recently, we decided to give differentiated tech PD (we do it for our kids, so why not teachers?), and again the data we received let us easily provide a wide range of PD opportunities.
Maybe it’s just me, but I have a problem with this type of quiz, as it does not really apply to my line of work. In EAL and workplace training, the emphasis is on language proficiency and productivity. We don’t use grading or scores. Rather, we assign general tasks and evaluate concepts like ability to address the task, develop ideas grammar etc. In his video under Best Practices, Shoettler discusses how feedback is key. Technology hasn’t developed to a point yet where it can give qualitative feedback on a piece of writing and speaking, pointing out students strengths and weaknesses. Analytics would be useful if it could collect data on how many students have trouble expressing main ideas, developing topics, most common grammar errors across a class, where vocabulary strengths and weaknesses are,and what they can do to get to the next skill level or achieve greater proficiency etc. But this technology still isn’t there, and I would have to enter in the data manually for each student, which I honestly don’t have the time, nor am I paid to do. Then there is the fact, that the majority of my students don’t write/type/record on computers, nor have access to them during class time.
Moreover, there are serious questions as to the validity/effectiveness of using MCQ and other right/wrong testing formats for receptive skills. In fact, recent research shows that multiple choice and true false tests do nothing to assess or improve learner reading and listening comprehension.
I could see this as a useful tool for learner needs and self-analysis, provided they were familiar with the technology or had access to it which unfortunately is a negative in many of my teaching contexts. I could also see this as a great task for learners to collaborate and create a survey, similar to what Jetz has done and use it to tabulate and analyze results.
I totally agree that this example does not necessarily follow best practices, unfortunately the reality right now is that the majority of schools and especially standardized test use multiple choose questions. So now we can use this type of data analysis to fight against the use of such archaic practices?
With reference to technology not being able to give qualitative feedback, have you had a chance to look at http://turnitin.com ? Although it doesn’t provide automatic feedback, teachers can now leave voice messages along with their written feedback. I know our English department is using a common rubric about some of the issues you mentioned (trouble expressing main ideas, developing topics, most common grammar errors across a class) and are collecting data about these points so they can improve their scaffolding through middle and high school.
Really well put. The lack of qualitative feedback is a key issue with bubble quizzes. I suppose LA Has the capacity to allow us to compile data and make judgements from said data. Presumably, as stated in the introduction, this requires the teacher to play the role of statistical analyzer, thus providing qualitative feedback based on the data that has been compiled. The unfortunate fact is the great majority of teachers are so far unprepared to utilize data in this manner, so, once again, your initial statement rings true.
I have also used Google forums before and have performed similar tests with my students but within the Edmodo website instead. I really like doing pre-tests this way as well as mini quizzes throughout units to check understanding. I especially like the flexibility of Google Forums, ease of accessibility and the fact that the students can see how they did compared to others.
The downfall of these types of tests, remain that they are limiting and like any test, the results will not represent the whole of the students knowledge or ability, but the data can be useful to guide learning and teaching.
Have you ever tried using these types of test as a pre-assessment before you start a unit? Some teacher at my school are starting to run simple pre-assessments to help them plan what areas they will be focusing on for their next unit. I know the test doesn’t give a complete assessment of a students abilities, however its fast and something is better than nothing.
Thanks for this intro and the example (Jetz66)! I had no idea this feature existed.
yes, Thank you! I had no idea it existed either. During this course and others in ETEC, I am constantly amazed at the number of applications, programs, and education technology that’s out there and am in awe with how informative everyone is. How does everyone find out about this stuff? I feel like medical education is light years behind everyone else. I am fighting feelings of inadequacy because I am expected to be a clinician (something I’ve been studying for 15 years so I’m adequately trained) AND an “educator” who has no background in education. It just doesn’t make sense!
Sorry for going off on a tangent. . . . Basically, I wanted to say thank you.
I’m not sure if the Quiz is generating grades. I was able to see that Flubaroo had accepted my quiz responses, but the grade tab did not show any data.
Did you make your own quiz or are you using the pre-made one from above?
Great add-on and walk through. I would love to try it out sometime. I’m not currently working as a teacher so that opportunity may have to wait.
However, while working in the middle east our ELT program with McGill was heavily test based. Weekly quizzes, midterms, final exams and lab grades were all compiled and weighed, eventually determining the students fate in passing the course. We were generally teaching very large classes, and only for 3 month stints, so knowing the students individual strengths and weaknesses was a bit of guesswork. As a result, a fair but of data analysis went into looking at areas of improvement, for the student, teacher, and program. At points throughout the program we were able to cross-analyse data sets and determine:
1. Issues with grading (ie. one teacher grading the essays far higher than others).
2. Concerns over the difficulty of tests (allowing us to apply adjusted grades so that too many students wouldn’t fall below the fail line).
3. Areas of improvement in methodology (for example, vocabulary and reading assessments rank comparatively higher than listen and speaking may have revealed too high and emphasis on textbook work)
However, it also sparked plenty of controversy, and competition among the staff. Overall, I think the data may have helped us as teachers in such a ridged, test based system, but I’m not convinced it helped the students as learners. That being said, “students as learners” was probably never the primary focus of the program anyways.
Brendan,
Interesting experience you just described. I could see your situation causing a lot of tension between teachers and the creation of class list a very interesting experience. I wonder how much competition is a good thing and when it just causes too many problems at work and then fudging of data starts so one secures a bonus or even their job.
I began using google forms in my resource room last year and thought it was really successful. I really enjoy having all of the student responses compiled simply within Google Drive. As many of the students I work with have written expression issues, the different response options were so helpful. I noticed that students generally enjoyed demonstrating their learning in this way and were always proud of themselves after completing a “quiz.” It terms of analysis, I could see immediately which questions were giving students issues. This was so valuable for me and allowed to me to reflect on the learning activities and approaches that went in to that topic. Thanks for the Flubaroo demo! I will definitely try using that as well!
I tend to write anecdotal reports for my students on modified program (not meeting grade level outcomes) and so data analysis tends to lean towards qualitative assessment. Although some of the grading features demonstrated may not be as applicable to my situation, I still benefit greatly from having all of the data stored in one place in a way that makes comparing and contrasting very simple. I agree that when qualitative data is being analyzed that the educator really must continue to perform the analysis themself and cannot rely on technology to perform this function.
Like others, I am amazed at all the tools that google drive offers. Thanks for the introduction to Flubaroo as well.
I can see the benefit to using these tools in preparing PD for colleagues.
It also has me thinking about the potential use in ‘customer satisfaction’ surveys we very occasionally do with our patients. Not much time for that in today’s healthcare setting….This would make it so much easier than a mailed out evaluation that would have to be mailed back again…..sounds pretty archaic, doesn’t it? We have not done a survey in 8+ years