One of the things I did while on administrative leave from the Centre for Teaching, Learning, and Technology is to review mission, vision, and values statements, as well as strategic goals and plans, from other Canadian centres for teaching and learning.
I focused on Canada partly because that’s where I’m situated, and also because there has recently been a book published about similar things in U.S. centres for teaching and learning (Wright, Centers for Teaching and Learning), and the post-secondary system in these two countries is fairly similar so this could possibly be an interesting point of comparison to see if there are any significant overlaps or differences. It would be even more interesting to review similar points for centres in other countries, but that’s for another time.
I thought I’d use this opportunity to test out a few generative AI platforms and tools to see whether they may or may not be helpful in this work. Short answer: either I’m doing it wrong (highly possible!) or things just aren’t quite there yet to be super helpful. I’ll explain more below.
Note: long post follows! I decided to put several different attempts all together into one post, which makes it very long. And also note: I did all this in mid-December 2024, but only got around to finalizing this post in early Jan. 2025. What worked/didn’t work as of that time may change very quickly!
Google’s Gemini 1.5 with Deep Research
First attempt
As I was thinking about this project, Google came out with a new version of Gemini, with “Deep Research.” I signed on for a free month of Gemini just to try it.
You provide it with a research question or topic, it generates a plan to search the web for information, and then produces a report. You don’t have control over the sections of that report; it just produces something that attempts to answer your question/instructions.
I didn’t have super high hopes for such an early-stage product, plus issues with generative AI making things up, etc., but I gave it a go.
Here was one of my early prompts (yes, it is probably bad…I am learning through this process that I should have paid more attention to prompting advice I heard in the past):
Please find web pages for centres of teaching and learning in post-secondary institutions in Canada that have mission, vision, and/or values statements for the centres, and create a table with the universities as rows and mission, vision, and values as columns. Also add a column with the URL for the web pages that have this information.
It came up with the following research plan:
This took quite a long time…so long I gave up and thought it would not work. It said the report would be ready in a few minutes but it was certainly more than that. I can’t tell how long b/c though there’s a time stamp on the report generation in the interface, there isn’t one on the prompt or when it started doing the research. I just know that I ended up cancelling and retrying a few times because it had been at least an hour.
One of the issues is that it was searching a LOT of websites. As noted in the research plan above, it was searching every post-secondary institution in Canada and looking for a centre for teaching and learning, then searching those sites for mission, vision, and values statements. That’s a lot. It kept getting hung up, and I understand why.
Second attempt
So I refined the prompt to a new version:
Please find web pages of centres for teaching and learning in post-secondary institutions in Canada that have mission, vision, and/or values statements. Find no more than 50 websites. Separate mission, vision, and values, and compare similarities and outliers among them.
The research plan it came up at first with didn’t limit to 50 websites, so I had to ask it to refine the research plan and add that in. Here’s the revised research plan:
- (1) Find up to 50 centres for teaching and learning in post-secondary institutions in Canada.
- (2) From the list in (1), find the websites for these centres.
- (3) From the list of websites in (2), find the mission, vision, and/or values statements.
- (4) Separate the mission, vision, and values statements into individual lists.
- (5) Compare the mission statements, vision statements, and values statements separately to find similarities and outliers.
- (6) Create a report summarizing the findings, including:
- (a) Common themes and trends in mission, vision, and values statements.
- (b) Outliers or unique statements that stand out.
- (c) Comparative analysis of similarities and differences.
- Analyze results
- Create report
And here is the report it generated for the second prompt. Once again I added the header info and the prompt. This is somewhat better in that it really did do a comparative analysis of mission, vision, and values for the centres, but as is evident from the table at the end, there are a lot of centres that either didn’t have mission, vision, or values statements, or that Gemini didn’t pick up on. For example, I know that Dalhousie University’s centre does have a list of values on its web page, but the table at the end of the report says that is “not available.”
So the analysis it did was, of course, only as good as the data it pulled, which was not great.
ChatGPT early attempts
I decided to go looking for the data itself and then work to build the analysis on top of better data. Here is what I first asked ChatGPT 4o (FYI: I have a paid subscription to ChatGPT Plus):
Please find web pages of centres for teaching and learning in Canadian post-secondary institutions, that have mission, vision, or values listed.
It found four such web pages and listed the mission, vision, and values statements, with links to each page. Seemed great! (note, the citations to the web pages don’t work in copy/paste)
Several Canadian post-secondary institutions have Centres for Teaching and Learning (CTLs) that articulate their mission, vision, and values. Here are some examples:
1. MacEwan University:
- Vision: To be a transformative leader in teaching excellence, empowering educators to explore and deliver impactful learning experiences.
- Values: Scholarship, Leadership, Growth, Creativity, Community, Empowerment, Adaptability, Excellence.
- Mission: To advance pedagogical practice at MacEwan by providing foundational instructional skills development, coaching, building communities, and developing educational leadership. citeturn0search0
2. University of Manitoba:
- Vision: Every teaching experience at the University of Manitoba is optimized for learning.
- Mission: To provide leadership, expertise, and support for the teaching and learning mission of the University of Manitoba and to advance and advocate for the practice and scholarship of teaching in higher education.
- Values: Collaboration, Excellence, Integrity, Innovation, Reciprocity, Respect, Transformative, Resurgence, Truth, Wholism. citeturn0search2
3. Trent University:
- Vision: To create vibrant, engaged, and sustainable communities of learning, teaching, and research committed to free enquiry and expression.
- Mission: To prepare students to make significant contributions to an increasingly complex world by providing them with a distinctive liberal arts, science, or professionally focused education, enhanced by global perspectives, experiential learning, and interdisciplinary approaches. citeturn0search4
4. University of British Columbia (UBC):
- Vision, Mission, and Values: UBC’s Centre for Teaching, Learning and Technology focuses on professional development in teaching and learning, integration of technology, and scholarly approaches to curriculum and pedagogy. citeturn0search5
These CTLs play a crucial role in enhancing teaching practices and fostering educational excellence within their respective institutions.
It did pretty well with getting the data from these pages, except the one from the CTLT at UBC Vancouver actually has mission, vision, and values laid out separately, and it didn’t find/list those.
At any rate, I thought I’d see if ChatGPT could find some more pages, and I asked a terrible prompt: “Can you find more such web pages?”
It actually did, and listed another 7. I didn’t fact check at the time, but looking over these now it’s clear there was some fabrication here and there, and some missing data (e.g., for the University of Lethbridge Teaching Centre page it listed only a set of goals, whereas there are clearly mission and vision statements provided).
Undaunted, I continued: “Please find more such web pages.” Which it happily did. Except, most of them were repeats of those in earlier lists.
Next prompt: “Please list URLs for webpages with mission, vision, and values for all teaching and learning centres in post-secondary institutions in Canada. Just give the URLs in a list.”
I got a list of a few, and then asked for more, which it also provided, and then more again a third time and asked it to not repeat any it had already listed. By the third round it had stopped finding pages with mission/vision/values and only provided the main pages for the centres themselves.
Well, here’s one mistake: I didn’t use the “web search” function in ChatGPT for the above. So I tried with that, and the following prompt:
Please find URLs of web pages for centres for teaching and learning in post-secondary institutions in Canada, that list their mission, vision, and/or values. Only provide the URLs for pages that have mission, vision, and/or values statements. Ignore centres that don’t have pages with such statements. Provide a list of URLs with the Institution name as well.
It provided a good list, but didn’t give the URLs, only those in-platform links. I asked it to re-do the list and provide the URLs, but got the same thing so I gave up on getting URLs listed out.
Then I asked for more: “Please find more such web pages with mission, vision, and/or values for teaching and learning centres in Canadian post-secondary institutions, and don’t repeat any that you have already listed above. Please include URLs and the names of the institutions.” It provided a few more, and this time it actually wrote out the URLs. Weird.
I tried a third time with the same prompt as the second time and got…repeated items from what had already been listed.
This wasn’t going so well, and I figured a different tactic was needed.
Scraping web pages & using NotebookLM
I had tried out Google’s NotebookLM before, and found it an interesting way to limit the LLM’s analysis to specific documents rather than the LLM’s training base and (in some cases) web searches. Which could hopefully help with fabrication, at least to some degree. But I needed the data on which it could work, so on the advice of a colleague, I asked a couple of LLMs for python code I could use to scrape text from the web pages with centres for teaching and learning’s mission, vision, and values statements.
I got python code from both Claude (free version) and ChatGPT (paid, Plus version) that worked to scrape text from websites. I used the list of web pages from centres for teaching and learning that I got from ChatGPT, plus others from Gemini, plus others from a plain old google web search. I think I got a fairly exhaustive list of web pages from CTL’s in Canada that list vision, mission, mandate, and/or values.
I tried multiple different version of web scraping python code, and asked the LLMs for help when the code was scraping too much or too little. I tried to get just the main body text from the web pages, but kept ending up with footers, menu items, etc. I went through I don’t know how many versions of code, and decided in the end to use one that still caught a lot of unwanted items but managed to catch all of the body text (some of the stricter versions missed a lot of the body text).
BTW, I used Visual Studio Code as a platform to run the python code, with Beautiful Soup imported. Learning how to set that up and get it working was a feat in itself. But it now works great, and outputs the scraped text to a .txt file on my laptop. Actually, I did it in two batches so there are two .txt files. And I had to do some manual cleanup to remove unwanted text from each of the page contents, like headers, footers, menu items, etc.
I took the cleaned-up text files and uploaded them to Google’s NotebookLM platform. This allowed me to ask multiple questions where the LLM just focused only on the content of the documents. The results were much better than earlier attempts. I was able to get pretty decent answers to questions like: what are some similarities and outliers amongst the centres’ mission statements? values statements? etc. Still, it did start pulling information from other parts of the centres’ web content rather than, e.g., just the values statements when I was asking about those. So I gave it prompts like this:
Please provide a list of common elements in these centres’ statements of values, as well as outliers. Use only the explicit values statements in each centre. Within each element, provide a list of universities that have that in their statements.
A very nice aspect of NotebookLM is that it will give you links to sections in the documents it is using, as reference points. So, for example, if it says University X has value Y, it will like to the place in the document where it got that information, so you can check its interpretation.
Still, there were issues with its interpretations of some aspects of the documents. For example, when I asked it to find commonalities and outliers amongst the vision statements in the text documents, it gave the following as a commonality:
Community and Collaboration: The importance of building a collaborative community of educators is highlighted in many vision statements. Examples include the University of Waterloo’s Centre for Teaching Excellence, which aims to “inspire teaching excellence, innovation, and inquiry”, and Thompson Rivers University’s Centre for Excellence in Learning and Teaching, which desires to be the “academic catalyst” that advances practices across the institution.
I mean, okay, I guess inspiring others is “community”? And being a catalyst is too somehow? But there were much better examples than this. Still, on the whole, the analyses I did like this were pretty good, even though sometimes when I asked it to focus on the “values” sections it pulled from the “vision” or other parts of the websites (and likewise for other analyses). So it’s not entirely accurate.
I also asked NotebookLM to generate a table:
Please create a table out of the information in the sources, with institutions and centres as rows and mission, vision, values, and the URL as columns. Be sure to make the URLs into hyperlinks.
This was surprisingly accurate. And it alphabetized the institutions to boot, which was really handy (and I didn’t ask it to do so). I wanted to download it as a google sheet or something but that wasn’t possible. The only options were to copy and paste it, or “save to note,” which creates a note in your notebook that is read-only. And tables are not formatted in these notes, so the table was a mess.
However, I was able to copy and paste the table into a Word doc, and from there into Excel. Which is roundabout manualness but it worked. Then I could fix errors in the Excel file, of which there weren’t very many. It did well with the French language websites too, keeping the original French and mostly putting things in the right boxes.
Here is the Excel file with CTLs mission, vision, and values statements as generated by NotebookLM, revised manually, and uploaded to my google drive.
I tried to create a public link to my Notebook with all of this information, but it seems I can only share with individual people using their google accounts.
Scraping web pages and using ChatGPT projects
I also tried uploading the same scraped text files to the new-ish functionality on ChatGPT, “projects.” With this functionality you can upload files for the LLM to focus on and have chats specifically organized in a folder with that project. You can also give it specific instructions for the project. Here are the project instructions I used:
You are a researcher who is analyzing websites from teaching and learning centres to find their mission, vision, and values statements and doing analyses on them. If you don’t find information in a particular category you say that no information was found; you don’t provide information that isn’t in the data files.
I then tried multiple chats on the text files. At first when I just asked it to give a list of universities’ centres for T&L and their mission, vision, and/or values statements, while just using the information in the text file, it paraphrased the mission, vision, and values, and frequently got things wrong. I then asked it not to paraphrase/summarize, but to use direct quotes, which it did pretty well–this helped with accuracy. However, it skipped multiple centres entirely.
I tried again, asking it to include all of the centres for teaching and learning, and it gave me a 7 of them. I then asked it to do the rest, without repeating the ones it had already listed. It did another 10. But I had over 40 websites scraped in the text files. At this point I stopped trying, though I probably could have kept going. Perhaps it keeps running up against some length limit in outputs?
I also asked it to do a bit of analysis on the two text files, which was more or less okay. I asked it to come up with some commonalities and outliers amongst the values statements, for example. And it did pretty well on some of them. But when I asked it to include quotes from the centres it added a few words to some of the quotes and also took quotes from one centre and said it was from another. This probably happened more than once, but I wasn’t interested in doing an exhaustive evaluation. Suffice it to say there were mistakes.
I tried to share these chats like you can with other chats–you can get a URL so others can view the chats publicly. But that seems not to be the case within ChatGPT projects; at least, I couldn’t figure out how to do it.
One nice thing about ChatGPT is that it can export directly into an Excel file. I asked it to create a spreadsheet like the one created above with NotebookLM:
Please create a spreadsheet using information from the two text files uploaded to this project. List universities as rows, and names of teaching and learning centres, mission statements, vision statements, and values statements as columns. Add a column with the URL for each centre’s webpage where this information can be found. If there is no information for one or more columns for a centre, write “not found” in the cell. Include all of the centres listed in the two text files into the spreadsheet.
Sadly, the column for the URL just says “link” on each row, with no hyperlink. Boo. And no matter how many times I ask it to include “all centres,” in this or other prompts, it still just gives me 7-10 or so. I am giving up on ChatGPT for this purpose.
Conclusion
All of the work I did did NOT save me time, but that wasn’t exactly the point. I was using this as an opportunity to test out various generative AI platforms and see what they could/could not do. In the end, I liked the scrape websites plus NoteboookLM best at the moment, for accuracy and for the links to the original document in the analyses it created. Gemini Deep Research did not fulfill the task I wanted, and ChatGPT either made some things up, missed things, or only provided partial lists when I asked it to list all centres in the two text files.
Still, NotebookLM’s save and export options are limited, and it still got a few things wrong. But perhaps, after all, I was using LLMs for a task that I shouldn’t use LLMs for! If I want accurate information I should use some other AI function, most likely. But sadly I don’t know what. Happy to hear ideas on what might work better for the sort of data collection and analysis I’m trying to do.
