The Framework
Below is a framework to support your critical analysis when using Chat GPT to generate Clinical Reasoning Activities for BCIT Nursing Knowledge courses.
*Content produced by Generative AI must be reviewed by the instructor prior to use with students
| Human Experience
(overall impression of content)[1] |
yes | no |
| Does the case study align with Nursing Knowledge 6000 content?
|
☐ | ☐ |
| Does the case study align with pre-existing clinical reasoning activities?
|
☐
|
☐
|
| Expertise | yes | no |
|
Does the content utilize Tanner’s Model appropriately?
|
☐ | ☐ |
| Does the content utilize NCLEX questioning appropriately?
|
☐ | ☐ |
| Do the questions align with the appropriate level of Bloom’s taxonomy?
|
☐ | ☐ |
| Accuracy[1]
|
||
| Accuracy of educational components | yes | no |
| Content aligns with learning outcomes[2] | ☐ | ☐ |
| The course level being taught was identified. | ☐ | ☐ |
| The level of higher-order thinking (Bloom’s) was identified[2] | ☐ | ☐ |
| Ensured Gen AI identifies the correct answer and provides a rationale to share with students | ☐ | ☐ |
| Accuracy of educational content (SIFT)[3] | yes | no |
| Stop: Content produced was reflected on critically and is accurate[3] | ☐ | ☐ |
| Investigate: Researched who created the information generated and why[3] | ☐ | ☐ |
| Find: Alternative sources of information were identified to see if there is consensus[3] | ☐ | ☐ |
| Trace: If claims are cited, original source was reviewed and repeat SIFT process[3] | ☐ | ☐ |
| Trust[1] | ||
| Alignment Indigenous data sovereignty, protocols and risks for harm[4] | yes | no |
| Advocating for Truth4]
o Content was critically reviewed to ensure the absence of false, stereotyped, discriminatory and harmful information about indigenous people and communities[4] o GenAI tools were not used to learn about indigenous histories and knowledges[4] |
☐ | ☐ |
| Indigenous Sovereignty[4]
o No data, knowledge, or creative works by Indigenous peoples of communities were utilized without permission/consult with indigenous knowledge experts when applicable[4] |
☐ | ☐ |
| Ethical considerations[4] | yes | no |
| o Potential for bias and harmful outputs that reproduce systemic inequities was considered[4]
o Gen AI outputs were reviewed for possible harm[4] o Equitable access to Gen AI tools exists (bandwidth, geographic location, cost)[4] |
☐ | ☐ |
| Privacy and Confidentiality[4] | yes | no |
| o No personal, private or confidential information was shared when interacting with Gen AI tools including student work[4]
o Only low risk (minimal harm if disclosed, or may be freely disclosed) information was used[4] |
☐ | ☐ |
| Copyright[4] | yes | no |
| o No third party material utilized (rationale – may constitute copyright infringement)
o Terms of service have been reviewed and complied with (rationale – it is crucial to understand use of the outputs and any attribution requirements) |
☐ | ☐ |
[1] Verhulsdonck, G., Weible, J., Stambler, D. M., Howard, T., & Tham, J. (2024). Incorporating Human Judgment in AI Assisted Content Development: The HEAT Heuristic. Technical Communication, 71(3), 60–72. https://doi.org/10.55177/tc286621
[2] British Columbia Institute of Technology. (2024, September). Generative AI to help instructors teach. BCIT Learning & Teaching Centre. https://www.bcit.ca/learning-teaching-centre/generative-ai/generative-ai-to help instructors-teach/
[3] The University of British Columbia Library. (n.d.). Guidelines for evaluating sources. UBC Library Research Guides. https://guides.library.ubc.ca/EvaluatingSources/Guidelines#s-lg-box-16579377
[4] The University of British Columbia, Office of Learning Technology. (2024, August). Principles for mitigating risks [PDF]. UBC. https://it‑genai‑2023.sites.olt.ubc.ca/files/2024/08/PRINCIPLES_FOR_MITIGATING_RISKS.pdf
The Reflection
In our Bachelor of Science in Nursing (BSN) program at BCIT, the use of ChatGPT is becoming increasingly prevalent. A common use of this platform is to create Clinical Reasoning Activities (CRAs) for our Nursing Knowledge courses. These activities are essentially an unfolding case study that utilizes NCLEX-style questioning. Here, students can apply Tanner’s clinical judgment model to a case study. One problem with our current use of Chat GPT, however, is the lack of a consistent and evidence-based approach to how Chat GPT is utilized. For this assignment, I created a template-like prompt for creating a CRA and then created a framework that an instructor can use to critique Chat GPT’s output.
The framework I created incorporates Verhulsdonck et al.’s (2024) HEAT Heuristic and is also supported by both UBC and BCIT’s institutional best practices for generative AI. Verhulsdonck et al. (2024) emphasize that algorithms require quality prompts from the user and a need for human judgment to critique AI-generated content. “AI acts like it has understood you but does not understand what it has generated” (Floridi as cited by Verhulsdonck et al., 2024, p. 62). There is a high risk for harmful stereotypes, content errors, and overall challenges with ethics and bias due to the content AI is drawing from (Verhulsdonck et al., 2024).
I found the framework I developed to be a helpful tool when analyzing the CRA generated by Chat GPT with my prompt.
Human Experience (overall impression of the content)
The content primarily aligned with the learning outcomes and focused on the pathophysiology and nursing management of COPD, CHF, and Atrial Fibrillation; however, it was not a classic unfolding case study, where there is a detailed description of how the patient’s status changes as the activity progresses. Perhaps providing clear direction on what an unfolding case study entails would have been helpful.
Expertise (appropriateness of content)
The CRA created utilized Tanner’s Model appropriately and included all five stages of it. It was also levelled to the appropriate level of Bloom’s as it asked the student to evaluate, prioritize and emphasize, which is all appropriate verbiage at the evaluation level of Bloom’s. Where the generative tool fell short was with the style of questioning. NCLEX utilizes multiple choice, fill-in-the-blank and matching style questions; however, the CRA generated only multiple choice questions.
Accuracy
Educational Components
The CRA aligned well with the learning outcomes provided in the prompt, and the answers and rationales supplied were accurate; however, some of the content in the CRA was more applicable to how a physician would interpret findings than to how a nurse would. For example, one correct answer required an understanding of how to assess for jugular vein distention, which would not be an expectation of a student nurse. The CRA also incorporated Arterial Blood Gas analysis, an advanced nursing skill, which did not align with the learning outcomes provided.
Educational Content
This aspect of the framework necessitates a critical analysis of the provided content. I was impressed with the content Chat GPT referenced for the most part. Two references were prevalent nursing textbooks that are commonly used in our curriculum. The other websites were also appropriate, but what was lacking was referencing current research and evidence. Perhaps including the required articles for this course content in the prompt would have helped.
Trust
Finally, but perhaps the most crucial critique ends here. As noted in my introduction, generative AI has the potential to perpetuate pre-existing human bias that is not rooted in reality (Verhulsdonck et al., 2024). This framework enables the instructor to critically reflect on potential biases or harmful outputs created, while also ensuring that important privacy, confidentiality, and copyright standards are met. As Canadians on a journey towards Truth and Reconciliation, we must also specifically analyze content for discriminatory language against Indigenous peoples. The case study provided by Chat GPT was brief, with limited information on demographics, and no discriminatory content was identified in the CRA. With more detailed case studies and CRAs, this may not be the case, and therefore, taking the time to stop, pause and reflect on the content is essential.
References
Verhulsdonck, G., Weible, J., Stambler, D. M., Howard, T., & Tham, J. (2024). Incorporating Human Judgment in AI Assisted Content Development: The HEAT Heuristic. Technical Communication, 71(3), 60–72. https://doi.org/10.55177/tc286621