Grading and Evaluation with AI

Artificial intelligence (AI) is transforming education, and one area where it holds significant potential is in grading student papers. With tools like ChatGPT, educators can streamline grading processes, reduce workload, and provide more timely, personalized feedback. However, successful integration requires thoughtful strategies. This webpage explores practical methods for using AI to assist with grading, including how to align AI with existing rubrics, create new grading frameworks with the help of AI, calibrate performance using model papers, and generate both quantitative scores and qualitative feedback. These strategies not only promote efficiency but also support deeper student engagement by focusing on meaningful feedback and personalized learning paths.

Adopting AI tools for grading is not without challenges, such as ensuring consistency and maintaining academic integrity. However, when implemented thoughtfully, these tools can complement traditional teaching practices rather than replace them. This resource guides educators through various strategies—from refining rubrics and using AI to analyze model papers, to tracking student progress over time.

This method involves using an established or AI-generated rubric to provide both numerical scores and qualitative feedback. Educators or students can input the rubric into an AI tool like ChatGPT or Gemini at the start of a chat thread. This sets clear expectations for the grading criteria, ensuring consistency across assignments.

Prompt Guide:

1. "I would like to use this thread to explore potential grades for my students. Use this rubric: [copy and paste rubric]."
   Optional: Develop a rubric with ChatGPT or Gemini: "I want to create a rubric for an argumentative paper. It will be out of 10 points and I will be looking for..."

2. Copy in student work. Omit student personal information for their safety and ALWAYS ask student permission to use their work in this manner; student work is submitted into the AI dataset. If I find an AI tool that can do this without submitting student work, I will post it.

3. Look at quantitative and qualitative metrics. In other words, ask for both numerical and written feedback and be CAREFUL about any numerical metrics for evaluating student work. Remember that you, as an educator or student, are always responsible and your judgment should subersede AI's regarding the quality of student work. 

Remember that educators should be cautious about data privacy when using AI tools, as student submissions may be processed and potentially incorporated into broader AI datasets. Teachers must inform students about the potential risks and secure their permission before submitting their work to these tools. (ACUE) (Harvard Law School).

This method allows educators or students to use a high-quality paper as a model for AI-based feedback. The model paper serves as a benchmark to guide students in improving their writing. Using ChatGPT or Gemini, the AI is prompted to identify key elements in the model paper and create criteria or rubrics that highlight what makes it exemplary.

Prompt Guide:

1. "I would like to use this model paper to develop grading criteria for future assignments. This paper received a 10/10 score. Please use this as the model paper for all feedback in this thread."
   Optional: Have ChatGPT or Gemini generate a rubric or scoring guide based on this paper. "I need a rubric based on this model paper for an argumentative essay. The rubric should evaluate the thesis, structure, evidence use, style, and engagement, each scored out of 10."

2. Submit student with no name or different names (with student permission). Calibrate feedback, especially early in the thread, to reflect your goals and standards.

This method is particularly beneficial for “sample teachers,” who prefer students to learn by analyzing examples rather than following step-by-step instructions. With AI-enabled tools, students can upload their own drafts and receive targeted feedback aligned with the high-quality model. The combination of AI feedback with strong models encourages students to critically reflect on their writing and make meaningful revisions

As always, omit personal information from student work for privacy and obtain student permission before using AI tools in this capacity, as their work may be incorporated into the AI’s broader dataset. This method emphasizes that educators are ultimately responsible for evaluating student work, and AI-based assessments should complement, not replace, professional judgment.

In this method, students take an active role in prompting AI tools like ChatGPT or Gemini to assess their work. This approach promotes self-reflection, helping students identify areas for improvement and develop their ability to engage meaningfully with AI. However, to ensure data privacy, students should avoid creating accounts whenever possible, and teachers must guide them through discussions on AI safety, privacy, and ethical use before they engage with the technology. I've created this page for support. 

This strategy requires an extended discussion (30-60 minutes) about how AI processes data and the risks involved, including the possibility of student work being incorporated into broader datasets. Emphasize transparency and responsible AI use to empower students with informed consent.

How to Implement This Strategy:

1. Students Frontload the AI with a Rubric or Desired Feedback:
   Students provide the AI with a rubric, sample criteria, or specific areas they would like feedback on.
   Sample Prompt:
   “This is my argumentative essay. I am focusing on the clarity of my thesis, use of evidence, and structure. Please provide feedback in these areas.”

2. Students Submit Their Work to AI (using no name or alternate name):
   Students input their writing directly into the AI for analysis.

   • AI provides approximate feedback aligned with the students’ prompts and rubrics, offering both strengths and areas for improvement.

3. Reflective Discussion and Teacher Support:
   After receiving AI feedback, students can engage in reflective discussions about what they learned and how they will revise their work. Teachers can facilitate these discussions, helping students evaluate the feedback and make thoughtful revisions.

Educational Benefits:

This method encourages metacognition by prompting students to reflect on their writing and assess their strengths and weaknesses. It also builds digital literacy, helping students develop competencies in using AI effectively and responsibly—an essential 21st-century skill emphasized by ISTE and Harvard’s Graduate School of Education​ (Harvard Graduate School of Education) (MDPI).

Challenges to Anticipate:

Ensure students understand the limitations of AI-generated feedback, as it is not a substitute for human judgment. Educators must stress that they retain ultimate responsibility for evaluating student work. Additionally, consult resources from the Consortium for School Networking (CoSN) to address privacy concerns and manage AI use effectively in educational settings​ (University of Waterloo).

This method builds upon the self-assessment strategies introduced in Method 3, adding a longitudinal element to track students' progress over time. The goal is for students to submit multiple assignments throughout the year, using AI feedback to monitor how their writing evolves. This approach helps students recognize specific skills they are developing and identify areas needing further attention. AI can offer valuable insights, but the instructor should emphasize the importance of analog reflection, where students set goals independently before consulting AI.

How to Implement This Strategy:

1. Planning and Structure:

   • Ensure that there is a catalog or digital footprint of student writing throughout the course of the year. This could look like a student portfolio, or typed essays on Google Classroom. I often have my students type up their hand-written, in-class essays as a reflection activity. Simply: Keep student writing throughout the year.

2. Student Goal Setting and Self-Reflection (Analog):

   • Before using AI, create a reflection activity to have students explore their own writing throughout the year and create lists of their strengths and weaknesses. This should be done with as little technology as possible to ensure that they develop metacognitive skills without over-relying on AI.

3. AI-Assisted Feedback:

   • After completing their self-reflection, students submit their essays to AI platforms like ChatGPT or Gemini to receive feedback. As usual: omit names, identifying information, and ask for student permission.
     Sample Prompt:
     “I would like to use this thread to evaluate my growth as a writer throughout the year. I will submit my essays and I would like you to evaluate their strengths and weaknesses with specific focus as to how I have grown as a writer."

4. Instructor Support and Goal Refinement:

   • Teachers guide students in comparing their analog reflections with AI feedback. The goal is to validate or adjust their assessments and use insights from AI as a supplement to their own observations.

Educational Benefits:

This method encourages continuous self-improvement and deeper engagement with learning processes. It also builds critical thinking skills by prompting students to compare their reflections with AI feedback, fostering a balanced approach to using technology. The focus on analog goal setting ensures students remain independent learners, capable of self-directed growth.

Considerations for Use:

While AI can provide useful metrics and insights, instructors must emphasize that students are responsible for their learning journey. As Harvard's AI Pedagogy Project and ISTE suggest, technology should complement—not replace—reflection and learning processes​ (Harvard Graduate School of Education) (OpenLearning). Additionally, to maintain student privacy, educators can follow best practices from CoSN, avoiding AI platforms that require student accounts and ensuring data is not added to AI datasets without consent​ (University of Waterloo).

By integrating analog self-reflection with AI feedback, this method helps students develop both independence and digital literacy. It reinforces the importance of reflection as a lifelong skill, empowering students to set meaningful goals and track their progress throughout the course.

AI tools like ChatGPT excel at providing qualitative feedback—commenting on the structure, coherence, and clarity of student work—where they mimic thoughtful reflections that teachers would otherwise write. However, quantitative grading introduces a subjective element under the illusion of objectivity. Assigning numerical values to components like thesis statements or argument development often distorts the reality of writing as a nuanced, creative endeavor.

The French philosopher Jacques Derrida argued that language is inherently unstable, with meaning constantly shifting depending on context, culture, and individual interpretation. Derrida’s theory of deconstruction reveals that any attempt to fix meaning—like assigning a numerical score to a subjective thesis statement—is fraught with instability. Writing, in essence, resists reductive quantification, just as Derrida argued meaning can never be fully pinned down​.

For educators, this means being cautious about numerical grades derived from AI. The "illusion of objectivity" risks misrepresenting nuanced writing as data. While rubrics provide structure, teachers must acknowledge the subjective nature of their evaluations and recognize that AI feedback mirrors these complexities rather than resolves them. AI can enhance grading but never replace the need for careful, reflective judgment from educators who are ultimately responsible for student outcomes.

Grading on a scale, such as assigning an 82 vs. 83/100, suggests precision where none exists. As scholars in educational assessment point out, writing is fundamentally interpretive, and scores often fail to capture the depth of a student’s work​. The Michelin star system is often cited as a parallel—a rigid attempt to quantify subjective experiences that cannot be boiled down to numbers.

Therefore, educators must calibrate AI-generated scores with their own professional judgment. Rely on AI’s insights to support your grading but never abdicate your responsibility as the final authority. If a student asks why they earned a certain grade, "AI graded it" is not an appropriate answer. Your name and reputation are attached to every grade, so you must own the decisions made.

To use AI effectively, educators must actively engage in calibration by debating and adjusting the feedback provided by the tool. Remember, AI systems like ChatGPT or Gemini can act as "yes men" unless properly challenged with precise prompts. Below are some examples for calibration:

• Prompt for Teachers:
  “This essay was previously scored as an 8/10. The feedback emphasized weak thesis development, but I believe the evidence use is more of an issue. Reanalyze the essay with an emphasis on how the evidence supports the thesis.”

• Prompt for Students:
  “I received feedback that my introduction needs work. However, I believe the introduction provides a clear roadmap for my argument. Please reevaluate the introduction with a focus on thesis clarity and alignment with the essay’s structure.”

These prompts encourage dialogue with AI, helping both students and teachers refine their understanding of what constitutes strong writing. Calibration is an ongoing process, and students should be empowered to disagree with AI feedback to develop critical thinking skills and writing independence.

Research in educational equity has highlighted the systemic biases present in grading, particularly in language assessments. English instruction, with its historic focus on canonical works by white, male authors, often imposes implicit standards that do not reflect the linguistic diversity of students. AI tools trained on such biased datasets risk reinforcing these disparities.

Studies have shown that students from marginalized backgrounds, particularly multilingual learners, are often penalized for not conforming to "standard" academic English. Scholars like Lisa Delpit and Gloria Ladson-Billings argue for more culturally responsive teaching practices that honor students' linguistic identities​ (Other People's Children). AI tools, when used critically, can help mitigate some biases by providing consistent feedback. However, educators must remain vigilant, ensuring that the feedback aligns with equitable practices and that AI recommendations are scrutinized for implicit biases.

In summary, while AI offers new opportunities for grading and feedback, these tools require thoughtful application. Educators must remain active participants in the grading process, balancing AI’s capabilities with the nuanced understanding and cultural awareness only a human educator can provide.