In order to evaluate AI output, we need subject matter expertise
Summary
- In order to evaluate AI output, we need subject matter expertise
Details
- The work of acquiring expertise requires us to be less skillful than AI for part of the process
- We still need to do the work to get the basic knowlege of a subject to become expert
References
Quotes
This is the paradox of knowledge acquisition in the age of AI: we may think we don’t need to work to memorize and amass basic skills, or build up a storehouse of fundamental knowledge—after all, this is what the AI is good at. Foundational skills, always tedious to learn, seem to be obsolete. And they might be, if there was a shortcut to being an expert. But the path to expertise requires a grounding in facts.
Learning any skill and mastering any domain requires rote memorization, careful skills building, and purposeful practice, and the AI (and future generations of AI) will undoubtedly be better than a novice at many early skills. For example, researchers at Stanford found that the GPT-4 AI scored higher than first- and second-year medical students at their final clinical reasoning exams. The temptation, then, might be to outsource these basic skills to the AI. After all, doctors are happy to use medical apps and the internet to help diagnose patients instead of simply memorizing medical information. Isn’t this the same thing?
The issue is that in order to learn to think critically, problem-solve, understand abstract concepts, reason through novel problems, and evaluate the AI’s output, we need subject matter expertise. An expert educator, with knowledge of their students and classroom, and with pedagogical content knowledge, can evaluate an AI-written syllabus or an AI-generated quiz; a seasoned architect, with a comprehensive grasp of design principles and building codes, can evaluate the feasibility of an AI-proposed building plan; a skilled physician, with extensive knowledge of human anatomy and diseases, can scrutinize an AI-generated diagnosis or treatment plan. The closer we move to a world of Cyborgs and Centaurs in which the AI augments our work, the more we need to maintain and nurture human expertise. We need expert humans in the loop.