Name of Project:
Prompt Engineering to Elicit Pedagogical Reasoning
Proposal in one sentence:
We want to experiment with diverse prompt engineering techniques using OpenAI’s API to build a LangChain application that powers a Discord bot competent in reading comprehension tutoring.
Description of the project and what problem it is solving:
Discourse about the role of AI in education orbits two equally myopic polarizations: either it needs to be banned to protect our academic integrity or it’s a foregone conclusion AI “will solve” education. Meanwhile, our institutions remain in a permanent state of distress with a dearth of realistic solutions at their disposal. What’s missing are ambitious, pedagogically sound projects building real solutions. Innovation in the way we learn is long overdue.
That’s why we’re iterating on prompt design to elicit conversational behaviors as good as a session with a private tutor. And a great place to start is advanced native language literacy, for which few effective tech tools exist, upon which symbol manipulation in every other domain of inquiry is built, and which overlaps appropriately with current LLM capability.
We’ve already shown that provided an input passage, a simple-prompt trained tutor can converse helpfully and knowledgeably with students to reach greater literary understanding. Instead of just supplying answers or interpretations, the tutor uses good educational practices to guide the student toward a fuller comprehension. Here’s an exchange based on this passage:
We can see the bot answers comprehensively and asks logical follow-up questions. This behavior is elicited only by few-shot examples of dialogues based on contexts plus a simple prefix instructing the model to ask a follow-up question. Our hypothesis is that effective, sound pedagogical reasoning is a task contained in the model’s pretraining and that it just needs further reinforcement to increase consistency.
What permutations of direct specification, memetic proxy, demonstration, or metaprompting will ultimately max out the limits of prompt engineering remains to be seen. But a grant from Algovera would give us the much-needed resources to use for OpenAI API credits to investigate this rigorously and fully.
Bloom’s 2-sigma problem notes that students with 1:1 tutoring access perform two standard deviations better than those without. So success here could not only mean a 100x reduction in the cost of Aristotelian tutoring, but a revolution in educational outcomes. Plastic Labs is in this for the long haul–prompt engineering for reading comprehension is just the first step.
Grant Deliverables:
- A prompt-engineered AI Discord bot capable of pedagogically-sound, natural language reading comprehension tutoring for a supplied passage
Squad
Squad Lead:
Vince Trost
- https://twitter.com/vintrotweets
- vintro#1873
Squad members:
Courtland Leer
- https://twitter.com/courtlandleer
- courtlandleer#9582