Four weeks in, Pham, a 23-year-old student living at the southern tip of Sweden, typed his way through the first test, trying to write a program that could draw waves of tiny blue diamonds across a black-and-white grid. Several days later, he received a detailed critique of his code. It applauded his work, but also pinpointed an error. “Seems like you have a small mistake,” the critique noted. “Perhaps you are running into the wall after drawing the third wave.”
The feedback was just what Pham needed. And it came from a machine. During this online class, a new kind of artificial intelligence offered feedback to Pham and thousands of other students who took the same test. Built by a team of Stanford researchers, this automated system points to a new future for online education, which can so easily reach thousands of people but does not always provide the guidance that many students need and crave.
“We’ve deployed this in the real world, and it works better than we expected,” said Chelsea Finn, a Stanford professor and A.I. researcher who helped build the new system. Dr. Finn and her team designed this system solely for Stanford’s programming class. But they used techniques that could automate student feedback in other situations, including for classes beyond programming.
Oren Etzioni, chief executive of the Allen Institute for Artificial Intelligence and a former professor of computer science at the University of Washington, cautioned that these techniques are a very long way from duplicating human instructors. Feedback and advice from professors, teaching assistants and tutors is always preferable to an automated critique. Still, Dr. Etzioni called the Stanford project a “step in an important direction,” with automated feedback better than none at all.
The online course taken by Pham and thousands of others this spring is based on a class that Stanford has offered for more than a decade. Each semester, the university gives students a midterm test filled with programming exercises, and it keeps a digital record of the results, including the reams of code written by students as well as pointed critiques of each program from university instructors. This decade of data is what drove the university’s new experiment in artificial intelligence. Dr. Finn and her team built a neural network, a mathematical system that can learn skills from vast amounts of data. By pinpointing patterns in thousands of cat photos, a neural network can learn to identify a cat. By analysing hundreds of old phone calls, it can learn to recognise spoken words. Or, by examining the way teaching assistants evaluate coding tests, it can learn to evaluate these tests on its own.
The Stanford system spent hours analysing examples from old midterms, learning from a decade of possibilities. Then it was ready to learn more. When given just a handful of extra examples from the new exam offered this spring, it could quickly grasp the task at hand. “It sees many kinds of problems,” said Mike Wu, another researcher who worked on the project. “Then it can adapt to problems it has never seen before.”
This spring, the system provided 16,000 pieces of feedback, and students agreed with the feedback 97.9 percent of the time, according to a study by the Stanford researchers. By comparison, students agreed with the feedback from human instructors 96.7 percent of the time.
Metz is a tech reporter with NYT©2021
The New York Times