Matter over mind: This robot can paint. But can it be considered art?

Oliver Whang

One day recently, on a table in Jean Oh’s lab in the Squirrel Hill neighborhood of Pittsburgh, a robot arm was busy at a canvas. Slowly, as if the air were viscous, it dipped a brush into a pool of light gray paint on a palette, swung around and stroked the canvas, leaving an inch-long mark amid a cluster of other brushstrokes. Then it pulled back and paused, as if to assess its work. The strokes, mostly different shades of gray, suggested something abstract — an anthill, maybe. Dr. Oh, the head of the roBot Intelligence Group at Carnegie Mellon University, dressed in a sweatshirt bearing the words “There Are Artists Among Us,” looked on with approval. Her doctoral student, Peter Schaldenbrand, stood alongside.

Dr. Oh’s work, which includes robot vision and topics in autonomous aviation, often touches on what is known as the sim-to-real gap: how machines trained in a simulated environment can act in the real world. In recent years, Schaldenbrand has led an effort to bridge the sim-to-real gap between sophisticated image-generation programs like Stable Diffusion and physical works of art like drawings and paintings.

This has mainly been manifest in the project known as FRIDA, the latest iteration of which was rhythmically whirring away in a corner of the lab. (FRIDA is an acronym for Framework and Robotics Initiative for Developing Arts, although the researchers chose the acronym, inspired by Frida Kahlo, before deciding what it stood for.) The process of moving from language prompts to pixelated images to brushstrokes can be complicated, as the robot must account for “the noise of the real world,” Dr. Oh said.

But she, Schaldenbrand and Jim McCann, a roboticist at Carnegie Mellon who also helped develop FRIDA, believe that the research is worth pursuing for two reasons: It could improve the interface between humans and machines, and it could, through art, help connect people to one another. “These models are trained based on everybody’s data,” Dr. McCann said, referring to the large language models that power tools like ChatGPT and DALL-E. “And so I still think we’re figuring out how projects like this, that use such models, can deliver value back to people.”

The sim-to-real gap provides a surprisingly tricky problem for roboticists and computer engineers. Some artificial intelligence systems can list the steps involved in walking (tighten your quadriceps and flex your tibialas posterior, tilt your weight back and tense your gluteus maximus) and can make a simulated body walk in a virtual world. So it’s tempting to think that these systems could easily make a physical body walk in the real world. Not so.

In the 1980s, the computer scientist Hans Moravec noted that A.I. was good at engaging in complicated reasoning and parsing vast amounts of data but that it was bad at simple physical activities, like picking up a bottle of water. This is known as Moravec’s paradox. (The physical superiority of humans might be explained by our body’s long evolutionary history; the tasks that are simple for us are supported by millions of years of Darwinian experimentation.)

Painting, which often mixes high-concept ideas and basic physical actions, throws the paradox into relief: How do we manage to capture the absurdity of human consciousness with the motions of an arm? A.I. image-generating tools like Midjourney, DALL-E and Stable Diffusion are trained by feeding neural networks massive databases of images and corresponding text descriptions.

Whang is a journalist with NYT©2023

The New York Times