How Microsoft Research Used ChatGPT to Guide Robots and Drones

What if you could control a robot by talking to it? Microsoft Research has published work on using ChatGPT to help control robots with ordinary language. A home assistant that can respond to a request like 'make my bed' feels a step closer, although the research demonstrations are narrower than that future scenario.

Today, controlling a robotic arm or a sophisticated robot often involves writing code or detailed instructions, a process that requires specialized technical skills. Software development has a similar barrier, and tools such as GitHub Copilot and ChatGPT can generate code from language descriptions. I see robotics as a natural extension of that idea.

The Microsoft team describes its method and the challenges of connecting ChatGPT to robots. One example asks a drone to find a drink, then asks it to find something with sugar and a red logo. The researchers' system uses available perception and robot functions; the demonstration does not mean a language model alone controls any arbitrary robot.

The idea is to put a person 'on the loop': rather than manually writing each iteration, the person supervises the result and gives feedback in language. I expect this could make iteration faster and potentially allow a less technical person to manage it. The human remains responsible for judging the output, however.

First, researchers define high-level robot APIs—the actions available to a particular robot, such as moving a drone. They then provide ChatGPT with the task and the available functions. It generates code, which a person reviews for accuracy, quality and safety before deployment. Microsoft also recommends simulation and careful analysis rather than directly deploying generated outputs.

In another example the drone takes a selfie using a reflective surface. I describe the request as simply 'take a selfie,' but the prompt shown on screen explicitly mentions a reflective surface. The generated code uses a mirror object. A separate request asks the drone to inspect a shelf in a lawnmower pattern, moving back and forth while increasing its height.

Another instruction asks the drone to inspect an orchid, fly in a half circle and avoid a chair. The team also asks ChatGPT for an algorithm to reach a goal while avoiding obstacles using a forward-facing distance sensor. The model writes much of the algorithm, but a human has to point out omissions, including the drone's orientation, through further conversation.

Microsoft also shows a simulated drone inspecting a wind turbine and a robot arm arranging colored blocks into the Microsoft logo. I say the model needed the logo described to it; Microsoft reports that the model recalled and drew the logo before composing the robot actions. These are research demonstrations, not evidence that a robot can safely perform arbitrary household tasks. I'm excited to imagine more robots following natural-language instructions in the future.