There’s a delicate art to teaching robots, even when you’re preparing them for predictable environments like factories, where they’ll repeat the same tasks a little differently depending on the obstacles they face. Whether a human is suddenly in their way or there’s new clutter, the machine must closely mimic its operator’s actions by staying on a trajectory (or motion path).A key issue is that many of the algorithms telling them where and when to move either aren’t flexible or get confused easily. Look at, say, time-dependent techniques, where robots work on a timer to know when to make certain movements. Any slight collision could throw the machine off schedule, though, and it won’t adjust. Time-independent methods help robots follow trajectories without time constraints by specifying where in space they should move in a certain direction; however, the bot often mixes up directions along paths that overlap at specific points (say, drawing an “H”).MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) researchers have developed an approach called “Cluster Alignment for Learned Motions” (CALM) that gives robots the flexibility and clarity they need. The system tracks a few human demonstrations for a single task using sensor data, visualizing each one as its own path. It then averages them into a general route, instead of giving the machine a rigid schedule or strict directions. It’s like a “beaten path” that the machine can easily return to when it falls off course — for example, colliding with an obstacle — when doing repetitive warehouse and household tasks.The project brings an algorithm originally designed to follow human motion to robots. Christopher Fourie PhD ‘24, a coauthor on a paper presenting CALM, originally built “TRACER” to closely draw out how a person moves, then cluster similar recordings to predict future movements. Lead author Alex Cuellar ‘21 SM ‘22, who is an MIT PhD student and CSAIL researcher, used the algorithm to “bridge motion prediction with robotics, giving machines the chance to pick up wherever they left off.”So how does CALM keep real-world robots on task? Users start by giving their machine kinesthetic demonstrations, physically guiding it to do things like brushing or wiping. After you show it multiple ways to do the same task, CALM traces your motions and groups similar ones into “clusters.” It averages the paths into a mean trajectory — basically, a broader motion plan that considers a few similar ways to do the same thing. CALM maintains a belief about where along a mean trajectory the robot is. Reliance on this belief, rather than a timer or its position in space, is what allows it to circumvent the issues of time-dependent and time-independent methods. This helps machines like robotic arms get back on course after encountering changes in their surroundings, such as a new object being placed on the table or a human suddenly in their path. “The goal is to help robots respond to disturbances more like humans do,” says Cuellar. “When we’re doing a chore and bump into something, for example, we go right back to what we’re doing because we have an intuitive sense of where along the task we got interrupted. We have a general, flexible plan in our mind, and we want robots that approach everyday tasks the same way.”Staying CALM under pressureA series of real-world experiments showed how CALM can help machines respond to obstacles that would otherwise throw them off. They first taught a robotic arm to wipe off some LEGOs with just six demonstrations, where it either verified if the brush was clean before brushing, or simply wiped down the bricks and put the brush away. The robot executed this task, despite a human pushing the machine around, while time-dependent and time-independent baselines failed to adjust.CALM helped the same robot hone rudimentary writing skills, too. Given only two demonstrations apiece to write the letters “A” and “C,” the robot could switch between writing either with a slight nudge from a human. The robot was just as effective at painting a box, where it was only trained on four strokes, three times each: an “X,” a “V”, up, and in a circle. The robot accurately mirrored each of the moves it was taught, intuitively understanding which stroke a human collaborator wanted with a little push.Before the researchers brought CALM to the real world, they tested its motion tracking skills on overlapping 2D trajectories — in other words, redrawing simple illustrations that represented motion paths. It recreated complex routes resembling things like brackets, snakes, and turnips so closely you’d think it literally traced them. But instead, CALM tried its best to redraw the lines, outdoing many comparable baselines.“What's distinctive about CALM is that the robot maintains a belief about where along the task it is, rather than where in space or when on a clock,” says senior author Julie Shah ‘04, SB ‘06, PhD ‘11, who is H.N. Slater Professor of Aeronautics and Astronautics and CSAIL principal investigator. “Giving the robot a notion of progress on a task is what enables a handful of demonstrations to handle the small surprises that happen outside the lab, and it's an exciting step toward robots that can work in everyday settings.”The team’s system may soon handle deeper layers of complexity within robot learning. Expanding beyond simple behaviors, the team intends to model tasks that require rotation, such as brushing a 3D surface, where the angle of a robot’s arm has to change over the course of a task. In the future, the team also plans to build a version of CALM that allows you to manipulate the mean trajectories themselves according to a machine’s environment. For example, if a robot needs to brush a block and that block is moved, that trajectory can be adjusted to track where the block is now.CALM might also see upgrades to its flexibility, building on an already adaptable approach. Researchers are hoping the system can learn to skip or jump backwards to specific parts of a trajectory, or even switch paths mid-task. The method could learn to alter its approach using either physical cues it sees via a computer vision system or verbal ones from users’ prompts.Cuellar, Fourie, and Shah’s work was supported, in part, by the National Science Foundation and Los Alamos National Laboratories. Their work was published in IEEE’s Robotics and Automation Letters (RA-L) last September and will be presented at the 2026 IEEE International Conference on Robotics and Automation (ICRA) in June.
Motion tracking system shows robots the path most traveled by, keeping them on task
There’s a delicate art to teaching robots, even when you’re preparing them for predictable environments like factories, where they’ll repeat the same tasks a little differently depending on the obstacles they face. Whether a human is suddenly in their way or there’s new clutter, the machine must closely mimic its operator’s actions by staying on a trajectory (or motion path).A key issue is that many of the algorithms telling them where and when to move either aren’t flexible or get confused easily. Look at, say, time-dependent techniques, where robots work on a timer to know when to make certain movements. Any slight collision could throw the machine off schedule, though, and it won’t adjust. Time-independent methods help robots follow trajectories without time constraints by specifying where in space they should move in a certain direction; however, the bot often mixes up directions along paths that overlap at specific points (say, drawing an “H”).MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) researchers have developed an approach called “Cluster Alignment for Learned Motions” (CALM) that gives robots the flexibility and clarity they need. The system tracks a few human demonstrations for a single task using sensor data, visualizing each one as its own path. It then averages them into a general route, instead of giving the machine a rigid schedule or strict directions. It’s like a “beaten path” that the machine can easily return to when it falls off course — for example, colliding with an obstacle — when doing repetitive warehouse and household tasks.The project brings an algorithm originally designed to follow human motion to robots. Christopher Fourie PhD ‘24, a coauthor on a paper presenting CALM, originally built “TRACER” to closely draw out how a person moves, then cluster similar recordings to predict future movements. Lead author Alex Cuellar ‘21 SM ‘22, who is an MIT PhD student and CSAIL researcher, used the algorithm to “bridge motion prediction with robotics, giving machines the chance to pick up wherever they left off.”So how does CALM keep real-world robots on task? Users start by giving their machine kinesthetic demonstrations, physically guiding it to do things like brushing or wiping. After you show it multiple ways to do the same task, CALM traces your motions and groups similar ones into “clusters.” It averages the paths into a mean trajectory — basically, a broader motion plan that considers a few similar ways to do the same thing. CALM maintains a belief about where along a mean trajectory the robot is. Reliance on this belief, rather than a timer or its position in space, is what allows it to circumvent the issues of time-dependent and time-independent methods. This helps machines like robotic arms get back on course after encountering changes in their surroundings, such as a new object being placed on the table or a human suddenly in their path. “The goal is to help robots respond to disturbances more like humans do,” says Cuellar. “When we’re doing a chore and bump into something, for example, we go right back to what we’re doing because we have an intuitive sense of where along the task we got interrupted. We have a general, flexible plan in our mind, and we want robots that approach everyday tasks the same way.”Staying CALM under pressureA series of real-world experiments showed how CALM can help machines respond to obstacles that would otherwise throw them off. They first taught a robotic arm to wipe off some LEGOs with just six demonstrations, where it either verified if the brush was clean before brushing, or simply wiped down the bricks and put the brush away. The robot executed this task, despite a human pushing the machine around, while time-dependent and time-independent baselines failed to adjust.CALM helped the same robot hone rudimentary writing skills, too. Given only two demonstrations apiece to write the letters “A” and “C,” the robot could switch between writing either with a slight nudge from a human. The robot was just as effective at painting a box, where it was only trained on four strokes, three times each: an “X,” a “V”, up, and in a circle. The robot accurately mirrored each of the moves it was taught, intuitively understanding which stroke a human collaborator wanted with a little push.Before the researchers brought CALM to the real world, they tested its motion tracking skills on overlapping 2D trajectories — in other words, redrawing simple illustrations that represented motion paths. It recreated complex routes resembling things like brackets, snakes, and turnips so closely you’d think it literally traced them. But instead, CALM tried its best to redraw the lines, outdoing many comparable baselines.“What's distinctive about CALM is that the robot maintains a belief about where along the task it is, rather than where in space or when on a clock,” says senior author Julie Shah ‘04, SB ‘06, PhD ‘11, who is H.N. Slater Professor of Aeronautics and Astronautics and CSAIL principal investigator. “Giving the robot a notion of progress on a task is what enables a handful of demonstrations to handle the small surprises that happen outside the lab, and it's an exciting step toward robots that can work in everyday settings.”The team’s system may soon handle deeper layers of complexity within robot learning. Expanding beyond simple behaviors, the team intends to model tasks that require rotation, such as brushing a 3D surface, where the angle of a robot’s arm has to change over the course of a task. In the future, the team also plans to build a version of CALM that allows you to manipulate the mean trajectories themselves according to a machine’s environment. For example, if a robot needs to brush a block and that block is moved, that trajectory can be adjusted to track where the block is now.CALM might also see upgrades to its flexibility, building on an already adaptable approach. Researchers are hoping the system can learn to skip or jump backwards to specific parts of a trajectory, or even switch paths mid-task. The method could learn to alter its approach using either physical cues it sees via a computer vision system or verbal ones from users’ prompts.Cuellar, Fourie, and Shah’s work was supported, in part, by the National Science Foundation and Los Alamos National Laboratories. Their work was published in IEEE’s Robotics and Automation Letters (RA-L) last September and will be presented at the 2026 IEEE International Conference on Robotics and Automation (ICRA) in June.
