Associate Professor Luca Carlone is working to give robots a more human-like awareness of their environment.
Robots have come a long way since the Roomba. Today, drones are starting to deliver door to door, self-driving cars are navigating some roads, robo-dogs are aiding first responders, and still more bots are doing backflips and helping out on the factory floor. Still, Luca Carlone thinks the best is yet to come.
While robots have grown by leaps and bounds in terms of their ability to detect and identify objects in their surroundings, they still have a lot to learn when it comes to making higher-level sense of their environment. As humans, we perceive objects with an intuitive sense of not just of their shapes and labels but also their physics — how they might be manipulated and moved — and how they relate to each other, their larger environment, and ourselves.
That kind of human-level perception is what Carlone and his group are hoping to impart to robots, in ways that enable them to safely and seamlessly interact with people in their homes, workplaces, and other unstructured environments.
“Perception is a big bottleneck toward getting robots to help us in the real world,” Carlone says. “If we can add elements of cognition and reasoning to robot perception, I believe they can do a lot of good.”
Expanding horizons
“It was love at first sight. Using algorithms and math to develop the brain of a robot and make it move and interact with the environment is one of the most fulfilling experiences,” Carlone says. “I immediately decided this is what I want to do in life.”
He went on to a dual-degree program at the Polytechnic University of Turin and the Polytechnic University of Milan, where he received master’s degrees in mechatronics and automation engineering, respectively. As part of this program, called the Alta Scuola Politecnica, Carlone also took courses in management, in which he and students from various academic backgrounds had to team up to conceptualize, build, and draw up a marketing pitch for a new product design. Carlone’s team developed a touch-free table lamp designed to follow a user’s hand-driven commands. The project pushed him to think about engineering from different perspectives.
“It was like having to speak different languages,” he says. “It was an early exposure to the need to look beyond the engineering bubble and think about how to create technical work that can impact the real world.”
The next generation
The topic in question was “simultaneous localization and mapping,” or SLAM — the problem of generating and updating a map of a robot’s environment while simultaneously keeping track of where the robot is within that environment.
In search of an answer, he accepted a postdoc position at Georgia Tech, where he dove into coding and computer vision — a field that, in retrospect, may have been inspired by a brush with blindness: As he was finishing up his PhD in Italy, he suffered a medical complication that severely affected his vision.
“For one year, I could have easily lost an eye,” Carlone says. “That was something that got me thinking about the importance of vision, and artificial vision.”
Spatial AI
Today, Carlone’s group is developing ways to represent a robot’s surroundings, beyond characterizing their geometric shape and semantics. He is utilizing deep learning and large language models to develop algorithms that enable robots to perceive their environment through a higher-level lens, so to speak. Over the last six years, his lab has released more than 60 open-source repositories, which are used by thousands of researchers and practitioners worldwide. The bulk of his work fits into a larger, emerging field known as “spatial AI.”
“Spatial AI is like SLAM on steroids,” Carlone says. “In a nutshell, it has to do with enabling robots to think and understand the world as humans do, in ways that can be useful.”
It’s a huge undertaking that could have wide-ranging impacts, in terms of enabling more intuitive, interactive robots to help out at home, in the workplace, on the roads, and in remote and potentially dangerous areas. Carlone says there will be plenty of work ahead, in order to come close to how humans perceive the world.
