How Generalist uses human demonstration data for robot learning

Generalist is built on research in training robots on real-world data. Source: Generalist Summer is about creating memories. And for robots, that translates into generating lots of data.

It’s also a time to catch up on reading. a. UMI).

The academic findings quickly became the foundation for a $2 billion unicorn robotics startup, Generalist. Essentially, teams at Toyota Research Institute ( TRI ), Columbia University , and Stanford University created a data-collection platform that uses puppet-like end effectors operated by people, with GoPro cameras capturing real-world tasks such as washing dishes and picking up objects. The resulting demonstrations become training data for robot foundation models, enabling collaborative robots to learn and take over these tasks much more quickly.

UMI’s paper demonstration image, illustrating human data capture to full autonomy on co-bots. io/ Models quickly generate polices for cobots This past June at Automate , Generalist demonstrated live how its models could quickly create policies for different cobot systems. On one side of the hall, the company demonstrated how it could make Universal Robots ( UR ) arms fold and build cardboard boxes, while across the McCormick Center, it used Flexiv arms to repair robot vacuums.

” Generalist uses a range of robotic hands and arms Then, almost a month later, the AI unicorn posted a blog missive titled “ Towards Machines with a Thousand Hands. ” To better understand the meaning of Generalist’s newest accomplishment of using different arms and grippers, including spatulas, across a plethora of use cases, I interviewed Samantha Castellanos, the company ‘s founding mechanical engineer. I wanted to know the process beyond research, especially how it translated into industrial implementations to accelerate adoption and expand usage into new applications.

As a starting point, Castellanos shared Generalist’s overall philosophy: “Our main goal is to make the best model in the world. ” The mechanical engineer was referring to the already crowded space that has cumulatively raised more than $4 billion, including standouts like Skild AI ($2 billion), Physical Intelligence ($1 billion), Field AI ($300 million), and RLWRLD ($41 million). Data and tools make models better “Everything we do is to make the model better,” she continued.

“All of the data collection, all the types of tool collection, different types of interacting with the world. That is all incredibly important. ” Tactically, Generalist’s approach to data collection that is their most distinctive feature, said Castellanos.

“From a hardware perspective, our gripper is just a one degree-of-freedom [DoF] gripper, and I think that’s very different from what some other folks have started with,” she said. “And I think there is an elegance to the simplicity of design. A lot of the time, simple things are often the most robust, and that’s definitely one of our core values from a hardware perspective.

“ “My focus as a hardware engineer is on simplicity and robustness, because you can’t have a line going down every five minutes for a gripper,” Castellanos added. “You need to be able to switch it out quickly. You need to get the line running again.

If it takes two minutes to swap out a finger, that’s great. ” Startup moves from data collection to a working robot arm It’s amazing how quickly Generalist is going from collection to working robot arm. “All of the data that we’ve done for this effort [a thousand hands demonstration] has been data that was collected in our own office in Boston, and some also in our California office, ” according to Castellanos.

“So for a number of hours, there is a spread. It’s all rather small. I wrote down all of my numbers.

We have about 80 hours at the most. The fewest hours were two. ” The process then went from human collection to robot policies and further refinement by the machines themselves.

“The human data collection between two and 80 hours is what we took for the various tasks, and then we also took robot data,” Castellanos recalled. ” “Usually, this is in minutes, no more than like 80 minutes, but also as little as four,” she elaborated. “Like for tape hand, it was a very quick task.

It was only about a second. So for robot data collection, I only collected about 50 perfect episodes of tape data collection on the robot, and that was only four minutes. ” The engineer illustrated the process further: “So I think the biggest surprise for this effort was that initially we didn’t know how well this was going to work.

We were assuming that, yes, our model is like a physics model where the end effector doesn’t matter; we could be gripper-agnostic. ” “So, the first tool we worked on was the screwdriver hand, and initially it wasn’t working, but it turns out I just didn’t train it long enough on my first attempt,” acknowledged Castellanos. ” “We have a lot of different kinds of customers coming to us.

Ones who want an all-in-one-based solution. We give them everything. They run our model.

We give them the hardware. They don’t need to do anything themselves,” Castellanos replied. ” “Nothing is off the table regarding how we’re going to integrate this,” she added.

“Also, the model is improving on a weekly and almost daily basis, so much that we’re still figuring out which business model makes the most sense. ” Recalibration is a differentiator Castellanos said one of the biggest distinctions for Generalist is how quickly its model recalibrates when something goes askew. “So it learns about recovering when things fall.

It learns to hand an object from one hand to the other. All that stuff is already baked in,” she said. “So there is a certain amount of recovery data already in place.

” “I find that interesting because it is using recovery behaviors that aren’t in its specific task dataset, like using the brush with the other hand; that’s an ambidextrous behavior that just came out of nowhere,” remarked Castellanos. ” “As for deployments, I think this is definitely helpful for recovering in real time when issues happen on the line, like when people or the robot drops something,” she declared. “When things don’t go quite perfectly, there’s definitely a chance.

” Mobile manipulation and humanoids are down the road Looking ahead, as Generalist deploys more of its models in real-world settings, I questioned if humanoids would be next. “We haven’t run on a humanoid yet, but we have our own computer that’s running all of our models,” responded Castellanos. “I still think it would be feasible for a mobile robot.

We would just need to be able to put that computer on the mobile platform. So there is some integration work that does need to be done. ” It should be noted that the UMI lineage is already extending into general-purpose, humanoid-like robots.

UMI co-authors Russ Tedrake and Ben Burchfiel, both formerly with TRI, went on to co-found Walden Robotics , which is applying large behavior models to mobile manipulators in real-world industrial environments. “I’m just here for the ride to see what happens, because sometimes it’s very hard to predict how well these models are getting,” Castellanos concluded. “I would love to see it do more of that on its own without prompting.

Being able to get to the point where our models need a couple of minutes of data and can just do the task, and recover from weird edge cases. ” Walden’s mobile manipulator demonstrates gripper assembly tasks. Source: Walden Robotics The post How Generalist uses human demonstration data for robot learning appeared first on The Robot Report .