Granted Patent B2

US 12,638,859 · App. 19/362,617 · Granted May 26, 2026

Inventors: Corey Lynch (San Jose, CA); Toki Migimatsu (San Jose, CA); Michael Ahn (San Jose, CA)

Assignee: FIGURE AI

G05D1/495B62D57/032G06F40/40G06V10/766G06V10/774G05D2101/15G05D2109/12G05D2111/52G05D2111/58

Abstract

Claims (82)

1 . A method for controlling a humanoid robot using a hierarchical bipedal action model (BAM), comprising:

obtaining a hierarchical BAM that is generated by:

collecting training data comprising: (i) obtaining video data from a 3 rd party database, and (ii) obtaining real-world robot demonstrations;

preprocessing the training data to form distinct segments that have natural language descriptions;

training, using the preprocessed training data, both a Beta model and an Alpha model; and

deploying the trained hierarchical BAM on a humanoid robot; and

controlling the humanoid robot to perform an autonomous task using the trained hierarchical BAM.

2 . The method of claim 1 , wherein deploying the hierarchical BAM comprises loading the Alpha model on a GPU that is positioned within the humanoid robot.

3 . The method of claim 1 , wherein at least one of the Alpha model and the Beta model is a diffusion model.

4 . The method of claim 1 , wherein the Beta model operates at a frequency that is less than 50 Hz.

5 . The method of claim 4 , wherein the Beta model is trained using a cross-entropy loss function and has more than 500 million parameters.

6 . The method of claim 1 , wherein the Alpha model operates at a frequency that is greater than 50 Hz.

7 . The method of claim 6 , wherein the Alpha model is trained using a regression-based loss function and has fewer than 500 million parameters.

8 . The method of claim 1 , wherein the Alpha and Beta models are trained end-to-end, and wherein said training includes allowing error gradients from the output of the Alpha model to be backpropagated through the Beta model.

9 . The method of claim 1 , further comprising the step of executing a safety verification that rejects or truncates outputs from the Alpha model that violate a predetermined constraint.

10 . The method of claim 1 , wherein the step of collecting training data includes capturing first-person video data synchronized with head and hand positions using a virtual reality or augmented reality headset.

11 . The method of claim 1 , wherein the natural language descriptions are generated using an AI model.

12 . The method of claim 1 , wherein obtaining a hierarchical BAM includes the step of obtaining at least one pre-trained model having a set of parameters; and

wherein training the hierarchical BAM includes using supervised learning to modify the set of parameters based in part on the preprocessed training data.

13 . The method of claim 2 , wherein deploying the trained hierarchical BAM includes loading the Beta model on a GPU that is not positioned within the humanoid robot.

14 . A method for controlling a robot using a hierarchical action model, comprising:

obtaining a hierarchical action model by:

collecting training data comprising obtaining: (i) video data from a 3 rd party database, (ii) first-person video data, and (iii) data from a robot demonstration;

training, using the training data, a hierarchical action model that includes an Alpha model and a Beta model, and wherein the Alpha model has a first number of parameters and the Beta model has a second number of parameters that is larger than the first number of parameters; and

deploying the trained hierarchical action model on a robot; and

controlling the robot to perform an autonomous task using the trained hierarchical action model.

15 . The method of claim 14 , wherein controlling the robot includes the step of using a retrieval-augmented generation technique to obtain additional real-time knowledge from external sources.

16 . The method of claim 14 , wherein at least one of the Alpha and Beta models is a diffusion model.

17 . The method of claim 16 , wherein the Alpha model is trained using reinforcement learning.

18 . The method of claim 17 , wherein the Beta model is trained using unsupervised learning.

19 . The method of claim 14 , wherein controlling the robot includes generating continuous actions comprising a first set of X, Y, Z floating-point coordinates and a first set of X, Y, Z floating-point rotations.

20 . The method of claim 14 , wherein deploying the hierarchical action model comprises loading the Alpha model on a GPU that is positioned within the robot.

21 . The method of claim 20 , wherein deploying the trained hierarchical action model includes loading the Beta model on a GPU that is not positioned within the robot.

22 . The method of claim 14 , wherein the first-person video data is captured using a virtual reality or augmented reality headset.

23 . A method for controlling a robot using an action model, comprising:

obtaining an action model by:

collecting training data comprising obtaining: (i) video data from a 3 rd party database, (ii) first-person video data, and (iii) data from a robot demonstration;

training an action model using the training data, and wherein the action model includes more than 1 billion parameters; and

deploying the trained action model on a robot; and

using the trained action model to generate continuous meter-actions.

24 . The method of claim 23 , wherein the first-person video data is captured using a virtual reality or augmented reality headset.

25 . The method of claim 23 , wherein the continuous actions include a first set of X, Y, Z floating-point coordinates and a first set of X, Y, Z floating-point rotations.

26 . The method of claim 23 , wherein the action model is trained using reinforcement learning and the deployment of the action model includes loading the model on a GPU located within the robot.

27 . The method of claim 23 , further comprising using a retrieval-augmented generation technique to obtain additional real-time knowledge from sources that are external to the robot.

28 . The method of claim 23 , further comprising the step of executing a safety verification that rejects or truncates outputs from the action model that violate a predetermined constraint.

29 . The method of claim 1 , wherein controlling the humanoid robot includes the step of using a retrieval-augmented generation technique to obtain additional real-time knowledge from external sources.

30 . The method of claim 1 , wherein the Beta model is trained using unsupervised learning and the Alpha model is trained using reinforcement learning.

31 . The method of claim 1 , wherein the hierarchical BAM further comprises a Gamma model that includes more than 1 trillion parameters, and wherein the Gamma model is not deployed on the humanoid robot.

32 . The method of claim 1 , wherein controlling the humanoid robot includes using the trained hierarchical BAM to generate continuous actions.

33 . The method of claim 1 , wherein controlling the humanoid robot includes using the trained hierarchical BAM to generate X, Y, Z floating-point coordinates and X, Y, Z floating-point rotations.

34 . The method of claim 1 , wherein the humanoid robot includes at least thirty electric actuators, and wherein at least ten of the thirty electric actuators include a strain wave gearbox.

35 . The method of claim 1 , wherein a majority of the humanoid robot is covered with a deformable textile that includes polymers.

36 . The method of claim 35 , wherein the humanoid robot includes an illumination assembly configured to communicate the humanoid robot's status.

37 . The method of claim 14 , wherein the Beta model operates at a frequency that is less than 50 Hz and has more than 500 million parameters.

38 . The method of claim 37 , wherein the Alpha model operates at a frequency that is greater than 50 Hz and has fewer than 500 million parameters.

39 . The method of claim 38 , wherein the Alpha and Beta models are trained end-to-end, and wherein said training includes allowing error gradients from the output of the Alpha model to be backpropagated through the Beta model.

40 . The method of claim 39 , wherein the Beta model is trained using a cross-entropy loss function and the Alpha model is trained using a regression-based loss function.

41 . The method of claim 14 , wherein the robot includes at least thirty electric actuators, and wherein at least one of the thirty electric actuators includes a strain wave gearbox.

42 . The method of claim 41 , wherein a majority of the robot is covered with a deformable textile that includes polymers.

43 . The method of claim 14 , wherein the hierarchical action model further comprises a Gamma model that includes more than 1 trillion parameters, and wherein the Gamma model is locally hosted relative to the robot.

44 . The method of claim 23 , wherein the action model includes an Alpha model that operates at a frequency that is greater than 50 Hz and has fewer than 500 million parameters.

45 . The method of claim 44 , wherein the action model includes a Beta model that operates at a frequency that is less than 50 Hz and has more than 500 million parameters.

46 . The method of claim 45 , wherein the Beta model is trained using a cross-entropy loss function and the Alpha model is trained using a regression-based loss function.

47 . The method of claim 23 , wherein the action model includes an Alpha model and a Beta model, and said Alpha and Beta models are trained end-to-end, and wherein said training includes allowing error gradients from the output of the Alpha model to be backpropagated through the Beta model.

48 . The method of claim 23 , wherein the robot includes at least thirty electric actuators, and wherein at least one of the thirty electric actuators includes a strain wave gearbox.

49 . The method of claim 48 , wherein a majority of the robot is covered with a deformable textile that includes polymers.

50 . The method of claim 23 , wherein the action model further comprises a Gamma model that includes more than 1 trillion parameters, and wherein the Gamma model is locally hosted relative to the robot.

51 . A method for controlling a robot using a model, comprising:

obtaining a model by:

collecting training data comprising obtaining: (i) video data from a 3 rd party database, and (ii) data from a robot demonstration;

training a model using the training data, and wherein the trained model includes more than 1 billion parameters; and

deploying the trained model on a robot; and

using the trained model to generate X, Y, Z floating-point coordinates and X, Y, Z floating-point rotations.

52 . The method of claim 51 , further comprising using a retrieval-augmented generation technique to obtain additional real-time knowledge from sources that are external to the robot.

53 . The method of claim 51 , wherein the model includes: (i) a Beta model that operates at a frequency that is less than 50 Hz, and (ii) an Alpha model that operates at a frequency that is more than 50 Hz.

54 . The method of claim 53 , wherein the Alpha and Beta models are trained end-to-end, and wherein said training includes allowing error gradients from the output of the Alpha model to be backpropagated through the Beta model.

55 . The method of claim 51 , wherein the model includes: (i) an Alpha model that has fewer than 500 million parameters, and (ii) a Beta model that has more than 500 million parameters.

56 . The method of claim 55 , wherein the Alpha and Beta models are trained end-to-end, and wherein said training includes allowing error gradients from the output of the Alpha model to be backpropagated through the Beta model.

57 . The method of claim 56 , wherein the model includes: (i) an Alpha model that is trained using a regression-based loss function, and (ii) a Beta model that is trained using a cross-entropy loss function.

58 . The method of claim 51 , wherein the robot includes at least thirty electric actuators, and wherein at least one of the thirty electric actuators includes a strain wave gearbox.

59 . The method of claim 58 , wherein a majority of the robot is covered with a deformable textile that includes polymers.

60 . The method of claim 51 , wherein the model further comprises a Gamma model that includes more than 1 trillion parameters, and wherein the Gamma model is locally hosted relative to the robot.

Assignments (1)

Continuity (10)

References Cited (327)