IP Library Granted Patent US 11,337,652
Granted Patent B2
US 11,337,652 · App. 15/659,504 · Granted May 24, 2022

System and method for measuring the movements of articulated rigid bodies

Inventors: Patrick Kaifosh (New York, NY); Timothy Machado (Palo Alto, CA); Thomas Reardon (New York, NY); Erik Schomburg (Brooklyn, NY); Calvin Tong (Seattle, WA)
Assignee: Facebook Technologies, LLC
A61B5/6813A61B5/1114A61B5/7275A61B5/744G01C21/18G06F3/011G06F3/014G06F3/017G06K9/00335G06K9/6287G16H50/50A61B5/1112A61B5/6824A61B2562/0219G01C17/38
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,337,652
App. No.
15/659,504
Granted
May 24, 2022
Kind
B2
Abstract

A method for determining spatial information for a multi-segment articulated rigid body system having at least an anchored segment and a non-anchored segment coupled to the anchored segment, each segment in the multi-segment articulated rigid body system representing a respective body part of a user, the method comprising: obtaining signals recorded by a first autonomous movement sensor coupled to a body part of the user represented by the non-anchored segment; providing the obtained signals as input to a trained statistical model and obtaining corresponding output of the trained statistical model; and determining, based on the corresponding output of the trained statistical model, spatial information for at least the non-anchored segment of the multi-segment articulated rigid body system. Determining the spatial information may include determining the position and/or orientation of the non-anchored segment relative to the anchor point and/or determining a spatial relationship between the anchored and non-anchored segments.

Claims (44)

1. A computerized system for determining spatial information for a multi-segment articulated rigid body system having at least an anchored segment and a non-anchored segment connected to the anchored segment, the anchored segment anchored to an anchor point to fix a position of the anchored segment relative to a torso of a user, each segment in the multi-segment articulated rigid body system representing a respective body part of the user, the computerized system comprising:

a first autonomous movement sensor;

at least one computer hardware processor; and

at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform:

obtaining signals recorded by the first autonomous movement sensor when the first autonomous movement sensor is coupled to a body part of the user represented by the non-anchored segment, the obtained signals indicating a position and an orientation of the non-anchored segment and not indicating both a position and orientation of the anchored segment;

providing the obtained signals as input to a trained statistical model and obtaining corresponding output of the trained statistical model; and

determining, spatial information for the anchored segment based at least in part on the corresponding output of the trained statistical model, wherein the spatial information comprises position and orientation information of the anchored segment of the multi-segment articulated rigid body system.

2. The computerized system of claim 1 , wherein determining the spatial information for the anchored segment and the non-anchored segment comprises:

determining a spatial relationship between the anchored segment and the non-anchored segment.

3. The computerized system of claim 2 , wherein determining the spatial relationship between the anchored segment and the non-anchored segment comprises:

determining a set of one or more joint angles describing the spatial relationship between the anchored segment and the non-anchored segment.

4. The computerized system of claim 1 , wherein the first autonomous movement sensor comprises an inertial measurement unit (IMU).

5. The computerized system of claim 1 , wherein the first autonomous movement sensor comprises at least one sensor selected from the group consisting of a gyroscope, an accelerometer, and a magnetometer.

6. The computerized system of claim 1 , wherein the trained statistical model comprises a trained non-linear regression model.

7. The computerized system of claim 1 , wherein the trained statistical model comprises a trained recurrent neural network.

8. The computerized system of claim 1 , wherein the trained statistical model comprises a trained variational autoencoder.

9. The computerized system of claim 1 , wherein the first autonomous movement sensor is arranged on a single wearable device configured to be worn on or around a body part of the user.

10. The computerized system of claim 9 , wherein the single wearable device comprises a flexible or elastic band configured to be worn around the body part of the user.

11. The computerized system of claim 1 , wherein the processor-executable instructions, when executed by the at least one computer hardware processor, further cause the at least one computer hardware processor to perform:

sending one or more control signals to a controller configured to instruct a device to perform an action based on the one or more control signals.

12. The computerized system of claim 11 , wherein the processor-executable instructions, when executed by the at least one computer hardware processor, further cause the at least one computer hardware processor to perform executing a computer application that provides a virtual reality environment,

wherein the controller comprises a display controller configured to instruct a display to display a visual representation of a character in the virtual reality environment, and

wherein the one or more control signals comprise signals to instruct the display controller to update in real time the visual representation of the character based, at least in part, on the determined spatial information.

13. The computerized system of claim 12 , wherein the virtual reality environment comprises a virtual object and wherein updating the visual representation of the character based on the determined spatial information comprises updating the visual representation such that the character interacts with the virtual object.

14. The computerized system of claim 13 , wherein interacting with the virtual object comprises an action selected from the group consisting of grasping the virtual object, dropping the virtual object, pushing the virtual object, throwing the virtual object, pulling the virtual object, opening the virtual object, and closing the virtual object.

15. The computerized system of claim 11 , wherein the controller includes a control interface for a physical device, and wherein the one or more control signals comprise signals to instruct at least one component of the physical device to move based on the determined spatial information.

16. The computerized system of claim 1 , wherein the processor-executable instructions, when executed by the at least one computer hardware processor, further cause the at least one computer hardware processor to perform:

updating a computer-generated skeletal representation of the multi-segment articulated rigid body system based, at least in part, on the determined spatial information; and

storing, on the at least one non-transitory computer-readable storage medium, the updated computer-generated representation of the multi-segment articulated rigid body system.

17. A method for determining spatial information for determining spatial information for a multi-segment articulated rigid body system having at least an anchored segment and a non-anchored segment connected to the anchored segment, the anchored segment anchored to an anchor point to fix a position of the anchored segment relative to a torso of a user, each segment in the multi-segment articulated rigid body system representing a respective body part of the user, the method comprising:

obtaining signals recorded by a first autonomous movement sensor when the first autonomous movement sensor is coupled to a body part of the user represented by the non-anchored segment, the obtained signals indicating a position and an orientation of the non-anchored segment and not indicating both a position and orientation of the anchored segment;

providing the obtained signals as input to a trained statistical model and obtaining corresponding output of the trained statistical model; and

determining spatial information for the anchored segment based at least in part on the corresponding output of the trained statistical model, wherein the spatial information comprises position and orientation information of the anchored segment of the multi-segment articulated rigid body system.

18. A computerized system for training a statistical model for generating spatial information for a multi-segment articulated rigid body system having at least an anchored segment and a non-anchored segment connected to the anchored segment, the anchored segment anchored to an anchor point to fix a position of the anchored segment relative to a torso of a user, each segment in the multi-segment articulated rigid body system representing a respective body part of the user, the computerized system comprising:

a plurality of autonomous movement sensors;

at least one computer hardware processor; and

at least one non-transitory computer-readable storage medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, causes the at least one computer hardware processor to perform:

obtaining movement signals recorded by the plurality of autonomous movement sensors when each of the plurality of autonomous movement sensors is coupled to a body part of a first user represented by a respective segment in the multi-segment articulated rigid body system, the movement signals indicating a position of an orientation of the non-anchored segment and not indicating both a position and orientation of the anchored segment;

generating training data using the obtained movement signals;

training the statistical model using at least some of the generated training data to output a trained statistical model, wherein the trained statistical model is configured to:

generate spatial information for a multi-segment articulated rigid body system using movement signals obtained by a single movement sensor coupled to a non-anchored segment of a body part of a second user, and

generate the spatial information for the anchored segment wherein the spatial information comprises position and orientation information of the anchored segment of the multi-segment articulated rigid body system; and

storing the trained statistical model.

19. The computerized system of claim 18 , wherein the first user and the second user are a same user.

Assignments (5)
CHANGE OF NAME Recorded May 26, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060199/0876 →
CORRECTIVE ASSIGNMENT TO CORRECT THE RECEIVING PARTY DATA PREVIOUSLY RECORDED AT REEL: 051649 FRAME: 0001. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Feb 10, 2020
From: CTRL-LABS CORPORATION
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 051867/0136 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 20, 2020
From: CTRL-LABS CORPORATION
To: FACEBOOK TECHNOLOGIES, INC.
Reel/Frame 051649/0001 →
CHANGE OF NAME Recorded Oct 11, 2017
From: COGNESCENT CORPORATION
To: CTRL-LABS CORPORATION
Reel/Frame 044278/0190 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 30, 2017
From: KAIFOSH, PATRICK; MACHADO, TIMOTHY; REARDON, THOMAS; SCHOMBURG, ERIK; TONG, CALVIN
To: COGNESCENT CORPORATION
Reel/Frame 043447/0723 →
Cited By (1)
US 12,641,204