Granted Patent B2

US 12,469,234 · App. 18/623,987 · Granted Nov 11, 2025

Inventors: Kai Zhou (Wiener Neudorf, AT); Adrian Schoisengeier (Vienna, AT)

Assignee: Snap Inc.

G06T19/006B25J13/08G06F3/012G06F3/017G06T17/00G06T19/20G06V20/64G06T2200/08G06T2210/56G06T2219/2016

Abstract

Claims (51)

1 . A method comprising:

displaying a virtual object in a display of a head-wearable device, the virtual object corresponding to a physical object in a physical environment;

tracking, using the head-wearable device, the virtual object by tracking hand gestures of a user of the head-wearable device that is co-located in a physical environment of the physical object;

detecting, using the head-wearable device, hands of the user grabbing the virtual object at a first location in the physical environment and releasing the virtual object at a second location in the physical environment;

defining an initial state of the virtual object at the first location and a final state of the virtual object at the second location based on the detecting; and

programming by demonstration a robotic system using the tracking of the virtual object, wherein the programming by demonstration comprises generating a robot program based on the initial state and the final state of the virtual object.

2 . The method of claim 1 , wherein tracking the virtual object comprises:

tracking a trajectory of the virtual object by tracking the trajectory of the head-wearable device in the physical environment.

3 . The method of claim 2 , wherein tracking the trajectory of the head-wearable device in the physical environment comprises: tracking, using a 6 degrees-of-freedom tracking system of the head-wearable device, a pose of the head-wearable device over a period of time,

wherein the trajectory of the head-wearable device is based on the pose of the head-wearable device over the period of time.

4 . The method of claim 3 , wherein the period of time is based on the hand gestures of the user of the head-wearable device,

wherein tracking the hand gestures of the user of the head-wearable device comprises: tracking a manipulation of the virtual object with respect to the physical environment.

5 . The method of claim 3 , wherein the period of time is based on:

detecting the hands of the user of the head-wearable device grabbing the virtual object corresponding to the first location in the physical environment, and

detecting the hands of the user of the head-wearable device releasing the virtual object corresponding to the second location in the physical environment.

6 . The method of claim 1 , wherein a perceived location of the virtual object corresponds to a physical location of the physical object in the physical environment.

7 . The method of claim 1 , wherein the virtual object is at an initial state at the first location, wherein the virtual object is at a final state at the second location,

wherein the programming by demonstration is based on the initial state and the final state of the virtual object.

8 . The method of claim 1 , wherein tracking the virtual object further comprises:

capturing three-dimensional spatial information of the physical environment with a sensor of the head-wearable device;

generating a three-dimensional point cloud based on the three-dimensional spatial information;

identifying the physical object from the three-dimensional point cloud;

rendering the virtual object based on the physical object identified from the three-dimensional point cloud; and

identifying the hand gestures of the user relative to the three-dimensional point cloud.

9 . A head-wearable device comprising:

a display;

a processor; and

a memory storing instructions that, when executed by the processor, configure the head-wearable device to perform operations comprising:

displaying a virtual object that corresponds to a physical object in a physical environment;

tracking, using the head-wearable device, the virtual object by tracking hand gestures of a user of the head-wearable device that is co-located in the physical environment of the physical object;

detecting, using the head-wearable device, the hands of the user grabbing the virtual object at a first location in the physical environment and releasing the virtual object at a second location in the physical environment;

defining an initial state of the virtual object at the first location and a final state of the virtual object at the second location based on the detecting; and

programming by demonstration a robotic system using the tracking of the virtual object, wherein the programming by demonstration comprises generating a robot program based on the initial state and the final state of the virtual object.

10 . The head-wearable device of claim 9 , wherein tracking the virtual object comprises:

tracking a trajectory of the virtual object by tracking the trajectory of the head-wearable device in the physical environment.

11 . The head-wearable device of claim 10 , wherein tracking the trajectory of the head-wearable device in the physical environment comprises: tracking, using a 6 degrees-of-freedom tracking system of the head-wearable device, a pose of the head-wearable device over a period of time,

wherein the trajectory of the head-wearable device is based on the pose of the head-wearable device over the period of time.

12 . The head-wearable device of claim 11 , wherein the period of time is based on the hand gestures of the user of the head-wearable device,

wherein tracking the hand gestures of the user of the head-wearable device comprises: tracking a manipulation of the virtual object with respect to the physical environment.

13 . The head-wearable device of claim 11 , wherein the period of time is based on:

detecting the hands of the user of the head-wearable device grabbing the virtual object corresponding to the first location in the physical environment, and

detecting the hands of the user of the head-wearable device releasing the virtual object corresponding to the second location in the physical environment.

14 . The head-wearable device of claim 9 , wherein a perceived location of the virtual object corresponds to a physical location of the physical object in the physical environment.

15 . The head-wearable device of claim 9 , wherein the virtual object is at an initial state at the first location, wherein the virtual object is at a final state at the second location,

wherein the programming by demonstration is based on the initial state and the final state of the virtual object.

16 . A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to perform operations comprising:

displaying a virtual object in a display of a head-wearable device, the virtual object corresponding to a physical object in a physical environment;

tracking, using the head-wearable device, the virtual object by tracking hand gestures of a user of the head-wearable device that is co-located in a physical environment of the physical object;

detecting, using the head-wearable device, hands of the user grabbing the virtual object at a first location in the physical environment and releasing the virtual object at a second location in the physical environment;

defining an initial state of the virtual object at the first location and a final state of the virtual object at the second location based on the detecting; and

programming by demonstration a robotic system using the tracking of the virtual object, wherein the programming by demonstration comprises generating a robot program based on the initial state and the final state of the virtual object.

Assignments (1)

Continuity (2)

References Cited (27)