Granted Patent B1

US 11,630,633 · App. 17/852,532 · Granted Apr 18, 2023

Inventor: Francis MacDougall (Milton, CA)

Assignee: PROMP, INC.

G06F3/1454G06F3/011G06F3/147

Abstract

Claims (64)

1. A computer-implemented method comprising:

receiving, by a collaborator computing device, streaming images taken by a streamer computing device, and depth data for an environment of the streamer computing device;

generating and providing, for output by the collaborator computing device, a collaborator user interface, the collaborator user interface including the received streaming images;

receiving, by the collaborator computing device, hand tracking data based on a user of the collaborator computing device reaching into a hand tracking area;

determining a total distance available for virtual reaches into the environment of the streamer computing device based on the depth data;

determining a virtual reach distance for the user of the collaborator computing device based at least on (i) a location of the reach of the user of the collaborator computing device into the hand tracking area, and (ii) the total distance available for virtual reaches into the environment of the streamer computing device;

updating, by the collaborator computing device, the collaborator user interface to include a representation of the hand tracking data, positioned based on the virtual reach distance; and

transmitting, by the collaborator computing device, the hand tracking data and the virtual reach distance to the streamer computing device for inclusion on a streamer user interface that is generated and provided for output by the streamer computing device.

2. The computer-implemented method of claim 1 , wherein the collaborator computing device is a virtual reality computing device and the hand tracking data is generated by the collaborator computing device.

3. The computer-implemented method of claim 1 , wherein:

the streaming images are taken at a location that is remote from the collaborator computing device;

the collaborator user interface can display the streaming images taken at the location that is remote from the collaborator computing device;

the hand tracking data is transmitted to the streamer computing device that is at the location that is remote from the collaborator computing device; and

the streamer user interface can display the streaming images taken at the location that is remote from the collaborator computing device along with a representation of the hand tracking data.

4. The computer-implemented method of claim 1 , wherein the hand tracking data is generated by a hand tracking device that is separate from the collaborator computing device.

5. The computer-implemented method of claim 1 , further comprising:

capturing, by the collaborator computing device, audio/video data of the user of the collaborator computing device; and

transmitting, by the collaborator computing device and to the streamer computing device, the audio/video data of the user of the collaborator computing device.

6. The computer-implemented method of claim 1 , wherein the streaming images include different sets of images, wherein each set of images is taken by a different camera.

7. The computer-implemented method of claim 6 , wherein generating the collaborator user interface comprises including, in the collaborator user interface, a separate portion of the collaborator user interface for each set of images of the different sets of images.

8. The computer-implemented method of claim 7 , further comprising:

receiving, by the collaborator computing device, a selection of a first portion of the collaborator user interface corresponding to a first set of images taken by a first camera;

updating, by the collaborator computing device, the first portion of the collaborator user interface to include the representation of the hand tracking data; and

transmitting, by the collaborator computing device and to the streamer computing device, in association with the transmitting of the hand tracking data, an indication of the selection of the first portion.

9. The computer-implemented method of claim 1 , further comprising:

receiving, by the collaborator computing device, second streaming images taken by a second streamer computing device;

updating, by the collaborator computing device, the collaborator user interface to include the streaming images in a first portion of the collaborator user interface and the second streaming images in a second portion of the collaborator user interface;

receiving, by the collaborator computing device, a selection of the second portion of the collaborator user interface;

updating, by the collaborator computing device, the second portion of the collaborator user interface, to include a representation of the hand tracking data; and

transmitting, by the collaborator computing device, the hand tracking data to the second streamer computing device.

10. A computer-implemented method comprising:

generating, by a streamer computing device, streaming images taken by the streamer computing device;

generating, by the streamer computing device, depth data for an environment of the streamer computing device;

generating and providing, for output by the streamer computing device, a streamer user interface, the streamer user interface including the generated streaming images;

transmitting, by the streamer computing device, the streaming images and the depth data to the collaborator computing device;

receiving, by the streamer computing device, hand tracking data and a virtual reach distance for a user of the collaborator computing device that was generated by the collaborator computing device, wherein the virtual reach distance is determined based at least on the depth data for the environment of the streamer computing device; and

updating, by the streamer computing device, the streamer user interface to include a representation of the hand tracking data, positioned based on the virtual reach distance, and the streaming images.

11. The computer-implemented method of claim 10 , wherein the streamer computing device is a mobile device.

12. The computer-implemented method of claim 11 , wherein the streamer computing device is a virtual reality device.

13. The computer-implemented method of claim 11 , wherein the streaming images are of the user of the streamer computing device captured using a front-facing camera of the streamer computing device.

14. The computer-implemented method of claim 11 , wherein the streaming images are captured using a rear-facing camera of the streamer computing device.

15. The computer-implemented method of claim 10 , wherein the streaming images are of a display screen of the streamer computing device.

16. The computer-implemented method of claim 10 , further comprising mapping, by the streamer computing device, received hand tracking data to the streamer user interface to generate mapped hand tracking data, and wherein updating the streamer user interface to include the representation of the hand tracking data comprises generating the representation of the hand tracking data based on the mapped hand tracking data.

17. The computer-implemented method of claim 10 , further comprising:

detecting, by the streamer computing device, a finger tap gesture from the hand tracking data;

determining, by the streamer computing device, a location within the streamer user interface that corresponds to the finger tap gesture; and

updating, by the streamer computing device, the streamer user interface to include an indicator of the finger tap gesture at the location within the streamer user interface that corresponds to the finger tap gesture.

18. The computer-implemented method of claim 10 , further comprising:

detecting, by the streamer computing device, a freeze-stream gesture from the hand tracking data;

updating, by the streamer computing device, the streamer user interface to display a single image of the streaming images;

receiving, by the streamer computing device, updated hand tracking data; and

displaying, by the streamer computing device and in the streamer user interface on the single image, an updated representation of hand tracking data, based on the updated hand tracking data.

19. The computer-implemented method of claim 10 , further comprising:

receiving, by the streamer computing device and from the collaborator computing device, audio/video data of the user of the collaborator computing device; and

updating, by the streamer computing device, the streamer user interface to present the audio/video data of the user of the collaborator computing device.

20. A computer-implemented method comprising:

generating, by a streamer computing device, streaming images taken by the streamer computing device;

generating and providing, for output by the streamer computing device, a streamer user interface, the streamer user interface including the streaming images as video content;

transmitting, by the streamer computing device, the streaming images to the collaborator computing device;

receiving, by the streamer computing device, hand tracking data for a user of the collaborator computing device that was generated by the collaborator computing device;

detecting, by the streamer computing device, a freeze-stream gesture from the hand tracking data that was generated by the collaborator computing device;

updating, by the streamer computing device, the streamer user interface to pause the video content by displaying a single, still image of the streaming images without displaying any other of the streaming images;

receiving, by the streamer computing device, updated hand tracking data that was generated by the collaborator computing device; and

displaying, by the streamer computing device and in the streamer user interface and overlaid on the single, still image of the streaming image, an updated representation of hand tracking data, based on the updated hand tracking data.

Assignments (1)

Continuity (1)

Cited By (10)

US 12,266,067 US 12,299,150 US 12,333,065 US 12,361,106 US 12,482,192 US 12,493,705 US 12,505,239 US 12,573,152 US 12,578,786 US 12,608,087