IP Library Granted Patent US 9,606,618
Granted Patent B2
US 9,606,618 · App. 15/057,382 · Granted Mar 28, 2017

Hand tracker for device with display

Inventors: Nadav Grossinger (Tel-Aviv, IL); Israel Grossinger (Karmei Yoseef, IL); Nitay Romano (Geva Binyamin, IL)
Assignee: Facebook, Inc.
G06F3/005G06F3/017
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,606,618
App. No.
15/057,382
Granted
Mar 28, 2017
Kind
B2
Abstract

An apparatus for remote hand control of a computer device, the apparatus comprising: a hand tracker, configured to track movement of a hand of a user of the computer device, an image presenter, associated with the hand tracker, configured to present a first image representative of the hand and a second image to the user, and to move the first image in concert with the tracked movement, and a computer controller, associated with the image presenter and configured to control the computer device in accordance with an interaction between the first image and the second image, thereby allowing the user to control the computer device, by moving the hand for bringing the first image and the second image into the interaction.

Claims (42)

1. A computer implemented method comprising:

obtaining a time series of images of a hand of a user of a computer device, the hand projected with a light structured in a pattern that enables tracking the hand, the pattern of the structured light comprises a plurality of stripes arranged substantially in parallel to each other forming a continuous feature of the structured light;

detecting segmentation of the stripes based on changes in intensity profiles of light reflected from the hand along the continuous feature;

tracking a movement of the hand using the images by identifying, in the pattern of the structured light, at least one cluster of stripe segments created by the segmentation of the stripes by at least one part of the hand and tracking the at least one cluster of stripe segments;

presenting, on a display of the computer device, a first image representative of the hand and a second image representative of an interface of a computer application running on the computer device;

moving the first image in concert with the tracked movement of the hand;

detecting an interaction between the first image and the second image, the interaction represented by the first image with respect to the second image on the display of the computer device; and

invoking a functionality of the computer application based on the detected interaction.

2. The method of claim 1 , wherein the tracking of the movement of the hand comprises using bi-dimensional video data.

3. The method of claim 1 , wherein the pattern of the structured light enables tracking movement of digits of the hand in bi-dimensional video data.

4. The method of claim 1 , wherein the pattern of the structured light enables tracking movement of digits and a palm of the hand in bi-dimensional video data based on distortions of the pattern by the digits and the palm.

5. The method of claim 1 , wherein the tracking of the movement of the hand further comprises identifying a depth movement of the at least one part of the hand, by detecting a change in a number of the stripe segments in the at least one tracked cluster.

6. The method of claim 1 , wherein the pattern of the structured light comprises a grid pattern, and the method further comprises:

detecting a predefined gesture in the movement of the hand by projecting the structured light having the grid pattern onto the hand.

7. An apparatus, comprising:

a light projector configured to project, onto a hand of a user of a computer device, a light structured in a pattern that enables tracking of the hand, the pattern of the structured light comprises a plurality of stripes arranged substantially in parallel to each other forming a continuous feature of the structured light;

a camera configured to capture a time series of images of the hand by capturing images of the hand projected with the pattern of the structured light;

a hand tracker, associated with the camera, configured to detect segmentation of the stripes based on changes in intensity profiles of light reflected from the hand along the continuous feature,

the hand tracker is further configured to track a movement of the hand using the images by identifying, in the pattern of the structured light, at least one cluster of stripe segments created by the segmentation of the stripes by at least one part of the hand and tracking the at least one cluster of stripe segments;

an image presenter, associated with the hand tracker, configured to present, on a display of the computer device, a first image representative of the hand and a second image representative of an interface of a computer application running on the computer device; and

a computer controller, associated with the image presenter, configured to move the first image in concert with the tracked movement of the hand, wherein the computer controller is further configured to

detect an interaction between the first image and the second image, the interaction represented by the first image with respect to the second image on the display of the computer device, and

invoke a functionality of the computer application based on the detected interaction.

8. The apparatus of claim 7 , wherein the hand tracker is further configured to use bi-dimensional video data for tracking the movement of the hand.

9. The apparatus of claim 7 , wherein the pattern of the structured light enables tracking movement of digits of the hand in bi-dimensional video data.

10. The apparatus of claim 7 , wherein the pattern of the structured light enables tracking movement of digits and a palm of the hand in bi-dimensional video data based on distortions of the pattern by the digits and the palm.

11. The apparatus of claim 7 , wherein the hand tracker is further configured to identify a depth movement of the at least one part of the hand, by detecting a change in a number of the stripe segments in the at least one tracked cluster.

12. The apparatus of claim 7 , wherein the camera is further configured to capture the first image by capturing an image of the hand projected with the pattern of the structured light.

13. The apparatus of claim 7 , wherein the light projector is remote from the computer device.

14. The apparatus of claim 7 , wherein the camera is remote from the computer device.

15. The apparatus of claim 7 , further comprising a gesture recognizer, associated with the hand tracker, configured to detect a predefined gesture in the tracked movement of the hand by projecting the structured light having a grid pattern onto the hand, wherein the computer controller is further configured to control the computer device in a manner predefined for the detected gesture.

16. The apparatus of claim 15 , wherein the image presenter is further configured to carry out one member of the group consisting of: aligning of the first image into a predefined position, upon the detection of the predefined gesture, resizing the first image, upon the detection of the predefined gesture, and resizing the second image, upon the detection of the predefined gesture.

17. The apparatus of claim 7 , wherein the computer controller is a part of one member of the group consisting of a mobile phone, a device installed in a car and a musical instrument.

18. The apparatus of claim 7 , wherein the hand tracker is remote from the computer device.

19. A non-transitory computer readable medium storing computer executable instructions for performing steps, the steps comprising:

obtaining a time series of images of a hand of a user of a computer device, the hand projected with a light structured in a pattern that enables tracking the hand, the pattern of the structured light comprises a plurality of stripes arranged substantially in parallel to each other forming a continuous feature of the structured light;

detecting segmentation of the stripes based on changes in intensity profiles of light reflected from the hand along the continuous feature;

tracking a movement of the hand using the images by identifying, in the pattern of the structured light, at least one cluster of stripe segments created by the segmentation of the stripes by at least one part of the hand and tracking the at least one cluster of stripe segments;

presenting, on a display of the computer device, a first image representative of the hand and a second image representative of an interface of a computer application running on the computer device;

moving the first image in concert with the tracked movement of the hand;

detecting an interaction between the first image and the second image, the interaction represented by the first image with respect to the second image on the display of the computer device; and

invoking a functionality of the computer application based on the detected interaction.

Assignments (6)
CHANGE OF NAME Recorded Jun 8, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060315/0224 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 30, 2018
From: FACEBOOK, INC.
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 047687/0942 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 7, 2016
From: GROSSINGER, NADAV; GROSSINGER, ISRAEL; ROMANO, NITAY
To: PEBBLESTECH LTD
Reel/Frame 040245/0813 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 7, 2016
From: PEBBLES LTD.
To: OCULUS VR, LLC
Reel/Frame 040245/0855 →
CHANGE OF NAME Recorded Nov 7, 2016
From: PEBBLESTECH LTD.
To: PEBBLES LTD
Reel/Frame 040654/0790 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 4, 2016
From: OCULUS VR, LLC
To: FACEBOOK, INC.
Reel/Frame 040196/0790 →
Continuity (3)
Continuation 13497589
Provisional Application 61244473 · Sep 22, 2009
Related Publication 20160179188A1 · Jun 23, 2016