IP Library Granted Patent US 10,877,605
Granted Patent B2
US 10,877,605 · App. 14/223,601 · Granted Dec 29, 2020

Method and device for detecting a touch between a first object and a second object

Inventor: Daniel Kurz (Munich, DE)
Assignee: Apple Inc.
G06F3/0425G06F3/011G06F3/017G06T19/006H04N5/33
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,877,605
App. No.
14/223,601
Granted
Dec 29, 2020
Kind
B2
Abstract

The present disclosure is related to a method and device for detecting a touch between at least part of a first object and at least part of a second object, wherein the at least part of the first object has a different temperature than the at least part of the second object. The method includes providing at least one thermal image of a portion of the second object, determining in at least part of the at least one thermal image a pattern which is indicative of a particular value or range of temperature or a particular value or range of temperature change, and using the determined pattern for detecting a touch between the at least part of the first object and the at least part of the second object.

Claims (48)

1. A method of detecting a touch between at least part of a first object and at least part of a second object, wherein the first object has a different temperature than the second object, comprising:

receiving, from a thermal camera, a thermal image of a portion of a first object and a portion of a second object;

identifying, based on the thermal image, a first temperature interval associated with the first object and a second temperature interval associated with the second object;

identifying a third temperature interval between the first temperature interval and the second temperature interval; and

detecting a touch between the first object and the second object in response to detecting that a number of pixels in a group of pixels in the thermal image satisfies a size threshold, each pixel in the group of pixels having a temperature value within the third temperature interval.

2. The method according to claim 1 , where the portion of the first object and the portion of the second object overlap.

3. The method according to claim 1 , wherein the first object is at least part of a human body.

4. The method according to claim 1 , further comprising determining a position, a size, an orientation, a direction, a trajectory, or a shape of the detected touch in the thermal image.

5. The method according to claim 1 , further comprising:

outputting first information related to the touch to a machine interface program configured to change state based on the first information; and

outputting, to the machine interface program, second information related to a position or orientation of at least one of a camera, an eye, or a display relative to the second object.

6. The method according to claim 1 , further comprising:

determining a position and orientation of the thermal camera in a coordinate system of the second object;

determining a position and an orientation of an image plane of the thermal image in the coordinate system of the second object;

projecting a ray onto a model of the second object, wherein the ray originates at an origin of the thermal camera in the coordinate system of the second object and passes through a portion of the image plane corresponding to the touch; and

identifying a point where the ray intersects the model as a location of the touch on the second object; and

outputting the location on the second object of the touch for use by a human machine interface for an Augmented Reality application.

7. The method according to claim 6 , wherein the human machine interface comprises an application using one of a video-see-through system, an optical-see-through system, or a projective Augmented Reality system.

8. The method according to claim 6 , wherein the human machine interface does not include a touch screen interface.

9. The method of claim 6 , further comprising creating the model of the second object based on depth information received from a depth sensor.

10. The method according to claim 1 , further comprising manipulating displayed virtual information based on a location of the touch.

11. The method according to claim 1 , wherein the thermal image was captured at a time that corresponds to a moment during which the touch occurred.

12. The method according to claim 1 , further comprising:

displaying an image of the second object as part of a human computer interface;

determining a first location of the touch relative to the second object;

determining a second location of the touch within the image; and

generating an input event associated with the human computer interface, the input event associated with the first location and having a type selected based on the second location.

13. The method of claim 12 , wherein the type of the input event corresponds to one of a left mouse click and a right mouse click.

14. The method of claim 1 , further comprising:

displaying an image of the second object as part of a human computer interface;

determining a first location of the touch with respect to the second object;

determining a second location of an eye of a user relative to the second object; and

updating the human computer interface based on the first location and the second location.

15. The method of claim 14 , wherein updating the human computer interface includes displaying an indication of the first location, the indication oriented based on the second location.

16. The method of claim 1 , further comprising determining a histogram of temperatures depicted in the thermal image, wherein the first temperature interval and the second temperature interval are determined based on the histogram.

17. The method of claim 1 , further comprising determining the size threshold based on a size of the first object.

18. The method of claim 1 , wherein the group of pixels is detected using a blob detection technique.

19. A non-transitory computer readable medium comprising code adapted to detect a touch between at least part of a first object and at least part of a second object, wherein the first object has a different temperature than the second object, wherein the code causes one or more processors to:

receive, from a thermal camera, a thermal image of a portion of a first object and a portion of a second object;

identify, based on the thermal image, a first temperature interval associated with the first object and a second temperature interval associated with the second object;

identify a third temperature interval between the first temperature interval and the second temperature interval; and

detect a touch between the portion of the first object and the portion of the second object in response to detecting that a number of pixels in a group of pixels in the thermal image satisfies a size threshold, each pixel in the group of pixels having a temperature value within the third temperature interval.

20. A device for detecting a touch between at least part of a first object and at least part of a second object, wherein the first object has a different temperature than the second object, comprising:

a processing device adapted to receive, from a thermal camera, a thermal image of a portion of a first object and a portion of a second object, the processing device configured to:

identify, based on the thermal image, a first temperature interval associated with the first object and a second temperature interval associated with the second object;

identify a third temperature interval between the first temperature interval and the second temperature interval; and

detect a touch between the portion of the first object and the portion of the second object in response to detecting that a number of pixels in a group of pixels in the thermal image satisfies a size threshold, each pixel in the group of pixels having a temperature value within the third temperature interval.

21. The device according to claim 20 , wherein the processing device is associated with a head-mounted display, a handheld device, or a projector for performing projector-based Augmented Reality.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 7, 2021
From: KURZ, DANIEL
To: APPLE INC.
Reel/Frame 057728/0688 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 6, 2016
From: METAIO GMBH
To: APPLE INC.
Reel/Frame 040821/0462 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 23, 2015
From: KURZ, DANIEL
To: METAIO GMBH
Reel/Frame 035884/0323 →
Continuity (2)
Continuation In Part PCTEP2014053017 · Feb 17, 2014
Related Publication 20150234454A1 · Aug 20, 2015
Cited By (2)
US 12,223,726 US 12,646,353