IP Library › Granted Patent US 12,399,572
Granted Patent B2
US 12,399,572 · App. 18/403,764 · Granted Aug 26, 2025

Methods and systems for touchless control with a mobile device

Inventors: Wang Fai Ng (Hong Kong, HK); Wing Hung Chan (Hong Kong, HK); Long Mak (Hong Kong, HK); Keng Fai Lee (Cupertino, CA); On Loy Sung (Hong Kong, HK)
Assignee: NEX Team Inc.
G06F3/017G06F3/04812G06F3/04815G06N3/08G06T7/73G06V40/28
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,399,572
App. No.
18/403,764
Granted
Aug 26, 2025
Kind
B2
Abstract

Methods and systems for touchless control of a device include receiving at least one image; determining a first map, the first map including a function that generates a correspondence between data associated with the image to a feature space; determining a second map including a representation of a user in the image based on the first map; determining a third map comprising a field associated with the data in the image; determining a characteristic of the user based on one or more of the second map and the third map; and tracking the characteristic of the user to generate a historical data of the characteristic for performing the touchless control of the device.

Claims (56)

1. A device, comprising:

at least one memory device that stores computer-executable instructions; and

at least one processor configured to access the memory device, wherein the processor is configured to execute the computer-executable instructions to:

receive at least one image;

determine a first map, the first map comprising a function that generates a correspondence between data associated with the image to a feature space;

determine a second map comprising a representation of a user in the image based on the first map;

determine a third map comprising a field associated with the data in the image;

determine a characteristic of the user based on one or more of the second map and the third map;

track the characteristic of the user for performing control of the device to generate a historical data of the characteristic;

determine, based on the tracking, a mapping between the characteristic and an input feature associated with the device;

modify the input feature to a first location based on the mapping; and

modify the input feature to a second location different from the first location based on the historical data.

2. The device of claim 1 , wherein the computer-executable instructions to determine the characteristic comprise computer-executable instructions to determine that the characteristic has an associated duration exceeding a predetermined threshold.

3. The device of claim 1 , wherein the computer-executable instructions to track the characteristic comprise computer-executable instructions to apply a filter to measurements during the tracking.

4. The device of claim 1 , further comprising computer-executable instructions to determine a second mapping between a second characteristic of the user and the input feature by applying a mathematical transformation to the characteristic.

5. The device of claim 4 , further comprising computer-executable instructions to modify the input feature at a different speed or at a different precision based on at least one of the mapping or the second mapping.

6. The device of claim 1 , further comprising computer-executable instructions to:

track the characteristic of the user at a frame rate;

determine that a value of a refresh rate associated with the device is different than a value of the frame rate; and

modify the input feature using a smoothing function, the smoothing function comprising parameters that are determined based on at least one of the frame rate and the refresh rate.

7. A method, comprising:

receiving at least one image;

determining a first map, the first map comprising a function that generates a correspondence between data associated with the image to a feature space;

determining a second map comprising a representation of a user in the image based on the first map;

determining a third map comprising a field associated with the data in the image;

determining a characteristic of the user based on one or more of the second map and the third map;

tracking the characteristic of the user for performing control of the device to generate a historical data of the characteristic;

determining, based on the tracking, a mapping between the characteristic and an input feature associated with the device;

modifying the input feature to a first location based on the mapping; and

modifying the input feature to a second location different from the first location based on the historical data.

8. The method of claim 7 , wherein the determining the characteristic further comprises determining that the characteristic has an associated duration exceeding a predetermined threshold.

9. The method of claim 7 , wherein the tracking the characteristic further comprises applying a filter to measurements during the tracking.

10. The method of claim 7 , further comprising determining a second mapping between a second characteristic of the user and the input feature by applying a mathematical transformation to the characteristic.

11. The method of claim 10 , further comprising modifying the input feature at a different speed or at a different precision based on at least one of the mapping or the second mapping.

12. The method of claim 7 , further comprising:

tracking the characteristic of the user at a frame rate;

determining that a value of a refresh rate associated with the device is different than a value of the frame rate; and

modifying the input feature using a smoothing function, the smoothing function comprising parameters that are determined based on at least one of the frame rate and the refresh rate.

13. A non-transitory computer-readable medium storing computer-executable instructions which, when executed by a processor, cause the processor to perform operations comprising:

receiving at least one image;

determining a first map, the first map comprising a function that generates a correspondence between data associated with the image to a feature space;

determining a second map comprising a representation of a user in the image based on the first map;

determining a third map comprising a field associated with the data in the image;

determining a characteristic of the user based on one or more of the second map and the third map;

tracking the characteristic of the user for performing control of the device to generate a historical data of the characteristic;

determining, based on the tracking, a mapping between the characteristic and an input feature associated with the device;

modifying the input feature to a first location based on the mapping; and

modifying the input feature to a second location different from the first location based on the historical data.

14. The non-transitory computer-readable medium of claim 13 , wherein the computer-executable instructions to determine the characteristic comprise computer-executable instructions to determine that the characteristic has an associated duration exceeding a predetermined threshold.

15. The non-transitory computer-readable medium of claim 13 , wherein the computer-executable instructions to track the characteristic comprise computer-executable instructions to apply a filter to measurements during the tracking.

16. The non-transitory computer-readable medium of claim 13 , further comprising computer-executable instructions to determine a second mapping between a second characteristic of the user and the input feature by applying a mathematical transformation to the characteristic.

17. The non-transitory computer-readable medium of claim 16 , further comprising computer-executable instructions to modify the input feature at a different speed or at a different precision based on at least one of the mapping or the second mapping.

18. The non-transitory computer-readable medium of claim 13 , further comprising computer-executable instructions to:

track the characteristic of the user at a frame rate;

determine that a value of a refresh rate associated with the device is different than a value of the frame rate; and

modify the input feature using a smoothing function, the smoothing function comprising parameters that are determined based on at least one of the frame rate and the refresh rate.

Assignments (2)
SECURITY INTEREST Recorded Oct 9, 2025
From: NEX TEAM INC.
To: SILVER LAKE WATERMAN FUND III, L.P., AS AGENT
Reel/Frame 072524/0882 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 4, 2024
From: NG, WANG FAI; CHAN, WING HUNG; MAK, LONG; LEE, KENG FAI; SUNG, ON LOY
To: NEX TEAM, INC.
Reel/Frame 066013/0985 →
Continuity (3)
Continuation 16824950 · Mar 20, 2020
Continuation 16432575 · Jun 5, 2019
Related Publication 20240134463A1 · Apr 25, 2024
References Cited (37)
US 7834847B2 · Boillot et al. · 2010 [cited by applicant]
US 8686943B1 · Rafii · 2014 [cited by applicant]
US 9886624B1 · Marty et al. · 2018 [cited by applicant]
US 10010778B2 · Marty et al. · 2018 [cited by applicant]
US 20070130547A1 · Boillot · 2007 [cited by applicant]
US 20100278384A1 · Shotton et al. · 2010 [cited by applicant]
US 20100295781A1 · Alameh · 2010 [cited by applicant]
US 20100309122A1 · Abe · 2010 [cited by applicant]
US 20110110560A1 · Adhikari · 2011 [cited by examiner]
US 20120070070A1 · Litvak · 2012 [cited by examiner]
US 20120239174A1 · Shotton · 2012 [cited by examiner]
US 20120277001A1 · Lansdale et al. · 2012 [cited by applicant]
US 20130303281A1 · Argiro · 2013 [cited by applicant]
US 20140104168A1 · Hegde · 2014 [cited by applicant]
US 20140212008A1 · Hatcher, II et al. · 2014 [cited by applicant]
US 20150261408A1 · Ostergren · 2015 [cited by applicant]
US 20170168586A1 · Sinha · 2017 [cited by applicant]
US 20180218243A1 · Felsen et al. · 2018 [cited by applicant]
US 20190004609A1 · Swissa · 2019 [cited by applicant]
US 20190025909A1 · Mittal et al. · 2019 [cited by applicant]
US 20190042838A1 · Bose et al. · 2019 [cited by applicant]
US 20190046886A1 · George et al. · 2019 [cited by applicant]
US 20190050111A1 · Shigeta · 2019 [cited by applicant]
US 20190228533A1 · Giurgica-Tiron · 2019 [cited by examiner]
US 20200012350A1 · Tay · 2020 [cited by examiner]
US 20200058148A1 · Blaylock · 2020 [cited by examiner]
US 20200387232A1 · Ng · 2020 [cited by examiner]
Krzysztof Przednowek, et al., “A System for Analysing the Basketball Free Throw Trajectory Based on Particle Swarm Optimization,” Applied Sciences, 2018, vol. 8, Issue 11, p. 2090, MDPI, Basel, Switzerland. [cited by applicant]
Simone Francia, “SpaceJam: a Dataset for Basketball Action Recognition,” Github Code Repository Page, available at: https://github.com/simonefrancia/SpaceJam, last access: Apr. 2, 2019. [cited by applicant]
Techsmith Corporation, “Coach's Eye,” Coach's Eye website, available at: https://www.coachseye.com/, last accessed: Feb. 18, 2019. [cited by applicant]
Stats LLC, “Stats SportVU Basketball Player Tracking,” SportVU website, available at: https://www.stats.com/sportvu-basketball/, last accessed: Feb. 12, 2019. [cited by applicant]
Vignesh Ramanathan, et al., “Detecting events and key actors in multi-person videos,” The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 3043-3053. [cited by applicant]
Mark Sandler, et al., “MobileNetV2: Inverted Residuals and Linear Bottlenecks,” The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 4510-4520, available at: https://arxiv.org/abs/1801.04381. [cited by applicant]
Wei Liu, et al., “SSD: Single Shot MultiBox Detector,” European conference on computer vision, pp. 21-37, Springer, Cham, 2016, available at: https://arxiv.org/abs/1512.02325. [cited by applicant]
Zhe Cao, et al., “Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields,” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7291-7299, 2017, available at: https://arxiv.org… [cited by applicant]
Andrew G. Howard, et al., “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,” arXiv preprint arXiv:1704.04861, 2017, available at: https://arxiv.org/abs/1704.04861. [cited by applicant]
Bluestack Systems, Inc., “Play mobile games hands-free on TV Minority Report style,” BlueStacks Blog, Mar. 21, 2016, available at: https://www.bluestacks.com/blog/bluestacks-exclusives/play-mobile-games-hands-free-on-tv… [cited by applicant]