IP Library Granted Patent US 10,121,063
Granted Patent B2
US 10,121,063 · App. 14/993,134 · Granted Nov 6, 2018

Wink gesture based control system

Inventor: Maximilian Ralph Peter von und zu Liechtenstein (Rorschach, CH)
Assignee: BMT BUSINESS MEETS TECHNOLOGY HOLDING AG
G06K9/00335G06F3/011G06F3/013G06F3/017G06F3/0233G06F3/0236G06F3/04842G06K9/00604G06F1/163G06K9/00671
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,121,063
App. No.
14/993,134
Granted
Nov 6, 2018
Kind
B2
Abstract

A wink gesture based control technique for selecting, controlling and manipulating virtual objects and smart applications on a display using facial gestures of the user, in particular winking, blinking and squinting movements of the eyes. The wink gesture based control technique utilizes wink gestures of a user to select, control, and manipulate virtual objects on a display. A head mounted device is adapted to allow detection and classification of specific wink gestures. The head mounted device in some embodiments may also be adapted to recognize certain characteristics of the wink gestures, such as duration or amplitude, to allow enhanced navigation of a computer interface.

Claims (51)

1. A method comprising:

detecting at least one wink gesture; and

determining parameters intrinsic to the at least one wink gesture; and

controlling a user interface of a computer system in response to detecting the at least one wink gesture; and

wherein controlling the user interface of a computer system in response to detecting the at least one wink gesture further comprises displaying list items as a plurality of virtual symbol cards attached to at least one virtual object, wherein movement of the at least one virtual object is controlled in response to the at least one wink gesture; and

wherein the direction of the movement of the at least one virtual object is determined by the side of the face on which the at least one wink gesture is made; and

wherein the movement of the at least one virtual object further comprises simulated torque being applied; wherein the magnitude of the simulated torque is derived from a function of the magnitude of the at least one wink gesture; and wherein the function of the magnitude of the at least one wink gesture comprises both linear and exponential functions.

2. The method of claim 1 further comprising:

selecting one of the at least two symbol cards when the movement of the at least one virtual object ceases; and

configuring the movement of the at least one virtual object so that the selecting of one of the at least two symbol cards always yields an unambiguous selection.

3. The method of claim 1 further comprising:

designating one of the plurality of virtual objects as the active object;

selecting one other of the plurality of virtual objects with the at least one wink gesture; and

designating the selected virtual object as the new active virtual object.

4. The method of claim 3 further comprising:

entering alphanumeric input into the user interface of the computer system, wherein entering the alphanumeric input comprises:

controlling both a simulated typewriter carriage and the at least one virtual object carrying the at least two symbol cards with the at least one wink gesture; and

transposing the symbol of the selected symbol card to the simulated print position on the simulated typewriter carriage; wherein the simulated typewriter carriage comprises a virtual object capable of attachment of a plurality symbol cards; wherein the simulated typewriter carriage is capable of movements from the list consisting of moving one print position to the left, moving one print position to the right, and returning n print position to the initial start position.

5. The method of claim 1 , wherein controlling the user interface of a computer system in response to detecting the at least one wink gesture further comprises a method for selecting an augmented reality object comprising:

defining a region of interest based on the at least one wink gesture;

displaying to the user a shape, wherein the shape outlines the region of interest;

displaying to the user a plurality of augmented reality objects, each of the plurality of augmented reality objects being associated with a target inside the region of interest; and

reducing or enlarging the size of the regions of interest based on the magnitude of the at least one wink gesture to form a reduced/enlarged-sized region of interest, wherein the reduced/enlarged-sized region of interest is used to select a specific augmented reality object from the plurality of augmented reality objects.

6. The method of claim 5 , wherein multiple augmentations are associated with the specific AR object, the method further comprising:

displaying to the user a corresponding augmentation from the multiple augmentations associated with the specific augmented reality object commanded by a second wink gesture.

7. The method of claim 5 , further comprising:

capturing text inside the reduced-sized region of interest and initializing translation based on the captured text commanded by a second wink gesture.

8. The method of claim 5 , further comprising:

performing a visual recognition search inside the reduced-sized region of interest commanded by a second wink gesture.

9. The method of claim 1 , further comprising:

iterating through a plurality of virtual objects, comprising the steps of:

deriving a vector from the at least one wink gesture, wherein the side of the face on which the at least one wink gesture takes place determines the direction of the vector and wherein the magnitude of the wink gesture determines the magnitude of the vector; and

controlling the direction of the iteration by the direction of the vector and; and

controlling the speed of the iteration by the magnitude of the vector.

10. A non-transitory computer-readable medium having stored therein instructions that, upon execution by a computing device, cause the computing device to perform functions comprising:

detecting a wink gesture at a head mounted device, wherein detecting a wink gesture comprises:

determining intrinsic parameters of the wink gesture wherein the intrinsic parameters are selected from the list comprising: duration of the wink gesture, magnitude of the wink gesture, side of the wink gesture; and

translating scalar values of the intrinsic parameters of the plurality of wink gestures to corresponding magnitudes of control inputs relating to input parameters of the user interface of the computing device; and

translating the directional component of the intrinsic parameters of the plurality of wink gestures to corresponding direction of control inputs relating to input parameters of the user interface of the computing device; and

translating the magnitude of a wink gesture to a corresponding control input relating to input parameters of the computing device using an exponential function.

11. A head mounted device system, comprising:

a head mounted device;

a wink detection system that is connected to the head mounted device, wherein the wink detection system is adapted to detect a wink gesture at the head mounted device, wherein the wink detection system comprises a plurality of sensors selected from the group consisting of electromyography sensors, distance sensors, motion sensors, infrared cameras, visual-range cameras and any combination thereof, wherein the wink gesture detected is selected from the group consisting of amplitude-dependent single-sided wink, duration-dependent single-sided wink, single-sided single-wink, single-sided double wink and double-sided single wink;

a display; and

a non-transitory computer-readable medium having stored therein program instructions that, upon execution by a computing device, cause the computing device to perform functions comprising:

allowing the user to control smart applications through control inputs comprising the wink gesture and wherein the wink gesture can be used to achieve tasks from the following group consisting of resizing a virtual object, iterating between different virtual objects, shifting the active focus to a different virtual object, controlling movable virtual objects, and making or confirming selections;

setting intrinsic HMD parameters comprising adjusting the speaker volume and adjusting the brightness of the display; and

wherein the response to a wink gesture input may take into account parameters selected from the group consisting of magnitude of the wink gesture, duration of the wink gesture, number of repetitions of the wink gesture, the side of the face on which the wink gesture was performed, and any combination of the said parameters.

12. The system of claim 11 further comprising a simulated mouse-operated user interface system comprising:

an eye movement detection system that is connected to the head mounted device, wherein when the head mounted device is worn the eye movement detection system is adapted to detect the movement of the user's eyes, wherein user interface of the head mounted device is adapted to display a mouse pointer, wherein the mouse pointer is configured to be controlled by the movement of the user's eyes; and

wherein the functions of the simulated mouse-operated user interface system comprise: simulating a mouse-click when the wink gesture is performed, wherein the wink gesture is selected from the group consisting of a single left wink, a single right wink, a double left wink, and a double right wink.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 12, 2016
From: VON UND ZU LIECHTENSTEIN, MAXIMILIAN RALPH PETER, MR.
To: BMT BUSINESS MEETS TECHNOLOGY HOLDING AG
Reel/Frame 037460/0152 →
Continuity (2)
Provisional Application 62102235 · Jan 12, 2015
Related Publication 20160203359A1 · Jul 14, 2016
Cited By (4)
US 12,256,198 US 12,360,591 US 12,535,891 US 12,566,489