IP Library Granted Patent US 11,009,951
Granted Patent B2
US 11,009,951 · App. 16/696,760 · Granted May 18, 2021

Wearable muscle interface systems, devices and methods that interact with content displayed on an electronic display

Inventors: Matthew Bailey (Kitchener, CA); Aaron Grant (Kitchener, CA); Stephen Lake (Kitchener, CA)
Assignee: Facebook Technologies, LLC
G06F3/015G02B27/017G06F3/011G06F3/014G06F3/016G06F3/017G06F3/0484H04W4/80
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,009,951
App. No.
16/696,760
Granted
May 18, 2021
Kind
B2
Abstract

Systems, devices and methods that enable a user to access and interact with content displayed on a portable electronic display in an inconspicuous, hands-free manner are described. There is disclosed a completely wearable system comprising a wearable muscle interface device and a wearable head-mounted display, as well as methods for using the wearable system to effect interactions between the user and content displayed on the wearable head-mounted display. The wearable muscle interface device includes muscle activity sensors worn on an arm of the user to detect muscle activity generated when the user performs a physical gesture. The wearable system is adapted to recognize a plurality of gestures made by the user and, in response to each recognized gesture, to effect one or more interaction(s) with content displayed on the wearable head-mounted display.

Claims (58)

1. A wearable muscle interface device that in use interacts with content displayed on a wearable head-mounted display, the wearable muscle interface device comprising:

a plurality of muscle activity sensors to be worn on an arm of a user, the muscle activity sensors responsive to electrical signals generated by muscles in the arm of the user;

at least one accelerometer;

a processor configured to:

generate, based on at least the electrical signals detected by the muscle activity sensors and motion of the arm of the user detected by the at least one accelerometer,

a first interpretation of a gesture performed by the user, and

a second interpretation of the gesture performed by the user, and

generate at least one control signal to request input from the user to select one of the first interpretation of the gesture or the second interpretation of the gesture, wherein the first interpretation of the gesture and the second interpretation of the gesture represent different gestures selected by the processor from a set of at least three defined gestures;

a transmitter communicatively coupled to the plurality of muscle activity sensors, wherein in use the transmitter transmits at least one signal from the wearable muscle interface device directly to a receiver on the wearable head-mounted display based on the signals detected by the muscle activity sensors;

wherein the at least one signal transmitted, in use, from the wearable muscle interface device directly to the receiver on the wearable head-mounted display effects at least one interaction with content displayed on the wearable head-mounted display including requesting the input from the user to select one of the first interpretation of the gesture performed by the user or the second interpretation of the gesture performed by the user.

2. The wearable muscle interface device of claim 1 ,

wherein the processor interprets the signals detected by the muscle activity sensors as gesture, wherein the processor is communicatively coupled in between the transmitter and the plurality of muscle activity sensors, and wherein the at least one signal that, in use, is transmitted from the wearable muscle interface device is based on the gesture interpreted by the processor of the wearable muscle interface device.

3. The wearable muscle interface device of claim 1 , wherein the wearable head-mounted display includes a processor communicatively coupled to the receiver of the wearable head-mounted display, and wherein the at least one signal that, in use, is transmitted from the wearable muscle interface device to the wearable head-mounted display is interpreted as a gesture by the processor of the wearable head-mounted display.

4. The wearable muscle interface device of claim 1 , further comprising a haptic feedback module that in use provides haptic feedback to the user, the haptic feedback module including a vibratory motor.

5. The wearable muscle interface device of claim 1 , wherein the plurality of muscle activity sensors includes at least one muscle activity sensor selected from the group consisting of: an electromyographic (EMG) sensor and a mechanomyographic (MMG) sensor.

6. The wearable muscle interface device of claim 1 ,

wherein the at least one accelerometer detects signals generated by motion of the arm of the user, the at least one accelerometer communicatively coupled to the transmitter, and wherein in use the at least one signal transmitted from the transmitter of the wearable muscle interface device directly to the receiver on the wearable head-mounted display is based on both the signals detected by the muscle activity sensors and the signals detected by the at least one accelerometer.

7. The wearable muscle interface device of claim 1 , wherein the transmitter includes a wireless transmitter.

8. A wearable system that in use provides hands-free access to and control of a portable electronic display, the wearable system comprising:

i) a wearable muscle interface device comprising:

a plurality of muscle activity sensors to be worn on an arm of a user, the muscle activity sensors responsive to electrical signals generated by muscles in the arm of the user;

at least one accelerometer;

a processor configured to:

generate, based on at least the electrical signals detected by the muscle activity sensors and motion of the arm of the user detected by the at least one accelerometer,

a first interpretation of a gesture performed by the user, and

a second interpretation of the gesture performed by the user, and

generate at least one control signal to request input from the user to select one of the first interpretation of the gesture or the second interpretation of the gesture, wherein the first interpretation of the gesture and the second interpretation of the gesture represent different gestures selected by the processor from a set of at least three defined gestures;

a transmitter communicatively coupled to the plurality of muscle activity sensors, wherein in use the transmitter transmits at least one signal from the wearable muscle interface device based on the signals detected by the muscle activity sensors;

and

ii) a wearable head-mounted display comprising:

at least one display screen to be worn on a head of the user, the at least one display screen arranged to be positioned in front of at least one eye of the user when worn on the head of the user;

a receiver communicatively coupled to the at least one display screen, wherein in use the receiver directly receives the at least one signal transmitted from the transmitter of the wearable muscle interface device; and

a processor communicatively coupled to the receiver and to the at least one display screen, wherein in use the at least one signal received directly from the transmitter of the wearable muscle interface device by the receiver of the wearable head-mounted display effects control of at least one function of the wearable head-mounted display including requesting the input from the user to select one of the first interpretation of the gesture performed by the user or the second interpretation of the gesture performed by the user.

9. The wearable system of claim 8 , wherein the transmitter of the wearable muscle interface device includes a wireless transmitter and the receiver of the wearable head-mounted display includes a wireless receiver.

10. The wearable system of claim 8 , wherein the processor interprets the signals detected by the muscle activity sensors as a gesture, wherein the processor of the wearable muscle interface device is communicatively coupled in between the transmitter and the plurality of muscle activity sensors, and wherein the at least one signal that, in use, is transmitted from the wearable muscle interface device is based on the gesture interpreted by the processor of the wearable muscle interface device.

11. The wearable system of claim 8 , wherein the plurality of muscle activity sensors includes at least one muscle activity sensor selected from the group consisting of: an electromyographic (EMG) sensor and a mechanomyographic (MMG) sensor.

12. The wearable system of claim 8 , wherein the accelerometer detects signals generated by motion of the arm of the user, the at least one accelerometer communicatively coupled to the transmitter, and wherein in use the at least one signal transmitted by the transmitter of the wearable muscle interface device is based on both the signals detected by the muscle activity sensors and the signals detected by the at least one accelerometer.

13. A method of using a wearable system to achieve hands-free access to and control of a portable electronic display, wherein the wearable system includes a wearable muscle interface device, at least one accelerometer, a processor, and a wearable head-mounted display, the method comprising:

detecting muscle activity corresponding to a physical gesture performed by a user of the wearable system by at least one muscle activity sensor of the wearable muscle interface device;

generating, based on at least the electrical signals detected by the at least one muscle activity sensor and motion of the arm of the user detected by the at least one accelerometer,

a first interpretation of a gesture performed by the user, and

a second interpretation of the gesture performed by the user, and

generating at least one control signal to request input from the user to select one of the first interpretation of the gesture or the second interpretation of the gesture, wherein the first interpretation of the gesture and the second interpretation of the gesture represent different gestures selected by the processor from a set of at least three defined gestures;

transmitting at least one signal from the wearable muscle interface device by a transmitter of the wearable muscle interface device based at least in part on the muscle activity detected by at least one muscle activity sensor of the wearable muscle interface device;

receiving the at least one signal directly from the wearable muscle interface device by a receiver of the wearable head-mounted display;

processing the at least one signal by a processor of the wearable head-mounted display and;

effecting at least one interaction between the user and the wearable head-mounted display by the processor of the wearable head-mounted display based on the processing of the at least one signal by the processor of the wearable head-mounted display including requesting the input from the user to select one of the first interpretation of the gesture performed by the user or the second interpretation of the gesture performed by the user.

14. The method of claim 13 , further comprising:

in response to detecting muscle activity corresponding to a physical gesture performed by a user of the wearable system by at least one muscle activity sensor of the wearable muscle interface device, processing the detected muscle activity by the processor of the wearable muscle interface device, and wherein transmitting at least one signal from the wearable muscle interface device by a transmitter of the wearable muscle interface device based at least in part on the muscle activity detected by at least one muscle activity sensor of the wearable muscle interface device includes transmitting at least one signal from the wearable muscle interface device by the transmitter of the wearable muscle interface device based at least in part on processing the detected muscle activity by the processor of the wearable muscle interface device.

15. The method of claim 13 , further comprising:

detecting motion of the wearable muscle interface device corresponding to the physical gesture performed by the user of the wearable system by the at least one accelerometer of the wearable muscle interface device, and wherein transmitting at least one signal from the wearable muscle interface device by a transmitter of the wearable muscle interface device based at least in part on the muscle activity detected by at least one muscle activity sensor of the wearable muscle interface device includes transmitting at least one signal from the wearable muscle interface device by the transmitter of the wearable muscle interface device based on both the muscle activity detected by at least one muscle activity sensor of the wearable muscle interface device and the motion detected by at least one accelerometer of the wearable muscle interface device.

16. The method of claim 13 , wherein:

transmitting at least one signal from the wearable muscle interface device by a transmitter of the wearable muscle interface device includes wirelessly transmitting at least one signal from the wearable muscle interface device by a wireless transmitter of the wearable muscle interface device; and

receiving the at least one signal directly from the wearable muscle interface device by a receiver of the wearable head-mounted display includes wirelessly receiving the at least one signal directly from the wearable muscle interface device by a wireless receiver of the wearable head-mounted display.

17. The method of claim 13 , wherein a haptic feedback module provides haptic feedback to the user.

18. The method of claim 17 , wherein the haptic feedback module includes a vibratory motor.

19. The method of claim 13 , wherein the at least one muscle activity sensor comprises a muscle activity sensor selected from the group consisting of: an electromyographic (EMG) sensor and a mechanomyographic (MMG) sensor.

20. The method of claim 13 , wherein the transmitter of the wearable muscle interface device includes a wireless transmitter and the receiver of the wearable head-mounted display includes a wireless receiver.

Assignments (6)
CHANGE OF NAME Recorded May 26, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060199/0876 →
CORRECTIVE ASSIGNMENT TO CORRECT THE RECEIVING PARTY DATA PREVIOUSLY RECORDED AT REEL: 051649 FRAME: 0001. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Feb 10, 2020
From: CTRL-LABS CORPORATION
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 051867/0136 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 20, 2020
From: CTRL-LABS CORPORATION
To: FACEBOOK TECHNOLOGIES, INC.
Reel/Frame 051649/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 18, 2019
From: BAILEY, MATTHEW; GRANT, AARON; LAKE, STEPHEN
To: THALMIC LABS INC.
Reel/Frame 051322/0199 →
CHANGE OF NAME Recorded Dec 18, 2019
From: THALMIC LABS INC.
To: NORTH INC.
Reel/Frame 051367/0046 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 18, 2019
From: NORTH INC.
To: CTRL-LABS CORPORATION
Reel/Frame 051367/0055 →
Cited By (5)
US 12,554,325 US 12,579,768 US 12,591,304 US 12,629,095 US 12,704,906