IP Library › Granted Patent US 11,020,654
Granted Patent B2
US 11,020,654 · App. 16/345,241 · Granted Jun 1, 2021

Systems and methods for interaction with an application

Inventor: Shufen Sun (Changchun, CN)
Assignee: SUZHOU YAOXINYAN TECHNOLOGY DEVELOPMENT CO., LTD.
A63F13/213A63F13/23G02B27/01G02B27/0172G06F3/012G06F3/0482G06K9/00261G06K9/00268G06K9/00302G06N3/02G06T7/248H04L67/38A63F2300/1025A63F2300/1087G02B2027/014G02B2027/0138G02B2027/0187G06F2203/011G06T2207/20084G06T2207/30201
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,020,654
App. No.
16/345,241
Granted
Jun 1, 2021
Kind
B2
Abstract

A system and method for interaction with a user are provided. The system may include a head-mounted display device, at least one sensor implemented on the head-mounted display device, at least one storage medium including a set of instructions, and at least one processor configured to communicate with the at least one storage medium. When executing the set of instructions, the system may be configured to: acquire, by the at least one sensor, a first facial image of a user; and determine, by the at least one processor, an expression of the user at least based on the first facial image of the user.

Claims (51)

1. A system, comprising:

a head-mounted display device;

at least one sensor implemented on the head-mounted display device;

at least one storage medium including a set of instructions; and

at least one processor configured to communicate with the at least one storage medium, wherein when executing the set of instructions, the system is configured to:

acquire, by the at least one sensor, a first facial image of a user; and

determine, by the at least one processor, an expression of the user at least based on the first facial image of the user, wherein to determine the expression of the user at least based on the first facial image of the user, the at least one processor is configured to:

extract a first feature from the first facial image, the first feature corresponding to a facial point of the user;

extract a second feature from a second facial image, the second facial image of the user being behind the first facial image of the user in terms of time, the second feature corresponding to the facial point of the user;

determine a movement of the facial point of the user based on the extracted first feature and the second feature; and

determine the expression of the user based on a neural network and the determined movement of the facial point of the user;

determine, by the at least one processor, an identification of the user based on the first facial image of the user; and

recommend, by the at least one processor, an item to the user based on the expression of the user and historical information associated with the identification of the user.

2. The system of claim 1 , wherein the system is further configured to:

store the determined expression of the user as a digital file in the storage medium; and

update the neural network model based on the determined expression of the user.

3. The system of claim 1 , wherein the system is further configured to:

acquire, by the at least one sensor, biological information of the user; and

process the biological information of the user and the facial image of the user to determine the expression of the user.

4. The system of claim 1 , wherein the system is further configured to:

determine an emotional state of the user based on the determined expression of the user; and

provide a feedback based on the determined emotional state of the user.

5. The system of claim 4 , wherein the system is further configured to:

provide a set of options related to the determined emotional state of the user for the user to choose; and

receive information related to a choice among the set of options of the user.

6. The system of claim 4 , wherein the system is further configured to:

adjust a running parameter of the application or the operating system.

7. The system of claim 6 , wherein the application is a game application, and the parameter is related to a rate of progress of the game application, or a difficulty level of the game application.

8. A method implemented on a head-mounted display device, comprising:

acquiring, by at least one sensor implemented on the head-mounted display device, a first facial image of a user; and

determining an expression of the user at least based on the first facial image of the user, wherein the determining the expression of the user at least based on the first facial image of the user comprises:

extracting a first feature from the first facial image, the first feature corresponding to a facial point of the user;

extracting a second feature from a second facial image, the second facial image of the user being behind the first facial image of the user in terms of time, the second feature corresponding to the facial point of the user;

determining a movement of the facial point of the user based on the extracted first feature and the second feature; and

determining the expression of the user based on a neural network and the determined movement of the facial point of the user;

determining an identification of the user based on the first facial image of the user; and

recommending an item to the user based on the expression of the user and historical information associated with the identification of the user.

9. The method of claim 8 , further comprising:

determining an emotional state of the user based on the determined expression of the user; and

providing a feedback based on the determined emotional state of the user.

10. The method of claim 9 , wherein the providing a feedback based on the determined emotional state of the user comprises:

providing a set of options related to the determined emotional state of the user for the user to choose; and

receiving information related to a choice among the set of options of the user.

11. The method of claim 9 , wherein the providing a feedback based on the determined emotional state of the user comprises:

adjusting a running parameter of an application or operating system.

12. A method implemented on a head-mounted display device having or being connected to at least one processor and storage, comprising:

obtaining, by at least one sensor implemented on the head-mounted display device, biological information of a user, wherein the biological information includes pupil information of the user;

determining, based on the obtained biological information of the user, an adjustment strategy corresponding to at least one running parameter of an application or an operating system;

adjusting the at least one running parameter of the application or the operating system;

determining an identification of the user based on the obtained biological information of the user; and

recommending an item to the user based on historical information associated with the identification of the user.

Assignments (3)
CORRECTIVE ASSIGNMENT TO CORRECT THE ASSIGNMENT PAGES PREVIOUSLY RECORDED AT REEL: 055361 FRAME: 0322. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Mar 23, 2021
From: CHANGCHUN RUIXINBOGUAN TECHNOLOGY DEVELOPMENT CO., LTD.
To: SUZHOU YAOXINYAN TECHNOLOGY DEVELOPMENT CO., LTD.
Reel/Frame 056786/0637 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 22, 2021
From: CHANGCHUN RUIXINBOGUAN TECHNOLOGY DEVELOPMENT CO., LTD.
To: SUZHOU YAOXINYAN TECHNOLOGY DEVELOPMENT CO., LTD.
Reel/Frame 055361/0322 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 28, 2019
From: SUN, SHUFEN
To: CHANGCHUN RUIXINBOGUAN TECHNOLOGY DEVELOPMENT CO., LTD.
Reel/Frame 049015/0298 →
Priority Claims (1)
CN 201621483468.1 · Dec 30, 2016 · national
Continuity (1)
Related Publication 20190275419A1 · Sep 12, 2019
Cited By (3)
US 12,313,431 US 12,320,888 US 12,433,854