IP Library › Granted Patent US 12,350,583
Granted Patent B2
US 12,350,583 · App. 18/519,445 · Granted Jul 8, 2025

Virtual image display program, virtual image display apparatus, and virtual image display method

Inventor: Masashi Watanabe (Tokyo, JP)
Assignee: GREE HOLDINGS, Inc.
A63F13/26A63F13/211A63F13/428G06F3/011G06F3/013G06F3/017G06F3/04842G06F9/453G06T19/006A63F2300/8082
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,350,583
App. No.
18/519,445
Granted
Jul 8, 2025
Kind
B2
Abstract

A game processing apparatus displays an image of virtual space on a display that displays an image by using a sensor for identifying the position and direction of a certain body part of a player and the display. A controller in the game processing apparatus displays the image of the virtual space on the display in accordance with the position and direction of the certain body part of the player, and, when an information providing condition regarding the position and direction of the certain body part of the player is satisfied, outputs to-be-provided information.

Claims (71)

1. A non-transitory computer-readable medium, on which is embodied a computer program product comprising code executable by a virtual image display apparatus having at least circuitry and a memory, the memory being configured to store an information providing condition of the virtual image display apparatus and being further configured to store to-be-provided information, to cause the virtual image display apparatus to carry out steps of:

detecting, with a sensor operationally linked to the virtual image display apparatus, a plurality of movements of a body part of a player including a number of times that the player has changed a direction of movement of the body part, the body part comprising at least one of a head of the player and an eye of the player, and the sensor being at least one of the set of: a gyro sensor configured to measure movement of the head of the player, an acceleration sensor configured to measure movement of the head of the player, a geomagnetic sensor configured to measure movement of the head of the player and a line-of-sight sensor configured to measure movement of the eye of the player;

determining, based on each of the plurality of movements of the body part of the player, a position and a direction of the body part of the player during said plurality of movements;

determining, based on the position and the direction of the body part of the player during said plurality of movements, whether each movement is a normal movement associated with gameplay, and determining, for movements that are not the normal movements associated with the gameplay, determining, based on whether the number of times that the player has changed the direction of movement of the body part is at least a threshold value, whether each movement of the body part of the player is a predetermined movement associated with the information providing condition, and classifying at least one first movement in the plurality of movements as a predetermined movement when the number of times that the player has changed the direction of movement of the body part is larger than the threshold value and at least one second movement in the plurality of movements as a normal movement when the at least one second movement is not the predetermined movement;

displaying, on a display operationally linked to the virtual image display apparatus, in accordance with the position and direction of the body part of the player, an image of a virtual space; and

triggering, upon classification of the at least one first movement of the body part of the player as the predetermined movement and upon satisfaction of the information providing condition, provision, with the virtual image display apparatus, of the to-be-provided information to the player by displaying the to-be-provided information.

2. The non-transitory computer-readable medium of claim 1 , wherein:

detecting the plurality of movements of the body part of the player further comprises detecting a movement speed of the head of the player; and

determining whether each movement of the body part of the player is the predetermined movement associated with the information providing condition further comprises determining whether the movement speed of the head of the player satisfies a movement speed condition.

3. The non-transitory computer-readable medium of claim 1 , further comprising:

based on the at least one second movement being classified as the normal movement, identifying, with the sensor operationally linked to the virtual image display apparatus, an object in the virtual space corresponding to the position and the direction of the body part of the player; and

with the sensor operationally linked to the virtual image display apparatus, determining, based on the plurality of movements of the body part of the player, whether the object in the virtual space has been selected by the player.

4. The non-transitory computer-readable medium of claim 3 , wherein determining whether the object in the virtual space has been selected by the player comprises:

determining a gaze time over which a gaze of the player has been focused on the object; and

comparing the gaze time to a predetermined time and determining that the object has been selected when the gaze time is at least the predetermined time.

5. The non-transitory computer-readable medium of claim 1 , wherein:

the virtual space includes a target object; and

providing the to-be-provided information to the player comprises identifying an area of the virtual space other than the target object and displaying the to-be-provided information in the area of the virtual space other than the target object.

6. The non-transitory computer-readable medium of claim 1 , wherein:

the virtual space includes a moving object; and

providing the to-be-provided information to the player comprises displaying the to-be-provided information in association with the moving object and moving a display of the to-be-provided information along with the moving object.

7. The non-transitory computer-readable medium of claim 1 , wherein:

the virtual image display apparatus is configured to execute a game, the game comprising a step of displaying the image of the virtual space; and

providing the to-be provided information to the player comprises providing a hint to the player related to a next step for clearing the game.

8. The non-transitory computer-readable medium of claim 1 , wherein:

the virtual image display apparatus is configured to execute an application, the application comprising a step of displaying the image of the virtual space; and

providing the to-be-provided information to the player comprises displaying a menu item of the application.

9. The non-transitory computer-readable medium of claim 1 , wherein:

detecting, with the sensor operationally linked to the virtual image display apparatus, the plurality of movements of the body part of the player including the number of times that the player has changed the direction of movement of the body part comprises detecting, with the sensor operationally linked to the virtual image display apparatus, the plurality of movements of the body part of the player including a number of times per unit time that the player has changed the direction of movement of the body part,

determining, based on whether the number of times that the player has changed the direction of movement of the body part is at least the threshold value, whether each movement of the body part of the player is the predetermined movement associated with the information providing condition comprises determining, based on whether the number of times per unit time that the player has changed the direction of movement of the body part is at least the threshold value, whether each movement of the body part of the player is the predetermined movement associated with the information providing condition, and

classifying the at least one first movement in the plurality of movements as the predetermined movement when the number of times that the player has changed the direction of movement of the body part is larger than the threshold value and the at least one second movement in the plurality of movements as the normal movement when the at least one second movement is not the predetermined movement comprises classifying the at least one first movement in the plurality of movements as the predetermined movement when the number of times per unit time that the player has changed the direction of movement of the body part is larger than the threshold value and at least one second movement in the plurality of movements as the normal movement when the at least one second movement occurs in a predetermined area.

10. A virtual image display method for displaying an image of virtual space using a virtual image display apparatus having circuitry and a memory, the memory comprising a recording of an information providing condition of the virtual image display apparatus and to-be-provided information, the method comprising:

detecting, with a sensor operationally linked to the virtual image display apparatus, a plurality of movements of a body part of a player including a number of times that the player has changed a direction of movement of the body part, the body part comprising at least one of a head of the player and an eye of the player, and the sensor being at least one of the set of: a gyro sensor configured to measure movement of the head of the player, an acceleration sensor configured to measure movement of the head of the player, a geomagnetic sensor configured to measure movement of the head of the player and a line-of-sight sensor configured to measure movement of the eye of the player;

determining, based on each of the plurality of movements of the body part of the player, a position and a direction of the body part of the player during said plurality of movements;

determining, based on the position and the direction of the body part of the player during said plurality of movements, whether each movement is a normal movement associated with gameplay, and determining, for movements that are not the normal movements associated with the gameplay, determining, based on whether the number of times that the player has changed the direction of movement of the body part is at least a threshold value, whether each movement of the body part of the player is a predetermined movement associated with the information providing condition, and classifying at least one first movement in the plurality of movements as a predetermined movement when the number of times that the player has changed the direction of movement of the body part is larger than the threshold value and at least one second movement in the plurality of movements as a normal movement when the at least one second movement is not the predetermined movement;

displaying, on a display operationally linked to the virtual image display apparatus, in accordance with the position and direction of the body part of the player, the image of the virtual space; and

with the virtual image display apparatus, triggering, upon classification of the at least one first movement of the body part of the player as the predetermined movement and upon satisfaction of the information providing condition, provision of the to-be-provided information to the player by displaying the to-be-provided information.

11. The method of claim 10 , wherein:

detecting the plurality of movements of the body part of the player further comprises detecting a movement speed of the head of the player; and

determining whether each movement of the body part of the player is the predetermined movement associated with the information providing condition further comprises determining whether the movement speed of the head of the player satisfies a movement speed condition.

12. The method of claim 10 , further comprising:

based on the at least one second movement being classified as the normal movement, identifying, with the sensor operationally linked to the virtual image display apparatus, an object in the virtual space corresponding to the position and the direction of the body part of the player; and

with the sensor operationally linked to the virtual image display apparatus, determining, based on the plurality of movements of the body part of the player, whether the object in the virtual space has been selected by the player.

13. The method of claim 12 , wherein determining whether the object in the virtual space has been selected by the player comprises:

determining a gaze time over which a gaze of the player has been focused on the object; and

comparing the gaze time to a predetermined time and determining that the object has been selected when the gaze time is at least the predetermined time.

14. The method of claim 10 , wherein:

the virtual space includes a target object; and

providing the to-be-provided information to the player comprises identifying an area of the virtual space other than the target object and displaying the to-be-provided information in the area of the virtual space other than the target object.

15. The method of claim 10 , wherein:

the virtual space includes a moving object; and

providing the to-be-provided information to the player comprises displaying the to-be-provided information in association with the moving object and moving a display of the to-be-provided information along with the moving object.

16. The method of claim 10 , wherein:

the virtual image display apparatus is configured to execute a game, the game comprising a step of displaying the image of the virtual space; and

providing the to-be provided information to the player comprises providing a hint to the player related to a next step for clearing the game.

17. The method of claim 10 , wherein:

the virtual image display apparatus is configured to execute an application, the application comprising a step of displaying the image of the virtual space; and

providing the to-be-provided information to the player comprises displaying a menu item of the application.

18. The method of claim 10 , wherein:

detecting, with the sensor operationally linked to the virtual image display apparatus, the plurality of movements of the body part of the player including the number of times that the player has changed the direction of movement of the body part comprises detecting, with the sensor operationally linked to the virtual image display apparatus, the plurality of movements of the body part of the player including a number of times per unit time that the player has changed the direction of movement of the body part,

determining, based on whether the number of times that the player has changed the direction of movement of the body part is at least the threshold value, whether each movement of the body part of the player is the predetermined movement associated with the information providing condition comprises determining, based on whether the number of times per unit time that the player has changed the direction of movement of the body part is at least the threshold value, whether each movement of the body part of the player is the predetermined movement associated with the information providing condition, and

classifying the at least one first movement in the plurality of movements as the predetermined movement when the number of times that the player has changed the direction of movement of the body part is larger than the threshold value and the at least one second movement in the plurality of movements as the normal movement when the at least one second movement is not the predetermined movement comprises classifying the at least one first movement in the plurality of movements as the predetermined movement when the number of times per unit time that the player has changed the direction of movement of the body part is larger than the threshold value and at least one second movement in the plurality of movements as the normal movement when the at least one second movement occurs in a predetermined area.

19. A virtual image display apparatus comprising circuitry, a memory, and a sensor, and configured to display an image of virtual space on a display, the memory being configured to store an information providing condition of the virtual image display apparatus and being further configured to store to-be-provided information, the virtual image display apparatus configured to execute steps of:

detecting, with the sensor, a plurality of movements of a body part of a player including a number of times that the player has changed a direction of movement of the body part, the body part comprising at least one of a head of the player and an eye of the player, and the sensor being at least one of the set of: a gyro sensor configured to measure movement of the head of the player, an acceleration sensor configured to measure movement of the head of the player, a geomagnetic sensor configured to measure movement of the head of the player and a line-of-sight sensor configured to measure movement of the eye of the player;

determining, based on each of the plurality of movements of the body part of the player, a position and a direction of the body part of the player during said plurality of movements;

determining, based on the position and the direction of the body part of the player during said plurality of movements, whether each movement is a normal movement associated with gameplay, and determining, for movements that are not the normal movements associated with the gameplay, determining, based on whether the number of times that the player has changed the direction of movement of the body part is at least a threshold value, whether each movement of the body part of the player is a predetermined movement associated with the information providing condition, and classifying at least one first movement in the plurality of movements as a predetermined movement when the number of times that the player has changed the direction of movement of the body part is larger than the threshold value and at least one second movement in the plurality of movements as a normal movement when the at least one second movement is not the predetermined movement;

displaying, on the display, in accordance with the position and direction of the body part of the player, the image of the virtual space; and

triggering, upon classification of the at least one first movement of the body part of the player as the predetermined movement and upon satisfaction of the information providing condition, provision, with the virtual image display apparatus, of the to-be-provided information to the player by displaying the to-be-provided information.

20. The virtual image display apparatus of claim 19 , wherein the memory is further configured to store at least one of:

instructions for executing a game, the game comprising a step of displaying the image of the virtual space, wherein providing the to-be provided information to the player comprises providing a hint to the player related to a next step for clearing the game; and

instructions for executing an application, the application comprising the step of displaying the image of the virtual space, wherein providing the to-be-provided information to the player comprises displaying a menu item of the application.

Assignments (3)
CORRECTIVE ASSIGNMENT TO CORRECT THE RECEIVING PARTY STREET ADDRESS AND ATTORNEY DOCKET NUMBER PREVIOUSLY RECORDED AT REEL: 71308 FRAME: 765. ASSIGNOR(S) HEREBY CONFIRMS THE CHANGE OF NAME. Recorded Jun 10, 2025
From: GREE, INC.
To: GREE HOLDINGS, INC.
Reel/Frame 071611/0252 →
CHANGE OF NAME Recorded May 16, 2025
From: GREE, INC.
To: GREE HOLDINGS, INC.
Reel/Frame 071308/0765 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 27, 2023
From: WATANABE, MASASHI
To: GREE, INC.
Reel/Frame 065666/0311 →
Priority Claims (1)
JP 2015-183379 · Sep 16, 2015 · national
Continuity (4)
Continuation 17703021 · Mar 24, 2022
Continuation 16524311 · Jul 29, 2019
Continuation 15196410 · Jun 29, 2016
Related Publication 20240091636A1 · Mar 21, 2024
References Cited (68)
US 7722465B2 · Sumi · 2010 [cited by applicant]
US 9021390B1 · Pidhajecky · 2015 [cited by applicant]
US 9392212B1 · Ross · 2016 [cited by applicant]
US 20100259471A1 · Takano et al. · 2010 [cited by applicant]
US 20110055919A1 · Hamilton, II et al. · 2011 [cited by applicant]
US 20130274016A1 · Gagner et al. · 2013 [cited by applicant]
US 20150143221A1 · Ahuja et al. · 2015 [cited by applicant]
US 20150153913A1 · Ballard et al. · 2015 [cited by applicant]
US 20150277552A1 · Wilairat et al. · 2015 [cited by applicant]
US 20160093105A1 · Rimon et al. · 2016 [cited by applicant]
US 20160259405A1 · Wilson et al. · 2016 [cited by applicant]
US 20170046881A1 · Kuribara · 2017 [cited by applicant]
US 20170285737A1 · Khalid et al. · 2017 [cited by applicant]
US 20170315825A1 · Gordon et al. · 2017 [cited by applicant]
US 20180004285A1 · Castleman · 2018 [cited by applicant]
US 20180024633A1 · Lo · 2018 [cited by applicant]
JP 07200162A · 1995 [cited by applicant]
JP 2002176661A · 2002 [cited by applicant]
JP 2005038008A · 2005 [cited by applicant]
JP 200815942A · 2008 [cited by applicant]
JP 2009123042A · 2009 [cited by applicant]
JP 2012063253A · 2012 [cited by applicant]
JP 2012213492A · 2012 [cited by applicant]
JP 2013083731A · 2013 [cited by applicant]
JP 2014187574A · 2014 [cited by applicant]
JP 5767386B1 · 2015 [cited by applicant]
JP 2019012551A · 2019 [cited by applicant]
“Social Virtual Reality”, GREE Channel, Published on Sep. 9, 2015, Accessed on Jun. 28, 2016, URL: https://www.youtube.com/watch?v=E9hqxgk3Gn0; 2 pgs. [cited by applicant]
Office Action issued May 29, 2018 in corresponding Japanese application No. JP2015-183379; 9 pgs. [cited by applicant]
Office Action issued on Nov. 12, 2019, in corresponding Japanese Application No. 2018-168549; 4 pages. [cited by applicant]
Office Action issued on Sep. 10, 2019, in corresponding Japanese Application No. 2018-168549; 6 pages. [cited by applicant]
Office Action issued on Jan. 12, 2021, in corresponding Japanese Application No. 2020-018652; 5 pages. [cited by applicant]
VR system “I tried Morpheus!” 15 demos of angry play impressions! [E3 2015], PlayStation.Blog [online], Jun. 29, 2015, [Search on Dec. 23, 2020], Internet <URL: https://blog.ja.playstation.com/2015/06/29 /20150629_e3-20… [cited by applicant]
Paper: Petitioner's Request for Rehearing Under 37 C.F.R. §42.71 dated Dec. 14, 2020, in connection with [cited by applicant]
Supercell Exhibit 1018: Decision, [cited by applicant]
Paper No. 10: Decision Denying Institution of Post-Grant Review 35 U.S.C. § 324(a) dated Nov. 12, 2020, in connection with [cited by applicant]
Paper No. 8: Patent Owner's Preliminary Response Pursuant to 37 C.F.R. § 42.207(a) dated Aug. 12, 2020, in connection with [cited by applicant]
Gree Exhibit 2001: Declaration of Michael Shamos, in connection with [cited by applicant]
Gree Exhibit 2002: “2019 Revised Patent Subject Matter Eligibility Guidance”, Federal Register, vol. 84, No. 4, Monday, Jan. 7, 2019, in connection with [cited by applicant]
Gree Exhibit 2003: Oct. 2019 Update: Subject Matter Eligibility, in connection with [cited by applicant]
Gree Exhibit 2004: USPTO Commissioner for Patents, Memorandum dated Apr. 19, 2018, Subject: Changes in Examination Procedure Pertaining to Subject Matter Eligibility, Recent Subject Matter Eligibility Decision (Berkheim… [cited by applicant]
Gree Exhibit 2005: USPTO “Subject Matter Eligibility Examples: Abstract Ideas”, dated Jan. 7, 2019, in connection with [cited by applicant]
Gree Exhibit 2009: “Rules and Regulations”, Federal Register, vol. 83, No. 197, dated Thursday, Oct. 11, 2018, in connection with [cited by applicant]
Gree Exhibit 2010: Yaman Terzioglu, Purdue University, “Immersion and identity in video games”, Theses and Dissertations, Purdue e-Pubs, URL: http://docs.lib.purdue.edu/open_access_theses, in connection with [cited by applicant]
Gree Exhibit 2011: Daniel Örtqvist, et al., “Immersion and Gameplay Experience: A Contingency Framework”, Research Article, International Journal of Computer Games Technology, Hindawi Publishing Company, vol. 2010, Arti… [cited by applicant]
Gree Exhibit 2012: Yale Song, et al., “Continuous Body and Hand Gesture Recognition for Natural Human-Computer Interaction”, Massachusetts Institute of Technology, ACM Transactions on Interactive Intelligent Systems, vo… [cited by applicant]
Gree Exhibit 2013: Carol Manetta, et al., “Glossary of Virtual Reality Terminology”, The International Journal of Virtual Reality, Fall 1995, in connection with [cited by applicant]
Gree Exhibit 2014: Paper: Report and Recommendation, Filed Under Seal, Case No. 2:19-cv-00070- JRG-RSP, in connection with [cited by applicant]
Paper: Petition for Post Grant Review of U.S. Pat. No. 10,406,432, dated May 8, 2020, in connection with [cited by applicant]
Supercell Exhibit 1001: USP 10,406,432 B2, dated Sep. 10, 2019, in connection with [cited by applicant]
Supercell Exhibit 1002: Prosecution history of USP 10,406,432, in connection with [cited by applicant]
Supercell Exhibit 1003: Declaration of Joseph J. LaViola, Ph.D., in connection with [cited by applicant]
Supercell Exhibit 1004: USP 9,392,212 B1, dated Jul. 12, 2016, in connection with [cited by applicant]
Supercell Exhibit 1005: Chris Lankford, “Effective Eye-Gaze Input Into Windows”, Eye Tracking Research & Applications Symposium 2000, a publication of ACM SIGGRAPH, in connection with [cited by applicant]
Supercell Exhibit 1006: US Patent Publication No. 2016/0093105 A1, dated Mar. 31, 2016, in connection with [cited by applicant]
Supercell Exhibit 1007: Paper: Eric Haines, ObjectLabel—Unify Community Wiki, dated Apr. 16, 2020, at URL: http://wiki.unity3d.com/index.php?title=ObjectLabel&oldid=18700, 3 pgs, in connection with [cited by applicant]
Supercell Exhibit 1008: U.S. Patent Publication No. 2015/0153913 A1, dated Jun. 4, 2015, in connection with [cited by applicant]
Supercell Exhibit 1009: Doug A. Bowman, et al., “3D User Interfaces, Theory and Practice”, 2005, Chapter 7-Wayfinding (pp. 227-253), and Chapter 8-System Control (pp. 255-285), in connection with [cited by applicant]
Supercell Exhibit 1010: Robert J.K. Jacob, “What You Look At Is What You Get: Eye Movement-Based Interaction Techniques”, CHI '90 Proceedings, Human-Computer Interaction Lab, Naval Research Laboratory, Washington, D.C.,… [cited by applicant]
Supercell Exhibit 1011: Linda E. Sibert, et al., “Evaluation of Eye Gaze Interaction”, CHI Letters, vol. 2, No. 1, CHI 2000 The Hague, Amsterdam, Apr. 1-6, 2000, in connection with [cited by applicant]
Supercell Exhibit 1012: Joseph J. LaViola, Jr., “3D Gestural Interaction: The State of the Field”, Review Article, 2013, Hindawi Publishing Corporation, ISRN Artificial Intelligence, vol. 2013, Article ID 514641, URL: h… [cited by applicant]
Supercell Exhibit 1013: Invalidity chart of U.S. Pat. No. 10,406,432 in view of U.S. Pat. No. 9,392,212 to Ross, in connection with [cited by applicant]
Supercell Exhibit 1014: Invalidity chart of U.S. Pat. No. 10,406,432 in view of U.S. Patent Publication No. 2015/0153913 to Ballard, in connection with [cited by applicant]
Supercell Exhibit 1015: Paper: Eric Haines, ObjectLabel—Unify Community Wiki, dated Apr. 16, 2020, at URL: http://wiki.unity3d.com/index.php?title=ObjectLabel&oldid=18700, 3 pgs, in connection with [cited by applicant]
Supercell Exhibit 1003: Declaration of Guang Y Zhang for Exhibit 1015, in connection with [cited by applicant]
Supercell Exhibit 1017: Curriculum Vitae of Joseph J. LaViola, Jr., Ph.D., in connection with [cited by applicant]
Office Action issued on Jun. 28, 2022, in connection with corresponding Japanese Application No. 2021-141059 (4 pp., including machine-generated English translation). [cited by applicant]
Office Action issued on Mar. 26, 2024, in corresponding Japanese Application No. 2022-202164, 3 pages. [cited by applicant]