IP Library Granted Patent US 12,625,521
Granted Patent B2
US 12,625,521 · App. 18/410,469 · Granted May 12, 2026

Virtual input device, virtual input system, virtual input method, and recording medium

Inventors: Yoshiaki Nozawa (Kanagawa, JP); Daisuke Shimada (Kanagawa, JP); Takuya Yagyu (Kanagawa, JP); Shinichi Yamanaka (Kanagawa, JP)
Assignee: NEC Platforms, Ltd.
G06F1/1673G02B27/0101G02B27/017G06F3/011G06F21/32G06F21/44G02B2027/0138G02B2027/014
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,625,521
App. No.
18/410,469
Granted
May 12, 2026
Kind
B2
Abstract

Provided is a virtual input device that includes an input/output unit that acquires image data captured by a camera of a virtual reality display device and output display information to be displayed on a display of the virtual reality display device to the virtual reality display device, a video analysis unit that analyzes the image data to specify a position at which a virtual terminal is to be displayed, and a display information generation unit that generates the display information including the virtual terminal based on user information of a user wearing the virtual reality display device.

Claims (55)

1 . A virtual input device comprising:

a memory storing instructions; and

a processor connected to the memory and configured to execute the instructions to:

acquire image data captured by a camera of a virtual reality display device and output display information to be displayed on a display of the virtual reality display device to the virtual reality display device;

analyze the image data to detect a physical operation terminal disposed in a real world from the image data;

specify a position of the physical operation terminal on the display of the virtual reality display device

determine a position of each key of the physical operation terminal based on at least one mark attached to the physical operation terminal;

generate the display information in which the virtual terminal having a key arrangement customized according to user information of a user wearing the virtual reality display device is displayed at the specified position of the physical operation terminal such that the virtual terminal is superimposed on the physical operation terminal;

display the generated display information including the virtual terminal superimposed on the physical operation terminal on the display of the virtual reality display device.

2 . The virtual input device according to claim 1 , wherein

the processor is configured to execute the instructions to

generate the display information in which a key to be operated by the user among a plurality of keys included in the virtual terminal is highlighted, according to the user information.

3 . The virtual input device according to claim 2 , wherein

the processor is configured to execute the instructions to

detect an operation terminal disposed in a real world from the image data,

specify a position of the operation terminal on the display of the virtual reality display device, and

generate the display information in which the virtual terminal is displayed at the specified position of the operation terminal.

4 . The virtual input device according to claim 1 , wherein

the processor is configured to execute the instructions to

analyze the image data and recognize an operation of the user, and

generate the display information based on the recognized operation of the user.

5 . The virtual input device according to claim 1 , wherein

the processor is configured to execute the instructions to

acquire associated information based on the user information, and

generate the display information in which a key associated to the associated information is highlighted.

6 . The virtual input device according to claim 1 , wherein

the processor is configured to execute the instructions to

acquire authentication information of the user who uses the virtual reality display device and perform authentication of the user based on the acquired authentication information, and

generate the display information for the virtual reality display device used by the authenticated user.

7 . A virtual input system comprising:

the virtual input device according to claim 1 ; and

a virtual reality display device that includes a camera and a display, wherein

the virtual reality display device is configured to

output image data captured by the camera to the virtual input device,

acquire display information output from the virtual input device, and

display an image based on the acquired display information on the display of the virtual reality display device.

8 . The virtual input system according to claim 7 , further comprising

an operation terminal that is disposed in a real world and operated by a user wearing the virtual input device, wherein

the processor of the virtual input device is configured to execute the instructions to

generate the display information in which a virtual terminal is displayed at a position of the operation terminal, and

output the generated display information to the virtual reality display device, and

the virtual reality display device is configured to

display an image based on the display information on the display of the virtual reality display device.

9 . A virtual input method executed by a computer, the method comprising:

acquiring image data captured by a camera of a virtual reality display device;

analyzing the image data to detect a physical operation terminal disposed in a real world from the image data;

specify a position of the physical operation terminal on the display of the virtual reality display device at which a virtual terminal is to be displayed; and

generating display information in which the virtual terminal having a key arrangement customized according to user information of a user wearing the virtual reality display device is displayed at the specified position of the physical operation terminal such that the virtual terminal is superimposed on the physical operation terminal;

displaying the generated display information including the virtual terminal superimposed on the physical operation terminal on the display of the virtual reality display device.

10 . A non-transitory recording medium recording a program for causing a computer to execute processing of:

acquiring image data captured by a camera of a virtual reality display device;

analyzing the image data to detect a physical operation terminal disposed in a real world from the image data;

specify a position of the physical operation terminal on the display of the virtual reality display device

generating display information in which the virtual terminal having a key arrangement customized according to user information of a user wearing the virtual reality display device is displayed at the specified position of the physical operation terminal such that the virtual terminal is superimposed on the physical operation terminal;

displaying the generated display information including the virtual terminal superimposed on the physical operation terminal on the display of the virtual reality display device.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 11, 2024
From: NOZAWA, YOSHIAKI; SHIMADA, DAISUKE; YAGYU, TAKUYA; YAMANAKA, SHINICHI
To: NEC PLATFORMS, LTD.
Reel/Frame 066102/0117 →
Priority Claims (1)
JP 2023-017951 · Feb 8, 2023 · national
Continuity (1)
Related Publication 20240264638A1 · Aug 8, 2024
References Cited (86)
US 5689669A · Lynch · 1997 [cited by examiner]
US 6002853A · de Hond · 1999 [cited by examiner]
US 6079982A · Meader · 2000 [cited by examiner]
US 6119147A · Toomey · 2000 [cited by examiner]
US 6179619B1 · Tanaka · 2001 [cited by examiner]
US 6219045B1 · Leahy · 2001 [cited by examiner]
US 6243091B1 · Berstis · 2001 [cited by examiner]
US 6271843B1 · Lection · 2001 [cited by examiner]
US 6362817B1 · Powers · 2002 [cited by examiner]
US 6396522B1 · Vu · 2002 [cited by examiner]
US 6414679B1 · Miodonski · 2002 [cited by examiner]
US 6570563B1 · Honda · 2003 [cited by examiner]
US 6573903B2 · Gantt · 2003 [cited by examiner]
US 6590593B1 · Robertson · 2003 [cited by examiner]
US 6621508B1 · Shiraishi · 2003 [cited by examiner]
US 6690393B2 · Heron · 2004 [cited by examiner]
US 6784901B1 · Harvey · 2004 [cited by examiner]
US 6961055B2 · Doak · 2005 [cited by examiner]
US 7119819B1 · Robertson · 2006 [cited by examiner]
US 7382288B1 · Wilson · 2008 [cited by examiner]
US 7414629B2 · Santodomingo · 2008 [cited by examiner]
US 7467356B2 · Gettman · 2008 [cited by examiner]
US 7542040B2 · Templeman · 2009 [cited by examiner]
US 7653877B2 · Matsuda · 2010 [cited by examiner]
US 7663625B2 · Chartier · 2010 [cited by examiner]
US 7746343B1 · Charaniya · 2010 [cited by examiner]
US 7788323B2 · Greenstein · 2010 [cited by examiner]
US 7804507B2 · Yang · 2010 [cited by examiner]
US 7814429B2 · Buffet · 2010 [cited by examiner]
US 7817150B2 · Reichard · 2010 [cited by examiner]
US 7844724B2 · Van Wie · 2010 [cited by examiner]
US 9244533B2 · Friend · 2016 [cited by examiner]
US 9696795B2 · Marcolina · 2017 [cited by examiner]
US 9996797B1 · Holz · 2018 [cited by examiner]
US 10516853B1 · Gibson · 2019 [cited by examiner]
US 11417054B1 · Tanner et al. · 2022 [cited by applicant]
US 20010018667A1 · Kim · 2001 [cited by examiner]
US 20020095463A1 · Matsuda · 2002 [cited by examiner]
US 20020113820A1 · Robinson · 2002 [cited by examiner]
US 20040113887A1 · Pair · 2004 [cited by examiner]
US 20040193441A1 · Altieri · 2004 [cited by examiner]
US 20050093719A1 · Okamoto · 2005 [cited by examiner]
US 20050128212A1 · Edecker · 2005 [cited by examiner]
US 20080030429A1 · Hailpern · 2008 [cited by examiner]
US 20080125218A1 · Collins · 2008 [cited by examiner]
US 20080235570A1 · Sawada · 2008 [cited by examiner]
US 20080246693A1 · Hailpern · 2008 [cited by examiner]
US 20090076791A1 · Rhoades · 2009 [cited by examiner]
US 20090091583A1 · McCoy · 2009 [cited by examiner]
US 20090287728A1 · Martine · 2009 [cited by examiner]
US 20090300528A1 · Stambaugh · 2009 [cited by examiner]
US 20100070378A1 · Trotman · 2010 [cited by examiner]
US 20100115428A1 · Shuping · 2010 [cited by examiner]
US 20100205541A1 · Rapaport · 2010 [cited by examiner]
US 20100214284A1 · Rieffel · 2010 [cited by examiner]
US 20100274567A1 · Carlson · 2010 [cited by examiner]
US 20100274627A1 · Carlson · 2010 [cited by examiner]
US 20110010636A1 · Hamilton, II · 2011 [cited by examiner]
US 20110041083A1 · Gabai · 2011 [cited by examiner]
US 20110225536A1 · Shams · 2011 [cited by examiner]
US 20120249741A1 · Maciocci · 2012 [cited by examiner]
US 20130304729A1 · Jiang · 2013 [cited by examiner]
US 20140035819A1 · Griffin · 2014 [cited by applicant]
US 20140082526A1 · Park · 2014 [cited by examiner]
US 20140282105A1 · Nordstrom · 2014 [cited by examiner]
US 20170185261A1 · Perez · 2017 [cited by examiner]
US 20170243403A1 · Daniels · 2017 [cited by examiner]
US 20180131907A1 · Schmirler · 2018 [cited by examiner]
US 20180357472A1 · Dreessen · 2018 [cited by examiner]
US 20190087015A1 · Lam · 2019 [cited by examiner]
US 20200110928A1 · Al Jazaery · 2020 [cited by examiner]
US 20200117267A1 · Gibson · 2020 [cited by examiner]
US 20200117270A1 · Gibson · 2020 [cited by examiner]
US 20200309944A1 · Thoresen · 2020 [cited by examiner]
US 20200356610A1 · Coimbra · 2020 [cited by examiner]
US 20200368616A1 · Delamont · 2020 [cited by examiner]
US 20210008413A1 · Asikainen · 2021 [cited by examiner]
US 20210041951A1 · Gibson · 2021 [cited by examiner]
JP H11272386A · 1999 [cited by applicant]
JP 2004164517A · 2004 [cited by applicant]
JP 2014192687A · 2014 [cited by applicant]
JP 2016177658A · 2016 [cited by applicant]
JP 6826029B · 2021 [cited by applicant]
JP 7176792B1 · 2022 [cited by applicant]
WO 2018225149A1 · 2018 [cited by applicant]
JP Office Action for JP Application No. 2023-017951, mailed on Mar. 18, 2025 with English Translation. [cited by applicant]