Overlay technology for enhancing connectivity and realism in interacting simulations
Method providing interactive simulation comprising a first computer displaying first simulation interacting with a first user, the first computer is connected to a first head mounted display, HMD, and a first interactive control, a second computer displaying a second simulation interacting with a second user, the second computer is connected to a second HMD and a second interactive control, wherein the first computer is connected to the second computer via a network.
1 . Method providing interactive simulation comprising; a first computer displaying a first simulation interacting with a first user, the first computer is connected to a first head mounted display (HMD), and a first interactive control; a second computer displaying a second simulation interacting with a second user, the second computer is connected to a second HMD and a second interactive control, wherein the first computer is connected to the second computer via a network, and wherein both the first and second simulations are not designed to interact by providing the behaviours or even the movements of the different users characterized by:
a first overlay program running on the first computer and second overlay program running on the second computer, wherein the first overly program utilizes a first overlay engine that writes graphic information into a graphic card memory of the first computer displayed in the first HMD, to at least partially overlay graphic information of the first simulation, and the overly program utilizes a second overlay engine that writes graphic information into a graphic card memory of the second computer displayed in the second HMD to at least partially overlay graphic information of the second simulation without any other connection of the second overlay to the second simulation;
a first network-enabled communication module that facilitates the exchange of simulation state information, user control inputs, and overlay rendering data between simulation software and overlay systems via standardized interfaces and protocols running on the first computer and a second such module running on the second computer, the first network-enabled communication module communicating with the first simulation, via application programming interfaces (APIs) or standard interfaces, for receiving and/or setting status information of the first simulation, the first network-enabled communication module further communicating with the first overlay program to provide first overlay information;
a second networked-enabled communication module communicating with the second simulation, via APIs or standard interfaces, for receiving and/or setting status information of the second simulation, the second network-enabled communication module further communicating with the second overlay program to provide second overlay information;
wherein the first and second network-enabled communication modules communicate with each other over the network to exchange status information and to generate graphic information representing the other user in the simulations, displaying the respective user and their operation as an overlay over the simulation on the corresponding HMD; and
wherein a number of computers, simulations, and HMDs is at least two, all of which communicating with each other and exchanging overlay information and simulation status information, to generate a common interactive simulation experience.
2 . The method according to claim 1 wherein the first and second overlay engines generate a partially transparent overlay, which only covers depending on a head position determined by the HMD of the user parts of the graphical output in the HMD of the simulation to integrate the other user and his operations with respect to the interactive control in the simulation.
3 . The method according to claim 2 wherein the first and second overly programs implement the first and second overlay engines using a game engine and uses a driver of the game engine for rendering the overlay on top of the output of the simulation, and wherein the first and second network-enabled communication modules that facilitate the exchange of simulation status information, user control inputs, and overlay rendering data between simulation software and overlay systems via standardized interfaces and protocols is implemented as part of the game engine for interaction of the overlay program with the simulations in accordance with user inputs of the interactive controls, standardized software development kits (SDKs) and application programming interfaces (APIs) are used for analyzing and transmitting user gesture or interaction data.
4 . The method according to claim 1 wherein a media player provides additional information to the first and second overlay programs via the first and second overlay engines being displayed in the HMD, providing instructions and information to the user, wherein the additional information is imported from third party content to the first and second overlay programs provided at runtime to control the simulation or to display information and wherein the media player reacts to user inputs and simulator outputs.
5 . The method according to claim 1 wherein the first and second overly programs via the first and second overlay engines renders virtual hands reflecting the hand movements and gestures of the users for operation of the interactive controls; and
wherein the hand movements and gestures interact with defined inputs of the first and second overlay programs; the hand movements and gestures being obtained from a software development kit (SDK) for analyzing and transmitting user hand-gesture data in a consistent format, with the capability to integrate and convert non-standard tracking input into the SDK-defined format for use by the first and second overlay programs.
6 . The method according to claim 1 , wherein the first and second overly programs via the first and second overlay engines uses environment information from the simulation so that no portion of a rendered hand is displayed when the rendered hand is located behind obstacles in the simulation.
7 . The method according to claim 1 wherein the first and second simulations are identical and one of simulations runs on one computer and displayed on the first and the second computers; or wherein the first simulation and the second simulation run separately on the first and second computers respectively, and information is exchanged between the simulations.
8 . The method according to claim 1 wherein the simulations use different technologies or implementations or software but are able to interact via the first and second networked-enabled communication modules.
9 . A system for providing interactive simulation comprising: a first computer displaying first simulation interacting with a first user, the first computer is connected to a first head mounted display (HMD), and a first interactive control; a second computer displaying a second simulation interacting with a second user, the second computer connected to a second HMD and a second interactive control; wherein the first computer is connected to the second computer via a network; and, wherein the system is configured to perform the method of claim 1 .