IP Library Granted Patent US 11,014,001
Granted Patent B2
US 11,014,001 · App. 15/911,780 · Granted May 25, 2021

Building virtual reality (VR) gaming environments using real-world virtual reality maps

Inventors: Miao Li (Carlsbad, CA); Shawn He (San Diego, CA)
Assignee: Sony Interactive Entertainment LLC
A63F13/65A63F13/352G06T17/00G06T19/00G06T19/006A63F2300/69A63F2300/8082
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,014,001
App. No.
15/911,780
Granted
May 25, 2021
Kind
B2
Abstract

A method for building a gaming environment. The method includes accessing a base VR model of a real-world environment from a third party mapping data store, wherein the real-world environment includes a plurality of real-world objects. The method includes augmenting a first object in the base VR model that corresponds to a first real-world object in the real-world environment. The method includes stitching in the first object that is augmented into the base VR model to generate an augmented base VR model. The method includes storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models for use as virtual environments of corresponding gaming applications storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models. The method includes using one or more of the plurality of augmented base VR models to define a virtual environment of a gaming application.

Claims (77)

1. A method, comprising:

accessing a base VR model of a real-world environment from a third party mapping data store, wherein the real-world environment includes a plurality of real-world objects;

detecting a first real-world object in the real-world environment based on one or more images or videos of the real-world environment provided as input into a deep learning engine including a plurality of learned models of objects, wherein the deep learning engine includes a network of interconnected nodes within an input layer, one or more hidden layers, and an output layer;

augmenting a first object in the base VR model that corresponds to the first real-world object in the real-world environment;

stitching in the first object that is augmented into the base VR model to generate an augmented base VR model;

storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models; and

using one or more of the plurality of augmented base VR models to define a virtual environment of a gaming application.

2. The method of claim 1 , further comprising:

receiving a request for the one or more of the plurality of augmented base VR models from a remote device; and

sending the one or more of the plurality of augmented base VR models to the remote device for interfacing with a game logic of the gaming application, wherein the game logic is executing on the remote device in association with a game play of a first user.

3. The method of claim 2 , wherein the remote device is a server configured to execute a plurality of game logics for a plurality of gaming applications for a plurality of users.

4. The method of claim 2 , wherein the remote device is a console of the first user.

5. The method of claim 1 , further comprising:

making available the augmented base VR model from the library data store through a software development kit (SDK) to a remote device for use in developing a game logic of the gaming application configured to interface with the augmented base VR model.

6. The method of claim 1 , wherein the augmenting the first object includes:

determining that a first surface model of the first real-world object in the real-world environment is incomplete in the base VR model, wherein the base VR model includes a plurality of surface models, each surface model associated with a corresponding real-world object;

accessing a plurality of images of the first real-world object from an object data store;

augmenting the first surface model that is incomplete with a surface segment built from the plurality of images of the first real-world object to generate an augmented first surface model; and

using the augmented first surface model for the first surface model that is incomplete when performing the stitching.

7. The method of claim 1 , wherein the augmenting the first object includes:

determining that a first surface model of the first real-world object in the real-world environment is incomplete in the base VR model, wherein the base VR model includes a plurality of surface models, each surface model associated with a corresponding real-world object;

accessing a previously generated complete first surface model from an object data store; and

using the previously generated complete first surface model to represent the first real- world object when performing the stitching.

8. The method of claim 1 , wherein the augmenting the first object includes:

determining that a first surface model of the first real-world object in the real-world environment is incomplete in the base VR model, wherein the base VR model includes a plurality of surface models, each surface model associated with a corresponding real-world object;

accessing a plurality of images of the first real-world object from an object data store;

generating a complete first surface model of the first real-world object using the plurality of images; and

using the complete first surface model to represent the first real-world object when performing the stitching.

9. The method of claim 1 , further comprising:

determining a gap region in the base VR model, wherein the gap region is devoid of at least one object;

accessing a filler object to fill in the gap region; and

stitching in the filler object into the base VR model to generate the augmented base VR model.

10. The method of claim 1 , wherein the real-world environment comprises one of a live event, a museum, and a city.

11. A non-transitory computer-readable medium storing a computer program, the computer-readable medium comprising:

program instructions for accessing a base VR model of a real-world environment from a third party mapping data store, wherein the real-world environment includes a plurality of real-world objects;

program instructions for detecting a first real-world object in the real-world environment based on one or more images or videos of the real-world environment provided as input into a deep learning engine including a plurality of learned models of objects, wherein the deep learning engine includes a network of interconnected nodes within an input layer, one or more hidden layers, and an output layer;

program instructions for augmenting a first object in the base VR model that corresponds to the first real-world object in the real-world environment;

program instructions for stitching in the first object that is augmented into the base VR model to generate an augmented base VR model;

program instructions for storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models; and

program instructions for using one or more of the plurality of augmented base VR models to define a virtual environment of a gaming application.

12. The computer-readable medium of claim 11 , further comprising:

program instructions for receiving a request for the one or more of the plurality of augmented base VR models from a remote device; and

program instructions for sending the one or more of the plurality of augmented base VR models to the remote device for interfacing with a game logic of the gaming application, wherein the game logic is executing on the remote device in association with a game play of a first user.

13. The computer-readable of claim 11 , further comprising:

program instructions for making available the augmented base VR model from the library data store through a software development kit (SDK) to a remote device for use in developing a game logic of the gaming application configured to interface with the augmented base VR model.

14. The computer-readable of claim 11 , wherein the program instructions for augmenting the first object comprises:

program instructions for determining that a first surface model of the first real-world object in the real-world environment is incomplete in the base VR model, wherein the base VR model includes a plurality of surface models, each surface model associated with a corresponding real-world object;

accessing a plurality of images of the first real-world object from an object data store;

augmenting the first surface model that is incomplete with a surface segment built from the plurality of images of the first real-world object to generate an augmented first surface model; and

using the augmented first surface model for the first surface model that is incomplete when performing the stitching in the first object that is augmented.

15. The computer-readable of claim 11 , further comprising:

program instructions for determining a gap region in the base VR model, wherein the gap region is devoid of at least one object;

program instructions for accessing a filler object to fill in the gap region; and

program instructions for stitching in the filler object into the base VR model to generate the augmented base VR model.

16. A computer system comprising:

a processor; and

memory coupled to the processor and having stored therein instructions that, if executed by the computer system, cause the computer system to execute a method comprising:

accessing a base VR model of a real-world environment from a third party mapping data store, wherein the real-world environment includes a plurality of real-world objects;

detecting a first real-world object in the real-world environment based on one or more images or videos of the real-world environment provided as input into a deep learning engine including a plurality of learned models of objects, wherein the deep learning engine includes a network of interconnected nodes within an input layer, one or more hidden layers, and an output layer;

augmenting a first object in the base VR model that corresponds to the first real-world object in the real-world environment;

stitching in the first object that is augmented into the base VR model to generate an augmented base VR model;

storing the augmented base VR model to a library data store comprising a plurality of augmented base VR models; and

using one or more of the plurality of augmented base VR models to define a virtual environment of a gaming application.

17. The computer system of claim 16 , wherein the method further comprises:

receiving a request for the one or more of the plurality of augmented base VR models from a remote device; and

sending the one or more of the plurality of augmented base VR models to the remote device for interfacing with a game logic of the gaming application, wherein the game logic is executing on the remote device in association with a game play of a first user.

18. The computer system of claim 16 , wherein the method further comprises:

making available the augmented base VR model from the library data store through a software development kit (SDK) to a remote device for use in developing a game logic of the gaming application configured to interface with the augmented base VR model.

19. The computer system of claim 16 , wherein the augmenting the first object in the method comprises:

determining that a first surface model of the first real-world object in the real-world environment is incomplete in the base VR model, wherein the base VR model includes a plurality of surface models, each surface model associated with a corresponding real-world object;

accessing a plurality of images of the first real-world object from an object data store;

augmenting the first surface model that is incomplete with a surface segment built from the plurality of images of the first real-world object to generate an augmented first surface model; and

using the augmented first surface model for the first surface model that is incomplete when performing the stitching.

20. The computer system of claim 16 , wherein the method further comprises:

determining a gap region in the base VR model, wherein the gap region is devoid of at least one object;

accessing a filler object to fill in the gap region; and

stitching in the filler object into the base VR model to generate the augmented base VR model.

Assignments (2)
MERGER Recorded Mar 30, 2020
From: SONY INTERACTIVE ENTERTAINMENT AMERICA LLC
To: SONY INTERACTIVE ENTERTAINMENT LLC
Reel/Frame 053323/0567 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 5, 2018
From: LI, MIAO; HE, SHAWN
To: SONY INTERACTIVE ENTERTAINMENT AMERICA LLC
Reel/Frame 045109/0651 →