IP Library Granted Patent US 10,769,843
Granted Patent B1
US 10,769,843 · App. 16/598,745 · Granted Sep 8, 2020

3D scene engineering simulation and real-life scene fusion system

Inventor: Shih-Cheng Wang (New Taipei, TW)
Assignee: HONGFUJIN PRECISION ELECTRONICS(TIANJIN)CO., LTD.
G06T15/20G06T2215/16
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,769,843
App. No.
16/598,745
Granted
Sep 8, 2020
Kind
B1
Abstract

A method performs 3D scene engineering simulation and real-life scene fusion. The method includes obtaining 3D coordinates of each point of a 3D model of a real-life scene, obtaining engineering data corresponding to the 3D coordinates of each point, acquiring location information of each point, establishing a correspondence between the 3D coordinates of each point and the location information of each point in the real-life scene, providing a scene fusion interactive interface on a mobile terminal, acquiring an image of a target scene in the real-life scene, acquiring location information and orientation information of the mobile terminal, determining 3D coordinates corresponding to the location information of the mobile terminal, determining a 3D coordinate range, and merging the engineering data corresponding to the 3D coordinate range into the image of the target scene.

Claims (63)

1. A three-dimensional scene engineering simulation and real-life scene fusion system (3D system) comprising:

at least one processor; and

at least one memory storing a plurality of instructions, which when executed by the at least one processor, cause the processor to:

obtain three-dimensional coordinates of each point of a three-dimensional model of a real-life scene;

obtain engineering data corresponding to the three-dimensional coordinates of each point analyzed according to the three-dimensional model;

acquire location information of at least one point of the real-life scene;

establish a correspondence between the three-dimensional coordinates of each point of the three-dimensional model and the location information of the at least one point of the real-life scene according to the acquired location information of the at least one point of the real-life scene and the three-dimensional coordinates corresponding to the at least one point;

provide a scene fusion interactive interface on a display screen of a mobile terminal;

acquire an image of a target scene in the real-life scene in real time in response to an operation of capturing an image on the display screen;

acquire location information and orientation information of the mobile terminal;

determine three-dimensional coordinates corresponding to the location information of the mobile terminal in the three-dimensional model according to the correspondence relationship and the location information of the mobile terminal;

determine a three-dimensional coordinate range corresponding to the acquired image according to the three-dimensional model, the determined three-dimensional coordinates, the orientation information, and a size of the display screen; and

merge the engineering data corresponding to the three-dimensional coordinate range into the image of the target scene according to an operation of merging the scene on the display screen to form a realistic scene engineering simulation image.

2. The 3D system of claim 1 , wherein:

the location information comprises a longitude and a latitude; and

the longitude and latitude are obtained from a Global Positioning System (GPS).

3. The 3D system of claim 1 , wherein:

the scene fusion interaction interface provides a plurality of fusion options; and

each fusion option represents different engineering parameters to be fused.

4. The 3D system of claim 1 , wherein the at least one processor is further configured to:

remove an object selected in the image of the target scene and fill in an empty space of the removed object according to a peripheral area around the removed object.

5. The 3D system of claim 1 , wherein the at least one processor is further configured to:

save the real-life scene simulation image in real time.

6. The 3D system of claim 1 , wherein the at least one processor is further configured to:

add additional information onto the real-life scene simulation image according to an operation of adding information on the display screen; and

automatically add identity information of a user who added the additional information and system time information.

7. The 3D system of claim 6 , wherein the at least one processor is further configured to:

query the additional information according to a corresponding operation on the display screen.

8. The 3D system of claim 6 , wherein the at least one processor is further configured to:

modify the additional information according to a corresponding operation on the display screen.

9. The 3D system of claim 1 , wherein:

the at least one memory comprises a first memory and a second memory;

the at least one processor comprises a first processor and a second processor;

the first memory and the first processor belong to a server;

the second memory and the second processor belong to a mobile terminal;

the server is communicatively coupled to the mobile terminal.

10. A method for performing three-dimensional scene engineering simulation and real-life scene fusion, the method comprising:

obtaining three-dimensional coordinates of each point of a three-dimensional model of a real-life scene;

obtaining engineering data corresponding to the three-dimensional coordinates of each point analyzed according to the three-dimensional model;

acquiring location information of at least one point of the real-life scene;

establishing a correspondence between the three-dimensional coordinates of each point of the three-dimensional model and the location information of the at least one point of the real-life scene according to the acquired location information of the at least one point of the real-life scene and the three-dimensional coordinates corresponding to the at least one point;

providing a scene fusion interactive interface on a display screen of a mobile terminal;

acquiring an image of a target scene in the real-life scene in real time in response to an operation of capturing an image on the display screen;

acquiring location information and orientation information of the mobile terminal;

determining three-dimensional coordinates corresponding to the location information of the mobile terminal in the three-dimensional model according to the correspondence relationship and the location information of the mobile terminal;

determining a three-dimensional coordinate range corresponding to the acquired image according to the three-dimensional model, the determined three-dimensional coordinates, the orientation information, and a size of the display screen; and

merging the engineering data corresponding to the three-dimensional coordinate range into the image of the target scene according to an operation of merging the scene on the display screen to form a realistic scene engineering simulation image.

11. The method of claim 10 , wherein:

the location information comprises a longitude and a latitude; and

the longitude and latitude are obtained from a Global Positioning System (GPS).

12. The method of claim 10 , wherein:

the scene fusion interaction interface provides a plurality of fusion options; and each fusion option represents different engineering parameters to be fused.

13. The method of claim 10 , further comprising:

removing an object selected in the image of the target scene and fill in an empty space of the removed object according to a peripheral area around the removed object.

14. The method of claim 10 , further comprising:

saving the real-life scene simulation image in real time.

15. The method of claim 10 , further comprising:

adding additional information onto the real-life scene simulation image according to an operation of adding information on the display screen; and

automatically adding identity information of a user who added the additional information and system time information.

16. The method of claim 10 , further comprising:

querying the additional information according to a corresponding operation on the display screen.

17. The method of claim 10 , further comprising:

modifying the additional information according to a corresponding operation on the display screen.

Assignments (2)
CHANGE OF NAME Recorded Mar 10, 2022
From: HONGFUJIN PRECISION ELECTRONICS(TIANJIN)CO.,LTD.
To: FULIAN PRECISION ELECTRONICS (TIANJIN) CO., LTD.
Reel/Frame 059620/0142 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 10, 2019
From: WANG, SHIH-CHENG
To: HONGFUJIN PRECISION ELECTRONICS(TIANJIN)CO.,LTD.
Reel/Frame 050683/0692 →
Priority Claims (1)
CN 2019 1 0702999 · Jul 31, 2019 · national