Granted Patent B2

US 12,230,041 · App. 17/775,937 · Granted Feb 18, 2025

Inventors: Jie Bai (Beijing, CN); Min Xiao (Beijing, CN); Yi Zhu (Beijing, CN); Yang Li (Beijing, CN)

Assignee: Realsee (Beijing) Technology Co., Ltd.

G06V20/64G06T7/75G06V10/774

Abstract

Claims (54)

1. An information playback method, comprising:

performing identification on a spatial image in a 3D model, thereby identifying an information display device and a display area of the information display device in the spatial image;

determining display position information corresponding to the display area; and

overlaying an information playback layer in the display area based on the display position information so as to play back display information in the information playback layer;

wherein playing back the display information in the information playback layer comprises:

obtaining current virtual user visual field information, wherein the visual field information comprises current position information of a virtual user and visual angle range information of the virtual user;

judging whether the information display device is within a visual field range of the virtual user; and

loading the display information on the information playback layer and playing back automatically, or playing back in response to a playback command of a user if the information display device is located within the visual field range of the virtual user; and

wherein judging whether the information display device is within the visual field range of the virtual user comprises:

obtaining spatial coordinates of end points of the information display device in the 3D model; and

determining that the information display device is within the visual field range of the virtual user when the number of the spatial coordinates of the end points falling within the visual field of the virtual user is greater than a preset threshold.

2. The method according to claim 1 , wherein performing the identification on the spatial image in the 3D model and thereby identifying the information display device and the display area in the spatial image comprise:

inputting the 3D model into an image identification model, identifying the information display device and the display area of the display device in the spatial image by using the image identification model, and determining positions of the information display device and the display area in the 3D model.

3. The method according to claim 2 , further comprising:

generating a training sample based on a 3D model sample in which 3D space information of the information display device is calibrated, wherein the display area of the information display device is calibrated in the 3D space information of the information display device; and

training a preset deep learning model by using a deep learning method and based on the training sample to obtain the image identification model.

4. The method according to claim 1 , wherein determining the display position information corresponding to the display area comprises:

obtaining 3D point cloud information corresponding to the 3D model; and

determining the display position information based on the 3D point cloud information and a position of the display area in the 3D model; wherein

the display position information comprises spatial coordinates of end points of the display area in the 3D model.

5. The method according to claim 1 , further comprising:

performing a corresponding interactive operation on the display information played back in the information playback layer in response to a playback control command of a user.

6. The method according to claim 1 , further comprising:

respectively controlling the display information played back in the information playback layer of each display area in a case that the identified display area comprises the plurality of display areas, wherein

optionally, respectively controlling the display information played back in the information playback layer of each display area comprises: controlling to play back different display information in the information playback layer of each display area.

7. The method according to claim 1 , further comprising:

determining target display areas corresponding to a plurality of 3D models and needing to play back the display information in a case that a user browses the plurality of 3D models, and respectively controlling display information played back in an information playback layer of each target display area, wherein

optionally, respectively controlling the display information played back in the information playback layer of each target display area comprises: controlling to play back different display information in the information playback layer of each target display area.

8. The method according to claim 1 , wherein

the display information comprises at least one of the following items: a static image, streaming media information, or a human-computer interaction interface.

9. The method according to claim 1 , wherein the display position information comprises spatial coordinates of end points of the display area in the 3D model, and the method further comprises:

dividing the display area determined based on the spatial coordinates of the end points into a plurality of sub display areas;

dividing the display information for being played back in the display area into a plurality of sub display information in one-to-one correspondence to the plurality of sub display areas on a display position; and

controlling to display the corresponding sub display information in each sub display area.

10. An electronic device, comprising:

a processor; and

a memory, configured to store executable instructions of the processor;

wherein the processor is configured to read the executable instructions from the memory, and execute the instructions so as to implement the following operations:

perform identification on a spatial image in a 3D model and thereby identify an information display device and a display area of the information display device in the spatial image;

determine display position information corresponding to the display area; and

overlay an information playback layer in the display area based on the display position information so as to play back display information in the information playback layer;

wherein the processor is configured to obtain current virtual user visual field information, wherein the visual field information comprises current position information of a virtual user and visual angle range information of the virtual user; judge whether the information display device is within a visual field range of the virtual user; and load the display information on the information playback layer and play back automatically, or play back in response to a playback command of a user if the information display device is located within the visual field range of the virtual user; and

wherein the processor is further configured to obtain spatial coordinates of end points of the information display device in the 3D model; and determine that the information display device is within the visual field range of the virtual user when the number of the spatial coordinates of the end points falling within a visual field of the virtual user is greater than a preset threshold.

11. The electronic device according to claim 10 , wherein

the processor is configured to input the 3D model into an image identification model, identify the information display device and the display area of the information display device in the spatial image by using the image identification model, and determine positions of the information display device and the display area in the 3D model.

12. The electronic device according to claim 11 , wherein

the processor is configured to generate a training sample based on a 3D model sample in which 3D space information of the information display device is calibrated, wherein the display area of the information display device is calibrated in the 3D space information of the information display device; and train a preset deep learning model by using a deep learning method and based on the training sample to obtain the image identification model.

13. The electronic device according to claim 10 , wherein

the display position determining module is configured to obtain 3D point cloud information corresponding to the 3D model; and determine the display position information based on the 3D point cloud information and a position of the display area in the 3D model, wherein the display position information comprises: spatial coordinates of end points of the display area in the 3D model.

14. The electronic device according to claim 10 , wherein the processor is further configured to perform a corresponding interactive operation on the display information played back in the information playback layer in response to a playback control command of a user.

15. The electronic device according to claim 10 , wherein the processor is configured to respectively control the display information played back in the information playback layer of each display area in a case that the identified display area comprises the plurality of display areas, wherein

optionally, respectively controlling the display information played back in the information playback layer of each display area comprises: controlling to play back different display information in the information playback layer of each display area.

16. The electronic device according to claim 10 , wherein the processor is configured to determine target display areas corresponding to a plurality of 3D models and needing to play back the display information in a case that a user browses the plurality of 3D models, and respectively control display information played back in an information playback layer of each target display area, wherein

optionally, respectively controlling the display information played back in the information playback layer of each target display area comprises: controlling to play back different display information in the information playback layer of each target display area.

Assignments (1)

Priority Claims (2)

Continuity (1)

References Cited (42)