Granted Patent B2

US 12,229,333 · App. 18/499,770 · Granted Feb 18, 2025

Inventors: Daniel Rinck (Forchheim, DE); Aniol Serra Juhe (Nuremberg, DE)

Assignee: SIEMENS HEALTHINEERS AG

G06F3/011G06F3/0304

Abstract

Claims (55)

1. A computer-implemented method for visualizing interactions in an extended reality (XR) scene, the computer-implemented method comprising:

receiving a first dataset, the first dataset representing the XR scene including at least a technical device;

displaying the XR scene on an XR headset or a head-mounted display (HMD);

providing a room for a user, wherein the user wears the XR headset or HMD to interact with the XR scene, wherein the XR scene is displayed on the XR headset or HMD, wherein the room includes a set of optical sensors, and wherein the set of optical sensors includes at least one optical sensor at a fixed location relative to the room;

detecting, via the set of optical sensors, optical sensor data of the user as a second dataset while the user is interacting in the room with the XR scene, wherein the XR scene is displayed on the XR headset or HMD; and

fusing the first dataset and the second dataset to generate a third dataset, wherein

the first dataset, the second dataset and the third dataset include a point cloud, and

the third dataset includes a fusion of point clouds of the first dataset and the second dataset.

2. The computer-implemented method according to claim 1 , wherein the set of optical sensors includes at least one depth camera to provide point cloud data.

3. The computer-implemented method according to claim 1 , further comprising:

generating real-time instructions based on the third dataset; and

providing the real-time instructions to the user.

4. The computer-implemented method according to claim 1 , wherein the computer-implemented method is used for at least one of product development of the technical device or for controlling the technical device.

5. A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed by a computing device, cause the computing device to perform the computer-implemented method according to claim 1 .

6. The computer-implemented method according to claim 2 , wherein the at least one depth camera includes an RGBD camera.

7. The computer-implemented method according to claim 1 , wherein a trained neural network is used to provide output data based on input data, wherein the input data includes the third dataset and the output data represents a semantic context of the optical sensor data of the user.

8. A computer-implemented method for visualizing interactions in an extended reality (XR) scene, the computer-implemented method comprising:

receiving a first dataset, the first dataset representing the XR scene including at least a technical device;

displaying the XR scene on an XR headset or a head-mounted display (HMD);

providing a room for a user, wherein the user wears the XR headset or HMD to interact with the XR scene, wherein the XR scene is displayed on the XR headset or HMD, wherein the room includes a set of optical sensors, and wherein the set of optical sensors includes at least one optical sensor at a fixed location relative to the room;

detecting, via the set of optical sensors, optical sensor data of the user as a second dataset while the user is interacting in the room with the XR scene, wherein the XR scene is displayed on the XR headset or HMD; and

fusing the first dataset and the second dataset to generate a third dataset, wherein

a trained neural network is used to provide output data based on input data, wherein the input data includes the third dataset and the output data represents a semantic context of the optical sensor data of the user.

9. The computer-implemented method according to claim 8 , wherein the trained neural network is trained by providing the input data including the third dataset, which is labeled with content data, wherein the content data represents a semantic context of user interaction.

10. The computer-implemented method according to claim 8 , further comprising:

pre-processing the third dataset before being used as the input data for the trained neural network, wherein the pre-processing includes application of an image-to-image transfer learning algorithm to the third dataset.

11. The computer-implemented method according to claim 8 , further comprising:

pre-processing the first dataset before fusion with the second dataset, wherein the pre-processing includes application of an image-to-image transfer learning algorithm to the first dataset, wherein the generating of the third dataset includes fusing the pre-processed first dataset and the second dataset.

12. The computer-implemented method according to claim 8 , wherein the trained neural network is further configured to receive, as input data, detected optical sensor data of a user interacting with a real-world scene, wherein the real-world scene includes a real-world technical device, wherein the real-world technical device corresponds to the technical device of the XR scene.

13. The computer-implemented method according to claim 11 , wherein the fusing includes applying a calibration algorithm, which utilizes at least one registration object deployed in the room.

14. The computer-implemented method according to claim 13 , wherein the calibration algorithm uses a set of registration objects, which are provided as real, physical objects in the room and which are provided as displayed virtual objects in the XR scene, wherein for registration purposes, the real, physical objects are moved to match the displayed virtual objects in the XR scene.

15. The computer-implemented method according to claim 14 , wherein the real, physical objects include a first set of spheres, and wherein the displayed virtual objects include a second set of spheres.

16. A computing device for visualizing interactions in an extended reality (XR) scene, the computing device comprising:

a first input interface configured to receive a first dataset, the first dataset representing the XR scene including at least a technical device;

a second input interface configured to receive, from a set of optical sensors, detected optical sensor data of a user as a second dataset while the user is interacting in a room with the XR scene, wherein the XR scene is displayed to the user on a XR headset or head-mounted display (HMD), and wherein the set of optical sensors includes at least one optical sensor at a fixed location relative to the room; and

at least one processor configured to fuse the first dataset and the second dataset to generate a third dataset, wherein

the first dataset, the second dataset and the third dataset include a point cloud, and

the third dataset includes a fusion of point clouds of the first dataset and the second dataset.

17. A system for visualizing interactions in an extended reality (XR) scene, the system comprising:

at least one of at least one XR headset or at least one head-mounted display (HMD);

an XR scene generating device configured to generate a first dataset, the first data set representing the XR scene including at least a technical device, wherein the XR scene is to be displayed on the at least one of the at least one XR headset or the at least one HMD;

a set of optical sensors configured to detect optical sensor data of a user as a second dataset, while the user is interacting in a room with the XR scene, the XR scene being displayable on the at least one of the at least one XR headset or the at least one HMD, and wherein the set of optical sensors includes at least one optical sensor at a fixed location relative to the room; and

a computing device configured to execute a fusing algorithm to fuse the first dataset with the second dataset to generate a third dataset, the computing device including

a first input interface configured to receive the first dataset,

a second input interface configured to receive, from the set of optical sensors, the optical sensor data of the user as the second dataset, and

at least one processor configured to execute the fuse algorithm to fuse the first dataset with the second dataset to generate the third dataset, wherein

at least one of (i) the at least one XR headset or the at least one HMD, (ii) the computing device, or (ii) the XR scene generating device are connected via a photon unity network (PUN).

18. The system according to claim 17 ,

wherein the first dataset, the second dataset and the third dataset include a point cloud, and

wherein the third dataset includes a fusion of point clouds of the first dataset and the second dataset.

19. A computing device for visualizing interactions in an extended reality (XR) scene, the computing device comprising:

a first input interface configured to receive a first dataset, the first dataset representing the XR scene including at least a technical device;

a second input interface configured to receive, from a set of optical sensors, detected optical sensor data of a user as a second dataset while the user is interacting in a room with the XR scene, wherein the XR scene is displayed to the user on a XR headset or head-mounted display (HMD), and wherein the set of optical sensors includes at least one optical sensor at a fixed location relative to the room; and

at least one processor configured to fuse the first dataset and the second dataset to generate a third dataset, wherein

a trained neural network is used to provide output data based on input data, wherein the input data includes the third dataset and the output data represents a semantic context of the detected optical sensor data of the user.

Assignments (4)

Priority Claims (1)

Continuity (1)

References Cited (29)