Extended reality-based user interface add-on, system and method for reviewing 3D or 4D medical image data

1 . A system for reviewing three-dimensional (3D), or four-dimensional (4D) medical image data, having a processing unit, a data storage and devices to allow user input and output, and having an extended reality environment, the system comprising:

a medical review application comprising a processing module configured to process a 3D or 4D dataset to generate 3D content, and a 2D user interface, wherein the 2D user interface is configured to display the 3D content generated by the processing module and to allow a user to generate user input commands;

an extended reality(XR)-based user interface add-on; and

a data exchange channel operatively coupled to the processing module, the data exchange channel being configured to interface the processing module with the extended reality, i.e. XR-based user interface add-on;

wherein the data exchange channel is adapted to direct the 3D content generated by the processing module to the extended reality-based user interface add-on;

wherein the extended reality-based user interface add-on is configured to interpret and process the 3D content and convert it to XR content displayable to the user in an extended reality environment;

wherein the extended reality environment is configured to allow a user to generate user input events, and the extended reality-based user interface add-on is configured to process the user input events and convert them to user input commands readable by the medical review application;

wherein the data exchange channel is adapted to direct user input commands from the extended reality-based user interface add-on to the medical review application; and

wherein the processing module processes the 3D content based on the user input commands to generate modified 3D content; and

wherein, responsive to a switch user interface command generated by a currently-active user interface among the extended reality-based user interface add-on and the 2D user interface, the processing module is adapted to stop responding to the user input commands from said user interface and to start responding to user input commands from the other user interface.

2 . The system according to claim 1 , wherein the medical review application comprises a data interface for the extended reality-based user interface add-on, wherein the data interface is operatively coupled to the data exchange channel, and is configured to allow the exchange of simplified and standardized operating actions and data between the medical review application and the extended reality-based user interface add-on.

3 . The system according to claim 1 , wherein the medical review application comprises a data interface for the XR-based user interface add-on, wherein the data interface is adapted to continuously synchronize corresponding user interface elements between the extended reality-based user interface add-on and the medical review application through the data exchange channel, wherein corresponding user interface elements comprise at least one of a value of a user interface element, an identifier of a selected frame of the 4D dataset, settings concerning the display of the 3D and/or XR content, and/or a 3D mouse position.

4 . The system according to claim 1 , wherein the XR-based user interface add-on is stateless, in that it does not have a memory of user input commands transferred to the processing module through the data exchange channel.

5 . The system according to claim 1 , wherein the 3D content generated by the processing module comprises a rendering of the 3D or 4D dataset, wherein the extended reality-based user interface add-on is configured to distort in perspective the rendered 3D content based on at least some of the user input and/or a user's current viewing perspective.

6 . The system according to claim 1 , wherein the 3D content generated by the processing module comprises at least one of an updated or data-compressed version of the 3D or 4D dataset, a rendering of the 3D or 4D dataset, a multi planar reconstruction, i.e. MPR, texture generated from the 3D or 4D dataset, a graphical primitive, a 3D or 4D model of an object, a mesh, a text of an annotation, and/or a number indicating a measurement.

7 . The system according to claim 1 , wherein the processing module is adapted to transfer a temporal and spatial reference system and at least one of the 3D or 4D dataset, an updated 3D or 4D dataset, a user interface element and/or configuration settings to the extended reality-based user interface add-on during establishment of a connection between the processing module and the extended reality-based user interface add-on.

8 . The system according to claim 1 , wherein the extended reality-based user interface add-on is adapted to transfer at least one of a 3D mouse position, a position and orientation of a multi planar reconstruction, i.e. MPR, plane, a screenshot, and/or a modified value of a user interface element, through the data exchange channel to the processing module during a connection between the processing module and the extended reality-based user interface add-on.

9 . The system of claim 1 , wherein the extended reality-based user interface add-on is configured to be used with XR hardware via an XR operating system, wherein the XR hardware comprises an XR headset and XR controllers.

10 . A tangible, non-transitorty computer readable medium adapted to store an extended reality-based user interface add-on configured to be operatively coupled via a data exchange channel to a medical review application having a 2D user interface and a processing module configured to process a 3D or 4D dataset to generate 3D content,

wherein the extended reality-based user interface add-on is configured to interpret the 3D content received from the processing module via the data exchange channel and convert it into XR content in a data format readable by an XR operating system of an XR hardware, wherein the XR hardware comprises an XR headset and XR controllers, such that the XR hardware can display the XR content generated by the processing module;

wherein the extended reality-based user interface add-on is configured to process any user input events generated using the XR controllers, and to convert the user input events into user input commands readable by the medical review application, and to transfer the user input commands via the data exchange channel to the medical review application;

wherein the processing module processes the 3D content based on the user input commands to generate modified 3D content; and

wherein, responsive to a switch user interface command generated by a currently-active user interface among the extended reality-based user interface add-on and the 2D user interface, the processing module is adapted to stop responding to the user input commands from said user interface and to start responding to user input commands from the other user interface.

11 . A method for analysing a 3D or 4D dataset using a medical review application comprising a processing module and a 2D user interface, a data exchange channel operatively coupled to the processing module, wherein the data exchange channel is configured to interface the processing module with an extended reality-based user interface add-on,

the method comprising:

processing the 3D dataset to generate 3D content on the processing module;

optionally, the 2D user interface displaying the 3D content;

the data exchange channel directing the 3D content to the extended reality-based user interface add-on and the extended reality-based user interface add-on interpreting and processing the 3D content and converting it to XR content displayable to a user by XR hardware;

receiving user input on one of the user interfaces;

directing user input commands to the processing module directly from the 2D user interface or via the data exchange channel from the extended reality-based user interface add-on;

the processing module processing the 3D content based on the user input commands to generate modified 3D content,

directing the modified 3D content to the data exchange channel and optionally to the 2D user interface;

optionally, the 2D user interface displaying the modified 3D content; and

the data exchange channel further directing the modified 3D content to the extended reality-based user interface add-on and the extended reality-based user interface add-on interpreting and processing the modified 3D content and converting it into modified XR content displayable to a user by XR hardware, wherein, responsive to a switch user interface command generated by a currently-active user interface among the extended reality-based user interface add-on and the 2D user interface, stop responding to the user input commands from said user interface and to start responding to user input commands from the other user interface.

12 . The method according to claim 11 , the method comprising:

processing the 3D dataset to generate a rendering of the 3D dataset to provide a 3D model of an anatomical structure depicted by said 3D dataset;

displaying the 3D model and the rendering via the extended reality-based user interface add-on;

allowing a user to check the 3D model on the extended reality-based user interface add-on and to provide user input to adjust the 3D model;

the data exchange channel directing the user input commands from the extended reality-based user interface add-on to the processing module;

the processing module processing the user input commands to generate a modified 3D model;

directing the modified 3D model to the 2D user interface, and the 2D user interface displaying the modified 3D model; and

optionally allowing a user to perform additional analysis and/or measurements on the modified 3D model in the 2D user interface.

13 . The method according to claim 11 , wherein the processing module is configured to process the 3D or 4D dataset and generate 3D content and/or XR content by at least one of data-compressing the 3D or 4D dataset, rendering the 3D dataset, volume rendering the 3D dataset, calculating a multi planar reconstruction, i.e. MPR, texture of an MPR plane through the 3D dataset, segmenting the 3D or 4D dataset, generating a 3D or 4D model of an object, generating a graphical primitive, and/or to taking a measurement responsive to user input.

14 . A tangible, non-transitory computer readable medium that stores program code instructions which, when executed by a processing unit, cause the processing unit to:

process a 3D dataset to generate 3D content;

optionally, cause a 2D user interface to display the 3D content;

receive user input commands directly from the 2D user interface or via a data exchange channel from an extended reality-based user interface add-on, wherein the data exchange channel is configured to direct the 3D content to the extended reality-based user interface add-on, and wherein the extended reality-based user interface add-on is configured to interpret and process the 3D content and convert it to XR content displayable to a user by XR hardware;

process the 3D content based on the user input commands to generate modified 3D content;

direct the modified 3D content to the data exchange channel and optionally to the 2D user interface;

optionally, cause the 2D user interface to display the modified 3D content;

cause the data exchange channel to further direct the modified 3D content to the extended reality-based user interface add-on;

cause the extended reality-based user interface add-on to interpret and process the modified 3D content and to convert it into modified XR content displayable to a user by XR hardware;

based on the user input commands to generate modified 3D content; and

responsive to a switch user interface command generated by a currently-active user interface among the extended reality-based user interface add-on and the 2D user interface, stop responding to the user input commands from said user interface and start responding to user input commands from the other user interface.

15 . The tangible, non-transitory computer readable medium according to claim 14 , wherein the medical review application comprises a data interface for the extended reality-based user interface add-on, wherein the data interface is operatively coupled to the data exchange channel, and is configured to allow the exchange of simplified and standardized operating actions, and data between the medical review application and the extended reality-based user interface add-on.

16 . The tangible, non-transitory computer readable medium according to claim 14 , wherein the medical review application comprises a data interface for the XR-based user interface add-on, wherein the data interface is adapted to continuously synchronize corresponding user interface elements between the extended reality-based user interface add-on and the medical review application through the data exchange channel, wherein corresponding user interface elements comprise at least one of a value of a user interface element, an identifier of a selected frame of the 4D dataset, settings concerning the display of the 3D and/or XR content, and/or a 3D mouse position.

17 . The tangible, non-transitory computer readable medium according to claim 14 , wherein the XR-based user interface add-on is stateless, and does not comprise a memory of user input commands transferred to the processing module through the data exchange channel.

18 . The tangible, non-transitory computer readable medium according to claim 14 , wherein the 3D content generated by the processing module comprises a rendering of the 3D or 4D dataset, wherein the extended reality-based user interface add-on is configured to distort in perspective, the rendered 3D content based on at least some of the user input and/or a user's current viewing perspective.