Method and system of displaying information during a medical procedure

1. A method for displaying information to a user during a medical procedure comprising the steps of:

a. determining a field of view of the user through a head mounted display device worn by the user using one or more sensors for determining the field of view, the one or more sensors for determining the field of view include active sensors for sensing head orientation of the user, head location of the user, eye or eyes location of the user, or the eye or eyes orientation of the user;

b. retrieving object information on one or more objects within the medical procedure within the field of view from the one or more sensors, the one or more objects within the medical procedure include one or more medical tools used in real time in the medical procedure and living bodies and/or living organs of the medical procedure, checking if the properties of the one or more object changed, the properties being selected from location, shape, size and orientation;

c. retrieving by a processor supplemental object information from data storage, said supplemental object information includes one or more 3D generic models of the one or more objects from step b. in the field of view;

d. integrating by a processor said retrieved object information on one or more objects within the field of view from the one or more sensors from step b. with the retrieved supplemental object information from step c. in real time to obtain models of the objects that can be displayed as a graphical representation with the same shape, size and orientation as through the said objects were visible to the user; and

e. displaying by a display processor the models of objects of step d. in 3D on one or more display elements on the head mounted display device in real time, the objects being in the field of view, and

repeating steps a. through e. for each movement of the user which changes said field of view of the user and every time said one or more objects changes a property selected from one or more of a location, orientation or shape,

wherein said head mounted display is transparent or semi-transparent to allow the integrated information to be overlaid on the field of view to allow the user to continue to view objects of the one or more medical tools used in real time in the medical procedure and the living bodies and/or living organs in the field of view during the medical procedure.

2. The method of claim 1 further comprising the step of:

calculating relationships between objects using the integrated information obtained in step d.

3. The method of claim 2 wherein the relationship is a distance between the one or more objects of the object information and the one or more objects of the supplemental information.

4. The method of claim 1 wherein said object information is selected from one or more of the group of a shape, size, location and orientation from the one or more sensors.

5. A system for displaying information to a user comprising:

a head mounted display device including

one or more sensors for determining a field of view of the user through the head mounted display device the one or more sensors including active sensors for sensing head orientation of the user, head location of the user, eye or eyes location of the user, or the eye or eyes orientation of the user;

one or more sensors for retrieving object information on one or more objects within the medical procedure within the field of view, the one or more objects within the medical procedure include one or more medical tools used in real time in the medical procedure and living bodies and/or living organs of the medical procedure, checking if the properties of the one or more object changed, the properties being selected from location, shape, size and orientation;

a processor, the processor retrieving supplemental object information from data storage, said supplemental information includes one or more 3D generic models of one or more objects from a medical procedure, the supplemental information includes information about the one or more objects of the medical procedure that are in the field of view, the one or more objects from the medical procedure include one or more medical tools used in the medical procedure in real time and living bodies and/or living organs of the medical procedure;

a processor, the processor integrating said retrieved object information on one or more objects within the field of view from the one or more sensors with the retrieved supplemental object information in real time and obtaining models of the said objects that can be displayed as a graphical representation with the same shape, size and orientation as though the said objects were visible to the user; and

a display processor, the display processor displaying in 3D the integrated information on one or more display elements on the head mounted display device, said head mounted display is transparent or semi-transparent to allow the integrated information to be overlaid on the field of view to allow the user to continue to view objects of the one or more medical tools used in real time in the medical procedure and the living bodies and/or living organs in the field of view during the medical procedure.

6. The system of claim 5 wherein said supplemental information includes information on shape, size, location or orientation from the one or more generic models of one or more objects.

7. The system of claim 5 wherein said object information includes information on a shape, size, location or orientation from the one or more sensors.

8. The system of claim 5 wherein the processor calculates relationships between said objects using the integrated information.

9. The system of claim 5 wherein the one or more of head orientation sensors and one or more head location sensors are one or more face forward cameras.

10. The system of claim 5 wherein the eye or eyes location sensors and/or eye or eyes orientation sensors is one or more eye facing cameras.