Mixed reality combination system
Disclosed herein is a system providing a mixed reality combination system that pairs augmented reality technology and an inertial measurement unit sensor with 3D printed objects such that user motions tracked by the inertial measurement unit as the user interacts with the 3D printed object is reflected in a virtual environment display of dynamic 3D imagery and augmented reality imagery.
1 . A mixed reality combination system comprising:
an augmented reality environment comprising one or more 3D models of target objects in a target environment;
a printed environment comprising one or more 3D printed objects;
a tracking environment for visually tracking a position and orientation of a tool within the target environment; and
a user interaction module specifying a user task and including visual aids for assisting a user in completion of the specified user task;
wherein the augmented reality environment provides dynamic augmented reality imagery aligned with and projected on the one or more 3D printed objects in a user field-of-view of the target environment in response to motion of a user as tracked by the tracking environment;
wherein the dynamic imagery shows movement of objects in the augmented reality environment in response to the user motions provided by the tracking environment; and
wherein the system predicts a future state of a target object based on the task specified by the user interaction module, manipulates the 3D model representing the target object to show the predicted state and projects imagery showing the predicted state onto the printed environment as the visual aid.
2 . The system of claim 1 wherein the augmented reality environment further comprises a virtual environment for superimposing the dynamic 3D imagery onto augmented reality imagery.
3 . The system of claim 2 wherein the virtual environment is projected on a heads-up display worn by the user.
4 . The system of claim 1 wherein the 3D printed model of an object is a bio specimen composed of collagen to mimic an anatomical structure.
5 . The system of claim 4 wherein the mixed reality combination system is used for surgical planning and practice and further wherein the 3D printed bio specimen is customized to reflect a surgical pathology.
6 . The system of claim 1 wherein the tracking environment comprises an inertial measurement unit to track movements of the user.
7 . The system of claim 6 wherein the inertial measurement unit is integrated with a hand-held tool used by the user to interact with the printed environment.
8 . The system of claim 7 wherein the tracking environment further comprises a spatial mapping component to visually track the handheld tool in a field of view of the user.
9 . The system of claim 8 wherein the printed environment comprises a bio specimen of an anatomical structure and further wherein the hand-held tool is a scalpel.
10 . The system of claim 1 further comprising an augmented reality device which places virtual environments in a physical environment.
11 . The system of claim 10 wherein the augmented reality device is a head mounted display worn by the user.
12 . The system of claim 1 wherein the dynamic 3D imagery is obtained by:
creating a series of 3D images wherein each 3D image in the series is created by stitching together multiple 2D images;
wherein each 3D image represents a state of an object as the object transitions from the prior state to the next state;
performing finite element modeling to create the dynamic 3D imagery from the series of 3D images.
13 . The system of claim 12 wherein the system provides a dynamic response when the user interacts with the virtual environment.
14 . The system of claim 13 wherein user interaction with the virtual environment is detected when the position of the user in relation to the printed environment crosses a boundary.
15 . The system of claim 13 wherein user interaction with the virtual environment is detected by using spatial mapping to relate positions of the virtual environment in relation to physical movements of the user.
16 . The system of claim 1 further comprising:
a processor; and
software, executing on the processor, the software integrating the augmented reality environment, the printed environment and the tracking environment and executing the user interaction module.
17 . The system of claim 16 wherein the processor is configured to exchange data with an inertial measurement unit as part of the tracking environment and with a heads-up display as part of the augmented reality environment.
18 . A system comprising:
a processor;
a head-mounted display, including a spatial mapping component, in communication with the processor;
one or more tools instrumented with inertial measurement units; and
software that, when executed by the processor, causes the system to:
provide dynamic 3D imagery projected on a 3D printed object in response to motion of a user tracked by the spatial mapping component and the inertial measurement units in the one or more tools.
19 . The system of claim 1 wherein finite element analysis is used to simulate multi physical responses of the 3D models to user interactions with the objects.
20 . The system of claim 19 wherein the multi physical responses are expected outcomes of user interactions with the projected augmented reality imagery.
21 . The system of claim 1 further comprising:
a user interaction platform for detecting user interactions with one of the target objects in a first state and matching dynamic responses from one or more of the 3D printed objects with a custom finite element analysis dynamic modeling outcome of a second state of the target object.
22 . The system of claim 1 wherein the system dynamically illustrates a desired user interaction with one or more target objects.
23 . The system of claim 22 wherein the illustration of the desired user interaction with one or more target objects remains displayed in the user field-of-view as the user dynamically interacts with the one or more target objects to complete the desired user interaction.
24 . The system of claim 1 wherein the one or more 3D models of target objects are created by stitching together multiple layers of 2D images.
25 . The system of claim 1 wherein the 3D model of the target object of the task is manipulated to show the predicted state using finite element analysis.