Procedural language and content generation environment for use in augmented reality/mixed reality systems to support laboratory and related operations

1. A method of operating a procedural language and content generation system comprises:

correlating environment objects and object movement detected through an image processor to input controls through operation of a correlator;

operating an interpreter to:

evaluate an input set from a protocol development user interface against a known objects library, a protocol library, and declared objects;

evaluate correlation of the input controls, the environment objects, and the object movement against the known objects library, the protocol library, and the declared objects; and

generate programmatic instructions;

storing the programmatic instructions as an instruction set within a protocol repository;

transforming the instruction set into executable commands through operation of a compiler;

configuring control logic to perform the executable commands in response to receiving detected environment objects and detected object movement from the image processor;

operating an audio device in response to at least one executed command performed by the control logic;

operating a constructor to generate an augmented reality layer in response to the at least one executed command performed by the control logic;

overlaying the augmented reality layer over an environment image layer through operation of a combiner; and

displaying the augmented reality layer above the environment image layer in a display device.

2. The method of claim 1 further comprises:

correlating the environment image layer and the augmented reality layer of an in progress instruction set to an edit control through operation of the correlator;

operating the interpreter to:

evaluate correlation of the edit control, the environment image layer, and the augmented reality layer against the known objects library, the protocol library, the declared objects, and stored programmatic instructions for the in progress instruction set; and

generate revised programmatic instructions;

storing the revised programmatic instructions as a revised instruction set within the protocol repository;

transforming the instruction set into the executable commands through operation of the compiler;

configuring the control logic to perform the executable commands in response to receiving the detected environment objects and the detected object movement from the image processor;

operating the audio device in response to the at least one executed command performed by the control logic;

operating the constructor to generate a revised augmented reality layer in response to the at least one executed command performed by the control logic;

overlaying the revised augmented reality layer over the environment image layer through operation of the combiner; and

displaying the revised augmented reality layer above the environment image layer in the display device.

3. The method of claim 1 , wherein the image processor comprises an object recognition engine and a spatiotemporal activity engine.

4. The method of claim 3 , wherein the object recognition engine and the spatiotemporal activity engine utilizes at least one neural network for detecting and classifying the environment objects and tracking the object movement.

5. The method of claim 1 , wherein the protocol development user interface generates a flow diagram.

6. The method of claim 1 , wherein the protocol development user interface generates a three dimensional protocol workflow.

7. A computing apparatus, the computing apparatus comprising:

a processor; and

a memory storing instructions that, when executed by the processor, configure the apparatus to:

correlate environment objects and object movement detected through an image processor to input controls through operation of a correlator;

operate an interpreter to:

evaluate an input set from a protocol development user interface against a known objects library, a protocol library, and declared objects;

evaluate correlation of the input controls, the environment objects, and the object movement against the known objects library, the protocol library, and the declared objects; and

generate programmatic instructions;

store the programmatic instructions as an instruction set within a protocol repository;

transform the instruction set into executable commands through operation of a compiler;

configure control logic to perform the executable commands in response to receiving detected environment objects and detected object movement from the image processor;

operate an audio device in response to at least one executed command performed by the control logic;

operate a constructor to generate an augmented reality layer in response to the at least one executed command performed by the control logic;

overlay the augmented reality layer over an environment image layer through operation of a combiner; and

display the augmented reality layer above the environment image layer in a display device.

8. The computing apparatus of claim 7 further comprises:

correlate the environment image layer and the augmented reality layer of an in progress instruction set to an edit control through operation of the correlator;

operate the interpreter to:

evaluate correlation of the edit control, the environment image layer, and the augmented reality layer against the known objects library, the protocol library, the declared objects, and stored programmatic instructions for the in progress instruction set; and

generate revised programmatic instructions;

store the revised programmatic instructions as a revised instruction set within the protocol repository;

transform the instruction set into the executable commands through operation of the compiler;

configure the control logic to perform the executable commands in response to receiving the detected environment objects and the detected object movement from the image processor;

operate the audio device in response to the at least one executed command performed by the control logic;

operate the constructor to generate a revised augmented reality layer in response to the at least one executed command performed by the control logic;

overlay the revised augmented reality layer over the environment image layer through operation of the combiner; and

display the revised augmented reality layer above the environment image layer in the display device.

9. The computing apparatus of claim 7 , wherein the image processor comprises an object recognition engine and a spatiotemporal activity engine.

10. The computing apparatus of claim 9 , wherein the object recognition engine and the spatiotemporal activity engine utilizes at least one neural network for detecting and classify the environment objects and tracking the object movement.

11. The computing apparatus of claim 7 , wherein the protocol development user interface generates a flow diagram.

12. The computing apparatus of claim 7 , wherein the protocol development user interface generates a three dimensional protocol workflow.

13. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:

correlate environment objects and object movement detected through an image processor to input controls through operation of a correlator;

operate an interpreter to:

evaluate an input set from a protocol development user interface against a known objects library, a protocol library, and declared objects;

evaluate correlation of the input controls, the environment objects, and the object movement against the known objects library, the protocol library, and the declared objects; and

generate programmatic instructions;

store the programmatic instructions as an instruction set within a protocol repository;

transform the instruction set into executable commands through operation of a compiler;

configure control logic to perform the executable commands in response to receiving detected environment objects and detected object movement from the image processor;

operate an audio device in response to at least one executed command performed by the control logic;

operate a constructor to generate an augmented reality layer in response to the at least one executed command performed by the control logic;

overlay the augmented reality layer over an environment image layer through operation of a combiner; and

display the augmented reality layer above the environment image layer in a display device.

14. The computer-readable storage medium of claim 13 further comprises:

correlate the environment image layer and the augmented reality layer of an in progress instruction set to an edit control through operation of the correlator;

operate the interpreter to:

evaluate correlation of the edit control, the environment image layer, and the augmented reality layer against the known objects library, the protocol library, the declared objects, and stored programmatic instructions for the in progress instruction set; and

generate revised programmatic instructions;

store the revised programmatic instructions as a revised instruction set within the protocol repository;

transform the instruction set into the executable commands through operation of the compiler;

configure the control logic to perform the executable commands in response to receiving the detected environment objects and the detected object movement from the image processor;

operate the audio device in response to the at least one executed command performed by the control logic;

operate the constructor to generate a revised augmented reality layer in response to the at least one executed command performed by the control logic;

overlay the revised augmented reality layer over the environment image layer through operation of the combiner; and

display the revised augmented reality layer above the environment image layer in the display device.

15. The computer-readable storage medium of claim 13 , wherein the image processor comprises an object recognition engine and a spatiotemporal activity engine.

16. The computer-readable storage medium of claim 15 , wherein the object recognition engine and the spatiotemporal activity engine utilizes at least one neural network for detecting and classify the environment objects and tracking the object movement.

17. The computer-readable storage medium of claim 13 , wherein the protocol development user interface generates a flow diagram.

18. The computer-readable storage medium of claim 13 , wherein the protocol development user interface generates a three dimensional protocol workflow.