Augmented reality display device with deep learning sensors

1. A system comprising:

a plurality of sensors for capturing different types of sensor data;

non-transitory memory configured to store executable instructions, and a neural network which receives the sensor data as input, wherein the neural network comprises:

an input layer for receiving input of the neural network,

a plurality of intermediate layers including a first intermediate layer of the plurality of intermediate layers coupled to the input layer, and

a plurality of head components including a head output node coupled to a last intermediate layer of the plurality of intermediate layers through a plurality of head component layers;

a hardware processor in communication with the plurality of sensors and the non-transitory memory, wherein the hardware processor is programmed by the executable instructions to:

receive the different types of sensor data from the plurality of sensors; and

determine output associated with the neural network.

2. The system of claim 1 , wherein the plurality of head components output results of the neural network, and wherein the outputs are associated with face recognition.

3. The system of claim 2 , wherein the outputs are further associated with lighting detection.

4. The system of claim 1 , wherein the hardware processor is further programmed to cause display of the determined output.

5. The system of claim 1 , wherein the sensor data is associated with a plurality of different types of events, and wherein the output indicates the different types of events which comprise face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization, or any combination thereof.

6. The system of claim 1 , wherein the sensors include one or more of: an inertial measurement unit, a depth sensing camera, a microphone, or an eye imaging camera.

7. The system of claim 1 , wherein the intermediate layers include a plurality of lower layers which are trained to extract lower level features and middle layers which are trained to extract higher level features.

8. The system of claim 7 , wherein the head components use respective subsets of the higher level features.

9. The system of claim 7 , wherein a number of weights associated with the plurality of lower layers is more than half of weights associated with the neural network.

10. The system of claim 7 , wherein a computation associated with the plurality of lower layers is more than half of a total computation associated with the neural network.

11. A method implemented by a display system of one or more processors, the display system comprising a plurality of sensors configured to capture different types of sensor data, and wherein the method comprises:

receiving sensor data from the plurality of sensors;

determining, based on the sensor data, output associated with a neural network, wherein the neural network comprises:

an input layer for receiving input of the neural network,

a plurality of intermediate layers including a first intermediate layer of the plurality of intermediate layers coupled to the input layer, and

a plurality of head components including a head output node coupled to a last intermediate layer of the plurality of intermediate layers through a plurality of head component layers; and

causing display, via the display system, of at least a portion of the determined output.

12. The method of claim 11 , wherein the plurality of head components output results of the neural network, and wherein the outputs are associated with face recognition.

13. The method of claim 12 , wherein the outputs are further associated with lighting detection.

14. The method of claim 11 , wherein the intermediate layers include a plurality of lower layers which are trained to extract lower level features and middle layers which are trained to extract higher level features.

15. The method of claim 14 , wherein a number of weights associated with the plurality of lower layers is more than half of weights associated with the neural network.

16. The method of claim 14 , wherein a computation associated with the plurality of lower layers is more than half of a total computation associated with the neural network.

17. Non-transitory computer storage media storing instructions for execution by a display system of one or more processors, the display system comprising a plurality of sensors configured to capture different types of sensor data, and wherein the instructions cause the one or more processors to perform operations comprising:

receiving sensor data from the plurality of sensors;

determining, based on the sensor data, output associated with a neural network, wherein the neural network comprises:

an input layer for receiving input of the neural network,

a plurality of intermediate layers including a first intermediate layer of the plurality of intermediate layers coupled to the input layer, and

a plurality of head components including a head output node coupled to a last intermediate layer of the plurality of intermediate layers through a plurality of head component layers; and

causing display, via the display system, of at least a portion of the determined output.

18. The computer storage media of claim 17 , wherein the plurality of head components output results of the neural network, and wherein the outputs are associated with face recognition and lighting detection.

19. The computer storage media of claim 17 , wherein the intermediate layers include a plurality of lower layers which are trained to extract lower level features and middle layers which are trained to extract higher level features.

20. The computer storage media of claim 19 , wherein a number of weights associated with the plurality of lower layers is more than half of weights associated with the neural network, or wherein a computation associated with the plurality of lower layers is more than half of a total computation associated with the neural network.