Augmented reality display device with deep learning sensors
A head-mounted augmented reality (AR) device can include a hardware processor programmed to receive different types of sensor data from a plurality of sensors (e.g., an inertial measurement unit, an outward-facing camera, a depth sensing camera, an eye imaging camera, or a microphone); and determining an event of a plurality of events using the different types of sensor data and a hydra neural network (e.g., face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization).
1. A system for training a neural network for determining a plurality of different types of events, the system comprising:
non-transitory computer-readable memory storing executable instructions; and
one or more processors programmed by the executable instructions to at least:
receive different types of sensor data,
wherein the sensor data is associated with a plurality of different types of events, and
wherein the different types of sensor data comprise inertial measurement unit data, image data, depth sensor data, sound data, voice data, or any combination thereof;
generate a training set comprising the different types of sensor data as input data and the plurality of different types of events as corresponding target output data; and
train a neural network, for determining a plurality of different types of events, using the training set,
wherein the neural network comprises an input layer for receiving input of the neural network, a plurality of intermediate layers, and a plurality of head components for outputting results of the neural network,
wherein the input layer is connected to a first layer of the plurality intermediate layers,
wherein a head component of the plurality of head components comprises a head output node, and
wherein the head output node is connected to a last layer of the intermediate layers through a plurality of head component layers.
2. The system of claim 1 , wherein the plurality of different types of events comprises face recognition, visual search, gesture identification, semantic segmentation, object detection, lighting detection, simultaneous localization and mapping, relocalization, or any combination thereof.
3. The system of claim 1 , wherein the plurality of intermediate layers comprises a plurality of lower layers and a plurality of middle layers,
wherein the plurality of lower layers is trained to extract lower level features from the different types of sensor data, and
wherein the plurality of middle layers is trained to extract higher level features from the lower level features extracted.
4. The system of claim 1 , wherein the plurality of intermediate layers or the plurality of head component layers comprises a convolution layer, a brightness normalization layer, a batch normalization layer, a rectified linear layer, an upsampling layer, a pooling layer, a concatenation layer, a fully connected layer, a linear fully connected layer, a softsign layer, a recurrent layer, or any combination thereof.
5. The system of claim 1 , wherein the one or more processors is further programmed by the executable instructions to at least:
receive a second different type of sensor data,
wherein the second different type of sensor data is associated with a second different type of events, and
wherein the second different type of sensor data comprises inertial measurement unit data, image data, depth sensor data, sound data, voice data, or any combination thereof;
generate a retraining set comprising the second different type of sensor data as input data and the second different type of events as corresponding target output data; and
retrain the neural network, for determining the second different type of events, using the retraining set,
wherein a second head component of the plurality of head components comprises a second head output node for outputting results associated with the second different type of events, and
wherein the head output node is connected to the last layer of the intermediate layers through a plurality of second head component layers.
6. The system of claim 5 , wherein to retrain the neural network, the one or more processors are programmed by the executable instructions to at least: update weights associated with the plurality of second head component layers.
7. The system of claim 5 , wherein the neural network is retrained without updating weights associated with the plurality of intermediate layers.
8. The system of claim 1 ,
wherein the plurality of different types of sensor data is associated with a second different type of events, and
wherein the one or more processors is further programmed by the executable instructions to at least:
generate a retraining set comprising the different types of sensor data as input data and the second different type of events as corresponding target output data; and
retrain the neural network, for determining the second different type of events, using the retraining set.