Camera based wearable devices with artificial intelligence assistants
An optical sensing system comprises a human interface structure to be applied to a user. The system comprises a time-of-flight sensor with an array of laser diodes with Bragg reflectors operating in the near-infrared, a detection system, and a processor generating a time-of-flight measurement and a first image. The system also includes a first camera system capturing a second image, and a second camera system illuminated by light emitting diodes to monitor one or more eyes of the user. In communication with or operating on the processor is an artificial intelligence assistant, which comprises a multi-modal generative artificial intelligence model. The system may be part of wearable eyewear such as smart glasses or extended reality goggles. The system may perform gesture analysis, eye tracking and speed recognition, and video and verbal outputs can be provided to the user. An artificial intelligence agent may have awareness of the environment.
1 . A remote sensing system comprising:
a housing configured to be physically applied to a user, the housing having an inward side facing the user and an outward side;
a time-of-flight sensor attached to the outward side, the time-of-flight sensor comprising:
an array of laser diodes configured to generate laser light having one or more optical wavelengths between 600 nanometers and 1000 nanometers, wherein a first laser diode of the array of laser diodes comprises one or more Bragg reflectors, and pulses at a modulation frequency between 10 Megahertz and 1 Gigahertz and has a phase associated with the pulses of the first laser diode;
at least a portion of the laser light generated by the array of laser diodes is configured to be directed to an object;
at least one photo-detector, provided with a lens and a spectral filter at an input to the at least one photo-detector,
a processor processing digitized signals received from the at least one photo-detector, the processor being configured to (i) measure a phase shift of at least a reflected portion of the laser light from the array of laser diodes reflected from the object relative to the at least a portion of the laser light generated by the array of laser diodes, (ii) measure a time-of-flight measurement of the at least the reflected portion of the laser light and (iii) generate a first image of the object based at least in part on an amplitude of the at least a reflected portion of the laser light;
wherein the photo-detector is configured to synchronize to the pulses of the first laser diode, and wherein the processor is configured to generate the time-of-flight measurement; and
a first camera attached to the outward side, the first camera coupled to a second lens and the processor, wherein the first camera is configured to capture a second image of at least a part of the object; and
the processor being coupled to a second camera attached to the inward side, the second camera being configured to be illuminated by one or more light emitting diodes generating incoherent light configured to operate in a near-infrared wavelength range, wherein the second camera is configured to at least in part monitor at least one eye of the user.
2 . The remote sensing system of claim 1 , further comprising:
a non-transitory computer readable medium coupled to the processor, wherein at least a portion of the time-of-flight measurement, at least a portion of the first image, and at least a portion of the second image are configured to serve as at least a part of an input to an artificial intelligence assistant configured to communicate with or operate on the processor, the artificial intelligence assistant comprising a multi-modal generative artificial intelligence model comprising a transformer including self-attention and positional encoding layers and configured to perform computer vision processing and natural language processing; and
wherein the multi-modal generative artificial intelligence model is configured to accept as prompt inputs comprising images, videos, text, and voice control or interactions and is configured to include memory that records prompt inputs.
3 . The remote sensing system of claim 2 , wherein the processor is configured to be at least a part of a smart glasses or an augmented, virtual, or mixed reality goggles, and wherein the glasses or goggles have a speaker, a microphone, and one or more displays, wherein the one or more displays are configured to be connected on the inward side and directed toward the at least one eye of the user.
4 . The remote sensing system of claim 3 , wherein the processor is configured to perform gesture analysis of user's bodily movements using the time-of-flight sensor or the first camera and is configured to track eye movements using the second camera.
5 . The remote sensing system of claim 4 , wherein the processor is configured to perform speech recognition and provide speech outputs configured to be sent to the speaker and video outputs configured to be sent to the one or more displays.
6 . The remote sensing system of claim 5 , wherein the multi-modal generative artificial model is configured to operate as an artificial intelligence agent that is configured to have awareness of an environment or context in which the agent interacts, and wherein the artificial intelligence assistant comprising the multi-modal generative artificial intelligence model is configured to at least in part operate on a remote computing apparatus comprising a cloud server, a smart phone, a tablet, or a computer, and the remote computing apparatus is configured to perform data fusion of an output from the processor with data from a plurality of other sensors.
7 . The remote sensing system of claim 6 , further comprising:
a light sensing system coupled to the processor, the light sensing system comprising:
one or more wavelength filters configured to separate incoming ambient light into one or more optical wavelength bands;
one or more detectors configured to capture at least some of the one or more optical wavelength bands; and
an analog-to-digital converter coupled to the one or more detectors that is configured to generate a color output; and
wherein the processor is configured to receive at least a portion of the color output and adjust a visual output from the one or more displays or a setting for the first camera.
8 . A wearable system comprising:
smart glasses configured to be worn by a user, the smart glasses comprising an outward side configured to be opposite an inward side facing the user;
one or more cameras that are CMOS back-illuminated sensors attached to the outward side of the smart glasses and configured to capture a camera image or a camera video including an object, and wherein the one or more cameras are also configured to perform hand tracking for gesture analysis;
the smart glasses comprising touch sensors configured to accept user inputs and motion tracking sensors configured to track user movements;
the smart glasses further comprising a processor, one or more displays attached to the inward side, speakers, and one or more microphones configured to receive sound inputs including voice control or interactions;
the smart glasses including the processor configured to be coupled to a smart phone or a tablet for further processing capabilities;
the processor being configured to be coupled to a non-transitory computer readable medium;
wherein smart glasses data comprises the camera image, the camera video, the sound inputs, the gesture analysis or the user inputs, and the smart glasses data is configured to serve as at least a part of an input to an artificial intelligence assistant configured to communicate with or operate on the processor, the artificial intelligence assistant comprising a multi-modal generative artificial intelligence model comprising a transformer including self-attention and positional encoding layers and configured to perform computer vision processing and natural language processing; and
wherein the multi-modal generative artificial intelligence model is configured to recognize at least some of the object captured in the camera image or the camera video.
9 . The wearable system of claim 8 , wherein the processor is configured to perform speech recognition and provide speech outputs configured to be sent to the speakers and video or image outputs configured to be sent to the one or more displays.
10 . The wearable system of claim 9 , wherein the multi-modal generative artificial intelligence model is configured to accept as prompt inputs images, videos, text, verbal inputs or sounds and configured to include memory that records prompt inputs, and wherein the multi-modal generative artificial intelligence model is configured to perform data fusion of the smart glasses data with an accelerometer input and a gyroscope sensor input.
11 . The wearable system of claim 10 , wherein the one or more cameras comprises a plurality of cameras attached to the outward side of the smart glasses, and wherein the plurality of cameras are configured for stereographic imaging.
12 . The wearable system of claim 10 , wherein the artificial intelligence assistant comprising the multi-modal generative artificial intelligence model is configured to at least in part operate on a remote computing apparatus comprising a cloud server or a computer, and wherein the multi-modal generative artificial model is configured to operate as an artificial intelligence agent that is configured to have awareness of an environment or context in which the agent interacts.
13 . The wearable system of claim 10 , wherein the processor is further configured to be coupled to a light sensing system, the light sensing system comprising:
one or more wavelength filters configured to separate incoming ambient light into one or more optical wavelength bands;
one or more detectors configured to capture at least some of the one or more optical wavelength bands; and
an analog-to-digital converter coupled to the one or more detectors that is configured to generate a color output;
wherein the processor is configured to receive at least a portion of the color output and adjust a visual output from the one or more displays or a setting for the one or more cameras.
14 . A measurement system comprising:
an array of laser diodes configured to generate a light having an initial light intensity and one or more optical wavelengths, wherein at least a portion of the one or more optical wavelengths is a near-infrared wavelength between 700 nanometers and 2500 nanometers, wherein at least a portion of the array of laser diodes comprises one or more Bragg reflectors, and wherein the array of laser diodes is further coupled to driver electronics and one or more safety shut-offs;
a beam splitter configured to receive at least a portion of the light from the array of laser diodes and to direct at least some of the light from the array of laser diodes to an object, wherein the beam splitter is further configured to separate the at least a portion of the light into a plurality of spatially separated lights;
a first camera, wherein the first camera further comprises one or more lenses and one or more spectral filters in front of the first camera, wherein the first camera is further coupled to a processor, wherein the processor is configured to be coupled to a non-transitory computer readable medium;
wherein the first camera is configured to receive at least a portion of the plurality of spatially separated lights reflected from the object and configured to capture a first image, and wherein the first camera is further configured to be synchronized to the at least a portion of the array of laser diodes comprising Bragg reflectors;
wherein the processor is configured to perform an analysis of a pattern of the received spatially separated lights reflected from the object in the first image and configured to analyze the first image; and
the processor further being coupled to a second camera, the second camera being configured to be illuminated by one or more light emitting diodes generating incoherent light and being configured to operate in a near-infrared wavelength range, wherein the second camera is configured to at least in part monitor at least one eye of a user.
15 . The measurement system of claim 14 , wherein at least a portion of the first image is configured to serve as at least a part of an input to an artificial intelligence assistant configured to communicate with or operate on the processor, the artificial intelligence assistant comprising a multi-modal generative artificial intelligence model comprising a transformer including self-attention and positional encoding layers and configured to perform computer vision processing and natural language processing; and
wherein the multi-modal generative artificial intelligence model is configured to accept as prompt inputs images, videos, text, and voice control or interactions and configured to include memory that records previous prompt inputs.
16 . The measurement system of claim 15 , wherein the measurement system including the processor is configured to perform gesture analysis of user's bodily movements using the first camera and configured to perform eye tracking using the second camera.
17 . The measurement system of claim 16 , wherein the processor is configured to be at least a part of smart glasses or augmented, virtual, or mixed reality goggles, and wherein the glasses or the goggles have a speaker, a microphone, and one or more displays configured to be directed toward the at least one eye of the user.
18 . The measurement system of claim 17 , wherein the measurement system including the processor is configured to perform speech recognition and provide speech outputs configured to be sent to the speaker and video outputs configured to be sent to the one or more displays.
19 . The measurement system of claim 18 , wherein the artificial intelligence assistant comprising the multi-modal generative artificial intelligence model is configured to at least in part operate on a remote computing apparatus comprising a cloud server, a smart phone, a tablet, or a computer;
wherein the remote computing apparatus is configured to perform data fusion of an output from the processor with data from a plurality of other sensors; and
wherein the multi-modal generative artificial model is configured to operate as an artificial intelligence agent that is configured to have awareness of an environment or context in which the agent interacts.
20 . The measurement system of claim 19 , wherein the processor is further configured to be coupled to a light sensing system comprising:
one or more wavelength filters configured to separate incoming ambient light into one or more optical wavelength bands;
one or more detectors configured to capture at least some of the one or more optical wavelength bands; and
an analog-to-digital converter coupled to the one or more detectors that is configured to generate a color output;
wherein the processor is configured to receive at least a portion of the color output and adjust a visual output from the one or more displays or a setting for the first camera.