Multi-purpose cameras for simultaneous capture and CV on wearable AR devices
Eyewear having an image signal processor (ISP) dynamically operable in a camera pipeline for augmented reality (AR) and computer vision (CV) systems. Multi-purpose cameras are used for simultaneous image capture and CV on wearable AR devices. The cameras are coupled to a frame and configured to generate images, wherein the cameras and the ISP operate in a first AR mode and capture images having a first resolution suitable for use in AR, and are configured to operate in a second CV mode to provide the images having a second resolution suitable for use in CV. The first resolution in the AR mode is higher than the second resolution in the CV mode, and the cameras and the ISP consume less power in the second CV mode than the first AR mode. The cameras and the ISP save significant system power by operating in the low power mode CV mode.
1 . Eyewear, comprising:
a frame;
a plurality of cameras coupled to the frame and configured to generate images; and
a pipeline including an image signal processor (ISP) configurable to process the images for augmented reality (AR) and for computer vision (CV), wherein the pipeline has a first portion configured to process the images in a first mode, and a second portion configured to process the images in a second mode;
wherein each of the plurality of cameras are configured to operate in the first mode and provide the images having a first resolution operable in AR, and wherein each of the plurality of cameras are configured to operate in the second mode and provide the images having a second resolution operable in CV, wherein the first resolution is higher than the second resolution;
wherein the cameras are configured to capture the images in the first resolution, and to downscale the images to the second resolution; and
wherein the ISP is configured to stitch and synchronize the images from each of the plurality of cameras in the second mode in CV, wherein the ISP is configured to match the frames from the plurality of cameras and put them in a buffer.
2 . The eyewear of claim 1 , wherein the ISP has CPU/DDR (central processing unit/double data read) rails, wherein the ISP is configured to scale up frequencies and voltages of the CPU/DDR rails in the first mode.
3 . The eyewear of claim 1 , wherein the ISP is configured to process the images and operate at a first power in the first mode and at a second power in the second mode, wherein the second power is lower than the first power.
4 . The eyewear of claim 1 , wherein the ISP is configured to selectively turn a first of the plurality of cameras off in the second mode.
5 . The eyewear of claim 4 , wherein the ISP is configured to selectively turn the first camera back on and synchronize the plurality of cameras.
6 . A method of operating eyewear comprising a frame, a plurality of cameras coupled to the frame and configured to generate images having frames, and a pipeline including an image signal processor (ISP) configurable to process the images in augmented reality (AR) and in computer vision (CV), wherein the pipeline has a first portion configured to process the images in a first mode, and a second portion configured to process the images in a second mode, wherein:
each of the plurality of cameras operate in thea first mode and generate the images having a first resolution operable in AR, and each of the plurality of cameras operate in thea second mode and provide the images having a second resolution operable in CV, wherein the first resolution is higher than the second resolution;
the cameras capture the images in the first resolution, and downscale the images to the second resolution; and
the ISP stitches and synchronizes the images from each of the plurality of cameras in the second mode in CV, wherein the ISP matches the frames from the plurality of cameras and puts them in a buffer.
7 . The method of claim 6 , wherein the ISP has CPU/DDR (central processing unit/double data read) rails, wherein the ISP scales up frequencies and voltages of the CPU/DDR rails in the first mode.
8 . The method of claim 6 , wherein the ISP processes the images and operates at a first power in the first mode and at a second power in the second mode, wherein the second power is lower than the first power.
9 . The method of claim 6 , wherein the ISP selectively turns a first of the plurality of cameras off in the second mode.
10 . The method of claim 9 , wherein the ISP selectively turns the first camera back on and synchronizes the plurality of cameras.
11 . A non-transitory computer-readable medium storing program code which, when executed by a pipeline including an image signal processor (ISP) of eyewear having a frame, and a plurality of cameras coupled to the frame and configured to generate images having frames, is operative to cause the ISP to selectively process the images in augmented reality (AR) and in computer vision (CV), wherein the pipeline has a first portion configured to process the images in a first mode, and a second portion configured to process the images in a second mode, to perform the steps of:
each of the plurality of cameras operating in the first mode such that each of the plurality of cameras generate the images having a first resolution operable in AR, and each of the plurality of cameras operating in the second mode such that each of the plurality of cameras provide the images having a second resolution operable in CV, wherein the first resolution is higher than the second resolution;
the cameras capturing the images in the first resolution, and downscaling the images to the second resolution; and
the ISP stitching and synchronizing the images from each of the cameras in the second mode in CV, wherein the ISP matches the frames from the plurality of cameras and puts them in a buffer.
12 . The non-transitory computer-readable medium of claim 11 , wherein the program code, when executed, is operative to cause the ISP to scale up frequencies and voltages of CPU/DDR (central processing unit/double data read) rails in the first mode.