Optimal sensor reading for low latency object detection
Methods and apparatus for matching portions of images are described as well as using depth information generated from the matching results. In various embodiments, lower portions of images of a scene area, which are more likely to include objects closer to a vehicle on which cameras are mounted, than are portions of images corresponding to an upper portion of the scene area, are processed and used for depth determination. In this way, depths to objects, which are likely to be closer to the vehicle than objects in the upper portion of the scene area, are determined first and used to control a vehicle reducing the risk of collision as compared to systems where the depth determination of an entire scene is completed before depth information is used for control purposes with the order of processing being such that nearer objects are likely to be detected prior to more distant objects.
1. A method of capturing and using images, the method comprising:
capturing images of a scene area using a set of camera devices, the set of camera devices including camera devices;
reading image data out of one or more of the camera devices beginning at a bottom of a captured scene area and progressing towards the top of the captured scene area;
performing depth processing on image data corresponding to a lower portion of scene area, the depth processing determining distances to one or more objects prior to performing depth processing on image data corresponding to an upper portion of scene area; and
performing a first vehicle control operation based on at least one of the determined distances.
2. The method of claim 1 ,
wherein cameras in the set of camera devices are part of a camera array mounted on the front, side, or rear of a vehicle; and
wherein the scene area is an environment in which the vehicle is located or moving.
3. The method of claim 2 , wherein the camera devices are angled or shaped such that in the case of level ground the lower portion of the scene area will include ground closer to the camera devices than the upper portion of the scene area.
4. The method of claim 1 ,
wherein performing the first vehicle control operation is based on at least one of the determined distances corresponding to the lower portion of the scene area.
5. The method of claim 4 , wherein the first vehicle control operation includes at least one of a braking, steering, or speed control operation.
6. The method of claim 4 , further comprising:
performing depth processing on image data corresponding to an upper portion of the scene area to determine distances to one or more objects shown in the upper portion of the scene area after performing the depth processing on image data corresponding to the lower portion of the scene area; and
performing a second vehicle control operation based on at least one of the determined distances to one or more objects shown in the upper portion of said scene area, the second vehicle control operation being performed after the first vehicle control operation.
7. The method of claim 4 , further comprising:
buffering data from multiple camera devices corresponding to the lower portion of scene area; and
outputting to one or more processors, which perform the depth processing, buffered data from the multiple camera devices corresponding to the lower portion of scene area, prior to outputting buffered data from the multiple camera devices corresponding to the upper portion of scene area.
8. The method of claim 1 , wherein capturing images of the scene area using the set of camera devices includes capturing images using multiple camera devices in parallel, the multiple camera devices including sensors with rolling shutters that capture the scene area beginning at the bottom of the scene area and progressing towards the top of the scene area.
9. A system for capturing and using images, the system comprising:
a set of camera devices, the set of camera devices including camera devices; and
a processor configured to:
operate the set of camera devices to capture images of a scene area;
read image data out of one or more of the camera devices beginning at a bottom of a captured scene area and progressing towards the top of the captured scene area;
perform depth processing on image data corresponding to a lower portion of scene area to determine distances to one or more objects prior to performing depth processing on image data corresponding to an upper portion of scene area; and
control the system to perform a first vehicle control operation based on at least one of the determined distances.
10. The system of claim 9 ,
wherein cameras in the set of camera devices are part of a camera array mounted on the front, side or rear of a vehicle; and
wherein the scene area is an environment in which the vehicle is located or moving.
11. The system of claim 10 , wherein the camera devices are angled or shaped such that in the case of level ground the lower portion of the scene area will include ground closer to the camera devices than the upper portion of the scene area.
12. The system of claim 9 , wherein the processor is further configured to:
control the system to perform the first vehicle control operation based on at least one of the determined distances corresponding to the lower portion of the scene area, as part of being configured to control the system to perform the first vehicle control operation.
13. The system of claim 12 , wherein the first vehicle control operation includes at least one of a braking, steering, or speed control operation.
14. The system of claim 12 , wherein the first vehicle control operation includes a vehicle suspension control operation.
15. The system of claim 12 , wherein the processor is further configured to:
perform depth processing on image data corresponding to an upper portion of the scene area to determine distances to one or more objects shown in the upper portion of the scene area after performing the depth processing on image data corresponding to the lower portion of the scene area; and
control the system to perform a second vehicle control operation based on at least one of the determined distances to one or more objects shown in the upper portion of the scene area, the second vehicle control operation being performed after the first vehicle control operation.
16. The system of claim 12 , further comprising:
memory; and
wherein the processor is further configured to:
buffer data from multiple camera devices corresponding to the lower portion of scene area into the memory; and
output from the memory to one or more processors which perform the depth processing, buffered data from the multiple camera devices corresponding to the lower portion of scene area, prior to outputting buffered data from the multiple camera devices corresponding to the upper portion of scene area.
17. The system of claim 9 , wherein the multiple camera devices including sensors with rolling shutters that capture the scene area beginning at the bottom of the scene area and progressing towards the top of the scene area, and
wherein the processor is configured to control the system to capture images from multiple camera devices in parallel, as part of being configured to operate the set of camera devices to capture images of a scene area.
18. A non-transitory computer readable medium including computer executable instructions which when executed by a processor control a system to perform the steps of:
capturing images of a scene area using a set of camera devices the set of camera devices including camera devices;
reading image data out of one or more of the camera devices beginning at a bottom of a captured scene area and progressing towards the top of the captured scene area,
performing depth processing on image data corresponding to a lower portion of scene area to determine distances to one or more objects prior to performing depth processing on image data corresponding to an upper portion of scene area; and
performing a first vehicle control operation based on at least one of the determined distances.