Vehicle sensor fusion
Various systems and methods for optimizing use of environmental and operational sensors are described herein. A system for improving sensor efficiency includes object recognition circuitry implementable in a vehicle to detect an object ahead of the vehicle, the object recognition circuitry configured to use an object detection operation to detect the object from sensor data of a sensor array, and the object recognition circuitry configured to use at least one object tracking operation to track the object between successive object detection operations; and a processor subsystem to: calculate a relative velocity of the object with respect to the vehicle; and configure the object recognition circuitry to adjust intervals between successive object detection operations based on the relative velocity of the object.
1. A system for improving sensor efficiency, the system comprising:
object recognition circuitry implementable in a vehicle to detect an object ahead of the vehicle, the object recognition circuitry configured to use an object detection operation to detect the object from sensor data of a sensor array, and the object recognition circuitry configured to use at least one object tracking operation to track the object between successive object detection operations; and
a processor subsystem to:
calculate a relative velocity of the object with respect to the vehicle; and
configure the object recognition circuitry to adjust intervals between successive object detection operations based on the relative velocity of the object;
wherein the object recognition circuitry is further configured to adjust the intervals between the successive object detection operations based on a determined number of object tracking operations, the number of object tracking operations being determined from each of: (i) a capture rate of the sensor data, (ii) a distance difference of the object over adjacent captures of the sensor data, and (iii) a threshold velocity for a given situation of an operational context of the vehicle.
2. The system of claim 1 , wherein the sensor array includes a visible light camera, and wherein the object detection operation includes:
obtaining an image from the visible light camera; and
performing, during the object detection operation, object recognition on the image to detect the object.
3. The system of claim 1 , wherein the sensor array includes a radar sensor, and wherein the object detection operation includes:
obtaining a radar scan from the radar sensor; and
performing, during the object detection operation, object recognition on the radar scan to detect the object.
4. The system of claim 1 , wherein to calculate the relative velocity of the object, the processor subsystem is to:
identify a current operating scenario of the vehicle;
assign weights to the sensor data from the sensor array based on the current operating scenario to create weighted sensor data; and
calculate the relative velocity of the object ahead of the vehicle using the weighted sensor data.
5. The system of claim 1 , wherein to calculate the relative velocity of the object, the processor subsystem is to:
calculate a first distance between the vehicle and the object in a first detection operation;
calculate a second distance between the vehicle and the object in a second detection operation;
calculate a difference between the first and second distances; and
calculate the relative velocity as the difference divided by a time interval between the first and second detection operations.
6. The system of claim 1 , wherein to configure the object recognition circuitry, the processor subsystem is to:
set the number of object tracking operations between successive object detection operations to adjust the intervals between successive object detection operations;
wherein the number of object tracking operations, n, is equal to:
n
=
[
d
*
f
v
max
]
-
1
where d is a distance difference of the object over adjacent detection frames, f is an input video stream capture rate, and vmax is the threshold velocity for the given situation of the operational context of the vehicle.
7. The system of claim 6 , wherein vmax is higher when the vehicle is travelling at a faster velocity, and vmax is lower when the vehicle is travelling at a slower velocity.
8. The system of claim 6 , wherein to configure the object recognition circuitry, the processor subsystem is to:
power off the object recognition circuitry during the intervals between successive detection operations.
9. The system of claim 1 , wherein the processor subsystem is to:
when the relative velocity is zero, use a pre-trained model to predict a time interval until a moving object is likely to appear; and
pause the object detection and object tracking operations for the time interval.
10. The system of claim 9 , wherein to pause the object detection and object tracking operations, the processor subsystem is to:
power off the object recognition circuitry for the time interval.
11. The system of claim 9 , wherein to pause the object detection and object tracking operations, the processor subsystem is to:
hibernate the object recognition circuitry for the time interval.
12. A method of improving sensor efficiency, the method comprising:
detecting, using object recognition circuitry installed in a vehicle, an object ahead of the vehicle, the object recognition circuitry configured to use an object detection operation to detect the object from sensor data of a sensor array, and the object recognition circuitry configured to use at least one object tracking operation to track the object between successive object detection operations;
calculating a relative velocity of the object with respect to the vehicle; and
configuring the object recognition circuitry to adjust intervals between successive object detection operations based on the relative velocity of the object;
wherein the object recognition circuitry is further configured to adjust the intervals between the successive object detection operations based on a determined number of object tracking operations, the number of object tracking operations being determined from each of: (i) a capture rate of the sensor data, (ii) a distance difference of the object over adjacent captures of the sensor data, and (iii) a threshold velocity for a given situation of an operational context of the vehicle.
13. The method of claim 12 , wherein the sensor array includes a visible light camera, and wherein detecting the object ahead of the vehicle comprises:
obtaining an image from the visible light camera; and
performing, during the object detection operation, object recognition on the image to detect the object.
14. The method of claim 12 , wherein the sensor array includes a radar sensor, and wherein detecting the object ahead of the vehicle comprises:
obtaining a radar scan from the radar sensor; and
performing, during the object detection operation, object recognition on the radar scan to detect the object.
15. The method of claim 12 , wherein calculating the relative velocity of the object comprises:
identifying a current operating scenario of the vehicle;
assigning weights to the sensor data from the sensor array based on the current operating scenario to create weighted sensor data; and
calculating the relative velocity of the object ahead of the vehicle using the weighted sensor data.
16. The method of claim 12 , wherein calculating the relative velocity of the object comprises:
calculating a first distance between the vehicle and the object in a first detection operation;
calculating a second distance between the vehicle and the object in a second detection operation;
calculating a difference between the first and second distances; and
calculating the relative velocity as the difference divided by a time interval between the first and second detection operations.
17. The method of claim 12 , wherein configuring the object recognition circuitry comprises:
setting the number of object tracking operations between successive object detection operations to adjust the intervals between successive object detection operations;
wherein the number of object tracking operations, n, is equal to:
n
=
[
d
*
f
v
max
]
-
1
where d is a distance difference of the object over adjacent detection frames, f is an input video stream capture rate, and vmax is the threshold velocity for the given situation of the operational context of the vehicle.
18. The method of claim 17 , wherein vrnax is higher when the vehicle is travelling at a faster velocity, and vmax is lower when the vehicle is travelling at a slower velocity.
19. The method of claim 17 , wherein configuring the object recognition circuitry comprises:
powering off the object recognition circuitry during the intervals between successive detection operations.
20. At least one non-transitory machine-readable medium comprising instructions for improving sensor efficiency, the instructions when executed by a machine, cause the machine to perform operations comprising:
detecting, using object recognition circuitry installed in a vehicle, an object ahead of the vehicle, the object recognition circuitry configured to use an object detection operation to detect the object from sensor data of a sensor array, and the object recognition circuitry configured to use at least one object tracking operation to track the object between successive object detection operations;
calculating a relative velocity of the object with respect to the vehicle; and
configuring the object recognition circuitry to adjust intervals between successive object detection operations based on the relative velocity of the object;
wherein the object recognition circuitry is further configured to adjust the intervals between the successive object detection operations based on a determined number of object tracking operations, the number of object tracking operations being determined from each of: (i) a capture rate of the sensor data, (ii) a distance difference of the object over adjacent captures of the sensor data, and (iii) a threshold velocity for a given situation of an operational context of the vehicle.
21. The non-transitory machine-readable medium of claim 20 , wherein the sensor array includes a visible light camera, and wherein detecting the object ahead of the vehicle comprises:
obtaining an image from the visible light camera; and
performing, during the object detection operation, object recognition on the image to detect the object.
22. The non-transitory machine-readable medium of claim 20 , wherein the sensor array includes a radar sensor, and wherein detecting the object ahead of the vehicle comprises:
obtaining a radar scan from the radar sensor; and
performing, during the object detection operation, object recognition on the radar scan to detect the object.
23. The non-transitory machine-readable medium of claim 20 , wherein calculating the relative velocity of the object comprises:
identifying a current operating scenario of the vehicle;
assigning weights to the sensor data from the sensor array based on the current operating scenario to create weighted sensor data; and
calculating the relative velocity of the object ahead of the vehicle using the weighted sensor data.
24. The non-transitory machine-readable medium of claim 20 , wherein configuring the object recognition circuitry comprises:
setting the number of object tracking operations between successive object detection operations to adjust the intervals between successive object detection operations;
wherein the number of object tracking operations, n, is equal to:
n
=
[
d
⋆
f
v
max
]
-
1
where d is a distance difference of the object over adjacent detection frames, f is an input video stream capture rate, and vmax is the threshold velocity for the given situation of the operational context of the vehicle.
25. The non-transitory machine-readable medium of claim 24 , wherein vmax is higher when the vehicle is travelling at a faster velocity, and vmax is lower when the vehicle is travelling at a slower velocity.