Sensor triggering based on sensor simulation
Described herein are systems, methods, and non-transitory computer readable media for triggering a sensor operation of a second sensor (e.g., a camera) based on a predicted time of alignment with a first sensor (e.g., a LiDAR), where operation of the second sensor is simulated to determine the predicted time of alignment. In this manner, the sensor data captured by the two sensors is ensured to be substantially synchronized with respect to the physical environment being sensed. This sensor data synchronization based on predicted alignment of the sensors solves the technical problem of lack of sensor coordination and sensor data synchronization that would otherwise result from the latency associated with communication between sensors and a centralized controller and/or between sensors themselves.
1. A computer-implemented method for triggering a sensor operation, the method comprising:
simulating a sensor operation of a first sensor, wherein the simulating the sensor operation of the first sensor comprises:
receiving a set of initial parameters associated with the operation of the first sensor;
generating a virtual sensor system based at least in part on the set of initial parameters; and
training a predictive model based at least in part on the set of initial parameters;
predicting, based at least in part on the predictive model, a time at which the first sensor will be aligned with a second sensor;
triggering the second sensor to perform the sensor operation comprising an image capture function relating to a common location for the first sensor and the second sensor based at least in part on the predicted time;
determining an error between the predicted time and an actual time that the first sensor was aligned with the second sensor;
providing feedback data comprising an indication of the error to the predictive model; and
re-training the predictive model based at least in part on the feedback data to improve a predictive capability of the predictive model.
2. The computer-implemented method of claim 1 , wherein the time is a first predicted time, the method further comprising:
predicting, using the re-trained predictive model, a second time at which the first sensor will be aligned with the second sensor; and
triggering the second sensor to perform the sensor operation based at least in part on the second predicted time.
3. The computer-implemented method of claim 2 , wherein the error between the first predicted time and a first actual time that the first sensor was aligned with the second sensor is a first error, the method further comprising:
determining a second error between the second predicted time and a second actual time that the first sensor was aligned with the second sensor,
wherein the second error is smaller than the first error.
4. The computer-implemented method of claim 1 , wherein the first sensor is a Light Detection and Ranging (LiDAR) sensor and the second sensor is a camera.
5. The computer-implemented method of claim 4 , wherein the sensor operation is an image capture function of the camera.
6. The computer-implemented method of claim 5 , further comprising determining a delay associated with the image capture function of the camera, wherein the delay results from a rolling shutter of the camera that causes the camera to capture different portions of image data of a scene at different times.
7. The computer-implemented method of claim 6 , wherein triggering the second sensor to perform the sensor operation comprises triggering the camera to perform the image capture function prior to or after the predicted time based at least in part on the delay.
8. The computer-implemented method of claim 1 , wherein triggering the second sensor to perform the sensor operation comprises triggering the second sensor to perform the sensor operation at the predicted time.
9. A system for triggering a sensor operation, the system comprising:
at least one processor; and
at least one memory storing computer-executable instructions, wherein the at least one processor is configured to access the at least one memory and execute the computer-executable instructions to:
simulate a sensor operation of a first sensor, wherein the simulating the sensor operation of the first sensor comprises:
receiving a set of initial parameters associated with the operation of the first sensor;
generating a virtual sensor system based at least in part on the set of initial parameters;
training a predictive model based at least in part on the set of initial parameters;
predict, based at least in part on the predictive model, a time at which the first sensor will be aligned with a second sensor;
trigger the second sensor to perform the sensor operation comprising an image capture function relating to a common location for the first sensor and the second sensor based at least in part on the predicted time;
determine an error between the predicted time and an actual time that the first sensor was aligned with the second sensor;
provide feedback data comprising an indication of the error to the predictive model; and
re-train the predictive model based at least in part on the feedback data to improve a predictive capability of the predictive model.
10. The system of claim 9 , wherein the time is a first predicted time and the error between the first predicted time and a first actual time that the first sensor was aligned with the second sensor is a first error, and wherein the at least one processor is further configured to execute the computer-executable instructions to:
predict, using the re-trained predictive model, a second time at which the first sensor will be aligned with the second sensor;
trigger the second sensor to perform the sensor operation based at least in part on the second predicted time;
determine a second error between the second predicted time and a second actual time that the first sensor was aligned with the second sensor,
wherein the second error is smaller than the first error.
11. The system of claim 9 , wherein the first sensor is a Light Detection and Ranging (LiDAR) sensor and the second sensor is a camera, and wherein the sensor operation is an image capture function of the camera.
12. The system of claim 11 , wherein the at least one processor is further configured to execute the computer-executable instructions to determine a delay associated with the image capture function of the camera, wherein the delay results from a rolling shutter of the camera that causes the camera to capture different portions of image data of a scene at different times.
13. The system of claim 12 , wherein the at least one processor is configured to trigger the second sensor to perform the sensor operation by executing the computer-executable instructions to trigger the camera to perform the image capture function prior to or after the predicted time based at least in part on the delay.
14. The system of claim 9 , wherein the at least one processor is configured to trigger the second sensor to perform the sensor operation by executing the computer-executable instructions to trigger the second sensor to perform the sensor operation at the predicted time.