Sensor synchronization system
Disclosed are systems and methods for a sensor synchronization system. Sensors of an autonomous vehicle (AV) are synchronized to cause sampling of a first sensor of the sensors to occur in alignment with sampling of a second sensor of the sensors. A sampling time point for the first sensor is determined based on the synchronizing the sensors, the sampling time point comprising a time when the first sensor is in alignment with the second sensor. The sampling time point is provided to the first sensor. A controller circuit of the first sensor determines an offset of the first sensor to apply to a local system time of the first sensor to cause a data acquisition of the first sensor to occur at the sampling time point. The data acquisition is performed at the first sensor using the sampling time point and the offset.
1 . A method, comprising:
synchronizing sensors of an autonomous vehicle (AV) to cause sampling of a first sensor to occur in alignment with sampling of a second sensor of the sensors, wherein the first sensor sweeps horizontally across a field of view of the second sensor and the second sensor includes a rolling shutter that sweeps vertically through N rows of the field of view of the second sensor;
determining an exposure time for the second sensor indicating a duration of time the second sensor is exposed to a scene, wherein the exposure time is dynamically adjustable between a minimum time duration and a maximum time duration based on an amount of ambient light;
determining a sampling time point for the first sensor based on a halfway point of an exposure duration that is an average of the minimum time duration and the maximum time duration;
providing the sampling time point to the first sensor;
determining, at a controller circuit of the first sensor, an offset of the first sensor to apply to a local system time of the first sensor to cause a data acquisition of the first sensor to occur at the sampling time point; and
causing the data acquisition to be performed at the first sensor using the sampling time point and the offset.
2 . The method of claim 1 , wherein synchronizing the sensors is based on a calibration of the sensors that identifies physical locations of the sensors on the AV with respect to each other.
3 . The method of claim 1 , wherein the sensors comprise at least one of Light Detection and Ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, Electromagnetic Detection and Ranging (EmDAR) sensors, Sound Navigation and Ranging (SONAR) sensors, Sound Detection and Ranging (SODAR) sensors, Global Navigation Satellite System (GNSS) receivers including Global Positioning System (GPS) receivers, cameras accelerometers, gyroscopes, inertial measurement units (IMU), infrared sensors, laser rangefinders, ultrasonic sensors, infrasonic sensors, or microphones.
4 . The method of claim 3 , wherein the first sensor comprises a Lidar sensor, and wherein the second sensor comprises a camera.
5 . The method of claim 1 , further comprising, prior to determining the offset, synchronizing the local system time of the first sensor to a unified time reference that is applied across each of the sensors.
6 . The method of claim 5 , wherein synchronizing the local system time further comprises:
receiving time synchronization events at the controller circuit of the first sensor, the time synchronization events corresponding to a time synchronization protocol for synchronizing to the unified time reference;
applying a local timestamp of the first sensor to the time synchronization events;
based on at least two of the time synchronization events and the corresponding local timestamps, determining frequency and offset adjustments to the local system time of the first sensor to cause the local system time to synchronize with the unified time reference;
applying the frequency and offset adjustments to the local system time to generate an adjusted local system time; and
utilizing the adjusted local system time to cause the data acquisition to be performed at the sampling time point using the offset.
7 . The method of claim 6 , wherein the time synchronization protocol comprises at least one of a global position system (GPS) protocol, a precision time protocol (PTP) protocol, a generic precision time protocol (gPTP) protocol, or a network time protocol (NTP) protocol.
8 . The method of claim 1 , wherein the sampling time point comprises the time when the first sensor is in alignment with the second sensor in accordance with temporal parameters and spatial parameters of the first sensor and the second sensor.
9 . The method of claim 1 , wherein the data acquisition of the first sensor is to be fused with another data acquisition of the second sensor occurring at the sampling time point.
10 . An apparatus, comprising:
at least one processor to:
synchronize sensors of an autonomous vehicle (AV) to cause sampling of a first sensor of the sensors to occur in alignment with sampling of a second sensor of the sensors, wherein the first sensor sweeps horizontally across a field of view of the second sensor and the second sensor includes a rolling shutter that sweeps vertically through N rows of the field of view of the second sensor;
determine an exposure time for the second sensor indicating a duration of time the second sensor is exposed to a scene, wherein the exposure time is dynamically adjustable between a minimum time duration and a maximum time duration based on an amount of ambient light;
determine a sampling time point for the first sensor based on a halfway point of an exposure duration that is an average of the minimum time duration and the maximum time duration;
provide the sampling time point to the first sensor;
determine, at a controller circuit of the first sensor, an offset of the first sensor to apply to a local system time of the first sensor to cause a data acquisition of the first sensor to occur at the sampling time point; and cause the data acquisition to be performed at the first sensor using the sampling time point and the offset.
11 . The apparatus of claim 10 , wherein the sensors are synchronized based on a calibration of the sensors that identifies physical locations of the sensors on the AV with respect to each other.
12 . The apparatus of claim 10 , wherein the sensors comprise at least one of Light Detection and Ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, Electromagnetic Detection and Ranging (EmDAR) sensors, Sound Navigation and Ranging (SONAR) sensors, Sound Detection and Ranging (SODAR) sensors, Global Navigation Satellite System (GNSS) receivers including Global Positioning System (GPS) receivers, cameras, accelerometers, gyroscopes, inertial measurement units (IMU), infrared sensors, laser rangefinders, ultrasonic sensors, infrasonic sensors, or microphones.
13 . The apparatus of claim 10 , wherein the processor is further to synchronize, prior to determining the offset, the local system time of the first sensor to a unified time reference that is applied across each of the sensors.
14 . The apparatus of claim 13 , wherein the processor to synchronize, prior to determining the offset, the local system time further comprises the processor to:
receive time synchronization events at the controller circuit of the first sensor, the time synchronization events corresponding to a time synchronization protocol for synchronizing to the unified time reference;
apply a local timestamp of the first sensor to the time synchronization events;
based on at least two of the time synchronization events and the corresponding local timestamps, determine frequency and offset adjustments to the local system time of the first sensor to cause the local system time to synchronize with the unified time reference;
apply the frequency and offset adjustments to the local system time to generate an adjusted local system time; and
utilize the adjusted local system time to cause the data acquisition to be performed at the sampling time point using the offset.
15 . The apparatus of claim 10 , wherein the sampling time point comprises the time when the first sensor is in alignment with the second sensor in accordance with temporal parameters and spatial parameters of the first sensor and the second sensor.
16 . A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to:
synchronize sensors of an autonomous vehicle (AV) to cause sampling of a first sensor of the sensors to occur in alignment with sampling of a second sensor of the sensors, wherein the first sensor sweeps horizontally across a field of view of the second sensor and the second sensor includes a rolling shutter that sweeps vertically through N rows of the field of view of the second sensor;
determine an exposure time for the second sensor indicating a duration of time the second sensor is exposed to a scene, wherein the exposure time is dynamically adjustable between a minimum time duration and a maximum time duration based on an amount of ambient light;
determine a sampling time point for the first sensor based on a halfway point of an exposure duration that is an average of the minimum time duration and the maximum time duration;
provide the sampling time point to the first sensor;
determine, at a controller circuit of the first sensor, an offset of the first sensor to apply to a local system time of the first sensor to cause a data acquisition of the first sensor to occur at the sampling time point; and cause the data acquisition to be performed at the first sensor using the sampling time point and the offset.
17 . The non-transitory computer-readable medium of claim 16 , wherein the sensors are synchronized based on a calibration of the sensors that identifies physical locations of the sensors on the AV with respect to each other.
18 . The non-transitory computer-readable medium of claim 16 , wherein the sensors comprise at least one of Light Detection and Ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, Electromagnetic Detection and Ranging (EmDAR) sensors, Sound Navigation and Ranging (SONAR) sensors, Sound Detection and Ranging (SODAR) sensors, Global Navigation Satellite System (GNSS) receivers including Global Positioning System (GPS) receivers, cameras, accelerometers, gyroscopes, inertial measurement units (IMU), infrared sensors, laser rangefinders, ultrasonic sensors, infrasonic sensors, or microphones.
19 . The non-transitory computer-readable medium of claim 16 , wherein the one or more processors to synchronize, prior to determining the offset, the local system time of the first sensor to a unified time reference that is applied across each of the sensors comprises the one or more processors to:
receive time synchronization events at the controller circuit of the first sensor, the time synchronization events corresponding to a time synchronization protocol for synchronizing to the unified time reference; apply a local timestamp of the first sensor to the time synchronization events;
based on at least two of the time synchronization events and the corresponding local timestamps, determine frequency and offset adjustments to the local system time of the first sensor to cause the local system time to synchronize with the unified time reference;
apply the frequency and offset adjustments to the local system time to generate an adjusted local system time; and
utilize the adjusted local system time to cause the data acquisition to be performed at the sampling time point using the offset.
20 . The non-transitory computer-readable medium of claim 16 , wherein the sampling time point comprises the time when the first sensor is in alignment with the second sensor in accordance with temporal parameters and spatial parameters of the first sensor and the second sensor.