Thermal sensor data vehicle perception
Provided are methods for thermal sensor data vehicle perception, which can include obtaining thermal sensor data, obtaining non-thermal sensor data, and determining, based on the thermal sensor data and the non-thermal sensor data, a perception parameter indicative of an object. Some methods described also include generating a trajectory for an autonomous vehicle. Systems and computer program products are also provided.
1 . A method comprising:
obtaining, by at least one processor, thermal sensor data associated with an environment in which an autonomous vehicle is operating;
obtaining, using the at least one processor, non-thermal sensor data associated with the environment;
identifying, using the at least one processor, a thermal reflection on a surface of a first object in the environment based at least in part on the thermal sensor data;
determining, using the at least one processor, a presence of a second object based at least in part on the identified thermal reflection on the surface of the first object; and
generating, using the at least one processor, a trajectory for the autonomous vehicle based at least in part on the determined presence of the second object.
2 . The method of claim 1 , the method comprising:
determining, using the at least one processor, based at least in part on the thermal sensor data, a reaction parameter for an emergency reaction; and
generating, using the at least one processor, based at least in part on the reaction parameter, a second trajectory for the autonomous vehicle.
3 . The method of claim 1 , wherein the second object is an agent.
4 . The method of claim 1 , wherein the thermal sensor data is obtained from one or more of: a thermal camera and an infrared thermal sensor.
5 . The method of claim 1 , wherein the non-thermal sensor data is obtained from one or more of: an image sensor, a Light Detection and Ranging (LiDAR) sensor, a RADAR sensor, and an ultrasonic sensor.
6 . The method of claim 1 , wherein the identified thermal reflection on the surface of the first object is indicative of a state of the second object.
7 . The method of claim 1 , wherein the identified thermal reflection on the surface of the first object is indicative of a feature of the second object.
8 . The method of claim 7 , wherein the feature is one or more of: a light, a doorknob, a window frame, and a window.
9 . The method of claim 1 , wherein the first object comprises a driving surface.
10 . The method of claim 1 , further comprising determining, using the at least one processor, based on the thermal sensor data and the identified thermal reflection on the surface of the first object, one or more segments of the second object.
11 . The method of claim 1 , wherein generating the trajectory for the autonomous vehicle comprises prioritizing, using the at least one processor, based at least in part on the identified thermal reflection on the surface of the first object, the second object.
12 . The method of claim 1 , the method comprising:
predicting, using the at least one processor, based at least in part on the identified thermal reflection on the surface of the first object, an object trajectory of the second object.
13 . The method of claim 12 , wherein predicting the object trajectory comprises:
overlaying, using the at least one processor, based at least in part on the identified thermal reflection on the surface of the first object, a representation of the thermal sensor data on a three-dimension reconstruction of the second object;
estimating, using the at least one processor, based at least in part on the identified thermal reflection on the surface of the first object, a location of the second object; and
predicting, using the at least one processor, based at least in part on the estimated location, the object trajectory of the second object.
14 . The method of claim 12 , wherein generating the trajectory for the autonomous vehicle comprises generating, using the at least one processor, based at least in part on the object trajectory, the trajectory for the autonomous vehicle.
15 . A non-transitory computer readable medium comprising instructions stored thereon that, when executed by at least one processor, cause the at least one processor to carry out operations comprising:
obtaining, by at least one processor, thermal sensor data associated with an environment in which an autonomous vehicle is operating;
obtaining, using the at least one processor, non-thermal sensor data associated with the environment;
identifying, using the at least one processor, a thermal reflection on a surface of a first object in the environment based at least in part on the thermal sensor data;
determining, using the at least one processor, a presence of a second object based at least in part on the identified thermal reflection on the surface of the first object; and
generating, using the at least one processor, a trajectory for the autonomous vehicle based at least in part on the determined presence of the second object.
16 . A system, comprising at least one processor; and at least one memory storing instructions thereon that, when executed by the at least one processor, cause the at least one processor to:
obtain thermal sensor data associated with an environment in which an autonomous vehicle is operating;
obtain non-thermal sensor data associated with the environment;
identify a thermal reflection on a surface of a first object in the environment based at least in part on the thermal sensor data;
determine, using the at least one processor, a presence of a second object based at least in part on the identified thermal reflection on the surface of the first object; and
generate a trajectory for the autonomous vehicle based at least in part on the determined presence of the second object.
17 . The system of claim 16 , wherein the at least one memory storing instructions, when executed by at least one processor, cause the at least one processor to:
predict, based at least in part on the identified thermal reflection on the surface of the first object, an object trajectory of the second object.
18 . The system of claim 17 , wherein to predict the object trajectory comprises to:
overlay, based at least in part on the identified thermal reflection on the surface of the first object, a representation of the thermal reflection on a three-dimension reconstruction of the second object;
estimate, based at least in part on the identified thermal reflection on the surface of the first object, a location of the second object; and
predict, based at least in part on the estimated location, the object trajectory of the second object.
19 . The method of claim 1 , wherein the second object is occluded from the non-thermal sensor data.
20 . The method of claim 1 , wherein the second object is occluded from the autonomous vehicle.