Navigating autonomous vehicles based on modulation of a world model representing traffic entities

1. A method comprising:

receiving, by an autonomous vehicle, sensor data from sensors mounted on the autonomous vehicle, the sensor data comprising one or more images;

generating a point cloud representation of the surroundings of the autonomous vehicle based on the sensor data;

identifying, one or more traffic entities based on the sensor data, the traffic entities representing non-stationary objects in traffic in which the autonomous vehicle is driving;

for each of the one or more traffic entities:

determining one or more motion parameters describing movement of the traffic entity;

providing an image of the one or more images, the image showing the traffic entity as input to a machine learning model configured to receive an input image showing an input traffic entity and output summary statistics of expected human responses describing a hidden context of the input traffic entity shown in the input image;

determining the hidden context of the traffic entity based on the output of the machine learning based model;

determining a region of the point cloud where the traffic entity is expected to reach within a threshold time interval; and

modifying the region based on the hidden context of the traffic entity; and

navigating the autonomous vehicle so that the autonomous vehicle stays at least a threshold distance away from the modified region of each of the one or more traffic entities.

2. The method of claim 1 , wherein the region is in a direction determined based on a motion vector of the traffic entity, wherein the traffic entity represents a user, wherein modifying the region based on the hidden context comprises:

responsive to determining based on the hidden context that the user represented by the traffic entity is likely to move in the direction having a component along the motion vector, extending the region along the direction of the motion vector.

3. The method of claim 1 , wherein the region is in a direction determined based on a motion vector of the traffic entity, wherein the traffic entity represents a user, wherein modifying the region based on the hidden context comprises:

responsive to determining based on the hidden context that the user represented by the traffic entity is likely to move in a direction having a component opposite to direction of the motion vector, decreasing a size of the region along the direction of the motion vector.

4. The method of claim 1 , wherein the hidden context represents a state of mind of a user represented by the traffic entity.

5. The method of claim 1 , wherein the hidden context represents a task that a user represented by the traffic entity is planning on accomplishing.

6. The method of claim 1 , wherein the hidden context represents a degree of awareness of the autonomous vehicle by a user represented by the traffic entity.

7. The method of claim 1 , wherein the hidden context represents a goal of a user represented by the traffic entity, wherein the user expects to achieve the goal within a threshold time interval.

8. The method of claim 1 , wherein navigating the autonomous vehicle comprises:

generating signals for controlling the autonomous vehicle based on the one or more motion parameters and the hidden context of each of the one or more traffic entities; and

sending the generated signals to controls of the autonomous vehicle.

9. The method of claim 1 , wherein the sensor data represents one or more images captured by a camera mounted on the autonomous vehicle or a lidar scan captured by a lidar mounted on the autonomous vehicle.

10. A non-transitory computer readable storage medium storing instructions, that when executed by a processor, cause the processor to perform steps comprising:

receiving, by an autonomous vehicle, sensor data from sensors mounted on the autonomous vehicle, the sensor data comprising one or more images;

generating a point cloud representation of the surroundings of the autonomous vehicle based on the sensor data;

identifying, one or more traffic entities based on the sensor data, the traffic entities representing non-stationary objects in traffic in which the autonomous vehicle is driving;

for each of the one or more traffic entities:

determining one or more motion parameters describing movement of the traffic entity;

providing an image of the one or more images, the image showing the traffic entity as input to a machine learning model configured to receive an input image showing an input traffic entity and output summary statistics of expected human responses describing a hidden context of the input traffic entity shown in the input image;

determining the hidden context of the traffic entity based on the output of the machine learning based model;

determining a region of the point cloud where the traffic entity is expected to reach within a threshold time interval; and

modifying the region based on the hidden context of the traffic entity; and

navigating the autonomous vehicle so that the autonomous vehicle stays at least a threshold distance away from the modified region of each of the one or more traffic entities.

11. The non-transitory computer readable storage medium of claim 10 , wherein the region is in a direction determined based on a motion vector of the traffic entity, wherein the traffic entity represents a user, wherein modifying the region based on the hidden context comprises:

responsive to determining based on the hidden context that the user represented by the traffic entity is likely to move in the direction having a component along the motion vector, extending the region along the direction of the motion vector.

12. The non-transitory computer readable storage medium of claim 10 , wherein the region is in a direction determined based on a motion vector of the traffic entity, wherein the traffic entity represents a user, wherein modifying the region based on the hidden context comprises:

responsive to determining based on the hidden context that the user represented by the traffic entity is likely to move in a direction having a component opposite to direction of the motion vector, decreasing a size of the region along the direction of the motion vector.

13. The non-transitory computer readable storage medium of claim 10 , wherein the hidden context represents a state of mind of a user represented by the traffic entity.

14. The non-transitory computer readable storage medium of claim 10 , wherein the hidden context represents a task that a user represented by the traffic entity is planning on accomplishing.

15. The non-transitory computer readable storage medium of claim 10 , wherein the hidden context represents a degree of awareness of the autonomous vehicle by a user represented by the traffic entity.

16. The non-transitory computer readable storage medium of claim 10 , wherein the hidden context represents a goal of a user represented by the traffic entity, wherein the user expects to achieve the goal within a threshold time interval.

17. The non-transitory computer readable storage medium of claim 10 , wherein navigating the autonomous vehicle comprises:

generating signals for controlling the autonomous vehicle based on the one or more motion parameters and the hidden context of each of the one or more traffic entities; and

sending the generated signals to controls of the autonomous vehicle.

18. The non-transitory computer readable storage medium of claim 10 , wherein the sensor data represents one or more images captured by a camera mounted on the autonomous vehicle or a lidar scan captured by a lidar mounted on the autonomous vehicle.

19. A computer system comprising:

a processor; and

a non-transitory computer readable storage medium storing instructions that when executed by the processor, cause the processor to perform steps comprising:

receiving, by an autonomous vehicle, sensor data from sensors mounted on the autonomous vehicle, the sensor data comprising one or more images;

generating a point cloud representation of the surroundings of the autonomous vehicle based on the sensor data;

identifying, one or more traffic entities based on the sensor data, the traffic entities representing non-stationary objects in traffic in which the autonomous vehicle is driving;

for each of the one or more traffic entities:

determining one or more motion parameters describing movement of the traffic entity;

providing an image of the one or more images, the image showing the traffic entity as input to a machine learning model configured to receive an input image showing an input traffic entity and output summary statistics of expected human responses describing a hidden context of the input traffic entity shown in the input image;

determining the hidden context of the traffic entity based on the output of the machine learning based model;

determining a region of the point cloud where the traffic entity is expected to reach within a threshold time interval; and

modifying the region based on the hidden context of the traffic entity; and

navigating the autonomous vehicle so that the autonomous vehicle stays at least a threshold distance away from the modified region of each of the one or more traffic entities.

20. The computer system of claim 19 , wherein the region is in a direction determined based on a motion vector of the traffic entity, wherein the traffic entity represents a user, wherein modifying the region based on the hidden context comprises:

responsive to determining based on the hidden context that the user represented by the traffic entity is likely to move in the direction having a component along the motion vector, extending the region along the direction of the motion vector.

21. The computer system of claim 19 , wherein the region is in a direction determined based on a motion vector of the traffic entity, wherein the traffic entity represents a user, wherein modifying the region based on the hidden context comprises:

responsive to determining based on the hidden context that the user represented by the traffic entity is likely to move in a direction having a component opposite to direction of the motion vector, decreasing a size of the region along the direction of the motion vector.

22. The computer system of claim 19 , wherein the hidden context represents a state of mind of a user represented by the traffic entity.

23. The computer system of claim 19 , wherein the hidden context represents a degree of awareness of the autonomous vehicle by a user represented by the traffic entity.