System and Methods for Automated Detection of Vehicle Cabin Events for Triggering Remote Operator Assistance

1 . A computer-implemented method comprising:

obtaining, by a computing system comprising one or more computing devices, sensor data associated with an interior of an autonomous vehicle;

determining, by the computing system and using the sensor data, that the interior of the autonomous vehicle contains one or more passengers;

in response to determining that the interior of the autonomous vehicle contains one or more passengers, analyzing, by the computing system, the sensor data to determine whether the one or more passengers are violating one or more passenger policies; and

in response to determining that the one or more passengers are violating one or more passenger policies, automatically initiating, by the computing system, a remote assistance session with a remote operator.

2 . The computer-implemented method of claim 1 , wherein the sensor data includes image data captured by a camera depicting the interior of the autonomous vehicle.

3 . The computer-implemented method of claim 2 , wherein determining that the one or more passengers are violating one or more passenger policies further comprises:

identifying, by the computing system and based on the image data, an item possessed by the one or more passengers;

determining, by the computing system, whether the item is included in a list of prohibited items; and

in response to determining that the item is included in the list of prohibited items, determining, by the computing system, that the one or more passengers are violating one or more passenger policies.

4 . The computer-implemented method of claim 3 , wherein determining that the one or more passengers are violating one or more passenger policies can be based both on an item possessed by the one or more passengers and pose data for the one or more passengers.

5 . The computer-implemented method of claim 2 , wherein determining that the one or more passengers are violating one or more passenger policies further comprises:

determining, by the computing system, a position and location of a body of a respective passenger of the one or more passengers based on the image data;

based on the determined position and location, determining, by the computing system, whether a portion of the body of the respective passenger is outside of an area of the autonomous vehicle designated for passengers; and

in response to determining that a portion of the body of a respective passenger is outside of the area of the autonomous vehicle designated for passengers, determining, by the computing system, that the one or more passengers are violating one or more passenger policies.

6 . The computer-implemented method of claim 5 , wherein the image data includes image data from more than one camera.

7 . The computer-implemented method of claim 2 , wherein determining that the one or more passengers are violating one or more passenger policies further comprises:

identifying, by the computing system, pose data for the one or more passengers based on image data;

determining, by the computing system, a list of potential passenger scenarios associated with the one or more passengers based on the pose data; and

generating, by the computing system, a confidence score for each potential passenger scenario from the list of potential scenarios, the confidence score for a particular potential passenger scenario from the list of potential passenger scenarios indicating an estimated likelihood that the one or more passengers is participating in the particular potential passenger scenario from the list of potential passenger scenarios.

8 . The computer-implemented method of claim 7 , further comprising:

determining, by the computing system for each respective potential passenger scenario in the list of potential passenger scenarios, whether the confidence score associated with the respective potential passenger scenario has a confidence score above a predetermined threshold; and

in response to determining that a respective potential passenger scenario has a confidence score that exceeds the predetermine threshold, determining, by the computing system, whether the respective potential passenger scenario is included in a list of banned passenger scenarios.

9 . The computer-implemented method of claim 8 , further comprising:

in accordance with a determination that at least one potential passenger scenario in the list of potential passenger scenarios is included in the list of banned passenger scenarios and has an associated confidence score that exceeds the predetermined threshold, determining, by the computing system, that the one or more passengers are violating one or more passenger policies.

10 . The computer-implemented method of claim 9 , wherein the sensor data includes audio data captured by a microphone.

11 . The computer-implemented method of claim 10 , wherein confidence scores are generated, at least in part, based on audio data.

12 . The computer-implemented method of claim 1 , wherein the obtained sensor data is received from sensors included in the autonomous vehicle.

13 . The computer-implemented method of claim 1 , wherein a data transfer rate at which sensor data is received from the autonomous vehicle is variable.

14 . The computer-implemented method of claim 12 , further comprising:

in response to input from the remote operator, transmitting, by the computing system, a command to the autonomous vehicle that results in a change to the data transfer rate at which sensor data is received from the autonomous vehicle from a first rate to a second rate.

15 . The computer-implemented method of claim 1 , wherein a resolution of the sensor data that is received from the autonomous vehicle is variable.

16 . The computer-implemented method of claim 1 , further comprising:

while a remote assistance session is ongoing, enabling, by the computing system, two-way communication between the one or more passengers and the remote operator.

17 . The computer-implemented method of claim 1 , further comprising:

while a remote assistance session is ongoing, in response to input from a remote operator, transmitting, by the computing system, one or more control commands to the autonomous vehicle.

18 . An autonomous vehicle, comprising:

one or more processors; and

one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising:

obtaining sensor data associated with an interior of an autonomous vehicle;

determining using the sensor data that the interior of the autonomous vehicle contains one or more passengers;

in response to determining that the interior of the autonomous vehicle contains one or more passengers, analyzing the sensor data to determine whether the one or more passengers are violating one or more passenger policies; and

in response to determining that the one or more passengers are violating one or more passenger policies, automatically initiating a remote assistance session with a remote operator.

19 . The autonomous vehicle of claim 18 , the operations further comprising:

while a remote assistance session is ongoing, in response to input from a remote operator, implementing one or more control commands for the autonomous vehicle based at least in part on the input from the remote operator.

20 . A computing system comprising:

one or more processors; and

one or more tangible, non-transitory, computer readable media that collectively store instructions that when executed by the one or more processors cause the computing system to perform operations comprising:

obtaining sensor data associated with an interior of an autonomous vehicle;

determining using the sensor data that the interior of the autonomous vehicle contains one or more passengers;

in response to determining that the interior of the autonomous vehicle contains one or more passengers, analyzing the sensor data to determine whether the one or more passengers are violating one or more passenger policies; and

in response to determining that the one or more passengers are violating one or more passenger policies, automatically initiating a remote assistance session with a remote operator.