IP Library Granted Patent US 11,577,756
Granted Patent B2
US 11,577,756 · App. 16/672,334 · Granted Feb 14, 2023

Detecting out-of-model scenarios for an autonomous vehicle

Inventors: John Hayes (Mountain View, CA); Volkmar Uhlig (Cupertino, CA); Akash J. Sagar (Redwood City, CA); Nima Soltani (Los Altos, CA); Feng Tian (Foster City, CA)
Assignee: GHOST AUTONOMY INC.
B60W60/00272B60R11/04B60W50/0097G05D1/0221G06N20/00G06V20/56B60R2300/804B60R2300/8086
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,577,756
App. No.
16/672,334
Granted
Feb 14, 2023
Kind
B2
Abstract

Detecting out-of-model scenarios for an autonomous vehicle including: determining, based on first sensor data from one or more sensors, an environmental state relative to the autonomous vehicle, wherein operational commands for the autonomous vehicle are based on a selected machine learning model, wherein the selected machine learning model comprises a first machine learning model; comparing the environmental state to a predicted environmental state relative to the autonomous vehicle; and determining, based on a differential between the environmental state and the predicted environmental state, whether to select a second machine learning model as the selected machine learning model.

Claims (40)

1. A method for detecting out-of-model scenarios for an autonomous vehicle, comprising:

determining, based on first sensor data from one or more sensors, an environmental state external to the autonomous vehicle, wherein the environmental state comprises an arrangement of one or more visual anchors identified in the first sensor data, wherein operational commands for the autonomous vehicle are output by a selected machine learning model, wherein the selected machine learning model comprises a first machine learning model;

comparing the environmental state to a predicted environmental state external to the autonomous vehicle, wherein the predicted environmental state comprises a predicted arrangement of the one or more visual anchors;

determining, based on a differential between the environmental state and the predicted environmental state, whether to select a second machine learning model as the selected machine learning model;

performing, by the autonomous vehicle, one or more operational commands output by the selected machine learning model; and

wherein the first machine learning model and the second machine learning model are included in a plurality of machine learning models, wherein each machine learning model of the plurality of machine learning models corresponds to a respective driving mode of the autonomous vehicle.

2. The method of claim 1 , further comprising determining the predicted environmental state based on second sensor data.

3. The method of claim 2 , wherein the second sensor data comprises sensor data associated with a time window ending at a time offset relative to a current time.

4. The method of claim 1 , wherein determining, based on the differential between the environmental state and the predicted environmental state, whether to select the second machine learning model as the selected machine learning model comprises determining to select the second machine learning model as the selected machine learning model in response to the differential meeting a threshold.

5. The method of claim 1 , wherein determining, based on the differential between the environmental state and the predicted environmental state, whether to select the second machine learning model as the selected machine learning model comprises determining to keep the first machine learning model as the selected machine learning model in response to the differential falling below a threshold.

6. The method of claim 1 , wherein comparing the environmental state to the predicted environmental state comprises comparing the arrangement of the one or more visual anchors in the environmental state to the predicted arrangement of the one or more visual anchors in the predicted environmental state.

7. The method of claim 6 , wherein comparing the arrangement of the one or more visual anchors in the environmental state to the predicted arrangement of the one or more visual anchors in the predicted environmental state comprises calculating a distance between a first multidimensional vector based on to the arrangement of the one or more visual anchors in the environmental state and a second multidimensional vector based on the predicted arrangement of the one or more visual anchors in the predicted environmental state.

8. An apparatus for detecting out-of-model scenarios for an autonomous vehicle, the apparatus configured to perform steps comprising:

determining, based on first sensor data from one or more sensors, an environmental state external to the autonomous vehicle, wherein the environmental state comprises an arrangement of one or more visual anchors identified in the first sensor data, wherein operational commands for the autonomous vehicle are output by a selected machine learning model, wherein the selected machine learning model comprises a first machine learning model;

comparing the environmental state to a predicted environmental state external to the autonomous vehicle, wherein the predicted environmental state comprises a predicted arrangement of the one or more visual anchors;

determining, based on a differential between the environmental state and the predicted environmental state, whether to select a second machine learning model as the selected machine learning model;

performing, by the autonomous vehicle, one or more operational commands output by the selected machine learning model; and

wherein the first machine learning model and the second machine learning model are included in a plurality of machine learning models, wherein each machine learning model of the plurality of machine learning models corresponds to a respective driving mode of the autonomous vehicle.

9. The apparatus of claim 8 , wherein the steps further comprise determining the predicted environmental state based on second sensor data.

10. The apparatus of claim 9 , wherein the second sensor data comprises sensor data associated with a time window ending at a time offset relative to a current time.

11. The apparatus of claim 8 , wherein determining, based on the differential between the environmental state and the predicted environmental state, whether to select the second machine learning model as the selected machine learning model comprises determining to select the second machine learning model as the selected machine learning model in response to the differential meeting a threshold.

12. The apparatus of claim 8 , wherein determining, based on the differential between the environmental state and the predicted environmental state, whether to select the second machine learning model as the selected machine learning model comprises determining to keep the first machine learning model as the selected machine learning model in response to the differential falling below a threshold.

13. The apparatus of claim 8 , wherein comparing the environmental state to the predicted environmental state comprises comparing the arrangement of the one or more visual anchors in the environmental state to the predicted arrangement of the one or more visual anchors in the predicted environmental state.

14. An autonomous vehicle for detecting out-of-model scenarios for an autonomous vehicle, comprising:

determining, based on first sensor data from one or more sensors, an environmental state external to the autonomous vehicle, wherein the environmental state comprises an arrangement of one or more visual anchors identified in the first sensor data, wherein operational commands for the autonomous vehicle are output by a selected machine learning model, wherein the selected machine learning model comprises a first machine learning model;

comparing the environmental state to a predicted environmental state external to the autonomous vehicle, wherein the predicted environmental state comprises a predicted arrangement of the one or more visual anchors;

determining, based on a differential between the environmental state and the predicted environmental state, whether to select a second machine learning model as the selected machine learning model;

performing, by the autonomous vehicle, one or more operational commands output by the selected machine learning model; and

wherein the first machine learning model and the second machine learning model are included in a plurality of machine learning models, wherein each machine learning model of the plurality of machine learning models corresponds to a respective driving mode of the autonomous vehicle.

15. The autonomous vehicle of claim 14 , wherein the steps further comprise determining the predicted environmental state based on second sensor data.

16. The autonomous vehicle of claim 15 , wherein the second sensor data comprises sensor data associated with a time window ending at a time offset relative to a current time.

17. The autonomous vehicle of claim 14 , wherein determining, based on the differential between the environmental state and the predicted environmental state, whether to select the second machine learning model as the selected machine learning model comprises determining to select the second machine learning model as the selected machine learning model in response to the differential meeting a threshold.

18. The autonomous vehicle of claim 14 , wherein determining, based on the differential between the environmental state and the predicted environmental state, whether to select the second machine learning model as the selected machine learning model comprises determining to keep the first machine learning model as the selected machine learning model in response to the differential falling below a threshold.

19. The autonomous vehicle of claim 14 , wherein comparing the environmental state to the predicted environmental state comprises comparing the arrangement of the one or more visual anchors in the environmental state to the predicted arrangement of the one or more visual anchors in the predicted environmental state.

20. A computer program product disposed upon a non-transitory computer readable medium, the computer program product comprising computer program instructions for detecting out-of-model scenarios for an autonomous vehicle that, when executed, cause a computer system of the autonomous vehicle to carry out the steps of:

determining, based on first sensor data from one or more sensors, an environmental state external to the autonomous vehicle, wherein the environmental state comprises an arrangement of one or more visual anchors identified in the first sensor data, wherein operational commands for the autonomous vehicle are output by a selected machine learning model, wherein the selected machine learning model comprises a first machine learning model;

comparing the environmental state to a predicted environmental state external to the autonomous vehicle, wherein the predicted environmental state comprises a predicted arrangement of the one or more visual anchors;

determining, based on a differential between the environmental state and the predicted environmental state, whether to select a second machine learning model as the selected machine learning model;

performing, by the autonomous vehicle, one or more operational commands output by the selected machine learning model; and

wherein the first machine learning model and the second machine learning model are included in a plurality of machine learning models, wherein each machine learning model of the plurality of machine learning models corresponds to a respective driving mode of the autonomous vehicle.

Assignments (3)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 18, 2024
From: GHOST AUTONOMY, INC.
To: APPLIED INTUITION, INC.
Reel/Frame 068982/0647 →
CHANGE OF NAME Recorded Aug 8, 2022
From: GHOST LOCOMOTION INC.
To: GHOST AUTONOMY INC.
Reel/Frame 061118/0665 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 17, 2020
From: HAYES, JOHN; UHLIG, VOLKMAR; SAGAR, AKASH J.; SOLTANI, NIMA; TIAN, FENG
To: GHOST LOCOMOTION INC.
Reel/Frame 054389/0654 →
Continuity (2)
Provisional Application 62900032 · Sep 13, 2019
Related Publication 20210078611A1 · Mar 18, 2021