IP Library Granted Patent US 11,802,968
Granted Patent B2
US 11,802,968 · App. 15/995,041 · Granted Oct 31, 2023

System of vehicles equipped with imaging equipment for high-definition near real-time map generation

Inventors: Ro Gupta (Brooklyn, NY); Justin Day (Brooklyn, NY); Chase Nicholl (Brooklyn, NY); Max Henstell (Portland, OR); Ioannis Stamos (Brooklyn, NY); David Boyle (Brooklyn, NY); Ethan Sorrelgreen (Seattle, WA); Huong Dinh (New York, NY)
Assignee: Woven by Toyota, U.S., Inc.
G01S17/88B60R1/00G01C21/3841G01C21/3885G01C21/3893G01S17/86G01S17/89G06T7/11G06V10/17G06V20/58G06V20/584H04W4/44B60R2300/207B60R2300/301B60R2300/302B60R2300/406B60R2300/50G01S17/42G06T2207/30236
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,802,968
App. No.
15/995,041
Granted
Oct 31, 2023
Kind
B2
Abstract

Described are street level intelligence platforms, systems, and methods that can include a fleet of swarm vehicles having imaging devices. Images captured by the imaging devices can be used to produce and/or be integrated into maps of the area to produce high-definition maps in near real-time. Such maps may provide enhanced street level intelligence useful for fleet management, navigation, traffic monitoring, and/or so forth.

Claims (60)

1. A non-transitory processor readable medium storing code configured to be executed by a processor, the code comprising code to cause the processor to:

capture, from a camera of a smartphone disposed with a vehicle, a video of a streetscape, the camera communicatively coupled to the processor, which is also disposed with the vehicle;

perform, locally, at the processor, a first pass of computer vision analysis on the video of the streetscape to identify a plurality of predefined characteristic signals, each predefined characteristic signal from the plurality of predefined characteristic signals indicating a candidate high-priority event from a plurality of candidate high-priority events;

perform, locally, at the processor, a second pass of computer vision analysis on the plurality of candidates of high-priority events to identify a high-priority event, the second pass of computer vision analysis consuming more computational resources than the first pass of computer vision analysis such that the second pass of computer vision analysis cannot be performed on the video in real time;

transmit, over a wireless data network and to a remote analysis service, an indication of the high-priority event, such that the remote analysis service can integrate the high-priority event into a map of the streetscape, the remote analysis service consuming more computational resources to integrate the high-priority event into the map than are available at the vehicle;

detect that the vehicle has been turned on;

boot an operating system of the smartphone based on detecting that the vehicle has been turned on;

configure the smartphone such that a call functionality is disabled; and

detect that the vehicle has left a home base, the video of the streetscape captured triggered by the detection that the vehicle has left the home base.

2. The non-transitory processor readable medium of claim 1 , the code further

comprising code to cause the processor to:

receive, from the remote analysis service, the map of the streetscape including the high-priority event.

3. The non-transitory processor readable medium of claim 1 , wherein the code to

cause the processor to perform the first pass of computer vision analysis includes code to cause the first pass of computer vision analysis to be performed in real time on the video of the streetscape.

4. The non-transitory processor readable medium of claim 1 , wherein the processor is located within a housing of the smartphone.

5. The non-transitory processor readable medium of claim 1 , the code further comprising code to cause the processor to:

detect that the vehicle has returned to the home base;

cease capture of video based on detecting that the vehicle has returned to the home base;

connect to a Wifi network based on detecting that the vehicle has returned to the home base; and

upload video captured from the camera to the remote analysis service via the WiFi network.

6. The non-transitory processor readable medium of claim 1 , the code further comprising code to cause the processor to:

detect that the vehicle has returned to the home base;

connect to a Wifi network based on detecting that the vehicle has returned to the home base;

uploading video captured from the camera to the remote analysis service via the Wifi network;

detect that the vehicle has been turned off;

enter a power saving state after video captured from the camera has been uploaded to the remote analysis service, based on detecting that the vehicle has been turned off.

7. The non-transitory processor readable medium of claim 1 , the code further comprising code to cause the processor to:

capture a still image from the camera;

divide the still image into a plurality of regions;

calculate a Laplacian variance for at least a subset of regions from the plurality of regions; and

compare the Laplacian variances to a predetermined threshold and store the still image if Laplacian variance for a majority of the subset of regions is less than the predetermined threshold or discard the still image if at least one Laplacian variance calculated for at least one region from the plurality of regions is greater than the predetermined threshold.

8. The non-transitory processor readable medium of claim 1 , wherein performing the first pass of computer vision analysis includes scanning the video in real time for orange features to identify candidate traffic cones.

9. The non-transitory processor readable medium of claim 1 , wherein:

performing the first pass of computer vision analysis includes scanning the video in real time for orange features to identify candidate traffic cones; and

performing the second pass of computer vision analysis includes confirming that the candidate traffic cones are actual traffic cones.

10. The non-transitory processor readable medium of claim 1 , wherein:

performing the first pass of computer vision analysis includes scanning the video in real time for orange features to identify candidate traffic cones;

performing the second pass of computer vision analysis includes confirming that the candidate traffic cones are actual traffic cones; and

the map of the streetscape includes an indication of a lane closure corresponding to a presence of the actual traffic cones.

11. The non-transitory processor readable medium of claim 1 , wherein:

performing the first pass of computer vision analysis includes scanning the video in real time for flashing lights to identify candidate emergency vehicles; and

performing the second pass of computer vision analysis includes identifying stationary emergency vehicles.

12. The non-transitory processor readable medium of claim 1 , wherein:

the plurality of predefined characteristic signals are associate with a subset of frames of the video; and

the second pass of computer vision analysis is performed only on the subset of frames.

13. The non-transitory processor readable medium of claim 1 , wherein the first pass of computer vision analysis is performed in real time at a frame rate lower than a rate at which the video is captured.

14. The non-transitory processor readable medium of claim 1 , wherein the video of the streetscape includes a plurality of still images, the code further comprising code to cause the processor to:

divide a still image from the plurality of still images into a plurality of regions;

calculate a blur metric for at least a subset of regions from the plurality of regions;

compare the blur metric for each of the subset of regions to a predetermined threshold; and

perform the first pass of computer vision analysis on the still image based on the blur metric for a majority of the subset of regions from the plurality of regions being above a predetermined threshold, at least one region from the subset of regions from the plurality of regions having a blur metric below the predetermined threshold.

15. A non-transitory processor readable medium storing code configured to be executed by a processor, the code comprising code to cause the processor to:

capture, from a camera of a smartphone disposed with a vehicle, a video of a streetscape, the camera communicatively coupled to the processor, which is also disposed with the vehicle;

perform, locally, at the processor, a first pass of computer vision analysis on the video of the streetscape to identify a plurality of predefined characteristic signals, each predefined characteristic signal from the plurality of predefined characteristic signals indicating a candidate high-priority event from a plurality of candidate high-priority events;

perform, locally, at the processor, a second pass of computer vision analysis on the plurality of candidates of high-priority events to identify a high-priority event, the second pass of computer vision analysis consuming more computational resources than the first pass of computer vision analysis such that the second pass of computer vision analysis cannot be performed on the video in real time;

transmit, over a wireless data network and to a remote analysis service, an indication of the high-priority event, such that the remote analysis service can integrate the high-priority event into a map of the streetscape, the remote analysis service consuming more computational resources to integrate the high-priority event into the map than are available at the vehicle;

detect that the vehicle has been turned on;

boot an operating system of the smartphone based on detecting that the vehicle has been turned on;

configure the smartphone such that a call functionality is disabled; and

configure the smartphone such that the smartphone periodically contacts the remote analysis service for instructions.

Assignments (5)
CHANGE OF NAME Recorded Jun 22, 2023
From: WOVEN PLANET NORTH AMERICA, INC.
To: WOVEN BY TOYOTA, U.S., INC.
Reel/Frame 064065/0601 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 22, 2022
From: CARMERA, LLC
To: WOVEN PLANET NORTH AMERICA, INC.
Reel/Frame 060857/0207 →
ENTITY CONVERSION Recorded Aug 22, 2022
From: CARMERA, INC.
To: CARMERA, LLC
Reel/Frame 061296/0032 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 1, 2021
From: SORRELGREEN, ETHAN; DINH, HUONG
To: CARMERA, INC.
Reel/Frame 055102/0599 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 1, 2021
From: GUPTA, RO; DAY, JUSTIN; NICHOLL, CHASE; HENSTELL, MAX; STAMOS, IOANNIS; BOYLE, DAVID
To: CARMERA, INC.
Reel/Frame 055103/0419 →
Continuity (2)
Provisional Application 62513056 · May 31, 2017
Related Publication 20180349715A1 · Dec 6, 2018
Cited By (1)
US 12,347,208