IP Library Granted Patent US 10,867,404
Granted Patent B2
US 10,867,404 · App. 16/116,417 · Granted Dec 15, 2020

Distance estimation using machine learning

Inventors: Rui Guo (Mountain View, CA); Babajide Ayinde (Louisville, TX); Hao Sun (St. Louis, MO); Kentaro Oguchi (Menlo Park, CA)
Assignee: TOYOTA JIDOSHA KABUSHIKI KAISHA
G06T7/74G06K9/00791G06K9/6262G06T5/008G06T7/50G06T2207/20081G06T2207/20084G06T2207/30256G06T2207/30261
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,867,404
App. No.
16/116,417
Granted
Dec 15, 2020
Kind
B2
Abstract

A method receives a captured image depicting image content including an object, the captured image being captured by an image sensor located at a sensor position; generates, using a trained first machine learning logic, a lighting-corrected image from an imitative simulation image depicting at least a portion of the image content of the captured image in a simulation style associated with an environment simulator; generates, using a trained second machine learning logic, a depth estimation image from the lighting-corrected image, the depth estimation image indicating a relative distance between the object depicted in the captured image and the sensor position of the image sensor; and determines an object position of the object depicted in the captured image based on the depth estimation image.

Claims (42)

1. A method comprising:

receiving a captured image depicting image content including an object, the captured image being captured by an image sensor located at a sensor position;

generating, using a trained first machine learning logic, a lighting-corrected image from an imitative simulation image depicting at least a portion of the image content of the captured image in a simulation style associated with an environment simulator;

generating, using a trained second machine learning logic, a depth estimation image from the lighting-corrected image, the depth estimation image indicating a relative distance between the object depicted in the captured image and the sensor position of the image sensor; and

determining an object position of the object depicted in the captured image based on the depth estimation image.

2. The method of claim 1 , wherein the image content of the captured image depicts a road segment and the object depicted in the captured image is a roadway object associated with the road segment.

3. The method of claim 1 , wherein generating the lighting-corrected image from the imitative simulation image includes:

detecting, using the trained first machine learning logic, one or more of a shadow region and a glare region in the imitative simulation image;

responsive to detecting the shadow region in the imitative simulation image, adjusting, using the trained first machine learning logic, the shadow region to reduce or remove a shadow in the shadow region of the imitative simulation image; and

responsive to detecting the glare region in the imitative simulation image, adjusting, using the trained first machine learning logic, the glare region to reduce or remove a glare in the glare region of the imitative simulation image.

4. The method of claim 1 , further comprising:

generating, using a trained third machine learning logic, the imitative simulation image from the captured image.

5. The method of claim 4 , wherein

the image content of the captured image depicts a real environment including the object;

the simulation style associated with the environment simulator includes a simulated color and a simulated texture generated by the environment simulator; and

generating the imitative simulation image from the captured image includes adjusting, using the trained third machine learning logic, one or more of a captured color of the object and a captured texture of the object to one or more of the simulated color and the simulated texture.

6. The method of claim 1 , wherein generating the depth estimation image from the lighting-corrected image includes:

mapping, using the trained second machine learning logic, one or more image pixels in the lighting-corrected image to one or more grayscale pixels in the depth estimation image.

7. The method of claim 1 , wherein

the trained first machine learning logic is a conditional-Generative Adversarial Network (GAN) and the trained second machine learning logic is another conditional-GAN; and the method includes

generating, using a trained third machine learning logic, the imitative simulation image from the captured image, the trained third machine learning logic being a cycle-GAN.

8. A system comprising:

one or more processors;

an environment simulator executable by the one or more processors to generate one or more simulation images;

a first machine learning logic executable by the one or more processors to receive a captured image depicting image content including an object, the captured image being captured by an image sensor located at a sensor position, and generate an imitative simulation image from the captured image, the imitative simulation image depicting at least a portion of the image content of the captured image in a simulation style associated with the environment simulator;

a second machine learning logic executable by the one or more processors to receive the imitative simulation image from the first machine learning logic, and generate a lighting-corrected image from the imitative simulation image;

a third machine learning logic executable by the one or more processors to receive the lighting-corrected image from the second machine learning logic, and generate a depth estimation image from the lighting-corrected image, the depth estimation image indicating a relative distance between the object depicted in the captured image and the sensor position of the image sensor; and

an object position calculator executable by the one or more processors to determine an object position of the object depicted in the captured image based on the depth estimation image.

9. The system of claim 8 , wherein the image content of the captured image depicts a road segment and the object depicted in the captured image is a roadway object associated with the road segment.

10. The system of claim 8 , wherein to generate the lighting-corrected image from the imitative simulation image, the second machine learning logic performs one or more operations including:

detecting one or more of a shadow region and a glare region in the imitative simulation image;

responsive to detecting the shadow region in the imitative simulation image, adjusting the shadow region to reduce or remove a shadow in the shadow region of the imitative simulation image; and

responsive to detecting the glare region in the imitative simulation image, adjusting the glare region to reduce or remove a glare in the glare region of the imitative simulation image.

11. The system of claim 8 , wherein

the image content of the captured image depicts a real environment including the object;

the simulation style associated with the environment simulator includes a simulated color and a simulated texture generated by the environment simulator; and

to generate the imitative simulation image from the captured image, the first machine learning logic performs one or more operations including adjusting one or more of a captured color of the object and a captured texture of the object to one or more of the simulated color and the simulated texture.

12. The system of claim 8 , wherein to generate the depth estimation image from the lighting-corrected image, the third machine learning logic performs one or more operations including:

mapping one or more image pixels in the lighting-corrected image to one or more grayscale pixels in the depth estimation image.

13. The system of claim 8 , wherein

the first machine learning logic is a cycle-Generative Adversarial Network (GAN); and

the second machine learning logic is a conditional-GAN and the third machine learning logic is another conditional-GAN.

Assignments (2)
CORRECTIVE ASSIGNMENT TO CORRECT THE CORRECT MISSING FILING DATE PREVIOUSLY RECORDED AT REEL: 047848 FRAME: 0060. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Aug 14, 2019
From: GUO, RUI; AYINDE, BABAJIDE; SUN, HAO; OGUCHI, KENTARO
To: TOYOTA JIDOSHA KABUSHIKI KAISHA
Reel/Frame 050054/0183 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 22, 2018
From: GUO, RUI; AYINDE, BABAJIDE; SUN, HAO; OGUCHI, KENTARO
To: TOYOTA JIDOSHA KABUSHIKI KAISHA
Reel/Frame 047848/0060 →
Continuity (1)
Related Publication 20200074674A1 · Mar 5, 2020