Granted Patent B2

US 12,450,784 · App. 17/650,201 · Granted Oct 21, 2025

Inventors: Takahiro Koguchi (Kariya, JP); Yousuke Hattori (Kariya, JP); Yusuke Sekikawa (Tokyo, JP)

Assignee: DENSO CORPORATION

G06T11/00G06T7/50G06T2207/10016G06T2207/10028G06T2207/20081G06T2207/20084G06T2207/30252

Abstract

Claims (57)

1. A peripheral video generation device comprising:

at least one memory having a set of computer-executable instructions stored thereon;

at least one processor communicable with the memory and a plurality of cameras; and

the set of computer-executable instructions stored on the memory cause the processor to implement:

inputting peripheral video data captured by the plurality of cameras;

compositing the peripheral video data to generate a composite video as viewed from a predetermined viewpoint;

estimating a three-dimensional shape of a peripheral object existing around the periphery of a vehicle to be parked based on the peripheral video data;

using an estimation result of the three-dimensional shape to estimate a shielded area which is shielded by the peripheral object of the estimated three-dimensional shape and is not visible from the predetermined viewpoint in the composite video;

inferring a video of the shielded area using deep learning, wherein:

the video of the shielded area is inferred using a generator and a discriminator;

the generator comprising a first neural network model configured to generate a false image intended to fool the discriminator; and

the discriminator comprising a second neural network model configured to discriminate between the false image generated by the generator and a correct image corresponding to true data,

wherein:

the generator is subjected to learning with updating of parameters of the generator such that the discriminator discriminates the false image as the correct image, the video of the shielded area is inferred using the correct image; and

the discriminator is subjected to learning with updating of parameters of the discriminator such that there is a large difference in output between when the discriminator is supplied the true data and when the discriminator is supplied the false image generated by the generator;

generating the correct image corresponding to the true data as the generator by alternately repeating the learning of the generator and the learning of the discriminator;

the video of the shielded area is inferred using the generator after learning, which can generate the correct image, and is generated using the correct image inferred by the generator; and

superimposing the video of the shielded area in the composite video, wherein

the processor transmits a control signal to the vehicle to automatically park the vehicle based on the composite video superimposed the video inferred and generated on the shielded area.

2. The peripheral video generation device according to claim 1 , wherein

the plurality of cameras is mounted in a vehicle, and

the set of computer-executable instructions further cause the processor to implement:

compositing a top-view video as viewed from above the vehicle.

3. The peripheral video generation device according to claim 1 , wherein the set of computer-executable instructions further cause the processor to implement:

superimposing the video inferred and generating on the shielded area in a display mode different from a display mode of the composite video.

4. A peripheral video generation method comprising:

inputting peripheral video data captured by a plurality of cameras;

compositing the peripheral video data to generate a composite video as viewed from a predetermined viewpoint;

estimating a three-dimensional shape of a peripheral object existing around the periphery of a vehicle to be parked based on the peripheral video data;

using an estimation result of the three-dimensional shape to estimate a shielded area which is shielded by the peripheral object of the estimated three-dimensional shape and is not visible from the predetermined viewpoint in the composite video;

inferring a video of the shielded area using deep learning, wherein:

the video of the shielded area is inferred using a generator and a discriminator;

the generator comprising a first neural network model configured to generate a false image intended to fool the discriminator; and

the discriminator comprising a second neural network model configured to discriminate between the false image generated by the generator and a correct image corresponding to true data,

wherein:

the generator is subjected to learning with updating of parameters of the generator such that the discriminator discriminates the false image as the correct image, the video of the shielded area is inferred using the correct image; and

the discriminator is subjected to learning with updating of parameters of the discriminator such that there is a large difference in output between when the discriminator is supplied the true data and when the discriminator is supplied the false image generated by the generator;

generating the correct image corresponding to the true data as the generator by alternately repeating the learning of the generator and the learning of the discriminator;

the video of the shielded area is inferred using the generator after learning, which can generate the correct image, and is generated using the correct image inferred by the generator; and

superimposing the inferred and generated video on the shielded area in the composite video, wherein

the vehicle is controlled to automatically park based on the composite video superimposed the video inferred and generated on the shielded area.

5. A non-transitory computer-readable storage medium containing thereon a program comprising instructions configured to cause a processor to execute a peripheral video generation process, the instructions comprising:

inputting peripheral video data captured by a plurality of cameras;

compositing the peripheral video data to generate a composite video as viewed from a predetermined viewpoint;

estimating a three-dimensional shape of a peripheral object existing around the periphery of a vehicle to be parked based on the peripheral video data;

using an estimation result of the three-dimensional shape to estimate a shielded area which is shielded by the peripheral object of the estimated three-dimensional shape and is not visible from the predetermined viewpoint in the composite video;

inferring a video of the shielded area using deep learning, wherein:

the video of the shielded area is inferred using a generator and a discriminator;

the generator comprising a first neural network model configured to generate a false image intended to fool the discriminator; and

the discriminator comprising a second neural network model configured to discriminate between the false image generated by the generator and a correct image corresponding to true data,

wherein:

the generator is subjected to learning with updating of parameters of the generator such that the discriminator discriminates the false image as the correct image, the video of the shielded area is inferred using the correct image; and

the discriminator is subjected to learning with updating of parameters of the discriminator such that there is a large difference in output between when the discriminator is supplied the true data and when the discriminator is supplied the false image generated by the generator;

generating the correct image corresponding to the true data as the generator by alternately repeating the learning of the generator and the learning of the discriminator;

the video of the shielded area is inferred using the generator after learning, which can generate the correct image, and is generated using the correct image inferred by the generator; and

superimposing the inferred and generated video on the shielded area in the composite video, wherein

the vehicle is controlled to automatically park based on the composite video superimposed the video inferred and generated on the shielded area.

Assignments (1)

Priority Claims (1)

Continuity (2)

References Cited (10)