Vehicle location calculation apparatus and vehicle location calculation method
View Patent ↗A vehicle location calculation apparatus includes: a model learning part configured to perform learning to output an invisible keypoint set in a model image in which each vehicle is modeled, based on a visible keypoint set in the model image; and a dataset calculation part configured to generate a dataset including a visible keypoint and an invisible keypoint of a target vehicle, by inputting the visible keypoint of the target vehicle to the model learning part so that the invisible keypoint of the target vehicle is output, the visible keypoint of the target vehicle being detected in an image of the target vehicle while driving.
1 . A vehicle location calculation apparatus, comprising:
at least one processor; and
at least one memory operably coupled with the at least one processor and storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:
performing learning of a model to output an invisible keypoint set in a model image in which each vehicle is modeled, based on a visible keypoint set in the model image; and
generating a dataset including a visible keypoint and an invisible keypoint of a target vehicle, by inputting the visible keypoint of the target vehicle to the model so that the invisible keypoint of the target vehicle is output, the visible keypoint of the target vehicle being detected in an image of the target vehicle while driving,
wherein the performing learning of the model comprises:
changing coordinate values of the visible keypoint of the modeled vehicle based on coordinate values of a visible keypoint of an image of an actual vehicle; and
inputting the changed coordinate values of the visible keypoint as training data into the model,
wherein the model is trained to output coordinate values of the invisible keypoint of the modeled vehicle by using the changed coordinate values as an input.
2 . The vehicle location calculation apparatus of claim 1 , wherein the coordinate values of the visible keypoint of the modeled vehicle are adjusted based on a Gaussian probability distribution according to distance to match the coordinate values of a corresponding visible keypoint in the actual vehicle image.
3 . The vehicle location calculation apparatus of claim 1 , wherein the performing learning of the model comprises performing learning of the visible keypoint in the image of the actual vehicle.
4 . The vehicle location calculation apparatus of claim 3 ,
wherein the operations further comprise detecting the visible keypoint of the target vehicle based on data regarding the visible keypoint learned by a keypoint learning part.
5 . The vehicle location calculation apparatus of claim 1 ,
wherein the operations further comprise determining spatial coordinates of the target vehicle, based on the dataset including the visible keypoint and the invisible keypoint of the target vehicle.
6 . The vehicle location calculation apparatus of claim 5 ,
wherein determining the spatial coordinates of the target vehicle comprises:
determining three-dimensional (3D) camera coordinate values of keypoints of front and rear wheels on first and second sides of the target vehicle, by use of two-dimensional (2D) image coordinate values of the keypoints of the front and rear wheels on the first and second sides of the target vehicle and an inverse matrix of intrinsic and extrinsic parameters of a camera, a height of the keypoints of the front and rear wheels on the first and second sides of the target vehicle being set as 0; and
determining an angle between an x-axis and a vector connecting a center point of the keypoints of the front and rear wheels on the first and second sides of the target vehicle and a center point of keypoints of the first and second front wheels of the target vehicle.
7 . The vehicle location calculation apparatus of claim 6 ,
wherein determining the spatial coordinates of the target vehicle comprises:
determining unknown values including a distance between first and second bumpers of the target vehicle and the first and second front wheels of the target vehicle, a height of the first and second bumpers from ground, and location values of the first and second bumpers disposed between the first and second front wheels, by use of 3D world coordinate values of the keypoints of the first and second front wheels, 2D image coordinate values of the first and second bumpers, the inverse matrix of the intrinsic and extrinsic parameters of the camera, and the angle, and
determining 3D camera coordinate values of the first and second bumpers based on the determined unknown values and 3D camera coordinate values of the keypoints of the first and second front wheels.
8 . The vehicle location calculation apparatus of claim 1 , wherein the operations further comprise
setting the visible keypoint and the invisible keypoint in the model image.
9 . The vehicle location calculation apparatus of claim 8 , wherein the operations further comprise:
placing the modeled vehicle in a 3D synthetic world;
projecting 3D keypoint coordinates of the modeled vehicle onto a plane; and
confirming the visible keypoint and the invisible keypoint of the modeled vehicle to perform labeling.
10 . A vehicle location calculation method, comprising:
setting, by a controller, a visible keypoint and an invisible keypoint in a model image in which each vehicle is modeled;
performing, by the controller, learning of a model to output the invisible keypoint based on the visible keypoint; and
generating, by the controller, a dataset including a visible keypoint and an invisible keypoint of a target vehicle, by inputting the visible keypoint of the target vehicle to the model so that the invisible keypoint of the target vehicle is output, the visible keypoint of the target vehicle being detected in an image of the target vehicle while driving,
wherein the performing learning of the model comprises:
changing coordinate values of the visible keypoint of the modeled vehicle based on coordinate values of a visible keypoint of an image of an actual vehicle; and
inputting the changed coordinate values of the visible keypoint as training data into the model,
wherein the model is trained to output coordinate values of the invisible keypoint of the modeled vehicle by using the changed coordinate values as an input.
11 . The vehicle location calculation method of claim 10 , wherein the setting includes:
setting a plurality of keypoint locations in the model image;
placing the modeled vehicle in a 3D synthetic world;
projecting 3D keypoint coordinates of the modeled vehicle onto a plane; and
confirming the visible keypoint and the invisible keypoint of the modeled vehicle to perform labeling.
12 . The vehicle location calculation method of claim 10 ,
wherein the coordinate values of the visible keypoint of the modeled vehicle are adjusted based on a Gaussian probability distribution according to distance to match the coordinate values of a corresponding visible keypoint in the actual vehicle image.
13 . The vehicle location calculation method of claim 10 , wherein the generating of the dataset includes learning the visible keypoint in the image of the actual vehicle.
14 . The vehicle location calculation method of claim 13 , wherein the generating of the dataset further includes detecting the visible keypoint of the target vehicle based on data regarding the learned visible keypoint.
15 . The vehicle location calculation method of claim 10 , further including:
determining spatial coordinates of the target vehicle, based on the dataset including the visible keypoint and the invisible keypoint of the target vehicle.
16 . The vehicle location calculation method of claim 15 , wherein the determining of the spatial coordinates includes:
determining 3D camera coordinate values of keypoints of front and rear wheels on first and second sides of the target vehicle, by use of 2D image coordinate values of the keypoints of the front and rear wheels on the first and second sides of the target vehicle and an inverse matrix of intrinsic and extrinsic parameters of a camera, a height of the keypoints of the front and rear wheels on the first and second sides of the target vehicle being set as 0; and
determining an angle between an x-axis and a vector connecting a center point of the keypoints of the front and rear wheels on the first and second sides of the target vehicle and a center point of keypoints of the first and second front wheels of the target vehicle.
17 . The vehicle location calculation method of claim 16 , wherein the determining of the spatial coordinates further includes:
determining unknown values including a distance between first and second bumpers of the target vehicle and the first and second front wheels of the target vehicle, a height of the first and second bumpers from ground, and location values of the first and second bumpers disposed between the first and second front wheels, by use of 3D world coordinate values of the keypoints of the first and second front wheels, 2D image coordinate values of the first and second bumpers, the inverse matrix of the intrinsic and extrinsic parameters of the camera, and the angle; and
determining 3D camera coordinate values of the first and second bumpers based on the determined unknown values and 3D camera coordinate values of the keypoints of the first and second front wheels.
18 . The vehicle location calculation method of claim 10 , wherein the controller includes:
a processor; and
a non-transitory storage medium on which a program for performing the vehicle location calculation method of claim 10 and for being executed by the processor is recorded.