Automating vehicle damage inspection using claims photos
In an implementation, a set of images of a vehicle, is captured as a captured set of images, where the set of images are captured from different viewpoints around the vehicle. The captured set of images are organized. The captured set of images are mapped, as a mapped image set, using an image classification model. The mapped image is aligned, as an aligned image set, onto a three-dimensional vehicle model for the vehicle. Image-level damage detection is performed on each image in the aligned image set to estimate a damage probability for each pixel in each image. Part damage and severity is predicted for each image.
1 . A computer-implemented method, comprising:
capturing, as a captured set of images, a set of images of a vehicle, wherein the set of images are captured from different viewpoints around the vehicle;
organizing the captured set of images into a first category and a second category using an image classification model, such that a first subset of the captured set of images is assigned to the first category and a second subset of the captured set of images is assigned to the second category;
mapping, as a mapped image set, the first subset of the captured set of images to the second subset of the captured set of images;
aligning, as an aligned image set, the mapped image set onto a three-dimensional vehicle model for the vehicle;
performing image-level damage detection on each image in the aligned image set at least in part by estimating a damage probability for each pixel in each image; and
predicting part damage and severity for each image.
2 . The computer-implemented method of claim 1 , wherein the first and second categories include: 1) Zoomed-out; and 2) Zoomed-in.
3 . The computer-implemented method of claim 2 , wherein each Zoomed-in image is mapped to a determined best matching Zoomed-out image using an image warping transformation.
4 . The computer-implemented method of claim 1 , wherein the aligning the mapped image set onto a three-dimensional vehicle model for the vehicle, comprises:
discretizing a surface of the three-dimensional vehicle model, wherein each discretized point is referred to as a voxel;
learning a high-dimensional embedding to represent each voxel and each image pixel, wherein a high similarity score in an embedding space yields a pixel-voxel correspondence;
determining an optimal camera pose using at least the pixel-voxel correspondence; and
determining a mapping between each image pixel and each voxel using ray tracing with the optimal camera pose.
5 . The computer-implemented method of claim 1 , wherein aligning, as an aligned image set, the mapped image set onto a three-dimensional vehicle model for the vehicle yields a part correspondence for each pixel in each image of the aligned image set, and wherein a deep neural network model is trained to estimate the damage probability for each pixel in each image.
6 . The computer-implemented method of claim 5 , wherein predicting part damage and severity for each image comprises estimating damage of a part using the damage probability and a part annotation.
7 . The computer-implemented method of claim 6 , wherein the part annotation comprises:
rendering, as rendered images, the three-dimensional vehicle model from different viewpoints around the vehicle;
using a model for part-segmentation to obtain parts corresponding to pixels of the three-dimensional vehicle model; and
determining a part annotation of voxels of the three-dimensional vehicle model using the parts corresponding to each pixel of each rendered image of the rendered images and ray tracing.
8 . A non-transitory, computer-readable medium storing one or more instructions executable by a computer system to perform operations, comprising:
capturing, as a captured set of images, a set of images of a vehicle, wherein the set of images are captured from different viewpoints around the vehicle;
organizing the captured set of images into a first category and a second category using an image classification model, such that a first subset of the captured set of images is assigned to the first category and a second subset of the captured set of images is assigned to the second category;
mapping, as a mapped image set, the first subset of the captured set of images to the second subset of the captured set of images
aligning, as an aligned image set, the mapped image set onto a three-dimensional vehicle model for the vehicle;
performing image-level damage detection on each image in the aligned image set at least in part by estimating a damage probability for each pixel in each image; and
predicting part damage and severity for each image.
9 . The non-transitory, computer-readable medium of claim 8 , wherein the first and second categories include: 1) Zoomed-out; and 2) Zoomed-in.
10 . The non-transitory, computer-readable medium of claim 9 , wherein each Zoomed-in image is mapped to a determined best matching Zoomed-out image using an image warping transformation.
11 . The non-transitory, computer-readable medium of claim 8 , wherein the aligning the mapped image set onto a three-dimensional vehicle model for the vehicle, comprises:
discretizing a surface of the three-dimensional vehicle model, wherein each discretized point is referred to as a voxel;
learning a high-dimensional embedding to represent each voxel and each image pixel, wherein a high similarity score in an embedding space yields a pixel-voxel correspondence;
determining an optimal camera pose using at least the pixel-voxel correspondence; and
determining a mapping between each image pixel and each voxel using ray tracing with the optimal camera pose.
12 . The non-transitory, computer-readable medium of claim 8 , wherein aligning, as an aligned image set, the mapped image set onto a three-dimensional vehicle model for the vehicle yields a part correspondence for each pixel in each image of the aligned image set, and wherein a deep neural network model is trained to estimate the damage probability for each pixel in each image.
13 . The non-transitory, computer-readable medium of claim 12 , wherein predicting part damage and severity for each image comprises estimating damage of a part using the damage probability and a part annotation.
14 . The non-transitory, computer-readable medium of claim 13 , wherein the part annotation comprises:
rendering, as rendered images, the three-dimensional vehicle model from different viewpoints around the vehicle;
using a model for part-segmentation to obtain parts corresponding to pixels of the three-dimensional vehicle model; and
determining a part annotation of voxels of the three-dimensional vehicle model using the parts corresponding to each pixel of each rendered image of the rendered images and ray tracing.
15 . A computer-implemented system, comprising:
one or more computers; and
one or more computer memory devices interoperably coupled with the one or more computers and having tangible, non-transitory, machine-readable media storing one or more instructions that, when executed by the one or more computers, perform one or more operations, comprising:
capturing, as a captured set of images, a set of images of a vehicle, wherein the set of images are captured from different viewpoints around the vehicle;
organizing the captured set of images into a first category and a second category using an image classification model, such that a first subset of the captured set of images is assigned to the first category and a second subset of the captured set of images is assigned to the second category;
mapping, as a mapped image set, the first subset of the captured set of images to the second subset of the captured set of images
aligning, as an aligned image set, the mapped image set onto a three-dimensional vehicle model for the vehicle;
performing image-level damage detection on each image in the aligned image set at least in part by estimating a damage probability for each pixel in each image; and
predicting part damage and severity for each image.
16 . The computer-implemented system of claim 15 , wherein the first and second categories include: 1) Zoomed-out; and 2) Zoomed-in.
17 . The computer-implemented system of claim 16 , wherein each Zoomed-in image is mapped to a determined best matching Zoomed-out image using an image warping transformation.
18 . The computer-implemented system of claim 15 , wherein the aligning the mapped image set onto a three-dimensional vehicle model for the vehicle, comprises:
discretizing a surface of the three-dimensional vehicle model, wherein each discretized point is referred to as a voxel;
learning a high-dimensional embedding to represent each voxel and each image pixel, wherein a high similarity score in an embedding space yields a pixel-voxel correspondence;
determining an optimal camera pose using at least the pixel-voxel correspondence; and
determining a mapping between each image pixel and each voxel using ray tracing with the optimal camera pose.
19 . The computer-implemented system of claim 15 , wherein aligning, as an aligned image set, the mapped image set onto a three-dimensional vehicle model for the vehicle yields a part correspondence for each pixel in each image of the aligned image set, and wherein a deep neural network model is trained to estimate the damage probability for each pixel in each image.
20 . The computer-implemented system of claim 19 , wherein predicting part damage and severity for each image comprises estimating damage of a part using the damage probability and a part annotation.