Granted Patent B2

US 12,586,313 · App. 18/633,446 · Granted Mar 24, 2026

Inventors: Stefan Johannes Josef Holzer (San Mateo, CA); Abhishek Kar (Berkeley, CA); Matteo Munaro (San Francisco, CA); Pavel Hanchar (Minsk, BY); Radu Bogdan Rusu (San Francisco, CA); Santi Arano (San Francisco, CA)

Assignee: Fyusion, Inc.

G06T17/00G01C21/32G06F9/453G06F16/29G06F17/18G06F30/15G06N3/02G06Q30/0278G06T7/0002G06T7/0004G06T7/001G06T7/593G06T7/70G06T15/10G06T15/205G06T19/003G06T19/006G06V10/82G06V20/64H04N13/243H04N13/271H04N23/633G06T2200/08G06T2200/24G06T2207/10016G06T2207/10028G06T2207/20076G06T2207/20081G06T2207/20084G06T2207/20224G06T2207/30108G06T2207/30244G06T2207/30248H04N2013/0081

Abstract

Claims (44)

1 . A method comprising:

determining an object model by applying a neural network to a first plurality of images, each of the first plurality of images being captured from a respective viewpoint, the object model including a plurality of object model components, each of the images corresponding to one or more of the object model components, each of the object model components corresponding to a respective portion of the designated object, wherein the object model is associated with calibration data;

determining respective component condition information for one or more of the object model components using the plurality of images, the component condition information indicating a characteristic of damage incurred by the respective object portion corresponding with the object model component, wherein a statistical estimate of damage triggers capture of additional images;

converting the characteristic of damage into a heatmap visual representation associated with estimated damage to the respective object portion, wherein the heatmap shows the probability and the severity of damage to the respective object portion;

overlaying the heatmap onto a Multi-View Interactive Digital Media Representation (MVIDMR) associated with the designated object; and

storing the component condition information on a storage device.

2 . The method of claim 1 , wherein the designated object is a vehicle.

3 . The method of claim 1 , wherein the characteristic of damage includes an estimated probability of damage to the respective object portion and an estimated severity of damage to the respective object portion.

4 . The method of claim 1 , wherein the MVIDMR is generated using the first plurality of images.

5 . The method recited in claim 1 , the method further comprising:

determining aggregated object condition information based on the component condition information, the aggregated object condition information indicating damage to the object as a whole.

6 . The method recited in claim 5 , the method further comprising:

based on the aggregated object condition information, determining a standard view of the object that includes a visual representation of damage to the object.

7 . The method recited in claim 6 , wherein the standard view of the object is selected from the group consisting of: a top-down view of the object, a multi-view representation of the object navigable in one or more directions, and a three-dimensional model of the object.

8 . The method recited in claim 1 , wherein determining the object model includes estimating pose information for a designated one of the plurality of images, the pose information including a location and angle of a camera with respect to the designated object for the designated image.

9 . The method recited in claim 8 , wherein determining the object model includes calibration data having a neural network trained to identify damage.

10 . The method recited in claim 9 , wherein the plurality of object model components are components of a vehicle.

11 . The method recited in claim 1 , wherein a designated one of the plurality of object model components corresponds with a designated subset of the images and a designated portion of the object, the method further comprising:

constructing a multi-view representation of the designated portion of the object at the handheld recording device based on the designated subset of the images, the multi-view representation being navigable in one or more directions.

12 . The method recited in claim 1 , wherein determining the component condition information comprises:

applying a neural network to a subset of the images corresponding with the respective object model component.

13 . The method recited in claim 12 , wherein the neural network receives as an input depth information captured from a depth sensor at the handheld recording device.

14 . The method recited in claim 12 , wherein determining the component condition information further comprises:

aggregating neural network results computed for separate images corresponding with the respective object model component.

15 . The method recited in claim 12 , the method further comprising:

constructing a multi-view representation of the designated object at the handheld recording device based on the plurality of images, the multi-view representation being navigable in one or more directions.

16 . The method recited in claim 12 , wherein the plurality of object model components include each of a left vehicle door, a right vehicle door, and a windshield.

17 . The method recited in claim 12 , determining that the plurality of images exhibit a coverage level that is beneath a designated threshold; and

providing a recording guidance instruction to capture additional image data.

18 . The method recited in claim 17 , wherein the plurality of images are captured at a first level of clarity or detail, and wherein the recording guidance instruction indicates a second level of clarity or detail.

19 . One or more non-transitory computer readable media having instructions stored thereon for performing a method, the method comprising:

determining an object model by applying a neural network to a first plurality of images, each of the first plurality of images being captured from a respective viewpoint, the object model including a plurality of object model components, each of the images corresponding to one or more of the object model components, each of the object model components corresponding to a respective portion of the designated object, wherein the object model is associated with calibration data;

determining respective component condition information for one or more of the object model components using the plurality of images, the component condition information indicating a characteristic of damage incurred by the respective object portion corresponding with the object model component, wherein a statistical estimate of damage triggers capture of additional images;

converting the characteristic of damage into a heatmap visual representation associated with estimated damage to the respective object portion, wherein the heatmap shows the probability and the severity of damage to the respective object portion;

overlaying the heatmap onto a Multi-View Interactive Digital Media Representation (MVIDMR) associated with the designated object; and

storing the component condition information on a storage device.

20 . A mobile computing device comprising:

a camera configured to capture a video of a designated object as it moves through space around the designated object;

a processor configured to:

determine an object model by applying a neural network to a first plurality of images, each of the first plurality of images being captured from a respective viewpoint, the object model including a plurality of object model components, each of the images corresponding to one or more of the object model components, each of the object model components corresponding to a respective portion of the designated object, wherein the object model is associated with calibration data;

determine respective component condition information for one or more of the object model components using the plurality of images, the component condition information indicating a characteristic of damage incurred by the respective object portion corresponding with the object model component, wherein a statistical estimate of damage triggers capture of additional images;

convert the characteristic of damage into a heatmap visual representation associated with estimated damage to the respective object portion, wherein the heatmap shows the probability and the severity of damage to the respective object portion; and

overlay the heatmap onto a Multi-View Interactive Digital Media Representation (MVIDMR) associated with the designated object; and

a storage device configured to store the component condition information.

Assignments (1)

Continuity (5)

References Cited (280)