Granted Patent B2

US 10,204,448 · App. 15/805,107 · Granted Feb 12, 2019

Inventors: Aryan Hazeghi (Palo Alto, CA); Giuliano Pasqualotto (Mountain View, CA); Keith Blackstone (Pacifica, CA); Carlo Dal Mutto (Sunnyvale, CA); Abbas Rafii (Palo Alto, CA); Jason Trachewsky (Menlo Park, CA); Jackson Masters (Redwood City, CA)

Assignee: AQUIFI, INC.

G06T17/20G03B17/55H04N5/2252H04N5/33H04N9/07H04N13/243H05K7/20409G03B17/563

Abstract

Claims (129)

1. A method for generating a three-dimensional model of an object, by a scanning system comprising a client-side device comprising:

an acquisition system configured to capture images; and

an interaction system comprising a display device and a network interface, the method comprising:

capturing a plurality of images of the object by the acquisition system, the images being captured from a plurality of different poses of the acquisition system;

computing depth maps from the images of the objects, each of the depth maps corresponding to one of the poses of the acquisition system;

combining the depth maps to generate a combined point cloud; and

displaying, on the display device, the combined point cloud or a 3D mesh model generated from the combined point cloud,

wherein the scanning system further comprises an offline processing system comprising a processor and memory, and

wherein the computing the depth maps from the images of the objects and the combining the depth maps to generate the combined point cloud are distributed between the client-side device and the offline processing system in accordance with a plurality of configuration settings.

2. The method of claim 1 ,

wherein the method further comprises:

computing, by the offline processing system, the 3D mesh model from the combined point cloud; and

receiving, by the client side device via the network interface, the 3D mesh model from the offline processing system.

3. The method of claim 1 ,

wherein the computing the depth maps from the images of the objects and the combining the depth maps to generate the combined point cloud are performed by the client-side device, and

wherein the method further comprises transmitting the combined point cloud to the offline processing system.

4. The method of claim 1 ,

wherein the computing the depth maps from the images of the objects is performed by the client-side device,

wherein the method further comprises transmitting the depth maps from the client-side device to the offline processing system, and

wherein the combining the depth maps to generate the combined point cloud is performed by the offline processing system.

5. The method of claim 1 ,

wherein the method further comprises transmitting the plurality of images of the object from the client-side device to the offline processing system via the network interface of the interaction system, and

wherein the computing the depth maps from the images of the objects and the combining the depth maps to generate the combined point cloud are performed by the offline processing system.

6. The method of claim 1 , wherein the configuration settings are provided in a manifest file, and

wherein the configuration settings are specified in terms of at least one of: model quality, client processing, turn-around time, client battery life, and server processing.

7. The method of claim 6 , wherein the model quality comprises geometric detail and surface texture fidelity, the geometric detail being defined based on number of polygons in the mesh model.

8. The method of claim 1 , wherein the configuration settings are dynamically configured in accordance with a plurality of current conditions of the client-side device, the current conditions comprising power availability and network accessibility.

9. The method of claim 1 , further comprising computing, by the client device, a real-time preview of the scan from the images of the objects; and

displaying the real time preview on the display device.

10. A method for generating a three-dimensional model of an object, by a scanning system comprising a client-side device comprising:

an acquisition system configured to capture images; and

an interaction system comprising a display device and a network interface, the method comprising:

capturing a plurality of images of the object by the acquisition system, the images being captured from a plurality of different poses of the acquisition system;

computing depth maps from the images of the objects, each of the depth maps corresponding to one of the poses of the acquisition system;

combining the depth maps to generate a combined point cloud; and

displaying, on the display device, the combined point cloud or a 3D mesh model generated from the combined point cloud,

wherein the acquisition system comprises:

a first infrared camera having a first optical axis and a first field of view,

a second infrared camera spaced apart from the first infrared camera and having a second optical axis substantially parallel to the first optical axis and a second field of view overlapping the first field of view;

a color camera having an optical axis substantially parallel to the first optical axis and the second optical axis and having a field of view overlapping the first field of view and the second field of view; and

an infrared illuminator configured to illuminate a region within the first field of view of the first infrared camera and the second field of view of the second infrared camera,

wherein the acquisition system is configured to control the first infrared camera, the second infrared camera, and the color camera to capture images substantially simultaneously.

11. A method for generating a three-dimensional model of an object, by a scanning system comprising a client-side device comprising:

an acquisition system configured to capture images; and

an interaction system comprising a display device and a network interface, the method comprising:

capturing a plurality of images of the object by the acquisition system, the images being captured from a plurality of different poses of the acquisition system;

computing depth maps from the images of the objects, each of the depth maps corresponding to one of the poses of the acquisition system;

combining the depth maps to generate a combined point cloud; and

displaying, on the display device, the combined point cloud or a 3D mesh model generated from the combined point cloud,

wherein the client-side device comprises:

a housing;

a grip coupled to the housing; and

wherein the acquisition system is housed in the housing, the acquisition system comprising a support bracket, a backplate coupled to the support bracket, and a plurality of electronic components between the support bracket and the backplate, and

wherein relative positions of the plurality of electronic components are fixed by the support bracket and the backplate.

12. The method of claim 11 , wherein the acquisition system further comprises:

a plurality of cooling fins on an outer surface of the backplate facing away from the support bracket;

a plurality of cooling fins on the support bracket; and

a plurality of thermal ports in the support bracket.

13. A three-dimensional (3D) scanning system comprising:

a client-side device comprising:

an acquisition system configured to capture images;

a processing system comprising a processor and memory; and

an interaction system comprising a display device and a network interface,

wherein the memory of the client-side device stores instructions that, when executed by the processor of the client-side device, cause the processing system to:

control the acquisition system to capture images of an object, the images being captured from a plurality of different poses of the acquisition system;

compute real-time feedback from the images of the object;

display the real-time feedback on the display device; and

display, on the display device, a 3D mesh model generated from the images,

further comprising an offline processing system comprising a processor and memory,

wherein the client-side device is configured to distribute, in accordance with a plurality of configuration settings, a computation pipeline for computing the 3D mesh model from the images of the object between the client-side device and the offline processing system,

wherein the computation pipeline comprises:

computing a plurality of depth maps from the images of the objects;

combining the depth maps to generate a combined point cloud; and

computing the 3D mesh model from the combined point cloud.

14. The system of claim 13 , wherein the memory of the client-side device further stores instructions that, when executed by the processor of the client-side device, cause the processor of the client-side device to compute the real-time feedback by:

computing a plurality of depth maps from the images;

decimating the depth maps; and

combining the depth maps to generate a preview point cloud, and

wherein the real-time feedback comprises the preview point cloud.

15. The system of claim 13 , wherein the memory of the client-side device further stores instructions that, when executed by the processor of the client-side device, cause the processor of the client-side device to:

transmit the images of the object to the offline processing system; and

receive, from the offline processing system, the 3D mesh model generated from the images of the object.

16. The system of claim 13 , wherein the memory of the client-side device further stores instructions that, when executed by the processor of the client-side device, cause the processor of the client-side device to:

compute a plurality of depth maps from the images of the objects;

transmit the depth maps of the object to the offline processing system; and

receive, from the offline processing system, the 3D mesh model generated from the depth maps of the object.

17. The system of claim 13 , wherein the memory of the client-side device further stores instructions that, when executed by the processor of the client-side device, cause the processor of the client-side device to:

compute a plurality of depth maps from the images of the objects;

combining the depth maps to generate a combined point cloud;

transmit the combined point cloud to the offline processing system; and

receive, from the offline processing system, the 3D mesh model generated from the combined point cloud.

18. The system of claim 13 , wherein the configuration settings are provided in a manifest file, and

wherein the configuration settings are specified in terms of at least one of: model quality, client processing, turn-around time, client battery life, and server processing.

19. The system of claim 18 , wherein the model quality comprises geometric detail and surface texture fidelity, the geometric detail being defined based on number of polygons in the mesh model.

20. The system of claim 13 , wherein the configuration settings are dynamically configured in accordance with a plurality of current conditions of the client-side device, the current conditions comprising power availability and network accessibility.

21. A three-dimensional (3D) scanning system comprising:

a client-side device comprising:

an acquisition system configured to capture images;

a processing system comprising a processor and memory; and

an interaction system comprising a display device and a network interface,

wherein the memory of the client-side device stores instructions that, when executed by the processor of the client-side device, cause the processing system to:

control the acquisition system to capture images of an object, the images being captured from a plurality of different poses of the acquisition system;

compute real-time feedback from the images of the object;

display the real-time feedback on the display device; and

display, on the display device, a 3D mesh model generated from the images,

wherein the acquisition system comprises:

a first infrared camera having a first optical axis and a first field of view,

a second infrared camera spaced apart from the first infrared camera and having a second optical axis substantially parallel to the first optical axis and a second field of view overlapping the first field of view;

a color camera having an optical axis substantially parallel to the first optical axis and the second optical axis and having a field of view overlapping the first field of view and the second field of view; and

an infrared illuminator configured to illuminate a region within the first field of view of the first infrared camera and the second field of view of the second infrared camera, wherein the acquisition system is configured to control the first infrared camera, the second infrared camera, and the color camera to capture images substantially simultaneously.

22. A three-dimensional (3D) scanning system comprising:

a client-side device comprising:

an acquisition system configured to capture images;

a processing system comprising a processor and memory; and

an interaction system comprising a display device and a network interface,

wherein the memory of the client-side device stores instructions that, when executed by the processor of the client-side device, cause the processing system to:

control the acquisition system to capture images of an object, the images being captured from a plurality of different poses of the acquisition system;

compute real-time feedback from the images of the object;

display the real-time feedback on the display device; and

display, on the display device, a 3D mesh model generated from the images,

wherein the client-side device comprises:

a housing;

a grip coupled to the housing; and

wherein the acquisition system is housed in the housing, the acquisition system comprising a support bracket, a backplate coupled to the support bracket, and a plurality of electronic components between the support bracket and the backplate, and

wherein relative positions of the plurality of electronic components are fixed by the support bracket and the backplate.

23. The system of claim 22 , wherein the acquisition system further comprises:

a plurality of cooling fins on an outer surface of the backplate facing away from the support bracket;

a plurality of cooling fins on the support bracket; and

a plurality of thermal ports in the support bracket.

Assignments (7)

Continuity (2)

Cited By (10)

US 12,190,916 US 12,261,990 US 12,380,634 US 12,381,995 US 12,432,327 US 12,495,134 US 12,525,045 US 12,541,933 US 12,657,238 US 12,665,986