IP Library Granted Patent US 11,069,134
Granted Patent B2
US 11,069,134 · App. 16/641,363 · Granted Jul 20, 2021

Methods, devices and computer program products for generation of mesh in constructed 3D images containing incomplete information

Inventors: Pal Szasz (Lund, SE); Johannes Elg (Helsingborg, SE); Fredrik Olofsson (Lund, SE); Lars Novak (Bjärred, SE); Fredrik Mattisson (Lund, SE)
Assignee: Sony Group Corporation
G06T17/20G06T19/20G06T2219/2021
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,069,134
App. No.
16/641,363
Granted
Jul 20, 2021
Kind
B2
Abstract

Methods, systems, and computer program products for removing unused portions of a 3D mesh representation of an object may include generating a first mesh representation of the object, the first mesh representation including a plurality of polygons, respective ones of the polygons including at least three vertices and at least three edges, wherein respective ones of the plurality of polygons are associated with a precision value that indicates an extent to which the respective ones of the plurality of polygons in the first mesh representation match the object, and adjusting the first mesh representation of the object to create a second mesh representation of the object by removing, front the first mesh representation, polygons of the plurality of polygons that are associated with precision values that have not been modified from an initial precision value.

Claims (65)

1. A method of creating a three-dimensional mesh representation of an object, the method comprising:

generating a first mesh representation of the object, the first mesh representation comprising a plurality of polygons, respective ones of the polygons comprising at least three vertices and at least three edges,

wherein the respective ones of the plurality of polygons are each associated with a precision value that indicates an extent to which the respective ones of the plurality of polygons in the first mesh representation match a surface of the object, and

wherein generating the first mesh representation of the object comprises modifying each precision value that is associated with one or more of the plurality of polygons from an initial precision value responsive to modifying the one or more of the plurality of polygons to match the surface of the object; and

adjusting the first mesh representation of the object to create a second mesh representation of the object by removing, from the first mesh representation, a first polygon of the plurality of polygons responsive to determining that the precision value that is associated with the first polygon has not been modified from the initial precision value.

2. The method of claim 1 , wherein generating the first mesh representation of the object further comprises:

setting the precision value associated with each of the plurality of polygons to the initial precision value;

receiving one or more images of a portion of the object;

identifying a plurality of points in the one or more images that correspond to the surface of the object;

associating the plurality of points with vertices of a first set of the plurality of polygons; and

modifying each precision value associated with the respective ones of the first set of the plurality of polygons responsive to modifying the first set of the plurality of polygons within the first mesh representation based on the plurality of points that are associated with the vertices of the first set of the plurality of polygons.

3. The method of claim 1 , wherein removing, from the first mesh representation, the first polygon of the plurality of polygons responsive to determining that the precision value that is associated with the first polygon has not been modified from the initial precision value comprises:

identifying a border in the first mesh representation between the vertices of a first group of the plurality of polygons whose precision values have not been modified from the initial precision value and the vertices of a second group of the plurality of polygons whose precision values have been modified from the initial precision value, wherein the first group of the plurality of polygons comprises the first polygon;

establishing a three-dimensional plane through the first mesh representation based on a location of the border; and

removing the first polygon of the first mesh representation on a side of the three-dimensional plane.

4. The method of claim 3 , wherein the three-dimensional plane through the first mesh representation at least partially intersects the border.

5. The method of claim 3 , wherein the border comprises a contiguously connected series of edges through vertices of the first group and the second group of the plurality of polygons.

6. The method of claim 3 , wherein establishing the three-dimensional plane through the first mesh representation comprises performing a random sample consensus (RANSAC) calculation on a plurality of vertices of the first group and the second group of the plurality of polygons that are on the border.

7. The method of claim 3 , wherein establishing the three-dimensional plane through the first mesh representation comprises repeating a plurality of times operations comprising:

randomly selecting at least three border vertices from a plurality of vertices of the first group and the second group of the plurality of polygons that are on the border;

constructing a temporary plane through the at least three border vertices; and

calculating a cumulative distance for respective ones of the plurality of vertices of the first group and the second group of the plurality of polygons that are on the border from the temporary plane; and

establishing the three-dimensional plane through the temporary plane that has a lowest calculated cumulative distance.

8. The method of claim 3 , wherein removing portions the first polygon of the first mesh representation on the side of the three-dimensional plane comprises removing the first polygon of the first mesh representation that is at least a first distance away from the three-dimensional plane in a line normal to the three-dimensional plane.

9. The method of claim 3 , wherein identifying the border in the first mesh representation between the vertices of the first group of the plurality of polygons whose precision values have not been modified from the initial precision value and the vertices of the second group of the plurality of polygons whose precision values have been modified from the initial precision value comprises:

identifying a contiguously connected series of edges through vertices of the first group and the second group of the plurality of polygons, wherein each edge of the contiguously connected series of edges comprises one vertex of a polygon of the second group whose precision value has been modified from the initial precision value and one vertex of a polygon of the first group whose precision value has not been modified from the initial precision value.

10. The method of claim 3 , wherein identifying the border in the first mesh representation between the vertices of the first group of the plurality of polygons whose precision values have not been modified from the initial precision value and the vertices of the second group of the plurality of polygons whose precision values have been modified from the initial precision value comprises:

repeatedly examining each respective edge of the plurality of polygons to identify border edges, wherein the border edges comprise one vertex of a polygon of the second group whose precision value has been modified from the initial precision value and one vertex of a polygon of the first group whose precision value has not been modified from the initial precision value; and

identifying a contiguously connected series of the border edges within the first mesh representation as the border.

11. An imaging system for processing images, the imaging system comprising:

a processor; and

a memory coupled to the processor and storing computer readable program code that when executed by the processor causes the processor to perform operations comprising:

generating, within the memory, a first mesh representation of an object, the first mesh representation comprising a plurality of polygons, respective ones of the polygons comprising at least three vertices and at least three edges,

wherein the respective ones of the plurality of polygons are each associated with a precision value that indicates an extent to which the respective ones of the plurality of polygons in the first mesh representation match a surface of the object, and

wherein generating the first mesh representation of the object comprises modifying each precision value that is associated with one or more of the plurality of polygons from an initial precision value responsive to modifying the one or more of the plurality of polygons to match the surface of the object; and

adjusting the first mesh representation of the object to create a second mesh representation of the object by removing, from the first mesh representation, polygons a first polygon of the plurality of polygons responsive to determining that the precision value that is associated with the first polygon has not been modified from the initial precision value.

12. The imaging system of claim 11 , wherein generating the first mesh representation of the object further comprises:

setting the precision value associated with each of the plurality of polygons to the initial precision value;

receiving one or more images of a portion of the object;

identifying a plurality of points in the one or more images that correspond to the surface of the object;

associating the plurality of points with vertices of a first set of the plurality of polygons; and

modifying each precision value associated with respective ones of the first set of the plurality of polygons responsive to modifying the first set of the plurality of polygons within the first mesh representation based on the plurality of points that are associated with the vertices of the first set of the plurality of polygons.

13. The imaging system of claim 11 , wherein the at least three vertices of the respective ones of the plurality of polygons comprise the precision value, and

wherein the respective ones of the plurality of polygons are associated with each of the precision values of the at least three vertices of the respective polygon.

14. The imaging system of claim 11 , wherein removing, from the first mesh representation, polygons the first polygon of the plurality of polygons responsive to determining that the precision value that is associated with the first polygon has not been modified from the initial precision value comprises:

identifying a border in the first mesh representation between the vertices of a first group of the plurality of polygons whose precision values have not been modified from the initial precision value and the vertices of a second group of the plurality of polygons whose precision values have been modified from the initial precision value, wherein the first group of the plurality of polygons comprises the first polygon;

establishing a three-dimensional plane through the first mesh representation based on a location of the border; and

removing the first polygon of the first mesh representation on a side of the three-dimensional plane.

15. The imaging system of claim 14 , wherein the border comprises a contiguously connected series of edges through vertices of the first group and the second group of the plurality of polygons.

16. The imaging system of claim 14 , wherein establishing the three-dimensional plane through the first mesh representation comprises repeating a plurality of times operations comprising:

randomly selecting at least three border vertices from a plurality of vertices of the first group and the second group of the plurality of polygons that are on the border;

constructing a temporary plane through the at least three border vertices; and

calculating a cumulative distance for respective ones of the plurality of vertices of the first group and the second group of the plurality of polygons that are on the border from the temporary plane; and

establishing the three-dimensional plane through the temporary plane that has a lowest calculated cumulative distance.

17. The imaging system of claim 14 , wherein removing the first polygon of the first mesh representation on the side of the three-dimensional plane comprises removing the first polygon of the first mesh representation that is at least a first distance away from the three-dimensional plane in a line normal to the three-dimensional plane.

18. The imaging system of claim 14 , wherein identifying the border in the first mesh representation between the vertices of the first group of the plurality of polygons whose precision values have not been modified from the initial precision value and the vertices of the second group of the plurality of polygons whose precision values have been modified from the initial precision value comprises:

identifying a contiguously connected series of edges through vertices of the first group and the second group of the plurality of polygons, wherein each edge of the contiguously connected series of edges comprises one vertex of a polygon of the second group whose precision value has been modified from the initial precision value and one vertex of a polygon of the first group whose precision value has not been modified from the initial precision value.

19. The imaging system of claim 14 , wherein identifying the border in the first mesh representation between the vertices of the first group of the plurality of polygons whose precision values have not been modified from the initial precision value and the vertices of the second group of the plurality of polygons whose precision values have been modified from the initial precision value comprises:

repeatedly examining each respective edge of the plurality of polygons to identify border edges, wherein the border edges comprise one vertex of a polygon of the second group whose precision value has been modified from the initial precision value and one vertex of a polygon of the first group whose precision value has not been modified from the initial precision value; and

identifying a contiguously connected series of the border edges within the first mesh representation as the border.

20. A computer program product for operating an imaging system, the computer program product comprising a non-transitory computer readable storage medium having computer readable program code embodied in the medium that when executed by a processor causes the processor to perform operations comprising:

generating a first mesh representation of an object, the first mesh representation comprising a plurality of polygons, respective ones of the polygons comprising at least three vertices and at least three edges,

wherein the respective ones of the plurality of polygons are each associated with a precision value that indicates an extent to which the respective ones of the plurality of polygons in the first mesh representation match a surface of the object, and

wherein generating the first mesh representation of the object comprises modifying each precision value that is associated with one or more of the plurality of polygons from an initial precision value responsive to modifying the one or more of the plurality of polygons to match the surface of the object; and

adjusting the first mesh representation of the object to create a second mesh representation of the object by removing, from the first mesh representation, a first polygon of the plurality of polygons responsive to determining that the precision value that is associated with the first polygon has not been modified from the initial precision value.

Assignments (3)
CHANGE OF NAME Recorded Jun 10, 2021
From: SONY CORPORATION
To: SONY GROUP CORPORATION
Reel/Frame 056539/0673 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 14, 2020
From: PZASZ, PAL; ELG, JOHANNES; OLOFSSON, FREDRIK; NOVAK, LARS; MATTISSON, FREDRIK
To: SONY MOBILE COMMUNICATIONS INC.
Reel/Frame 052393/0711 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 14, 2020
From: SONY MOBILE COMMUNICATIONS INC.
To: SONY CORPORATION
Reel/Frame 052393/0777 →
Continuity (1)
Related Publication 20200226829A1 · Jul 16, 2020