System and method for optimizing dynamic point clouds based on prioritized transformations
View Patent ↗An example disclosed method includes (i) transmitting a first point cloud to a client, wherein the first point cloud corresponds to a reference point cloud, (ii) receiving a second point cloud, and (iii) hierarchically determining changes in the second point cloud from the reference point cloud, wherein hierarchically determining the changes includes (a) identifying first areas in the second point cloud that have changed from the reference point cloud, and (b) for a first area having a highest priority, determining a first rigid 3D transformation that approximates a first change from the reference point cloud, and if the first rigid 3D transformation cannot be determined, further determining first points to be used to modify the reference point cloud, wherein the first points are representative of the first change.
1. A method, comprising:
obtaining a first point cloud in a time sequence of point clouds, the first point cloud corresponding to a first time in the time sequence of point clouds;
obtaining a second point cloud in the time sequence of point clouds, the second point cloud corresponding to a later time in the time sequence of point clouds than the first point cloud;
determining areas of deviation between the first point cloud and the second point cloud;
selecting a first area of deviation from among the determined areas, wherein selecting the first area of deviation is based on respective importance scores;
generating a representation of the changes between the first point cloud and the second point cloud in the first area of deviation, wherein the representation of the changes comprises at least one of a 3D transformation, a set of points to be added, and a set of points to be removed;
updating the first point cloud using the representation;
repeating the determining, the selecting, the generating, and the updating steps to obtain a series of representations of the changes between the first point cloud and the second point cloud; and
transmitting the series of representations of the changes to a client device.
2. The method of claim 1 ,
wherein the generated representation comprises a first representation that captures coarse changes, and
a second representation that captures finer residual changes.
3. The method of claim 1 , wherein the first point cloud and the second point cloud are obtained from one or more 3D sensors.
4. The method of claim 1 , wherein the time sequence of point clouds is accessed from a storage medium.
5. The method of claim 1 , wherein the 3D transformation comprises rotation and translation.
6. The method of claim 1 , wherein the 3D transformation is represented using a transformation matrix.
7. The method of claim 1 , wherein the areas of deviation between the first point cloud and the second point cloud are determined using a clustering technique.
8. The method of claim 1 , wherein an importance score for an area of deviation reflects a visual importance of the changes between first and second point clouds in the area of deviation.
9. The method of claim 1 , further comprising
determining a budget for representing the changes between the first point cloud and the second point cloud; and
repeating the determining, selecting, generating and updating steps until the budget is exhausted.
10. The method of claim 9 , wherein the budget comprises at least one of an amount of time, an amount of processing, or an amount of bandwidth available to update the first point cloud based on the changes.
11. A method, comprising:
obtaining a first point cloud in a time sequence of point clouds, the first point cloud corresponding to a first time in the time sequence of point clouds;
receiving from a server a series of representations of changes to the first point cloud, wherein each representation of the series of representations is applicable to an area of deviation between the first point cloud and a second point cloud corresponding to a later time in the time sequence of point clouds, wherein each area of deviation is selected by the server from among areas of deviations between the first and the second point clouds based on respective importance scores, and wherein each representation of the series comprises at least one of a 3D transformation, a set of points to be added, and a set of points to be removed; and
updating the first point cloud by sequentially applying the representations of the series of representations to the first point cloud to obtain a version of the second point cloud.
12. The method of claim 11 , wherein obtaining the first point cloud comprises receiving a version of the first point cloud from the server.
13. The method of claim 11 , wherein the 3D transformations comprises rotation and translation.
14. The method of claim 11 , wherein a first representation and a second representation, of the series of representations, capture first changes and second changes, respectively, to an area of the first point cloud,
wherein the first changes comprise coarse changes to the area and the second changes comprise finer residual changes to the area.
15. The method of claim 11 , wherein receiving the series of representations comprises receiving an indication of the area of the first point cloud to which a 3D transformation is to be applied.
16. The method of claim 11 , further comprising transmitting to the server information about a viewpoint of a user currently viewing a rendering of the point cloud.
17. An apparatus comprising:
a processor; and
a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to:
obtain a first point cloud in a time sequence of point clouds, the first point cloud corresponding to a first time in the time sequence of point clouds;
obtain a second point cloud in the time sequence of point clouds, the second point cloud corresponding to a later time in the time sequence of point clouds than the first point cloud;
determine areas of deviation between the first point cloud and the second point cloud;
select a first area of deviation from among the determined areas, wherein selecting the first area of deviation is based on respective importance scores;
generate a representation of the changes between the first point cloud and the second point cloud in the first area of deviation, wherein the representation of the changes comprises at least one of a 3D transformation, a set of points to be added, and a set of points to be removed;
update the first point cloud using the representation;
repeat the determining, the selecting, the generating, and the updating steps to obtain a series of representations of the changes between the first point cloud and the second point cloud; and
transmit the series of representations of the changes to a client device.
18. The apparatus of claim 17 , wherein the generated representation comprises a first representation that captures coarse changes and a second representation that captures finer residual changes.
19. The apparatus of claim 17 , wherein the 3D transformation comprises rotation and translation.
20. The apparatus of claim 17 , wherein an importance score for an area of deviation reflects the visual importance of the changes between first and second point clouds in the area of deviation.
21. The apparatus of claim 17 , further comprising
determining a budget for representing the changes between the first point cloud and the second point cloud; and
repeating the determining, selecting, generating and updating steps until the budget is exhausted.
22. The apparatus of claim 21 , wherein the budget comprises at least one of an amount of time, an amount of processing, or an amount of bandwidth available to update the first point cloud based on the changes.
23. An apparatus comprising:
a processor; and
a non-transitory computer-readable medium storing instructions operative, when executed by the processor, to cause the apparatus to:
obtain a first point cloud in a time sequence of point clouds, the first point cloud corresponding to a first time in the time sequence of point clouds,
receive from a server a series of representations of changes to the first point cloud, wherein each representation of the series of representations is applicable to an area of deviation between the first point cloud and a second point cloud corresponding to a later time in the time sequence of point clouds, wherein each area of deviation is selected by the server from among areas of deviations between the first and the second point clouds based on respective importance scores, and wherein each representation of the series comprises at least one of a 3D transformation, a set of points to be added, and a set of points to be removed, and
update the first point cloud by sequentially applying the representations of the series of representations to the first point cloud to obtain a version of the second point cloud.
24. The apparatus of claim 23 , wherein obtaining the first point cloud comprises receiving a version of the first point cloud from the server.
25. The apparatus of claim 23 , wherein the 3D transformation comprises rotation and translation.
26. The apparatus of claim 23 , wherein a first representation and a second representation, of the series of representations, capture first changes and second changes, respectively, to an area of the first point cloud, wherein the first changes comprise coarse changes to the area and the second changes comprise finer residual changes to the area.
27. The apparatus of claim 23 , wherein receiving the series of representations comprises receiving an indication of an area of the first point cloud to which a 3D transformation is to be applied.
28. The apparatus of claim 23 , further comprising transmitting to the server information about a viewpoint of a user currently viewing a rendering of the point cloud.