System and method for procedurally colorizing spatial data
Systems and methods are described for compressing color information in point cloud data. In some embodiments, point cloud data includes point position information and point color information for each of a plurality of points. The point position information is provided to a neural network, and the neural network generates predicted color information (e.g. predicted luma and chroma values) for respective points in the point cloud. A prediction residual is generated to represent the difference between the predicted color information and the input point color position. The point position information (which may be in compressed form) and the prediction residual are encoded in a bitstream. In some embodiments, color hint data is encoded to improve color prediction.
1. A method comprising:
receiving a bitstream that encodes at least (i) geometry information for a point cloud, (ii) neural network parameter data, and (iii) a residual color signal;
producing color prediction data for the point cloud by supplying the geometry information as input to a neural network characterized by the received neural network parameter data; and
adding the residual color signal to the color prediction data to generate a reconstructed color signal for the point cloud.
2. The method of claim 1 , further comprising rendering a representation of the point cloud using the reconstructed color signal.
3. The method of claim 1 , wherein the neural network parameter data comprises a set of neural network weights.
4. The method of claim 1 , wherein the neural network parameter data comprises information identifying a stored set of neural network weights.
5. The method of claim 1 , wherein the bitstream further encodes color hint data, and wherein producing color prediction data further comprising supplying the color hint data as input to the neural network.
6. The method of claim 1 , wherein the bitstream further encodes local color hint data comprising at least one color sample of at least one respective position in the point cloud, and wherein producing color prediction data further comprises supplying the local color hint data as input to the neural network.
7. The method of claim 1 , wherein the bitstream further encodes global color hint data comprising color histogram data, and wherein producing color prediction data further comprises supplying the global color hint data as input to the neural network.
8. The method of claim 1 , wherein the bitstream further encodes global color hint data comprising color saturation data, and wherein producing color prediction data further comprises supplying the global color hint data as input to the neural network.
9. The method of claim 1 , wherein producing color prediction data further comprises supplying a previously-reconstructed color signal of a previously-reconstructed point cloud as input into the neural network.
10. The method of claim 1 , wherein the color prediction data produced for the point cloud includes luma and chroma information for each of a plurality of points in the point cloud.
11. The method of claim 1 , wherein the geometry information is encoded in the bitstream in a compressed form, and wherein the method further comprises decompressing the geometry information.
12. The method of claim 1 , wherein the geometry information for the point cloud comprises position information for each of a plurality of points in the point cloud.
13. An apparatus comprising:
a processor configured to perform at least:
receiving a bitstream that encodes at least (i) geometry information for a point cloud, (ii) neural network parameter data, and (iii) a residual color signal;
producing color prediction data for the point cloud by supplying the geometry information as input to a neural network characterized by the received neural network parameter data; and
adding the residual color signal to the color prediction data to generate a reconstructed color signal for the point cloud.
14. The apparatus of claim 13 , wherein the neural network parameter data comprises a set of neural network weights.
15. The apparatus of claim 13 , wherein the bitstream further encodes color hint data, and wherein producing color prediction data further comprising supplying the color hint data as input to the neural network.
16. The apparatus of claim 13 , wherein the neural network parameter data comprises information identifying a stored set of neural network weights.
17. The apparatus of claim 13 , wherein the bitstream further encodes local color hint data comprising at least one color sample of at least one respective position in the point cloud, and wherein producing color prediction data further comprises supplying the local color hint data as input to the neural network.
18. The apparatus of claim 13 , wherein the bitstream further encodes global color hint data comprising color histogram data, and wherein producing color prediction data further comprises supplying the global color hint data as input to the neural network.
19. The apparatus of claim 13 , wherein the bitstream further encodes global color hint data comprising color saturation data, and wherein producing color prediction data further comprises supplying the global color hint data as input to the neural network.
20. The apparatus of claim 13 , wherein producing color prediction data further comprises supplying a previously-reconstructed color signal of a previously-reconstructed point cloud as input into the neural network.