Systems and methods for real-time streaming of three-dimensional content using different encoding formats

1 . A method comprising:

receiving a request for three-dimensional (3D) content;

retrieving at least a first 3D asset and a second 3D asset that form the 3D content;

determining a first priority associated with the first 3D asset and a second priority associated with the second 3D asset;

streaming the first 3D asset in a first encoding format based on the first priority and the second 3D asset in a second encoding format based on the second priority, wherein the first encoding format uses more network resources than the second encoding format and presents the first 3D asset at a higher fidelity than the second encoding format;

determining that a priority of the second 3D asset changes from the second priority to the first priority between a first frame and a subsequent frame of the 3D content; and

streaming the second 3D asset in the first encoding format for the subsequent frame of the 3D content, wherein the first encoding format increases a fidelity at which the second 3D asset is rendered in the 3D content relative to the second encoding format.

2 . The method of claim 1 further comprising:

assigning the first priority to the first 3D asset in response to the first 3D asset being positioned in one or more of a center or a foreground of the 3D content; and

assigning the second priority to the second 3D asset in response to the second 3D asset being positioned in one or more of a periphery or a background of the 3D content.

3 . The method of claim 1 further comprising:

assigning the first priority to the first 3D asset based on the first 3D asset occupying a larger first portion of a field-of-view; and

assigning the second priority to the second 3D asset based on the second 3D asset occupying a second portion of the field-of-view that is smaller than the first portion of the field-of-view.

4 . The method of claim 1 further comprising:

determining an amount of available bandwidth for streaming the 3D content; and

determining that the amount of available bandwidth is sufficient to stream a total amount of data associated with the first 3D asset in the first encoding format and the second 3D asset in the second encoding format at a desired frame rate or without interrupting a real-time experience.

5 . The method of claim 1 further comprising:

determining a reduction in rendering resources at a client device requesting the 3D content between a first frame and a second frame of the 3D content, wherein the second frame is one or more frames after the first frame; and

streaming the first 3D asset in the second encoding format for the second frame of the 3D content, wherein the second encoding format is rendered using fewer rendering resources of the client device than the first encoding format.

6 . The method of claim 1 ,

wherein the first encoding format corresponds to one of a point cloud format, a Gaussian splat format, or a mesh format, and

wherein the second encoding format corresponds to a different one of the Gaussian splat format, the mesh format, or a two-dimensional (2D) image format.

7 . The method of claim 1 further comprising:

encoding the first 3D asset with a first plurality of primitives of the first encoding format; and

encoding the second 3D asset with a second plurality of primitives of the second encoding format, wherein the first plurality of primitives reconstruct the first 3D asset with less loss than the second plurality of primitives reconstruct the second 3D asset.

8 . The method of claim 1 further comprising:

defining the first 3D asset in the first encoding format with one of a plurality of meshes, a plurality of splats, or a plurality of points; and

defining the second 3D asset in the second encoding format with a different one the plurality of meshes, the plurality of splats, or the plurality of points.

9 . The method of claim 1 , wherein streaming the first 3D asset and the second 3D asset comprises:

streaming a first plurality of primitives encoding the first 3D asset in the first encoding format and a second plurality of primitives encoding the second 3D asset in the second encoding format for a first key frame of the 3D content; and

streaming changes to the first plurality of primitives and to the second plurality of primitives in frames that follow the first key frame.

10 . The method of claim 9 , wherein determining that the priority of the second 3D asset changes comprises:

changing a priority of the second 3D asset from the second priority to the first priority before a second key frame of the 3D content; and

streaming a third plurality of primitives encoding the second 3D asset in the first encoding format for the second key frame of the 3D content.

11 . The method of claim 10 , wherein changing the priority comprises:

determining that a position of the second 3D asset changes from a periphery or a background to a center or a foreground from the first key frame to the second key frame.

12 . The method of claim 10 , wherein changing the priority comprises:

determining that a size at which the second 3D asset is presented in the 3D content increases from the first key frame to the second key frame.

13 . A streaming system comprising:

one or more hardware processors configured to:

receive a request for three-dimensional (3D) content;

retrieve at least a first 3D asset and a second 3D asset that form the 3D content;

determine a first priority associated with the first 3D asset and a second priority associated with the second 3D asset;

stream the first 3D asset in a first encoding format based on the first priority and the second 3D asset in a second encoding format based on the second priority, wherein the first encoding format uses more network resources than the second encoding format and presents the first 3D asset at a higher fidelity than the second encoding format;

determine that a priority of the second 3D asset changes from the second priority to the first priority between a first frame and a subsequent frame of the 3D content; and

stream the second 3D asset in the first encoding format for the subsequent frame of the 3D content, wherein the first encoding format increases a fidelity at which the second 3D asset is rendered in the 3D content relative to the second encoding format.

14 . The streaming system of claim 13 , wherein the one or more hardware processors are further configured to:

assign the first priority to the first 3D asset in response to the first 3D asset being positioned in one or more of a center or a foreground of the 3D content; and

assign the second priority to the second 3D asset in response to the second 3D asset being positioned in one or more of a periphery or a background of the 3D content.

15 . The streaming system of claim 13 , wherein the one or more hardware processors are further configured to:

assign the first priority to the first 3D asset based on the first 3D asset occupying a first portion of a field-of-view; and

assign the second priority to the second 3D asset based on the second 3D asset occupying a second portion of the field-of-view that is smaller than the first portion of the field-of-view.

16 . The streaming system of claim 13 , wherein the one or more hardware processors are further configured to:

determine an amount of available bandwidth for streaming the 3D content; and

determine that the amount of available bandwidth is sufficient to stream a total amount of data associated with the first 3D asset in the first encoding format and the second 3D asset in the second encoding format at a desired frame rate or without interrupting a real-time experience.

17 . A method comprising:

receiving a request for three-dimensional (3D) content from a client device;

retrieving at least a first 3D asset and a second 3D asset that form the 3D content;

determining a first priority associated with the first 3D asset and a second priority associated with the second 3D asset;

streaming, to the client device, the first 3D asset in a first encoding format based on the first priority and the second 3D asset in a second encoding format based on the second priority, wherein the first encoding format uses more network resources than the second encoding format and presents the first 3D asset at a higher fidelity than the second encoding format;

determining a reduction in rendering resources at the client device between a first frame and a subsequent frame of the 3D content; and

streaming the first 3D asset in the second encoding format for the subsequent frame of the 3D content, wherein the second encoding format is rendered using fewer rendering resources of the client device than the first encoding format.

18 . A streaming system comprising:

one or more hardware processors configured to:

receive a request for three-dimensional (3D) content from a client device;

retrieve at least a first 3D asset and a second 3D asset that form the 3D content;

determine a first priority associated with the first 3D asset and a second priority associated with the second 3D asset;

stream, to the client device, the first 3D asset in a first encoding format based on the first priority and the second 3D asset in a second encoding format based on the second priority, wherein the first encoding format uses more network resources than the second encoding format and presents the first 3D asset at a higher fidelity than the second encoding format;

determine a reduction in rendering resources at the client device between a first frame and a subsequent frame of the 3D content; and

stream the first 3D asset in the second encoding format for the subsequent frame of the 3D content, wherein the second encoding format is rendered using fewer rendering resources of the client device than the first encoding format.

19 . A method comprising:

receiving a request for three-dimensional (3D) content;

retrieving at least a first 3D asset and a second 3D asset that form the 3D content;

determining a first priority associated with the first 3D asset and a second priority associated with the second 3D asset;

encoding the first 3D asset with a first plurality of primitives based on a first encoding format associated with the first priority;

encoding the second 3D asset with a second plurality of primitives based on a second encoding format associated with the second priority, wherein the first plurality of primitives reconstruct the first 3D asset with less loss than the second plurality of primitives reconstruct the second 3D asset; and

streaming the first 3D asset with the first plurality of primitives in the first encoding format based on the first priority and the second 3D asset with the second plurality of primitives in the second encoding format based on the second priority, wherein the first encoding format uses more network resources than the second encoding format and presents the first 3D asset at a higher fidelity than the second encoding format.

20 . A streaming system comprising:

one or more hardware processors configured to:

receive a request for three-dimensional (3D) content;

retrieve at least a first 3D asset and a second 3D asset that form the 3D content;

determine a first priority associated with the first 3D asset and a second priority associated with the second 3D asset;

encode the first 3D asset with a first plurality of primitives based on a first encoding format associated with the first priority;

encode the second 3D asset with a second plurality of primitives based on a second encoding format associated with the second priority, wherein the first plurality of primitives reconstruct the first 3D asset with less loss than the second plurality of primitives reconstruct the second 3D asset; and

stream the first 3D asset with the first plurality of primitives in the first encoding format based on the first priority and the second 3D asset with the second plurality of primitives in the second encoding format based on the second priority, wherein the first encoding format uses more network resources than the second encoding format and presents the first 3D asset at a higher fidelity than the second encoding format.

21 . A method comprising:

receiving a request for three-dimensional (3D) content;

retrieving at least a first 3D asset and a second 3D asset that form the 3D content;

determining a first priority associated with the first 3D asset and a second priority associated with the second 3D asset; and

streaming the first 3D asset in a first encoding format based on the first priority and the second 3D asset in a second encoding format based on the second priority, wherein the first encoding format uses more network resources than the second encoding format and presents the first 3D asset at a higher fidelity than the second encoding format, wherein streaming the first 3D asset in the first encoding format and the second 3D asset in the second encoding format comprises:

streaming a first plurality of primitives encoding the first 3D asset in the first encoding format and a second plurality of primitives encoding the second 3D asset in the second encoding format for a first key frame of the 3D content; and

streaming changes to the first plurality of primitives and to the second plurality of primitives in frames that follow the first key frame.

22 . A streaming system comprising:

one or more hardware processors configured to:

receive a request for three-dimensional (3D) content;

retrieve at least a first 3D asset and a second 3D asset that form the 3D content;

determine a first priority associated with the first 3D asset and a second priority associated with the second 3D asset; and

stream the first 3D asset in a first encoding format based on the first priority and the second 3D asset in a second encoding format based on the second priority, wherein the first encoding format uses more network resources than the second encoding format and presents the first 3D asset at a higher fidelity than the second encoding format, wherein streaming the first 3D asset in the first encoding format and the second 3D asset in the second encoding format comprises:

streaming a first plurality of primitives encoding the first 3D asset in the first encoding format and a second plurality of primitives encoding the second 3D asset in the second encoding format for a first key frame of the 3D content; and

streaming changes to the first plurality of primitives and to the second plurality of primitives in frames that follow the first key frame.