Reference of neural network model by immersive media for adaptation of media for streaming to heterogenous client end-points
A method, computer program, and computer system is provided for streaming immersive media. Content is ingested in a first two-dimension format or a first three-dimensional format, whereby the format references a neural network. The ingested content is converted to a second two-dimensional or a second three-dimensional format based on the referenced neural network. The converted content is streamed to a client end-point, such as a television, a computer, a head-mounted display, a lenticular light field display, a holographic display, an augmented reality display, or a dense light field display.
1. A method of streaming immersive media, executable by a processor, comprising:
ingesting content in a first two-dimensional format or a first three-dimension format, the content comprising one or more SEI messages referencing a neural network and parameters associated with the neural network using a location of the neural network;
converting the ingested content to a second two-dimensional or a second three-dimensional format based on the referenced neural network; and
streaming the converted content to a client end-point.
2. The method of claim 1 , further comprising adapting the content into a scene-specific volumetric format associated with the client end-point based on inferring depth information from the content.
3. The method of claim 1 , wherein the neural network is referenced by metadata that identifies the location of the model.
4. The method of claim 1 , wherein the neural network is referenced by a universal resource identifier corresponding to the metadata describing the ingested content.
5. The method of claim 1 , wherein the neural network is trained prior to ingesting the content based on priors corresponding to objects within the content.
6. The method of claim 1 , wherein the ingested content is converted based on characteristics of the client end-point.
7. The method of claim 1 , wherein the one or more client end-points include one or more from among a television, a computer, a head-mounted display, a lenticular light field display, a holographic display, an augmented reality display, and a dense light field display.
8. A computer system for streaming immersive media, the computer system comprising:
one or more computer-readable non-transitory storage media configured to store computer program code; and
one or more computer processors configured to access said computer program code and operate as instructed by said computer program code, said computer program code including:
ingesting code configured to cause the one or more computer processors to ingest content in a first two-dimensional format or a first three-dimension format, the content comprising one or more SEI messages referencing a neural network and parameters associated with the neural network using a location of the neural network;
converting code configured to cause the one or more computer processors to convert the ingested content to a second two-dimensional or a second three-dimensional format based on the referenced neural network; and
streaming code configured to cause the one or more computer processors to stream the converted content to a client end-point.
9. The computer system of claim 8 , further comprising adapting the content into a scene-specific volumetric format associated with the client end-point based on inferring depth information from the content.
10. The computer system of claim 8 , wherein the neural network is referenced by metadata that identifies the location of the model.
11. The computer system of claim 8 , wherein the neural network is referenced by a universal resource identifier corresponding to metadata describing the content.
12. The computer system of claim 8 , wherein the neural network is trained prior to ingesting the content based on priors corresponding to objects within the content.
13. The computer system of claim 8 , wherein the ingested content is converted based on characteristics of the client end-point.
14. The computer system of claim 8 , wherein the one or more client end-points include one or more from among a television, a computer, a head-mounted display, a lenticular light field display, a holographic display, an augmented reality display, and a dense light field display.
15. A non-transitory computer readable medium having stored thereon a computer program for streaming immersive media, the computer program configured to cause one or more computer processors to:
ingest content in a first two-dimensional format or a first three-dimension format, the content comprising one or more SEI messages referencing a neural network and parameters associated with the neural network using a location of the neural network;
converting code configured to cause the one or more computer processors to convert the ingested content to a second two-dimensional or a second three-dimensional format based on the referenced neural network; and
streaming code configured to cause the one or more computer processors to stream the converted content to a client end-point.
16. The computer readable medium of claim 15 , further comprising adapting the content into a scene-specific volumetric format associated with the client end-point based on inferring depth information from the content.
17. The computer readable medium of claim 15 , wherein the neural network is referenced by metadata that identifies the location of the model.
18. The computer readable medium of claim 15 , wherein the neural network is referenced by a universal resource identifier corresponding to metadata describing the content.
19. The computer readable medium of claim 15 , wherein the neural network is trained prior to ingesting the content based on priors corresponding to objects within the content.
20. The computer readable medium of claim 15 , wherein the ingested content is converted based on characteristics of the client end-point.