Video quality improvements system and method for virtual reality
A system and method for improving the quality of video for virtual reality systems uses several different techniques to improve the quality of the video. The different techniques may include quantization parameter maps, gradient scaling, using analytics to identify most view scenes and encode the most viewed scenes with better quality and adaptively shaping the field of view.
1. A method for improved virtual reality data sent to a virtual reality device, comprising:
receiving a virtual reality data request from a virtual reality device, the virtual reality data request including viewing data for the virtual reality device that issued the virtual reality data request;
generating, at a virtual reality backend remote from the virtual reality device, a gradient scaling for each frame of the virtual reality data that is a 360 degree view for a view point of a user, wherein the gradient scaling gradually decreases a pixel-scaling ratio across each frame of virtual reality data, wherein the pixel-scaling ratio is a ratio of a number of pixels in a portion of the frame before scaling to a number of pixels in the portion of the frame after scaling and wherein the gradual decrease in the pixel-scaling ratio monotonically decreases from a 1:1 pixel-scaling ratio at the center of the frame to a reduced pixel-scaling ratio at an edge of the frame of the virtual reality data and the pixel-scaling ratios are the same for concentric locations in the frame;
generating one or more fields of views for each frame of the virtual reality data based on the viewing data for the virtual reality device;
encoding the virtual reality data for the virtual reality device based on the gradient scaling and the one or more generated field of views; and
communicating the encoded virtual reality data to the virtual reality device.
2. The method of claim 1 further comprising adjusting, at the virtual reality backend remote from the virtual reality device, a set of quantization parameters that form a quantization parameter map that are used to encode the virtual reality data sent back to the virtual reality device using more bits at a center of a field of view of the virtual reality data for the particular virtual reality device and fewer bits at locations away from the center of the field of view, wherein adjusting the set of quantization parameters further comprises determining a characteristic of a lens of the virtual reality device and adjusting the set of quantization parameters based on the characteristic of the lens of the virtual reality device.
3. The method of claim 2 , wherein adjusting the set of quantization parameters further comprises adjusting the set of quantization parameters based on the characteristic of the lens of the virtual reality device and a characteristics of the human eye.
4. The method of claim 3 , wherein adjusting the set of quantization parameters further comprises generating a quantization parameter map.
5. The method of claim 1 , wherein generating the one or more field of views further comprises generating one or more field of views having one of different sizes and different shapes.
6. The method of claim 1 , wherein generating the one or more field of views further comprises generating the one or more field of views using viewing data over a period of time.
7. The method of claim 6 , wherein the viewing data further comprises analytic data that is one of a most viewed scene of the virtual reality data, a heat map of the virtual reality data and a movement of each virtual reality device indicating a scene of the virtual reality data currently being viewed by each virtual reality device to determine an area of the scene in the virtual reality data most viewed and wherein encoding the virtual reality data further comprises encoding the area of the scene in the virtual reality data most viewed at a higher quality based on the analytic data and encoding a plurality of scenes in the virtual reality data that are not most viewed at a lower quality.
8. A virtual reality system, comprising:
a plurality of virtual reality devices wherein each virtual reality device has a display that displays virtual reality data to a user;
a virtual reality backend coupled to each virtual reality device;
each virtual reality device sending a virtual reality data request to the virtual reality backend, the virtual reality data request including viewing data for the virtual reality device that issued the virtual reality data request;
the virtual reality backend having a processor, a memory and a plurality of lines of computer code so that the virtual reality backend is configured to:
generate a gradient scaling for each frame of the virtual reality data that is a 360 degree view for a view point of a user, wherein the gradient scaling gradually decreases a pixel-scaling ratio across each frame of virtual reality data, wherein the pixel-scaling ratio is a ratio of a number of pixels in a portion of the frame before scaling to a number of pixels in the portion of the frame after scaling and wherein the gradual decrease in the pixel-scaling ratio monotonically decreases from a 1:1 pixel-scaling ratio at the center of the frame to a reduced pixel-scaling ratio at an edge of the frame of the virtual reality data and the pixel-scaling ratios are the same for concentric locations in the frame;
generate one or more fields of views for each frame of the virtual reality data based on the viewing data for a particular virtual reality device;
encode the virtual reality data for the particular virtual reality device based on the gradient scaling and the one or more generated field of views; and
send the encoded virtual reality data to the particular virtual reality device to improve the quality of the virtual reality data for the particular virtual reality device.
9. The system of claim 8 , wherein the virtual reality backend is further configured to adjust one or more quantization parameters that are used to encode the virtual reality data sent back to a particular virtual reality device and determine a characteristic of a lens of the particular virtual reality device and adjust the set of quantization parameters based on the characteristic of the lens of the particular virtual reality device.
10. The system of claim 9 , wherein the virtual reality backend is further configured to adjust the set of quantization parameters based on the characteristic of the lens of the particular virtual reality device and a characteristics of the human eye.
11. The system of claim 10 , wherein the virtual reality backend is further configured to generate a quantization parameter map.
12. The system of claim 8 , wherein the virtual reality backend is further configured to generate one or more field of views having one of different sizes and different shapes.
13. The system of claim 8 , wherein the virtual reality backend is further configured to generate the one or more field of views using viewing data over a period of time.
14. The system of claim 13 , wherein the viewing data further comprises analytic data that is one of a most viewed scene of the virtual reality data, a heat map of the virtual reality data and a movement of each virtual reality device indicating a scene of the virtual reality data currently being viewed by each virtual reality device to determine an area of the scene in the virtual reality data most viewed and wherein the virtual reality backend is further configured to encode the area of the scene in the virtual reality data most viewed at a higher quality based on the analytic data and encode a plurality of scenes in the virtual reality data that are not most viewed at a lower quality.
15. A virtual reality backend having a processor and memory, comprising:
the processor executing a plurality of lines of instructions of a video quality improvement element and the processor is configured to:
generate a gradient scaling for each frame of the virtual reality data that is a 360 degree view for a view point of a user, wherein the gradient scaling gradually decreases a pixel-scaling ratio across each frame of virtual reality data, wherein the pixel-scaling ratio is a ratio of a number of pixels in the frame before scaling to a number of pixels after scaling and wherein the gradual decrease in the pixel-scaling ratio monotonically decreases from a 1:1 pixel-scaling ratio at the center of the frame to a reduced pixel-scaling ratio at an edge of the frame of the virtual reality data and the pixel-scaling ratios are the same for concentric locations in the frame;
generate one or more fields of views for each frame of a virtual reality data based on viewing data for the virtual reality device wherein the gradient scaling and the one or more generated field of views improve the quality of the virtual reality data for the virtual reality device.
16. The virtual reality backend of claim 15 , wherein the processor is further configured to adjust one or more quantization parameters that are used to encode virtual reality data sent back to the virtual reality device and determine a characteristic of a lens of the virtual reality device and adjust the set of quantization parameters based on the characteristic of the lens of the virtual reality device.
17. The virtual reality backend of claim 16 , wherein the processor is further configured to adjust the set of quantization parameters based on the characteristic of the lens of the virtual reality device and a characteristics of the human eye.
18. The virtual reality backend of claim 17 , wherein the processor is further configured to generate a quantization parameter map.
19. The virtual reality backend of claim 15 , wherein the processor is further configured to generate one or more field of views having one of different sizes and different shapes.
20. The virtual reality backend of claim 15 , wherein the processor is further configured to generate the one or more field of views using viewing data over a period of time.
21. The virtual reality backend of claim 20 , wherein the viewing data further comprises analytic data that is one of a most viewed scene of the virtual reality data, a heat map of the virtual reality data and a movement of each virtual reality device indicating a scene of the virtual reality data currently being viewed by each virtual reality device to determine an area of the scene in the virtual reality data most viewed and wherein the video quality improvement element encodes the area of the scene in the virtual reality data most viewed at a higher quality based on the analytic data and encodes a plurality of scenes in the virtual reality data that are not most viewed at a lower quality.