Video pipeline
A mixed reality system that includes a device and a base station that communicate via a wireless connection The device may include sensors that collect information about the user's environment and about the user. The information collected by the sensors may be transmitted to the base station via the wireless connection. The base station renders frames or slices based at least in part on the sensor information received from the device, encodes the frames or slices, and transmits the compressed frames or slices to the device for decoding and display. The base station may provide more computing power than conventional stand-alone systems, and the wireless connection does not tether the device to the base station as in conventional tethered systems. The system may implement methods and apparatus to maintain a target frame rate through the wireless link and to minimize latency in frame rendering, transmittal, and display.
1 . A system, comprising:
a base station comprising one or more processors; and
a device, comprising:
one or more processors;
one or more cameras configured to capture frames that include views of a user's environment; and
a display subsystem for displaying a 3D virtual view to the user;
wherein the device is configured to:
transmit frames captured by the one or more cameras to the base station over a wireless connection;
wherein the base station is configured to:
render the frames or frame portions;
compress the rendered frames or frame portions; and
transmit the compressed frames or frame portions to the device over the wireless connection;
wherein the device is configured to decompress the compressed frames or frame portions received from the base station and provide the rendered frames or frame portions to the display subsystem for display; and
wherein the device is configured to:
monitor the wireless connection to the base station; and
in response to detecting that the wireless connection to the base station has been lost:
render, by the device, one or more frames that include views of the user's environment based on the frames captured by the one or more cameras; and
provide the rendered one or more frames to the display subsystem for display.
2 . The system of claim 1 , wherein the base station is configured to:
monitor a rate at which the frames or the frame portions are rendered by a rendering application on the base station; and
in response to detecting that the rendering rate is below a threshold, direct the rendering application to reduce a complexity of one or more rendering processes, wherein reducing the complexity of the one or more rendering processes reduces a resolution of the rendered frames or frame portions and increases a rate at which the frames or the frame portions are rendered.
3 . The system of claim 1 , wherein the base station is configured to:
monitor bandwidth usage on the wireless connection between the device and the base station; and
in response to detecting that the bandwidth usage is above a threshold, direct the rendering application to reduce a complexity of one or more rendering processes, wherein reducing the complexity of the one or more rendering processes reduces a resolution of the rendered frames or frame portions.
4 . The system of claim 1 , wherein the base station is configured to:
monitor a rate at which frames or frame portions are rendered and/or monitor bandwidth usage on the wireless connection; and
in response to detecting that the rendering rate is below a rendering threshold or that the bandwidth usage is above a bandwidth threshold, adjust one or more compression processes on the base station to increase a compression ratio at which the rendered frames or frame portions are compressed.
5 . The system of claim 1 , wherein the device or the base station is configured to:
monitor a frame rate or latency of transmission of the frames, or frame portions, or the compressed frames, or the compressed frame portions; and
adjust a compression process, a rendering process, or a transmission process with regard to the frames or the frame portions such that the frame rate or the latency of transmission satisfy one or more thresholds.
6 . The system of claim 5 , wherein the monitored latency of transmission of the frames or frame portions comprises:
a latency to transmit the captured frames from the device to the base station;
a latency to generate the compressed frames or frame portions at the base station; and
a latency to transmit the compressed frames or frame portions to the device.
7 . The system of claim 1 , wherein the rendered frames or frame portions comprise a mixed reality view.
8 . The system of claim 1 , wherein the device comprises:
a headset; or
glasses.
9 . The system of claim 1 , wherein the base station is:
a desktop computer;
a notebook or laptop computer;
a pad or tablet device;
a smartphone; or
a hand-held computing device.
10 . A device, comprising:
one or more processors;
one or more cameras configured to capture frames that include views of a user's environment; and
a display subsystem for displaying a 3D virtual view to the user;
wherein the device is configured to:
transmit frames captured by the one or more cameras to a base station over a wireless connection;
receive compressed rendered frames or frame portions from the base station over the wireless connection; and
decompress the compressed frames or frame portions received from the base station and provide the rendered frames or frame portions to the display subsystem for display,
wherein the device is configured to:
monitor the wireless connection to the base station; and
in response to detecting that the wireless connection to the base station has been lost:
render, by the device, one or more frames that include views of the user's environment based on the frames captured by the one or more cameras; and
provide the rendered one or more frames to the display subsystem for display.
11 . The device of claim 10 , S wherein the device is further configured to:
monitor a frame rate or a latency of transmission of the frames, or frame portions, or the compressed frames, or the compressed frame portions; and
cause a compression process, a rendering process, or a transmission process to be adjusted with regard to the frames or the frame portions such that the frame rate or the latency of transmission satisfy one or more thresholds.
12 . The device of claim 10 , wherein the rendered frames or frame portions comprise a mixed reality view.
13 . The device of claim 10 , wherein the device comprises:
a headset; or
glasses.
14 . A computing device, comprising:
one or more processors to:
receive frames captured by one or more cameras of a head-mounted display device, wherein the frames are received, by the computing device over a wireless connection;
render the frames or frame portions;
compress the rendered frames or frame portions;
transmit the compressed frames or frame portions over the wireless connection to the head-mounted display device;
monitor the wireless connection to the head-mounted display device; and
in response to detecting that the wireless connection to the head-mounted display has been restored, resume transmitting the compressed frames or frame portions over the wireless connection to the head-mounted display device,
wherein while the wireless connection is lost, the head-mounted display device:
renders one or more frames that include views of a user's environment based on the frames captured by the one or more cameras; and
provides the rendered one or more frames that have been rendered to a display subsystem of the head-mounted device for display.
15 . The computing device of claim 14 , wherein the computing device is configured to:
monitor a rate at which the frames or the frame portions are rendered by a rendering application on the computing device; and
in response to detecting that the rendering rate is below a threshold, direct the rendering application to reduce a complexity of one or more rendering processes, wherein reducing the complexity of the one or more rendering processes reduces a resolution of the rendered frames or frame portions and increases the rate at which frames or frame portions are rendered.
16 . The computing device of claim 14 , wherein the computing device is configured to:
monitor bandwidth usage on the wireless connection between the head-mounted display device and the computing device; and
in response to detecting that the bandwidth usage is above a threshold, direct the rendering application to reduce a complexity of one or more rendering processes, wherein reducing the complexity of the one or more rendering processes reduces a resolution of the rendered frames or frame portions.
17 . The computing device of claim 14 , wherein the computing device is configured to:
monitor a rate at which frame portions are rendered and/or monitor bandwidth usage on the wireless connection; and
in response to detecting that the rendering rate is below a rendering threshold or that the bandwidth usage is above a bandwidth threshold, adjust one or more compression processes to increase a compression ratio at which the rendered frames or frame portions are compressed.
18 . The computing device of claim 14 , wherein the one or more processors are further configured to:
monitor a frame rate or latency of transmission over the wireless connection; and
adjust a compression process, a rendering process, or a transmission process with regard to the frames or the frame portions such that the frame rate or the latency of transmission satisfy one or more thresholds,
wherein the monitored latency of transmission comprises:
a latency to transmit the captured frames from the head-mounted device to the computing device;
a latency to generate the compressed frames or frame portions at the computing device; and
a latency to transmit the compressed frames or frame portions to the head-mounted device.
19 . The computing device of claim 18 , wherein said adjusting the compression process, the rendering process, or the transmission process is performed in response to the latency exceeding a threshold, wherein the threshold is 1 millisecond or less.
20 . The computing device of claim 14 , wherein the head-mounted display comprises:
a headset; or
glasses.