Fault recovery of video bitstream in remote sessions
Fault recovery of a video bitstream can be performed in a remote session. A decoder of a remote display protocol client can be configured to process a video bitstream that is transferred over a virtual channel of a remote session. As part of processing the video bitstream prior to passing it to a hardware decoder, the decoder can detect the occurrence of an error in the bitstream and then cause the remote display protocol client to employ the remote display protocol which encapsulates the virtual channel to request an I-frame from the remote display protocol service. This will force the encoder to send an I-frame even if the desktop has low entropy thereby minimizing how long a glitch may persist. Because the remote display protocol itself is employed to request the I-frame, there is no requirement to modify the video stream protocol.
1. A method for performing fault recovery of a video bitstream in a remote session comprising:
employing, by a remote display protocol client that executes on a client terminal, a remote display protocol to establish a remote display protocol connection with a remote display protocol service executing on a remote session host to thereby establish and communicate with a remote session;
establishing, by the remote display protocol client, a virtual channel within the remote display protocol connection between a decoder that executes on the client terminal as part of the remote display protocol client and an encoder of the remote display protocol service;
receiving, at the decoder and from the encoder, a video bitstream via the virtual channel, the video bitstream including the remote session's graphics data;
as part of decoding the video bitstream, processing, by the decoder, the video bitstream to identify the occurrence of an error;
in response to identifying an error in the video bitstream, modifying, by the decoder, the decoded video bitstream to include an indication of the error, wherein modifying the decoded video bitstream to include the indication of the error comprises setting an identifier of a slice defined in the video bitstream to an invalid value;
as part of examining the decoded video bitstream before the decoded video bitstream is provided to a hardware decoder for display, detecting, by the remote display protocol client, that the decoded video bitstream includes the indication of the error, wherein detecting that the decoded video bitstream includes the indication of the error comprises detecting that the identifier of the slice is set to the invalid value;
in response to detecting that the decoded video bitstream includes the indication of the error, sending, by the remote display protocol client and via the remote display protocol connection, a key frame request to the remote display protocol service such that the key frame request is not sent by the decoder and is not sent via the virtual channel to the encoder; and
in conjunction with sending the key frame request, preventing, by the remote display protocol client, the slice with the identifier set to the invalid value from being passed to the hardware decoder.
2. The method of claim 1 , wherein the video bitstream comprises an H.264 or H.265 bitstream.
3. The method of claim 1 , wherein the virtual channel is a Remote Desktop Protocol: Graphics Pipeline Extension virtual channel.
4. The method of claim 3 , wherein the key frame request is in the form of a Client Refresh Rect Protocol Data Unit (PDU) of Remote Desktop Protocol: Basic Connectivity and Graphics Remoting.
5. The method of claim 1 , wherein processing the video bitstream comprises extracting and caching individual data units that are defined in the video bitstream.
6. The method of claim 1 , wherein processing the video bitstream to identify the occurrence of an error comprises:
detecting that a start code is not present within the video bitstream.
7. The method of claim 1 , wherein processing the video bitstream to identify the occurrence of an error comprises:
detecting that a raw byte sequence payload (RBSP) in the video bitstream has a size that is less than a defined threshold.
8. The method of claim 1 , wherein processing the video bitstream to identify the occurrence of an error comprises:
determining that a picture parameter set defines a current picture as a top field or a bottom field and a slice pertaining to the current picture has a field_pic_flag bit set to 1; and
wherein modifying the decoded video bitstream to include the indication of the error comprises setting a slice_id of the slice to an invalid value.
9. The method of claim 1 , wherein processing the video bitstream to identify the occurrence of an error comprises:
determining that a current sequence parameter set indicates that a frame contains only macroblocks and a slice pertaining to the current picture has a field_pic_flag bit set to 1; and
wherein modifying the decoded video bitstream to include the indication of the error comprises setting a slice_id of the slice to an invalid value.
10. The method of claim 1 , wherein processing the video bitstream to identify the occurrence of an error comprises:
determining that a current sequence parameter set indicates that a frame does not contain only macroblocks and has a direct_8×8_inference_flag bit set to 0; and
wherein modifying the decoded video bitstream to include the indication of the error comprises setting a slice_id of any slice that references the current sequence parameter set to an invalid value.
11. The method of claim 1 , wherein processing the video bitstream to identify the occurrence of an error comprises:
determining that a first macroblock ID in a slice header of a subsequent slice is not greater than one or more macroblock IDs in a slice header of a previous slice; and
wherein modifying the decoded video bitstream to include the indication of the error comprises setting a slice_id of the subsequent slice to an invalid value.
12. One or more computer storage media storing computer executable instructions which when executed on a client terminal implement a method for performing fault recovery of a video bitstream in a remote session, the method comprising:
employing, by a remote display protocol client that executes on a client terminal, a remote display protocol to establish a remote display protocol connection with a remote display protocol service executing on a remote session host to thereby establish and communicate with a remote session executing on the remote session host;
establishing, by the remote display protocol client, a virtual channel within the remote display protocol connection between a decoder that executes on the client terminal as part of the remote display protocol client and an encoder of the remote display protocol service;
receiving, at the decoder and from the encoder, a video bitstream via the virtual channel, the video bitstream including the remote session's graphics data, the video bitstream defining slices of frames;
as part of decoding the video bitstream, processing, by the decoder, the video bitstream to identify the occurrence of an error;
in response to identifying an error in the video bitstream, modifying, by the decoder, the decoded video bitstream to include an indication of the error, wherein modifying the video bitstream to include the indication of the error comprises setting a slice_id of a slice defined in the video bitstream to an invalid value;
as part of examining the decoded video bitstream before the decoded video bitstream is provided to a hardware decoder for display, detecting, by the remote display protocol client, that the decoded video bitstream includes the indication of the error, wherein detecting that the decoded video bitstream includes the indication of the error comprises detecting that the slice_id of the slice defined in the video bitstream is set to the invalid value;
in response to detecting that the decoded video bitstream includes the indication of the error, sending, by the remote display protocol client and via the remote display protocol connection, a key frame request to the remote display protocol service such that the key frame request is not sent by the decoder and is not sent via the virtual channel to the encoder; and
providing slices that have slice_ids set to a valid value to the hardware decoder while blocking slices that have slice-ids set to an invalid value from being provided to the hardware decoder.
13. The computer storage media of claim 12 , wherein the key frame request comprises a Client Refresh Rect Protocol Data Unit (PDU) of Remote Desktop Protocol: Basic Connectivity and Graphics Remoting.
14. The computer storage media of claim 12 , wherein processing the video bitstream comprises extracting and caching individual data units that are defined in the video bitstream.
15. A method for processing a video bitstream comprising:
employing, by a remote display protocol client that executes on a client terminal, a remote display protocol to establish a remote display protocol connection with a remote display protocol service executing on a remote session host to thereby establish and communicate with a remote session executing on the remote session host;
establishing, by the remote display protocol client, a virtual channel within the remote display protocol connection between a decoder that executes on the client terminal as part of the remote display protocol client and an encoder of the remote display protocol service;
receiving, at the decoder and from the encoder, a video bitstream via the virtual channel, the video bitstream including the remote session's graphics data;
as part of decoding the video bitstream:
extracting, by the decoder, slices from the video bitstream;
evaluating, by the decoder, a slice header of each extracted slice; and
in response to detecting an error in a first slice, setting, by the decoder, a slice_id in the slice header of the first slice to an invalid value;
as part of examining the decoded video bitstream before the decoded video bitstream is provided to a hardware decoder for display, detecting, by the remote display protocol client, that the decoded video bitstream includes the first slice with the slice_id set to the invalid value;
in response to detecting that the decoded video bitstream includes the first slice with the slice_id set to the invalid value, sending, by the remote display protocol client and via the remote display protocol connection, a key frame request to the remote display protocol service such that the key frame request is not sent by the decoder and is not sent via the virtual channel to the encoder; and
in conjunction with sending the key frame request, preventing, by the remote display protocol client, the first slice with the slice_id set to the invalid value from being passed to the hardware decoder.