Method and system for enhancing user experience for remoting technologies

1. A computer-implemented method, comprising:

receiving, by a client device, from a remoting server, a sequence of frame updates for a display screen;

determining, at the client device, a sequence of frames corresponding to the received frame updates;

monitoring, at the client device, the received sequence of frame updates to observe the available network bandwidth, frame refresh rate, and image quality corresponding to the received sequence of frame updates; and

executing an enhancement application on the client device, the enhancement application adaptively applying, at the client device, one or more image enhancing techniques to the sequence of frames based on at least one of:

the observed available network bandwidth;

the observed frame refresh rate; and

the observed image quality;

wherein applying the one or more image enhancement techniques comprises:

detecting a user interaction with a user-interface object;

generating a predicted frame based on the user interaction; and

appending the predicted frame to the received sequence of frame updates;

wherein generating the predicted frame based on the user interaction with the user-interface object comprises:

generating a predicted motion vector corresponding to the user interaction with the user-interface object;

generating the predicted frame based on the predicted motion vector; and

wherein the user interaction with the user-interface object comprises at least one of:

a window-scrolling operation;

a window-resizing operation;

a text entry into a text field;

a mouse-pointer movement;

an button click; and

a drag operation that moves the object to a new screen position.

2. The method of claim 1 , wherein adaptively applying, at the client device, the one or more image enhancing techniques comprises:

determining whether the observed available network bandwidth, observed frame refresh rate, or observed image quality is sufficiently low to apply the one or more image enhancing techniques; and

selecting an image enhancement technique based on a type of content distributed by the remoting server.

3. The method of claim 1 , further comprising:

in response to the received frame updates being associated with a streamed video, applying an image enhancement technique that comprises interpolating a frame based on at least two buffered frames.

4. The method of claim 1 , wherein adaptively applying the one or more image enhancement techniques further comprises:

determining whether a respective image enhancement technique sufficiently improves user experience; and

in response to the image enhancement technique not sufficiently improving user experience, discontinuing application of the image enhancement technique.

5. The method of claim 1 , wherein applying the one or more image enhancement techniques further comprises reducing appearance of artifacts in a frame that the client device received from the remoting server, and wherein reducing appearance of artifacts in the received frame comprises:

detecting an edge in the frame;

determining whether the edge belongs to a user-interface object or is caused by artifacts;

in response to the edge being caused by artifacts, blurring the edge; and

in response to the belonging to a user-interface object, preserving the edge.

6. The method of claim 5 , wherein determining whether the edge belongs to the user-interface object or is caused by artifacts comprises:

determining whether a color variance along the edge is greater than a predetermined threshold; and

in response to the color variance being greater than the threshold, identifying the edge as being caused by artifacts.

7. A non-transitory storage medium storing instructions which when executed by a processor cause the processor to perform a method, the method comprising:

receiving, by a client device, from a remoting server, a sequence of frame updates for a display screen;

determining, at the client device, a sequence of frames corresponding to the received frame updates;

monitoring, at the client device, the received sequence of frame updates to observe the available network bandwidth, frame refresh rate, and image quality corresponding to the received sequence of frame updates; and

executing an enhancement application on the client device, the enhancement application adaptively applying, at the client device, one or more image enhancing techniques to the sequence of frames based on at least one of:

the observed available network bandwidth;

the observed frame refresh rate; and

the observed image quality;

wherein applying the one or more image enhancement techniques comprises:

detecting a user interaction with a user-interface object;

generating a predicted frame based on the user interaction; and

appending the predicted frame to the received sequence of frame updates;

wherein generating the predicted frame based on the user interaction with the user-interface object comprises:

generating a predicted motion vector corresponding to the user interaction with the user-interface object;

generating the predicted frame based on the predicted motion vector; and

wherein the user interaction with the user-interface object comprises at least one of:

a window-scrolling operation;

a window-resizing operation;

a text entry into a text field;

a mouse-pointer movement;

an button click; and

a drag operation that moves the object to a new screen position.

8. The non-transitory storage medium of claim 7 , wherein adaptively applying, at the client device, the one or more image enhancing techniques comprises:

determining whether the observed available network bandwidth, observed frame refresh rate, or observed image quality is sufficiently low to apply the one or more image enhancing techniques; and

selecting an image enhancement technique based on a type of content distributed by the remoting server.

9. The non-transitory storage medium of claim 7 , wherein the method further comprises:

in response to the received frame updates being associated with a streamed video, applying an image enhancement technique that comprises interpolating a frame based on at least two buffered frames.

10. The non-transitory storage medium of claim 7 , wherein adaptively applying the one or more image enhancement techniques further comprises:

determining whether a respective image enhancement technique sufficiently improves user experience; and

in response to the image enhancement technique not sufficiently improving user experience, discontinuing application of the image enhancement technique.

11. The non-transitory storage medium of claim 7 , wherein applying the one or more image enhancement techniques further comprises reducing appearance of artifacts in a frame that the client device received from the remoting server, and wherein reducing appearance of artifacts in the received frame comprises:

detecting an edge in the frame;

determining whether the edge belongs to a user-interface object or is caused by artifacts;

in response to the edge being caused by artifacts, blurring the edge; and

in response to the belonging to a user-interface object, preserving the edge.

12. The non-transitory storage medium of claim 11 , wherein determining whether the edge belongs to the user-interface object or is caused by artifacts comprises:

determining whether a color variance along the edge is greater than a predetermined threshold; and

in response to the color variance being greater than the threshold, identifying the edge as being caused by artifacts.

13. A computer system, comprising:

a processor;

memory coupled to the processor; and

a storage device storing instructions which when loaded into the memory and executed by the processor cause the processor to perform a method, the method comprising:

receiving, by a client device, from a remoting server, a sequence of frame updates for a display screen;

determining, at the client device, a sequence of frames corresponding to the received frame updates;

monitoring, at the client device, the received sequence of frame updates to observe the available network bandwidth, frame refresh rate, and image quality corresponding to the received sequence of frame updates; and

executing an enhancement application on the client device, the enhancement application adaptively applying, at the client device, one or more image enhancing techniques to the sequence of frames based on at least one of:

the observed available network bandwidth;

the observed frame refresh rate; and

the observed image quality;

wherein applying the one or more image enhancement techniques comprises:

detecting a user interaction with a user-interface object;

generating a predicted frame based on the user interaction; and

appending the predicted frame to the received sequence of frame updates;

wherein generating the predicted frame based on the user interaction with the user-interface object comprises:

generating a predicted motion vector corresponding to the user interaction with the user-interface object;

generating the predicted frame based on the predicted motion vector; and

wherein the user interaction with the user-interface object comprises at least one of:

a window-scrolling operation;

a window-resizing operation;

a text entry into a text field;

a mouse-pointer movement;

an button click; and

a drag operation that moves the object to a new screen position.

14. The computer system of claim 13 , wherein adaptively applying, at the client device, the one or more image enhancing techniques comprises:

determining whether the observed available network bandwidth, observed frame refresh rate, or observed image quality is sufficiently low to apply the one or more image enhancing techniques; and

selecting an image enhancement technique based on a type of content distributed by the remoting server.

15. The computer system of claim 13 , wherein the method further comprises:

in response to the received frame updates being associated with a streamed video, applying an image enhancement technique that comprises interpolating a frame based on at least two buffered frames.

16. The computer system of claim 13 , wherein adaptively applying the one or more image enhancement techniques further comprises:

determining whether a respective image enhancement technique sufficiently improves user experience; and

in response to the image enhancement technique not sufficiently improving user experience, discontinuing application of the image enhancement technique.

17. The computer system of claim 13 , wherein applying the one or more image enhancement techniques further comprises reducing appearance of artifacts in a frame that the client device received from the remoting server, and wherein reducing appearance of artifacts in the received frame comprises:

detecting an edge in the frame;

determining whether the edge belongs to a user-interface object or is caused by artifacts;

in response to the edge being caused by artifacts, blurring the edge; and

in response to the belonging to a user-interface object, preserving the edge.

18. The computer system of claim 17 , wherein determining whether the edge belongs to the user-interface object or is caused by artifacts comprises:

determining whether a color variance along the edge is greater than a predetermined threshold; and

in response to the color variance being greater than the threshold, identifying the edge as being caused by artifacts.