Ultra-low latency remote application access
An ultra-low latency remote access system may stream video images from a remote application server to a client device, but may render a cursor separately on the client device. The remote application may send cursor bitmaps and other cursor information to the client device, which may be rendered by the client device to give a near-native response time and feel to the user experience. The remote access system may operate in a browser environment, with executable code in the browser to capture user input, including pointer and keyboard inputs. Executable code on the application server may encode screen images into video, which may be transmitted and rendered on the client device. In some cases, the rendered video may be larger than the displayable area of the client device, allowing for panning and zooming to be performed on the client device.
1. An ultra-low latency server system comprising:
a first hardware platform comprising at least one processor and an operating system;
a first network connection;
an ultra-low latency remote service executing within said operating system and on said at least one processor, said ultra-low latency remote service that:
receives cursor information from a remote browser operating on a second hardware platform, wherein the cursor information comprises location information associated with the cursor in the remote browser;
transmits said cursor information to said operating system executing an application;
receives displayable screen dimensions from said remote browser;
receives a stream of rendered images generated by said operating system, said rendered images being configured to be displayed within said displayable screen dimensions;
determines a cursor location within said stream of rendered images;
changes said stream of rendered images at said cursor location to create a minimized cursor, generating a stream of updated rendered images;
generates a video stream comprising said stream of updated rendered images, said video stream comprising said minimized cursor;
generates one or more alphanumeric cursor identifiers that identifies a type of cursor to be displayed at the remote browser based on the location information, wherein if the cursor in the remote browser is at a first location, a first alphanumeric cursor identifier associated with a first cursor is generated, and if the cursor is at a second location, a second alphanumeric cursor identifier associated with a second cursor is generated, and
transmits said video stream and the one or more alphanumeric cursor identifiers over said first network connection to said remote browser on said second hardware platform.
2. The ultra-low latency server system of claim 1 , said ultra-low latency remote service that further:
receives a connection request from said remote browser, said connection request comprising said displayable screen dimensions on said remote browser.
3. The ultra-low latency server system of claim 2 , said ultra-low latency remote service that further:
generates said video stream to match said displayable screen dimensions.
4. The ultra-low latency server system of claim 2 , said ultra-low latency remote service that further:
causes said stream of rendered images to match said displayable screen dimensions.
5. The ultra-low latency server system of claim 2 , said displayable screen dimensions being smaller than dimensions of said rendered image.
6. The ultra-low latency server system of claim 5 , said remote browser being configured to pan said video stream.
7. The ultra-low latency server system of claim 5 , said remote browser being configured to zoom said video stream.
8. The ultra-low latency server system of claim 1 , said ultra-low latency remote service that further:
receives an updated displayable screen dimensions from said remote browser; and
changes said video stream to match said updated displayable screen dimensions.
9. The ultra-low latency server system of claim 1 , said minimized cursor being a complete removal of said cursor from said rendered image.
10. The ultra-low latency server system of claim 1 , said minimized cursor being a smaller cursor than said cursor from said rendered image.
11. The ultra-low latency server system of claim 1 , said ultra-low latency remote service that further:
determines a cursor identifier; and
transmits said cursor identifier.
12. The ultra-low latency server system of claim 1 , said application being a computer desktop.
13. The ultra-low latency server of claim 12 , said computer desktop comprising a second application rendered on said computer desktop.
14. A method performed on a hardware platform comprising at least one
processor, said method comprising:
receiving cursor information from a remote browser operating on a second hardware platform, said cursor information comprising input information and location information associated with the cursor in the remote browser;
transmitting said cursor information to an operating system executing on said hardware platform, said operating system executing an application;
receiving a stream of rendered images of said application, said stream of rendered images being updated by a response by said application to said input information;
identifying a rendered cursor within said stream of rendered images;
removing said rendered cursor from said stream of rendered images to create an updated stream of rendered images;
generating a video stream from said stream of updated rendered images;
generating one or more alphanumeric cursor identifiers that identifies a type of cursor to be displayed at the remote browser based on the location information, wherein if the cursor in the remote browser is at a first location, a first alphanumeric cursor identifier associated with a first cursor is generated, and if the cursor is at a second location, a second alphanumeric cursor identifier associated with a second cursor is generated, and
transmitting said video stream and the one or more alphanumeric cursor identifiers over said first network connection to said remote browser.
15. The method of claim 14 , said application being executed on said at least one processor.
16. The method of claim 14 further comprising:
receiving a connection request from said remote browser, said connection request comprising a displayable screen dimensions on said remote browser.
17. The method of claim 16 further comprising:
generating said video stream to match said displayable screen dimensions.
18. The method of claim 16 further comprising:
causing said rendered image to match said displayable screen dimensions.
19. The method of claim 14 , said application being a computer desktop.
20. The method of claim 19 , said computer desktop comprising a second application rendered on said computer desktop.
21. A method performed on a first hardware platform comprising at least
one processor, said method comprising:
receiving a video stream and one or more alphanumeric cursor identifiers from an application server operating on a second hardware platform, said video stream comprising a stream of rendered images, said rendered images comprising successive renderings of an application on a computer desktop, wherein the one or more alphanumeric cursor identifiers identifies types of cursor to be displayed at the remote browser based on the location information;
displaying said video stream within a display viewport of a remote browser operating on said first hardware platform;
capturing cursor location coordinates said first hardware platform, said cursor location coordinates being within said display viewport;
capturing cursor input from said display viewport;
rendering said cursor at said cursor location on said first hardware platform wherein the cursor to be rendered is selected based on the one or more alphanumeric cursor identifiers received from the application server; and
transmitting said cursor location coordinates and said cursor input to said application server.
22. The method of claim 21 further comprising:
receiving a current cursor image and causing said current cursor image to be displayed within said display viewport of said remote browser.
23. The method of claim 22 , said client device having a current cursor location, said current cursor image being displayed as a cursor at said current cursor location when said current cursor location is within said display viewport of said remote browser.
24. The method of claim 22 , said client device having a local cursor image, said local cursor image being displayed as said cursor at said current cursor location when said current cursor location is outside said display viewport of said remote browser.