WebRTC API redirection with screen sharing
A computing system includes a first client computing device, and a virtual desktop server. The first client computing device displays a local client surface, and a virtual client surface provided by the virtual desktop server. The virtual desktop server provides a real-time media application to provide real-time communications (RTC), and an API code redirection module to redirect intercepted APIs of the real-time media application intended for a native RTC engine based on redirection code injected into the real-time media application so that the portion of the real-time media application is redirected. The injected redirection code enumerates the local and virtual client surfaces. The first client computing device shares the local and virtual client surfaces with a second client computing device based on the intercepted APIs enumerating the local and virtual client surfaces.
1. A computing system comprising: a first client computing device configured to display a local client surface and a virtual client surface, with the local client surface and the virtual client surface to be shared with a second client computing device having peer-to-peer communications with the first client computing device;
a virtual desktop server comprising a processor configured to communicate with said first client computing device through a virtual channel to provide the virtual client surface, and comprising:
an application framework comprising a real-time media application to provide real-time communications (RTC), and a native RTC engine to execute a portion of the real-time media application when received by the native RTC engine, and
an API code redirection module to redirect intercepted APIs of the real-time media application intended for the native RTC engine based on redirection code injected into the real-time media application so that the portion of the real-time media application is redirected, with the injected redirection code enumerating the local and virtual client surfaces; and
said first client computing device comprising a client RTC API engine communicating configured to communicate with the API code redirection module through the virtual channel to execute the redirected portion of the real-time media application, with the redirected portion of the real-time media application invoked by the intercepted APIs corresponding to real-time media processing being off-loaded to said first client computing device, and configured to share the local and virtual client surfaces with the second client computing device based on the intercepted APIs enumerating the local and virtual client surfaces, wherein the redirected portion of the real-time media application is optimized to the capabilities of said first client computing device.
2. The computing system according to claim 1 wherein the real-time media application is configured to provide to said first client computing device a single list of enumerated sources of shared content for the local and virtual client surfaces.
3. The computing system according to claim 2 wherein the sources of shared content comprise at least one of applications and their windows, monitors and desktops.
4. The computing system according to claim 1 wherein the virtual client surface comprises at least one window that is to be shared with the second client computing device, and wherein said virtual desktop server is configured to provide pixels of the at least one window to the second client computing device.
5. The computing system according to claim 4 wherein said virtual desktop server is configured to compress the pixels of the at least one window.
6. The computing system according to claim 1 wherein the real-time media application is configured to prompt a user of said first client computing device to confirm that the local and virtual client surfaces are to be shared.
7. The computing system according to claim 1 wherein the virtual client surface comprises a plurality of overlapping windows, and wherein said first client computing device is configured to obtain, based on selection of one of the plurality of overlapping windows, pixels of the one selected window from said virtual desktop server.
8. The computing system according to claim 1 wherein said virtual desktop server is configured to share a whole virtual desktop of said virtual desktop server by capturing pixels of the virtual desktop and delivering the captured pixels to the second client computing device.
9. The computing system according to claim 1 wherein the virtual client surface comprises at least one window that is to be shared with the second client computing device, and wherein said virtual desktop server is configured to compress pixels of the at least one window before sending to said first client computing device for sharing with the second client computing device.
10. A method for operating a computing system comprising a virtual desktop server further comprising an application framework and an API code redirection module, with the application framework further comprising a real-time media application and a native RTC engine, and a first client computing device further comprising a client RTC API engine,
operating the first client computing device to display a local client surface and a virtual client surface, with the local client surface and the virtual client surface to be shared with a second client computing device having peer-to-peer communications with the first client computing device;
operating the virtual desktop server to communicate with the first client computing device through a virtual channel to provide the virtual client surface;
providing real-time communications (RTC) based on operation of the real-time media application, with a portion of the real-time media application to be executed by the native RTC engine when received by the native RTC engine;
redirecting by the API code redirection module intercepted APIs of the real-time media application intended for the native RTC engine based on redirection code injected into the real-time media application so that the portion of the real-time media application is redirected, with the injected redirection code enumerating the local and virtual client surfaces; and
operating the client RTC API engine communicating with the API code redirection module through the virtual channel to execute the redirected portion of the real-time media application, with the redirected portion of the real-time media application invoked by the intercepted APIs corresponding to real-time media processing being off-loaded to said first client computing device, and to share the local and virtual client surfaces with the second client computing device based on the intercepted APIs enumerating the local and virtual client surfaces, wherein the redirected portion of the real-time media application is optimized to the capabilities of said first client computing device.
11. The method according to claim 10 wherein the real-time media application provides to said first client computing device a single list of enumerated sources of shared content for the local and virtual client surfaces.
12. The method according to claim 11 wherein the sources of shared content comprise at least one of applications and their windows, monitors and desktops.
13. The method according to claim 10 wherein the virtual client surface comprises at least one window that is to be shared with the second client computing device, with pixels of the shared window to be provided by the virtual desktop server to the second client computing device.
14. The method according to claim 13 wherein the virtual desktop server compresses the pixels of the at least one window.
15. The method according to claim 10 wherein the real-time media application prompts a user of the first client computing device to confirm that the local and virtual client surfaces are to be shared.
16. The method according to claim 10 wherein the virtual client surface comprises a plurality of overlapping windows, and if one of the overlapping windows is selected to be shared with the second client computing device, then pixels of the selected window are obtained from the virtual desktop server.
17. The method according to claim 10 wherein if a whole virtual desktop of said virtual desktop server is to be shared, then pixels of the virtual desktop are captured on the first client computing device to be delivered to the second client computing device.
18. The method according to claim 10 wherein the virtual client surface comprises at least one window that is to be shared with the second client computing device, with the pixels of the shared window to be compressed by the virtual desktop server before sending to the first client computing device for sharing with the second client computing device.
19. A non-transitory computer readable medium for operating a virtual desktop server within a computing system comprising a client computing device comprising a client RTC API engine, with the first client computing device to display a local client surface and a virtual client surface, with the local client surface and the virtual client surface to be shared with a second client computing device having peer-to-peer communications with the first client computing device, with the virtual desktop server comprising an application framework and an API code redirection module, with the application framework comprising a real-time media application and a native RTC engine, and with the non-transitory computer readable medium having a plurality of computer executable instructions for causing the virtual desktop server to perform steps comprising:
communicating with the first client computing device through a virtual channel to provide the virtual client surface;
providing real-time communications (RTC) based on operation of the real-time media application, with a portion of the real-time media application to be executed by the native RTC engine when received by the native RTC engine; and
redirecting by the API code redirection module intercepted APIs of the real-time media application intended for the native RTC engine based on redirection code injected into the real-time media application so that the client RTC API engine communicating with the API code redirection module through a virtual channel executes the redirected portion of the real-time media application, with the redirected portion of the real-time media application invoked by the intercepted APIs corresponding to real-time media processing being off-loaded to said first client computing device, and with the injected redirection code enumerating the local and virtual client surfaces so that the first client computing device shares the local and virtual client surfaces with the second client computing device based on the intercepted APIs enumerating the local and virtual client surfaces, wherein the redirected portion of the real-time media application is optimized to the capabilities of said first client computing device.
20. The non-transitory computer readable medium according to claim 19 wherein the real-time media application provides to the first client computing device a single list of enumerated sources of shared content for the local and virtual client surfaces.