WebRTC API redirection with window monitoring/overlay detection
A computing system includes at least one video source, a virtual desktop server and a client computing device. The virtual desktop server includes a real-time media application to provide real-time communications (RTC), a native RTC engine, a geometry tracking module, and an API code redirection module to redirect intercepted APIs of the real-time media application intended for the native RTC engine based on redirection code injected into the real-time media application. The injected redirection code defines a placeholder to indicate positioning geometry of a video stream within an RTC window. The geometry tracking module detects the placeholder within the injected redirection code. The client computing device includes a display composition module to receive the video stream and the positioning geometry of the placeholder, and to overlay the video stream over the placeholder within the displayed RTC window based on the positioning geometry.
1. A computing system comprising:
at least one video source to provide at least one video stream;
a virtual desktop server comprising:
an application framework comprising a real-time media application to provide real-time communications (RTC), and a native RTC engine to execute a portion of the real-time media application when received by the native RTC engine,
an API code redirection module to redirect intercepted APIs of the real-time media application intended for the native RTC engine based on redirection code injected into the real-time media application so that the portion of the real-time media application is redirected, with the injected redirection code defining at least one placeholder to indicate positioning geometry of the at least one video stream within an RTC window, and
a geometry tracking module to detect the at least one placeholder within the injected redirection code, and to provide the positioning geometry associated therewith; and
a client computing device comprising
a display to display the RTC window,
a client RTC API engine communicating with the API code redirection module through a virtual channel to execute the redirected portion of the real-time media application, and
and a display composition module to receive the at least one video stream and the positioning geometry of the at least one placeholder, and to overlay the at least one video stream over the at least one placeholder within the displayed RTC window based on the positioning geometry.
2. The computing system according to claim 1 wherein the geometry tracking module provides the positioning geometry to be included in the redirected portion of the real-time media application to the client RTC API engine, with the display composition module receiving the positioning geometry from the client RTC API engine.
3. The computing system according to claim 1 wherein the geometry tracking module provides the positioning geometry directly to the display composition module over a different virtual channel.
4. The computing system according to claim 1 wherein the displayed RTC window includes a composited display of non-accelerated graphics and accelerated graphics, with the accelerated graphics defined by the video stream.
5. The computing system according to claim 4 wherein the injected redirection code only generates the non-accelerated graphics, and wherein the geometry tracking module analyzes the non-accelerated graphics for the at least one placeholder.
6. The computing system according to claim 1 wherein the injected redirection code defines the at least one placeholder with at least one color, and wherein the geometry tracking module uses the at least one color to detect the at least one placeholder.
7. The computing system according to claim 6 wherein the injected redirection code changes the at least one color to at least one different color after detection by the geometry tracking module so that the at least one placeholder is displayed on the RTC window is in the at least one different color.
8. The computing system according to claim 1 wherein the injected redirection code defines the at least one placeholder with at least one pattern, and wherein the geometry tracking module uses the at least one pattern to detect the at least one placeholder.
9. The computing system according to claim 1 wherein the injected redirection code defines the at least one placeholder with at least one bar code, and wherein the geometry tracking module uses the at least one bar code to detect the at least one placeholder.
10. The computing system according to claim 9 wherein the at least one bar code assigns at least one ID for the at least one video stream.
11. The computing system according to claim 1 wherein the injected redirection code assigns at least one ID for the at least one placeholder that corresponds to the at least one video stream, and wherein the geometry tracking module associates the at least one ID with the at least one video stream.
12. The computing system according to claim 1 wherein the at least one video source comprises a camera coupled to said client computing device.
13. The computing system according to claim 1 wherein the at least one video source comprises a camera coupled to a different client computing device in peer-to-peer communications with said client computing device.
14. A method for operating a computing system comprising at least one video source, a virtual desktop server and a client computing device comprising a client RTC API engine, a display and a display composition module, with the virtual desktop server comprising a geometry tracking module, an application framework and an API code redirection module, and with the application framework comprising a real-time media application and a native RTC engine, the method comprising:
providing at least one video stream from the at least one video source;
providing real-time communications (RTC) based on operation of the real-time media application, with a portion of the real-time media application to be executed by the native RTC engine when received by the native RTC engine;
redirecting by the API code redirection module intercepted APIs of the real-time media application intended for the native RTC engine based on redirection code injected into the real-time media application so that the portion of the real-time media application is redirected, with the injected redirection code defining at least one placeholder to indicate positioning geometry of the at least one video stream within an RTC window;
operating the geometry tracking module to detect the at least one placeholder within the injected redirection code, and to provide the positioning geometry associated therewith;
displaying the RTC window on the display;
operating the client RTC API engine communicating with the API code redirection module through a virtual channel to execute the redirected portion of the real-time media application; and
operating the display composition module to receive the at least one video stream and the positioning geometry of the at least one placeholder, and to overlay the at least one video stream over the at least one placeholder within the displayed RTC window based on the positioning geometry.
15. The method according to claim 14 wherein the displayed RTC window includes a composited display of non-accelerated graphics and accelerated graphics, with the accelerated graphics defined by the video stream.
16. The method according to claim 15 wherein the injected redirection code only generates the non-accelerated graphics, and wherein the geometry tracking module analyzes the non-accelerated graphics for the at least one placeholder.
17. The method according to claim 14 wherein the injected redirection code defines the at least one placeholder with at least one color, and wherein the geometry tracking module uses the at least one color to detect the at least one placeholder.
18. The method according to claim 14 wherein the injected redirection code defines the at least one placeholder with at least one pattern, and wherein the geometry tracking module uses the at least one pattern to detect the at least one placeholder.
19. The method according to claim 14 wherein the injected redirection code defines the at least one placeholder with at least one bar code, and wherein the geometry tracking module uses the at least one bar code to detect the at least one placeholder.
20. A non-transitory computer readable medium for operating a virtual desktop server within a computing system comprising at least one video source, and a client computing device comprising a display, a client RTC API engine and a display composition module, with the virtual desktop server comprising a geometry tracking module, an application framework and an API code redirection module, with the application framework comprising a real-time media application and a native RTC engine, and with the non-transitory computer readable medium having a plurality of computer executable instructions for causing the virtual desktop server to perform steps comprising:
providing real-time communications (RTC) based on operation of the real-time media application, with a portion of the real-time media application to be executed by the native RTC engine when received by the native RTC engine;
redirecting by the API code redirection module intercepted APIs of the real-time media application intended for the native RTC engine based on redirection code injected into the real-time media application so that the client RTC API engine communicating with the API code redirection module through a virtual channel executes the redirected portion of the real-time media application, with the injected redirection code defining at least one placeholder to indicate positioning geometry of the at least one video stream within an RTC window; and
operating the geometry tracking module to detect the at least one placeholder within the injected redirection code, and to provide the positioning geometry associated therewith so that the client computing device operates the client RTC API engine to communicate with the API code redirection module through a virtual channel to execute the redirected portion of the real-time media application, and for the display composition module to receive the at least one video stream and the positioning geometry of the at least one placeholder, and to overlay the at least one video stream over the at least one placeholder within the displayed RTC window based on the positioning geometry.