System, method, and computer program product for remote graphics processing
View Patent ↗A system, method, and computer program product are provided for remote rendering of computer graphics. The system includes a graphics application program resident at a remote server. The graphics application is invoked by a user or process located at a client. The invoked graphics application proceeds to issue graphics instructions. The graphics instructions are received by a remote rendering control system. Given that the client and server differ with respect to graphics context and image processing capability, the remote rendering control system modifies the graphics instructions in order to accommodate these differences. The modified graphics instructions are sent to graphics rendering resources, which produce one or more rendered images. Data representing the rendered images is written to one or more frame buffers. The remote rendering control system then reads this image data from the frame buffers. The image data is transmitted to the client for display or processing. In an embodiment of the system, the image data is compressed before being transmitted to the client. In such an embodiment, the steps of rendering, compression, and transmission can be performed asynchronously in a pipelined manner.
1. A system comprising:
a server system comprising one or more processors and memory storing one or more programs executable by the one or more processors, the one or more programs including instructions for:
receiving, at a graphics application, a command from a client, the command related to a graphics rendering operation performed by the graphics application, and, responsive to the command, generating an application programming interface call for producing rendered image data;
receiving, at a remote rendering control system, the application programming interface call from the graphics application and modifying one or more parameters of the application programming interface call based upon one or more differences between the client and the server in at least one of graphics processing capabilities and graphics contexts;
performing, with a rendering process, the rendering operation using the modified parameters; and
transmitting, with a transmission process, the rendered image data to the client for display without rendering the image data on the client;
wherein the remote rendering control system compresses, by a compression process, the image data and wherein the rendering process of a first block of data occurs during a first time period, the compression process of a second block of data occurs during a second time period, and the transmission process of a third block of data occurs during a third time period so that the rendering process, the compression process, and the transmission process are pipelined; and
wherein the remote rendering control system discards the first block of data if the rendering process of the first block of data is completed before the compression process of the second block of data and discards the second block of data if the compression process of the second block of data is completed before the transmission process of the third block of data.
2. The system of claim 1 , wherein the application programming interface is an OPENGL application programming interface.
3. The system of claim 1 , wherein the remote rendering control system converts the at least one of the graphics processing capabilities and the graphics contexts appearing in the application programming interface call from the at least one of the graphics processing capabilities and the graphics contexts of the server to the at least one of the imaging processing capabilities and the graphics contexts of the client to generate the modified application programming interface call based upon the differences, between the client and the server, in the at least one of the graphics processing capabilities and the graphic contexts.
4. The system of claim 1 , wherein the remote rendering control system further modifies the application programming interface call by modifying function calls and references in the application programming interface call to make them suitable to the client.
5. The system of claim 4 , wherein the remote rendering control system is further intercepts the function calls and references in the application programming interface call.
6. A method for rendering image data, performed at an electronic device having one or more processors and memory storing one or more programs executable by the one or more processors, the method comprising:
receiving, at a server, a command from a client, the command related to a graphics rendering operation performed by the graphics application;
generating, at the server and in response to the command, an application programming interface call for producing rendered image data;
modifying, at the server, one or more parameters of the application programming interface call based upon one or more differences between the client and the server in at least one of graphics processing capabilities and graphics contexts;
performing with a rendering operation using the modified parameters;
transmitting, with a transmission process, the rendered image data to the client for display without rendering the image data on the client;
compressing, by a compression process at the server, the image data, wherein the rendering process of a first block of data occurs during a first time period, the compression of a second block of data occurs during a second time period, and the transmission process of a third block of data occurs during a third time period so that the rendering process, the compression process, and the transmission process are pipelined;
discarding, at the server, the first block of data if the rendering process of the first block of data is completed before the compression process of the second block of data; and
discarding, at the server, the second block of data if the compression process of the second block of data is completed before the transmission process of the third block of data.
7. The method of claim 6 , wherein modifying the one or more parameters of the application programming interface call comprises converting the at least one of the graphics processing capabilities and the graphics contexts appearing in the application programming interface call from the at least one of the graphics processing capabilities and the graphics contexts of the server to the at least one of the graphics processing capabilities and the graphics contexts of the client.
8. The method of claim 6 , wherein modifying the application programming interface call further includes modifying function calls and references in the application programming interface call to make them suitable to the client.
9. The method of claim 8 , further including intercepting the function calls and references in the application programming interface call.
10. A non-transitory computer readable storage medium having stored thereon computer-executable instructions that, responsive to execution by a computing device, cause the computing device to perform operations comprising:
receiving a command from a client, the command related to a graphics rendering operation performed by the graphics application;
generating, in response to the command, an application programming interface call for producing rendered image data;
modifying one or more parameters of the application programming interface call based upon one or more differences between the client and a server in at least one of graphics processing capabilities and graphics contexts;
performing, with a rendering process, the rendering operation using the modified parameters;
transmitting, with a transmission process, the rendered image data to the client for display without rendering the image data on the client;
compressing the image data, wherein the rendering the image data for a first block occurs during a first time period, the compressing for a second block of data occurs during a second time period, and the transmitting the image data for a third block of data occurs during a third time period so that the rendering, the compressing, and the transmitting are pipelined;
discarding the first block of data if the rendering the image data for the first block of data is completed before the compressing for the second block of data, and
discarding the second block of data if the compressing for the second block of data is completed before the transmitting the image of data for the third block of data.
11. The non-transitory computer readable storage medium of claim 10 , wherein the application programming interface is an OPENGL application programming interface.
12. The non-transitory computer readable storage medium of claim 10 , wherein modifying the one or more parameters of the application programming interface call comprises converting the at least one of the graphics processing capabilities and the graphics contexts appearing in the application programming interface call from the at least one of the graphics processing capabilities and the graphics contexts of the server to the at least one of the graphics processing capabilities and the graphics contexts of the client.
13. The non-transitory computer readable storage medium of claim 10 , wherein modifying the application programming interface call further includes modifying function calls and references in the application programming interface call to make them suitable to the client.
14. The non-transitory computer readable storage medium of claim 13 , further including intercepting the function calls and references in the application programming interface call.