Optimization of direct memory accesses for real-time data streaming applications
Disclosed are apparatuses, systems, and techniques that improve memory and computational efficiency of remote direct memory accesses into a memory of a graphics processing unit. The techniques include but are not limited to receiving packets with video frame data, storing the plurality of packets in a memory of a network controller, processing the packets to obtain unit(s) of the video frame, storing the unit(s) representative of the video frame in a memory of a graphics processing unit (GPU), and extracting the data of the video frame from the units representative of the video frame, stored in the memory of the GPU, to render the video frame.
1 . A method comprising:
receiving, using a network controller of a first device, from a second device remote to the network controller of the first device and executing a remote direct memory access operation to a graphics processing unit (GPU) of the first device (GPU RDMA operation) a plurality of packets comprising data of a video frame, wherein the plurality of packets are associated, by the second device, with at least one memory address of a memory of the GPU, and wherein the network controller is configured to perform packet-to-video frame processing;
storing the plurality of packets using a memory of the network controller;
processing, using the network controller, the plurality of packets to obtain one or more units representative of the video frame;
completing the GPU RDMA operation by storing the one or more units representative of the video frame in the memory of the GPU; and
extracting, using the GPU, the data of the video frame from the one or more units representative of the video frame stored in the memory of the GPU to render the video frame.
2 . The method of claim 1 , wherein one or more packets of the plurality of packets comprise Real-time Transport Protocol (RTP) packets.
3 . The method of claim 1 , wherein the one or more units representative of the video frame comprise at least one of an H.264 network abstraction layer unit (NALU), an H.265 NALU, an AV1 Open Bitstream unit (OBU), or a VP9 OBU.
4 . The method of claim 1 , wherein processing the plurality of packets comprises:
processing two or more packets of the plurality of packets to obtain a combined unit representative of the video frame, wherein the combined unit comprises a unit header generated based at least on headers of at least two packets of the two or more packets, and wherein a data payload of the combined unit is obtained by combining data payloads of the at least two packets.
5 . The method of claim 1 , wherein a subset of the one or more units representative of the video frame is obtained by processing a single packet of the plurality of packets, wherein a unit header of at least one unit of the subset is generated based at least on a header of the single packet, and wherein a data payload of the at least one unit of the subset is obtained by splitting a data payload of the single packet.
6 . The method of claim 1 , wherein processing the plurality of packets to obtain one or more units representative of the video frame is performed using a coprocessor of the network controller.
7 . The method of claim 1 , wherein extracting the data of the video frame comprises:
decoding the one or more units representative of the video frame in accordance with a codec standard used to encode the data of the video frame.
8 . The method of claim 1 , wherein processing the plurality of packets is performed responsive to instructions for parsing of headers and data payloads of the plurality of packets, wherein the instructions are provided to a parsing logic from an application library via an application programming interface (API).
9 . A system comprising:
a graphics processing unit (GPU) comprising:
a GPU memory;
a network controller (NC) configured to perform packet-to-video frame processing and coupled to the GPU memory, the NC comprising:
an NC memory; and
a processing device, communicatively coupled to the NC memory, the processing device to:
receive, using the NC, from an external device remote to the NC, executing a remote direct memory access operation to the GPU (GPU RDMA operation), a plurality of packets comprising data of a video frame, wherein the plurality of packets are associated, by the external device, with at least one memory address of the GPU memory;
store the plurality of packets using the NC memory;
process the plurality of packets to extract one or more units representative of the video frame; and
complete the GPU RDMA operation by storing the one or more units representative of the video frame in the GPU memory,
wherein the GPU is to:
render the video frame using the one or more units representative of the video frame stored in the GPU memory.
10 . The system of claim 9 , wherein one or more packets of the plurality of packets comprise Real-time Transport Protocol (RTP) packets.
11 . The system of claim 9 , wherein the one or more units representative of the video frame comprise at least one of an H.264 network abstraction layer unit (NALU), an H.265 NALU, an AV1 Open Bitstream unit (OBU), or a VP9 OBU.
12 . The system of claim 9 , wherein to process the plurality of packets, the processing device is to:
obtain a combined unit representative of the video frame, wherein the combined unit comprises a unit header generated based on headers of each of two or more packets of the plurality of packets, and wherein a data payload of the combined unit is obtained by combining data payloads of the two or more packets.
13 . The system of claim 9 , wherein to extract a subset of the one or more units representative of the video frame, the processing device is to:
process a single packet of the plurality of packets;
generate a unit header of at least one unit of the subset based on a header of the single packet; and
split a data payload of the single packet to obtain a data payload of the at least one unit of the subset.
14 . The system of claim 9 , wherein to render the video frame, the GPU is to decode the one or more units representative of the video frame in accordance with a codec standard used to encode the data of the video frame.
15 . The system of claim 9 , wherein the processing device is to process the plurality of packets responsive to instructions for parsing of headers and data payloads of
the plurality of packets, wherein the instructions are provided from an application library via an application programming interface (API).
16 . A system comprising:
one or more processing devices to:
receive, via a network controller, from an external device remote to the network controller executing a remote direct memory access operation to a graphics processing unit (GPU) of the system (GPU RDMA operation), a plurality of packets comprising data of a video frame, wherein the plurality of packets are associated, by the external device, with at least one memory address of a memory of the GPU, and wherein the network controller is configured to perform packet-to-video frame processing;
store the plurality of packets using a memory of the network controller;
process, using the network controller, the plurality of packets to obtain one or more units representative of the video frame;
complete the GPU RDMA operation by storing the one or more units representative of the video frame in the memory of the GPU; and
extract, using the GPU, the data of the video frame from the one or more units representative of the video frame stored using the memory of the GPU to render the video frame.
17 . The system of claim 16 , wherein one or more packets of the plurality of packets comprise Real-time Transport Protocol (RTP) packets.
18 . The system of claim 16 , wherein the one or more units representative of the video frame comprise at least one of an H.264 network abstraction layer unit (NALU), an H.265 NALU, an AV1 Open Bitstream unit (OBU), or a VP9 OBU.
19 . The system of claim 16 , wherein to process the plurality of packets, the one or more processing devices are to:
obtain a combined unit representative of the video frame, wherein the combined unit comprises a unit header generated based on headers of at least two packets of the plurality of packets, and wherein a data payload of the combined unit is obtained by combining data payloads of the at least two packets.
20 . The system of claim 16 , wherein to extract a subset of the one or more units representative of the video frame, the one or more processing devices are to:
process a single packet of the plurality of packets;
generate a unit header of at least one unit of the subset based on a header of the single packet; and
split a data payload of the single packet to obtain a data payload of the at least one unit of the subset.
21 . The system of claim 16 , wherein the system comprises at least one of:
a system to provide streaming services;
a system to provide video monitoring services;
a system to provide computer-vision based services;
a system to provide artificial intelligence and machine learning services;
a system to provide mapping services;
a system to provide autonomous machine control or operation services;
a system to provide online gaming services;
a system to provide database services;
a system to provide online library services;
a system to provide cloud-based computing services;
a system to provide content creation for 3D assets services;
a system to provide conversational AI services;
a system to provide virtual reality or augmented reality services;
a system to provide digital avatar services; or
a system to provide human-machine interface services.