Data processing and transmission using hardware serialization and deserialization functions
Various embodiments described herein support or provide for data processing and data transmission using hardware serialization and hardware deserialization functions, such as receiving structured data from a software application, using the data communication hardware to perform operations including: serializing the structured data to generate a serialized byte string based on the one or more hardware parameters; generate one or more data packets based on the serialized byte string; appending one or more headers to the one or more data packets; and transmitting the one or more data packets to a device based on a plurality of network parameters.
1. A method comprising:
receiving, at data communication hardware of a first electronic device, structured data from a software application that is operating by a processor of the first electronic device, the structured data being associated with one or more hardware parameters; and
using the data communication hardware to perform operations comprising:
serializing the structured data to generate a serialized byte string based on the one or more hardware parameters,
generate one or more data packets based on the serialized byte string,
appending one or more headers to the one or more data packets, the one or more headers including a plurality of network parameters, and
transmitting, via a network operatively coupled to the data communication hardware, the one or more headers and the one or more data packets to a second electronic device based on the plurality of network parameters.
2. The method of claim 1 , wherein the data communication hardware performs the operations further comprise:
generating one or more messages that include the serialized byte string, each message including a plurality of variables;
determining a variable length value for each of the plurality of variables; and
encoding the variable length value for each of the plurality of variables into the serialized byte string, the variable length value representing a length of data content associated with a corresponding variable.
3. The method of claim 2 , further comprising:
determining a message length value based on a sum of a plurality of variable length values determined for the plurality of variables.
4. The method of claim 3 , wherein the data communication hardware performs the operations further comprise:
generating a lengths list based on the plurality of variable length values determined for the plurality of variables.
5. The method of claim 4 , wherein an order of the plurality of variable length values arranged in the lengths list is different from an order of the plurality of variable length values used for serialization.
6. The method of claim 1 , wherein the data communication hardware performs the operations further comprise:
generating one or more messages that include the serialized byte string;
generating one or more tags for the one or more messages, each tag representing a message type and a message value of a corresponding message; and
encoding the one or more tags into the serialized byte string.
7. The method of claim 1 , wherein the structured data comprises one or more data structures, the method further comprising:
using the data communication hardware associated with the software application to convert the one or more data structures into the serialized byte string.
8. The method of claim 1 , wherein the data communication hardware of the first electronic device comprises a network interface controller of the first electronic device.
9. The method of claim 1 , wherein the one or more headers comprise one or more serialization protocol-specific headers and one or more network parameters that are associated with one or more network protocols, the one or more network protocols comprising at least one of Remote Direct Memory Access (RDMA) over Converged Ethernet Version 2 (RoCEv2) network protocol, User Datagram Protocol (UDP), Internet Protocol version 4 (IPv4), Internet Protocol version 4 (IPv6), or Ethernet protocol.
10. The method of claim 9 , wherein the one or more serialization protocol-specific headers comprise one or more protocol parameters associated with at least one of Scalable Service-Oriented Middleware over IP (SOME/IP) protocol, Protocol Buffers protocol, or Thrift protocol.
11. A system comprising:
a memory storing instructions; and
one or more hardware processors communicatively coupled to the memory and configured by the instructions to perform operations comprising:
receiving, at data communication hardware of a first electronic device, structured data from a software application that is operating by a processor of the first electronic device, the structured data being associated with one or more hardware parameters; and
using the data communication hardware to perform operations comprising:
serializing the structured data to generate a serialized byte string based on the one or more hardware parameters,
generate one or more data packets based on the serialized byte string,
appending one or more headers to the one or more data packets, the one or more headers including a plurality of network parameters, and
transmitting, via a network operatively coupled to the data communication hardware, the one or more headers and the one or more data packets to a second electronic device based on the plurality of network parameters.
12. The system of claim 11 , wherein the data communication hardware performs the operations further comprise:
generating one or more messages that include the serialized byte string, each message including a plurality of variables;
determining a variable length value for each of the plurality of variables; and
encoding the variable length value for each of the plurality of variables into the serialized byte string, the variable length value representing a length of data content associated with a corresponding variable.
13. The system of claim 12 , wherein the operations further comprise:
determining a message length value based on a sum of a plurality of variable length values determined for the plurality of variables.
14. The system of claim 13 , wherein the data communication hardware performs the operations further comprise:
generating a lengths list based on the plurality of variable length values determined for the plurality of variables.
15. The system of claim 14 , wherein an order of the plurality of variable length values arranged in the lengths list is different from an order of the plurality of variable length values used for serialization.
16. The system of claim 11 , wherein the data communication hardware performs the operations further comprise:
generating one or more messages that include the serialized byte string;
generating one or more tags for the one or more messages, each tag representing a message type and a message value of a corresponding message; and
encoding the one or more tags into the serialized byte string.
17. The system of claim 11 , wherein the structured data comprises one or more data structures, and wherein the operations further comprise:
using the data communication hardware associated with the software application to convert the one or more data structures into the serialized byte string.
18. The system of claim 11 , wherein the data communication hardware of the first electronic device comprises a network interface controller of the first electronic device.
19. The system of claim 11 , wherein the one or more headers comprise one or more serialization protocol-specific headers and one or more network parameters that are associated with one or more network protocols, the one or more network protocols comprising at least one of Remote Direct Memory Access (RDMA) over Converged Ethernet Version 2 (RoCEv2) network protocol, User Datagram Protocol (UDP), Internet Protocol version 4 (IPv4), Internet Protocol version 4 (IPv6), or Ethernet protocol.
20. A non-transitory computer-readable medium comprising instructions that, when executed by a hardware processor of a device, cause the device to perform operations comprising:
receiving, at data communication hardware of a first electronic device, structured data from a software application that is operating by a processor of the first electronic device, the structured data being associated with one or more hardware parameters; and
using the data communication hardware to perform operations comprising:
serializing the structured data to generate a serialized byte string based on the one or more hardware parameters,
generate one or more data packets based on the serialized byte string,
appending one or more headers to the one or more data packets, the one or more headers including a plurality of network parameters, and
transmitting, via a network operatively coupled to the data communication hardware, the one or more headers and the one or more data packets to a second electronic device based on the plurality of network parameters.