Method for dynamic binary message decoding
Dynamic translation of data records is performed. A client of a translation service may request translation of binary data according to a binary data protocol to be output according to an output schema different from the binary data protocol. A dynamic translation engine may be configured to connect to an input data stream implementing the binary data protocol and connect to an output data consumer or storage device. The dynamic translation engine may then identify records on the source data stream according to the binary protocol, extract keys and values for individual fields of records, then output key-value pairs or alternative binary encodings to output records formatted according to the output schema. The binary data protocol and output schema may be dynamically defined and optionally updated during translation.
1 . A system, comprising:
one or more processors; and
a memory storing program instructions that, when executed on the one or more processors, implement a binary translator for a stream processing service of a provider network, the binary translator configured to:
receive, via a first programmatic interface, a request to decode a binary data stream, wherein the request identifies a binary schema and an output format, wherein the binary data stream comprises data for one or more events of a second programmatic interface, and responsive to receiving the request:
establish a streaming connection to the second programmatic interface of a data source;
identify an input record, corresponding to an event of the one or more events of the second programmatic interface, within the binary data stream received from the data source on behalf of a client according to the binary schema identified in the request;
determine respective keys and respective values for one or more fields of the input record according to the binary schema identified in the request; and
output, as key-value pairs, at least a portion of the determined values as an output record using the output format different than the binary schema.
2 . The system of claim 1 , wherein the binary schema comprises an error detection code field, and wherein the translator is further configured to verify the record prior to outputting at least of portion of the determined values.
3 . The system of claim 1 , wherein the output format is one or more of a JavaScript Object Notation (JSON) schema, a Yet Another Markup Language (YAML) schema, a Standard Generalized Markup Language (SGML) schema and an Extensible Markup Language (XML) schema, and wherein the output format identified in the request.
4 . The system of claim 1 , wherein at least one of the input record and output record comprise either encrypted or compressed data.
5 . A method, comprising:
performing by one or more computing devices implementing a translation engine:
receiving, via a first programmatic interface, a request to decode a data stream, wherein the request identifies a binary schema and an output format, wherein the data stream comprises data for one or more events of a second programmatic interface, and responsive to receiving the request:
establishing a streaming connection to the second programmatic interface of a data source;
identifying a record, corresponding to an event of the one or more events of the second programmatic interface, within the data stream received from the data source on behalf of a client according to the schema identified in the request;
determining respective values for one or more fields of the record according to the schema identified in the request; and
outputting at least a portion of the determined values of the record using the output format different than the schema.
6 . The method of claim 5 , wherein the method further comprises verifying the respective values for one or more fields of the record prior to outputting at least of portion of the determined values.
7 . The method of claim 6 , wherein the schema comprises a verification field, and wherein the method further comprises verifying the record prior to outputting at least of portion of the determined values.
8 . The method of claim 5 , wherein the output format is one or more of a JavaScript Object Notation (JSON) schema, a Yet Another Markup Language (YAML) schema, a Standard Generalized Markup Language (SGML) schema and an Extensible Markup Language (XML) schema, and wherein the output format identified in the request.
9 . The method of claim 5 , further comprising:
receiving, via the programmatic interface or another programmatic interface, a request to update the schema, and responsive to receiving the request:
performing the identifying, determining and outputting for at least another record of the one or more records within the data stream.
10 . The method of claim 5 , wherein outputting a value of the at least of portion of the determined values comprises:
determining a key associated with the value according to the schema; and
outputting the values as a key-value pair.
11 . The method of claim 5 , wherein the data stream is an internal message stream of an application of the client.
12 . The method of claim 5 , wherein the data stream is a human-readable data stream.
13 . The method of claim 5 , wherein the translation engine is implemented as part of a stream processing service of a provider network.
14 . One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more computing devices cause the one or more computing devices to implement a decoder to perform:
receiving, via an application programming interface (API), a request to decode a binary data stream, wherein the request identifies a binary protocol and an output format, wherein the binary data stream comprises data for one or more events of a programmatic interface, and responsive to receiving the request:
establishing a streaming connection to the programmatic interface of a data source;
identifying a record, corresponding to an event of the one or more events of the programmatic interface, within the binary data stream received from the data source on behalf of a client according to the binary protocol identified in the request;
determining respective values for one or more fields of the record according to the binary protocol identified in the request; and
outputting at least a portion of the determined values using the output format different than the binary protocol.
15 . The one or more non-transitory computer-accessible storage media of claim 14 , wherein the binary protocol comprises and validation field, and wherein the method further comprises verifying the record prior to outputting at least of portion of the determined values.
16 . The one or more non-transitory computer-accessible storage media of claim 14 , wherein the output format is one or more of a JavaScript Object Notation (JSON) schema, a Yet Another Markup Language (YAML) schema, a Standard Generalized Markup Language (SGML) schema and an Extensible Markup Language (XML) schema, and wherein the output format identified in the request.
17 . The one or more non-transitory computer-accessible storage media of claim 16 , wherein the output format is a Comma Separated Variable (CSV) format.
18 . The one or more non-transitory computer-accessible storage media of claim 14 , wherein outputting a value of the at least of portion of the determined values comprises:
determining a key associated with the value according to the binary schema/protocol; and
outputting the values as a key-value pair.
19 . The one or more non-transitory computer-accessible storage media of claim 14 , wherein the internal message stream is a bidirectional message stream of an API provided by an application of the client.
20 . The one or more non-transitory computer-accessible storage media of claim 14 , wherein the output format is a binary format.