Encoding and transmitting stream data without prior knowledge of data size
Implementations are provided herein for encoding and transmitting streaming data from a client application to a server for storage without prior knowledge of the size of the streaming data. A header can be sent that includes a batch size chunk size. Raw streaming data can be packaged into the chunk. Chunks can be packaged and sent at any time prior to filling up with streaming data, by padding the chunk and including a footer that delineates the amount of raw stream data in the chunk. Chunks that are full can have a footer that delineates the entire chunk is raw stream data. It can be appreciated that you do not need to buffer data on the client side as chunks do not need to be full to send. Latency on processing streaming data can also be reduced by limited or eliminated buffering.
1. A method comprising:
ingesting raw stream data by a client application;
generating a chunk header that includes a batch size chunk payload size, wherein generating the chunk header includes determining a total batch size and setting the batch size chunk payload size based on the total batch size;
writing the raw stream data into each of a plurality of chunks, wherein the raw stream data fills a first portion of each chunk, the first portion corresponding to an offset related to the batch size chunk payload size;
filling a second portion of each chunk with padded data;
generating a chunk footer for each chunk that delineates an amount of the raw stream data written into each chunk;
in response to a size of each chunk ready for sending being less than the batch size chunk payload size, padding each chunk with zeroes to fill each chunk to the batch size chunk payload size;
generating a data size integer based on the size of each chunk and the padding; and
using the data size integer as each chunk footer; and
sending each chunk to a controller for storage without buffering at the client application and based on a time associated with a dynamic ingestion of the raw stream data.
2. The method of claim 1 , further comprising:
in response to the raw stream data filling each chunk to an offset associated with the batch size chunk payload size, using the batch size chunk payload size as each chunk footer.
3. The method of claim 1 , wherein the dynamically generating each chunk footer and dynamically sending each chunk to the controller for storage occur based on a time associated with the dynamically ingesting raw stream data.
4. The method of claim 1 , wherein the batch size chunk payload size is 32,000 bytes.
5. The method of claim 1 , further comprising:
receiving each chunk by the controller;
determining a stream data size based on each chunk header and each chunk footer; and
storing a subset of each chunk in storage based on the stream data size.
6. A system comprising at least one storage device and at least one hardware processor configured to:
ingest raw stream data by a client application;
generate a chunk header that includes a batch size chunk payload size for each of a plurality of chunks, wherein generating the chunk header includes determining a total batch size and setting the batch size chunk payload size based on the total batch size;
write the raw stream data into each chunk of the plurality of chunks, wherein the raw stream data fills a first portion of each chunk, the first portion corresponding to an offset related to the batch size chunk payload size;
generate a chunk footer for each chunk that delineates an amount of the raw stream data written into each chunk;
in response to a size of each chunk ready for sending being less than the batch size chunk payload size, pad each chunk with zeroes to fill each chunk to the batch size chunk payload size;
generate a data size integer based on the size of each chunk and the padding; and
use the data size integer as each chunk footer; and
send each chunk to a controller for storage without buffering at the client application and based on a time associated with a dynamic ingestion of the raw stream data, wherein each chunk is sent regardless of an amount of raw stream data written into each chunk.
7. The system of claim 6 , further configured to:
in response to the raw stream data filling each chunk to an offset associated with the batch size chunk payload size, use the batch size chunk payload size as each chunk footer.
8. The system of claim 6 , wherein the dynamically generating each chunk footer and dynamically sending each chunk to the controller for storage occur based on a time associated with the dynamically ingesting raw stream data.
9. The system of claim 6 , wherein the batch size chunk payload size is 32,000 bytes.
10. The system of claim 6 , further configured to:
receive each chunk by the controller;
determine a stream data size based on each chunk header and each chunk footer; and
store a subset of each chunk in storage based on the stream data size.
11. A non-transitory computer readable medium with program instructions stored thereon to perform the following acts:
ingest raw stream data by a client application;
generate a chunk header that includes a batch size chunk payload size for each of a plurality of chunks, wherein generating the chunk header includes determining a total batch size and setting the batch size chunk payload size based on the total batch size;
write the raw stream data into each chunk of the plurality of chunks, wherein the raw stream data fills a first portion of each chunk, the first portion corresponding to an offset related to the batch size chunk payload size;
generate a chunk footer for each chunk that delineates an amount of the raw stream data written into each chunk;
in response to a size of each chunk ready for sending being less than the batch size chunk payload size, padding each chunk with zeroes to fill each chunk to the batch size chunk payload size;
generating a data size integer based on the size of each chunk and the padding; and
using the data size integer as each chunk footer; and
send each chunk to a controller for storage without buffering at the client application and based on a time associated with a dynamic ingestion of the raw stream data, wherein each chunk is sent regardless of an amount of raw stream data written into each chunk.
12. The non-transitory computer readable medium of claim 11 , with program instructions stored thereon to further perform the following acts:
in response to the raw stream data filling each chunk to an offset associated with the batch size chunk payload size, using the batch size chunk payload size as each chunk footer.
13. The non-transitory computer readable medium of claim 11 , wherein the dynamically generating each chunk footer and dynamically sending each chunk to the controller for storage occur based on a time associated with the dynamically ingesting raw stream data.
14. The non-transitory computer readable medium of claim 11 , wherein the batch size chunk payload size is 32,000 bytes.
15. The non-transitory computer readable medium of claim 11 , with program instructions stored thereon to further perform the following acts:
receiving each chunk by the controller;
determining a stream data size based on each chunk header and each chunk footer; and
storing a subset of each chunk in storage based on the stream data size.