Cloud-based streaming data receiver and persister
The present system receives streaming raw data and inserts context into the raw data. The context raw data may be partitioned into sub-batches and transmitted to a data receiver and persister. The raw data may include context information as well as child-parent information to assist with persisting data. The context may be used to place the data in buckets without analysis of the data, thereby saving time and resources while storing the data batches.
1. A method for streaming and storing data, comprising:
automatically generating a context for each application from amongst a plurality of applications that transmitted an initiation message, wherein each context for a particular application is distinct from other context associated with other applications;
receiving raw streaming data from the one or more applications;
associating a corresponding context data to a batch of the raw data sent from a particular application, wherein the associating includes:
creating a batch table, and
updating the batch table with the context data associated with the batch of raw data sent from the particular application; and
transmitting the batch of raw data and corresponding associated context data to a remote server, wherein the remote server stores the batch of raw data in memory based on the associated context data without further processing of the raw data.
2. The method of claim 1 , further comprising:
sending a start batch signal to the remote server; and receiving the context data from the remote server.
3. The method of claim 1 , wherein the batch table includes the context data and a corresponding batch identifier.
4. The method of claim 1 , further comprising associating parent-child information to the batch of raw data.
5. The method of claim 4 , wherein the transmitted batch of raw data configured to be persisted by the remote server based at least in part on the parent-child information.
6. The method of claim 1 , further including partitioning the batch of data into sub-batches.
7. The method of claim 6 , wherein the sub-batches are sent in parallel to the remote server.
8. A computer readable non-transitory storage medium having embodied thereon a program, the program being executable by a processor to perform a method for streaming and storing data, the method comprising:
automatically generating a context for each application from amongst a plurality of applications that transmitted an initiation message, wherein each context for a particular application is distinct from other context associated with other applications;
receiving raw streaming data from the one or more applications;
associating a corresponding context data to a batch of the raw data sent from a particular application, wherein the associating includes:
creating a batch table, and
updating the batch table with the context data associated with the batch of raw data sent from the particular application; and
transmitting the batch of raw data and corresponding associated context data to a remote server, wherein the remote server stores the batch of raw data in memory based on the associated context data without further processing of the raw data.
9. The computer readable non-transitory storage medium of claim 8 , the method further comprising:
sending a start batch signal to the remote server; and
receiving the context data from the remote server.
10. The computer readable non-transitory storage medium of claim 8 , wherein the batch table includes the context data and a corresponding batch identifier.
11. The computer readable non-transitory storage medium of claim 8 , the method further comprising associating parent-child information to the batch of raw data.
12. The computer readable non-transitory storage medium of claim 11 , the transmitted batch of raw data configured to be persisted by the remote server based at least in part on the parent-child information.
13. The computer readable non-transitory storage medium of claim 8 , the method further including partitioning the batch of data into sub-batches.
14. The computer readable non-transitory storage medium of claim 13 , wherein the sub-batches are sent in parallel to the remote server.
15. A system for streaming and storing data, comprising:
a memory;
a processor; and
one or more modules stored in memory and executable by the processor to:
automatically generate a context for each application from amongst a plurality of applications that transmitted an initiation message, wherein each context for a particular application is distinct from other context associated with other applications;
receive raw streaming data from the one or more applications;
associate a corresponding context data to a batch of the raw data sent from a particular application, wherein the associating includes:
creating a batch table, and
updating the batch table with the context data associated with the batch of raw data sent from the particular application; and
the batch of raw data and corresponding associated context data to a remote server, wherein the remote server stores the batch of raw data in memory based on the associated context data without further processing of the raw data.
16. The system of claim 15 , further comprising:
sending a start batch signal to the remote server, and
receiving the context data from the remote server.
17. The system of claim 15 , wherein the batch table includes the context data and a corresponding batch identifier.
18. The system of claim 15 , the one or more modules further executable to associate parent-child information to the batch of raw data.
19. The system of claim 18 , the transmitted batch of raw data configured to be persisted by the remote server based at least in part on the parent-child information.
20. The system of claim 15 , the one or more modules further executable to partition the batch of data into sub-batches.
21. The system of claim 20 , wherein the sub-batches are sent in parallel to the remote server.