Unified storage for event streams in an anomaly detection framework
Providing unified storage for event streams in an anomaly detection framework, including: receiving, by an event streaming platform, first event data encoded in a first file format; converting, by the event streaming platform, the first event data into second event data encoded in a second file format for storage in a first remote storage system; and providing, to the first remote storage system, a command to load the second event data into the first remote storage system.
1. A method of providing unified storage for event streams in an anomaly detection framework, the method comprising:
receiving, by an event streaming platform, first event data encoded in a first file format;
converting, by the event streaming platform, the first event data into second event data encoded in a second file format for storage in a first remote storage system, wherein converting the first event data into the second event data comprises combining the first event data with data from one or more other sources by performing one or more enrichment operations;
providing, to the first remote storage system, a command to load the second event data into the first remote storage system; and
providing a query layer for accessing data in the second file format across a plurality of data sources including the first remote storage system.
2. The method of claim 1 , further comprising:
storing the second event data into a second remote storage system; and
wherein providing the command to load the second event data into the first remote storage system causes the first remote storage system to load the second event data from the second remote storage system.
3. The method of claim 1 , wherein the first remote storage system comprises a data warehouse.
4. The method of claim 1 , the first remote storage system comprises a data lakehouse.
5. The method of claim 1 , wherein the second remote storage system comprises a cloud-based object storage system.
6. The method of claim 1 , wherein converting the first event data into the second event data comprises performing one or more filtering operations.
7. The method of claim 1 , wherein converting the first event data into the second event data comprises performing one or more aggregation operations.
8. A computer program product for providing unified storage for event streams in an anomaly detection framework, the computer program product disposed on a non-transitory computer readable medium, the computer program product including computer program instructions configurable to carry out the steps of:
receiving, by an event streaming platform, first event data encoded in a first file format;
converting, by the event streaming platform, the first event data into second event data encoded in a second file format for storage in a first remote storage system, wherein converting the first event data into the second event data comprises combining the first event data with data from one or more other sources by performing one or more enrichment operations;
providing, to the first remote storage system, a command to load the second event data into the first remote storage system; and
providing a query layer for accessing data in the second file format across a plurality of data sources including the first remote storage system.
9. The computer program product of claim 8 , wherein the steps further comprise:
storing the second event data into a second remote storage system; and
wherein providing the command to load the second event data into the first remote storage system causes the first remote storage system to load the second event data from the second remote storage system.
10. The computer program product of claim 8 , wherein the first remote storage system comprises a data warehouse.
11. The computer program product of claim 8 , the first remote storage system comprises a data lakehouse.
12. The computer program product of claim 8 , wherein the second remote storage system comprises a cloud-based object storage system.
13. The computer program product of claim 8 , wherein converting the first event data into the second event data comprises performing one or more filtering operations.
14. The computer program product of claim 8 , wherein converting the first event data into the second event data comprises performing one or more aggregation operations.
15. A system for providing unified storage for event streams in an anomaly detection framework, the system including computer program instructions stored in a memory coupled with a processor that, when executed, cause the system to carry out the steps of:
receiving, by an event streaming platform, first event data encoded in a first file format;
converting, by the event streaming platform, the first event data into second event data encoded in a second file format for storage in a first remote storage system, wherein converting the first event data into the second event data comprises combining the first event data with data from one or more other sources by performing one or more enrichment operations;
providing, to the first remote storage system, a command to load the second event data into the first remote storage system; and
providing a query layer for accessing data in the second file format across a plurality of data sources including the first remote storage system.
16. The system of claim 15 , wherein the steps further comprise:
storing the second event data into a second remote storage system; and
wherein providing the command to load the second event data into the first remote storage system causes the first remote storage system to load the second event data from the second remote storage system.
17. The system of claim 15 , wherein the first remote storage system comprises a data warehouse or a data lakehouse.
18. The system of claim 15 , wherein the second remote storage system comprises a cloud-based object storage system.
19. The system of claim 15 , wherein converting the first event data into the second event data comprises performing one or more filtering operations.
20. The system of claim 15 , wherein converting the first event data into the second event data comprises performing one or more aggregation operations.