Systems and methods for data analytics
A serverless data processing architecture may be a high-velocity, cost-efficient and massively scalable log ingest application and event parser that may act as the intermediary between data collection and data analysis and may handle the receipt and parsing of incoming event data generated by collector and sensor applications. The log analytics data science portion of an ingest process may determine if arbitrarily structured log data is anomalous and/or security relevant via single event or multievent algorithms. There are numerous applications for both the single and multi-event log analytics platforms. Any application involving reliably finding anomalous pieces of semi-structured or unstructured text data may provide potential use cases. These processes may be particularly widely applicable because these processes are agnostic to and require no prior knowledge of the structure of the data.
1. A data analytics method, comprising:
receiving event data and authenticating one or more events by extracting metadata and one or more validation tokens and comparing the one or more validation tokens against a list of authorized tokens and identifying one or more authenticated events if the one or more validation tokens to identify one or more events in the event data;
extracting key data from the one or more authenticated events using serverless functions, constructing an object for each of the one or more authenticated events and batching similar objects of the one or more events into one or more batches;
storing the one or more batches of objects in a distributed NoSQL data store cluster to facilitate rapid searches of the one or more batches of objects; and
storing in parallel the one or more batches of objects in a data lake and to facilitate rapid access to large volumes of data comprising the one or more batches of objects that are packaged together, compressed and delivered to a real-time data stream.
2. The data analytics method of claim 1 , wherein the event data comprises event log data.
3. The data analytics method of claim 1 , wherein the authorized tokens are stored in a serverless NoSQL key value database.
4. The data analytics method of claim 1 , further comprising a series of serverless event streams used to store one or more incoming events after they have been authenticated, wherein the events are batched together based on similarity and subsequently extracted from the stream in batches for processing.
5. The data analytics method of claim 1 , wherein the distributed NoSQL data store cluster is serverless.
6. The data analytics method of claim 5 , wherein the serverless distributed NoSQL data store cluster provides highly scalable storage of the one or more batches of objects.
7. The data analytics method of claim 1 , wherein the object constructed for each of the one or more authenticating events is a JSON object.
8. The data analytics method of claim 1 , wherein the data lake is serverless.