Systems and methods for security observability model and design for agents
An architecture of observability can be viewed as consisting of: Data observation and aggregation Data transmission Data storage Data visualizations and intelligence insights Agents are concerned with “Data observation and aggregation” and “Data Transmission”. The backend SaaS service is concerned with “Data Transmission”, “Data Storage”, and “Data visualizations and intelligence insights”. The scope of this article will specifically focus on Agent concerns.
1 . A system for security observability for agents, comprising:
with at least one hard-ware computer processor; and
a non-transitory machine-readable medium storing computer-executable instructions that, when executed by the at least one hard-ware processor, configure the at least one hard-ware processor to perform operations comprising:
at least one agent configured to gather security relevant data from an application using a plurality of sensors that generate events that snapshot information from within the application;
an aggregation operation configured to record the events and to compress the events into meaningful metrics including one or more metric data streams;
a transmission operation configured to transmit the one or more metric data streams to a collector according to transmission intervals that are decoupled from an activity on the application under observation; and
a backend software-as-a-service (SaaS) service configured to perform data storage and to provide data visualizations and intelligence insights based on the transmitted data,
wherein the transmission operation uses a communication protocol that is abstracted from a backend data model to enable agents to send arbitrary observations to the collector without an ingestion point rejecting a request because of unknown data, and to enable a mix of agents sending data in an older model and other agents sending data in a newer model.
2 . The system of claim 1 , wherein the plurality of sensors capture at least one of an incoming hypertext transfer protocol (HTTP) parameter or details of a structured query language (SQL) query made to a database.
3 . The system of claim 2 , wherein the aggregation operation aggregates at least one event into multiple forms and outputs the at least one event as separate metric data streams.
4 . The system of claim 3 , wherein the backend software-as-a-service (SaaS) service is concerned with data transmission, data storage, and data visualizations and intelligence insights.
5 . The system of claim 4 , wherein the transmission operation transmits the one or more metric data streams in a manner that mitigates a denial-of-service condition caused by a plurality of agents attempting to send production-load activity to the collector.
6 . The system of claim 5 , wherein the transmission operation transmits the one or more metric data streams as data streams that are decomposable to enable horizontal scaling at an ingestion point.