Configuration and operational network observability using sliding window for append-only telemetry log
Techniques are described for storing and processing network data for responding to queries for such network data. Operational network data is separated from configuration network data so that they can be processed and stored separately. A sliding window cache is used to continually, temporarily store network data objects having time stamps falling within the time range of the sliding window cache. Network data objects stored within the sliding window cache are then moved to computer memory for storage and later retrieval. In response to a query for network data, network data objects can be retrieved from the sliding window cache and also from the computer memory based on time stamps of the network data objects and on the time range of the query.
1 . A method comprising:
receiving configuration data and operational data,
wherein the configuration data comprises data created by an end user, and
wherein the operational data comprises data created internally in a network system and not by and not by the end user;
separating the configuration data and operational data to generate a configuration data stream and an operational data stream;
using a first processor to process the configuration data stream, wherein processing the configuration data stream comprises storing the configuration data in one or more files in a memory; and
using a second processor to process the operational data stream, wherein processing the operational data stream comprises:
temporarily caching operational data objects from the operational data stream in a sliding window cache of a predetermined time length;
generating a log database table for the sliding window cache, the log database table comprising log entries for the cached operational data objects, wherein the log entries each comprise an identifier associated with a cached operational data object, an incremented number and a timestamp associated with the cached operational data object; and
storing the operational data stream in an append-only telemetry log; and
using the log database table to process queries for cached operational data objects occurring within the sliding window cache.
2 . The method of claim 1 , wherein the identifier is associated with the cached operational data object's parent object class.
3 . The method of claim 1 , wherein the append-only telemetry log comprises one or more of an audit log database, an event log database, or an access log database.
4 . The method of claim 3 , wherein the access log database comprises an nginx access log.
5 . The method of claim 3 , wherein the audit log database comprises an records objects that are created, modified, or deleted, and corresponding times.
6 . The method of claim 3 , wherein the event log database records conditions expressed by Boolean expressions over object properties.
7 . The method of claim 1 , further comprising:
receiving a query for network data occurring during a query duration, wherein the query duration exceeds a duration of the sliding window cache;
retrieving cached operational data objects that are associated with timestamps within the duration of the sliding window cache from the sliding window cache;
retrieving non-cached operational data objects that are associated with timestamps outside the duration of the sliding window cache from outside the sliding window cache; and
responding to the query with the retrieved cached operational data objects and the retrieved non-cached operational data objects.
8 . The method as in claim 1 , further comprising:
receiving, from a subscription service, a query triggered by an object change event; and
determining whether to use the sliding window cache in connection with responding to the query.
9 . The method of claim 1 , further comprising:
receiving a query for network data; and
performing a binary search to locate at least a portion of the network data in the sliding window cache.
10 . A network data processing system comprising:
one or more processors; and
one or more non-transitory computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
receiving configuration data and operational data,
wherein the configuration data comprises data created by an end user, and
wherein the operational data comprises data created internally in a network system and not by and not by the end user;
separating the configuration data and operational data to generate a configuration data stream and an operational data stream;
using a first processor to process the configuration data stream, wherein processing the configuration data stream comprises storing the configuration data in one or more files in a memory; and
using a second processor to process the operational data stream, wherein processing the operational data stream comprises:
temporarily caching operational data objects from the operational data stream in a sliding window cache;
generating a log database table for the sliding window cache, the log database table comprising log entries for the cached operational data objects, wherein the log entries each comprise an incremented number and a timestamp; and
storing the operational data stream in an append-only telemetry log; and
using the log database table to process queries for cached operational data objects occurring within the sliding window cache.
11 . The network data processing system of claim 10 , wherein the log entries each further comprise an identifier associated with a cached operational data object's parent object class.
12 . The network data processing system of claim 10 , wherein the operations further comprise, after temporarily caching the operational data objects in the sliding window cache, moving the operational data objects to one or more of an audit log database, an event log database, or an access log database.
13 . The network data processing system of claim 10 , wherein the operations further comprise:
receiving a query for network data occurring during a query duration, wherein the query duration exceeds a duration of the sliding window cache;
retrieving cached operational data objects that are associated with timestamps within the duration of the sliding window cache from the sliding window cache;
retrieving non-cached operational data objects that are associated with timestamps outside the duration of the sliding window cache from outside the sliding window cache; and
responding to the query with the retrieved cached operational data objects and the retrieved non-cached operational data objects.
14 . The network data processing system of claim 10 , wherein the operations further comprise:
receiving, from a subscription service, a query triggered by an object change event; and
determining whether to use the sliding window cache in connection with responding to the query.
15 . The network data processing system of claim 10 , wherein the operations further comprise:
receiving a query for network data; and
performing a binary search to locate at least a portion of the network data in the sliding window cache.
16 . The network data processing system of claim 10 , wherein the operations further comprise:
receiving, from a subscription service, a query triggered by an object change event; and
determining whether to use the sliding window cache in connection with responding to the query.
17 . One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:
receiving configuration data and operational data,
wherein the configuration data comprises data created by an end user, and
wherein the operational data comprises data created internally in a network system and not by and not by the end user;
separating the configuration data and operational data to generate a configuration data stream and an operational data stream;
using a first processor to process the configuration data stream, wherein processing the configuration data stream comprises storing the configuration data in one or more files in a memory; and
using a second processor to process the operational data stream, wherein processing the operational data stream comprises:
temporarily caching operational data objects from the operational data stream in a sliding window cache;
generating a log database table for the sliding window cache, the log database table comprising log entries for the cached operational data objects, wherein the log entries each comprise an incremented number and a timestamp; and
storing the operational data stream in an append-only telemetry log comprising one or more of an audit log database, an event log database, or an access log database.
18 . The one or more non-transitory computer-readable media of claim 17 , wherein the operations further comprise:
using the log database table for the sliding window cache to process queries for cached operational data objects occurring within the sliding window cache.
19 . The one or more non-transitory computer-readable media of claim 17 , wherein the log entries each further comprise an identifier associated with a cached operational data object's parent object class.
20 . The one or more non-transitory computer-readable media of claim 17 , wherein the operations further comprise:
receiving a query for network data occurring during a query duration, wherein the query duration exceeds a duration of the sliding window cache;
retrieving cached operational data objects that are associated with timestamps within the duration of the sliding window cache from the sliding window cache;
retrieving non-cached operational data objects that are associated with timestamps outside the duration of the sliding window cache from outside the sliding window cache; and
responding to the query with the retrieved cached operational data objects and the retrieved non-cached operational data objects.