LOG DATA MANAGEMENT
This disclosure relates generally to efficiently storing and retrieving indexless log data. In particular, during ingestion of the log data metadata describing the log data and its location within a first data storage system is saved in a second data storage system to assist in efficient retrieval of the log data.
1 . A non-transitory computer-readable storage medium storing instructions configured to be executed by one or more processors to carry out steps that include:
receiving a request to retrieve log data from a first data storage system containing a plurality of log data files;
identifying a subset of the plurality of log data files that contain the requested log data by sending the request to a second data storage system, separate and distinct from the first data storage system, that runs a log data indexing service and stores metadata describing contents of the plurality of log data files stored in the first data storage system;
transmitting a query to the first data storage system with instructions to search only the subset of the plurality of log data files for the requested log data; and
receiving the requested log data from the first data storage system.
2 . The non-transitory computer-readable storage medium of claim 1 , wherein the query transmitted to the first data storage system comprises instructions to create a plurality of subqueries for concurrent execution by a plurality of compute units associated with the first data storage system.
3 . The non-transitory computer-readable storage medium of claim 2 , wherein the plurality of log data files is stored in a hierarchical data structure organized by time of generation.
4 . The non-transitory computer-readable storage medium of claim 1 , wherein the query comprises instructions specifying how many compute units the first data storage system should apply to the query based on an urgency of the request.
5 . The non-transitory computer-readable storage medium of claim 4 , wherein the query includes instructions for how to divide the query into sub-queries for assignment to the specified number of compute units.
6 . The non-transitory computer-readable storage medium of claim 1 , wherein identifying the subset of the plurality of log data files comprises receiving the identification of the subset of the plurality of log data files from the log data indexing service.
7 . The non-transitory computer-readable storage medium of claim 1 , wherein the metadata comprises file size, time range and URIs where a log data file can be accessed within the first data storage system for each of the plurality of log data files.
8 . The non-transitory computer-readable storage medium of claim 1 , wherein the metadata is stored by the log data indexing service at the second data storage system prior to storing the log data within the first data storage system.
9 . A log data retrieval system, comprising:
one or more processors; and
memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for:
receiving a request to retrieve log data from a first data storage system containing a plurality of log data files;
identifying a subset of the plurality of log data files that contain the requested log data by sending the request to a second data storage system, separate and distinct from the first data storage system, that runs a log data indexing service and stores metadata describing contents of the plurality of log data files stored in the first data storage system;
transmitting a query to the first data storage system with instructions to search only the subset of the plurality of log data files for the requested log data; and
receiving the requested log data from the first data storage system.
10 . The log data retrieval system of claim 9 , wherein the query transmitted to the first data storage system comprises instructions to create a plurality of subqueries for concurrent execution by a plurality of compute units associated with the first data storage system.
11 . The log data retrieval system of claim 10 , wherein the plurality of log data files is stored in a hierarchical data structure organized by time of generation.
12 . The log data retrieval system of claim 9 , wherein the query comprises instructions specifying how many compute units the first data storage system should apply to the query based on an urgency of the request.
13 . The log data retrieval system of claim 12 , wherein the query includes instructions for how to divide the query into sub-queries for assignment to the specified number of compute units.
14 . The log data retrieval system of claim 9 , wherein identifying the subset of the plurality of log data files comprises receiving the identification of the subset of the plurality of log data files from the log data indexing service.
15 . The log data retrieval system of claim 9 , wherein the metadata comprises file size, time range and URIs where a log data file can be accessed within the first data storage system for each of the plurality of log data files.
16 . A method of retrieving data from a log data storage system, the method comprising:
receiving a request to retrieve log data from a first data storage system containing a plurality of log data files;
identifying a subset of the plurality of log data files that contain the requested log data by sending the request to a second data storage system, separate and distinct from the first data storage system, that runs a log data indexing service and stores metadata describing contents of the plurality of log data files stored in the first data storage system;
transmitting a query to the first data storage system with instructions to search only the subset of the plurality of log data files for the requested log data; and
receiving the requested log data from the first data storage system.
17 . The method of claim 16 , wherein the query transmitted to the first data storage system comprises instructions to create a plurality of subqueries for concurrent execution by a plurality of compute units associated with the first data storage system.
18 . The method of claim 17 , wherein the plurality of log data files is stored in a hierarchical data structure organized by time of generation.
19 . The method of claim 16 , wherein the query includes instructions specifying how many compute units the first data storage system should apply to the query based on an urgency of the request.
20 . The method of claim 19 , wherein the query includes instructions for how to divide the query into sub-queries for assignment to the specified number of compute units.