Systems and methods for indexing and aggregating data records
View Patent ↗Computer-implemented systems and methods are disclosed for indexing and aggregating proxy server data. In accordance with some embodiments, the systems comprise a memory device configured to store a set of instructions, and one or more processors configured to execute the set of instructions to receive a plurality of log lines of proxy server data comprising a plurality of data components and index one or more of the plurality of data components for the plurality of log lines. The processors also generate a plurality of log data records, each including the one or more indexed data components, and generate a summarized data record by aggregating one or more log data records having at least one matching indexed data component.
1. A system for reducing storage space requirements by aggregating similar structured data records, the system comprising:
a memory device configured to store a set of instructions; and
one or more processors configured to execute the set of instructions to:
access a plurality of structured data records each comprising a plurality of data values in a corresponding plurality of data fields, the data fields including at least a first data field and a second data field;
identify those of the plurality of structured data records having a first data value for each of the corresponding first data fields, wherein second data values of corresponding second data fields of the identified plurality of structured data records are unique;
determine a quantity of the identified structured data records having the first data value for each of the corresponding first data fields; and
generate, for storage in place of the identified structured data records, a summarized data record indicating the corresponding first data value common among each of the identified structured data records and the quantity of identified structured data records, wherein the summarized data record does not include the second data values of the identified structured data records.
2. The system of claim 1 , wherein, the quantity of identified structured data records is included in an additional data field of the summarized data record.
3. The system of claim 1 , wherein the summarized data record further comprises an aggregate value of each of a particular data field of the identified structured data records.
4. The system of claim 1 , wherein the one or more processors are further configured to:
receive a search request relating to at least one of data fields; and
provide instructions to display information relating to the request based on a search of summarized data records.
5. The system of claim 1 , wherein the plurality of structured data records summarized in the summarized data record are limited to those associated with a certain period of time.
6. The system of claim 1 , wherein the one or more processors are further configured to:
identify those of a plurality of summarized data records each a common value for a first data field; and
generate, for storage in place of the identified summarized data records, a second summarized data record indicating the common value.
7. The system of claim 1 , wherein the plurality of structured data records correspond to a plurality of indexed log lines of proxy server data, and the first data field corresponds to a remote address or domain.
8. A computer-implemented method for comprising:
access a plurality of structured data records, each comprising a plurality of data values in a corresponding plurality of data fields, the data fields including at least a first data field and a second data field;
identify those of the plurality of structured data records having a first data value for each of the corresponding first data fields, wherein second data values of corresponding second data fields of the identified plurality of structured data records are unique;
determine a quantity of the identified structured data records having the first data value for each of the corresponding first data fields; and
generate, for storage in place of the identified structured data records, a summarized data record, indicating the corresponding first data value common among each of the identified structured data records and the quantity of identified structured data record, wherein the summarized data record does not include the second data values of the identified structured data records;
wherein the method is performed using one or more processors.
9. The computer-implemented method of claim 8 , wherein the quantity of identified structured data records is included in an additional data field of the summarized data record.
10. The computer-implemented method of claim 8 , wherein the summarized data record further comprises an aggregate value of each of a particular data field of the identified structured data records.
11. The computer-implemented method of claim 8 , further comprising:
receiving a search request relating to at least one of the data fields; and
providing instructions to display information relating to the request based on a search of summarized data records.
12. The computer-implemented method of claim 8 , wherein the plurality of structured data records summarized in the summarized data record are limited to those associated with a certain period of time.
13. The computer-implemented method of claim 8 , further comprising:
identifying those of a plurality of summarized data records each a common value for a first data fields; and
generating, for storage in place of the identified summarized data records, a second summarized data record indicating the common value.
14. The computer-implemented method of claim 8 , wherein the plurality of structured data records correspond to a plurality of indexed log lines of proxy server data, and the first data field corresponds to a remote address or domain.
15. A non-transitory computer-readable medium storing a set of instructions that are executable by one or more processors to cause a computing system to:
access a plurality of structured data records, each comprising a plurality of data values in a corresponding plurality of data fields, the data fields including at least a first data field and a second data field;
identify those of the plurality of structured data records having a first data value for each of the corresponding first data fields, wherein second data values of corresponding second data fields of the identified plurality of structured data records are unique;
determine a quantity of the identified structured data records having the first data value for each of the corresponding first data fields; and
generate, for storage in place of the identified structured data records, a summarized data record indicating the corresponding first data value common among each of the identified structured data records and the quantity of identified structured data record, wherein the summarized data record does not include the second data values of the identified structured data records.
16. The non-transitory computer-readable medium of claim 15 , wherein the summarized data record is generated to include an additional data field indicative of the quantity of identified structured data records.
17. The non-transitory computer-readable medium of claim 15 , wherein the method further comprises:
identifying those of a plurality of summarized data records each a common value for a first data field; and
generating, for storage in place of the identified summarized data records, a second summarized data record indicating the common value.