Structural deduplication of network events
Various embodiments include systems and methods pertaining to a security service platform that detects security threats based on a security service that operates on structurally deduplicated network data. Based on efficient processing of structurally deduplicated data, the security service platform may use a larger ruleset to increase coverage of threat detection. Additional performance improvements based on the security service platform using deduplicated event data according to a structure of a data model of the event data is that the security service platform may use a single instance of structurally deduplicated event data to represent multiple network events.
1 . A method comprising:
performing, by one or more network sensors located in a local area network:
analyzing network data in the local area network to determine a plurality of network events;
determining, based on the plurality of network events, a plurality of event references associated with event data, wherein the event data is structured in accordance with a data model comprising one or more fields;
determining, based on the plurality of network events, a plurality of deduplicated field groups associated with one or more fields of the data model that are redundant with other field groups of the event data;
determining, based on the plurality of network events, a plurality of deduplicated values associated with one or more values in one or more fields of the event data that are redundant with one or more other values in one or more other fields of the event data;
generating structurally deduplicated data indicative of the plurality of network events based on the plurality of event references, of the plurality of deduplicated field groups, and of the plurality of deduplicated values, wherein the structurally deduplicated data is generated as a file in a format that removes redundant field groups and redundant values in the event data, wherein the format includes:
(a) a plurality of deduplication tables including (i) a strings table that stores the deduplicated values and (ii) a plurality of field group tables that stores the deduplicated field groups based on references to the deduplicated values in the strings table,
(b) an events table that stores the network events based on references to the deduplicated field groups in the field group tables, and
(c) a file header that indicates respective table sizes of the deduplication tables and the events table; and
uploading the file to a security service, wherein the security service is remote from the local area network and configured to assess the structurally deduplicated data in the file according to the format and generate an alert when a cyberattack on the local network is detected based on the assessment.
2 . The method of claim 1 , wherein the format of the structurally deduplicated data is based on a structure of the data model.
3 . The method of claim 1 , wherein the field group tables corresponding to the deduplicated field groups include two or more of:
a JA3 hash values table,
a clients table,
a servers table,
a certificates table, and
a connections table.
4 . The method of claim 3 , wherein the servers table references the certificates table and the clients table references to the JA3 hash values table.
5 . The method of claim 1 , wherein a quantity of levels of the deduplicated field groups is based on a quantity of nesting levels of a data model representing a network event.
6 . The method of claim 1 , wherein the file header indicates an initial event timestamp of the plurality of events indicated in the file.
7 . The method of claim 1 , wherein event data included within the structurally deduplicated data comprises a transport layer security (TLS) event, a domain name system (DNS) event, or a flow event.
8 . A system comprising:
a memory storing executable instructions; and
one or more processors that execute the executable instructions to implement one or more networks sensors located in a local area network and cause the one or more networks sensors to:
analyze network data in the local area network to determine a plurality of network events;
determine, based on the plurality of network events, a plurality of event references associated with event data, wherein the event data is structured in accordance with a data model comprising one or more fields;
determine, based on the plurality of network events, a plurality of deduplicated field groups associated with one or more fields of the data model that are redundant with other field groups of the event data;
determine, based on the plurality of network events, a plurality of deduplicated values associated with one or more values in one or more fields of the event data that are redundant with one or more other values in one or more other fields of the event data;
generate structurally deduplicated data indicative of the plurality of network events based on the plurality of event references, of the plurality of deduplicated field groups, and of the plurality of deduplicated values, wherein the structurally deduplicated data is generated as a file in a format removes redundant field groups and redundant values in the event data, wherein the format includes:
(a) a plurality of deduplication tables including (i) a strings table that stores the deduplicated values and (ii) a plurality of field group tables that stores the deduplicated field groups based on references to the deduplicated values in the strings table, and
(b) an events table that stores the network events based on references to the deduplicated field groups in the field group tables, and
(c) a file header that indicates respective table sizes of the deduplication tables and the events table; and
upload the file to a security service, wherein the security service is remote from the local area network and configured to assess the structurally deduplicated data in the file according to the format and generate an alert when a cyberattack on the local network is detected based on the assessment.
9 . The system of claim 8 , wherein the format of the structurally deduplicated data is based on a structure of the data model.
10 . The system of claim 8 , wherein the field group tables corresponding to the deduplicated field groups include two or more of:
a JA3 hash values table,
a clients table,
a servers table,
a certificates table, and
a connections table.
11 . The system of claim 10 , wherein the servers table references the certificates table and the clients table references to the JA3 hash values table.
12 . The system of claim 8 , wherein a quantity of levels of the deduplicated field groups is based on a quantity of nesting levels of a data model representing a network event.
13 . The system of claim 8 , wherein the file header indicates an initial event timestamp of the plurality of events indicated in the file.
14 . The system of claim 8 , wherein event data included within the structurally deduplicated data comprises a transport layer security (TLS) event, a domain name system (DNS) event, or a flow event.
15 . One or more non-transitory computer-accessible storage media storing executable instructions that, when executed by one or more processors, implement one or more networks sensors located in a local area network and cause the one or more networks sensors to:
analyze network data in the local area network to determine a plurality of network events;
determine, based on the plurality of network events, a plurality of event references associated with event data, wherein the event data is structured in accordance with a data model comprising one or more fields;
determine, based on the plurality of network events, a plurality of deduplicated field groups associated with one or more fields of the data model are redundant with other field groups of the event data;
determine, based on the plurality of network events, a plurality of deduplicated values associated with one or more values in one or more fields of the event data that are redundant with one or more other values in one or more other fields of the event data;
generate structurally deduplicated data indicative of the plurality of network events based on the plurality of event references, of the plurality of deduplicated field groups, and of the plurality of deduplicated values, wherein the structurally deduplicated data is generated as a file in a format removes redundant field groups and redundant values in the event data, wherein the format includes:
(a) a plurality of deduplication tables including (i) a strings table that stores the deduplicated values and (ii) a plurality of field group tables that stores the deduplicated field groups based on references to the deduplicated values in the strings table,
(b) an events table that stores the network events based on references to the deduplicated field groups in the field group tables, and
(c) a file header that indicates respective table sizes of the deduplication tables and the events table; and
upload the file to a security service, wherein the security service is remote from the local area network and configured to assess the structurally deduplicated data in the file according to the format and generate an alert when a cyberattack on the local network is detected based on the assessment.
16 . The one or more non-transitory computer-accessible storage media of claim 15 , wherein the format of the structurally deduplicated data is based on a structure of the data model.
17 . The one or more non-transitory computer-accessible storage media of claim 16 , wherein the field group tables corresponding to the deduplicated field groups include two or more of:
a JA3 hash values table,
a clients table,
a servers table, a certificates table, and
a connections table.
18 . The one or more non-transitory computer-accessible storage media of claim 17 , wherein the servers table references the certificates table and the clients table references to the JA3 hash values table.
19 . The one or more non-transitory computer-accessible storage media of claim 15 , wherein a quantity of levels of the deduplicated field groups is based on a quantity of nesting levels of a data model representing a network event.
20 . The one or more non-transitory computer-accessible storage media of claim 15 , wherein the file header indicates an initial event timestamp of the plurality of events indicated in the file.