Security service platform for rule matching
Various embodiments include systems and methods pertaining to a security service platform that detects security threats based on a security service that operates on structurally deduplicated network data. The security service platform, based on using the structure, or data model, of data being deduplicated, generates structurally deduplicated event data that is more compact than traditionally compressed data or traditionally deduplicated data stored in a structured data format. The security service may perform a security analysis that includes rule matching to detect threats to a network, where the rule matching operates on the structurally deduplicated data.
1 . A method comprising:
performing, by a network sensor in a local area network:
determining, based on deep packet inspection (DPI) analysis of network data observed in the local area network, a plurality of network events, wherein the plurality of network events are matched against a first ruleset of the network sensor to monitor for security incidents or threats in the local area network;
generating structurally deduplicated data for the plurality of network events, the structurally deduplicated data comprising:
a plurality of event references associated with event data, wherein the event data is structured in accordance with a data model comprising one or more fields,
a plurality of deduplicated field groups associated with one or more fields of the data model that include redundant data with other field groups of the event data, and
a plurality of deduplicated values associated with one or more values in one or more fields of the event data that are redundant with one or more other values in one or more other fields of the event data,
wherein the structurally deduplicated data is compressed using a compression algorithm;
uploading the structurally deduplicated data to a security service, wherein the security service is configured to assess the structurally deduplicated data for security incidents or threats using an expanded ruleset larger than the first ruleset;
performing, by the security service in response to the upload:
identifying, based on a single rule in the expanded ruleset matching a single instance of event data indicated by the structurally deduplicated data, one or more network events indicative of a cyberattack on the local area network; and
initiating, based on the one or more network events indicative of a cyberattack, one or more remediation operations.
2 . The method of claim 1 , wherein the security service is implemented using one or more cloud services operating within a remote cloud computing environment.
3 . The method of claim 2 , further comprising the security service:
repeatedly receiving and assessing uploads of structurally deduplicated data of network events from the local area network.
4 . The method of claim 3 , further comprising the security service:
assessing the uploads of structurally deduplicated data to perform different types of security monitoring, including two or more of:
malware detection,
phishing detection, and
intrusion detection.
5 . The method of claim 1 , further comprising the security service:
in response to the identifying the one or more network events indicative of the cyberattack:
reconstructing, based on the structurally deduplicated data comprising the plurality of deduplicated field groups, an instance of a network event associated with the cyberattack; and
determining, based on the instance of the network event, the one or more remediation operations.
6 . The method of claim 1 , wherein;
the structurally deduplicated data is stored in a formatted file,
the deduplicated field groups of redundant fields in the event data and the deduplicated values of redundant values in the event data are stored in one or more deduplication tables in the formatted file, and
the expanded ruleset is a compiled ruleset executable to match rules against values in the one or more deduplication tables.
7 . The method of claim 1 , wherein event data included within the structurally deduplicated data comprises:
a transport layer security (TLS) event;
a domain name system (DNS) event; or
a flow event.
8 . A system comprising:
a memory storing executable instructions that implement a network sensor; and
one or more processors that execute the executable instructions to cause the network sensor to:
determine, based on deep packet inspection (DPI) analysis of network data observed in a local area network, a plurality of network events, wherein the plurality of network events are matched against a first ruleset of the network sensor to monitor for security incidents or threats in the local area network;
generate structurally deduplicated data for the plurality of network events, the structurally deduplicated data comprising:
a plurality of event references associated with event data, wherein the event data is structured in accordance with a data model comprising one or more fields,
a plurality of deduplicated field groups associated with one or more fields of the data model that include redundant data with other field groups of the event data, and
a plurality of deduplicated values associated with one or more values in one or more fields of the event data that are redundant with one or more other values in one or more other fields of the event data,
wherein the structurally deduplicated data is compressed using a compression algorithm;
upload the structurally deduplicated data to a security service, wherein the security service is configured to assess the structurally deduplicated data for security incidents or threats using an expanded ruleset larger than the first ruleset;
cause the security service to, in response to the upload:
identify, based on a single rule in the expanded ruleset matching a single instance of event data indicated by the structurally deduplicated data, one or more network events indicative of a cyberattack on the local area network; and
initiate, based on the one or more network events indicative of a cyberattack, one or more remediation operations.
9 . The system of claim 8 , wherein the security service is implemented using one or more cloud services operating within a remote cloud computing environment.
10 . The system of claim 9 , wherein the security service is configured to:
repeatedly receive and assess uploads of structurally deduplicated data of network events from the local area network.
11 . The system of claim 10 , wherein the security service is configured to:
assess the uploads of structurally deduplicated data to perform different types of security monitoring, including two or more of:
malware detection,
phishing detection, and
intrusion detection.
12 . The system of claim 8 , wherein the security service is configured to:
in response to the identifying the one or more network events indicative of the cyberattack:
reconstruct, based on the structurally deduplicated data comprising the plurality of deduplicated field groups, an instance of a network event associated with the cyberattack; and
determine, based on the instance of the network event, the one or more remediation operations.
13 . The system of claim 8 , wherein:
the structurally deduplicated data is stored in a formatted file,
the deduplicated field groups of redundant fields in the event data and the deduplicated values of redundant values in the event data are stored in one or more deduplication tables in the formatted file, and
the expanded ruleset is a compiled ruleset executable to match rules against values in the one or more deduplication tables.
14 . The system of claim 8 , wherein event data included within the structurally deduplicated data comprises:
a transport layer security (TLS) event;
a domain name system (DNS) event; or
a flow event.
15 . One or more non-transitory computer-accessible storage media storing executable instructions that, when executed by one or more processors, implement a network sensor and cause the network sensor to:
determine, based on deep packet inspection (DPI) analysis of network data observed in a local area network, a plurality of network events, wherein the plurality of network events are matched against a first ruleset of the network sensor to monitor for security incidents or threats in the local area network;
generate structurally deduplicated data for the plurality of network events, the structurally deduplicated data comprising:
a plurality of event references associated with event data, wherein the event data is structured in accordance with a data model comprising one or more fields,
a plurality of deduplicated field groups associated with one or more fields of the data model that include redundant data with other field groups of the event data, and
a plurality of deduplicated values associated with one or more values in one or more fields of the event data that are redundant with one or more other values in one or more other fields of the event data,
wherein the structurally deduplicated data is compressed using a compression algorithm;
upload the structurally deduplicated data to a security service, wherein the security service is configured to assess the structurally deduplicated data for security incidents or threats using an expanded ruleset larger than the first ruleset;
cause the security service to, in response to the upload:
identify, based on a single rule in the expanded ruleset matching a single instance of event data indicated by the structurally deduplicated data, one or more network events indicative of a cyberattack on the local area network; and
initiate, based on the one or more network events indicative of a cyberattack, one or more remediation operations.
16 . The one or more non-transitory computer-accessible storage media of claim 15 , wherein the security service is implemented using one or more cloud services operating within a remote cloud computing environment.
17 . The one or more non-transitory computer-accessible storage media of claim 16 , wherein the security service is configured to:
repeatedly receive and assess uploads of structurally deduplicated data of network events from the local area network.
18 . The one or more non-transitory computer-accessible storage media of claim 17 , wherein the security service is configured to:
assess the uploads of structurally deduplicated data to perform different types of security monitoring, including two or more of:
malware detection,
phishing detection, and
intrusion detection.
19 . The one or more non-transitory computer-accessible storage media of claim 15 , wherein the security service is configured to:
in response to the identifying the one or more network events indicative of the cyberattack:
reconstruct, based on the structurally deduplicated data comprising the plurality of deduplicated field groups, an instance of a network event associated with the cyberattack; and
determine, based on the instance of the network event, the one or more remediation operations.
20 . The one or more non-transitory computer-accessible storage media of claim 16 , wherein:
the structurally deduplicated data is stored in a formatted file,
the deduplicated field groups of redundant fields in the event data and the deduplicated values of redundant values in the event data are stored in one or more deduplication tables in the formatted file, and
the expanded ruleset is a compiled ruleset executable to match rules against values in the one or more deduplication tables.