Modular cybersecurity engine in a data intelligence system
Methods, systems, and computer storage media for providing a modular cybersecurity platform are described. The modular cybersecurity platform is implemented using a modular cybersecurity engine that operates based on an analytical framework for dynamic data analysis and data management in a data intelligence system. In particular, the analytical framework is based on complementary modular components that are designed to interoperate in the modular cybersecurity engine. The modular cybersecurity engine includes a modular distributed system, a credential detection system, and a credential semantic graph system. The modular cybersecurity engine supports cybersecurity and sensitive data management scenarios that can empower investigators in various investigations, and provide automated flows that are highly scalable and support different types of functionality (e.g., priority embedding pipeline, credential scanning, and credential semantic graph analysis). The utility of the modular cybersecurity engine is demonstrated by its wide-ranging application in addressing complex cybersecurity challenges and sensitive data management tasks.
1 . A computerized system comprising:
one or more computer processors; and
computer memory storing computer-useable instructions that, when used by the one or more computer processors, cause the one or more computer processors to perform operations, the operations comprising:
accessing, at a priority embedder pipeline, a dataset comprising a plurality of data items, the plurality of data items comprising a first data item and a second data item;
assigning corresponding priority values to a first set of content instances in the first data item;
identifying a first set of unique content instances in the first data item based on de-duplicating the first set of content instances based on the corresponding priority values and identified matching content instances in the first set of content instances;
assigning corresponding priority values to a second set of content instances in the second data item;
identifying a second set of unique content instances in the second data item based on de-duplicating the second set of content instances based on the corresponding priority values and identified matching content instances in the second set of content instances;
identifying a third set of unique content instances in the first data item and the second data item based on de-duplicating the first set of unique content instances and the second set of unique content instances based on the corresponding priority values and identified matching content instances in the first set of unique content instances and the second set of unique content instances; and
generating embedding vectors for the third set of unique content instances, wherein the third set of unique content instances are higher-priority content instances identified for vector embedding priority in the priority embedder pipeline.
2 . The system of claim 1 , wherein assigning a priority value to a content instance is based on contextual data associated with a data item and a content instance, wherein contextual data comprises an attribute of the data item or content instance that is used in calculate the priority value.
3 . The system of claim 1 , wherein the third set of unique content instances are stored in an embedding queue that supports dynamic reordering of content instances in the embedding queue,
wherein a first unique content instance in the third set of unique content instances is reordered based on a new unique content instance that is added to the embedded queue, and
wherein the new unique content instance is associated with a higher priority value and matches the first unique content instance.
4 . The system of claim 1 , the operations further comprising:
bypassing generating an embedding vector for a first unique content instance in the third set of unique content instances based on determining an embedding vector of the first unique content instance already exists; and
generating a semantic index based on the a plurality of embedding vectors associated with the dataset, wherein the plurality of data items are documents and a content instance is a sentence, wherein an embedding vector represents the content instance associated with contextual data in a fixed dimensional space for computation analysis.
5 . The system of claim 1 , wherein the priority embedder pipeline de-duplicates content instances within a data item and across data items based on uniqueness of the content instances and corresponding priority values of the content instances, wherein the priority values are calculated based on contextual data.
6 . The system of claim 1 , wherein the dataset is associated with a modular distributed engine that supports interoperability between different cybersecurity tools and provides a centralized hub and cybersecurity workflows for programmatic interactions with the modular distributed engine.
7 . The system of claim 6 , wherein the modular distributed engine further comprises a permissions engine that manages and enforces access rights and permissions; and an audit trail engine that tracks and audits user actions.
8 . The system of claim 6 , wherein the modular distributed engine further comprises a priority embedder that supports prioritizing content instances of data items and generating embedding vectors for content instances.
9 . The system of claim 1 , the operations further comprising generating a semantic index comprising the embedding vectors, wherein the semantic index supports executing semantic searches using the embedding vectors.
10 . The system of claim 1 , wherein the embedding vectors are sentence embeddings comprising numerical representations of sentences extracted from a breached dataset to support detection of sensitive information.