Detection and mitigation of data compromises in adversarial environments
Detection and mitigation of data source compromises in an adversarial information environment, featuring the ability to scan for, ingest and process, and then use relational, wide column, and graph stores for capturing entity data, their relationships, and actions associated with them. Metadata is gathered and linked to the ingested data, which provides a broader contextual view of the environment leading up to and during an event of interest. Data quality analysis is conducted as data is ingested in order to identify if a data source may be compromised. The results are used to manage the reputation of the contributing data sources.
1 . A system for detection and mitigation of data compromises in adversarial environments, comprising one or more computers with executable instructions that, when executed, cause the system to:
extract metadata from each of a plurality of data pulls from application programming interfaces (APIs) of a plurality of data services, wherein the metadata identifies the respective data service and a content of the data in the respective data pull;
for each data pull from each API, use the data and the extracted metadata to identify and measure a plurality of data quality metrics;
establish a reputation score for the data in each data pull by:
comparing the extracted metadata for identifying the content of the data against a breach content database;
comparing the extracted metadata for identifying a source of the data against a vulnerabilities and exploits database;
comparing the plurality of data quality metrics for determining if a data source has been compromised against a baseline database; and
generating a component reputation score for that data pull based on the comparisons;
updating the breach content database with a new reputation score; and
publishing the update on a first publication and subscription data feed for the breach content database; and
generate a recommendation of data services based on the new reputation score.
2 . The system of claim 1 , wherein the system is further caused to:
for each component reputation score generated:
create a new node in a reputation relationship graph representing the component reputation score, and
associate the new node via one or more edges with one or more existing nodes in the reputation relationship graph for the data service from which the data was pulled;
generate a new reputation score for each data service from which data was pulled by analyzing the new nodes and edges of the reputation relationship graph by, for each data pull from each data service:
updating the vulnerabilities and exploits database with the new reputation score; and
publishing the update on a second publication and subscription data feed for the vulnerabilities and exploits database;
wherein the reputation relationship graph logically organizes the plurality of data services into a distributed collaborative database with a reliability of each data service being indicated by its reputation score.
3 . The system of claim 1 , wherein the breach content database is stored in a non-volatile storage device of a cloud computing platform, the breach content database comprising historical data breach records.
4 . The system of claim 1 , wherein the vulnerability and exploits database stored in a non-volatile storage device of a cloud computing platform, the vulnerability and exploits database comprising information about vulnerabilities and exploits associated with a data service.
5 . The system of claim 1 , wherein the system is further caused to:
send the recommendation of data services to an end user;
receive feedback from the end user; and
use the feedback to update the new reputation score.
6 . The system of claim 1 , wherein the system is further caused to:
extract data from external score and metric databases;
extract user data from internal databases;
generate a cyber score from the combination of external data and internal data, wherein the cyber score is calculated from one or more of a plurality of scoring metrics;
send the cyber score to the cyber open market exchange; and
wherein the cyber open market exchange facilitates transactional behavior among market participants.
7 . The system of claim 1 , wherein the plurality of data quality metrics comprises at least one of data source relevance, data source consistency, data source connectedness, data source timing, data source completeness, data source reliability, data source accuracy, and data source market reputation.
8 . A method for detection and mitigation of data compromises in adversarial environments, comprising the steps of:
extracting metadata from each of a plurality of data pulls from application programming interfaces (APIs) of a plurality of data services, wherein the metadata identifies the respective data service and a content of the data in the respective data pull;
for each data pull from each API, using the data and the extracted metadata to identify and measure a plurality of data quality metrics;
establishing a reputation score for the data in each data pull by:
comparing the extracted metadata for identifying the content of the data against a breach content database;
comparing the extracted metadata for identifying a source of the data against a vulnerabilities and exploits database;
comparing the plurality of data quality metrics for determining if a data source has been compromised against a baseline database; and
generating a component reputation score for that data pull based on the comparisons;
updating the breach content database with a new reputation score; and
publishing the update on a first publication and subscription data feed for the breach content database;
generating a recommendation of data services based on the new reputation score.
9 . The method of claim 8 , further comprising the steps of:
for each component reputation score generated:
creating a new node in a reputation relationship graph representing the component reputation score, and
associating the new node via one or more edges with one or more existing nodes in the reputation relationship graph for the data service from which the data was pulled;
generating a new reputation score for each data service from which data was pulled by analyzing the new nodes and edges of the reputation relationship graph by, for each data pull from each data service:
updating the vulnerabilities and exploits database with the new reputation score; and
publishing the update on a second publication and subscription data feed for the vulnerabilities and exploits database;
wherein the reputation relationship graph logically organizes the plurality of data services into a distributed collaborative database with a reliability of each data service being indicated by its reputation score.
10 . The method of claim 8 , further comprising the step of storing the breach content database in a cloud computing platform, the breach content database comprising historical data breach records.
11 . The method of claim 8 , further comprising the step of storing the vulnerability and exploits database in a cloud computing platform, the vulnerability and exploits database comprising information about vulnerabilities and exploits associated with a data service.
12 . The method of claim 8 , further comprising the steps of:
sending the recommendation of data services to an end user;
receiving feedback from the end user; and
using the feedback to update the new reputation score.
13 . The method of claim 8 , further comprising the steps of:
extracting data from external score and metric databases;
extracting user data from internal databases;
generating a cyber score from the combination of external data and internal data, wherein the cyber score is calculated from a one or more of plurality of scoring metrics; and
sending the cyber score to the cyber open market exchange;
wherein the cyber open market exchange facilitates transactional behavior among market participants.
14 . The method of claim 8 , wherein the plurality of data quality metrics comprises at least one of data source relevance, data source consistency, data source connectedness, data source timing, data source completeness, data source reliability, data source accuracy, and data source market reputation.
15 . A computing system for detection and mitigation of data compromises in adversarial environments, the computing system comprising:
one or more hardware processors configured for:
extracting metadata from each of a plurality of data pulls from application programming interfaces (APIs) of a plurality of data services, wherein the metadata identifies the respective data service and a content of the data in the respective data pull;
for each data pull from each API, using the data and the extracted metadata to identify and measure a plurality of data quality metrics;
establishing a reputation score for the data in each data pull by:
comparing the extracted metadata for identifying the content of the data against a breach content database;
comparing the extracted metadata for identifying a source of the data against a vulnerabilities and exploits database;
comparing the data quality metrics for determining if a data source has been compromised against a baseline database; and
generating a component reputation score for that data pull based on the comparisons;
updating the breach content database with a new reputation score; and
publishing the update on a first publication and subscription data feed for the breach content database; and
generating a recommendation of data services based on the new reputation score.
16 . The computing system of claim 15 , wherein the one or more hardware processors are further configured for:
for each component reputation score generated:
creating a new node in a reputation relationship graph representing the component reputation score, and
associating the new node via one or more edges with one or more existing nodes in the reputation relationship graph for the data service from which the data was pulled;
generating a new reputation score for each data service from which data was pulled by analyzing the new nodes and edges of the reputation relationship graph by, for each data pull from each data service:
updating the vulnerabilities and exploits database with the new reputation score; and
publishing the update on a second publication and subscription data feed for the vulnerabilities and exploits database;
wherein the reputation relationship graph logically organizes the plurality of data services into a distributed collaborative database with a reliability of each data service being indicated by its reputation score.
17 . The computing system of claim 15 , wherein the computing system wherein the one or more hardware processors are further configured for storing the breach content database in a cloud computing platform, the breach content database comprising historical data breach records.
18 . The computing system of claim 15 , wherein the computing system wherein the one or more hardware processors are further configured for storing the vulnerability and exploit database in a cloud computing platform, the vulnerability and exploits database comprising information about vulnerabilities and exploits associated with a data service.
19 . The computing system of claim 15 , wherein the computing system wherein the one or more hardware processors are further configured for:
sending the recommendation of data services to an end user;
receiving feedback from the end user; and
using the feedback to update the new reputation score.
20 . The computing system of claim 15 , wherein the computing system wherein the one or more hardware processors are further configured for:
extracting data from external score and metric databases;
extracting user data from internal databases;
generating a cyber score from the combination of external data and internal data, wherein the cyber score is calculated from one or more of a plurality of scoring metrics; and
sending the cyber score to the cyber open market exchange;
wherein the cyber open market exchange facilitates transactional behavior among market participants.
21 . The computing system of claim 15 , wherein the plurality of data quality metrics comprises at least one of data source relevance, data source consistency, data source connectedness, data source timing, data source completeness, data source reliability, data source accuracy, and data source market reputation.
22 . Non-transitory, computer-readable storage media having computer-executable instructions embodied thereon that, when executed by one or more processors of a computing system for detection and mitigation of data compromises in adversarial environments, cause the computing system to:
extract metadata from each of a plurality of data pulls from application programming interfaces (APIs) of a plurality of data services, wherein the metadata identifies the respective data service and a content of the data in the respective data pull;
for each data pull from each API, use the data and the extracted metadata to identify and measure a plurality of data quality metrics;
establish a reputation score for the data in each data pull by:
comparing the extracted metadata for identifying the content of the data against a breach content database;
comparing the extracted metadata for identifying a source of the data against a vulnerabilities and exploits database;
comparing the data quality metrics for determining if a data source has been compromised against a baseline database; and
generating a component reputation score for that data pull based on the comparisons;
updating the breach content database with a new reputation score;
publishing the update on a first publication and subscription data feed for the breach content database; and
generating a recommendation of data services based on the new reputation score.
23 . The non-transitory, computer-readable storage media of claim 22 , wherein the computing system is further caused to:
for each component reputation score generated:
create a new node in a reputation relationship graph representing the component reputation score; and
associate the new node via one or more edges with one or more existing nodes in the reputation relationship graph for the data service from which the data was pulled; and
generate a new reputation score for each data service from which data was pulled by analyzing the new nodes and edges of the reputation relationship graph by, for each data pull from each data service:
updating the vulnerabilities and exploits database with the new reputation score; and
publishing the update on a second publication and subscription data feed for the vulnerabilities and exploits database;
wherein the reputation relationship graph logically organizes the plurality of data services into a distributed collaborative database with a reliability of each data service being indicated by its reputation score.
24 . The non-transitory, computer-readable storage media of claim 22 , wherein the breach content database is stored in a non-volatile storage device of a cloud computing platform, the breach content database comprising historical data breach records.
25 . The non-transitory, computer-readable storage media of claim 22 , wherein the vulnerability and exploits database stored in the non-volatile storage device of a cloud computing platform, the vulnerability and exploits database comprising information about vulnerabilities and exploits associated with a data service.
26 . The non-transitory, computer-readable storage media of claim 22 , wherein the computing system is further caused to:
send the recommendation of data services to an end user;
receive feedback from the end user; and
use the feedback to update the new reputation score.
27 . The non-transitory, computer-readable storage media of claim 22 , wherein the computing system is further caused to:
extract data from external score and metric databases;
extract user data from internal databases;
generate a cyber score from the combination of external data and internal data, wherein the cyber score is calculated from one or more of a plurality of scoring metrics;
send the cyber score to the cyber open market exchange; and
wherein the cyber open market exchange facilitates transactional behavior among market participants.
28 . The non-transitory, computer-readable storage media of claim 22 , wherein the plurality of data quality metrics comprises at least one of data source relevance, data source consistency, data source connectedness, data source timing, data source completeness, data source reliability, data source accuracy, and data source market reputation.