Method for reducing false-positives for identification of digital content
Many areas of investigation require searching through data that may be of interest. In a first method step, a digital content element is provided. The digital content element may have any suitable format or data structure of interest to a searching entity. The digital content element may be a particular data file that is of interest to a searching entity. In a second step, the digital content element is compared with a first set of data provided by a combination of a second set of data and a third set of data. The first set of data is a collection of known digital content elements that are of interest to a searching entity, for example contraband digital content elements or digital content elements owned by or represented by the searching entity. In a third method step, the digital content element is identified as known if the digital content element is detected within the first set of data.
1 . A method for identifying at least one digital content element, the digital content element forming a part of a set of digital content, the method comprising:
providing the digital content element;
comparing the digital content element with a first set of data provided by a combination of a second set of data and a third set of data, wherein the second set of data represents known data of interest to a searching entity and wherein the third set of data represents known non-identifying digital content elements including one or more non-unique digital content elements relating to a file structure or meta data;
if the digital content element is detected within the first set of data, then identifying the digital content element as digital content of interest; and
adjusting the second set of data or the third set of data to include one or more additional data elements based on determining whether the one or more additional data elements are not already included in either of the second set of data and the third set of data.
2 . The method according to claim 1 , wherein the step of comparing comprises:
comparing the digital content element with the second set of data; and
if the digital content element is detected within the second set of data, then comparing the digital content element with the third set of data, and
wherein the step of identifying comprises:
if the digital content element is detected within the second set of data and if the digital content element is not detected within the third set of data, then identifying the digital content element as digital content of interest.
3 . The method according to claim 1 , wherein the step of comparing comprises:
comparing the digital content element with the third set of data; and
if the digital content element is not detected within the third set of data, then comparing the digital content element with the second set of data, and wherein the step of identifying comprises:
if the digital content element is detected within the second set of data and if the digital content is not detected within the third set of data, then identifying the digital content as digital content of interest.
4 . The method according to claim 1 , wherein the step of comparing comprises:
creating the first set of data by subtracting the third set of data from the second set of data; and
comparing the digital content element with the first set of data, and wherein the step of identifying comprises:
if the digital content element is detected within the first set of data, then identifying the digital content element as digital content of interest.
5 . The method according to claim 1 , wherein creating the first set of data comprises:
comparing each element in the second set of data with each element in the third set of data; and
if an element of the second set of data is not detected within the third set of data, then adding the element to the first set of data.
6 . The method according to claim 1 , wherein the set of digital content comprises:
at least one data file, and wherein the digital content element is a fragment of the data file.
7 . The method according to claim 1 , wherein the digital content element is defined in the structure of the set of digital content.
8 . The method according to claim 1 , wherein the digital content element is a block.
9 . The method according to claim 8 , wherein the block corresponds to a network packet or a payload portion of a network packet.
10 . The method according to claim 8 , wherein the block corresponds to one of: a memory block; a disk storage; a disk storage sector; or a block comprising at least one data file.
11 . The method according to claim 8 , wherein the block has a fixed size.
12 . The method according to claim 1 , wherein the digital content element has been encoded by the way of one of: a hashing function; or a locality-sensitive hashing function.
13 . The method according to claim 1 , wherein at least one of the second set of data or the third set of data have been encoded by way of a hashing function.
14 . The method according to claim 1 , wherein the second set of data and the third set of data is one of: a cuckoo filter; or a bloom filter.
15 . The method according to claim 1 , further comprising: receiving a fourth set of data identified as known, wherein the fourth set of data comprises misidentified digital content elements;
comparing the fourth set of data with the second set of data; and
if a misidentified digital content element is detected within the second set of data, adding the misidentified digital content element to the third set of data.
16 . The method according to claim 1 , wherein at least one of the second set of data and the third set of data comprises a plurality of respective subsets of data.
17 . The method of claim 1 , wherein the one or more additional data elements comprise at least one set of population data comprising a plurality of population data elements, and adjusting the second set of data or the third set of data comprises:
comparing each population data element with the third set of data;
if a population data element is not detected within the third set of data, then compare the population data element with the second set of data;
if a population data element is not detected within the second set of data, then adding the population data element of the second set of data;
and if a population data element is detected within the second set of data, then adding the population data element to the third set of data.
18 . The method according to claim 17 , wherein the step of providing comprises:
comparing the at least one set of population data with a population database, the population database comprising at least one known sets of population data;
if a set of population data is not detected in the population database, then adding the set of population data to the population database;
and if a set of population data is detected in the population database, then ignoring the set of population data.
19 . The method according to claim 18 , wherein the population database comprises at least one representation of at least one known sets of population data, and wherein the step of comparing comprises:
providing a representation of each of the at least one set of population data; and
comparing the representation of each of the at least one set of population data with each of the at least one representation of the at least one known sets of population data.
20 . The method of claim 1 , wherein the one or more additional data elements comprise at least one set of population data comprising a plurality of population data elements, and adjusting the second set of data or the third set of data comprises:
comparing each population data element with the second set of data;
if a population data element is not detected within the second set of data, then adding the population data element to the second set of data; and
if a population data element is detected within the second set of data, then adding the population data element to the third set of data.