System and method of detecting common patterns within unstructured data elements retrieved from big data sources
A method and system for detection of common patterns within unstructured data elements. The method includes searching a plurality of unstructured data elements extracted from big data sources to identify a plurality of patches; extracting the identified plurality of patches; generating, by a signature generator system, at least one signature for each patch; identifying common patterns among the at least one generated signature; clustering the signatures having the identified common patterns; and correlating the generated clusters to identify associations between the respective identified common patterns.
1. A method for detection of common patterns within unstructured data elements, comprising:
searching a plurality of unstructured data elements extracted from big data sources to identify a plurality of patches;
extracting the plurality of patches that were identified;
generating, by a signature generator system, at least one signature for each one patch of the plurality of patches to generate a plurality of signatures for the plurality of patches, wherein the signature generator system includes a plurality of computational cores configured to receive the plurality of patches, each one computational core of the plurality of computational cores having properties that are at least partly statistically independent of other ones of the plurality of computational cores, wherein the properties of the one computational core are set independently of each other computational core of the plurality of computational cores;
identifying common patterns among the plurality of signatures;
clustering the plurality of signatures having the common patterns that were identified to generate a plurality of clusters; and
correlating the plurality of clusters to identify associations between the respective common patterns that were identified.
2. The method of claim 1 , further comprising:
re-clustering clusters of the plurality of clusters having associations between their respective common patterns to create a common concept.
3. The method of claim 2 , further comprising:
storing, in a database, at least one of: the common concept, the plurality of signatures, the plurality of clusters, and the common patterns that were identified.
4. The method of claim 1 , wherein the unstructured data elements are: specific interest unstructured data elements; or higher interest unstructured data elements than other unstructured data elements in the big data sources.
5. The method of claim 4 , wherein each one of the specific interest unstructured data elements has an entropy level above a predefined threshold.
6. The method of claim 1 , wherein the unstructured data elements are retrieved from a plurality of big data sources having similar classifications.
7. The method of claim 1 , wherein each of the plurality of unstructured data elements is at least one of: a multimedia content, a book, a document, metadata, a collection of health records, audio, video, analog data, a file, unstructured text, or a web page.
8. The method of claim 1 , wherein identifying the common patterns further comprises:
matching the plurality of signatures to each other to generate a plurality of matched signatures;
assigning a matching score to each pair of the plurality of matched signatures;
comparing the matching score of each pair of the plurality of matched signatures to a preconfigured threshold; and
upon determining that the matching scores of one pair of the plurality of matched signatures is above the predefined threshold, determining the one pair to have the common pattern.
9. The method of claim 1 , wherein correlating the generated clusters further comprises:
determining, for each one pair of the plurality of clusters, if the one pair contains at least a preconfigured number of matching signatures.
10. A non-transitory computer readable medium having stored thereon instructions for causing one or more processing units to execute the method according to claim 1 .
11. A system for analyzing unstructured data, comprising:
a network interface for allowing connectivity to a plurality of big data sources;
a processing unit; and
a memory connected to the processing unit, the memory containing instructions that, when executed by the processing unit, configure the system to:
search a plurality of unstructured data elements extracted from the big data sources to identify a plurality of patches;
extract the plurality of patches that were identified;
generate, by a signature generator system, at least one signature for each one patch of the plurality of patches to generate a plurality of signatures for the plurality of patches, wherein the signature generator system includes a plurality of computational cores configured to receive the plurality of patches, each one computational core of the plurality of computational cores having properties that are at least partly statistically independent of other ones of the plurality of computational cores, wherein the properties of the one computational core are set independently of each other computational core of the plurality of computational cores;
identify common patterns among the plurality of signatures;
cluster the plurality of signatures having the common patterns that were identified to generate a plurality of clusters; and
correlate the plurality of clusters to identify associations between the respective common patterns that were identified.
12. The system of claim 11 , wherein the system is further configured to:
re-cluster clusters of the plurality of clusters having associations between their respective common patterns to create a common concept.
13. The system of claim 12 , wherein the system is further configured to:
store, in a database, at least one of: the common concept, the plurality of signatures, the plurality of clusters, and the common patterns that were identified.
14. The system of claim 11 , wherein the unstructured data elements are: specific interest unstructured data elements; or higher interest unstructured data elements than other unstructured data elements in the big data sources.
15. The system of claim 14 , wherein each one of the specific interest unstructured data elements has an entropy level above a predefined threshold.
16. The system of claim 11 , wherein the unstructured data elements are retrieved from a plurality of big data sources having similar classifications.
17. The system of claim 11 , wherein each of the plurality of unstructured data elements is at least one of: a multimedia content, a book, a document, metadata, a collection of health records, audio, video, analog data, a file, unstructured text, or a web page.
18. The system of claim 11 , wherein the system is further configured to:
match the plurality of signatures to each other to generate a plurality of matched signatures;
assign a matching score to each pair of the plurality of matched signatures;
compare the matching score of each pair of the plurality of matched signatures to a preconfigured threshold; and
upon determining that the matching scores of one pair of the plurality of matched signatures is above the predefined threshold, determine the one pair to have the common pattern.
19. The system of claim 11 , wherein the system is further configured to:
determine, for each one pair of the plurality of clusters, if the one pair contains at least a preconfigured number of matching signatures.