System and method of detecting common patterns within unstructured data elements retrieved from big data sources
A method for detection of common patterns within unstructured data elements. The method includes extracting a plurality of unstructured data elements retrieved from a plurality of big data sources; generating at least one signature for each of the plurality of unstructured data elements; identifying common patterns among the generated signatures; clustering the signatures identified to have common patterns; and correlating the generated clusters to identify associations between their respective identified common patterns.
1. A method for detection of common patterns within unstructured data elements, comprising:
extracting a plurality of unstructured data elements retrieved from a plurality of big data sources;
generating, by a signature generator system, at least one signature for each of the plurality of unstructured data elements, wherein the signature generator system includes a plurality of computational cores configured to receive the plurality of unstructured data elements, each computational core of the plurality of computational cores having properties that are at least partly statistically independent of other of the computational cores, the properties are set independently of each other core;
identifying common patterns among the generated signatures;
clustering the signatures identified to have common patterns; and
correlating the generated clusters to identify associations between their respective identified common patterns.
2. The method of claim 1 , further comprising:
re-clustering clusters having association between their respective common patterns to create a common concept.
3. The method of claim 2 , further comprising:
storing in a database at least one of the common concept, the generated signatures, the clusters, and the identified common patterns.
4. The method of claim 1 , wherein the extracted unstructured data elements are of specific interest, or otherwise of higher interest than other elements comprised in a collected unstructured data.
5. The method of claim 1 , wherein each of the at least one generated signature is robust to noise and distortion.
6. The method of claim 1 , wherein the plurality of unstructured data elements unstructured data elements are retrieved from the plurality of big data sources having a similar classification.
7. The method of claim 1 , wherein each of the plurality of unstructured data elements is at least one of: a multimedia content, a book, a document, metadata, a collection of health records, audio, video, analog data, a file, unstructured text, and a web page.
8. The method of claim 1 , wherein identifying the common patterns further comprises:
matching the generated signatures to each other;
assigning a matching score for each two matched signatures;
comparing the matching score to a preconfigured threshold; and
determining the two signatures to have the common pattern when their respective matching score is greater than the preconfigured threshold.
9. The method of claim 1 , wherein the correlating of the generated clusters further comprises:
determining for each at least two clusters of the generated clusters if they contain a preconfigured number of matching signatures.
10. A non-transitory computer readable medium having stored thereon instructions for causing one or more processing units to execute the method according to claim 1 .
11. A system for analyzing unstructured data, comprising:
a network interface for allowing connectivity to a plurality of big data sources;
a signature generator system;
a processor; and
a memory connected to the processor, the memory contains instructions that when executed by the processor, configure the system to:
extract a plurality of unstructured data elements retrieved from a plurality of big data sources;
generate by the signature generator system at least one signature for each of the plurality of unstructured data elements, wherein the signature generator system includes a plurality of computational cores enabled to receive the plurality of unstructured data elements, each computational core of the plurality of computational cores having properties that are at least partly statistically independent of other of the computational cores, the properties are set independently of each other core;
identify common patterns among the generated signatures;
cluster the signatures identified to have common patterns; and
correlate the generated clusters to identify associations between their respective identified common patterns.
12. The system of claim 11 , wherein the system is further configured to re-cluster clusters having association between their respective common patterns to create a common concept.
13. The system of claim 12 , wherein the system further comprises a database for storage of the common concept, the generated signatures, the clusters, and the identified common patterns.
14. The system of claim 11 , wherein the extracted unstructured data elements are of specific interest, or otherwise of higher interest than other elements comprised in a collected unstructured data.
15. The system of claim 11 , wherein the plurality of unstructured data elements are retrieved from the plurality of big data sources having a similar classification.
16. The system of claim 11 , wherein each of the plurality of unstructured data elements is at least one of: a multimedia content, a book, a document, metadata, a collection of health records, audio, video, analog data, a file, unstructured text, and a web page.
17. The system of claim 11 , wherein the each of the at least one generated signature is robust to noise and distortion.
18. The system of claim 11 , wherein the system is further configured to:
match the generated signatures to each other;
assign a matching score for each of two matched signatures;
compare the matching score to a preconfigured threshold; and
determine the two signatures to have the common pattern when their respective matching score is greater than the preconfigured threshold.
19. The system of claim 11 , wherein the system is further configured to determine for each at least two clusters of the generated clusters if they contain a preconfigured number of matching signatures.