SCALABLE DARKWEB ANALYTICS
Scalable darkweb analytics are provided by crawling content offered by a plurality of onion services; sanitizing the content from each onion service of the plurality of onion services into sanitized content that represents potentially malicious content in a non-malicious form; storing, in a database, the sanitized content in association with a unique identity for each onion service of the plurality of onion services; receiving a request for information related to a given onion service of the plurality of onion services; and providing, in response to the request, the information based on the sanitized content
1 . An onion service analysis system comprising:
a memory; and
a processor in communication with the memory, the processor configured to:
execute an onion service analysis model, wherein the onion service analysis model comprises a crawling pipeline, an analysis pipeline, and a distributed datastore;
extract information from onion services using the crawling pipeline;
store the extracted information in the distributed datastore;
analyze the stored extracted information using the analysis pipeline of the onion service analysis model; and
determine a first property based on the analyzed information.
2 . The system of claim 1 , wherein the onion service analysis model further comprises an applications module, and wherein the processor is further configured to run at least one application hosted with the applications module.
3 . The system of claim 1 , wherein the onion service analysis model further comprises an applications module, and wherein the processor is further configured to run an analytical search engine hosted with the applications module.
4 . The system of claim 1 , wherein the first property includes a language of an onion service.
5 . The system of claim 1 , wherein the first property includes an illicitness status.
6 . The system of claim 1 , wherein the first property includes an onion services' cryptocurrency address.
7 . The system of claim 1 , wherein the analysis pipeline includes a plurality of artificial neural networks.
8 . The system of claim 1 , wherein the processor is further configured to determine a second property based on the analyzed information.
9 . A method, comprising:
crawling content offered by a plurality of onion services;
sanitizing the content from each onion service of the plurality of onion services into sanitized content that represents potentially malicious content in a non-malicious form;
storing, in a database, the sanitized content in association with a unique identity for each onion service of the plurality of onion services;
receiving a request for information related to a given onion service of the plurality of onion services; and
providing, in response to the request, the information based on the sanitized content.
10 . The method of claim 9 , wherein the malicious content includes images that are sanitized by:
hashing the images via difference and perceptual hashing to capture features/scenes present in the images.
11 . The method of claim 9 , wherein the malicious content includes illicit images.
12 . The method of claim 9 , further comprising:
pre-filtering images included in the content that are below a given number of pixels to omit from the content that is sanitized.
13 . The method of claim 9 , further comprising:
fingerprinting a source camera for any images in the content via photo-response non-uniformity (PRNU) noise hashing.
14 . The method of claim 9 , further comprising:
wherein the request for information includes a reverse image search request.
15 . A non-transitory computer readable device including instructions that, when executed by a processor, perform operations comprising:
crawling content offered by a plurality of onion services;
sanitizing the content from each onion service of the plurality of onion services into sanitized content that represents potentially malicious content in a non-malicious form;
storing, in a database, the sanitized content in association with a unique identity for each onion service of the plurality of onion services;
receiving a request for information related to a given onion service of the plurality of onion services; and
providing, in response to the request, the information based on the sanitized content.
16 . The device of claim 15 , wherein the malicious content includes images that are sanitized by:
hashing the images via difference and perceptual hashing to capture features/scenes present in the images.
17 . The device of claim 15 , wherein the malicious content includes illicit images.
18 . The device of claim 15 , the operations further comprising:
pre-filtering images included in the content that are below a given number of pixels to omit from the content that is sanitized.
19 . The device of claim 15 , the operations further comprising:
fingerprinting a source camera for any images in the content via photo-response non-uniformity (PRNU) noise hashing.
20 . The device of claim 15 , the operations further comprising:
wherein the request for information includes a reverse image search request.