File collection method for subsequent malware detection
A computer-implemented method for collecting files transmitted on a network for subsequent malware analysis is disclosed. The method comprises determining, by a sensor, that a file has been transmitted on the network; transmitting, from the sensor to an intermediate agent, the file; storing, by the intermediate agent, the file, pending a determination whether metadata of the file indicates an identical copy of the file has likely already been stored by a source collection subsystem; responsive to a determination that the file has not likely already been stored by the source collection subsystem, transmitting the file from the intermediate agent to the source collection subsystem; verifying, by the source collection subsystem, that the file has not likely already been stored by the source collection subsystem; and storing the file for future analysis.
1. A computer-implemented method for collecting files transmitted on a network for subsequent malware analysis, comprising:
determining, by a sensor, that a file has been transmitted on the network;
transmitting, from the sensor to an intermediate agent, the file;
storing, by the intermediate agent, the file, pending a determination whether metadata of the file indicates an identical copy of the file has already been stored by a source collection subsystem;
responsive to a determination that the file has not already been stored by the source collection subsystem, transmitting the file from the intermediate agent to the source collection subsystem;
verifying, by the source collection subsystem, that the file has not already been stored by the source collection subsystem; and
storing the file for future analysis.
2. The method of claim 1 wherein both the intermediate agent and the source collection subsystem track a set of files that have been transmitted to the source collection subsystem.
3. The method of claim 2 , wherein the intermediate agent sends the metadata to the source collection subsystem if the intermediate agent does not find the file in the set of files that have been transmitted to the source collection subsystem, and does not send the metadata to the source collection subsystem if the intermediate agent does find the file in the set of files that have been transmitted to the source collection subsystem.
4. The method of claim 2 , wherein the set of files that have been transmitted to the source collection subsystem is curated to store only a set of files seen within a particular recent interval of time.
5. The method of claim 4 , wherein the intermediate agent sends the metadata to the source collection subsystem if the intermediate agent does not find the file in the set of files that have been transmitted to the source collection subsystem, and does not send the metadata to the source collection subsystem if the intermediate agent does find the file in the set of flies that have been transmitted to the source collection subsystem.
6. The method of claim 1 , wherein recursive deduplication is performed by the source collection subsystem when one or more contents of the file themselves are themselves extractable files.
7. The method of claim 1 , wherein the intermediate agent or the source collection subsystem tracks received files using a deduplication technique.
8. The method of claim 1 , wherein, responsive to a determination that the file has already been stored by the source collection subsystem, the file is deleted by the intermediate agent or allowed to be overwritten in memory.
9. The method of claim 1 , wherein, responsive to a lack of a determination whether the file has already been stored by the source collection subsystem during a predetermined or adaptive window of time after receipt of the file, the file is deleted by the intermediate agent or allowed to be overwritten in memory.
10. The method of claim 1 , wherein the collection subsystem deletes the file from memory if it does not verify that the file has not already been stored by the source collection subsystem.
11. A system for collecting files transmitted on a network for subsequent malware analysis, comprising:
a sensor connected to the network;
an intermediate agent computing device;
a source collection subsystem;
one or more databases;
one or more processors; and
non-transitory memory comprising instructions that, when executed by the one or more processors, cause the one or more processors to:
determine, by a sensor, that a file has been transmitted on the network;
transmit, from the sensor to an intermediate agent computing device, the file;
store, by the intermediate agent computing device, the file, pending a determination whether metadata of the file indicates an identical copy of the file has already been stored by the source collection subsystem;
responsive to a determination that the file has not already been stored by the source collection subsystem transmit the file from the intermediate agent computing device to the source collection subsystem;
verify, by the source collection subsystem, that the file has not already been stored by the source collection subsystem; and
store the file in at least one of the one or more databases for future analysis.
12. The system of claim 11 , wherein both the intermediate agent computing device and the source collection subsystem track a set of files that have been transmitted to the source collection subsystem.
13. The system of claim 12 , wherein the intermediate agent computing device sends the metadata to the source collection subsystem if the intermediate agent computing device does not find the file in the set of files that have been transmitted to the source collection subsystem, and does not send the metadata to the source collection subsystem if the intermediate agent computing device does find the file in the set of files that have been transmitted to the source collection subsystem.
14. The system of claim 12 , wherein the set of files that have been transmitted to the source collection subsystem is curated to store only a set of files seen within a particular recent interval of time.
15. The system of claim 14 , wherein the intermediate agent sends the metadata to the source collection subsystem if the intermediate agent does not find the file in the set of files that have been transmitted to the source collection subsystem, and does not send the metadata to the source collection subsystem if the intermediate agent does find the file in the set of files that have been transmitted to the source collection subsystem.
16. The system of claim 11 , wherein recursive deduplication is performed by the source collection subsystem when one or more contents of the file themselves are themselves extractable files.
17. The system of claim 11 , wherein the intermediate agent computing device or the source collection subsystem tracks received files using a deduplication technique.
18. The system of claim 11 , wherein, responsive to a determination that the file has already been stored by the source collection subsystem, the file is deleted by the intermediate agent computing device or allowed to be overwritten in memory.
19. The system of claim 11 , wherein, responsive to a lack of a determination whether the file has likely already been stored by the source collection subsystem during a predetermined or adaptive window of time after receipt of the file, the file is deleted by the intermediate agent computing device or allowed to be overwritten in memory.
20. The system of claim 11 , wherein the source collection subsystem deletes the file from memory if it does not verify that the file has not already been stored by the source collection subsystem.