System and method for information slingshot over a network tapestry and granularity of a tick
Systems and methods for file transfer and processing in a network environment are disclosed. In one embodiment, the system may comprise one or more processors. The one or more processors may be coupled to a first device. The one or more processors may be configured to retrieve a file from a file queue. The file may be stored in a local store of the first device. The file may be transferred from a second remote device via Remote Direct Memory Access. The one or more processors may further be configured to determine if the file is complete. The one or more processors may further be configured to remove the file from the file queue, if the file is determined to be complete.
1. A method for transferring data from remote network locations, comprising:
repetitively determining, by one or more hardware processors, a current time interval associated with batch processing files transferred from a remote network location to a destination location on a local file system, wherein repetitively determining comprises determining the current time interval based at least in part on past timing data for batch processing files transferred from the remote network location; and
repetitively, based upon the current time interval as determined at the time of each repetition:
selecting and retrieving, by the one or more hardware processors, a batch of files from the destination location on the local file system, the batch of files comprising files that have completed transfer from the remote network location and that have not been previously batch processed; and
batch processing, by the one or more hardware processors, the batch of files.
2. The method of claim 1 , wherein the batch of files is transferred from the remote network location to the destination location on the local file system via Remote Direct Memory Access.
3. The method of claim 2 , wherein the Remote Direct Memory Access is implemented based on at least one of Infiniband, fiber channel, or another Remote Direct Memory Access-enabled protocol.
4. The method of claim 1 , wherein the local file system is a parallel file system.
5. The method of claim 1 , further comprising:
repetitively, based upon the current time interval as determined at the time of each repetition:
checking, by the one or more hardware processors, whether each file in the batch of files is complete;
in response to determining that a first file in the batch of files is complete, marking, by the one or more hardware processors, the first file as being complete; and
in response to determining that a second file in the batch of files is not complete, passing on, by the one or more hardware processors, the second file to a next batch of files to be retrieved from the destination location at a next time interval.
6. The method of claim 5 , wherein checking whether each file in the batch of files is complete comprises, for each given file in the batch of files:
calculating a respective checksum for a payload for the given file; and
comparing the calculated respective checksum with a checksum embedded in a footer of the given file.
7. The method of claim 1 , wherein repetitively determining a current time interval is performed at a time interval longer than the current time interval, and the current time interval is fixed between repetitive determinations of the current time interval.
8. The method of claim 1 , wherein batch processing comprises, for a batch comprising a plurality of files, processing multiple files of the plurality of files in parallel.
9. The method of claim 1 , wherein repetitively determining the current time interval comprises calculating a variable time interval to a next batch processing time based at least in part on an amount of file processing to be completed in a current time interval.
10. The method of claim 9 , wherein calculating the variable time interval is further based on a minimum delay between consecutive batches of processing.
11. The method of claim 1 , further comprising, based at least on a value of the current time interval, limiting a size of the batch of files selected for batch processing during the current time interval to less than all files that have completed transfer from the remote network location and have not yet been processed.
12. The method of claim 1 , further comprising concurrently and repetitively batch processing files transferred from the remote network location to a different destination location on the file system based on a time interval different than the current time interval.
13. The method of claim 1 , wherein determining the current time interval based at least in part on past timing data comprises performing artificial intelligence analysis on past processing time intervals to estimate a processing time required for one or more future time intervals.
14. The method of claim 1 , wherein determining the current time interval is further based at least in part on monitoring system resources to determine system resources available to batch process files in a future time interval.
15. The method of claim 1 , wherein determining the current time interval is further based at least in part on predictive analysis to estimate the system resources required to process files for a time interval.
16. A system for synchronization of data transfers from remote network locations, comprising one or more processors configured to:
repetitively determine a current time interval associated with batch processing files transferred from a remote network location to a destination location on a local file system, wherein to repetitively determine comprises to determine the current time interval based at least in part on past timing data for batch processing files transferred from the remote network location; and
repetitively, based upon the current time interval as determined at the time of each repetition:
select and retrieve a batch of files from the destination location on the local file system, the batch of files comprising files that have completed transfer from the remote network location and that have not been previously batch processed; and
batch process the batch of files.
17. The system of claim 16 , wherein the batch of files is transferred from the remote network location to the destination location on the local file system via Remote Direct Memory Access.
18. The system of claim 17 , wherein the Remote Direct Memory Access is implemented based on at least one of Infiniband, fiber channel, or another Remote Direct Memory Access-enabled protocol.
19. The system of claim 16 , wherein the local file system is a parallel file system.
20. The system of claim 16 , wherein the one or more processors is further configured to:
repetitively, based upon the current time interval as determined at the time of each repetition:
check whether each file in the batch of files is complete;
in response to determining that a first file in the batch of files is complete, mark the first file as being complete; and
in response to determining that a second file in the batch of files is not complete, pass on the second file to a next batch of files to be retrieved from the destination location at a next time interval.
21. The system of claim 20 , wherein checking whether each file in the batch of files is complete comprises, for each given file in the batch of files:
calculating a respective checksum for a payload for the given file; and
comparing the calculated respective checksum with a checksum embedded in a footer of the given file.
22. The system of claim 16 , wherein repetitively determining a current time interval is performed at a time interval longer than the current time interval, and the current time interval is fixed between repetitive determinations of the current time interval.
23. A non-transitory computer readable medium storing a computer-readable program for synchronization of data transfers from remote network locations, the computer-readable program comprising:
computer-readable instructions to repetitively determine a time interval associated with batch processing files transferred from a remote network location to a destination location on a local file system, wherein to repetitively determine comprises to determine the current time interval based at least in part on past timing data for batch processing files transferred from the remote network location; and
computer-readable instructions to repetitively, based upon the current time interval as determined at the time of each repetition:
select and retrieve a batch of files from the destination location on the local file system, the batch of files comprising files that have completed transfer from the remote network location and that have not been previously batch processed; and
batch process the batch of files.