Methods and systems for stream-processing of biomedical data
A method for stream-processing biomedical data includes receiving, by a file system on a computing device, a first request for access to at least a first portion of a file stored on a remotely located storage device. The method includes receiving, by the file system, a second request for access to at least a second portion of the file. The method includes determining, by a pre-fetching component executing on the computing device, whether the first request and the second request are associated with a sequential read operation. The method includes automatically retrieving, by the pre-fetching component, a third portion of the requested file, before receiving a third request for access to least the third portion of the file, based on a determination that the first request and the second request are associated with the sequential read operation.
1. A method for stream-processing of data, the method comprising:
receiving, by a file system on a computing device, from an application executing on the computing device, a first request for access to at least a first portion of a file stored on a remotely located storage device;
receiving, by the file system, a second request for access to at least a second portion of the file;
determining, by a pre-fetching component of the file system and executing on the computing device, whether the first request and the second request are associated with a sequential read operation; and
retrieving, by the pre-fetching component, metadata associated with the file and stored in a metadata database, the metadata including at least one key-value pair associated with the file that identifies one or more locations storing one or more portions of data for the file;
automatically retrieving, by the pre-fetching component, via a network connection to the remotely located storage device, using the metadata, a third portion of the requested file, after receiving the first request and after receiving the second request and before receiving a third request for the third portion of the file, based on a determination, by the pre-fetching component of the file system, that the first request and the second request are associated with the sequential read operation.
2. The method of claim 1 further comprising determining a location of the first portion of the requested file on the remotely located storage device.
3. The method of claim 1 further comprising determining, by a cache management component, that the first portion of the file is not stored on the computing device.
4. The method of claim 3 further comprising retrieving, by the pre-fetching component, from the remotely located storage device, the first portion of the requested file.
5. The method of claim 1 , wherein determining further comprises:
analyzing, by the pre-fetching component, the first request and the second request; and
confirming, by the pre-fetching component, based on the analysis, that the first request and the second request are associated with the sequential read operation.
6. The method of claim 5 , wherein determining that the second request is for a sequential read operation further comprises determining that the second request is for a portion of the file that sequentially follows the first portion.
7. The method of claim 1 further comprising storing, by the pre-fetching component a retrieved portion of the requested file in a cache local to the computing device.
8. The method of claim 7 , further comprising removing a retrieved portion of the requested file from the cache.
9. The method of claim 8 , wherein retrieved portions of the requested file in the cache are removed according to a Last In, First Out (LIFO) policy.
10. The method of claim 1 , further comprising automatically retrieving a fourth portion of the requested file before receiving a fourth request access to at least a fourth portion of a file.
11. A system for stream-processing of data, the system comprising:
a file system on a computing device receiving, from an application executing on the computing device, a first request for access to at least a portion of a file stored on a remotely located storage device and receiving a second request for access to at least a second portion of the file; and
a pre-fetching component of the file system: (i) executing on the computing device, (ii) determining whether the first request and the second request are associated with a sequential read operation, (iii) retrieving metadata associated with the file and stored in a metadata database, the metadata including at least one key-value pair associated with the file that identifies one or more locations storing one or more portions of data for the file, and (iv) automatically retrieving, via a network connection to the remotely located storage device, using the metadata, a third portion of the requested file, after receiving the first request and after receiving the second request and before receiving a third request for the third portion of the file, based on a determination, by the pre-fetching component of the file system that the first request and the second request are associated with the sequential read operation.
12. The system of claim 11 , wherein the file is a file containing biomedical data.
13. The system of claim 11 , wherein the file system is provided by a kernel module executing in a user space provided by an operating system executing on the computing device.
14. The system of claim 11 further comprising a cache local to the computing device and storing at least one pre-fetched portion of a file.
15. The system of claim 11 further comprising a data structure including an identifier of a portion of the file.
16. The system of claim 11 further comprising a data structure including an identification of a storage location of a portion of the file.
17. A non-transitory, computer-readable medium comprising computer program instructions tangibly stored on the non-transitory computer-readable medium, wherein the instructions are executable by at least one processor to perform a method for stream-processing of data the method comprising:
receiving, by a file system on a computing device, from an application executing on the computing device, a first request for access to at least a first portion of a file stored on a remotely located storage device;
receiving, by a file system on a computing device, a second request for access to at least a second portion of a file stored on the remotely located storage device;
determining, by a pre-fetching component of the file system and executing on the computing device, whether the first request and the second request are associated with a sequential read operation;
retrieving, by the pre-fetching component, metadata associated with the file and stored in a metadata database, the metadata including at least one key-value pair associated with the file that identifies one or more locations storing one or more portions of data for the file; and
automatically retrieving, by the pre-fetching component, using the metadata, via a network connection to the remotely located storage device, a third portion of the requested file, after receiving the first request and after receiving the second request and before receiving a third request for the third portion of the file, based on a determination, by the pre-fetching component of the file system, that the first request and the second request are associated with the sequential read operation.
18. The non-transitory, computer-readable medium of claim 17 , wherein determining further comprises:
analyzing, by the pre-fetching component, the first request and the second request; and
confirming, by the pre-fetching component, based on the analysis, that the first request and the second request are associated with the sequential read operation.