Systems and methods for file triggers in a converged compute and file system
A hot folder mechanism is employed to provide a truly integrated architecture for easy-to-use, easy-to-deploy scale-out computation and scale-out storage. A folder of an Ingest, Transform, Store (ITS)-Network attached storage (NAS) system can be configured as “hot.” The configured hot folder can then detect changes on its content, analyze such content, perform transform services on the content, and output the computation results as files on other specified output folders. In one aspect, file system nodes of the ITS-NAS can present the content to high performance computing (HPC) compute nodes of the ITS-NAS as well as to dispatch the execution of transform services to the HPC compute nodes.
1. A method, comprising:
receiving, by a system comprising a processor, configuration data that configures a first hot folder, wherein the first hot folder is a folder within a network attached storage node device that comprises first file data that is to be processed in response to receiving a first request to access the folder, wherein the configuration data comprises information indicative of an output folder within the network attached storage node device that is to be employed to store a result of processing of the first file data, and wherein the output folder is a second hot folder that comprises second file data that is to be processed in response to receiving a second request to access the second hot folder; and
in response to determining that request data indicative of the first request has been received, instructing, by the system, a high-performance computing node device to process the first file data, wherein the high-performance computing node device and the network attached storage node device operate under a common operating system.
2. The method of claim 1 , further comprising:
monitoring, by the system, input data received from a client device, and wherein the input data comprises the request data.
3. The method of claim 1 , wherein the receiving the configuration data comprises receiving criterion information indicative of a criterion to processes the first file data.
4. The method of claim 3 , wherein the receiving the configuration data comprises receiving service information indicative of a transformation service that is employable to process the first file data in response to determining that the criterion has been satisfied.
5. The method of claim 3 , wherein the receiving the criterion information comprises receiving the criterion information indicative of a file criterion associated with at least one of a size of the first file data, a type of the first file data, metadata associated with the first file data, or an attribute associated with the first file data.
6. The method of claim 3 , wherein the receiving the criterion information comprises receiving the criterion information indicative of a resource criterion associated with a computing resource of the high-performance computing node device that is to be utilized to process the first file data.
7. The method of claim 3 , wherein the receiving the criterion information comprises receiving the criterion information indicative of an availability criterion associated with an availability of resources of at least one of the high-performance computing node device or the network attached storage node device.
8. The method of claim 1 , wherein the receiving the configuration data comprises receiving information indicative of a splitter service that is employable to split the first file data into a set of logical addresses that are to be utilized by a set of high-performance computing node device to process the first file data in a distributed manner.
9. The method of claim 1 , wherein the determining comprises intercepting the request data prior to a transmission of the request data to a file system driver of the network attached storage node device.
10. A system, comprising:
a processor; and
a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations, comprising:
determining that a first request to access a first hot folder has been received, wherein the first hot folder is a folder comprising first content stored within a network attached storage node device that has been configured to be processed in response to receiving the first request; and
facilitating a transformation of the first content based on an execution of a specified service via a high-performance computing node device, wherein the high-performance computing node device and the network attached storage node device operate under a common operating system, wherein the transformation of the first content results in second content that is to be stored within a second hot folder within the network attached storage node device, and wherein the second content is configured to be processed in response to receiving a second request to access the second hot folder.
11. The system of claim 10 , wherein the operations further comprise:
monitoring input data received from a client device, and wherein the input data comprises request data indicative of the first request.
12. The system of claim 10 , wherein the facilitating comprises facilitating the transformation in response to determining that a defined criterion associated with the first hot folder has been satisfied.
13. The system of claim 12 , wherein the defined criterion is associated with at least one of a size of the first content, a type of the first content, metadata associated with the first content, or an attribute of the first content.
14. The system of claim 12 , wherein the defined criterion is associated with a computing resource of the high-performance computing node device that is to be utilized for the execution of the specified service.
15. The system of claim 10 , wherein the facilitating the transformation comprises facilitating a transmission of instruction data to the high-performance computing node device via a private communication network link, and wherein the instruction data triggers the execution.
16. A computer-readable storage medium comprising instructions that, in response to execution, cause a device comprising a processor to perform operations, comprising:
receiving request data representing a first request to access a first hot folder, wherein the first hot folder is a folder of a network attached storage node device that comprises a first file that is to be transformed in response to being accessed; and
based on configuration data associated with the first hot folder, facilitating a transformation of the file via a high-performance computing node device, wherein a common operating system manages the high-performance computing node device and the network attached storage node device, wherein the configuration data specifies an output folder within the network attached storage node device that is to be employed to store a result of transformation of the first file, and wherein the output folder is defined as a second hot folder that comprises a second file that is to be transformed in response to being accessed.
17. The computer-readable storage medium of claim 16 , wherein the configuration data comprises service data that specifies a transformation service that is to be executed by the high-performance computing node device to facilitate the transformation.
18. The computer-readable storage medium of claim 16 , wherein the configuration data comprises criterion data that represents a criterion associated with the first file that is to be determined to be satisfied to permit the facilitating.
19. The computer-readable storage medium of claim 18 , wherein the criterion is associated with at least one of a size of the first file, a type of the first file, metadata associated with the first file, or an attribute of the first file.
20. The computer-readable storage medium of claim 18 , wherein the criterion is associated with a computing resource of the high-performance computing node device that is to be utilized for facilitating the transformation.