Systems and methods for facilitating analytics on data sets stored in remote monolithic files
The disclosed computer-implemented method for facilitating analytics on data sets stored in remote monolithic files may include (1) identifying, within a secondary storage system, a secondary copy of a data set duplicated from a primary copy of the data set stored in a primary storage system, (2) generating a set of virtual objects that represent at least a portion of the secondary copy of the data set, (3) exposing the set of virtual objects to a remote analytics engine via a network such that the set of individual data objects appears to be stored locally on the remote analytics engine, and then (4) enabling the remote analytics engine to perform at least one analytics job on the set of individual data objects by way of the set of virtual objects via the network. Various other methods, systems, and computer-readable media are also disclosed.
1. A computer-implemented method for facilitating analytics on data sets stored in remote monolithic files, at least a portion of the method being performed by a computing device comprising at least one processor, the method comprising:
identifying, within a secondary storage system, a monolithic file that:
includes a secondary copy of a data set duplicated from a primary copy of the data set stored in a primary storage system;
stores the secondary copy of the data set as a singular data block;
identifying a set of individual data objects within the secondary copy of the data set included in the monolithic file;
generating a set of virtual objects that represent the set of individual data objects identified within the secondary copy of the data set included in the monolithic file;
exposing the set of virtual objects to a remote analytics engine via a network such that the individual data objects appear to the remote analytics engine to be stored locally on the remote analytics engine, wherein the remote analytics engine comprises a computer cluster that:
includes a plurality of nodes;
implements a distributed file system that manages data across the plurality of nodes;
is unable to natively open or read monolithic files that store data sets as singular data blocks;
enabling the remote analytics engine to perform at least one analytics job on the set of individual data objects identified within the secondary copy of the data set by way of the set of virtual objects exposed to the remote analytics engine via the network.
2. The method of claim 1 , wherein generating the set of virtual objects that represent the set of individual data objects comprises:
providing a user with a user interface that enables the user to select the set of individual data objects;
detecting the user's selection of the set of individual data objects via the user interface;
generating the set of virtual objects that represent the set of individual data objects in response to the user's selection.
3. The method of claim 1 , wherein generating the set of virtual objects that represent the set of individual data objects comprises:
extracting information about the set of individual data objects from the monolithic file identified within the secondary storage system;
generating, based at least in part on the information extracted from the monolithic file, the set of virtual objects that represent the set of individual data objects.
4. The method of claim 1 , further comprising:
identifying another monolithic file that includes a secondary copy of another data set duplicated from a primary copy of the other data set stored in the primary storage system;
identifying another set of individual data objects within the secondary copy of the other data set included in the other monolithic file;
generating another set of virtual objects that represent the other set of individual data objects identified within the secondary copy of the other data set included in the other monolithic file;
wherein:
exposing the set of virtual objects to the remote analytics engine via the network comprises exposing the set of virtual objects and the other set of virtual objects as a virtual file system to the remote analytics engine via the network;
enabling the remote analytics engine to perform at least one analytics job on the set of individual data objects comprises enabling the remote analytics engine to perform the analytics job on the set of individual data objects and the other set of individual data objects by way of the virtual file system exposed to the remote analytics engine via the network.
5. The method of claim 1 , wherein:
exposing the set of virtual objects to the remote analytics engine via the network comprises exposing the set of virtual objects as a virtual file system to the remote analytics engine via the network;
enabling the remote analytics engine to perform at least one analytics job on the set of individual data objects comprises enabling the remote analytics engine to perform the analytics job on the set of individual data objects by way of the virtual file system exposed to the remote analytics engine via the network.
6. The method of claim 1 , wherein:
exposing the set of virtual objects to the remote analytics engine via the network comprises exposing the set of virtual objects to the remote analytics engine without moving or copying the set of individual data objects to the remote analytics engine via the network;
enabling the remote analytics engine to perform at least one analytics job on the set of individual data objects comprises enabling the remote analytics engine to perform the analytics job on the set of individual data objects without moving or copying the set of individual data objects to the remote analytics engine.
7. The method of claim 1 , wherein:
exposing the set of virtual objects to the remote analytics engine via the network comprises exposing the set of virtual objects to the remote analytics engine without moving or copying an equivalent set of individual data objects included in the primary copy of the data set to the remote analytics engine via the network;
enabling the remote analytics engine to perform at least one analytics job on the set of individual data objects comprises enabling the remote analytics engine to perform the analytics job on the set of individual data objects without moving or copying the equivalent set of individual data objects included in the primary copy of the data set to the remote analytics engine via the network.
8. The method of claim 1 , wherein:
exposing the set of virtual objects to the remote analytics engine via the network comprises exposing the set of virtual objects to the remote analytics engine via the network by way of a file system plug-in that interfaces with a file system of the remote analytics engine;
enabling the remote analytics engine to perform at least one analytics job on the set of individual data objects comprises enabling the remote analytics engine to perform the analytics job on the set of individual data objects via the network by way of the file system plug-in that interfaces with the file system of the remote analytics engine.
9. The method of claim 8 , further comprising:
receiving, by the file system plug-in, at least one request to perform an Input/Output (I/O) operation on the set of individual data objects;
generating, based at least in part on the request to perform the I/O operation, a notification of an anticipated future I/O operation likely to be performed on the set of individual data objects in connection with the analytics job;
forwarding the notification of the anticipated future I/O operation from the file system plug-in to the secondary storage system to facilitate prefetching at least a portion of the set of individual data objects in connection with the analytics job.
10. The method of claim 8 , further comprising:
receiving, by the file system plug-in, at least one request to perform a write operation on at least a portion of the set of individual data objects;
generating, by the file system plug-in, a data representation of the write operation based at least in part on the request to perform the write operation;
caching, by the file system plug-in, the data representation of the write operation in memory accessible to the file system plug-in;
providing, after completion of the analytics job, the data representation of the write operation to the secondary storage system to facilitate updating the portion of the set of individual data objects.
11. The method of claim 1 , further comprising:
identifying another secondary copy of the data set within another secondary storage system;
optimizing at least a portion of the analytics job by providing substantially parallel access to the secondary copy of the data set and the other secondary copy of the data set in connection with the analytics job.
12. A system for facilitating analytics on remotely stored data sets, the system comprising:
an identification module, stored in memory, that:
identifies, within a secondary storage system, a monolithic file that:
includes a secondary copy of a data set duplicated from a primary copy of the data set stored in a primary storage system;
stores the secondary copy of the data set as a singular data block;
identifies a set of individual data objects within the secondary copy of the data set included in the monolithic file;
a generation module, stored in memory, that generates a set of virtual objects that represent the set of individual data objects identified within the secondary copy of the data set included in the monolithic file;
an exposure module, stored in memory, that:
exposes the set of virtual objects to a remote analytics engine via a network such that the individual data objects appear to the remote analytics engine to be stored locally on the remote analytics engine, wherein the remote analytics engine comprises a computer cluster that:
includes a plurality of nodes;
implements a distributed file system that manages data across the plurality of nodes;
is unable to natively open or read monolithic files that store data sets as singular data blocks;
enables the remote analytics engine to perform at least one analytics job on the set of individual data objects identified within the secondary copy of the data set by way of the set of virtual objects exposed to the remote analytics engine via the network;
at least one physical processor that executes the identification module, the generation module, and the exposure module.
13. The system of claim 12 , further comprising a providing module, stored in memory, that provides a user with a user interface that enables the user to select the set of individual data objects;
wherein:
the identification module detects the user's selection of the set of individual data objects via the user interface;
the generation module generates the set of virtual objects that represent the set of individual data objects in response to the user's selection.
14. The system of claim 12 , wherein the exposure module:
exposes the set of virtual objects as a virtual file system to the remote analytics engine via the network;
enables the remote analytics engine to perform the analytics job on the set of individual data objects by way of the virtual file system exposed to the remote analytics engine via the network.
15. The system of claim 12 , wherein the exposure module:
exposes the set of virtual objects to the remote analytics engine without moving or copying the set of individual data objects to the remote analytics engine via the network;
enables the remote analytics engine to perform the analytics job on the set of individual data objects without moving or copying the set of individual data objects to the remote analytics engine.
16. The system of claim 12 , wherein the exposure module:
exposes the set of virtual objects to the remote analytics engine without moving or copying an equivalent set of individual data objects included in the primary copy of the data set to the remote analytics engine via the network;
enables the remote analytics engine to perform the analytics job on the set of individual data objects without moving or copying the equivalent set of individual data objects included in the primary copy of the data set to the remote analytics engine via the network.
17. The system of claim 12 , wherein the exposure module:
exposes the set of virtual objects to the remote analytics engine via the network by way of a file system plug-in that interfaces with a file system of the remote analytics engine;
enables the remote analytics engine to perform the analytics job on the set of individual data objects via the network by way of the file system plug-in that interfaces with the file system of the remote analytics engine.
18. The system of claim 17 , further comprising a receiving module, stored in memory, that receives at least one request to perform an Input/Output (I/O) operation on the set of individual data objects;
wherein the generation module generates, based at least in part on the request to perform the I/O operation, a notification of an anticipated future I/O operation likely to be performed on the set of individual data objects in connection with the analytics job;
further comprising a forwarding module, stored in memory, that forwards the notification of the anticipated future I/O operation from the file system plug-in to the secondary storage system to facilitate prefetching at least a portion of the set of individual data objects in connection with the analytics job.
19. A non-transitory computer-readable medium comprising one or more computer-executable instructions that, when executed by at least one processor of a computing device, cause the computing device to:
identify, within a secondary storage system, a monolithic file that:
includes a secondary copy of a data set duplicated from a primary copy of the data set stored in a primary storage system;
stores the secondary copy of the data set as a singular data block;
identify a set of individual data objects within the secondary copy of the data set included in the monolithic file;
generate a set of virtual objects that represent the set of individual data objects identified within the secondary copy of the data set included in the monolithic file;
expose the set of virtual objects to a remote analytics engine via a network such that the individual data objects appear to the remote analytics engine to be stored locally on the remote analytics engine, wherein the remote analytics engine comprises a computer cluster that:
includes a plurality of nodes;
implements a distributed file system that manages data across the plurality of nodes;
is unable to natively open or read monolithic files that store data sets as singular data blocks;
enable the remote analytics engine to perform at least one analytics job on the set of individual data objects identified within the secondary copy of the data set by way of the set of virtual objects exposed to the remote analytics engine via the network.