Enabling use of analytic functions for distributed storage system data
Various systems and methods for enabling use of analytic functions for distributed storage data are described. For example, one method involves generating an index for a data stream that includes a plurality of objects. The index indicates a location of each of the plurality of objects. The method also involves performing a plurality of first functions that generate a plurality of first outputs. Performing these first functions includes accessing objects based on the index and generating an output for each object. Each output also includes the respective object. The method also involves performing one or more second functions, where the second functions use the first outputs.
1. A method comprising:
generating, using one or more processors, an index, wherein
the index is generated for a data stream comprising a plurality of objects, and
the index indicates a location of each of the plurality of objects;
performing a plurality of first functions, wherein
the plurality of first functions are performed substantially concurrently,
each of the plurality of first functions accesses, based on the index, a respective object of the plurality of objects,
the performing the plurality of first functions generates a plurality of first outputs, and
the performing the each of the plurality of first functions comprises
accessing a respective object of the plurality of objects, wherein
the accessing is based on the index, and
generating a respective first output of the plurality of first outputs, wherein
the respective first output comprises the respective object; and
performing one or more second functions, wherein
the one or more second functions use the plurality of first outputs.
2. The method of claim 1 , further comprising:
storing the data stream using a distributed storage system, wherein
the storing the data stream comprises
generating boundary information for the plurality of objects,
the generating the index is based, at least in part, on the boundary information.
3. The method of claim 2 , wherein
the generating the index is performed substantially during the storing the data stream.
4. The method of claim 1 , further comprising:
receiving a plurality of metadata, wherein
the plurality of metadata indicates one or more characteristics of the data stream, and
the generating the index is based, at least in part, on the plurality of metadata.
5. The method of claim 1 , further comprising:
performing a backup of a plurality of original data, wherein
the performing the backup is based on a backup policy, and
generating a plurality of data, wherein
the generating the plurality of data is based, at least in part, on the original data.
6. The method of claim 5 , wherein
the plurality of original data is generated by one or more applications, and
the plurality of objects are associated with the one or more applications.
7. The method of claim 5 , wherein
the performing the plurality of first functions further comprises
accessing the plurality of objects in the data stream based on the locations indicated by the index, wherein
the data stream comprises the plurality of data, and wherein
each object of the plurality of objects comprises a respective subset of the plurality of data.
8. The method of claim 1 , wherein
the performing the one or more second functions further comprises
aggregating the plurality of objects into one or more aggregated outputs.
9. The method of claim 8 , further comprising:
generating a sequence file using the one or more aggregated outputs.
10. The method of claim 8 , further comprising:
performing one or more analytic functions on the plurality of objects in the data stream, wherein the performing uses the one or more aggregated outputs.
11. A system, comprising:
one or more processors;
a software agent, wherein
the software agent is configured to generate an index for a data stream,
the data stream comprises a plurality of objects,
the index is configured to indicate a respective location of each of the plurality of objects, and
the software agent is configured to be executed using the one or more processors; and
a plurality of nodes, wherein
the plurality of nodes are configured to
store the data stream,
perform a plurality of mapping functions, wherein
the plurality of mapping functions are configured to be performed substantially concurrently,
each of the plurality of mapping functions are configured to access, based on the index, a respective object of the plurality of objects,
the performing the each of the plurality of mapping functions is configured to
access a respective object of the plurality of objects, wherein
the accessing is based on the index, and
generate a respective first output of a plurality of first outputs, wherein
the respective first output comprises the respective object; and
perform one or more reduce functions, wherein
the one or more reduce functions are configured to use the plurality of first outputs.
12. The system of claim 11 , wherein
the software agent is configured to generate the index using the data stream substantially during the plurality of nodes performing the storage of the data stream.
13. The system of claim 11 , wherein
the software agent is configured to
receive a plurality of metadata, wherein
the plurality of metadata indicates one or more characteristics of the data stream, and
the software agent is configured to generate the index based, at least in part, on the plurality of metadata.
14. The system of claim 11 , further comprising:
backup software, wherein
the backup software is configured to perform a backup of a plurality of original data,
the backup software is configured to perform the backup based on a backup policy, and
generating a plurality of data, wherein
the generating the plurality of data is based, at least in part, on the original data.
15. The system of claim 14 , wherein
the plurality of original data is generated by one or more applications, and
the plurality of objects are associated with the one or more applications.
16. The system of claim 11 , wherein
the plurality of nodes are configured to perform the plurality of mapping functions, further comprising
accessing the plurality of objects in the data stream based on the respective locations indicated by the index, wherein
the data stream comprises a plurality of data, and
each object of the plurality of objects comprises a respective subset of the plurality of data.
17. A computer program product comprising:
a plurality of instructions, comprising
a first set of instructions, executable on a computer system, configured to generate an index, wherein
the index is generated for a data stream comprising a plurality of objects, and
the index indicates a location of each of the plurality of objects,
a second set of instructions, executable on the computer system, configured to perform a plurality of first functions, wherein
the plurality of first functions are performed substantially concurrently,
each of the plurality of first functions accesses, based on the index, a respective object of the plurality of objects,
the performing the plurality of first functions generates a plurality of first outputs, and
the performing the each of the plurality of first functions comprises
accessing a respective object of the plurality of objects, wherein the accessing is based on the index, and
generating a respective first output of the plurality of first outputs, wherein
the respective first output comprises the respective object,
a third set of instructions, executable on the computer system, configured to, perform one or more second functions, wherein
the one or more second functions use the plurality of first outputs; and
a non-transitory computer-readable storage medium, wherein the instructions are encoded in the computer-readable storage medium.
18. The computer program product of claim 17 , wherein
the first set of instructions is further configured to generate the index substantially during storing of the data stream.
19. The computer program product of claim 17 , wherein
the instructions further comprise
a fourth set of instructions, executable on the computer system, configured to receive a plurality of metadata, wherein
the plurality of metadata indicates one or more characteristics of the data stream, and wherein
the first set of instructions is further configured to generate the index based, at least in part, on the plurality of metadata.
20. The computer program product of claim 17 , wherein the instructions further comprise:
a fourth set of instructions, executable on the computer system, configured to perform a backup of a plurality of original data, wherein
the performance of the backup is based on a backup policy, and
a fifth set of instructions, executable on the computer system, configured to generate a plurality of data, wherein
the generation of the plurality of data is based, at least in part, on the original data.
21. The method of claim 1 , wherein
a second function of the one or more second functions is performed on at least a subset of the plurality of the first outputs.