Unhandled data protection for programmatic input/output routing to datasets with user-defined partitions
Systems and methods are described for implementing programmatic input/output (I/O) routing to datasets with user-defined partitions while providing unhandled data protection. As disclosed herein, a user may define a dataset as including one or more partitions, each partition including criteria for storing data objects written to the partitioned dataset in the individual partitions. Data objects written to the dataset can then be evaluated according to the criteria, and routed to an appropriate partition. To provide unhandled data protection, a dataset definition can include a default partition to which data objects are routed when the data object fails to satisfy the criteria of any of the set of user-defined partitions identified in the specification. Processing I/O operations according to a user-defined partitioning schema can enable data objects to be arranged according to any partitioning schema without tethering the partitioning to a particular underlying storage system.
1 . A computer-implemented method comprising:
obtaining, by a data intake and query system, specification of a partitioned dataset in a data store,
the specification identifying a set of user-defined partitions included in the partitioned dataset, wherein the partitioned dataset is represented as a tree data structure, wherein each partition in the set of user-defined partitions corresponds to a child node, of a root node of the tree data structure, that is associated with user-defined criteria for data objects to be written to a corresponding partition, and wherein each child node corresponds to one or more leaf nodes identifying a data storage destination;
obtaining, by the data intake and query system, a request to write one or more data objects to the partitioned dataset, the request comprising an identifier of the partitioned dataset; and
writing, by the data intake and query system, the one or more data objects to the partitioned dataset, wherein writing the one or more data objects to the partitioned dataset comprises:
analyzing a plurality of user-defined criteria associated with a plurality of child nodes of the root node of the tree data structure to determine that a data object of the one or more data objects satisfies a particular user-defined criteria, of a particular child node, for the data object to be written to the corresponding partition, the particular user-defined criteria specifying a value of a field associated with the data object;
based on the data object satisfying the particular user-defined criteria of the particular child node, identifying the one or more leaf nodes corresponding to the particular child node, and
for each leaf node of the identified one or more leaf nodes, generating and executing a request to write the data object to the data storage destination corresponding to the leaf node.
2 . The computer-implemented method of claim 1 , further comprising:
obtaining, by the data intake and query system, a request to read the one or more data objects from the partitioned dataset, the request comprising an identifier of the partitioned dataset;
reading, by the data intake and query system, the one or more data objects from the partitioned dataset, wherein reading the one or more data objects from the partitioned dataset comprises generating a query to read from a union of data storage destination corresponding to each leaf node of the tree data structure; and
providing results of the query as a response to the read request.
3 . The computer-implemented method of claim 1 , wherein obtaining the request to write the one or more data objects to the partitioned dataset includes obtaining a stream of data objects to be written to the partitioned dataset.
4 . The computer-implemented method of claim 1 , wherein obtaining the request to write the one or more data objects to the partitioned dataset includes obtaining a query to the data intake and query system instructing to write the one or more data objects specified in the query to the partitioned dataset.
5 . The computer-implemented method of claim 1 , wherein the partitioned dataset further includes a default partition, wherein the default partition is not included in the set of user-defined partitions, wherein the default partition corresponds to a first child node, and wherein writing the one or more data objects to the partitioned dataset further comprises:
when a second data object of the one or more data objects fails to satisfy the user-defined criteria for any child node corresponding to a partition in the set of user-defined partitions:
identifying a set of leaf nodes corresponding to the default partition; and
for each leaf node of the identified set of leaf nodes corresponding to the default partition, generating and executing a request to write the data object to the data storage destination corresponding to the leaf node.
6 . The computer-implemented method of claim 1 , wherein obtaining the specification of the partitioned dataset comprises:
obtaining specification of a processing pipeline including criteria for the data objects to be processed by the processing pipeline; and
generating a first partition of the set of user-defined partitions, with criteria corresponding to the criteria of the processing pipeline.
7 . The computer-implemented method of claim 6 further comprising deploying the processing pipeline to a stream data processing system, wherein deploying the processing pipeline comprises:
configuring the stream data processing system to write an input data stream including the one or more data objects to the partitioned dataset;
modifying the processing pipeline to read from the first partition of the set of user-defined partitions, wherein modifying the processing pipeline results in a modified processing pipeline; and
configuring the stream data processing system to implement the modified processing pipeline.
8 . The computer-implemented method of claim 1 , wherein at least one child node is a leaf node.
9 . The computer-implemented method of claim 1 , wherein at least one child node is a parent of the one or more leaf nodes.
10 . The computer-implemented method of claim 1 , wherein the specification is stored as a first data object in an object notation format.
11 . The computer-implemented method of claim 1 , wherein the data storage destination is one of: a substrate storage system or another partitioned dataset that itself identifies one or more substrate storage systems.
12 . The computer-implemented method of claim 11 , wherein the substrate storage system is a file system or an object storage system.
13 . The computer-implemented method of claim 1 , wherein the user-defined criteria of each child node is either exclusive among child nodes at a level of the tree data structure or non-exclusive among child nodes at the level of the tree data structure.
14 . A system, comprising:
a data store storing computer-executable instructions; and
a processor configured to execute the computer-executable instructions, wherein execution of the computer-executable instructions causes the system to:
obtain, by a data intake and query system, specification of a partitioned dataset in a data store,
the specification identifying a set of user-defined partitions included in the partitioned dataset, wherein the partitioned dataset is represented as a tree data structure, wherein each partition in the set of user-defined partitions corresponds to a child node, of a root node of the tree data structure, that is associated with user-defined criteria for data objects to be written to a corresponding partition, and wherein each child node corresponds to one or more leaf nodes identifying a data storage destination;
obtain, by the data intake and query system, a request to write one or more data objects to the partitioned dataset, the request comprising an identifier of the partitioned dataset; and
write, by the data intake and query system, the one or more data objects to the partitioned dataset, wherein to write the one or more data objects to the partitioned dataset, the execution of the computer-executable instructions causes the system to:
analyze a plurality of user-defined criteria associated with a plurality of child nodes of the root node of the tree data structure to determine that a data object of the one or more data objects satisfies a particular user-defined criteria, of a particular child node, for the data object to be written to the corresponding partition, the particular user-defined criteria specifying a value of a field associated with the data object based on the data object satisfying the particular user-defined criteria of the particular child node, identify the one or more leaf nodes corresponding to the particular child node, and
for each leaf node of the identified one or more leaf nodes, generate and execute a request to write the data object to the data storage destination corresponding to the leaf node.
15 . The system of claim 14 , wherein the execution of the computer-executable instructions further causes the system to:
obtain, by the data intake and query system, a request to read the one or more data objects from the partitioned dataset, the request comprising an identifier of the partitioned dataset;
read, by the data intake and query system, the one or more data objects from the partitioned dataset, wherein to read the one or more data objects from the partitioned dataset, execution of the computer-executable instructions further causes the system to generate a query to read from a union of data storage destination corresponding to each leaf node of the tree data structure; and
provide results of the query as a response to the read request.
16 . The system of claim 14 , wherein to obtain the request to write the one or more data objects to the partitioned dataset, the execution of the computer-executable instructions further causes the system to obtain a stream of data objects to be written to the partitioned dataset.
17 . The system of claim 14 , wherein to obtain the request to write the one or more data objects to the partitioned dataset, the execution of the computer-executable instructions further causes the system to obtain a query to the data intake and query system instructing to write one or more data objects specified in the query to the partitioned dataset.
18 . The system of claim 14 , wherein the partitioned dataset further includes a default partition, wherein the default partition is not included in the set of user-defined partitions, wherein the default partition corresponds to a first child node, and wherein to write the one or more data objects to the partitioned dataset, the execution of the computer-executable instructions further causes the system to:
when a second data object of the one or more data objects fails to satisfy the user-defined criteria for any child node corresponding to a partition in the set of user-defined partitions:
identify a set of leaf nodes corresponding to the default partition; and
for each leaf node of the identified set of leaf nodes corresponding to the default partition, generate and execute a request to write the data object to the data storage destination corresponding to the leaf node.
19 . One or more non-transitory computer-readable media comprising computer-executable instructions, that, when executed, cause a system to:
obtain, by a data intake and query system, specification of a partitioned dataset in a data store,
the specification identifying a set of user-defined partitions included in the partitioned dataset, wherein the partitioned dataset is represented as a tree data structure, wherein each partition in the set of user-defined partitions corresponds to a child node, of a root node of the tree data structure, that is associated with user-defined criteria for data objects to be written to a corresponding partition, and wherein each child node corresponds to one or more leaf nodes identifying a data storage destination;
obtain, by the data intake and query system, a request to write one or more data objects to the partitioned dataset, the request comprising an identifier of the partitioned dataset; and
write, by the data intake and query system, the one or more data objects to the partitioned dataset, wherein to write the one or more data objects to the partitioned dataset, execution of the computer-executable instructions causes the system to:
analyze a plurality of user-defined criteria associated with a plurality of child nodes of the root node of the tree data structure to determine that a data object of the one or more data objects satisfies a particular user-defined criteria, of a particular child node, for the data object to be written to the corresponding partition, the particular user-defined criteria specifying a value of a field associated with the data object;
based on the data object satisfying the particular user-defined criteria of the particular child node, identify the one or more leaf nodes corresponding to the particular child node, and
for each leaf node of the identified one or more leaf nodes, generate and execute a request to write the data object to the data storage destination corresponding to the leaf node.
20 . The one or more non-transitory computer-readable media of claim 19 , wherein execution of the computer-executable instructions further causes the system to:
obtain, by the data intake and query system, a request to read the one or more data objects from the partitioned dataset, the request comprising an identifier of the partitioned dataset;
read, by the data intake and query system, the one or more data objects from the partitioned dataset, wherein to read the one or more data objects from the partitioned dataset, execution of the computer-executable instructions further causes the system to generate a query to read from a union of data storage destination corresponding to each leaf node of the tree data structure; and
provide results of the query as a response to the read request.