Distributed computing framework and distributed computing method
A distributed computing framework and a distributed computing method are provided. A specific embodiment of the distributed computing framework includes: a parsing unit, configured to parse an expression of a distributed computing task, and determine an operator and a field corresponding to the operator; and an operator unit, configured to provide the operator, input parameters of the operator including: the field and a field-type distributed dataset. The type of parameters received and returned by any operator may be the field-type distributed dataset, and any operator may operate on the data corresponding to the field in the field-type distributed dataset. Therefore, any operator needs to be implemented once to realize the reuse of the operator. The distributed computing task is expressed in a simple expression, which simplifies the complexity of writing a distributed computing program with the distributed computing framework used by the user.
1. A distributed computing system, the distributed computing framework comprising:
at least one processor; and
a memory storing instructions, the instructions when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising:
parsing an expression of a distributed computing task, and determining an operator and a field corresponding to the operator, the field indicating an attribute of data in a distributed dataset the operator acting on;
generating input parameters of the operator, the input parameters of the operator comprising: the field and a field-type distributed dataset, the field-type distributed dataset containing at least one element, the element containing a plurality of key-value pairs, a key of each of the plurality of key-value pairs being the field, and a value of each of the plurality of key-value pairs being the data corresponding to the field in the distributed dataset;
converting the distributed dataset corresponding to the data required by the distributed computing task into the field-type distributed dataset based on the field, a given element in the field-type distributed dataset containing a plurality of key-value pairs, the key of each of the plurality of key-value pairs being the field, and the value of each of the plurality of key-value pairs being the data corresponding to the field in the distributed dataset, wherein the plurality of key-value pairs comprised in the given element comprises first key-value pairs, and a second key-value pair, wherein first keys of the first key-value pairs are identical, first values of the first key-value pairs are different, and a second key of the second key-value pair is different from the first key;
forming new elements by combining the first key-value pairs with the second key-value pair respectively, and using the new elements as elements of a field-type distributed dataset, wherein each of the new elements consists of the second key-value pair and one respective first key-value pair; and
performing distributed computing by the operator based on the input parameters.
2. The distributed computing framework according to claim 1 , wherein the operations further comprise:
grouping the elements in the field-type distributed dataset based on the value corresponding to the key indicated by a grouping field in the elements in the field-type distributed dataset, to obtain a plurality of grouped field-type distributed datasets, wherein each of the values corresponding to the key indicated by the grouping field corresponds to each of the plurality of grouped field-type distributed datasets, and the each of the plurality of grouped field-type distributed dataset comprises at least one element containing the value corresponding to the key indicated by the grouping field.
3. The distributed computing system according to claim 2 , wherein the operations further comprise:
performing, based on a traversal field, operations on the value corresponding to the key indicated by the traversal field in the elements in the each of the plurality of grouped field-type distributed datasets.
4. The distributed computing system according to claim 3 , wherein the operations further comprise:
aggregating results obtained by performing operations on the value corresponding to the key indicated by the traversal field in the elements in the each of the plurality of grouped field-type distributed datasets.
5. A distributed computing method, the method comprising:
parsing an expression of a distributed computing task, and determining an operator and a field corresponding to the operator, the field indicating an attribute of data in a distributed dataset the operator acting on;
generating input parameters of the operator, the input parameters of the operator comprising: the field and a field-type distributed dataset, the field-type distributed dataset containing at least one element, the element containing a plurality of key-value pairs, a key of each of the plurality of key-value pairs being the field, and a value of each of the plurality of key-value pairs being the data corresponding to the field in the distributed dataset; converting the distributed dataset corresponding to the data required by the distributed computing task into the field-type distributed dataset based on the field, a given element in the field-type distributed dataset containing a plurality of key-value pairs, the key of each of the plurality of key-value pairs being the field, and the value of each of the plurality of key-value pairs being the data corresponding to the field in the distributed dataset, wherein the plurality of key-value pairs comprised in the given element comprises first key-value pairs, and a second key-value pair, wherein first keys of the first key-value pairs are identical, first values of the first key-value pairs are different, and a second key of the second key-value pair is different from the first key;
forming new elements by combining the first key-value pairs with the second key-value pair respectively, and using the new elements as elements of a field-type distributed dataset, wherein each of the new elements consists of the second key-value pair and one respective first key-value pair; and
performing distributed computing by the operator based on the input parameters,
wherein the method is performed by at least one hardware processor.
6. The method according to claim 5 , wherein the method further comprises:
grouping the elements in the field-type distributed dataset based on the value corresponding to the key indicated by a grouping field in the elements in the field-type distributed dataset, to obtain a plurality of grouped field-type distributed datasets, wherein each of the values corresponding to the key indicated by the grouping field corresponds to each of the plurality of grouped field-type distributed datasets, and the each of the plurality of grouped field-type distributed dataset comprises at least one element containing the value corresponding to the key indicated by the grouping field.
7. The method according to claim 6 , wherein the method further comprises:
performing, based on a traversal field, operations on the value corresponding to the key indicated by the traversal field in the elements in the each of the plurality of grouped field-type distributed datasets.
8. The method according to claim 7 , wherein the method further comprises:
aggregating results obtained by performing operations on the value corresponding to the key indicated by the traversal field in the elements in the each of the plurality of grouped field-type distributed datasets.
9. A non-transitory computer-readable storage medium storing a computer program, the computer program when executed by one or more processors, causes the one or more processors to perform operations, the operations comprising:
parsing an expression of a distributed computing task, and determining an operator and a field corresponding to the operator, the field indicating an attribute of data in a distributed dataset the operator acting on;
generating input parameters of the operator, the input parameters of the operator comprising: the field and a field-type distributed dataset, the field-type distributed dataset containing at least one element, the element containing a plurality of key-value pairs, a key of each of the plurality of key-value pairs being the field, and a value of each of the plurality of key-value pairs being the data corresponding to the field in the distributed dataset;
converting the distributed dataset corresponding to the data required by the distributed computing task into the field-type distributed dataset based on the field, a given element in the field-type distributed dataset containing a plurality of key-value pairs, the key of each of the plurality of key-value pairs being the field, and the value of each of the plurality of key-value pairs being the data corresponding to the field in the distributed dataset, wherein the plurality of key-value pairs comprised in the given element comprises first key-value pairs, and a second key-value pair, wherein first keys of the first key-value pairs are identical, first values of the first key-value pairs are different, and a second key of the second key-value pair is different from the first key;
forming new elements by combining the first key-value pairs with the second key-value pair respectively, and using the new elements as elements of a field-type distributed dataset, wherein each of the new elements consists of the second key-value pair and one respective first key-value pair; and
performing distributed computing by the operator based on the input parameters.