Mapping data set(s) to canonical phrases using natural language processing model(s)
Techniques are disclosed that enable generating a unified data set by mapping a set of item description phrases, describing entries in a data set, to a set of canonical phrases. Various implementations include generating a similarity measure between each item description phrase and each canonical phrase by processing the corresponding item description phrase and the corresponding canonical phrase using a natural language processing model. Additional or alternative implementations include generating a bipartite graph based on the set of item description phrases, the set of canonical phrases, and the similarity measures. The mapping can be generated based on the bipartite graph.
1 . A method implemented by one or more processors, the method comprising:
with a Bidirectional Encoder Representation of Transformers (“BERT”) natural language processing (“NLP”) model, processing (a) a set of canonical phrases and (b) a set of item description phrases in an agricultural data set, where each item in the agricultural data set is described by an item description phrase in the set of item description phrases,
wherein the processing generates a similarity measure between each item description phrase in the set of item description phrases, and each canonical phrase in the set of canonical phrases;
generating a bipartite graph based on (a) the set of item description phrases, (b) the set of canonical phrases, and (c) the similarity measure between each item description phrase and each canonical phrase;
generating a mapping between the set of canonical phrases and the set of item description phrases based on the bipartite graph,
wherein the mapping between each item description phrase and the corresponding canonical phrase is based on comparing the edge value, of each edge between the corresponding item description vertex and each of the canonical phrase vertices, and
wherein the mapping between each item description phrase and the corresponding canonical phrase is automatically generated to combine the agricultural data set with an additional agricultural data set;
with the BERT NLP model, processing (a) the set of canonical phrases and (b) a set additional of additional item description phrases in the additional agricultural data set, where each item in the additional agricultural data set is described by an additional item description phrase in the set of additional item description phrases, to generate an additional similarity measure between each additional item description phrase in the set of item description phrases, and each canonical phrase in the set of canonical phrases;
generating an additional bipartite graph based on (a) the set of additional item description phrases, (b) the set of canonical phrases, and (c) the additional similarity measure between each item description phrase and each canonical phrase;
generating an additional mapping from the set of canonical phrases to the set of additional item description phrases based on the additional bipartite graph,
wherein the mapping between each item description phrase and the corresponding canonical phrase is automatically generated to combine the agricultural data set with the additional agricultural data set;
generating a unified agricultural data set based on (a) the set of canonical phrases, (b) the mapping corresponding to the agricultural data set, and (c) the additional mapping corresponding to the additional agricultural data set;
training a crop yield model based on the agricultural data set, the additional agricultural data set, the mapping, and the additional mapping; and
generating a predicted crop yield for a crop captured in the agricultural data set and/or the additional agricultural data set based on processing the unified agricultural data set using the trained crop yield prediction model.
2 . The method of claim 1 , wherein generating the bipartite graph based on (a) the set of item description phrases, (b) the set of canonical phrases, and (c) the similarity measure between each item description phrase and each canonical phrase comprises:
generating the bipartite graph where the set of item description phrases is a first set of vertices, the canonical phrases is a second set of vertices, and where a weight of an edge between each item description phrase and each canonical phrase is the corresponding similarity measure between the item description phrase and the canonical phrase.
3 . The method of claim 2 , wherein generating the mapping between the set of canonical phrases and the set of item description phrases based on the bipartite graph comprises globally optimizing the bipartite graph.
4 . The method of claim 3 , wherein globally optimizing the bipartite graph comprises:
globally optimizing the bipartite graph by processing the bipartite graph using a bipartite graph perfect match process, a Hungarian algorithm, or a Hopcroft-Karp algorithm.
5 . The method of claim 1 , wherein the agricultural data set is a third party agricultural data set, and the method further comprises onboarding the agricultural data set to a database that is indexed by the set of canonical phrases based on the mapping.
6 . The method of claim 1 , wherein the agricultural data set includes data captured from a first farm in an agricultural marketplace, wherein the additional data set includes data captured from a second farm in the agricultural marketplace, and wherein the predicted crop yield generated based on the unified agricultural data set is a predicted crop yield for the agricultural marketplace.
7 . The method of claim 6 , further comprising:
processing the predicted crop yield for the agricultural marketplace to plan a crop harvest of the first farm.
8 . The method of claim 1 , wherein generating the unified agricultural data set based on (a) the set of canonical phrases, (b) the bipartite graph corresponding to the agricultural data set, and (c) the additional bipartite graph corresponding to the additional agricultural data set comprises:
determining whether the mapping between the set of canonical phrases and the set of item description phrases based on the bipartite graph satisfies a threshold confidence value;
determining whether the additional mapping between the set of canonical phrases and the additional item description phrases based on the additional bipartite graph satisfies the threshold confidence value;
in response to determining the mapping between the set of canonical phrases and the set of item description phrases based on the bipartite graph satisfies the threshold confidence value and in response to determining the additional mapping between the set of canonical phrases and the additional item description phrases based on the additional bipartite graph satisfies the threshold confidence value:
generating the unified agricultural data set based on (a) the set of canonical phrases, (b) the bipartite graph corresponding to the agricultural data set, and (c) the additional bipartite graph corresponding to the additional agricultural data set.
9 . A non-transitory computer-readable medium comprising instructions that when executed by one or more processors of a computing system, cause the computing system to perform a method of:
with a Bidirectional Encoder Representation of Transformers (“BERT”) natural language processing (“NLP”) model, processing (a) a set of canonical phrases and (b) a set of item description phrases in an agricultural data set, where each item in the agricultural data set is described by an item description phrase in the set of item description phrases,
wherein the processing generates a similarity measure between each item description phrase in the set of item description phrases, and each canonical phrase in the set of canonical phrases;
generating a bipartite graph based on (a) the set of item description phrases, (b) the set of canonical phrases, and (c) the similarity measure between each item description phrase and each canonical phrase;
generating a mapping between the set of canonical phrases and the set of item description phrases based on the bipartite graph,
wherein the mapping between each item description phrase and the corresponding canonical phrase is based on comparing the edge value, of each edge between the corresponding item description vertex and each of the canonical phrase vertices, and
wherein the mapping between each item description phrase and the corresponding canonical phrase is automatically generated to combine the agricultural data set with an additional agricultural data set;
with the BERT NLP model, processing (a) the set of canonical phrases and (b) a set additional of additional item description phrases in the additional agricultural data set, where each item in the additional agricultural data set is described by an additional item description phrase in the set of additional item description phrases, to generate an additional similarity measure between each additional item description phrase in the set of item description phrases, and each canonical phrase in the set of canonical phrases;
generating an additional bipartite graph based on (a) the set of additional item description phrases, (b) the set of canonical phrases, and (c) the additional similarity measure between each item description phrase and each canonical phrase;
generating an additional mapping from the set of canonical phrases to the set of additional item description phrases based on the additional bipartite graph,
wherein the mapping between each item description phrase and the corresponding canonical phrase is automatically generated to combine the agricultural data set with the additional agricultural data set;
generating a unified agricultural data set based on (a) the set of canonical phrases, (b) the mapping corresponding to the agricultural data set, and (c) the additional mapping corresponding to the additional agricultural data set;
training a crop yield model based on the agricultural data set, the additional agricultural data set, the mapping, and the additional mapping; and
generating a predicted crop yield for a crop captured in the agricultural data set and/or the additional agricultural data set based on processing the unified agricultural data set using the trained crop yield prediction model.
10 . The non-transitory computer-readable medium of claim 9 , wherein generating the bipartite graph based on (a) the set of item description phrases, (b) the set of canonical phrases, and (c) the similarity measure between each item description phrase and each canonical phrase comprises:
generating the bipartite graph where the set of item description phrases is a first set of vertices, the canonical phrases is a second set of vertices, and where a weight of an edge between each item description phrase and each canonical phrase is the corresponding similarity measure between the item description phrase and the canonical phrase.
11 . The non-transitory computer-readable medium of claim 10 , wherein generating the mapping between the set of canonical phrases and the set of item description phrases based on the bipartite graph comprises globally optimizing the bipartite graph.
12 . The non-transitory computer-readable medium of claim 11 , wherein globally optimizing the bipartite graph comprises:
globally optimizing the bipartite graph by processing the bipartite graph using a bipartite graph perfect match process, a Hungarian algorithm, or a Hopcroft-Karp algorithm.
13 . The non-transitory computer-readable medium of claim 9 , wherein the agricultural data set is a third party agricultural data set, and the method further comprises onboarding the agricultural data set to a database that is indexed by the set of canonical phrases based on the mapping.
14 . A system comprising one or more processors configured to perform the following operations:
with a Bidirectional Encoder Representation of Transformers (“BERT”) natural language processing (“NLP”) model, processing (a) a set of canonical phrases and (b) a set of item description phrases in an agricultural data set, where each item in the agricultural data set is described by an item description phrase in the set of item description phrases,
wherein the processing generates a similarity measure between each item description phrase in the set of item description phrases, and each canonical phrase in the set of canonical phrases;
generating a bipartite graph based on (a) the set of item description phrases, (b) the set of canonical phrases, and (c) the similarity measure between each item description phrase and each canonical phrase;
generating a mapping between the set of canonical phrases and the set of item description phrases based on the bipartite graph,
wherein the mapping between each item description phrase and the corresponding canonical phrase is based on comparing the edge value, of each edge between the corresponding item description vertex and each of the canonical phrase vertices, and
wherein the mapping between each item description phrase and the corresponding canonical phrase is automatically generated to combine the agricultural data set with an additional agricultural data set;
with the BERT NLP model, processing (a) the set of canonical phrases and (b) a set additional of additional item description phrases in the additional agricultural data set, where each item in the additional agricultural data set is described by an additional item description phrase in the set of additional item description phrases, to generate an additional similarity measure between each additional item description phrase in the set of item description phrases, and each canonical phrase in the set of canonical phrases;
generating an additional bipartite graph based on (a) the set of additional item description phrases, (b) the set of canonical phrases, and (c) the additional similarity measure between each item description phrase and each canonical phrase;
generating an additional mapping from the set of canonical phrases to the set of additional item description phrases based on the additional bipartite graph,
wherein the mapping between each item description phrase and the corresponding canonical phrase is automatically generated to combine the agricultural data set with the additional agricultural data set;
generating a unified agricultural data set based on (a) the set of canonical phrases, (b) the mapping corresponding to the agricultural data set, and (c) the additional mapping corresponding to the additional agricultural data set;
training a crop yield model based on the agricultural data set, the additional agricultural data set, the mapping, and the additional mapping; and
generating a predicted crop yield for a crop captured in the agricultural data set and/or the additional agricultural data set based on processing the unified agricultural data set using the trained crop yield prediction model.