Artificial intelligence based metadata semantic enrichment
Mechanisms are provided for automatically generating semantical enhanced metadata for a structured data structure. Multi-task machine learning training is performed, based on data comprising separate sets of training data samples for each of a plurality of semantic metadata enhancement tasks, of a base artificial intelligence (AI) computer model to thereby generate a fine-tuned AI computer model trained to specifically generate semantically enhanced metadata for structured data structures. A prompt is received that specifies a structure of an input structured data structure and requests a semantic metadata enhancement task from the plurality of semantic metadata enhancement tasks. The fine-tuned AI computer model processes the prompt to generate semantically enhanced metadata for the structure of the input structured data structure and provide it to a downstream computing system for performing a downstream computing operation based on the semantically enhanced metadata.
1 . A method, in a data processing system, for automatically generating semantical enhanced metadata for a structured data structure, the method comprising:
generating multi-task machine learning training data comprising separate sets of training data samples for each of a plurality of semantic metadata enhancement tasks;
executing multi-task machine learning training of a base artificial intelligence (AI) computer model based on the multi-task machine learning training data to thereby generate a fine-tuned AI computer model trained to specifically generate semantically enhanced metadata for structured data structures, wherein the fine-tuned AI computer model is trained to perform the plurality of semantic metadata enhancement tasks;
receiving a prompt specifying a structure of an input structured data structure and requesting at least one of the semantic metadata enhancement tasks from the plurality of semantic metadata enhancement tasks;
processing, by the fine-tuned AI computer model, the prompt to generate semantically enhanced metadata for the structure of the input structured data structure; and
providing the semantically enhanced metadata to a downstream computing system for performing a downstream computing operation based on the semantically enhanced metadata.
2 . The method of claim 1 , wherein the input structured data structure is a table data structure and the structure of the input structured data structure comprises column names of the table data structure.
3 . The method of claim 2 , wherein the semantic metadata enhancement tasks comprise at least one of generating table captions, generating column descriptions, generating column datatypes, or generating tags for portions of the table data structure.
4 . The method of claim 1 , wherein the base AI computer model is a previously trained foundation model that is trained on a large and varied set of data for general applicability.
5 . The method of claim 4 , wherein the previously trained foundation model is one of a large language model (LLM), a generative adversarial network (GAN), or a variational autoencoder (VAE).
6 . The method of claim 1 , wherein executing multi-task machine learning training of a base artificial intelligence (AI) computer model based on the multi-task machine learning training data comprises generating a plurality of batches of training data samples from the multi-task machine learning training data, wherein each batch comprises a proportional number of training data samples for each of the semantic metadata enhancement tasks.
7 . The method of claim 6 , wherein each batch of training data samples comprises at least two training data samples for opposite semantic metadata enhancement tasks.
8 . The method of claim 1 , wherein the multi-task machine learning training data is generated by curating a larger size training data to a selected subset of training data at least by filtering the larger size training data based on filtering criteria that eliminates structured data structures, or portions of structured data structures, that are determined to be not informative with regard to the structure of the structured data structure.
9 . The method of claim 2 , wherein the downstream computing operation comprises at least one of correlating the table data structure, or portions of the table data structure, with one of dictionaries or ontologies, or performing a table data structure similarity evaluation with one or more other table data structures.
10 . The method of claim 1 , wherein processing the prompt is performed without accessing the data within the structured input structured data structure.
11 . A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:
generate multi-task machine learning training data comprising separate sets of training data samples for each of a plurality of semantic metadata enhancement tasks;
execute multi-task machine learning training of a base artificial intelligence (AI) computer model based on the multi-task machine learning training data to thereby generate a fine-tuned AI computer model trained to specifically generate semantically enhanced metadata for structured data structures, wherein the fine-tuned AI computer model is trained to perform the plurality of semantic metadata enhancement tasks;
receive a prompt specifying a structure of an input structured data structure and requesting at least one of the semantic metadata enhancement tasks from the plurality of semantic metadata enhancement tasks;
process, by the fine-tuned AI computer model, the prompt to generate semantically enhanced metadata for the structure of the input structured data structure; and
provide the semantically enhanced metadata to a downstream computing system for performing a downstream computing operation based on the semantically enhanced metadata.
12 . The computer program product of claim 11 , wherein the input structured data structure is a table data structure and the structure of the input structured data structure comprises column names of the table data structure.
13 . The computer program product of claim 12 , wherein the semantic metadata enhancement tasks comprise at least one of generating table captions, generating column descriptions, generating column datatypes, or generating tags for portions of the table data structure.
14 . The computer program product of claim 11 , wherein the base AI computer model is a previously trained foundation model that is trained on a large and varied set of data for general applicability.
15 . The computer program product of claim 14 , wherein the previously trained foundation model is one of a large language model (LLM), a generative adversarial network (GAN), or a variational autoencoder (VAE).
16 . The computer program product of claim 11 , wherein executing multi-task machine learning training of a base artificial intelligence (AI) computer model based on the multi-task machine learning training data comprises generating a plurality of batches of training data samples from the multi-task machine learning training data, wherein each batch comprises a proportional number of training data samples for each of the semantic metadata enhancement tasks.
17 . The computer program product of claim 16 , wherein each batch of training data samples comprises at least two training data samples for opposite semantic metadata enhancement tasks.
18 . The computer program product of claim 11 , wherein the multi-task machine learning training data is generated by curating a larger size training data to a selected subset of training data at least by filtering the larger size training data based on filtering criteria that eliminates structured data structures, or portions of structured data structures, that are determined to be not informative with regard to the structure of the structured data structure.
19 . The computer program product of claim 11 , wherein processing the prompt is performed without accessing the data within the structured input structured data structure.
20 . An apparatus comprising:
at least one processor; and
at least one memory coupled to the at least one processor, wherein the at least one memory comprises instructions which, when executed by the at least one processor, cause the at least one processor to:
generate multi-task machine learning training data comprising separate sets of training data samples for each of a plurality of semantic metadata enhancement tasks;
execute multi-task machine learning training of a base artificial intelligence (AI) computer model based on the multi-task machine learning training data to thereby generate a fine-tuned AI computer model trained to specifically generate semantically enhanced metadata for structured data structures, wherein the fine-tuned AI computer model is trained to perform the plurality of semantic metadata enhancement tasks;
receive a prompt specifying a structure of an input structured data structure and requesting at least one of the semantic metadata enhancement tasks from the plurality of semantic metadata enhancement tasks;
process, by the fine-tuned AI computer model, the prompt to generate semantically enhanced metadata for the structure of the input structured data structure; and
provide the semantically enhanced metadata to a downstream computing system for performing a downstream computing operation based on the semantically enhanced metadata.