Method, apparatus, and computer-readable medium for intent classification of natural language queries in a generative artificial intelligence platform

1 . A method executed by one or more computing devices for intent classification of natural language queries in a generative artificial intelligence platform, the method comprising:

receiving, by at least one of the one or more computing devices, a natural language query from a user in a user interface of a generative artificial intelligence platform, the generative artificial intelligence platform being coupled to an enterprise data catalog and the natural language query comprising a plurality of query terms;

querying, by at least one of the one or more computing devices, an entity index based at least in part on one or more query terms in the plurality of query terms to identify one or more entities corresponding to the natural language query;

identifying, by at least one of the one or more computing devices, one or more assets in the enterprise data catalog corresponding to the one or more entities;

identifying, by at least one of the one or more computing devices, a metadata context corresponding to the one or more assets;

retrieving, by at least one of the one or more computing devices, one or more previous conversations of the user on the generative artificial intelligence platform;

generating, by at least one of the one or more computing devices, a structured prompt based at least in part on the natural language query, the one or more assets, the metadata context corresponding to the one or more assets, and the one or more previous conversations; and

determining, by at least one of the one or more computing devices, an intent associated with the natural language query by inputting the structured prompt into a fine-tuned large language model, the fine-tuned large language model being trained based at least in part on a plurality training queries on the enterprise data catalog, a plurality of training conversations, and a plurality of training metadata contexts.

2 . The method of claim 1 , further comprising:

generating, by at least one of the one or more computing devices, a response to the natural language query based at least in part on the determined intent; and

transmitting, by at least one of the one or more computing devices, the response in the user interface of the generative artificial intelligence platform.

3 . The method of claim 2 , wherein generating a response to the natural language query based at least in part on the determined intent comprises:

generating one or more backend queries based at least in part on the natural language query and the determined intent;

executing the one or more backend queries to identify one or more results; and

generating the response to the natural language query based at least in part on the identified one or more results.

4 . The method of claim 1 , wherein the entity index is generated based at least in part on an entity graph corresponding to the enterprise data catalog, the entity graph comprising a plurality of nodes and a plurality of edges between the plurality of nodes, wherein the plurality of nodes correspond to a plurality of concepts and the plurality of edges correspond to a plurality of relationships between the plurality of concepts.

5 . The method of claim 1 , wherein identifying metadata context corresponding to the one or more assets comprises:

identifying one or more related asset types for each asset in the one or more assets based at least in part on one or more of the entity index or a metadata index generated based at least in part on the enterprise data catalog; and

identifying one or more joinable assets for each asset in the one or more assets based at least in part on a primary key-foreign key relationship between the asset and the other assets in the enterprise data catalog.

6 . The method of claim 5 , wherein the structured prompt comprises:

the natural language query;

at least a portion of the one or more previous conversations;

one or more asset identifiers correspond to the one or more assets;

one or joinable asset identifiers corresponding to the one or more joinable assets for each asset in the one or more assets; and

the one or more related asset types for each asset in the one or more assets.

7 . The method of claim 1 , wherein the fine-tuned large language model is configured to identify an intent from a plurality of potential intents, the plurality of potential intents comprising data catalog discovery, metadata exploration, data exploration, data transformation, and product help.

8 . An apparatus for intent classification of natural language queries in a generative artificial intelligence platform, the apparatus comprising:

one or more processors; and

one or more memories operatively coupled to at least one of the one or more processors and having instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:

receive a natural language query from a user in a user interface of a generative artificial intelligence platform, the generative artificial intelligence platform being coupled to an enterprise data catalog and the natural language query comprising a plurality of query terms;

query an entity index based at least in part on one or more query terms in the plurality of query terms to identify one or more entities corresponding to the natural language query;

identify one or more assets in the enterprise data catalog corresponding to the one or more entities;

identify a metadata context corresponding to the one or more assets;

retrieve one or more previous conversations of the user on the generative artificial intelligence platform;

generate a structured prompt based at least in part on the natural language query, the one or more assets, the metadata context corresponding to the one or more assets, and the one or more previous conversations; and

determine an intent associated with the natural language query by inputting the structured prompt into a fine-tuned large language model, the fine-tuned large language model being trained based at least in part on a plurality training queries on the enterprise data catalog, a plurality of training conversations, and a plurality of training metadata contexts.

9 . The apparatus of claim 8 , wherein at least one of the one or more memories has further instructions stored thereon that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to:

generate a response to the natural language query based at least in part on the determined intent; and

transmit the response in the user interface of the generative artificial intelligence platform.

10 . The apparatus of claim 9 , wherein the instructions that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to generate a response to the natural language query based at least in part on the determined intent further cause at least one of the one or more processors to:

generate one or more backend queries based at least in part on the natural language query and the determined intent;

execute the one or more backend queries to identify one or more results; and

generating the response to the natural language query based at least in part on the identified one or more results.

11 . The apparatus of claim 8 , wherein the entity index is generated based at least in part on an entity graph corresponding to the enterprise data catalog, the entity graph comprising a plurality of nodes and a plurality of edges between the plurality of nodes, wherein the plurality of nodes correspond to a plurality of concepts and the plurality of edges correspond to a plurality of relationships between the plurality of concepts.

12 . The apparatus of claim 8 , wherein the instructions that, when executed by at least one of the one or more processors, cause at least one of the one or more processors to identify metadata context corresponding to the one or more assets further cause at least one of the one or more processors to:

identify one or more related asset types for each asset in the one or more assets based at least in part on one or more of the entity index or a metadata index generated based at least in part on the enterprise data catalog; and

identify one or more joinable assets for each asset in the one or more assets based at least in part on a primary key-foreign key relationship between the asset and the other assets in the enterprise data catalog.

13 . The apparatus of claim 12 , wherein the structured prompt comprises:

the natural language query;

at least a portion of the one or more previous conversations;

one or more asset identifiers correspond to the one or more assets;

one or joinable asset identifiers corresponding to the one or more joinable assets for each asset in the one or more assets; and

the one or more related asset types for each asset in the one or more assets.

14 . The apparatus of claim 8 , wherein the fine-tuned large language model is configured to identify an intent from a plurality of potential intents, the plurality of potential intents comprising data catalog discovery, metadata exploration, data exploration, data transformation, and product help.

15 . At least one non-transitory computer-readable medium storing computer-readable instructions for intent classification of natural language queries in a generative artificial intelligence platform that, when executed by one or more computing devices, cause at least one of the one or more computing devices to:

receive a natural language query from a user in a user interface of a generative artificial intelligence platform, the generative artificial intelligence platform being coupled to an enterprise data catalog and the natural language query comprising a plurality of query terms;

query an entity index based at least in part on one or more query terms in the plurality of query terms to identify one or more entities corresponding to the natural language query;

identify one or more assets in the enterprise data catalog corresponding to the one or more entities;

identify a metadata context corresponding to the one or more assets;

retrieve one or more previous conversations of the user on the generative artificial intelligence platform;

generate a structured prompt based at least in part on the natural language query, the one or more assets, the metadata context corresponding to the one or more assets, and the one or more previous conversations; and

determine an intent associated with the natural language query by inputting the structured prompt into a fine-tuned large language model, the fine-tuned large language model being trained based at least in part on a plurality training queries on the enterprise data catalog, a plurality of training conversations, and a plurality of training metadata contexts.

16 . The at least one non-transitory computer-readable medium of claim 15 , wherein at least one of the one or more memories has further instructions stored thereon that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to:

generate a response to the natural language query based at least in part on the determined intent; and

transmit the response in the user interface of the generative artificial intelligence platform.

17 . The at least one non-transitory computer-readable medium of claim 16 , wherein the instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to generate a response to the natural language query based at least in part on the determined intent further cause at least one of the one or more computing devices to:

generate one or more backend queries based at least in part on the natural language query and the determined intent;

execute the one or more backend queries to identify one or more results; and

generating the response to the natural language query based at least in part on the identified one or more results.

18 . The at least one non-transitory computer-readable medium of claim 15 , wherein the entity index is generated based at least in part on an entity graph corresponding to the enterprise data catalog, the entity graph comprising a plurality of nodes and a plurality of edges between the plurality of nodes, wherein the plurality of nodes correspond to a plurality of concepts and the plurality of edges correspond to a plurality of relationships between the plurality of concepts.

19 . The at least one non-transitory computer-readable medium of claim 15 , wherein the instructions that, when executed by at least one of the one or more computing devices, cause at least one of the one or more computing devices to identify metadata context corresponding to the one or more assets further cause at least one of the one or more computing devices to:

identify one or more related asset types for each asset in the one or more assets based at least in part on one or more of the entity index or a metadata index generated based at least in part on the enterprise data catalog; and

identify one or more joinable assets for each asset in the one or more assets based at least in part on a primary key-foreign key relationship between the asset and the other assets in the enterprise data catalog.

20 . The at least one non-transitory computer-readable medium of claim 19 , wherein the structured prompt comprises:

the natural language query;

at least a portion of the one or more previous conversations;

one or more asset identifiers correspond to the one or more assets;

one or joinable asset identifiers corresponding to the one or more joinable assets for each asset in the one or more assets; and

the one or more related asset types for each asset in the one or more assets.

21 . The at least one non-transitory computer-readable medium of claim 15 , wherein the fine-tuned large language model is configured to identify an intent from a plurality of potential intents, the plurality of potential intents comprising data catalog discovery, metadata exploration, data exploration, data transformation, and product help.