Artificial intelligence based generation of data connectors
A device that generates a data connector. The device accesses an index based on representations of documents storing information describing APIs for accessing data source systems, and generates a prompt for a LLM based on information extracted from the index that requests information for accessing data from the data source system. The device provides the generated prompt to the LLM and receives a response describing information associated with the data source. The device determines a measure of ambiguity for the response and, responsive to the measure of ambiguity indicating more than a threshold ambiguity, generates an additional prompt that requests more specific information compared to the information received in the response. The device generates a data connector based on a connector knowledge representation corresponding to the response and deploys the data connector for accessing data from the data source system.
1 . A computer-implemented method for generation of a data connector, the method comprising:
accessing an index based on representations of documents storing information describing APIs (Application Programming Interfaces) for accessing data source systems;
determining a subset of documents referenced by the index having at least a threshold similarity to required connector information;
generating a prompt for a large language model (LLM) based on information extracted from the index, the prompt requesting information associated with building a new data connector for accessing data from the data source systems;
providing the generated prompt to the large language model along with instructions to query the subset of documents with the generated prompt;
receiving a respective interim response from each respective document of the subset;
receiving, from the large language model, a response describing information associated with the new data connector, the response comprising each interim response;
determining a measure of ambiguity for the response;
responsive to the measure of ambiguity indicating more than a threshold ambiguity in the response based on two or more interim responses comprising conflicting information, generating an additional prompt for the large language model the additional prompt automatically requesting more specific information compared to the information received in the response, the more specific information requested to resolve the conflicting information, wherein as further conflicting information is received, further prompts requesting further resolution are requested until an ultimate response without an ambiguity is received;
generating the data connector based on a connector knowledge representation corresponding to the ultimate response; and
automatically deploying the data connector for accessing data from the data source systems.
2 . The method of claim 1 , further comprising, repeatedly generating subsequent prompts, each subsequent prompt requesting more specific information compared to previous prompts and providing the subsequent prompts to the large language model.
3 . The method of claim 1 , further comprising:
initializing a knowledge graph as the connector knowledge representation, the knowledge graph including nodes corresponding to requirements for generating the data connector; and
accessing topics of the index based on the requirements for generating the data connector.
4 . The method of claim 3 , wherein the topics correspond to topics extracted from cleansed document headers crawled from API documentation.
5 . The method of claim 1 , wherein determining the response from the interim response comprises:
determining the measure of ambiguity to be below the threshold ambiguity based on determining that a first of the interim responses is not a null response and other interim responses are null responses; and
determining the response to be equivalent to a result of the first of the interim responses.
6 . The method of claim 1 , wherein the additional prompt includes context of the conflicting information.
7 . A non-transitory computer-readable medium comprising memory with instructions encoded thereon for generation of a data connector, the instructions comprising instructions to:
access an index based on representations of documents storing information describing APIs (Application Programming Interfaces) for accessing data source systems;
determine a subset of documents referenced by the index having at least a threshold similarity to required connector information;
generate a prompt for a large language model (LLM) based on information extracted from the index, the prompt requesting information associated with building a new data connector for accessing data from the data source systems;
provide the generated prompt to the large language model along with instructions to query the subset of documents with the generated prompt;
receive a respective interim response from each respective document of the subset;
receive, from the large language model, a response describing information associated with the new data connector, the response comprising each interim response;
determine a measure of ambiguity for the response;
responsive to the measure of ambiguity indicating more than a threshold ambiguity in the response based on two or more interim responses comprising conflicting information, generate an additional prompt for the large language model the additional prompt automatically requesting more specific information compared to the information received in the response, the more specific information requested to resolve the conflicting information, wherein as further conflicting information is received, further prompts requesting further resolution are requested until an ultimate response without an ambiguity is received;
generate the data connector based on a connector knowledge representation corresponding to the ultimate response; and
automatically deploy the data connector for accessing data from the data source systems.
8 . The non-transitory computer-readable medium of claim 7 , the instructions further comprising instructions to repeatedly generate subsequent prompts, each subsequent prompt requesting more specific information compared to previous prompts and providing the subsequent prompts to the large language model.
9 . The non-transitory computer-readable medium of claim 7 , the instructions further comprising instructions to:
initialize a knowledge graph as the connector knowledge representation, the knowledge graph including nodes corresponding to requirements for generating the data connector; and
access topics of the index based on the requirements for generating the data connector.
10 . The non-transitory computer-readable medium of claim 9 , wherein the topics correspond to topics extracted from cleansed document headers crawled from API documentation.
11 . The non-transitory computer-readable medium of claim 7 , wherein the instructions to determine the response from the interim response comprise instructions to:
determine the measure of ambiguity to be below the threshold ambiguity based on determining that a first of the interim responses is not a null response and other interim responses are null responses; and
determine the response to be equivalent to a result of the first of the interim responses.
12 . The non-transitory computer-readable medium of claim 7 , wherein the additional prompt includes context of the conflicting information.
13 . A system for generation of a data connector, the system comprising:
memory with instructions encoded thereon; and
one or more processors that, when executing the instructions, are caused to perform operations comprising:
accessing an index based on representations of documents storing information describing APIs (Application Programming Interfaces) for accessing data source systems;
determining a subset of documents referenced by the index having at least a threshold similarity to required connector information;
generating a prompt for a large language model (LLM) based on information extracted from the index, the prompt requesting information associated with building a new data connector for accessing data from the data source systems;
providing the generated prompt to the large language model along with instructions to query the subset of documents with the generated prompt;
receiving a respective interim response from each respective document of the subset;
receiving, from the large language model, a response describing information associated with the new data connector, the response comprising each interim response;
determining a measure of ambiguity for the response;
responsive to the measure of ambiguity indicating more than a threshold ambiguity in the response based on two or more interim responses comprising conflicting information, generating an additional prompt for the large language model the additional prompt automatically requesting more specific information compared to the information received in the response, the more specific information requested to resolve the conflicting information, wherein as further conflicting information is received, further prompts requesting further resolution are requested until an ultimate response without an ambiguity is received;
generating the data connector based on a connector knowledge representation corresponding to the ultimate response; and
automatically deploying the data connector for accessing data from the data source systems.
14 . The system of claim 13 , the operations further comprising:
initializing a knowledge graph as the connector knowledge representation, the knowledge graph including nodes corresponding to requirements for generating the data connector; and
accessing topics of the index based on the requirements for generating the data connector.
15 . The system of claim 14 , wherein the topics correspond to topics extracted from cleansed document headers crawled from API documentation.