Requirements discovery for generative ai software development assistant
Techniques for leveraging a large language model (LLM) in software development are described. A description of a software development task is received from a user. Data associated with the software system is obtained from a data source. An LLM is prompted to identify at least one aspect of the task which requires clarification from the user, at least partly by providing the obtained data to the LLM and asking the LLM to identify a question for the user which remains unanswered by the obtained data. The question is presented to the user. An answer to the question is received from the user. The LLM is prompted to respond to propose an implementation of the task at least partly based on the data associated with the software system and the answer received from the user. The proposed implementation is received from the LLM and caused to be displayed to the user.
1 . A system comprising:
a large language model (LLM) hosted in a multi-tenant provider network; and
a software development service (SDS) provided by the multi-tenant provider network, the SDS executing SDS code using one or more processors to cause the SDS to:
receive, by a system design agent of the executing SDS code from a user, a description of a software development task to make a change to a software system;
prompt, by the system design agent of the executing SDS code, the LLM with a first prompt to generate a set of questions related to the software development task, wherein the first prompt includes the description of the software development task;
receive, by the system design agent of the executing SDS code, the set of questions from the LLM;
obtain, by one or more data-gathering context aggregators of the executing SDS code in response to receiving the description of the software development task to make the change to the software system, data associated with at least the software system by one or more of (i) accessing source code of the software system, (ii) accessing documentation about the software system, or (iii) accessing information about resources in the multi-tenant provider network which are used to run the software system;
prompt, by the system design agent of the executing SDS code, the LLM with a second prompt to answer individual questions in the set of questions using the obtained data;
receive answers to some questions in the set of questions from the LLM;
determine that a particular question in the set of questions remains unanswered;
request a response from the user to the particular question;
receive an answer to the particular question from the user;
prompt, by the system design agent of the executing SDS code, the LLM with a third prompt to respond with a proposed solution to the software development task, wherein the third prompt includes the answers received from the LLM and the answer to the particular question received from the user as well as at least some of the obtained data;
receive a response to the software development task from the LLM; and
send the response to the software development task to the user.
2 . The system of claim 1 , further comprising a chat-based interface configured to receive the description of the software development task from the user and output the response to the software development task to the user.
3 . The system of claim 1 , wherein the first prompt includes a limit on a maximum number of questions that can be included in the set of questions.
4 . A system comprising:
a large language model (LLM) in a multi-tenant provider network; and
a software development service (SDS) in the multi-tenant provider network, the SDS executing SDS code using one or more processors to cause the SDS to:
receive, by a system design agent of the executing SDS code from a user, a description of a software development task to make a change to a software system;
obtain, by one or more data-gathering context aggregators of the executing SDS code in response to receiving the description of the software development task to make the change to the software system, data associated with at least the software system from a data source by one or more of (i) accessing source code of the software system, (ii) accessing documentation about the software system, or (iii) accessing information about resources in the multi-tenant provider network which are used to run the software system;
prompt, by the system design agent of the executing SDS code, the LLM to identify at least one aspect of implementing the change to the software system which requires clarification from the user, at least partly by providing the obtained data to the LLM and asking the LLM to identify a question for the user which remains unanswered by the obtained data;
present the question to the user;
receive an answer to the question from the user;
prompt, by the system design agent of the executing SDS code, the LLM to respond to propose an implementation of the change to the software system at least partly based on the data associated with the software system and the answer received from the user;
receive the proposed implementation of the change to the software system from the LLM; and
cause display of the proposed implementation of the change to the software system to the user.
5 . The system of claim 4 , wherein to prompt the LLM to identify at least one aspect of implementing the change which requires clarification from the user, the SDS is to:
prompt the LLM with a first prompt to generate a set of questions, wherein the first prompt includes the description of the software development task;
receive the set of questions from the LLM; and
prompt the LLM with a second prompt to answer questions in the set of questions using the obtained data, wherein the question for the user which remains unanswered is a question in the set of questions.
6 . The system of claim 5 , wherein the obtained data is also associated with a plurality of other software systems accessible to the user, and wherein to ask the LLM to identify the question for the user which remains unanswered by the obtained data, the SDS is to:
prompt the LLM to indicate whether there is a consensus across answers to the question based on each software system represented in the obtained data including the software system and the other software systems; and
receive, from the LLM, an indication that there is not consensus in answers to the question.
7 . The system of claim 5 , wherein to ask the LLM to identify a question for the user which remains unanswered by the obtained data, the SDS is to:
prompt the LLM to provide a confidence score associated with an LLM-provided answer to the question for the software system;
receive the confidence score from the LLM; and
determine that the confidence score does not satisfy a threshold.
8 . The system of claim 5 , wherein the first prompt includes a limit on a maximum number of questions that can be included in the set of questions.
9 . The system of claim 8 , wherein the SDS is further configured to:
prompt the LLM with a third prompt to identify an estimated number of questions related to design considerations to respond to the software development task; and
receive the estimated number of questions from the LLM, wherein the limit is the estimated number of questions.
10 . The system of claim 5 , wherein an application including a chat-based interface provides an interface for the user, the description of the software development task received from the application, and the proposed implementation of the change to the software system sent to the application.
11 . The system of claim 5 , wherein the SDS is further configured to:
prompt the LLM with a validation prompt to check whether the set of questions conforms with a response definition, the validation prompt including the response definition and the set of questions; and
receive, from the LLM, an indication that the set of questions conforms with the response definition.
12 . The system of claim 4 , wherein the SDS is further configured to:
prompt the LLM with a sanitization prompt to check whether the answer from the user includes objectionable material; and
receive, from the LLM, an indication that the answer does not include objectionable material.
13 . A computer-implemented method comprising:
receiving, by a system design agent implemented by a software development service (SDS) executing SDS code using one or more processors, from a user, a description of a software development task to make a change to a software system;
obtaining, by one or more data-gathering context aggregators of the executing SDS code in response to receiving the description of the software development task to make the change to the software system, data associated with at least the software system from a data source by one or more of (i) accessing source code of the software system, (ii) accessing documentation about the software system, or (iii) accessing information about resources in a multi-tenant provider network which are used to run the software system;
prompting, by the system design agent of the executing SDS code, a large language model (LLM) to identify at least one aspect of implementing the change to the software system which requires clarification from the user, at least partly by providing the obtained data to the LLM and asking the LLM to identify a question for the user which remains unanswered by the obtained data;
presenting the question to the user;
receiving an answer to the question from the user;
prompting, by the system design agent of the executing SDS code, the LLM to respond to propose an implementation of the change to the software system at least partly based on the data associated with the software system and the answer received from the user;
receiving the proposed implementation of the change to the software system from the LLM; and
causing display of the proposed implementation of the change to the software system to the user.
14 . The computer-implemented method of claim 13 , wherein to prompt the LLM to identify at least one aspect of implementing the change which requires clarification from the user includes:
prompting the LLM with a first prompt to generate a set of questions, wherein the first prompt includes the description of the software development task;
receiving the set of questions from the LLM; and
prompting the LLM with a second prompt to answer questions in the set of questions using the obtained data, wherein the question for the user which remains unanswered is a question in the set of questions.
15 . The computer-implemented method of claim 14 , wherein the obtained data is also associated with a plurality of other software systems accessible to the user, and wherein to ask the LLM to identify the question for the user which remains unanswered by the obtained data includes:
prompting the LLM to indicate whether there is a consensus across answers to the question based on each software system represented in the obtained data including the software system and the other software systems; and
receiving, from the LLM, an indication that there is not consensus in answers to the question.
16 . The computer-implemented method of claim 14 , wherein to ask the LLM to identify the question for the user which remains unanswered by the obtained data includes:
prompting the LLM to provide a confidence score associated with an LLM-provided answer to the question for the software system;
receiving the confidence score from the LLM; and
determining that the confidence score does not satisfy a threshold.
17 . The computer-implemented method of claim 14 , wherein the first prompt includes a limit on a maximum number of questions that can be included in the set of questions.
18 . The computer-implemented method of claim 17 , further comprising:
prompting the LLM with a third prompt to identify an estimated number of questions related to design considerations to respond to the software development task; and
receiving the estimated number of questions from the LLM, wherein the limit is the estimated number of questions.
19 . The computer-implemented method of claim 14 , wherein an application including a chat-based interface provides an interface for the user, the description of the software development task received from the application, and the proposed implementation of the change to the software system sent to the application.
20 . The computer-implemented method of claim 13 , further comprising:
prompting the LLM with a sanitization prompt to check whether the answer from the user includes objectionable material; and
receiving, from the LLM, an indication that the answer does not include objectionable material.