Reducing latency by processing parts of a language model query in parallel
A technique partitions a user's original query into plural smaller component queries, each of which has a common part and an instance-specific part. The technique distributes the component queries to plural processor instances of a processor. The plural processor instances transform the respective component queries into query-component responses by acting in parallel, independent of each other. The technique generates a final response based on the query-component responses, e.g., by assembling the component-query responses into the final response. The technique reduces latency because the processor instances work on parts of the user's original query at the same time, rather than as a single stream of consecutive tokens. The plural processor instances have access to a shared cache memory, and utilize relevant data that has been computed in response to previous queries.
1. A method for processing a query using a machine-trained language model, comprising:
receiving an original query;
generating component queries based on the original query, the component queries having a same common part, and the component queries having different respective instance-specific parts;
distributing the component queries to respective processor instances, the processor instances being instances of one or more processors, each processor instance executing an instance of the machine-trained language model,
the processor instances generating respective component-query responses in parallel based on the plural component queries, and based on intermediate results previously generated by the machine-trained language model and stored in a cache memory, the previously generated intermediate results including key-value information used in performing an attention operation in the machine-trained language model;
receiving the component-query responses;
generating a final response based on the component-query responses; and
generating output information based on the final response.
2. The method of claim 1 ,
wherein the original query includes a question and source text, the source text serving as context for use by the language model in answering the question, and
wherein the instance-specific parts are associated with respective selected source parts of the source text.
3. The method of claim 2 , wherein the selected source parts are a subset of the source text that are collectively less than an entirety of the source text.
4. The method of claim 2 , wherein the source text includes plural consecutive source parts, and wherein the selected source parts include at least a contiguous portion of the plural consecutive source parts.
5. The method of claim 2 , wherein the selected source parts are automatically selected based on a determination that the selected source parts have a greatest relevance to the question.
6. The method of claim 1 , further comprising assigning a common query identifier to the component queries, wherein the component-query responses are associated with the common query identifier.
7. The method of claim 1 , further comprising determining whether it is appropriate to partition the original query into the component queries by determining whether the original query includes a predetermined key term and/or a predetermined semantic concept, and/or matches a predetermined structure.
8. The method of claim 1 , wherein the generating a final response comprises assembling the component-query responses into the final response in an order in which the component queries appear in the original query.
9. The method of claim 1 , wherein the generating a final response comprises comparing information imparted by at least two of the component-query responses, and generating an output result that expresses a result of the comparing.
10. The method of claim 1 , further comprising detecting a predetermined key term and/or a predetermined semantic concept in the original query and/or the component query responses, wherein the generating a final response is controlled based on the key term and/or semantic concept that has been detected by the detecting.
11. The method of claim 1 , further comprising: determining that at least one of the component-query responses includes invocation information associated with a particular function; invoking the particular function; and receiving supplemental data in response to the invoking.
12. The method of claim 1 , wherein the method further comprises:
automatically generating additional component queries based on at least one of the component-query responses, the additional component queries having a child relationship with respect to said at least one of the component-query responses; and
instructing the plural processor instances to generate another set of component-query responses based on the additional component queries.
13. The method of claim 12 , wherein the additional component queries include supplemental data obtained in response to invoking a particular function, the particular function being invoked in response to invocation information provided by the component-query responses.
14. The method of claim 1 , further comprising selecting the one or more processors from among a group of candidate processors based on a determination that the original query contains tokens that have been previously processed by the one or more processors.
15. A computing system for processing a query using a machine-trained language model, comprising:
an instruction data store for storing computer-readable instructions; and
a processing system for executing the computer-readable instructions in the data store, to perform operations including:
receiving an original query that includes a question and source text;
determining that the original query is capable of being partitioned into component queries;
in response to the determining, generating the component queries based on the question and the source text, the component queries having a same common part that expresses the question, and the component queries expressing respective selected source parts of the source text;
distributing the component queries to respective processor instances, the processor instances being instances of one or more processors, each processor instance executing an instance of the machine-trained language model,
the processor instances generating respective component-query responses in parallel based on the plural component queries, and based on previously-generated intermediary results stored in a shared cache memory of the one or more processors,
the previously-generated intermediate results including key-value information used in performing an attention operation in the machine-trained language model;
receiving the component-query responses;
generating a final response based on the component-query responses; and
generating output information based on the final response.
16. The computing system of claim 15 , wherein the determining that the query is capable of being partitioned involves determining whether the original query includes a predetermined key term and/or a predetermined semantic concept, and/or matches a predetermined structure.
17. A computer-readable storage medium for storing computer-readable instructions, a processing system executing the computer-readable instructions to perform operations, the operations comprising each of:
receiving an original query that includes a question and a separate source text that accompanies the question and is separate therefrom,
the question containing a reference to the source text,
the source text including plural source parts that serve as context for use by a machine-trained language model in answering the question;
upon determining that the original query is severable into independent parts, generating component queries based on the original query, the component queries having a same common part, and the component queries expressing respective selected source parts of the plural source parts;
distributing the component queries to respective processor instances, the processor instances being instances of one or more processors, each processor instance executing an instance of the machine-trained language model,
the processor instances auto-regressively generating respective component-query responses in parallel based on the plural component queries;
receiving the component-query responses;
generating a final response based on the component-query responses; and
generating output information based on the final response.
18. The computer-readable storage medium of claim 17 , wherein the selected source parts are a subset of the source text that are collectively less than an entirety of the source text.
19. The computer-readable storage medium of claim 18 , wherein the operations further include automatically selecting the source parts from the source text based on a determination that the source parts have a greatest relevance to the question.
20. The computer-readable storage medium of claim 17 , wherein the processor instances generate the respective component-query responses based, in part, on intermediate results previously generated by the machine-trained language model and stored in a cache memory, the previously generated intermediate results including key-value information used in performing an attention operation in the machine-trained language model.