Reduction of latency in retriever-reader architectures
Disclosed is a technical solution to reduce latency in reader-retriever architectures of an open domain query response system. An example apparatus is to reduce latency in a retriever-reader architecture comprising at least one memory, machine readable instructions, and processor circuitry to at least one of instantiate or execute the machine readable instructions to identify a query, the query having data associated with the query, create a data array based on the data associated with the query, and deploy a machine learning model. The machine learning model is to assign a score to an index of the data array based on a likelihood that a response to the query exists in the index, determine an action to perform on the index based on a comparison of the score and a ranking threshold, and produce the response from the data array based on the action.
1 . An apparatus to reduce latency in a retriever-reader architecture comprising:
at least one memory;
machine readable instructions; and
at least one programmable circuit to at least one of instantiate or execute the machine readable instructions to:
identify a query, the query having data associated with the query;
create a data array based on the data associated with the query; and
deploy a machine learning model including a plurality of encoder layers, the machine learning model to at least:
assign a score to an index of the data array based on a likelihood that a response to the query exists in the index;
filter, at one or more of the plurality of encoder layers, the data array by removal of the index from the data array based on a comparison of the score and a layer-specific ranking threshold, the layer-specific ranking threshold determined based on a filtering rate for the encoder layer; and
produce the response from the data array after the filtering.
2 . The apparatus of claim 1 , wherein one or more of the at least one programmable circuit is to reduce a size of the data array after removal of the index.
3 . The apparatus of claim 1 , wherein one or more of the at least one programmable circuit is to determine the filtering rate based on a decay function for the identified encoder layer.
4 . The apparatus of claim 1 , wherein one or more of the at least one programmable circuit is to utilize a decay function to identify the encoder layer.
5 . The apparatus of claim 1 , wherein one or more of the at least one programmable circuit is to produce the response by utilizing a decoder of the retriever-reader architecture to decode the data array.
6 . The apparatus of claim 1 , wherein the one or more of the at least one programmable circuit is to update the machine learning model based on the response produced.
7 . The apparatus of claim 1 , wherein the filtering rate differs between at least two of the plurality of encoder layers.
8 . The apparatus of claim 1 , wherein the layer-specific ranking threshold increases or decreases monotonically across successive encoder layers.
9 . The apparatus of claim 1 , wherein the filtering rate controls an extent of index removal independently of the comparison of the score and the layer-specific ranking threshold.
10 . The apparatus of claim 1 , wherein the filtering rate specifies a proportion of indices removed from the data array at a respective encoder layer.
11 . At least one non-transitory machine readable storage medium comprising instructions that, when executed, cause processor circuitry to at least:
identify a query, the query having data associated with the query;
create a data array based on data associated with the query;
assign a score to an index of the data array based on a likelihood that a response to the query exists in the index;
filter, at one or more of a plurality of encoder layers of a machine learning model, the data array by removal of the index from the data array based on a comparison of the assigned score and a layer-specific ranking threshold, the layer-specific ranking threshold determined based on a filtering rate for the encoder layer; and
produce the response from the data array after filtering.
12 . The at least one non-transitory machine readable storage medium of claim 11 , wherein the instructions, when executed, further cause the processor circuitry to reduce a size of the data array after removal of the index.
13 . The at least one non-transitory machine readable storage medium of claim 11 , wherein the instructions, when executed, further cause the processor circuitry to determine the filtering rate based on a decay function for the identified encoder layer.
14 . The at least one non-transitory machine readable storage medium of claim 11 , wherein the instructions, when executed, further cause the processor circuitry to utilize a decay function to identify the encoder layer.
15 . The at least one non-transitory machine readable storage medium of claim 11 , wherein the instructions, when executed, further cause the processor circuitry to produce the response by utilizing a decoder of a retriever-reader architecture to decode the data array.
16 . The at least one non-transitory machine readable storage medium of claim 11 , wherein the instructions, when executed, further cause the processor circuitry to update the machine learning model based on the response produced.
17 . The at least one non-transitory machine readable storage medium of claim 11 , wherein the instructions, when executed, cause the processor circuitry to utilize a filtering rate that differs between at least two of the plurality of encoder layers.
18 . The at least one non-transitory machine readable storage medium of claim 11 , wherein the instructions, when executed, cause the processor circuitry to utilize a layer-specific ranking threshold that increases or decreases monotonically across successive encoder layers.
19 . The at least one non-transitory machine readable storage medium of claim 11 , wherein the instructions, when executed, cause the processor circuitry to control an extent of index removal using the filtering rate independently of the comparison of the assigned score and the layer-specific ranking threshold.
20 . An apparatus to reduce latency in a reader-retriever architecture comprising:
means for accessing a data array, the data array including data associated with a query;
means for assigning a score to an index of the data array based on a likelihood that a response to the query exists in the index;
means for removing the index, at one or more of a plurality of encoder layers of a machine learning model, from the data array based on a comparison of the assigned score and a layer-specific ranking threshold, the layer-specific ranking threshold determined based on a filtering rate for the encoder layer; and
means for producing the response from the data array after filtering.
21 . The apparatus of claim 20 , wherein the means for removing is to reduce a size of the data array after removing the index.
22 . The apparatus of claim 20 , wherein the means for accessing is to identify at least one filtering layer of an encoder of the machine learning model.
23 . A method for reducing latency in a reader-retriever architecture using a machine learning model comprising:
creating, by executing an instruction with at least one processor, a data array based on data associated with a query;
assigning a score to an index of the data array based on a likelihood that a response to the query exists in the index;
filtering, by executing an instruction with at least one processor, at one or more of a plurality of encoder layers of a machine learning model, the data array by removal of the index from the data array based on a comparison of the assigned score and a layer-specific ranking threshold, the layer-specific ranking threshold determined based on a filtering rate for the encoder layer; and
producing the response from the data array after filtering.
24 . The method of claim 23 , further including reducing a size of the data array after removal of the index.
25 . The method of claim 23 , further including identifying at least one filtering layer of an encoder of the machine learning model.