Method for embedding rows prefetching in recommendation models
A system and method for efficiently accessing sparse data for a workload are described. In various implementations, a computing system includes an integrated circuit and a memory for storing tasks of a workload that includes sparse accesses of data items stored in one or more tables. The integrated circuit receives a user query, and generates a result based on multiple data items targeted by the user query. To reduce the latency of processing the workload even with sparse lookup operations performed on the one or more tables, a prefetch engine of the integrated circuit stores a subset of data items in prefetch data storage. The prefetch engine also determines which data items to store in the prefetch data storage based on one or more of a frequency of reuse, a distance or latency of access of a corresponding table of the one more tables, or other.
1 . An integrated circuit comprising:
a prefetch engine comprising circuitry, wherein responsive to receipt of a user query that targets two or more data items, the circuitry is configured to:
retrieve, from a prefetch data storage, a first group of one or more data items of the two or more data items, identified as being stored in the prefetch data storage;
send the first group of data items to a data processing stage comprising processing circuitry, prior to sending remaining data items of the two or more data items;
retrieve, from a data storage different from the prefetch data storage, a second group of one or more data items of the two or more data items, identified as not being stored in the prefetch data storage; and
send the second group of data items to the data processing stage.
2 . The integrated circuit as recited in claim 1 , wherein each of the first group of data items is stored in a non-contiguous manner.
3 . The integrated circuit as recited in claim 1 , wherein the prefetch engine is further configured to reorder a priority of data processing of a list of a plurality of data item identifiers specifying data items targeted by a given user query, wherein reordering is based on access statistics of data items currently stored in the prefetch data storage.
4 . The integrated circuit as recited in claim 1 , wherein the prefetch engine is further configured to replace a first data item currently stored in the prefetch data storage with a second data item not stored in the prefetch data storage based on determining that a priority of the second data item is greater than a priority of the first data item.
5 . The integrated circuit as recited in claim 4 , wherein a priority of a given data item is based on a frequency of reuse of the given data item by a plurality of user queries.
6 . The integrated circuit as recited in claim 4 , wherein a priority of a given data item is based on an access latency of a corresponding one of a data storage area that stores the given data item.
7 . The integrated circuit as recited in claim 1 , wherein:
data storage areas targeted by the user query are embedding tables configured to store embedding vectors used by a data model implementing a neural network; and
each data item is an embedding vector.
8 . A method comprising:
responsive to receiving a user query that targets two or more data items:
retrieving, from a prefetch data storage, a first group of one or more data items of the two or more data items, identified as being stored in the prefetch data storage;
sending the first group of data items to a data processing stage comprising processing circuitry, prior to sending remaining data items of the two or more data items;
retrieving, from a data storage different than the prefetch data storage, a second group of one or more data items of the two or more data items, identified as not being stored in the prefetch data storage; and
sending the second group of data items to the data processing stage.
9 . The method as recited in claim 8 , wherein each of the first group of data items is stored in a non-contiguous manner.
10 . The method as recited in claim 8 , further comprising reordering a priority of data processing of a list of a plurality of data item identifiers specifying data items targeted by a given user query, wherein reordering is based on access statistics of data items currently stored in the prefetch data storage.
11 . The method as recited in claim 8 , further comprising replacing a first data item currently stored in the prefetch data storage with a second data item not stored in the prefetch data storage based on determining that a priority of the second data item is greater than a priority of the first data item.
12 . The method as recited in claim 11 , further comprising assigning a priority of a given data item based on a frequency of reuse of the given data item by a plurality of user queries.
13 . The method as recited in claim 11 , further comprising assigning a priority of a given data item based on an access latency of a data storage area that stores the given data item.
14 . The method as recited in claim 8 , wherein:
data storage areas targeted by the user query are embedding tables configured to store embedding vectors used by a data model implementing a neural network; and
each data item is an embedding vector.
15 . A computing system comprising:
a memory configured to store instructions of one or more tasks and source data to be processed by the one or more tasks;
an integrated circuit configured to execute the instructions using the source data, wherein the integrated circuit comprises:
a prefetch engine comprising circuitry, wherein responsive to receipt of a user query that targets two or more data items, the circuitry is configured to:
retrieve, from a prefetch data storage, a first group of one or more data items of the two or more data items, identified as being stored in the prefetch data storage;
send the first group of data items to a data processing stage comprising processing circuitry, prior to sending remaining data items of the two or more data items;
retrieve, from a data storage different than the prefetch data storage, a second group of one or more data items of the two or more data items, identified as not being stored in the prefetch data storage; and
send the second group of data items to the data processing stage.
16 . The computing system as recited in claim 15 , wherein each of the first group of data items is stored in a non-contiguous manner.
17 . The computing system as recited in claim 15 , wherein the prefetch engine is further configured to reorder a priority of data processing of a list of a plurality of data item identifiers specifying data items targeted by a given user query, wherein reordering is based on access statistics of data items currently stored in the prefetch data storage.
18 . The computing system as recited in claim 15 , wherein the prefetch engine is further configured to replace a first data item currently stored in the prefetch data storage with a second data item not stored in the prefetch data storage based on determining that a priority of the second data item is greater than a priority of the first data item.
19 . The computing system as recited in claim 18 , wherein a priority of a given data item is based on an access latency of a data storage area that stores the given data item.
20 . The computing system as recited in claim 15 , wherein:
data storage areas targeted by the user query are embedding tables configured to store embedding vectors used by a data model implementing a neural network; and
each data item is an embedding vector.