Systems and methods for grouping embedding vectors in a storage device for improved read latency
A method of grouping embedding vectors in a storage device includes receiving, at a storage device, an inference query request including a first access index associated with a first embedding vector, determining, with a classification model, a first classification of the first access index, based on the first classification, searching for the first embedding vector in a cache of the storage device, receiving, at the storage device, a second access index associated with a second embedding vector, determining, with the classification model, a second classification of the second access index, and based on the second classification, searching for the second embedding vector in a persistent storage of the storage device.
1 . A method, comprising:
receiving, at a storage device, an inference query request comprising a first access index to locate a first embedding vector, the first embedding vector representing an object for processing by a machine-learning (ML) model;
determining, from a first output of a classification model, a first location of the first embedding vector;
searching for the first embedding vector in the first location;
receiving, at the storage device, a second access index to locate a second embedding vector;
determining, from a second output of the classification model, a second location of the second embedding vector; and
searching for the second embedding vector in the second location.
2 . The method of claim 1 , wherein:
the first output indicates a first access frequency; and
the second output indicates a second access frequency that is less than the first access frequency.
3 . The method of claim 1 , wherein:
the determining, from the first output of the classification model, comprises comparing an access frequency associated with the first access index with a reference access frequency; and
the determining, from the second output of the classification model, comprises comparing an access frequency associated with the second access index with the reference access frequency.
4 . The method of claim 1 , wherein the classification model comprises a machine-learning (ML) grouping model.
5 . The method of claim 4 , further comprising generating, by the machine-learning (ML) grouping model, the first output based on the first access index.
6 . The method of claim 1 , wherein the storage device is configured to communicate with a host via a cache coherent protocol.
7 . The method of claim 1 , wherein the second embedding vector is prevented from being stored in the first location of the storage device based on a reference access frequency.
8 . The method of claim 1 , wherein:
the first location of the first embedding vector comprises a cache; and
the second location of the second embedding vector comprises a persistent storage.
9 . A hardware storage device, comprising:
a first location; and
a second location,
wherein, the hardware storage device is configured to:
receive an inference query request comprising a first access index to locate a first embedding vector, the first embedding vector representing an object for processing by a machine-learning (ML) model;
determine, from a first output of a classification model, to search the first location for the first embedding vector;
search for the first embedding vector in the first location;
receive a second access index to locate a second embedding vector;
determine, from a second output of the classification model, to search the second location for the second embedding vector; and
search for the second embedding vector in the second location.
10 . The hardware storage device of claim 9 , wherein:
the first output indicates a first access frequency; and
the second output indicates a second access frequency that is less than the first access frequency.
11 . The hardware storage device of claim 9 , wherein:
the determining, from the first output of the classification model, comprises comparing an access frequency associated with the first access index with a reference access frequency; and
the determining, from the second output of the classification model, comprises comparing an access frequency associated with the second access index with the reference access frequency.
12 . The hardware storage device of claim 9 , wherein the classification model comprises a machine-learning (ML) grouping model.
13 . The hardware storage device of claim 12 , wherein the hardware storage device is configured to generate the first output from the machine-learning (ML) grouping model based on the first access index.
14 . The hardware storage device of claim 9 , wherein the second embedding vector is prevented from being stored in the first location based on a reference access frequency.
15 . A system, comprising:
a processor; and
a memory storing instructions, which, when executed by the processor, cause the processor to perform:
receiving an inference query request comprising a first access index to locate a first embedding vector, the first embedding vector representing an object for processing by a machine-learning (ML) model;
determining, from a first output of a classification model, a first location of the first embedding vector;
searching for the first embedding vector in the first location;
receiving a second access index to locate a second embedding vector;
determining, from a second output of the classification model, a second location of the second embedding vector; and
searching for the second embedding vector in the second location.
16 . The system of claim 15 , wherein:
the first output indicates a first access frequency; and
the second output indicates a second access frequency that is less than the first access frequency.
17 . The system of claim 15 , wherein:
the determining, from the first output of the classification model, comprises comparing an access frequency associated with the first access index with a reference access frequency; and
the determining, from the second output of the classification model, comprises comparing an access frequency associated with the second access index with the reference access frequency.
18 . The system of claim 15 , wherein the classification model comprises a machine-learning (ML) grouping model.
19 . The system of claim 18 , wherein the machine-learning (ML) grouping model generates the first output based on the first access index.
20 . The system of claim 15 , wherein the second embedding vector is prevented from being stored in the first location based on a reference access frequency.