Vector dataset index parameter determination
Vector dataset index parameter determination is performed by building an index of a vector dataset according to each build parameter group among a plurality of build parameter groups, generating search parameter groups, submitting each vector query in a vector query set to each index to obtain, for each vector query, a result content and a result delay value, wherein an accuracy value is determined based on the result content, correlating build parameter values and search parameter values with accuracy values and result delay values, fixing build parameter values based on the correlating, and determining a mathematical relationship expressing a correlation of the search depth parameter values, the numbers of requested results, and the accuracy values obtained in response to vector queries submitted to the index having the fixed build parameter values.
1. A non-transitory computer-readable medium including instructions executable by a processor to cause the processor to perform operations comprising:
building an index of a vector dataset according to each build parameter group among a plurality of build parameter groups, each build parameter group including a plurality of candidate build parameter values;
generating search parameter groups, each search parameter group including a plurality of candidate search parameter values, wherein the plurality of candidate search parameter values includes a search depth parameter value and a number of requested results;
submitting each vector query in a vector query set to each index according to the candidate search parameter values from each search parameter group to obtain, for each vector query, a result content and a result delay value, wherein an accuracy value is determined based on the result content;
correlating build parameter values and search parameter values with accuracy values and result delay values;
fixing build parameter values based on the correlating; and
determining a mathematical relationship expressing a correlation of the search depth parameter values, the numbers of requested results, and the accuracy values obtained in response to vector queries submitted to the index having the fixed build parameter values.
2. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise:
generating, after the fixing, additional search parameter groups based on the fixed build parameter values.
3. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise:
recording, for each submission, the vector query and the search parameter values along with a result content and a result delay.
4. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise:
sampling vectors among a vector dataset for use as the vector query set.
5. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise:
determining the accuracy value based on ground truth.
6. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise:
deploying the index for the vector dataset built according to the fixed build parameter values.
7. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise:
determining a search depth parameter value based on a target accuracy and a target number of results based on the mathematical relationship;
performing a query, according to the determined search depth parameter value, with the index having the fixed build parameter values; and
transmitting a result content of the query.
8. The non-transitory computer-readable medium of claim 1 , wherein the operations further comprise:
determining the plurality of candidate build parameter values of each build parameter group based on an index type.
9. A method comprising:
building an index of a vector dataset according to each build parameter group among a plurality of build parameter groups, each build parameter group including a plurality of candidate build parameter values;
generating search parameter groups, each search parameter group including a plurality of candidate search parameter values, wherein the plurality of candidate search parameter values includes a search depth parameter value and a number of requested results;
submitting each vector query in a vector query set to each index according to the candidate search parameter values from each search parameter group to obtain, for each vector query, a result content and a result delay value, wherein an accuracy value is determined based on the result content;
correlating build parameter values and search parameter values with accuracy values and result delay values;
fixing build parameter values based on the correlating; and
determining a mathematical relationship expressing a correlation of the search depth parameter values, the numbers of requested results, and the accuracy values obtained in response to vector queries submitted to the index having the fixed build parameter values.
10. The method of claim 9 , further comprising:
generating, after the fixing, additional search parameter groups based on the fixed build parameter values.
11. The method of claim 9 , further comprising:
recording, for each submission, the vector query and the search parameter values along with a result content and a result delay.
12. The method of claim 9 , further comprising:
sampling vectors among a vector dataset for use as the vector query set.
13. The method of claim 9 , further comprising:
determining the accuracy value based on ground truth.
14. The method of claim 9 , further comprising:
deploying the index for the vector dataset built according to the fixed build parameter values.
15. The method of claim 9 , further comprising:
determining a search depth parameter value based on a target accuracy and a target number of results based on the mathematical relationship;
performing a query, according to the determined search depth parameter value, with the index having the fixed build parameter values; and
transmitting a result content of the query.
16. The method of claim 9 , further comprising:
determining the plurality of candidate build parameter values of each build parameter group based on an index type.
17. An apparatus comprising:
a controller including circuitry configured to:
build an index of a vector dataset according to each build parameter group among a plurality of build parameter groups, each build parameter group including a plurality of candidate build parameter values;
generate search parameter groups, each search parameter group including a plurality of candidate search parameter values, wherein the plurality of candidate search parameter values includes a search depth parameter value and a number of requested results;
submit each vector query in a vector query set to each index according to the candidate search parameter values from each search parameter group to obtain, for each vector query, a result content and a result delay value, wherein an accuracy value is determined based on the result content;
correlate build parameter values and search parameter values with accuracy values and result delay values;
fix build parameter values based on the correlating; and
determine a mathematical relationship expressing a correlation of the search depth parameter values, the numbers of requested results, and the accuracy values obtained in response to vector queries submitted to the index having the fixed build parameter values.
18. The apparatus of claim 17 , wherein the circuitry is further configured to:
generate, after the fixing, additional search parameter groups based on the fixed build parameter values.
19. The apparatus of claim 17 , wherein the circuitry is further configured to:
record, for each submission, the vector query and the search parameter values along with a result content and a result delay.
20. The apparatus of claim 17 , wherein the circuitry is further configured to:
sample vectors among a vector dataset for use as the vector query set.