Data query apparatus, method, and storage medium
A database system that, after receiving the query request, adopts a pre-trained deep learning model to predict probing cardinality corresponding to the query request based on the query vector in the query request, the number of vectors of the query result, and the vector corresponding to the distances between the query vector and the center vector of a plurality of data partitions stored in the storage device. The system then determines the target data partitions according to the probing cardinality, and obtains the query result from the target data partitions. Since the probing cardinality is dynamically determined based on the query request, decreases to both the query efficiency due to a larger setting of the probing cardinality, and the query recall due to a smaller setting of the probing cardinality are avoided, which is beneficial in reducing the average number of target data partitions and improving query efficiency.
1. An apparatus comprising:
a memory storing instructions; and
a processor configured to execute the instructions to:
receive a query request for a storage device storing vector data, wherein:
the vector data is divided into a plurality of data partitions, each of the data partitions includes a center vector, and
the query request comprises a query vector and a result number;
predict, via a pre-trained deep learning model, a number of the plurality of data partitions to be queried based on the query vector, the result number, and a vector corresponding to one or more distances between the query vector and the center vector of each of the data partitions,
determine from the plurality of data partitions at least one target data partition having a corresponding center vector that is least distant from the query vector, wherein the number of the at least one target data partitions is the same as the number of data partitions to be queried,
determine a query result corresponding to the query request from the at least one target data partition,
determine an intermediate query result, wherein a number of result vectors included in the intermediate query result is the same as the result number;
determine first type vectors of each at least one target data partition, wherein:
an absolute value of difference between a distance between a corresponding first type vector and a corresponding center vector of a corresponding data partition of the at least one target data partition storing the corresponding first type vector and a first distance is smaller than the distance between the corresponding first type vector and the query vector,
the first type vectors are a vector type having a largest distance from the query vector in the intermediate query result, and
the first distance is the distance between the center vector of the corresponding data partition of the at least one target data partition storing the corresponding first type vector and the query vector;
determine distances between the query vector and the first vector types;
update the intermediate query result according to the distances between the query vector and the first type vectors; and
take the intermediate query result as the query result of the query request after at least one target data partition are queried.
2. The apparatus of claim 1 , wherein the processor is further configured to:
determine, via a first coding network, a first embedded feature of the query vector based on coding a plurality of sub-vectors of the query vector;
determine, via a second encoding network, a second embedded feature of the result number;
determine, via a third coding network, a third embedded feature of the vector corresponding to the distances between the query vector and the center vector of each of the data partitions; and
predict, via a decoding network, the number of data partitions based on the first embedded feature, the second embedded feature, and the third embedded feature.
3. The apparatus of claim 2 , wherein:
the pre-trained deep learning model is trained based at least partially on the following loss function:
L
=
1
m
[
∑
j
:
p
-
p
j
≥
p
-
s
j
γ
(
p
-
s
j
-
p
-
p
j
)
2
+
∑
j
:
p
-
p
j
<
p
-
s
j
(
1
-
γ
)
(
p
-
s
j
-
p
-
p
j
)
2
]
,
wherein L is a loss function, m is a number of query requests, p-p j is the number of data partitions to be queried corresponding to the j-th query request predicted via the pre-trained deep learning model, p-s j is the number of query data partitions to be queried with reference to the j-th query request, and γ is a super parameter smaller than 0.5.
4. The apparatus of claim 1 , wherein:
the vector data is partitioned into the plurality of data partitions by a hierarchical clustering method.
5. The apparatus of claim 1 , wherein:
the vector data in each of the plurality of data partitions are arranged in ascending order of distances between the vector data and the center vector of a corresponding data partition.
6. The apparatus of claim 1 , wherein the processor is further configured to:
update a second type vector into the intermediate query result and delete the first type vector from the intermediate query result when the distance between the second type vector and the query vector is smaller than the distance between the first type vector and the query vector.
7. The apparatus of claim 1 , wherein:
the distance between the query vector and each of the first type vectors is an asymmetric distance.
8. The apparatus of claim 7 , wherein the processor is further configured to determine the distance between the query vector and each of the first type vectors by:
sequentially accumulating distances between M sub-vectors of the query vector and sub-vectors corresponding to a third type vector to obtain a distance between the query vector and the third vector, and
terminating the accumulating if a value after accumulating the distance between the Nth sub-vector of the query vector and the corresponding sub-vector is greater than the distance between the first type vector and the query vector, wherein M>1, 1≤N<M.
9. The apparatus of claim 1 , wherein the processor is a component of a one or more query devices.
10. The apparatus of claim 9 , wherein the one or more query devices communicate with one or more storage devices via network.
11. A method executed by one or more processors, the method comprising:
receiving a query request for one or more storage device storing vector data, wherein:
the vector data is divided into a plurality of data partitions, each of the plurality of data partitions including a center vector, and
the query request comprises a query vector and a result number;
predicting, via a pre-trained deep learning model, a number of the plurality of data partitions to be queried based on the query vector, the result number, and a vector corresponding to one or more distances between the query vector and the center vector of each of the plurality of data partitions;
determining from the plurality of data partitions at least one target data partition whose corresponding center vector is least distant from a corresponding query vector, wherein a number of target data partitions is the same as the predicted number of data partitions to be queried; and
determining the query result corresponding to the query request from the at least one target data partition, wherein further comprises:
determining an intermediate query result, wherein a number of result vectors included in the intermediate query result is the same as the result number;
determining first type vectors of each at least one target data partition, wherein:
an absolute value of difference between a distance between a corresponding first type vector and a corresponding center vector of a corresponding data partition of the at least one target data partition storing the corresponding first type vector and a first distance is smaller than the distance between the corresponding first type vector and the query vector,
the first type vectors are a vector type having a largest distance from the query vector in the intermediate query result, and
the first distance is the distance between the center vector of the corresponding data partition of the at least one target data partition storing the corresponding first type vector and the query vector;
determining distances between the query vector and the first vector types;
updating the intermediate query result according to the distances between the query vector and the first type vectors; and
taking the intermediate query result as the query result of the query request after at least one target data partition are queried.
12. The method of claim 11 , wherein predicting, via a pre-trained deep learning model, the number of the plurality of data partitions, comprises:
determining, via a first coding network, a first embedded feature of the query vector based on coding a plurality of sub-vectors of the query vector;
determining, via a second encoding network, a second embedded feature of the result number;
determining, via a third coding network, a third embedded feature of the vector corresponding to the distances between the query vector and the center vector of each of the data partitions; and
predicting, via a decoding network, the number of data partitions based on the first embedded feature, the second embedded feature, and the third embedded feature.
13. The method of claim 11 , wherein updating the intermediate query result comprises:
updating a second type vector into the intermediate query result and deleting the first type vector from the intermediate query result when the distance between the second type vector and the query vector is smaller than the distance between the first type vector and the query vector.
14. The method of claim 13 wherein:
the distance between the query vector and each of the first type vectors is an asymmetric distance; and the method further comprises:
sequentially accumulating distances between M sub-vectors of the query vector and sub-vectors corresponding to a third type vector to obtain a distance between the query vector and the third vector, and
terminating the accumulating if a value after accumulating the distance between the Nth sub-vector of the query vector and the corresponding sub-vector is greater than the distance between the first type vector and the query vector, wherein M>1, 1≤N<M.
15. A non-transitory computer-readable storage medium storing instructions, wherein the instructions, when executed on a machine, cause the machine to:
receive a query request for one or more storage device storing vector data, wherein the query request comprises a query vector and a result number;
predict, via a pre-trained deep learning model, a number of the plurality of data partitions to be queried based on the query vector, the result number, and a vector corresponding to one or more distances between the query vector and the center vector of each of the plurality of data partitions;
determine from the plurality of data partitions at least one target data partition whose corresponding center vector is least distant from a corresponding query vector, wherein a number of target data partitions is the same as the predicted number of data partitions to be queried;
determine the query result corresponding to the query request from the at least one target data partition;
determine an intermediate query result, wherein a number of result vectors included in the intermediate query result is the same as the result number;
determine first type vectors of each at least one target data partition, wherein:
an absolute value of difference between a distance between a corresponding first type vector and a corresponding center vector of a corresponding data partition of the at least one target data partition storing the corresponding first type vector and a first distance is smaller than the distance between the corresponding first type vector and the query vector,
the first type vectors are a vector type having a largest distance from the query vector in the intermediate query result, and
the first distance is the distance between the center vector of the corresponding data partition of the at least one target data partition storing the corresponding first type vector and the query vector;
determine distances between the query vector and the first vector types;
update the intermediate query result according to the distances between the query vector and the first type vectors; and
take the intermediate query result as the query result of the query request after at least one target data partition are queried.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the instructions cause the machine to:
determine, via a first coding network, a first embedded feature of the query vector based on coding a plurality of sub-vectors of the query vector;
determine, via a second encoding network, a second embedded feature of the result number;
determine, via a third coding network, a third embedded feature of the vector corresponding to the distances between the query vector and the center vector of each of the data partitions; and
predict, via a decoding network, the number of data partitions based on the first embedded feature, the second embedded feature, and the third embedded feature.
17. The non-transitory computer-readable storage medium of claim 15 , wherein the instructions cause the machine to:
update a second type vector into the intermediate query result and delete the first type vector from the intermediate query result when the distance between the second type vector and the query vector is smaller than the distance between the first type vector and the query vector.