System and method for identifying approximate k-nearest neighbors in web scale clustering
The present teaching relates to method, system, medium, and implementations for identifying k nearest neighbors. One or more KNN lists corresponding to one or more source data points are received. Each KNN list includes K neighbors of a source data point and each of the K neighbors is a data point represented by an index. Neighbor pairs and reverse neighbor pairs are generated based on the one or more KNN lists. The neighbor pairs and reverse neighbor pairs having the same source data point are grouped to generate a grouped pairs of neighbors for the source data point. A local join operation is performed based on grouped pairs of neighbors for each source data point to generate a combined neighborhood for the source data point, which is then sent to a KNN server, where combined neighborhoods generated by multiple local join executors are integrated to update a plurality of global KNN lists.
1. A method implemented on at least one machine including at least one processor, memory, and communication platform capable of connecting to a network for identifying k-nearest neighbors (KNN), the method comprising:
receiving, by a local join executor, one or more KNN lists corresponding to one or more source data points, each of the KNN lists including K neighbors of a source data point, each of the K neighbors is a data point represented by an index pointing thereto;
generating neighbor pairs and reverse neighbor pairs based on the one or more KNN lists;
grouping neighbor pairs and reverse neighbor pairs having a same source data point to generate a grouped pairs of neighbors for the source data point;
performing a local join operation based on grouped pairs of neighbors for each of the one or more source data points to generate a combined neighborhood for the source data point; and
transmitting the combined neighborhood of each of the one or more source data points to a KNN server, where combined neighborhoods generated by a plurality of local join executors are integrated to update a plurality of global KNN lists.
2. The method of claim 1 , wherein the step of generating neighbor pairs and reverse neighbor pairs comprises, with respect to each of the one or more KNN lists for a source data point,
obtaining K neighbor pairs, each of which includes the source data point and one of the neighbors in the KNN list;
obtaining reverse neighbor pairs, each of which is obtained by reversing a corresponding one of the K neighbor pairs.
3. The method of claim 2 , further comprising sampling the K neighbor pairs and the reverse neighbor pairs based on some predetermined criterion to generate the neighbor pairs and reverse neighbor pairs.
4. The method of claim 1 , further comprising
receiving a maximum distance map (maxDistMap) having a plurality of maximum distance (maxDist) values, each of which relating to one of a plurality of KNN lists for a plurality of source data points, wherein a combined neighborhood for a source data point is derived based on a corresponding maxDist value in the maxDistMap.
5. The method of claim 4 , wherein a neighbor in the combined neighborhood for a source data point has a distance to the source data point not exceeding the maxDist value associated with a KNN List for the source data point.
6. The method of claim 4 , wherein a maxDist value relating to a KNN list for a source data point corresponds to a maximum distance between the source data point to a neighbor in the KNN list.
7. The method of claim 1 , wherein the steps of receiving, generating, grouping, performing, and transmitting are carried out in each iteration of an iterative process of identifying KNN and each iteration is activated upon receiving an iteration activation signal generated based on a count of changes occurred in updating the plurality of global KNN lists.
8. Machine readable and non-transitory medium having information recorded thereon for identifying k-nearest neighbors (KNN), wherein the information, once read by the machine, causes the machine to perform the following steps:
receiving one or more KNN lists corresponding to one or more source data points, each of the KNN lists including K neighbors of a source data point, each of the K neighbors is a data point represented by an index pointing thereto;
generating neighbor pairs and reverse neighbor pairs based on the one or more KNN lists;
grouping neighbor pairs and reverse neighbor pairs having a same source data point to generate a grouped pairs of neighbors for the source data point;
performing a local join operation based on grouped pairs of neighbors for each of the one or more source data points to generate a combined neighborhood for the source data point; and
transmitting the combined neighborhood of each of the one or more source data points to a KNN server, where combined neighborhoods generated by a plurality of local join executors are integrated to update a plurality of global KNN lists.
9. The medium of claim 8 , wherein the step of generating neighbor pairs and reverse neighbor pairs comprises, with respect to each of the one or more KNN lists for a source data point,
obtaining K neighbor pairs, each of which includes the source data point and one of the neighbors in the KNN list;
obtaining reverse neighbor pairs, each of which is obtained by reversing a corresponding one of the K neighbor pairs.
10. The medium of claim 9 , wherein the information, once read by the machine, further causes the machine to perform the step of sampling the K neighbor pairs and the reverse neighbor pairs based on some predetermined criterion to generate the neighbor pairs and reverse neighbor pairs.
11. The medium of claim 8 , wherein the information, once read by the machine, further causes the machine to perform the step of receiving a maximum distance map (maxDistMap) having a plurality of maximum distance (maxDist) values, each of which relating to one of a plurality of KNN lists for a plurality of source data points, wherein a combined neighborhood for a source data point is derived based on a corresponding maxDist value in the maxDistMap.
12. The medium of claim 11 , wherein a neighbor in the combined neighborhood for a source data point has a distance to the source data point not exceeding the maxDist value associated with a KNN List for the source data point.
13. The medium of claim 11 , wherein a maxDist value relating to a KNN list for a source data point corresponds to a maximum distance between the source data point to a neighbor in the KNN list.
14. The medium of claim 8 , wherein the steps of receiving, generating, grouping, performing, and transmitting are carried out in each iteration of an iterative process of identifying KNN and each iteration is activated upon receiving an iteration activation signal generated based on a count of changes occurred in updating the plurality of global KNN lists.
15. A system for identifying k-nearest neighbors (KNN), comprising:
a KNN list receiver implemented by a processor and configured for receiving one or more KNN lists corresponding to one or more source data points, each of the KNN lists including K neighbors of a source data point, each of the K neighbors is a data point represented by an index pointing thereto;
a neighbor pair identifier implemented by a processor and configured for generating neighbor pairs and reverse neighbor pairs based on the one or more KNN lists;
a grouping unit implemented by a processor and configured for grouping neighbor pairs and reverse neighbor pairs having a same source data point to generate a grouped pairs of neighbors for the source data point; and
a local join unit implemented by a processor and configured for
performing a local join operation based on grouped pairs of neighbors for each of the one or more source data points to generate a combined neighborhood for the source data point, and
transmitting the combined neighborhood of each of the one or more source data points to a KNN server, where combined neighborhoods generated by a plurality of local join executors are integrated to update a plurality of global KNN lists.
16. The system of claim 15 , wherein the neighbor pair identifier generates neighbor pairs and reverse neighbor pairs by, with respect to each of the one or more KNN lists for a source data point,
obtaining K neighbor pairs, each of which includes the source data point and one of the neighbors in the KNN list;
obtaining reverse neighbor pairs, each of which is obtained by reversing a corresponding one of the K neighbor pairs.
17. The system of claim 16 , further comprising a sampling unit configured for sampling the K neighbor pairs and the reverse neighbor pairs based on some predetermined criterion to generate the neighbor pairs and reverse neighbor pairs.
18. The system of claim 15 , further comprising a maxDistMap receiver configured for receiving a maximum distance map (maxDistMap) having a plurality of maximum distance (maxDist) values, each of which relating to one of a plurality of KNN lists for a plurality of source data points, wherein a combined neighborhood for a source data point is derived based on a corresponding maxDist value in the maxDistMap, wherein a neighbor in the combined neighborhood for a source data point has a distance to the source data point not exceeding the maxDist value associated with a KNN List for the source data point.
19. The system of claim 18 , wherein a maxDist value relating to a KNN list for a source data point corresponds to a maximum distance between the source data point to a neighbor in the KNN list.
20. The system of claim 15 , wherein the steps of receiving, generating, grouping, performing, and transmitting are carried out in each iteration of an iterative process of identifying KNN and each iteration is activated upon receiving an iteration activation signal generated based on a count of changes occurred in updating the plurality of global KNN lists.