Method, electronic device, and computer program product for data indexing
Embodiments of the present disclosure provide a method, an electronic device, and a computer program product for data indexing. The method for data indexing includes obtaining a plurality of data blocks for storage. The method further includes generating a plurality of vectorized representations of the plurality of data blocks so that a difference between vectorized representations of data blocks with a higher similarity among the plurality of data blocks is smaller. The method further includes storing the plurality of vectorized representations for indexing the plurality of data blocks respectively. The vectorized representations serve as indexes of the data blocks, and the vectorized representations may be used to effectively measure similarities between the data blocks, thereby providing the possibility of performing a fuzzy search of the data blocks based on the similarities.
1. A method for data indexing, comprising:
obtaining a plurality of data blocks for storage;
generating a plurality of vectorized representations of the plurality of data blocks using a representation generation model so that a difference between vectorized representations of data blocks with a higher similarity among the plurality of data blocks is smaller, wherein the representation generation model is trained based on a training triplet, the training triplet comprising a first training data block, a second training data block similar to the first training data block, and a third training data block dissimilar to the first training data block; and
storing the plurality of vectorized representations for indexing the plurality of data blocks respectively.
2. The method according to claim 1 , wherein
the representation generation model is a neural network model.
3. The method according to claim 1 , wherein
a training target of the representation generation model is to make a difference between vectorized representations generated by the representation generation model for the first training data block and the second training data block less than a difference between vectorized representations generated for the first training data block and the third training data block.
4. The method according to claim 3 , wherein the first training data block and the third training data block are selected from a data block set, and the second training data block is generated by modifying a part of the first training data block.
5. The method according to claim 1 , wherein the plurality of vectorized representations are represented by numerical vectors of a same number of dimensions.
6. The method according to claim 1 , wherein storing the plurality of vectorized representations comprises:
selecting a data structure suitable for a neighbor search; and
storing the plurality of vectorized representations according to the selected data structure.
7. The method according to claim 1 , further comprising:
obtaining a search associated with a search data block;
generating a search vectorized representation of the search data block;
searching the plurality of vectorized representations for at least one vectorized representation with a difference between the at least one vectorized representation and the search vectorized representation that is less than a difference threshold; and
determining a response to the search based on at least one data block indexed by the at least one vectorized representation among the plurality of data blocks.
8. The method according to claim 7 , wherein generating the search vectorized representation of the search data block comprises:
generating the search vectorized representation of the search data block using the representation generation model.
9. An electronic device, comprising:
at least one processor; and
at least one memory storing computer program instructions, the at least one memory and the computer program instructions being configured to cause, with the at least one processor, the electronic device to perform actions comprising:
obtaining a plurality of data blocks for storage;
generating a plurality of vectorized representations of the plurality of data blocks using a representation generation model so that a difference between vectorized representations of data blocks with a higher similarity among the plurality of data blocks is smaller, wherein the representation generation model is trained based on a training triplet, the training triplet comprising a first training data block, a second training data block similar to the first training data block, and a third training data block dissimilar to the first training data block; and
storing the plurality of vectorized representations for indexing the plurality of data blocks respectively.
10. The device according to claim 9 , wherein
the representation generation model is a neural network model.
11. The device according to claim 10 , wherein
a training target of the representation generation model is to make a difference between vectorized representations generated by the representation generation model for the first training data block and the second training data block less than a difference between vectorized representations generated for the first training data block and the third training data block.
12. The device according to claim 11 , wherein the first training data block and the third training data block are selected from a data block set, and the second training data block is generated by modifying a part of the first training data block.
13. The device according to claim 9 , wherein the plurality of vectorized representations are represented by numerical vectors of a same number of dimensions.
14. The device according to claim 9 , wherein storing the plurality of vectorized representations comprises:
selecting a data structure suitable for a neighbor search; and
storing the plurality of vectorized representations according to the selected data structure.
15. The device according to claim 9 , wherein the actions further comprise:
obtaining a search associated with a search data block;
generating a search vectorized representation of the search data block;
searching the plurality of vectorized representations for at least one vectorized representation with a difference between the at least one vectorized representation and the search vectorized representation that is less than a difference threshold; and
determining a response to the search based on at least one data block indexed by the at least one vectorized representation among the plurality of data blocks.
16. The device according to claim 15 , wherein generating the search vectorized representation of the search data block comprises:
generating the search vectorized representation of the search data block using the representation generation model.
17. A computer program product tangibly stored on a non-volatile computer-readable medium and comprising computer-executable instructions, wherein when executed, the computer-executable instructions cause a device to perform actions comprising:
obtaining a plurality of data blocks for storage;
generating a plurality of vectorized representations of the plurality of data blocks using a representation generation model so that a difference between vectorized representations of data blocks with a higher similarity among the plurality of data blocks is smaller, wherein the representation generation model is trained based on a training triplet, the training triplet comprising a first training data block, a second training data block similar to the first training data block, and a third training data block dissimilar to the first training data block; and
storing the plurality of vectorized representations for indexing the plurality of data blocks respectively.
18. The computer program product according to claim 17 , wherein the representation generation model is a neural network model.
19. The computer program product according to claim 18 , wherein
a training target of the representation generation model is to make a difference between vectorized representations generated by the representation generation model for the first training data block and the second training data block less than a difference between vectorized representations generated for the first training data block and the third training data block.
20. The computer program product according to claim 19 , wherein the first training data block and the third training data block are selected from a data block set, and the second training data block is generated by modifying a part of the first training data block.