IP Library › Granted Patent US 12,675,521
Granted Patent B2
US 12,675,521 · App. 17/804,992 · Granted Jul 7, 2026

Composite code sparse autoencoders for approximate neighbor search

Inventors: Carlos Lassance (Grenoble, FR); Stéphane Clinchant (Meylan, FR); Thibault Formal (Grenoble, FR)
Assignee: NAVER CORPORATION
G06F16/41G06F16/435G06N3/0455G06N3/048
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,675,521
App. No.
17/804,992
Granted
Jul 7, 2026
Kind
B2
Abstract

Information retrieval methods employ a neural network encoder configured to receive a dense representation and generate a composite code comprising C clusters of dimension L from the dense representation. An activation function is configured to generate a sparse composite code from the composite code. The sparse composite code comprises a binary representation. An index can be generated using the sparse composite code.

Claims (102)

1 . A method for creating an index for information retrieval, the method being implemented by a processor and memory, the method comprising:

(a) receiving a set of input data, wherein each element of data in the set is a dense representation;

(b) generating, using a trained encoder implemented using the processor and including one or more layers, a composite code from the dense representation, the composite code comprising a vector of dimension D decomposed into C clusters of dimension L, where C and L are at least two;

(c) using an activation function to generate a sparse representation from the composite code, the sparse representation comprising a binary representation comprising C clusters of dimension L, where each cluster sums to one;

(d) repeating steps (a)-(c) for each element of data in the set of input data;

(e) creating, using the processor, an index based on each of the sparse representations generated at step (c) corresponding to each element of data in the set of input data; and

(f) outputting the index for information retrieval.

2 . The method of claim 1 , wherein each of the dense representations represent one or more of text data, image data and audio data;

wherein in the composite code of each dense representation data elements are represented by values of each of L subconcepts or classifications for each of C concepts.

3 . The method of claim 1 , wherein each of the dense representations represent text data and the dense representation is generated using a probabilistic model; wherein the probabilistic model comprises a Transformer model.

4 . The method of claim 1 , wherein the sparse representation is a sparse composite code where each of the C clusters forming part of the composite code has exactly one non-zero binary representation.

5 . The method of claim 1 , wherein the activation function is applied using an activation layer in a neural network.

6 . The method of claim 5 , wherein the activation layer comprises a softmax layer.

7 . The method of claim 1 , wherein information retrieval is one of text retrieval, image retrieval, and voice retrieval.

8 . The method of claim 1 , further comprising:

generating one or more posting lists from the sparse representation, wherein the posting list associated with a dimension comprises a list of documents having a non-zero binary representation in that dimension in the sparse composite code.

9 . A method for creating an index for information retrieval, the method being implemented by a processor and memory, the method comprising:

(a) receiving a set of input data, wherein each element of data in the set is a dense representation;

(b) generating, using a trained encoder implemented using the processor and including one or more layers, a composite code from the dense representation, the composite code comprising a vector of dimension D decomposed into C clusters of dimension L, where C and L are at least two;

(c) using an activation function to generate a sparse binary representation from the composite code, the sparse representation comprising C clusters of dimension L;

(d) repeating steps (a)-(c) for each element of data in the set of input data;

(e) creating, using the processor, an index based on each of the sparse representations generated at step (c) corresponding to each element of data in the set of input data; and

(f) outputting the index for information retrieval;

wherein the method further comprises:

regularizing the sparse binary representation to enforce load-balancing between different dimensions.

10 . The method of claim 9 , wherein said regularizing minimizes a loss based on a reconstruction loss combined with an error between an optimal mean activation of the sparse representation and a mean amount of activations of each dimension, weighted by a regularization factor;

wherein said regularizing is unsupervised.

11 . The method of claim 10 , wherein the error is approximated based on a batch of documents.

12 . An autoencoder for information retrieval implemented by a processor and a memory comprising:

a neural network encoder implemented using the processor, the neural network encoder being trained to receive a dense representation and generate a composite code comprising a vector of dimension D decomposed into C clusters of dimension L from the dense representation, where C and L are at least two, and

an activation function implemented using the processor, the activation function being configured to generate a sparse composite code from the composite code, the sparse composite code comprising a binary representation comprising C clusters of dimension L, where each cluster sums to one;

wherein the sparse composite code is used to generate, supplement, or search an index for information retrieval.

13 . The autoencoder of claim 12 further comprising:

a decoder implemented using the processor and configured to decode the sparse composite code into an original dense representation during training of the neural network encoder.

14 . The autoencoder of claim 12 , wherein the decoder comprises a linear decoder.

15 . The autoencoder of claim 12 , wherein the activation function comprises a Gumbel-softmax model.

16 . The autoencoder of claim 12 , wherein the encoder comprises a one-layer encoder.

17 . The autoencoder of claim 12 , further comprising:

a batch normalizer implemented using the processor and configured to reduce variance in the dense representation.

18 . An autoencoder for information retrieval implemented by a processor and a memory comprising:

a neural network encoder implemented using the processor, the neural network encoder being trained to receive a dense representation and generate a composite code from the dense representation, the composite code comprising a vector of dimension D decomposed into C clusters of dimension L, where C and L are at least two, and

an activation function implemented using the processor, the activation function being configured to generate a sparse composite code from the composite code, the sparse representation being a binary representation comprising C clusters of dimension L;

wherein the sparse composite code is used to generate, supplement, or search an index for information retrieval;

further comprising:

a neural regularizer implemented using the processor and configured to regularize the sparse representation to enforce load-balancing between different dimensions;

wherein in the composite code of each dense representation data elements are represented by values of each of Z subconcepts or classifications for each of C concepts.

19 . An information retrieval architecture implemented by one or more processors in combination with a memory, the architecture comprising:

an encoding phase implemented using the one or more processors and having one or more layers, the encoding phase being trained to receive a dense representation corresponding to a query, generate a composite code comprising a vector of dimension D decomposed into C clusters of dimension L from the dense representation and generate a sparse composite code from the composite code using an activation function, the sparse composite code comprising a binary representation comprising C clusters of dimension L, where each cluster sums to one, where C and L are at least two;

a scoring phase implemented using the one or more processors and configured to search an index based on the sparse composite code and output scores for a plurality of documents based on the index; and

a sorting phase implemented using the one or more processors and configured to rank a subset of the plurality of documents based on the scores and output one or more relevant documents from the ranked subset.

20 . The architecture of claim 19 , further comprising a probabilistic model implemented using the one or more processors and configured to:

receive the query;

encode the query into the dense representation; and

output the dense representation to the encoding phase.

21 . The architecture of claim 19 , further comprising:

a thresholding phase implemented using the one or more processors and configured to:

receive the scores from the scoring phase;

generate a subset of k documents from the scored documents based on a threshold; and

output the scores for the subset of k documents to the sorting phase;

wherein the sorting phase sorts the top-k documents based on the output scores.

22 . The architecture of claim 19 , wherein the scoring phase comprises an Approximate Nearest Neighbor (ANN) search model for information retrieval.

23 . The architecture of claim 19 , wherein the one or more processors of the encoding phase comprise at least one graphics processing unit (GPU) or central processing unit (CPU), and wherein the one or more processors of the scoring phase comprise at least one central processing unit (CPU).

24 . The architecture of claim 19 , wherein the architecture is further configured to determine one or more posting lists to score from the sparse composite code, and wherein the scoring phase searches the index based on the determined posting lists.

25 . The architecture of claim 19 , wherein the one or more processors scoring phase comprises a plurality of CPUs operating in parallel.

26 . A method for information retrieval implemented by one or more processors and a memory, the method comprising:

receiving a dense representation of a query;

generating, using a trained encoder implemented using the one or more processors and including one or more layers, a composite code comprising a vector of dimension D decomposed into C clusters of dimension L from the dense representation, where C and L are at least two;

generating a sparse composite code from the composite code model using an activation implemented using the one or more processors, the sparse composite code comprising a binary representation comprising C clusters of dimension L, where each cluster sums to one;

generating scores for a plurality of documents in an index based on the sparse composite code;

ranking a subset of the plurality of documents based on the scores; and

outputting the ranked subset.

27 . The method of claim 26 , wherein the received dense representation is generated by a probabilistic model processing the query.

28 . The method of claim 26 , wherein the query comprises a text query and the index represents one or more of text data, image data and audio data;

wherein in the composite code of each dense representation data elements are represented by values of each of L subconcepts or classifications for each of C concepts.

29 . The method of claim 26 , wherein the query comprises one or more of a search query, a request for a recommendation, a question, or a sequence to be translated.

30 . The method of claim 26 , wherein the index represents documents, and wherein the documents comprise one or more of text data, image data and audio data;

wherein in the composite code of each dense representation data elements are represented by values of each of L subconcepts or classifications for each of C concepts.

31 . The method of claim 26 , wherein the scoring comprises searching the index on the sparse composite code.

32 . The method of claim 26 , wherein the scoring comprises searching the index based on one or more posting lists determined using the composite code.

33 . The method of claim 26 , further comprising, creating the index by:

(a) receiving a set of input data, wherein each element of data in the set is a dense representation;

(b) generating a composite code from the dense representation using the trained encoder, the composite code comprising a vector of dimension D decomposed into C clusters of dimension L, where C and L are at least two;

(c) using an activation function implemented using the one or more processors to generate a sparse representation from the composite code, the sparse representation comprising a binary representation comprising C clusters of dimension L, where each cluster sums to one;

(d) repeating steps (a)-(c) for each element of data in the set of input data;

(e) creating an index based on each of the sparse representations generated at step (c) corresponding to each element of data in the set of input data; and

(f) outputting the index for information retrieval.

34 . An apparatus for transforming one or more dense representations corresponding to one or more documents into a sparse composite code for information retrieval comprising:

a non-transitory computer-readable medium having executable instructions stored thereon for causing a processor and a memory to:

for each of the one or more dense representations:

receive the dense representation;

generate, using a trained encoder including one or more layers, a composite code from the dense representation, the composite code comprising a vector of dimension D decomposed into C clusters of dimension L, where C and L are at least two; and

use an activation layer to generate the sparse composite code from the composite code, the sparse composite code comprising a binary representation comprising C clusters of dimension L, where each cluster sums to one; and

outputting the sparse composite codes for the one or more documents for generating, supplementing, or searching an index for information retrieval.

35 . The apparatus of claim 34 , wherein the processor and memory is further caused to:

generate or supplement an index using the one or more output sparse composite codes.

36 . The apparatus of claim 34 , wherein the processor and memory is further caused to:

search an index using the output sparse composite codes.

37 . The apparatus of claim 34 , wherein the dense representation is generated from a probabilistic model.

38 . The apparatus of claim 34 , wherein the one or more documents comprise one or more of text data, image data and audio data;

wherein in the composite code of each dense representation data elements are represented by values of each of L subconcepts or classifications for each of C concepts.

39 . The apparatus of claim 34 , wherein the executable instructions further cause a processor and a memory to:

regularize the sparse composite code to enforce load-balancing between different dimensions.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 7, 2023
From: LASSANCE, CARLOS; CLINCHANT, STÉPHANE; FORMAL, THIBAULT
To: NAVER CORPORATION
Reel/Frame 062618/0282 →
Continuity (2)
Provisional Application 63220300 · Jul 9, 2021
Related Publication 20230021996A1 · Jan 26, 2023
References Cited (77)
US 10832139B2 · Yan et al. · 2020 [cited by applicant]
US 10970629B1 · Dirac · 2021 [cited by examiner]
US 20180046900A1 · Dally · 2018 [cited by examiner]
US 20180285734A1 · Chen et al. · 2018 [cited by applicant]
US 20190370254A1 · Maxwell · 2019 [cited by examiner]
US 20210109958A1 · Behtash · 2021 [cited by examiner]
US 20210351791A1 · Lee · 2021 [cited by examiner]
US 20220114479A1 · Zhao · 2022 [cited by examiner]
US 20220253435A1 · Chen · 2022 [cited by examiner]
CN 104166684A · 2014 [cited by examiner]
WO WO2014123353A1 · 2014 [cited by examiner]
Ge, et al., Optimized Product Quantization for Approximate Nearest Neighbor Search. IEEE 2013. [online] [retrieved Sep. 28, 2025]. Retrieved from the Internet <URL:https://ieeexplore.ieee.org/document/6619223> (Year: 20… [cited by examiner]
Shu, R., et al., “Compressing Word Embeddings via Deep Compositional Code Learning,” International Conference on Learning 2018, Feb. 15, 2018, 13 pages. [cited by applicant]
Sobroza, M., et al., “Sparse associative memory based on contextual code learning for disambiguating word senses,” published on arXiv:1911.06415v1, Nov. 14, 2019, 6 pages. [cited by applicant]
Tu, Z., et al., “Approximate Nearest Neighbor Search and Lightweight Dense Vector Reranking in Multi-Stage Retrieval Architectures,” ICTIR '20, Sep. 14-17, 2020, Virtual Event, Norway, pp. 97-100. [cited by applicant]
Vorhees, E. et al., “NIST Special Publication: SP 500-331,” Proceedings of the Twenty-Eighth Text Retrieval Conference, TREC 2019, Gaithersburg, Maryland, USA, Nov. 13-15, 2019. NIST Special Publication, National Instit… [cited by applicant]
Xiong, L., et al., “Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Tetrieval,” published on arXiv:2007.00808v2, Oct. 20, 2020, 16 pages. [cited by applicant]
Xiong, L., et al., “Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval,” published on arXiv:2007.00808v1, Jul. 1, 2020, 15 pages. [cited by applicant]
Xiong, L., et al., “Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval,” International Conference on Learning Representations, 2021, 16 pages. [cited by applicant]
Yang, P., et al., “Anserini: Enabling the Use of Lucene for Information Retrieval Research,” Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Aug. 7-11, 20… [cited by applicant]
Zamani, H., et al., “From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing,” CIKM'18, Oct. 22-26, 2018, Torino, Italy, pp. 497-506. [cited by applicant]
Zhao, T., et al., “SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval,” published on arXiv:2009.13013v1, Sep. 28, 2020, 11 pages. [cited by applicant]
Amati, G., et al., “Probabilistic Models of Information Retrieval Based on Measuring the Divergence from Randomness,” ACM Transactions on Information Systems, vol. 20, No. 4, Oct. 2002, pp. 357-389. [cited by applicant]
Azad, H., et al., “Query Expansion Techniques for Information Retrieval: a Survey,” Information Processing & Management 56, No. 5, Sep. 2019, pp. 1698-1735, published on arXiv:1708.00247v2, Jun. 20, 2019. [cited by applicant]
Babenko, A., et al., “The Inverted Multi-Index,” IEEE Transactions on Pattern Analysis and Machine Intelligence 37, No. 6, Jun. 2015, pp. 1247-1260. [cited by applicant]
Bai, Y., et al., “SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval,” published on arXiv:2010.00768v1, Oct. 2, 2020, 8 pages. [cited by applicant]
Bajaj, P., et al., “MS MARCO: A Human Generated MAchine Reading Comprehension Dataset,” published on arXiv:1611.09268, Oct. 31, 2018, 11 pages. [cited by applicant]
Baranchuk, D., et al., “Learning to Route in Similarity Graphs,” published on arXiv:1905.10987v1, May 27, 2019, 10 pages. [cited by applicant]
Bengio, Y., et al., “Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation,” published on arXiv:1308.3432v1, Aug. 15, 2013, 12 pages. [cited by applicant]
Berger, A., et al., “Information Retrieval as Statistical Translation,” ACM SIGIR Forum 51, No. 2, Jul. 2017, pp. 219-226. [cited by applicant]
Boytsov, L., “Efficient and Accurate Non-Metric k-NN Search with Applications to Text Matching,” Ph.D. Dissertation, Carnegie Mellon University, School of Computer Science, CMU-LTI-18-006, 2018, 205 pages. [cited by applicant]
Boytsov, L., et al., “Flexible retrieval with NMSLIB and FlexNeuART,” published on arXiv:2010.14848v2, Nov. 17, 2020, 12 pages. [cited by applicant]
Chatfield, K., et al., “Return of the Devil in the Details: Delving Deep into Convolutional Nets,” published on arXiv:1405.3531v4, Nov. 5, 2014, 11 pages. [cited by applicant]
Craswell, N., et al., “Overview of the TREC 2019 Deep Learning Track,” published on arXiv:2003.07820v2, Mar. 18, 2020, 22 pages. [cited by applicant]
Dai, Z., et al., “Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval,” published on arXiv:1910.10687v2, Nov. 26, 2019, 9 pages. [cited by applicant]
Devlin, J., et al., “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” published on arXiv:1810.04805v1, Oct. 11, 2018, 14 pages. [cited by applicant]
Ge, T., et al., “Optimized Product Quantization,” IEEE Transactions on Pattern Analysis and Machine Intelligence 36, No. 4, Apr. 2014, pp. 744-755. [cited by applicant]
Guu, K., et al., “REALM: Retrieval-Augmented Language Model Pre-Training,” published on arXiv:2002.08909v1, Feb. 10, 2020, 12 pages. [cited by applicant]
Harris, C., et al., “Array programming with NumPy,” Nature, vol. 585, 6825, Sep. 17, 2020, pp. 357-362. [cited by applicant]
Hofstätter, S., et al., “Let's measure run time!,” published on arXiv:1907.04614v1, Jul. 10, 2019, 5 pages. [cited by applicant]
Hofstätter, S., et al., “Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation,” published on arXiv:2010.02666v2, Jan. 22, 2021, 10 pages. [cited by applicant]
Joffe, S., et al., “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift,” International conference on machine learning, 2015, PMLR, pp. 448-456, downloaded from arXiv:1502.03167v… [cited by applicant]
Jain, H., et al., “SUBIC: A supervised, structured binary code for image search,” published on arXiv:1708.02932v1, Aug. 9, 2017, 10 pages. [cited by applicant]
Jang, E., et al., “Categorical Reparameterization with Gumbel-Softmax,” published on arXiv:1611.01144v5, Aug. 5, 2017, 13 pages. [cited by applicant]
Jégou, H., et al., “Product Quantization for Nearest Neighbor Search,” IEEE Trans. Pattern Anal. Mach. Intell. vol. 33, No. 1, 2011, pp. 117-128. [cited by applicant]
Johnson, J., et al., “Billion-scale similarity search with GPUs,” published on arXiv:1702.08734v1, Feb. 28, 2017, 12 pages. [cited by applicant]
Kalantidis, Y., et al., “Locally Optimized Product Quantization for Approximate Nearest Neighbor Search,” 2014 IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, USA: IEEE, 2014, pp. 2329-2336. [cited by applicant]
Khattab, O., et al., “ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT,” Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information… [cited by applicant]
Kingma, D., et al., “Adam: A method for stochastic optimization,” published on arXiv:1412.6980v9, Jan. 30, 2017, 15 pages. [cited by applicant]
Klein, B., et al., “End-To-End Supervised Product Quantization for Image Search and Retrieval,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, (CVPR), 2019, pp. 5041-5050. [cited by applicant]
Kurland, O., et al., “Corpus Structure, Language Models, and Ad Hoc Information Retrieval,” SIGIR'04, Sheffeld, South Yorkshire, UK, Jul. 25-29, 2004, 8 pages. [cited by applicant]
Lam, S., et al., “Numba: A LLVM-based Python JIT Compiler,” Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC, Nov. 15-20, 2015, pp. 1-6. [cited by applicant]
Lee, S., et al., “Partitioned Parallel Radix Sort,” High Performance Computing, Lecture Notes in Computer Science, Berlin, Heidelberg: Springer Berlin Heidelberg, vol. 1940, pp. 2000, 160-171. [cited by applicant]
Li, H., “Learning to Rank for Information Retrieval and Natural Language Processing,” Synthesis Lectures on Human Language Technologies 12, San Rafael, Calif., Morgan & Claypool, 2011, 94 pages. [cited by applicant]
Lin, S., et al., “Distilling Dense Representations for Ranking using Tightly-Coupled Teachers,” published on arXiv:2010.11386v1, Oct. 22, 2020, 7 pages. [cited by applicant]
Macavaney, S., et al., “Efficient Document Re-Ranking for Transformers by Precomputing Term Representations,” Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retriev… [cited by applicant]
Mackenzie, J., et al., “Efficiency Implications of Term Weighting for Passage Retrieval,” Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, C… [cited by applicant]
Maddison, C., et al., “A* Sampling,” Proceedings of the 27th International Conference on Neural Information Processing Systems, vol. 2, 2014, pp. 3086-3094. [cited by applicant]
Malkov, Y., et al., “Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs,” IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 2020, pp. 824-836. [cited by applicant]
Mallia, A., et al., “PISA: Performant Indexes and Search for Academia,” Proceedings of the Open-Source IR Replicability Challenge, Paris, France, Jul. 21-25, 2019, 7 pages. [cited by applicant]
Medini, T., et al., “SOLAR: Sparse Orthogonal Learned and Random Embeddings,” published on arXiv:2008.13225v1, Aug. 30, 2020, 14 pages. [cited by applicant]
Nogueira, R., et al., “Passage Re-Ranking With BERT,” published on arXiv:1901.04085v5, Apr. 14, 2020, 5 pages. [cited by applicant]
Nogueira, R., et al., “From doc2query to docTTTTTquery,” MS MARCO Passage Retrieval Task, 2019, 3 pages. [cited by applicant]
Nogueira, R., et al., “Document Expansion by Query Prediction,” published on arXiv:1904.08375v2, Sep. 25, 2019, 7 pages. [cited by applicant]
Nogueira, R., et al., “Multi-Stage Document Ranking with BERT,” published on arXiv:1910.14424v1, Oct. 31, 2019, 13 pages. [cited by applicant]
Paria, B., et al., “Minimizing FLOPs to Learn Efficient Sparse Representations,” International Conference on Learning Representations, Sep. 25, 2019, 20 pages. [cited by applicant]
Paszke, A., et al., “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” 33rd Conference on Neural Information Processing Systems (NeurIPS.43), Vancouver, Canada, 2019, 12 pages. [cited by applicant]
Qu, Y., et al., “RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering,” Proceedings of the 2021 Conference of the North American Chapter of the Association for Computati… [cited by applicant]
Raffel, C., et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,” Journal of Machine Learning Research 21, 2020, pp. 1-67, published on arXiv:1910.10683v3, Jul. 28, 2020. [cited by applicant]
Reimers, N., et al., “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, N… [cited by applicant]
Robertson, S., et al., “The Probabilistic Relevance Framework: BM25 and Beyond,” Foundations and Trends in Information Retrieval 3, No. 4, 2009, pp. 333-389. [cited by applicant]
Sanh, V., et al., “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” published on arXiv:1910.01108v2, Oct. 16, 2019, 5 pages. [cited by applicant]
Office Action received in Korean Application No. 10-2022-0084314, dated Feb. 21, 2025, 16 pages. [cited by applicant]
Luan, Y, et al., “Sparse, Dense, and Attentional Representations for Text Retrieval,” Transactions of the Association for Computational Linguistics, Apr. 2021, vol. 9, pp. 329-345. [cited by applicant]
Negri, M., et al., “Natural Representation of Composite Data With Replicated Autoencoders,” published on arXiv:1909.13327v1, Sep. 29, 2019, 18 pages. [cited by applicant]
Zhang, T., et al., “Sparse Composite Quantization,” Proceedings ofthe IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 4548-4556. [cited by applicant]
Written Decision on Registration received for Korean Application No. 10-2022-0084314, dated Oct. 30, 2025. [cited by applicant]