Composite code sparse autoencoders for approximate neighbor search

1 . A method for creating an index for information retrieval, the method being implemented by a processor and memory, the method comprising:

(a) receiving a set of input data, wherein each element of data in the set is a dense representation;

(b) generating, using a trained encoder implemented using the processor and including one or more layers, a composite code from the dense representation, the composite code comprising a vector of dimension D decomposed into C clusters of dimension L, where C and L are at least two;

(c) using an activation function to generate a sparse representation from the composite code, the sparse representation comprising a binary representation comprising C clusters of dimension L, where each cluster sums to one;

(d) repeating steps (a)-(c) for each element of data in the set of input data;

(e) creating, using the processor, an index based on each of the sparse representations generated at step (c) corresponding to each element of data in the set of input data; and

(f) outputting the index for information retrieval.

2 . The method of claim 1 , wherein each of the dense representations represent one or more of text data, image data and audio data;

wherein in the composite code of each dense representation data elements are represented by values of each of L subconcepts or classifications for each of C concepts.

3 . The method of claim 1 , wherein each of the dense representations represent text data and the dense representation is generated using a probabilistic model; wherein the probabilistic model comprises a Transformer model.

4 . The method of claim 1 , wherein the sparse representation is a sparse composite code where each of the C clusters forming part of the composite code has exactly one non-zero binary representation.

5 . The method of claim 1 , wherein the activation function is applied using an activation layer in a neural network.

6 . The method of claim 5 , wherein the activation layer comprises a softmax layer.

7 . The method of claim 1 , wherein information retrieval is one of text retrieval, image retrieval, and voice retrieval.

8 . The method of claim 1 , further comprising:

generating one or more posting lists from the sparse representation, wherein the posting list associated with a dimension comprises a list of documents having a non-zero binary representation in that dimension in the sparse composite code.

9 . A method for creating an index for information retrieval, the method being implemented by a processor and memory, the method comprising:

(a) receiving a set of input data, wherein each element of data in the set is a dense representation;

(b) generating, using a trained encoder implemented using the processor and including one or more layers, a composite code from the dense representation, the composite code comprising a vector of dimension D decomposed into C clusters of dimension L, where C and L are at least two;

(c) using an activation function to generate a sparse binary representation from the composite code, the sparse representation comprising C clusters of dimension L;

(d) repeating steps (a)-(c) for each element of data in the set of input data;

(e) creating, using the processor, an index based on each of the sparse representations generated at step (c) corresponding to each element of data in the set of input data; and

(f) outputting the index for information retrieval;

wherein the method further comprises:

regularizing the sparse binary representation to enforce load-balancing between different dimensions.

10 . The method of claim 9 , wherein said regularizing minimizes a loss based on a reconstruction loss combined with an error between an optimal mean activation of the sparse representation and a mean amount of activations of each dimension, weighted by a regularization factor;

wherein said regularizing is unsupervised.

11 . The method of claim 10 , wherein the error is approximated based on a batch of documents.

12 . An autoencoder for information retrieval implemented by a processor and a memory comprising:

a neural network encoder implemented using the processor, the neural network encoder being trained to receive a dense representation and generate a composite code comprising a vector of dimension D decomposed into C clusters of dimension L from the dense representation, where C and L are at least two, and

an activation function implemented using the processor, the activation function being configured to generate a sparse composite code from the composite code, the sparse composite code comprising a binary representation comprising C clusters of dimension L, where each cluster sums to one;

wherein the sparse composite code is used to generate, supplement, or search an index for information retrieval.

13 . The autoencoder of claim 12 further comprising:

a decoder implemented using the processor and configured to decode the sparse composite code into an original dense representation during training of the neural network encoder.

14 . The autoencoder of claim 12 , wherein the decoder comprises a linear decoder.

15 . The autoencoder of claim 12 , wherein the activation function comprises a Gumbel-softmax model.

16 . The autoencoder of claim 12 , wherein the encoder comprises a one-layer encoder.

17 . The autoencoder of claim 12 , further comprising:

a batch normalizer implemented using the processor and configured to reduce variance in the dense representation.

18 . An autoencoder for information retrieval implemented by a processor and a memory comprising:

a neural network encoder implemented using the processor, the neural network encoder being trained to receive a dense representation and generate a composite code from the dense representation, the composite code comprising a vector of dimension D decomposed into C clusters of dimension L, where C and L are at least two, and

an activation function implemented using the processor, the activation function being configured to generate a sparse composite code from the composite code, the sparse representation being a binary representation comprising C clusters of dimension L;

wherein the sparse composite code is used to generate, supplement, or search an index for information retrieval;

further comprising:

a neural regularizer implemented using the processor and configured to regularize the sparse representation to enforce load-balancing between different dimensions;

wherein in the composite code of each dense representation data elements are represented by values of each of Z subconcepts or classifications for each of C concepts.

19 . An information retrieval architecture implemented by one or more processors in combination with a memory, the architecture comprising:

an encoding phase implemented using the one or more processors and having one or more layers, the encoding phase being trained to receive a dense representation corresponding to a query, generate a composite code comprising a vector of dimension D decomposed into C clusters of dimension L from the dense representation and generate a sparse composite code from the composite code using an activation function, the sparse composite code comprising a binary representation comprising C clusters of dimension L, where each cluster sums to one, where C and L are at least two;

a scoring phase implemented using the one or more processors and configured to search an index based on the sparse composite code and output scores for a plurality of documents based on the index; and

a sorting phase implemented using the one or more processors and configured to rank a subset of the plurality of documents based on the scores and output one or more relevant documents from the ranked subset.

20 . The architecture of claim 19 , further comprising a probabilistic model implemented using the one or more processors and configured to:

receive the query;

encode the query into the dense representation; and

output the dense representation to the encoding phase.

21 . The architecture of claim 19 , further comprising:

a thresholding phase implemented using the one or more processors and configured to:

receive the scores from the scoring phase;

generate a subset of k documents from the scored documents based on a threshold; and

output the scores for the subset of k documents to the sorting phase;

wherein the sorting phase sorts the top-k documents based on the output scores.

22 . The architecture of claim 19 , wherein the scoring phase comprises an Approximate Nearest Neighbor (ANN) search model for information retrieval.

23 . The architecture of claim 19 , wherein the one or more processors of the encoding phase comprise at least one graphics processing unit (GPU) or central processing unit (CPU), and wherein the one or more processors of the scoring phase comprise at least one central processing unit (CPU).

24 . The architecture of claim 19 , wherein the architecture is further configured to determine one or more posting lists to score from the sparse composite code, and wherein the scoring phase searches the index based on the determined posting lists.

25 . The architecture of claim 19 , wherein the one or more processors scoring phase comprises a plurality of CPUs operating in parallel.

26 . A method for information retrieval implemented by one or more processors and a memory, the method comprising:

receiving a dense representation of a query;

generating, using a trained encoder implemented using the one or more processors and including one or more layers, a composite code comprising a vector of dimension D decomposed into C clusters of dimension L from the dense representation, where C and L are at least two;

generating a sparse composite code from the composite code model using an activation implemented using the one or more processors, the sparse composite code comprising a binary representation comprising C clusters of dimension L, where each cluster sums to one;

generating scores for a plurality of documents in an index based on the sparse composite code;

ranking a subset of the plurality of documents based on the scores; and

outputting the ranked subset.

27 . The method of claim 26 , wherein the received dense representation is generated by a probabilistic model processing the query.

28 . The method of claim 26 , wherein the query comprises a text query and the index represents one or more of text data, image data and audio data;

wherein in the composite code of each dense representation data elements are represented by values of each of L subconcepts or classifications for each of C concepts.

29 . The method of claim 26 , wherein the query comprises one or more of a search query, a request for a recommendation, a question, or a sequence to be translated.

30 . The method of claim 26 , wherein the index represents documents, and wherein the documents comprise one or more of text data, image data and audio data;

wherein in the composite code of each dense representation data elements are represented by values of each of L subconcepts or classifications for each of C concepts.

31 . The method of claim 26 , wherein the scoring comprises searching the index on the sparse composite code.

32 . The method of claim 26 , wherein the scoring comprises searching the index based on one or more posting lists determined using the composite code.

33 . The method of claim 26 , further comprising, creating the index by:

(a) receiving a set of input data, wherein each element of data in the set is a dense representation;

(b) generating a composite code from the dense representation using the trained encoder, the composite code comprising a vector of dimension D decomposed into C clusters of dimension L, where C and L are at least two;

(c) using an activation function implemented using the one or more processors to generate a sparse representation from the composite code, the sparse representation comprising a binary representation comprising C clusters of dimension L, where each cluster sums to one;

(d) repeating steps (a)-(c) for each element of data in the set of input data;

(e) creating an index based on each of the sparse representations generated at step (c) corresponding to each element of data in the set of input data; and

(f) outputting the index for information retrieval.

34 . An apparatus for transforming one or more dense representations corresponding to one or more documents into a sparse composite code for information retrieval comprising:

a non-transitory computer-readable medium having executable instructions stored thereon for causing a processor and a memory to:

for each of the one or more dense representations:

receive the dense representation;

generate, using a trained encoder including one or more layers, a composite code from the dense representation, the composite code comprising a vector of dimension D decomposed into C clusters of dimension L, where C and L are at least two; and

use an activation layer to generate the sparse composite code from the composite code, the sparse composite code comprising a binary representation comprising C clusters of dimension L, where each cluster sums to one; and

outputting the sparse composite codes for the one or more documents for generating, supplementing, or searching an index for information retrieval.

35 . The apparatus of claim 34 , wherein the processor and memory is further caused to:

generate or supplement an index using the one or more output sparse composite codes.

36 . The apparatus of claim 34 , wherein the processor and memory is further caused to:

search an index using the output sparse composite codes.

37 . The apparatus of claim 34 , wherein the dense representation is generated from a probabilistic model.

38 . The apparatus of claim 34 , wherein the one or more documents comprise one or more of text data, image data and audio data;

wherein in the composite code of each dense representation data elements are represented by values of each of L subconcepts or classifications for each of C concepts.

39 . The apparatus of claim 34 , wherein the executable instructions further cause a processor and a memory to:

regularize the sparse composite code to enforce load-balancing between different dimensions.