Supervised semantic indexing and its extensions

1. A method for determining a similarity between a document and a query, comprising:

building a weight vector for each of a plurality of documents in a corpus of documents stored in memory;

building a weight vector for a query input into a document retrieval system;

generating a weight matrix which distinguishes between relevant documents and lower ranked documents by comparing document/query tuples using a gradient step approach; and

determining a similarity score between weight vectors of the query and documents in a corpus by determining a product of a document weight vector, a query weight vector and the weight matrix.

2. The method as recited in claim 1 , wherein the weight vectors include Term Frequency Inverse Document Frequency (TFIDF) weights.

3. The method as recited in claim 1 , further comprising sparsifying the weight matrix through feature selection.

4. The method as recited in claim 3 , wherein sparsifying includes:

finding active elements in the weight matrix with smallest weight values;

constraining the active elements in the weight matrix with smallest weight values to equal zero;

training a model with the weight matrix; and

if the weight matrix includes more than p non-zero terms, returning to the finding step.

5. The method as recited in claim 1 , wherein building a weight vector for each of a plurality of documents includes performing a linear embedding to transform the weight vector to embedding space using a matrix U.

6. The method as recited in claim 5 , wherein building a weight vector for a query includes performing a linear embedding to transform the weight vector to embedding space using a matrix V.

7. The method as recited in claim 6 , wherein determining a similarity score includes taking a product between transformed document and query weight vectors to simplify computations.

8. The method as recited in claim 1 , wherein building a weight vector for each of a plurality of documents in a corpus of document includes mapping the weight vectors for the plurality of documents to multiple embedding spaces to handle models having polynomial functions of n-order.

9. The method as recited in claim 1 , further comprising combining functional embedding with transductive embedding to transform weight vectors to model both new queries and improve performance in existing databases.

10. The method as recited in claim 1 , wherein building a weight vector includes mapping the weight vector using a sliding window while embedding words or grams to consider other words or grams in proximity thereof.

11. The method as recited in claim 1 , wherein generating a weight matrix includes:

assigning infrequently occurring words to a reduced dimension representation; and

mapping the reduced dimension representation to a full dimension representation when the infrequently occurring words are encountered.

12. A system for determining a similarity between a document and a query, comprising:

a user interface configured to permit a user to pose a query to find relevant documents, each document having a weight vector stored in memory;

a processing device configured to build a weight vector for the query;

a weight matrix stored in memory and configured to distinguish between relevant documents and lower ranked documents by comparing document/query tuples using a gradient step approach; and

a program configured to determine a similarity score between weight vectors of the query and documents in a corpus by determining a product of a document weight vector, a query weight vector and the weight matrix.

13. The system as recited in claim 12 , wherein the program includes one or more of a method for sparsification of the weight matrix, a method to implement low ranking approximations, a method to perform correlated feature hashing (CFH), a method for polynomial modeling of word features, a method for half-transductive modeling, a method for sliding window mapping, and a method for flexible dimensionality for word embedding.