METHODS AND SYSTEMS FOR MAPPING DATA ITEMS TO SPARSE DISTRIBUTED REPRESENTATIONS
A method enables identification of a similarity level between a user-provided data item and a data item within a set of data documents. The method includes a representation generator determining, for each term in an enumeration of terms, occurrence information. The representation generator generates, for each term, a sparse distributed representation (SDR) using the occurrence information. The method includes receiving, by a filtering module, a filtering criterion. The method includes generating, by the representation generator, for the filtering criterion, at least one SDR. The method includes generating, by the representation generator, for a first of a plurality of streamed documents received from a data source, a compound SDR. The method includes determining, by a similarity engine executing on the second computing device, a distance between the filtering criterion SDR and the generated compound SDR. The method includes acting on the first streamed document, based upon the determined distance.
1 . A computer-implemented method for identifying a level of similarity between a user-provided data item and a data item within a set of data documents, the method comprising:
clustering, by a reference map generator executing on a first computing device, in a two-dimensional metric space, a set of data documents selected according to at least one criterion, generating a semantic map;
associating, using the semantic map, a coordinate pair with each data document in the set of data documents;
generating, by a parser executing on the first computing device, an enumeration of terms occurring in the set of data documents;
determining, by a representation generator executing on the first computing device, for each term in the enumeration, occurrence information including: (i) a number of data documents in which the term occurs, (ii) a number of occurrences of the term in each data document, and (iii) the coordinate pair associated with each data document in which the term occurs;
generating, by the representation generator, for each term in the enumeration, a sparse distributed representation (SDR) using the occurrence information;
storing, in an SDR database, each of the generated SDRs;
receiving, by a query module executing on a second computing device and in communication with the first computing device, from a third computing device, a first term and a plurality of preference documents;
generating, by the representation generator, a compound SDR using the plurality of preference documents;
transmitting, by the query generator, the first term to a similarity engine executing on a fourth computing device;
determining, by the similarity engine, a level of semantic similarity between a first SDR generated based on the first term and a second SDR of a second term, the second SDR retrieved from the SDR database;
receiving, by the query module, from the similarity engine, the first term and the second term;
transmitting, by the query module, to a full-text search system, a query for an identification of each of a subset of a second set of documents containing at least one term similar to at least one of the first term and the second term;
transmitting, by the query module, to a full text search system, a query for an identification of each of a set of results documents similar to the first term;
receiving, by the query module, the set of results documents from the full-text search system;
generating, by the representation generator, an SDR for each of the documents identified in the set of results documents;
determining, by a similarity engine executing on the second computing device, a level of semantic similarity between each SDR generated for the each of the set of results documents and the compound SDR;
modifying, by a ranking module executing on the second computing device, an order of at least one document in the set of results documents, based on the determined level of semantic similarity; and
providing, by the query module, to the third computing device, the identification of each of the set of search results documents in the modified order.