Real time content searching in social network
View Patent ↗Indexing and retrieving real time content in a social networking system is disclosed. A user-term index includes user-term partitions, each user-term partition comprising temporal databases. As a post is received from a user, a user identifier, a post identifier, and a post is extracted. An object store communicatively coupled to a temporal database for recently received content is queried to determine whether terms in the post has already been stored. A term identifier is stored in the user-term index with the user and post identifiers. A forward index stores the post by post identifier. Responsive to a search query, the user-term index is searched by the user's connections and the terms. A real time search engine compiles the results of the user-term index query and retrieves the stored posts from the forward index. The search results may then be ranked and cached before presentation to the searching user.
1. A method comprising:
receiving a query comprising a term from a user;
determining a user identifier of the user and a term identifier associated with the term;
gathering post identifiers and user identifiers that are associated with the term identifier in a user-term index comprising time-ordered database shards of records, where the user-term index is organized by a plurality of user identifiers associated with a plurality of users of a social networking system and a plurality of term identifiers associated with a plurality of terms used by the plurality of users;
retrieving posts from an index based upon the gathered post identifiers for presentation to the user;
determining user identifiers of connections of the user; and
selecting partitions that are associated with the user identifiers of the connections of the user from among a plurality of partitions of the user-term index,
wherein gathering post identifiers comprises gathering post identifiers that are associated with the term identifier in the selected partitions of the user-term index.
2. The method of claim 1 , wherein gathering the post identifiers that are associated with the term identifier in the user-term index further comprises:
selecting a record in a most recent database shard of the user-term index, the record comprising the term identifier and a list of post identifiers; and
compiling the lists of post identifiers.
3. The method of claim 1 , wherein gathering the post identifiers that are associated with the term identifier in the selected partitions of the user-term index comprises:
for each of the selected partitions, selecting a record in a most recent database shard of the selected partition, the record comprising a user identifier matching a user identifier for at least one of the connections, the term identifier, and a list of post identifiers; and
compiling the lists of post identifiers.
4. The method of claim 1 , wherein selecting partitions that are associated with the user identifiers of the connections of the user comprises:
performing a hash function to associate each of the connections' user identifiers to respective partitions of the user-term index; and
selecting the respective partitions of the user-term index.
5. The method of claim 1 , wherein the presentation of the retrieved posts to the user comprises applying a ranking to the retrieved posts.
6. The method of claim 1 , further comprising:
storing the retrieved posts in a global cache; and
responsive to receiving subsequent queries matching the query, retrieving the posts from the global cache.
7. The method of claim 1 , wherein the connections comprise other users of the social networking system that are connected to the user.
8. The method of claim 1 , wherein the presentation of the retrieved posts to the user is limited by privacy settings.
9. The method of claim 5 , wherein the ranking comprises ranking the retrieved posts based at least in part on reputations of users.
10. The method of claim 5 , wherein the ranking comprises ranking the retrieved posts based at least in part on popularity of users.
11. The method of claim 5 , wherein the ranking comprises ranking the retrieved posts based at least in part on similarity of users.
12. The method of claim 5 , wherein the ranking comprises ranking the retrieved posts based at least in part on proximity of users.
13. The method of claim 5 , wherein the ranking comprises ranking the retrieved posts based at least in part on affinities.
14. A system for retrieving information in a social networking system, the system comprising:
a server configured to receive a query comprising a term from a user;
a real time search engine, communicatively coupled to the server, comprising an aggregator module configured to determine a user identifier of the user and a term identifier associated with the term;
an index configured to store posts based upon post identifiers; and
a user-term index comprising time-ordered database shards of records, the user-term index organized by a plurality of user identifiers associated with a plurality of users of the social networking system and a plurality of term identifiers associated with a plurality of terms used by the plurality of users,
wherein the aggregator module is further configured to:
gather post identifiers and user identifiers that are associated the term identifier in the user-term index,
retrieve posts from the index based upon the gathered post identifiers for presentation to the user,
determine user identifiers of connections of the user, and
select partitions that are associated with the user identifiers of the connections of the user from among a plurality of partitions of the user-term index, wherein aggregator module gathers post identifiers that are associated with the term identifier in the selected partitions of the user-term index.
15. The system of claim 14 , wherein the aggregator module is further configured to:
select a record in a most recent database shard of the user-term index, the record comprising the term identifier and a list of post identifiers; and
compile the lists of post identifiers.
16. The system of claim 14 , wherein the aggregator module is further configured to:
for each of the selected partitions, select a record in a most recent database shard of the selected partition, the record comprising a user identifier matching one of the user identifiers of the user's connections, the term identifier, and a list of post identifiers; and
compile the lists of post identifiers.