Real time content searching in social network
View Patent ↗Indexing and retrieving real time content in a social networking system is disclosed. A user-term index includes user-term partitions, each user-term partition comprising temporal databases. As a post is received from a user, a user identifier, a post identifier, and a post is extracted. An object store communicatively coupled to a temporal database for recently received content is queried to determine whether terms in the post has already been stored. A term identifier is stored in the user-term index with the user and post identifiers. A forward index stores the post by post identifier. Responsive to a search query, the user-term index is searched by the user's connections and the terms. A real time search engine compiles the results of the user-term index query and retrieves the stored posts from the forward index. The search results may then be ranked and cached before presentation to the searching user.
1. A method comprising:
receiving a post from a user;
determining a user identifier associated with the user, a post identifier associated with the post, and a term identifier associated with a term in the post;
selecting a partition of a user-term index that is associated with the user identifier from among a plurality of partitions of the user-term index, the partition comprising a plurality of database shards organized by time;
selecting a record in a most recent database shard of the of a plurality of database shards, the record comprising the user identifier, the term identifier, and a list of post identifiers; and
adding the post identifier into the list of post identifiers of the selected record in the most recent database shard.
2. The method of claim 1 , wherein determining the term identifier comprises:
performing a hash function on the term to generate the term identifier;
querying for the term identifier in an object store associated with the most recent database shard in the selected partition of the user-term index; and
allocating memory in the object store for the term identifier responsive to not finding the term identifier in the object store.
3. The method of claim 1 , wherein selecting the partition of the user-term index that is associated with the user identifier comprises:
performing a hash function to associate the user identifier to a particular partition of the user-term index; and
selecting the particular partition of the user-term index.
4. The method of claim 3 , wherein the hash function takes a modulo of the user identifier by the number of partitions.
5. The method of claim 1 , wherein the plurality of database shards comprises a shard for each day of the month, a shard for each month of the year, or a shard for each hour of the day.
6. The method of claim 1 , further comprising:
determining that all of the shards are filled to capacity; and
creating a new shard, and setting the new shard as the most recent database shard.
7. The method of claim 6 , further comprising:
deleting the oldest shard, of the plurality of shards; and
deleting an object store associated with the oldest shard.
8. A system comprising:
a server configured to receive a post from a user;
a real time search engine comprising an indexing module configured to determine a post identifier of the post and a term identifier of a term in the post and an aggregator module configured to determine a user identifier of the user,
a forward index configured to store the received post based upon the post identifier; and
a user-term index comprising a plurality of partitions, and each partition comprises a respective plurality of database shards organized by time
wherein,
the aggregator module is further configured to:
select a partition of the user-term index that is associated with the user identifier from among the plurality of partitions of the user-term index,
select a record in a most recent database shard of a plurality of database shards that comprise the selected partition, and
the indexing module is further configured to:
add the post identifier into the list of post identifiers of the selected record in the most recent database shard.
9. The system of claim 8 , wherein the indexing module is further configured to:
perform a hash function on the term to generate the term identifier;
query the term identifier in an object store associated with the most recent database shard in the selected partition of the user-term index; and
allocate memory in the object store for the term identifier responsive to not finding the term identifier in the object store.
10. The system of claim 8 , wherein the indexing module is configured to:
perform a hash function to associate the user identifier to a particular partition of the user-term index; and
select the particular partition of the user-term index.
11. The system of claim 10 , wherein the hash function takes the modulo of the user identifier by the number of partitions.
12. The system of claim 8 , wherein the plurality of database shards comprises a shard for each day of the month, a shard for each month of the year, or a shard for each hour of the day.
13. The system of claim 8 , wherein the number of database shards fluctuates over time.
14. The system of claim 8 , wherein the indexing module is further configured to:
determine that all of the shards are filled to capacity; and
create a new shard, and setting the new shard as the most recent database shard.
15. The system of claim 14 , wherein the indexing module is further configured to:
delete the oldest shard, of the plurality of shards; and
delete an object store associated with the oldest shard.
16. A computer program product comprising a computer-readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to perform steps comprising:
receiving a post from a user;
determining a user identifier associated with the user, a post identifier associated with the post, and a term identifier associated with a term in the post;
selecting a partition of a user-term index that is associated with the user identifier from among a plurality of partitions of the user-term index, the partition comprising a plurality of database shards organized by time;
selecting a record in a most recent database shard of the of a plurality of database shards, the record comprising the user identifier, the term identifier, and a list of post identifiers; and
adding the post identifier into the list of post identifiers of the selected record in the most recent database shard.
17. The computer program product of claim 16 , wherein determining the term identifier comprises:
performing a hash function on the term to generate the term identifier;
querying for the term identifier in an object store associated with the most recent database shard in the selected partition of the user-term index; and
allocating memory in the object store for the term identifier responsive to not finding the term identifier in the object store.
18. The computer program product of claim 16 , wherein selecting the partition of the user-term index that is associated with the user identifier comprises:
performing a hash function to associate the user identifier to a particular partition of the user-term index; and
selecting the particular partition of the user-term index.
19. The computer program product of claim 18 , wherein the hash function takes a modulo of the user identifier by the number of partitions.
20. The computer program product of claim 16 , wherein the plurality of database shards comprises a shard for each day of the month, a shard for each month of the year, or a shard for each hour of the day.