System and method for updating a search index

1. A method implemented on at least one machine comprising at least one processor, memory, and a communication platform connected to a network for updating a search index, the method comprising:

obtaining event data representing actions performed on content items managed via the search index;

filtering the event data based on pre-determined criteria to generate updated event data, wherein the updated event data represents relevant actions performed on at least one of the content items and are associated with the search index, and wherein the pre-determined criteria comprises (i) a list of content items represented within the search index including the at least one of the content items, and (ii) an action associated with the at least one of the content items corresponds to one of the relevant actions; and

generating an updated search index based on the updated event data to reflect the action associated with the at least one of the content items.

2. The method of claim 1 , wherein filtering comprises:

retrieving the list of content items represented within the search index;

determining that the list of content items comprises the at least one of the content items; and

generating the updated event data.

3. The method of claim 1 , wherein retrieving the list of content items comprises:

determining, for each of the actions, a corresponding identifier associated with a content item; and

identifying the list of content items from a plurality of lists of content items based on the identifier.

4. The method of claim 1 , further comprising:

receiving content items from one or more content feeds;

extracting a semantic entity for at least one of the content items;

adding an identifier associated with the at least one of the content items to a list of content items; and

generating the search index comprising one or more lists of identifiers ranked based on a relevancy of a corresponding content item to the semantic entity.

5. The method of claim 1 , wherein obtaining the event data comprises:

determining an epoch start time;

receiving, from one or more content feeds, the event data;

determining an epoch end time; and

storing the event data within an entry in an event log, the entry having a temporal duration equal to a difference between the epoch end time and the epoch start time.

6. The method of claim 1 , wherein the event data is stored within an event log located on a content grid, and the search index is located on a remote system, the method further comprising:

sending the updated search index to the remote system to perform an update to the search index.

7. The method of claim 6 , wherein the event log, a content log, and an extraction log are located on the content grid, the content log being configured to store information associated with the content items, and the extraction log being configured to store entities extracted from at least some of the content items.

8. The method of claim 7 , wherein, the content log, the extraction log, and the event log are log-structured file systems, filtering further comprises:

generating the pre-determined criteria, the pre-determined criteria comprising a Bloom Filter indicating a list of content items represented within the search index; and

applying the Bloom Filter to the event data to generate the updated event data.

9. A method implemented on at least one machine comprising at least one processor, memory, and a communication platform connected to a network for updating a search index, the method comprising:

obtaining event data representing actions performed on a plurality of content items managed via the search index;

filtering the event data based on pre-determined criteria to generate updated event data, wherein the updated event data represents relevant actions performed on at least one of the plurality of content items and are associated with the search index, and wherein the pre-determined criteria comprises (i) a list of content items represented within the search index including the at least one of the plurality of content items, and (ii) an action associated with the at least one of the plurality of content items corresponds to one of the relevant actions; and

sending the updated event data to a content grid for generating an updated search index.

10. The method of claim 9 , wherein filtering comprises:

retrieving the list of content items represented within the search index;

determining that the list of content items comprises the at least one of the plurality of content items; and

generating the updated event data, wherein the search index is to be updated to reflect the action associated with the at least one of the plurality of content items.

11. The method of claim 9 , wherein obtaining the event data comprises:

determining an epoch start time and an epoch end time associated with each of the actions represented by the event data; and

providing the event data to the content grid for storing within an entry of an event log, the entry having a temporal duration equal to a difference between the epoch end time and the epoch start time and including each action that occurred during the temporal duration.

12. The method of claim 9 , wherein filtering comprises:

generating the pre-determined criteria, the pre-determined criteria further comprising a Bloom Filter indicating the list of content items represented within the search index; and

applying the Bloom Filter to the event data to generate the updated event data.

13. A system for updating a search index, comprising:

memory comprising one or more computer program instructions; and

at least one processor configured, upon execution of the one or more computer program instructions, to:

obtain event data representing actions performed on a plurality of content items managed via the search index;

filter the event data based on pre-determined criteria to generate updated event data, wherein the updated event data represents relevant actions performed on at least one of the plurality of content items and are associated with the search index, and wherein the pre-determined criteria comprises (i) a list of content items represented within the search index including the at least one of the plurality of content items, and (ii) an action associated with the at least one of the plurality of content items corresponds to one of the relevant actions; and

generate an updated search index based on the updated event data to reflect the action associated with the at least one of the plurality of content items.

14. The system of claim 13 , wherein the at least one processor, upon execution of the one or more computer program instructions associated with the event data being filtered, is configured to:

retrieve the list of content items represented within the search index;

determine that the list of content items comprises the at least one of the plurality of content items; and

generate the updated event data.

15. The system of claim 13 , wherein the at least one processor, upon execution of the one or more computer program instructions comprising the list of content items being retrieved, is configured to:

determine, for each of the actions, a corresponding identifier associated with a content item; and

identify the list of content items from a plurality of lists of content items based on the identifier.

16. The system of claim 13 , wherein the at least one processor, upon execution of the one or more computer program instructions, is further configured to:

receive content items from one or more content feeds;

extract a semantic entity for at least one of the content items;

add an identifier associated with the at least one of the content items to the list of content items; and

generate the search index comprising one or more lists of identifiers ranked based on a relevancy of a corresponding content item to the semantic entity.

17. The system of claim 13 , wherein the at least one processor, upon execution of the one or more computer program instructions comprising the event data being obtained, is configured to:

determine an epoch start time;

receive, from one or more content feeds, the event data;

determine an epoch end time; and

store the event data within an entry in an event log, the entry having a temporal duration equal to a difference between the epoch end time and the epoch start time.

18. The system of claim 13 , wherein the event data is stored within an event log located on a content grid, and the search index is located on a remote system, the at least one processor, upon execution of the one or more computer program instructions, is further configured to:

send the updated search index to the remote system to perform an update to the search index.

19. The system of claim 18 , wherein the event log, a content log, and an extraction log are located on the content grid, the content log being configured to store information associated with the plurality of content items, and the extraction log being configured to store entities extracted from at least some of the plurality of content items.

20. The system of claim 19 , wherein, the content log, the extraction log, and the event log are log-structured file systems, the at least one processor, upon execution of the one or more computer program instructions comprising the event data being filters, is configured to:

generate the pre-determined criteria, the pre-determined criteria comprising a Bloom Filter indicating the list of content items represented within the search index; and

apply the Bloom Filter to the event data to generate the updated event data.

21. A system for updating a search index, comprising:

memory comprising one or more computer program instructions; and

at least one processor configured, upon execution of the one or more computer program instructions, to:

obtain event data representing actions performed on a plurality of content items managed via the search index;

filter the event data based on pre-determined criteria to generate updated event data, wherein the updated event data represents relevant actions performed on at least one of the plurality of content items and are associated with the search index, and wherein the pre-determined criteria comprises (i) a list of content items represented within the search index including the at least one of the plurality of content items, and (ii) an action associated with the at least one of the plurality of content items corresponds to one of the relevant actions; and

send the updated event data to a content grid for generating an updated search index.

22. The system of claim 21 , wherein the at least one processor, upon execution of the one or more computer program instructions comprising the event data being filtered, is configured to:

retrieve the list of content items represented within the search index;

determine that the list of content items comprises the at least one of the plurality of content items; and

generate the updated event data.

23. The system of claim 21 , wherein the at least one processor, upon execution of the one or more computer program instructions comprising the event data being obtained, is configured to:

determine an epoch start time and an epoch end time associated with each of the actions represented by the event data; and

provide the event data to the content grid for storing within an entry of an event log, the entry having a temporal duration equal to a difference between the epoch end time and the epoch start time and including each action that occurred during the temporal duration.

24. The system of claim 21 , wherein the at least one processor, upon execution of the one or more computer program instructions comprising the event data being filtered, is configured to:

generate the pre-determined criteria, the pre-determined criteria comprising a Bloom Filter indicating the list of content items represented within the search index; and

apply the Bloom Filter to the event data to generate the updated event data.

25. A non-transitory computer readable medium comprising instructions that, when executed by at least one processor of a computing system, effectuate operations comprising:

obtaining event data representing actions performed on a plurality of content items managed via a search index;

filtering the event data based on pre-determined criteria to generate updated event data, wherein the updated event data represents relevant actions performed on at least one of the plurality of content items and are associated with the search index, and wherein the pre-determined criteria comprises (i) a list of content items represented within the search index including the at least one of the plurality of content items, and (ii) an action associated with the at least one of the plurality of content items corresponds to one of the relevant actions; and

generating an updated search index based on the updated event data to reflect the action associated with the at least one of the plurality of content items.

26. A non-transitory computer readable medium comprising instructions that, when executed by at least one processor of a computing system, effectuate operations comprising:

obtaining event data representing actions performed on a plurality of content items managed via a search index;

filtering the event data based on pre-determined criteria to generate updated event data, wherein the updated event data represents relevant actions performed on at least one of the plurality of content items and are associated with the search index, and wherein the pre-determined criteria comprises (i) a list of content items represented within the search index including the at least one of the plurality of content items, and (ii) an action associated with the at least one of the plurality of content items corresponds to one of the relevant actions; and

sending the updated event data to a content grid for generating an updated search index.