Method and system for data mining
The present teaching relates to method and system for generating a stream of content items. A plurality of entities associated with a time-window are obtained, wherein each entity of the plurality of entities is associated with at least one content item. For each entity, a first parameter with respect to the time-window, a second parameter with respect to previous time-windows, and a trendiness score based on a function of the first parameter and the second parameter are respectively calculated. A graph based on one or more entity-pairs is generated, wherein each entity-pair of the one or more entity-pairs satisfies a first criterion. A stream of content items is generated based on the graph, wherein each content item in the stream of content items corresponds to at least one of the one or more entity pairs.
1 . A method, implemented on a machine having at least one processor, storage, and a communication platform capable of connecting to a network for generating a stream of web-based content items, the method comprising:
crawling, by an engine, in accordance with a configuration model controlling a number of hyperlinks crawled from a webpage, web-based published content items;
extracting, by the engine, in accordance with a machine learning model, a plurality of entities from each of the web-based published content items, wherein each entity is within a corresponding time window;
determining, for each of the published content items, a corresponding category from a plurality of categories;
appending, via a web-application embedded in a webpage, each of the published content items with metadata in the web-application, wherein the metadata includes one or more of: a corresponding publisher, the corresponding entities, the corresponding category, and a count of the corresponding entities;
clustering the appended content items based on the categories to generate one or more clusters;
storing the time windows, the extracted entities, and the clustered content items in a repository;
obtaining, from the repository, entities stored in their corresponding time-windows to calculate, for each of the extracted entities, a corresponding trendiness score as a function including a numerical difference between a number of occurrences of the entity in the time window and an average number of occurrences of the entity in previous time windows;
determining one or more entity-pairs with respect to the extracted entities, wherein a co-occurrence count of entities included in each of the one or more entity-pairs exceeds a threshold;
ranking stored content items in each of the one or more clusters based on an average trendiness score of entities included in each of the one or more entity-pairs in the cluster;
generating, based on a query for content items that are trending, a stream of web-based content items by selecting a highest ranked content item from each of the one or more clusters to be included in the stream; and
scaling up the stream of web-based content items across the plurality of categories.
2 . The method of claim 1 , wherein the clustering the appended content items is in accordance with a clustering model.
3 . The method of claim 1 , wherein the metadata further comprises a publish time of the corresponding content item.
4 . The method of claim 3 , wherein the generating the stream of content items is further based on the publish time.
5 . The method of claim 1 , wherein the plurality of categories comprise at least one of finance, politics, sports, and entertainment.
6 . A non-transitory machine-readable medium having information recorded thereon for generating a stream of web-based content items, wherein the information, when read by a machine, causes the machine to perform operations comprising:
crawling, by an engine, in accordance with a configuration model controlling a number of hyperlinks crawled from a webpage, web-based published content items;
extracting, by the engine, in accordance with a machine learning model, a plurality of entities from each of the web-based published content items, wherein each entity is within a corresponding time window;
determining for each of the published content items, a corresponding category from a plurality of categories;
appending, via a web-application embedded in a webpage, each of the published content items with metadata in the web-application, wherein the metadata includes one or more of: a corresponding publisher, the corresponding entities, the corresponding category, and a count of the corresponding entities;
clustering the appended content items based on the categories to generate one or more clusters;
storing the time windows, the extracted entities, and the clustered content items in a repository;
obtaining, from the repository, entities stored in their corresponding time-windows to calculate, for each of the extracted entities, a corresponding trendiness score as a function including a numerical difference between a number of occurrences of the entity in the time window and an average number of occurrences of the entity in previous time windows;
determining one or more entity-pairs with respect to the extracted entities, wherein a co-occurrence count of entities included in each of the one or more entity-pairs exceeds a threshold;
ranking stored content items in each of the one or more clusters based on an average trendiness score of entities included in each of the one or more entity-pairs in the cluster;
generating, based on a query for content items that are trending, a stream of web-based content items by selecting a highest ranked content item from each of the one or more clusters to be included in the stream; and
scaling up the stream of web-based content items across the plurality of categories.
7 . The medium of claim 6 , wherein the clustering the appended content items is in accordance with a clustering model.
8 . The medium of claim 6 , wherein the metadata further comprises a publish time of the corresponding content item.
9 . The medium of claim 8 , wherein the generating the stream of content items is further based on the publish time.
10 . The medium of claim 6 , wherein the plurality of categories comprise at least one of finance, politics, sports, and entertainment.
11 . A system for generating a stream of web-based content items, the system comprising:
memory storing computer program instructions; and
one or more processors that, in response to executing the computer program instructions, effectuate operations comprising:
crawling, by an engine, in accordance with a configuration model controlling a number of hyperlinks crawled from a webpage, web-based published content items;
extracting, by the engine, in accordance with a machine learning model, a plurality of entities from each of the web-based published content items, wherein each entity is within a corresponding time window;
determining, for each of the published content items, a corresponding category from a plurality of categories;
appending, via a web-application embedded in a webpage, each of the published content items with metadata in the web-application, wherein the metadata includes one or more of: a corresponding publisher, the corresponding entities, the corresponding category, and a count of the corresponding entities;
clustering the appended content items based on the categories to generate one or more clusters;
storing the time windows, the extracted entities, and the clustered content items in a repository;
obtaining, from the repository, entities stored in their corresponding time-windows to calculate, for each of the extracted entities, a corresponding trendiness score as a function including a numerical difference between a number of occurrences of the entity in the time window and an average number of occurrences of the entity in previous time windows;
determining one or more entity-pairs with respect to the extracted entities, wherein a co-occurrence count of entities included in each of the one or more entity-pairs exceeds a threshold;
ranking stored content items in each of the one or more clusters based on an average trendiness score of entities included in each of the one or more entity-pairs in the cluster;
generating, based on a query for content items that are trending, a stream of web-based content items by selecting a highest ranked content item from each of the one or more clusters to be included in the stream; and
scaling up the stream of web-based content items across the plurality of categories.
12 . The system of claim 11 , wherein the clustering the appended content items is in accordance with a clustering model.
13 . The system of claim 11 , wherein the metadata further comprises a publish time of the corresponding content item.
14 . The system of claim 13 , wherein the generating the stream of content items is further based on the publish time.