Generic scheduling
A system and method for customized scheduling of sources, including breaking down a source of content into at least two categories, including posts and engagements, and gathering content related to a specific source. A scheduler handles scheduling of posts and engagement for a single source and entities that are due to be crawled are sent to a scheduling queue, in which each content type for a source can have its own queue. A process points to the correct scheduler queue in order to request content to be crawled, attaches to the proper queue, processes requests, queries the social network for content, parses the response and sends any new data to be saved to the system.
1. A method for setting a schedule of a crawl of a content from a web page, the method comprising:
parsing, by a processor, the content into a first portion and a second portion, the first portion being associated with a new content to the web page, the second portion being associated with updated content related to the first portion;
identifying, by the processor, whether the web page comprises a pattern of content being a first static type or a second non-static type;
in response to a determination that the web page is the pattern of content of the first static type, causing the processor to refrain from fetching the static content until the processor determines the content of the web page has been modified; and
setting, by the processor, the schedule according to a type of the web page, the type being the first static type or the second non-static type.
2. The method of claim 1 , further comprising rescheduling, by the processor and in response to a determination that the web page is the second non-static type, the crawl of the content of the web page.
3. The method of claim 1 , further comprising sending, by the processor to a scheduling queue, the schedule of the crawl of the content from the web page.
4. The method of claim 3 , wherein the scheduling queue comprises:
a first queue for the first static type; and
a second queue for the second non-static type.
5. The method of claim 3 , wherein the scheduling queue comprises:
a first queue for the new content in web page; and
a second queue for the updated content related to the new content in the web page.
6. The method of claim 1 , further comprising querying, by the processor, a dynamics ADS active table in conjunction with rescheduling the crawl of the updated content.
7. A non-transitory computer-readable medium storing computer code for setting a schedule of a crawl of a content from a web page, the computer code including instructions to cause a processor to:
parse the content into a first portion and a second portion, the first portion being associated with a new content to the web page, the second portion being associated with updated content related to the first portion;
identify whether the web page comprises a pattern of content being a first static type or a second non-static type;
causing, in response to a determination that the web page is the pattern of content of the first static type, to refrain from fetching the static content until the processor determines that a new content on the web page is available; and
set the schedule according to a type of the web page, the type being the first static type or the second non-static type.
8. A system for setting a schedule of a crawl of a content from a web page, comprising:
a memory configured to store the content; and
a processor configured to:
parse the content into a first portion and a second portion, the first portion being associated with a new content to the web page, the second portion being associated with updated content related to the first portion;
identify whether the web page comprises a pattern of content being a first static type or a second non-static type;
causing, in response to a determination that the web page is the pattern of content of the first static type, to refrain from fetching the static content until the processor
determines that a new content on the web page is available; and
set the schedule according to a type of the web page, the type being the first static type or the second non-static type.