Efficient crawling using path scheduling, and applications thereof
The present disclosure is directed to systems and methods for extracting unstructured data from a data source in a structure manner. Embodiments provide ways to retrieve unstructured data along from data sources not optimized for automated retrieval. For example, embodiments may generate a branched tree for each data source that maps out paths to individual sites of, for example, a healthcare provider listing the unstructured data. Using this branched tree, tasks can be generated to navigate along a path with the data source to each site and extract the unstructured data from the data source. In this way, embodiments provide the ability to navigate through a site from a base site to a site that has the relevant data.
1. A computer-implemented method for extracting data from a plurality of data sources, comprising:
generating a decision tree for a data source of the plurality of data sources, wherein the decision tree comprises a base website at a root node and a plurality of respective sites represented as corresponding leaf nodes of the decision tree, wherein the decision tree specifies one or more paths to navigate from the base website to the plurality of respective sites containing unstructured demographic data for a healthcare provider, wherein each of the plurality of respective sites and the corresponding leaf nodes of the decision tree represent a step in the decision tree, wherein a respective site is a website accessible from the base website of the data source;
generating, based on the decision tree, a list of tasks corresponding to each of the plurality of data sources, wherein each task includes instructions for how to extract the unstructured demographic data corresponding to the respective site corresponding to one of the one or more paths;
generating a user interface to be presented on a display, wherein the user interface indicates the list of tasks to be performed for each of the plurality of data sources and a status for the list of tasks;
navigating, based on the decision tree, within a corresponding data source from the base website to the respective site as specified by a specified path, wherein the specified path is a plurality of steps to navigate from the base website to the respective site where the unstructured demographic data is located;
parsing the unstructured demographic data from the respective site into separate categories;
storing the parsed unstructured demographic data in separate databases based on the separate categories; and
generating a report based on the parsed unstructured demographic data that displays the parsed unstructured demographic data in a structured format.
2. The method of claim 1 , wherein the navigating comprises iteratively accessing the respective site for a predetermined number of attempts when the corresponding data source or the respective site is initially inaccessible.
3. The method of claim 2 , further comprising receiving an error notification when the corresponding data source or the respective site is inaccessible after completing the predetermined number of attempts.
4. The method of claim 1 , wherein the user interface further indicates a color code indicator of a priority level of each of the plurality of data sources.
5. The method of claim 1 , further comprising managing a plurality of data extractors performing tasks on each of the plurality of data sources.
6. The method of claim 5 , wherein managing the plurality of data extractors comprises managing a maximum number of data extractors performing tasks on each of the plurality of data sources.
7. The method of claim 6 , wherein when the maximum number of data extractors for a first data source of the plurality of data sources is reached, the method further comprises assigning tasks of a second data source of the plurality of data sources having a same priority level as the first data source.
8. The method of claim 6 , wherein when the maximum number of data extractors for a first data source is reached, the method further comprises assigning tasks of a second data source of the plurality of data sources having a different priority level as the first data source.
9. The method of claim 5 , wherein managing the plurality of data extractors comprises periodically adjusting the plurality of data extractors performing tasks on the corresponding data source.
10. A non-transitory program storage device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform a method, the method comprising:
generating a decision tree for a data source of a plurality of data sources, wherein the decision tree comprises a base website at a root node and a plurality of respective sites represented as corresponding leaf nodes of the decision tree, wherein the decision tree specifies one or more paths to navigate from the base website to the plurality of respective sites containing unstructured demographic data for a healthcare provider, wherein each of the plurality of respective sites and the corresponding leaf nodes of the decision tree represent a step in the decision tree, wherein a respective site is a website accessible from the base website of the data source;
generating, based on the decision tree, a list of tasks corresponding to each of the plurality of data sources, wherein each task includes instructions for how to extract the unstructured demographic data corresponding to the respective site corresponding to one of the one or more paths;
generating a user interface to be presented on a display, wherein the user interface indicates the list of tasks to be performed for each of the plurality of data sources and a status for the list of tasks;
navigating, based on the decision tree, within a corresponding data source from the base website to the respective site as specified by a specified path, wherein the specified path is a plurality of steps to navigate from the base website to the respective site where the unstructured demographic data is located;
parsing the unstructured demographic data from the respective site into separate categories;
storing the parsed unstructured demographic data in separate databases based on the separate categories; and
generating a report based on the parsed unstructured demographic data that displays the parsed unstructured demographic data in a structured format.
11. The program storage device of claim 10 , wherein the navigating comprises iteratively accessing the respective site for a predetermined number of attempts when the corresponding data source or the respective site is initially inaccessible.
12. The program storage device of claim 11 , the instructions further comprising receiving an error notification when the corresponding data source or the respective site is inaccessible after completing the predetermined number of attempts.
13. The program storage device of claim 10 , wherein the user interface further indicates a color code indicator of a priority level of each of the plurality of data sources.
14. The program storage device of claim 10 , the instructions further comprising managing a plurality of data extractors performing tasks on each of the plurality of data sources.
15. The program storage device of claim 14 , wherein managing the plurality of data extractors comprises managing a maximum number of data extractors performing tasks on each of the plurality of data sources.
16. The program storage device of claim 15 , wherein when the maximum number of data extractors for a first data source of the plurality of data sources is reached, the method further comprises assigning tasks of a second data source of the plurality of data sources having a same priority level as the first data source.
17. The program storage device of claim 15 , wherein when the maximum number of data extractors for a first data source is reached, the method further comprises assigning tasks of a second data source of the plurality of data sources having a different priority level as the first data source.
18. The program storage device of claim 14 , wherein managing the plurality of data extractors comprises periodically adjusting the plurality of data extractors performing tasks on the corresponding data source.
19. A system comprising:
a first computing device comprising:
a first memory; and
a first processor communicatively coupled to the first memory and configured to:
generate a decision tree for a data source of a plurality of data sources, wherein the decision tree comprises a base website at a root node and a plurality of respective sites represented as corresponding leaf nodes of the decision tree, wherein the decision tree specifies one or more paths to navigate from the base website to the plurality of respective sites containing unstructured demographic data for a healthcare provider, wherein each of the plurality of respective sites and the corresponding leaf nodes of the decision tree represent a step in the decision tree, wherein a respective site is a website accessible from the base website of the data source;
generate a list of tasks for each of the plurality of data sources based on the decision tree, wherein each task includes instructions for how to extract the unstructured demographic data corresponding to the respective site of the one or more paths and comprises instructions for extracting the unstructured demographic data from the respective site;
assign a task from the list of tasks to a second computing device based on a priority level of a corresponding data source;
generate a user interface to be presented on a display, wherein the user interface indicates the list of tasks to be performed for each of the plurality of data sources and a status for the list of tasks; and
transmit the assigned task to the second computing device; the second computing device comprising:
a second memory; and
a second processor communicatively coupled to the second memory and configured to:
execute the assigned task to navigate, based on the decision tree, the corresponding data source to the respective site and extract the unstructured demographic data from the respective site based on the assigned task; and
transmit the extracted unstructured demographic data to the first computing device, wherein upon receipt of the extracted unstructured demographic data, the first processor is further configured to:
parse the extracted unstructured demographic data into separate categories;
store the parsed unstructured demographic data in separate databases based on the separate categories; and
generate a report based on the parsed unstructured demographic data that displays the parsed unstructured demographic data in a structured format.
20. The system of claim 19 , wherein the user interface further indicates a color code indicator of a priority level of each of the plurality of data sources.