Automated financial data aggregation

1. A computer-implemented method, comprising:

analyzing, in a computer, data for each webpage of a plurality of webpages that are not in a financial institution website;

generating, in the computer, a site map of a financial institution website based on the analyzed data, wherein generating the site map includes (i) crawling the financial institution website to identify one or more web pages in the financial institution website and (ii) determining respective categorizations for the one or more identified web pages based on (a) document structure of each webpage of the one or more identified web pages and (b) the analyzed data;

generating, automatically and in the computer, a site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website; and

extracting, in the computer, for one or more users, financial data from the financial institution website using the generated site-specific script.

2. The method of claim 1 , wherein determining, respective categorizations for the one or more identified web pages in the financial institution website further comprises:

determining a respective plurality of scores for each web page in the one or more identified web pages, each score in the plurality of scores indicating a confidence that the web page corresponds to a particular category;

determining, for each web page in the one or more identified web pages, whether a score in the respective plurality of scores satisfies a threshold; and

in response to determining that a score in the respective plurality of scores for a web page satisfies a threshold, associating the web page in the one or more web pages with a particular category corresponding to the score.

3. The method of claim 2 , further comprising:

in response to determining that no score in the respective plurality of scores for a web page satisfies a threshold, providing the web page to a user for manual categorization; and

associating the web page with a category specified by the user.

4. The method of claim 1 , wherein analyzing data for each webpage of the plurality of webpages that are not in a financial institution website comprises:

generating, based on one or more machine learning techniques, a data model that models a relationship between a serialized document object model (DOM) of web pages to respective categories of the web pages.

5. The method of claim 4 , wherein determining respective categorizations for the one or more identified web pages based on (a) document structure of each webpage in the one or more identified web pages and (b) the analyzed data comprises:

using the data model to categorize web pages in the financial institution website.

6. The method of claim 1 , wherein generating the site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website does not require human input.

7. The method of claim 1 , further comprising:

determining, in the computer, that a site-specific script for extracting financial data from the financial institution website is not available.

8. A computer-implemented method, comprising:

analyzing, in a computer, data for each web page portion in each webpage of a plurality of webpages that are not in a financial institution website;

generating, in the computer, a site map of a financial institution website based on the analyzed data, wherein generating the site map includes (i) crawling the financial institution website to identify one or more web page portions in one or more web pages in the financial institution website and (ii) determining respective categorizations for the one or more identified web page portions based on (a) document structure of each web page portion of the one or more identified web page portions and (b) the analyzed data;

generating, automatically and in the computer, a site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website; and

extracting, in the computer, for one or more users, financial data from the financial institution website using the generated site-specific script.

9. The method of claim 8 , wherein determining, respective categorizations for the one or more identified web pages portions in the one or more web pages in the financial institution website further comprises:

determining a respective plurality of scores for each web page portion in the one or more web pages, each score in the plurality of scores indicating a confidence that a web page portion corresponds to a particular category;

determining, for each web page portion, whether a score in the respective plurality of scores satisfies a threshold; and

in response to determining that a score in the respective plurality of scores for a web page portion satisfies a threshold, associating the web page portion with a particular category corresponding to the score.

10. The method of claim 9 , further comprising:

in response to determining that no score in the respective plurality of scores for a web page portion satisfies a threshold, providing the web page portion to a user for manual categorization; and

associating the web page with a category specified by the user.

11. The method of claim 8 , wherein analyzing, in the computer, data for each web page portion in each webpage of a plurality of webpages that are not in a financial institution website comprises:

generating, based on one or more machine learning techniques, a data model that models a relationship between a serialized document object model (DOM) of web page portions to respective categories of the web page portions.

12. The method of claim 11 , wherein determining respective categorizations for the one or more identified web page portions based on (a) document structure of each web page portion of the one or more identified web page portions and (b) the analyzed data comprises:

using the data model to categorize web page portions in the financial institution website.

13. The method of claim 8 , wherein generating the site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website does not require human input.

14. The method of claim 8 , further comprising:

determining, in the computer, that a site-specific script for extracting financial data from the financial institution website is not available.

15. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

analyzing, in a computer, data for each webpage of a plurality of webpages that are not in a financial institution website;

generating, in the computer, a site map of a financial institution website based on the analyzed data, wherein generating the site map includes (i) crawling the financial institution website to identify one or more web pages in the financial institution website and (ii) determining respective categorizations for the one or more identified web pages based on (a) document structure of each webpage of the one or more identified web pages and (b) the analyzed data;

generating, automatically and in the computer, a site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website; and

extracting, in the computer, for one or more users, financial data from the financial institution website using the generated site-specific script.

16. The computer storage medium of claim 15 , wherein determining, respective categorizations for the one or more identified web pages in the financial institution web site further comprises:

determining a respective plurality of scores for each web page in the one or more identified web pages, each score in the plurality of scores indicating a confidence that the web page corresponds to a particular category;

determining, for each web page in the one or more identified web pages, whether a score in the respective plurality of scores satisfies a threshold; and

in response to determining that a score in the respective plurality of scores for a web page satisfies a threshold, associating the web page in the one or more web pages with a particular category corresponding to the score.

17. The computer storage medium of claim 16 , wherein the operations further comprise:

in response to determining that no score in the respective plurality of scores for a web page satisfies a threshold, providing the web page to a user for manual categorization; and

associating the web page with a category specified by the user.

18. The computer storage medium of claim 15 , wherein analyzing, in the computer, data for each webpage of the plurality of webpages that are not in a financial institution website comprises:

generating, based on one or more machine learning techniques, a data model that models a relationship between a serialized document object model (DOM) of web pages to respective categories of the web pages.

19. The computer storage medium of claim 18 , wherein determining respective categorizations for the one or more identified web pages based on (a) document structure of each webpage in the one or more identified web pages and (b) the analyzed data comprises:

using the data model to categorize web pages in the financial institution website.

20. The computer storage medium of claim 15 , wherein generating the site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website does not require human input.

21. The computer storage medium of claim 15 , wherein the operations further comprise:

determining, in the computer, that a site-specific script for extracting financial data from the financial institution website is not available.

22. A computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

analyzing, in a computer, data for each web page portion in each webpage of a plurality of webpages that are not in a financial institution website;

generating, in the computer, a site map of a financial institution website based on the analyzed data, wherein generating the site map includes (i) crawling the financial institution website to identify one or more web page portions in one or more web pages in the financial institution website and (ii) determining respective categorizations for the one or more identified web page portions based on (a) document structure of each web page portion of the one or more identified web page portions and (b) the analyzed data;

generating, automatically and in the computer, a site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website; and

extracting, in the computer, for one or more users, financial data from the financial institution website using the generated site-specific script.

23. The computer storage medium of claim 22 , wherein determining, respective categorizations for the one or more identified web pages portions in the one or more web pages in the financial institution website further comprises:

determining a respective plurality of scores for each web page portion in the one or more web pages, each score in the plurality of scores indicating a confidence that a web page portion corresponds to a particular category;

determining, for each web page portion, whether a score in the respective plurality of scores satisfies a threshold; and

in response to determining that a score in the respective plurality of scores for a web page portion satisfies a threshold, associating the web page portion with a particular category corresponding to the score.

24. The computer storage medium of claim 23 , wherein the operations further comprise:

in response to determining that no score in the respective plurality of scores for a web page portion satisfies a threshold, providing the web page portion to a user for manual categorization; and

associating the web page with a category specified by the user.

25. The computer storage medium of claim 22 , wherein analyzing, in the computer, data for each web page portion in each webpage of a plurality of webpages that are not in a financial institution web site comprises:

generating, based on one or more machine learning techniques, a data model that models a relationship between a serialized document object model (DOM) of web page portions to respective categories of the web page portions.

26. The computer storage medium of claim 25 , wherein determining respective categorizations for the one or more identified web page portions based on (a) document structure of each web page portion of the one or more identified web page portions and (b) the analyzed data comprises:

using the data model to categorize web page portions in the financial institution website.

27. The computer storage medium of claim 22 , wherein generating the site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website does not require human input.

28. The computer storage medium of claim 22 wherein the operations further comprise:

determining, in the computer, that a site-specific script for extracting financial data from the financial institution website is not available.

29. A system comprising one or more computers programmed to perform operations comprising:

analyzing, in a computer, data for each webpage of a plurality of webpages that are not in a financial institution website;

generating, in the computer, a site map of a financial institution website based on the analyzed data, wherein generating the site map includes (i) crawling the financial institution website to identify one or more web pages in the financial institution website and (ii) determining respective categorizations for the one or more identified web pages based on (a) document structure of each webpage of the one or more identified web pages and (b) the analyzed data;

generating, automatically and in the computer, a site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website; and

extracting, in the computer, for one or more users, financial data from the financial institution website using the generated site-specific script.

30. The system of claim 29 , wherein determining, respective categorizations for the one or more identified web pages in the financial institution website further comprises:

determining a respective plurality of scores for each web page in the one or more identified web pages, each score in the plurality of scores indicating a confidence that the web page corresponds to a particular category;

determining, for each web page in the one or more identified web pages, whether a score in the respective plurality of scores satisfies a threshold; and

in response to determining that a score in the respective plurality of scores for a web page satisfies a threshold, associating the web page in the one or more web pages with a particular category corresponding to the score.

31. The system of claim 30 , wherein the operations further comprise:

in response to determining that no score in the respective plurality of scores for a web page satisfies a threshold, providing the web page to a user for manual categorization; and

associating the web page with a category specified by the user.

32. The system of claim 29 , wherein analyzing, in the computer, data for each webpage of the plurality of webpages that are not in a financial institution website comprises:

generating, based on one or more machine learning techniques, a data model that models a relationship between a serialized document object model (DOM) of web pages to respective categories of the web pages.

33. The system of claim 32 , wherein determining respective categorizations for the one or more identified web pages based on (a) document structure of each webpage in the one or more identified web pages and (b) the analyzed data comprises:

using the data model to categorize web pages in the financial institution website.

34. The system of claim 29 , wherein generating the site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website does not require human input.

35. The system of claim 29 , wherein the operations further comprise:

determining, in the computer, that a site-specific script for extracting financial data from the financial institution website is not available.

36. A system comprising one or more computers programmed to perform operations comprising:

analyzing, in a computer, data for each web page portion in each webpage of a plurality of webpages that are not in a financial institution website;

generating, in the computer, a site map of a financial institution website based on the analyzed data, wherein generating the site map includes (i) crawling the financial institution website to identify one or more web page portions in one or more web pages in the financial institution website and (ii) determining respective categorizations for the one or more identified web page portions based on (a) document structure of each web page portion of the one or more identified web page portions and (b) the analyzed data;

generating, automatically and in the computer, a site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website; and

extracting, in the computer, for one or more users, financial data from the financial institution website using the generated site-specific script.

37. The system of claim 36 , wherein determining, respective categorizations for the one or more identified web pages portions in the one or more web pages in the financial institution website further comprises:

determining a respective plurality of scores for each web page portion in the one or more web pages, each score in the plurality of scores indicating a confidence that a web page portion corresponds to a particular category;

determining, for each web page portion, whether a score in the respective plurality of scores satisfies a threshold; and

in response to determining that a score in the respective plurality of scores for a web page portion satisfies a threshold, associating the web page portion with a particular category corresponding to the score.

38. The system of claim 37 , wherein the operations further comprise:

in response to determining that no score in the respective plurality of scores for a web page portion satisfies a threshold, providing the web page portion to a user for manual categorization; and

associating the web page with a category specified by the user.

39. The system of claim 36 , wherein analyzing, in the computer, data for each web page portion in each webpage of a plurality of webpages that are not in a financial institution website comprises:

generating, based on one or more machine learning techniques, a data model that models a relationship between a serialized document object model (DOM) of web page portions to respective categories of the web page portions.

40. The system of claim 39 , wherein determining respective categorizations for the one or more identified web page portions based on (a) document structure of each web page portion of the one or more identified web page portions and (b) the analyzed data comprises:

using the data model to categorize web page portions in the financial institution website.

41. The system of claim 36 , wherein generating the site-specific script for extracting financial data from the financial institution website based on the site map of the financial institution website does not require human input.

42. The system of claim 36 wherein the operations further comprise:

determining, in the computer, that a site-specific script for extracting financial data from the financial institution website is not available.