Mobile web scraping
Methods, systems and computer program products implementing data aggregation using distributed Web scraping are disclosed. A mobile device can scrape one or more target sites to collect data from accounts of a particular user. The scraping can occur under scraping conditions as specified by the user. The scraping conditions can include conditions based on time, power, bandwidth, usage, or any combination of the above. The scraping conditions can ensure that the scraping occurs at time that is most convenient to the user, e.g., when sufficient bandwidth is available to the mobile device or the mobile device is not performing other tasks. The mobile device can upload the scraped data to a data aggregation server under submission conditions as specified by the user. The data aggregation server can aggregate the scraped data, enrich the aggregated data, and provide the enriched data to the user through Web access.
1. A method comprising:
receiving, by a mobile client device of a user and from a data aggregation server, a request to scrape account data associated with the user at one or more target sites corresponding to one or more service providers;
determining, by the mobile client device of the user based on a target database, a number of target sites from which to scrape the account data associated with the user of the mobile client device;
determining, by the mobile client device and at a first time, that an amount of time for scraping the number of target sites exceeds a pre-specified time;
in response to determining that the amount of time for scraping the number of target sites exceeds the pre-specified time, tracking, by the mobile client device, usage of the mobile client device;
determining, by the mobile client device and based on the usage, a second time for scraping the number of target sites;
in response to determining the second time for scraping the number of target sites, scraping, by the mobile client device and at the second time, the account data associated with the user of the mobile client device from the target sites, including providing respective user credentials to each target site, navigating pages of each target site, and retrieving the account data from the pages;
determining, by the mobile client device, whether a data submission condition is satisfied; and
in response to determining that the data submission condition is satisfied, providing the scraped account data associated with the user of the mobile client device from the mobile client device of the user to the data aggregation server.
2. The method of claim 1 , wherein the request is obtained by a scraping scheduler of the mobile client device of the user.
3. The method of claim 1 , wherein the request is sent from the data aggregation server according to a schedule specified by the user of the mobile client device.
4. The method of claim 1 , wherein:
the method further comprises determining, by the mobile client device of the user, whether a data scraping condition is satisfied;
the data scraping condition is specified on the mobile client device of the user and includes at least one of a first time condition, a first power condition, a first bandwidth condition, or a first device usage condition, and
the data submission condition is specified on the mobile client device of the user and includes at least one of a second time condition, a second power condition, a second bandwidth condition, or a second device usage condition.
5. The method of claim 1 , wherein:
the method further comprises determining, by the mobile client device of the user, whether a data scraping condition is satisfied; and
the data scraping condition specifies whether the scraping should occur after wakeup or before sleep.
6. The method of claim 1 , wherein:
the method further comprises determining, by the mobile client device of the user, whether a data scraping condition is satisfied; and
the data scraping condition specifies that the scraping shall occur when a Wi-Fi connection is present.
7. The method of claim 1 , wherein each condition is based on a user-specified timing mode, and automatic timing mode, and a mixed timing mode.
8. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors of a mobile client device, cause the one or more processors to perform operations comprising:
receiving, by the mobile client device of a user and from a data aggregation server, a request to scrape account data associated with the user at one or more target sites corresponding to one or more service providers;
determining, by the mobile client device of the user based on a target database, a number of target sites from which to scrape the account data associated with the user of the mobile client device;
determining, by the mobile client device and at a first time, that an amount of time for scraping the number of target sites exceeds a pre-specified time;
in response to determining that the amount of time for scraping the number of target sites exceeds the pre-specified time, tracking, by the mobile client device, usage of the mobile device;
determining, by the mobile client device and based on the usage, a second time for scraping the number of target sites;
in response to determining the second time for scraping the number of target sites, scraping, by the mobile client device and at the second time, the account data associated with the user of the mobile client device from the one or more target sites, including providing respective user credentials to each target site, navigating pages of each target site, and retrieving the account data from the pages;
determining, by the mobile client device whether a data submission condition is satisfied; and
in response to determining that the data submission condition is satisfied, providing the scraped account data associated with the user of the mobile client device from the mobile client device of the user to the data aggregation server.
9. The non-transitory computer-readable medium of claim 8 , wherein the request is obtained by a scraping scheduler of the mobile client device of the user.
10. The non-transitory computer-readable medium of claim 8 , wherein the request is sent from the data aggregation server according to a schedule specified by the user of the mobile client device.
11. The non-transitory computer-readable medium of claim 8 , wherein:
the operations further comprise determining, by the mobile client device of the user, whether a data scraping condition is satisfied;
the data scraping condition is specified on the mobile client device of the user and includes at least one of a first time condition, a first power condition, a first bandwidth condition, or a first device usage condition, and
the data submission condition is specified on the mobile client device of the user and includes at least one of a second time condition, a second power condition, a second bandwidth condition, or a second device usage condition.
12. The non-transitory computer-readable medium of claim 8 , wherein:
the operations further comprise determining, by mobile client device of the user, whether a data scraping condition is satisfied; and
the data scraping condition specifies whether the scraping should occur after wakeup or before sleep.
13. The non-transitory computer-readable medium of claim 8 , wherein:
the operations further comprise determining, by the mobile client device of the user, whether a data scraping condition is satisfied; and
the data scraping condition specifies that the scraping shall occur when a Wi-Fi connection is present.