Efficient data scraping and deduplication system for registered sex offender queries
An improved query system in which registered sex offender information published in state and territory databases is obtained using an improved data scraper, merged, cleaned, and/or enriched to remove inaccuracies is described herein. Users can then perform a single search or multiple searches simultaneously (e.g., perform batch searches) and view the results in a single content page. The improved query system can use the merged and/or cleaned data to generate a more limited set of search results, reducing the amount of mismatched or inaccurate information that a user may have to navigate through. Thus, users can perform one or more searches and view accurate search results without having to navigate through many different windows or content pages or perform additional navigational steps to identify relevant information.
1. A system comprising:
a processed data store configured to store data records corresponding to a plurality of registered sex offenders; and
a computing system comprising a processor and computer-executable instructions, wherein the computer-executable instructions, when executed, cause the computing system to:
obtain first content page source code from a first external data source;
generate a first document object model (DOM) tree using the first content page source code, wherein the DOM tree comprises a first path corresponding to a first registered sex offender and a second path corresponding to a second registered sex offender;
request second content page source code of a content page corresponding to the first registered sex offender;
obtain the second content page source code;
modify the first path of the DOM tree to include a pointer;
scrape the second content page source code to identify first data corresponding to the first registered sex offender;
request third content page source code of a content page corresponding to the second registered sex offender;
pause, at a first time, a scraping process in response to the third content page source code not being obtained;
using the pointer included in the first path of the DOM tree, determine that the second content page source code corresponding to the first registered sex offender has been obtained and request the third content page source code corresponding to the second registered sex offender at a second time after the first time; and
resume the scraping process.
2. The system of claim 1 , wherein the computer-executable instructions, when executed, further cause the computing system to:
ping a server hosting the content page;
determine that the server is operational in response to the ping; and
resume the scraping process in response to the determination that the server is operational.
3. The system of claim 2 , wherein the second time occurs after the server is operational.
4. The system of claim 1 , wherein the computer-executable instructions, when executed, further cause the computing system to:
scrape the third content page source code to identify second data corresponding to the second registered sex offender; and
modify the second path of the DOM tree to include the pointer.
5. The system of claim 1 , wherein the third content page source code is not obtained because a server hosting the content page has failed.
6. The system of claim 1 , wherein the pointer being included in the first path of the DOM tree indicates that the second content page source code associated with the first registered sex offender has been scraped.
7. The system of claim 1 , wherein the computer-executable instructions, when executed, further cause the computing system to:
parse the DOM tree to identify that the pointer is included in the first path of the DOM tree; and
request the third content page source code corresponding to the second registered sex offender in response to a determination that the pointer is located in a portion of the DOM tree corresponding to the first registered sex offender.
8. The system of claim 1 , wherein the first path and the second path are different paths.
9. The system of claim 1 , wherein the first data comprises a first name, a middle initial, and a last name.
10. A computer-implemented method comprising:
obtaining first content page source code from a first external data source;
generating a first document object model (DOM) tree using the first content page source code, wherein the DOM tree comprises a first path corresponding to a first registered sex offender and a second path corresponding to a second registered sex offender;
requesting second content page source code of a content page corresponding to the first registered sex offender;
obtaining the second content page source code;
modifying the first path of the DOM tree to include a pointer;
scraping the second content page source code to identify first data corresponding to the first registered sex offender;
requesting third content page source code of a content page corresponding to the second registered sex offender;
pausing, at a first time, a scraping process in response to the third content page source code not being obtained;
using the pointer included in the first path of the DOM tree, determining that the second content page source code corresponding to the first registered sex offender has been obtained and requesting the third content page source code corresponding to the second registered sex offender at a second time after the first time; and
resuming the scraping process.
11. The computer-implemented method of claim 10 , wherein resuming the scraping process further comprises:
pinging a server hosting the content page;
determining that the server is operational in response to the ping; and
resuming the scraping process in response to the determination that the server is operational.
12. The computer-implemented method of claim 11 , wherein the second time occurs after the server is operational.
13. The computer-implemented method of claim 10 , further comprising:
scraping the third content page source code to identify second data corresponding to the second registered sex offender; and
modifying the second path of the DOM tree to include the pointer.
14. The computer-implemented method of claim 10 , wherein the third content page source code is not obtained because a server hosting the content page has failed.
15. The computer-implemented method of claim 10 , wherein the pointer being included in the first path of the DOM tree indicates that the second content page source code associated with the first registered sex offender has been scraped.
16. The computer-implemented method of claim 10 , wherein requesting the third content page source code further comprises:
parsing the DOM tree to identify that the pointer is included in the first path of the DOM tree; and
requesting the third content page source code corresponding to the second registered sex offender in response to a determination that the pointer is located in a portion of the DOM tree corresponding to the first registered sex offender.
17. The computer-implemented method of claim 10 , wherein the first path and the second path are different paths.
18. The computer-implemented method of claim 10 , wherein the first data comprises a first name, a middle initial, and a last name.
19. Non-transitory, computer-readable storage media comprising computer-executable instructions for obtaining registered sex offender data, wherein the computer-executable instructions, when executed by a computer system, cause the computer system to:
obtain first content page source code from a first external data source;
generate a first document object model (DOM) tree using the first content page source code, wherein the DOM tree comprises a first path corresponding to a first registered sex offender and a second path corresponding to a second registered sex offender;
request second content page source code of a content page corresponding to the first registered sex offender;
obtain the second content page source code;
modify the first path of the DOM tree to include a pointer;
scrape the second content page source code to identify first data corresponding to the first registered sex offender;
request third content page source code of a content page corresponding to the second registered sex offender;
pause, at a first time, a scraping process in response to the third content page source code not being obtained;
using the pointer included in the first path of the DOM tree, determine that the second content page source code corresponding to the first registered sex offender has been obtained and request the third content page source code corresponding to the second registered sex offender at a second time after the first time; and
resume the scraping process.
20. The non-transitory, computer-readable storage media of claim 19 , wherein the computer-executable instructions, when executed, further cause the computer system to:
ping a server hosting the content page;
determine that the server is operational in response to the ping; and
resume the scraping process in response to the determination that the server is operational.