Discovery of new business openings using web content analysis
In general, embodiments of the present invention provide systems, methods and computer readable media for identifying a new business based on programmatically analyzing content received from online sources and, as a result, discovering one or more references to the business. In embodiments, the system stores historical data representing previously identified new businesses and then uses attributes of those businesses in search queries to receive related content. Additionally or alternatively, the system stores data representing online sources that historically provided content containing references to new businesses and then continues to access those sources for additional content. In embodiments, the system performs content analysis on structured and/or unstructured content. In some embodiments, analysis of content received from a particular online source includes a source-specific algorithm that takes a source-specific representation of the content as input and produces a result indicating the likelihood that the content includes a new business reference.
1. An apparatus comprising at least one processor and at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to:
retrieve search results comprising content data from an online source, the search results returned by a search engine;
extract, using a pattern recognition algorithm, references to new businesses that are included in the content data, wherein a new business is one of a newly opened business or a business that is about to open;
verify each of a set of new businesses in the extracted references by determining whether data representing the new business is stored in a repository;
update a source search index based on source data quality signals calculated based upon verification of the content data;
determine, based on at least a content feed, an initial confidence rating associated with the online source;
update, by increasing or decreasing, the initial confidence rating associated with the online source based on at least a verification result; and
modify, based on the updated initial confidence rating, the pattern recognition algorithm.
2. The apparatus of claim 1 , wherein the content data comprises an address of the new business, and wherein verifying a new business comprises a determination of a valid address.
3. The apparatus of claim 1 , wherein the content data comprises an opening date of the new business, and wherein verifying a new business comprises a determination that the opening date is a first opening.
4. The apparatus of claim 1 , wherein the content data comprises an opening date of the new business, and wherein verifying a new business comprises determining if the opening date is from content received during a predetermined time period.
5. The apparatus of claim 1 , wherein when a respective reference to a previously unknown business is verified, data representing the previously unknown business is stored in the repository.
6. The apparatus of claim 1 , further caused to transmit to a first device at least a link to the online source and the updated initial confidence rating associated with the online source.
7. A computer-implemented method comprising:
retrieving search results comprising content data from an online source, the search results returned by a search engine;
extracting, using a pattern recognition algorithm, references to new businesses that are included in the content data, wherein a new business is one of a newly opened business or a business that is about to open;
verifying each of a set of new businesses in the extracted references by determining whether data representing the new business is stored in a repository;
updating a source search index based on source data quality signals calculated based upon verification of the content data;
determining, based on at least a content feed, an initial confidence rating associated with the online source;
updating, by increasing or decreasing, the initial confidence rating associated with the online source based on at least a verification result; and
modifying, based on the updated initial confidence rating, the pattern recognition algorithm.
8. The method of claim 7 , wherein the content data comprises an address of the new business, and wherein verifying a new business comprises a determination of a valid address.
9. The method of claim 7 , wherein the content data comprises an opening date of the new business, and wherein verifying a new business comprises a determination that the opening date is a first opening.
10. The method of claim 7 , wherein the content data comprises an opening date of the new business, and wherein verifying a new business comprises determining if the opening date is from content received during a predetermined time period.
11. The method of claim 7 , wherein when a respective reference to a previously unknown business is verified, data representing the previously unknown business is stored in the repository.
12. The method of claim 7 , further comprising:
transmitting to a first device at least a link to the online source and the updated initial confidence rating associated with the online source.
13. A computer program product comprising at least one non-transitory storage medium for storing computer program code, that, when executed by an apparatus, cause the apparatus to:
retrieve search results comprising content data from an online source, the search results returned by at least a search engine;
extract, using a pattern recognition algorithm, references to new businesses that are included in the content data, wherein a new business is one of a newly opened business or a business that is about to open;
verify each of a set of new businesses in the extracted references by determining whether data representing the new business is stored in a repository;
update a source search index based on source data quality signals calculated based upon verification of the content data;
determine, based on at least a content feed, an initial confidence rating associated with the online source;
update, by increasing or decreasing, the initial confidence rating associated with the online source based on at least a verification result;
modify, based on the updated initial confidence rating, the pattern recognition algorithm; and
transmit to a first device at least a link to the online source and the updated initial confidence rating associated with the online source.
14. The computer program product of claim 13 , wherein the content data comprises an address of the new business; and
wherein verifying a new business comprises a determination of a valid address.
15. The computer program product of claim 13 , wherein the content data comprises an opening date of the new business; and
wherein verifying a new business comprises a determination that the opening date is a first opening.
16. The computer program product of claim 13 , wherein the content data comprises an opening date of the new business; and
wherein verifying a new business comprises determining if the opening date is from content received during a predetermined time period.
17. The computer program product of claim 13 , wherein when a respective reference to a previously unknown business is verified, data representing the previously unknown business is stored in the repository.