Discovery of new business openings using web content analysis
View Patent ↗In general, embodiments of the present invention provide systems, methods and computer readable media for identifying a new business based on programmatically analyzing content received from online sources and, as a result, discovering one or more references to the business. In embodiments, the system stores historical data representing previously identified new businesses and then uses attributes of those businesses in search queries to receive related content. Additionally or alternatively, the system stores data representing online sources that historically provided content containing references to new businesses and then continues to access those sources for additional content. In embodiments, the system performs content analysis on structured and/or unstructured content. In some embodiments, analysis of content received from a particular online source includes a source-specific algorithm that takes a source-specific representation of the content as input and produces a result indicating the likelihood that the content includes a new business reference.
1. A computer-implemented method for automatically identifying references to a new business within content returned from an online source, the method comprising:
receiving content data from the online source;
automatically determining, using an analysis of the content data, whether the content data includes at least one reference to the new business, wherein the analysis of the content data includes implementing a particular pattern recognition algorithm that is configured to process one or more text patterns extracted from the content data;
in an instance in which the content data includes the new business reference, determining whether data representing the new business is already stored in a business repository; and
in an instance in which the data representing the new business is not already stored in the business repository,
automatically determining, based on at least one data quality signal associated with the content data, whether the new business reference is verified; and
storing data representing the new business in the business repository in an instance in which the new business reference is verified.
2. The method of claim 1 , further comprising:
in an instance in which the online source has provided content referencing at least one previously unknown new business, and wherein a calculated confidence rating is associated with the online source,
updating the confidence rating associated with the online source based in part on the data quality signal.
3. The method of claim 2 , wherein calculating the confidence rating associated with the online source comprises:
periodically receiving content data from the online source within a predetermined time period;
calculating a total of references to different verified new businesses within the content data received within the time period; and
calculating the confidence rating associated with the online source based in part on the total of references.
4. The method of claim 1 , further comprising:
in an instance in which the new business reference is not verified, not storing the data representing the new business in the business repository.
5. The method of claim 1 , further comprising:
determining whether data representing the online source is stored in a source search index; and
in an instance in which the data representing the online source is not stored in the source search index, updating the source search index by storing the data representing the online source in the source search index.
6. The method of claim 5 , further comprising:
determining whether the analysis of the content data includes a source-specific pattern analysis algorithm;
in an instance in which the analysis includes the source-specific pattern analysis algorithm,
generating a source-specific representation of the new business reference that is included in the content data; and
updating the source-specific pattern analysis algorithm using the source-specific representation.
7. The method of claim 6 , wherein the source-specific pattern analysis algorithm is a trainable pattern recognition algorithm, and wherein updating the source-specific pattern analysis algorithm using the source-specific representation of the new business reference comprises:
updating a training data set using the source-specific representation of the new business reference; and
updating the trainable pattern recognition algorithm using the updated training data set.
8. The method of claim 1 , wherein the content data is unstructured.
9. The method of claim 1 , wherein preceding the receiving of the content data comprises:
receiving data representing a new business;
generating a business query, wherein the business query includes at least one search term derived from the data representing the new business;
submitting the business query to at least one search engine; and
responsive to receiving search results from the business query, extracting the content data from the search results.
10. An apparatus comprising:
at least one processor; and
at least one memory including computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to perform operations for automatically identifying references to a new business within content returned from an online source, the operations comprising:
receiving content data from the online source;
automatically determining, using an analysis of the content data, whether the content data includes at least one reference to the new business, wherein the analysis of the content data includes implementing a particular pattern recognition algorithm that is configured to process one or more text patterns extracted from the content data;
in an instance in which the content data includes the new business reference, determining whether data representing the new business is already stored in a business repository; and
in an instance in which the data representing the new business is not already stored in the business repository,
automatically determining, based on at least one data quality signal associated with the content data, whether the new business reference is verified; and
storing data representing the new business in the business repository in an instance in which the new business reference is verified.
11. The apparatus of claim 10 , wherein the operations further comprise:
in an instance in which the online source has provided content referencing at least one previously unknown new business, and wherein a calculated confidence rating is associated with the online source,
updating the confidence rating associated with the online source based in part on the data quality signal.
12. The apparatus of claim 10 , wherein the operations further comprise:
in an instance in which the new business reference is not verified, not storing the data representing the new business in the business repository.
13. The apparatus of claim 12 , wherein calculating the confidence rating associated with the online source comprises:
periodically receiving content data from the online source within a predetermined time period;
calculating a total of references to different verified new businesses within the content data received within the time period; and
calculating the confidence rating associated with the online source based in part on the total of references.
14. The apparatus of claim 10 , wherein the operations further comprise:
determining whether data representing the online source is stored in a source search index; and
in an instance in which the data representing the online source is not stored in the source search index, updating the source search index by storing the data representing the online source in the source search index.
15. The apparatus of claim 14 , wherein the operations further comprise:
determining whether the analysis of the content data includes a source-specific pattern analysis algorithm;
in an instance in which the analysis includes the source-specific pattern analysis algorithm,
generating a source-specific representation of the new business reference that is included in the content data; and
updating the source-specific pattern analysis algorithm using the source-specific representation.
16. The apparatus of claim 15 , wherein the source-specific pattern analysis algorithm is a trainable pattern recognition algorithm, and wherein updating the source-specific pattern analysis algorithm using the source-specific representation of the new business reference comprises:
updating a training data set using the source-specific representation of the new business reference; and
updating the trainable pattern recognition algorithm using the updated training data set.
17. The apparatus of claim 10 , wherein the content data is unstructured.
18. The apparatus of claim 10 , wherein the operations further comprise:
receiving data representing a new business;
generating a business query, wherein the business query includes at least one search term derived from the data representing the new business;
submitting the business query to at least one search engine; and
responsive to receiving search results from the business query, extracting the content data from the search results.
19. A computer program product comprising:
at least one computer readable non-transitory memory medium having program code instructions stored thereon, the program code instructions which when executed by an apparatus cause the apparatus at least to perform operations for automatically identifying references to a new business within content returned from an online source, the operations comprising:
receiving content data from the online source;
automatically determining, using an analysis of the content data, whether the content data includes at least one reference to the new business wherein the analysis of the content data includes implementing a particular pattern recognition algorithm that is configured to process one or more text patterns extracted from the content data;
in an instance in which the content data includes the new business reference, determining whether data representing the new business is already stored in a business repository; and
in an instance in which the data representing the new business is not already stored in the business repository,
automatically determining, based on at least one data quality signal associated with the content data, whether the new business reference is verified; and
storing data representing the new business in the business repository in an instance in which the new business reference is verified.
20. The computer program product of claim 19 , wherein the operations further comprise:
determining whether data representing the online source is stored in a source search index;
in an instance in which the data representing the online source is not stored in the source search index, updating the source search index by storing the data representing the online source in the source search index;
determining whether the analysis of the content data includes a source-specific pattern analysis algorithm;
in an instance in which the analysis includes the source-specific pattern analysis algorithm,
generating a source-specific representation of the new business reference that is included in the content data; and
updating the source-specific pattern analysis algorithm using the source-specific representation.
21. The computer program product of claim 20 , wherein the source-specific pattern analysis algorithm is a trainable pattern recognition algorithm, and wherein updating the source-specific pattern analysis algorithm using the source-specific representation of the new business reference comprises:
updating a training data set using the source-specific representation of the new business reference; and
updating the trainable pattern recognition algorithm using the updated training data set.
22. The computer program product of claim 19 , wherein the content data is unstructured.
23. The computer program product of claim 19 , wherein the operations further comprise:
receiving data representing a new business;
generating a business query, wherein the business query includes at least one search term derived from the data representing the new business;
submitting the business query to at least one search engine; and
responsive to receiving search results from the business query, extracting the content data from the search results.