Methods and systems for aggregating data from webpages using path attributes
Methods and systems that improve the ability of a media guidance application to aggregate information from one or more sources and one or more mediums. For example, the media guidance application may automatically collect information based on attributes associated with information of a particular type. Specifically, the media guidance application may determine based on comparison with verified training data that one source or medium typically associates information of a particular type, for example, “Actor,” with one or more path attributes, for example, a location in a directory structure. The media guidance application may then search the source or medium for the one or more path attributes. Upon detecting the one or more path attributes, the media guidance application may designate any sub-set of information associated with the one or more path attributes as a particular type of information.
1. A method of automatically aggregating data from a webpage using a path attribute, the method comprising:
retrieving a first webpage of a website, wherein the first webpage is associated with a first media asset;
selecting, from a database of a plurality of verified media descriptors, a verified media descriptor for the first media asset, wherein each of the plurality of verified media descriptors is associated with one of a plurality of media descriptor types;
locating a webpage element of the first webpage by matching the verified media descriptor to a first text in the webpage element;
identifying a first path attribute of the webpage element;
in response to identifying the first path attribute of the webpage element, generating a mapping of the first path attribute to a media descriptor type, wherein the media descriptor type is associated with the verified media descriptor;
retrieving a second webpage of the website, wherein the second webpage is associated with a second media asset;
identifying a second webpage element on the second webpage by locating the first path attribute in the second webpage element;
extracting a second text from the second webpage element;
associating a second media descriptor with the media descriptor type based on the mapping, wherein the second media descriptor comprises the second text; and
storing the association of the second media descriptor with the media descriptor type in a database of learned media descriptors for the second media asset.
2. The method of claim 1 , wherein the first path attribute includes:
a path; and
a format attribute which includes at least one of:
a location;
a font;
a size; and
a color.
3. The method of claim 1 , wherein the database of the plurality of verified media descriptors for the first media asset is populated with manually verified media descriptors for the first media asset.
4. The method of claim 1 , wherein generating the mapping of the first path attribute to the media descriptor type further comprises generating code that indicates that the first media descriptor type is associated with the first path attribute in any webpage of the website.
5. The method of claim 1 , further comprising:
comparing the second media descriptor in the database of learned media descriptors for the second media asset to a second verified media descriptor of the media descriptor type from a database of a plurality of verified media descriptors for the second media asset; and
determining a first accuracy of the mapping based on the comparison.
6. The method of claim 5 , further comprising:
comparing the first accuracy to a second accuracy of a different mapping, wherein the different mapping maps a second path attribute to the media descriptor type; and
in response to determining that the first accuracy is greater than the second accuracy, selecting the mapping for use in associating the second media descriptor with the media descriptor type.
7. The method of claim 6 , wherein the first path attribute is randomly selected from a plurality of path attributes associated with a first media descriptor, wherein the first media descriptor is associated with the first media descriptor type.
8. The method of claim 7 , wherein the first media descriptor is randomly selected from a plurality of media descriptors for the first media asset.
9. The method of claim 1 , further comprising:
accessing a media guidance database containing metadata used to generate media asset listings in an electronic program guide;
comparing entries in the media guidance database with the database of learned media descriptors;
determining whether a first field in the media guidance database corresponding to the second media asset and a second field in the database of learned media descriptors corresponding to the second media asset are different; and
in response to determining that the first field is different than the second field, updating the first field in the media guidance database to match the second field in the database of learned media descriptors.
10. The method of claim 9 , wherein determining whether the first field in the media guidance database corresponding to the second media asset and the second field in the database of learned media descriptors corresponding to the second media asset are different comprises:
retrieving a first stored descriptor from the first field;
retrieving a second stored descriptor from the second field;
comparing corresponding characters of the first stored descriptor with those of the second stored descriptor; and
in response to a first character of the first stored descriptor not corresponding to a second character of the second stored descriptor, determining the first field and the second field are different.
11. A system for automatically aggregating data from a webpage using a path attribute, the system comprising:
control circuitry configured to:
retrieve a first webpage of a website, wherein the first webpage is associated with a first media asset;
select, from a database of a plurality of verified media descriptors, a verified media descriptor for the first media asset, wherein each of the plurality of verified media descriptors is associated with one of a plurality of media descriptor types;
locate a webpage element of the first webpage by matching the verified media descriptor to a first text in the webpage element;
identify a first path attribute of the webpage element;
in response to identifying the first path attribute of the webpage element,
generate a mapping of the first path attribute to a media descriptor type, wherein the media descriptor type is associated with the verified media descriptor;
retrieve a second webpage of the website, wherein the second webpage is associated with a second media asset;
identify a second webpage element on the second webpage by locating the first path attribute in the second webpage element;
extract a second text from the second webpage element;
associate a second media descriptor with the media descriptor type based on the mapping, wherein the second media descriptor comprises the second text; and
storage circuitry configured to:
store the association of the second media descriptor with the media descriptor type in a database of learned media descriptors for the second media asset.
12. The system of claim 11 , wherein the first path attribute includes:
a path; and
a format attribute which includes at least one of:
a location;
a font;
a size; and
a color.
13. The system of claim 11 , wherein the database of the plurality of verified media descriptors for the first media asset is populated with manually verified media descriptors for the first media asset.
14. The system of claim 11 , wherein the control circuitry is configured to generate the mapping of the first path attribute to the media descriptor type by generating code that indicates that the first media descriptor type is associated with the first path attribute in any webpage of the website.
15. The system of claim 11 , wherein the control circuitry is further configured to:
compare the second media descriptor in the database of learned media descriptors for the second media asset to a second verified media descriptor of the media descriptor type from a database of a plurality of verified media descriptors for the second media asset; and
determine a first accuracy of the mapping based on the comparison.
16. The system of claim 15 , wherein the control circuitry is further configured to:
compare the first accuracy to a second accuracy of a different mapping, wherein the different mapping maps a second path attribute to the media descriptor type; and
in response to determining that the first accuracy is greater than the second accuracy, selecting the mapping for use in associating the second media descriptor with the media descriptor type.
17. The system of claim 16 , wherein the first path attribute is randomly selected from a plurality of path attributes associated with a first media descriptor, wherein the first media descriptor is associated with the first media descriptor type.
18. The system of claim 17 , wherein the first media descriptor is randomly selected from a plurality of media descriptors for the first media asset.
19. The system of claim 11 , wherein the control circuitry is further configured to:
access a media guidance database containing metadata used to generate media asset listings in an electronic program guide;
compare entries in the media guidance database with the database of learned media descriptors;
determine whether a first field in the media guidance database corresponding to the second media asset and a second field in the database of learned media descriptors corresponding to the second media asset are different; and
in response to determining that the first field is different than the second field, update the first field in the media guidance database to match the second field in the database of learned media descriptors.
20. The system of claim 19 , wherein the control circuitry configured to determine whether the first field in the media guidance database corresponding to the second media asset and the second field in the database of learned media descriptors corresponding to the second media asset are different is further configured to:
retrieve a first stored descriptor from the first field;
retrieve a second stored descriptor from the second field;
compare corresponding characters of the first stored descriptor with those of the second stored descriptor; and
in response to a first character of the first stored descriptor not corresponding to a second character of the second stored descriptor, determine the first field and the second field are different.