Systems and methods for discovery and tracking of web-based advertisements
Systems and methods are provided for discovering advertisements on publisher web pages and for identifying placement pathways by which discovered advertisements have been placed on the publisher web pages. An advertisement tracking and discovery system may use multiple web crawler applications to explore multiple publisher websites. The web crawler applications may gather advertisement data that includes times associated with each request made by the web crawler application. The system may use the gathered advertisement data and the times associated with each request to determine the placement pathways by which discovered advertisements have been placed. Each placement pathway may include one or more advertising channels or combinations of advertising channels. The system may accumulate and aggregate advertising data associated with the advertisements and the placement pathways and display the aggregated advertising data to a customer.
1. A system, comprising:
memory storing code; and
processing circuitry configured to run the code to:
load a plurality of linked resources included in a web page that has been loaded by a browser;
obtain, for each linked resource, a response body and a request time;
identify an advertisement based on the response bodies;
identify, in the request times, an advertisement request time associated with the advertisement;
identify a subset of the response bodies that have associated request times that are earlier than the advertisement request time;
extract a reference to the advertisement in the subset of the response bodies; and
identify a placement pathway for the advertisement based on the reference.
2. The system of claim 1 wherein the processing circuitry is configured to run the code to identify the placement pathway by determining a source that provided the advertisement responsive to the web page being loaded by the browser.
3. The system of claim 1 wherein the processing circuitry is configured to run the code to:
send a request to a publisher server associated with the web page;
receiving a response from the publisher server; and
load the plurality of linked resources by sending additional requests based on the received response from the publisher server.
4. The system of claim 3 , wherein processing circuitry is configured to run the code to identify the placement pathway by determining a cause and effect relationship between at least some of the additional requests based on the request times.
5. The system of claim 1 , wherein the placement pathway comprises an advertising channel, an advertising exchange, or a demand side platform.
6. The system of claim 5 , further comprising data processing and storage equipment, wherein:
the processing circuitry is further configured to run the code to identify an advertisement block of the web page; and
the data processing and storage equipment is configured to store ad block data for the identified advertisement block.
7. The system of claim 6 , wherein the processing circuitry is further configured to run the code to identify an advertiser associated with the identified advertisement block, wherein the ad block data includes a name of the advertiser.
8. The system of claim 7 , wherein the processing circuitry is further configured to run the code to identify an advertiser creative associated with the identified advertiser, wherein the ad block data includes a file including the creative.
9. The system of claim 5 , wherein the advertising channel, the advertising exchange, and the demand side platform are each different from the advertiser.
10. The system of claim 1 , further comprising data analysis and storage equipment configured to:
retrieve the identified placement pathway from a database; and
generate aggregated advertisement data by aggregating at least the identified placement pathway retrieved from the database with additional advertisement information.
11. The system of claim 10 , further comprising user interface equipment, wherein the data analysis and storage equipment is configured to provide the aggregated advertisement data to the user interface equipment, and wherein the user interface equipment is configured to display the aggregated advertisement data.
12. A method, comprising:
with a web browser application, loading linked resources included in a web page that has been loaded by the web browser application;
obtaining, with the web browser application for each of the linked resources, a response body and a request time;
with computing equipment that is associated with the web browser application, identifying an advertisement included in the web page based on the response bodies;
with the computing equipment, identifying, in the request times, an advertisement request time associated with the identified advertisement;
with the computing equipment, identifying a subset of the response bodies that have associated request times that are earlier than the advertisement request time;
with the computing equipment, extracting a reference to the identified advertisement from the identified subset; and
with the computing equipment, identifying a placement pathway by which the identified advertisement was included in the web page based on the extracted reference.
13. The method of claim 12 , wherein identifying the placement pathway further comprises determining a cause and effect relationship between at least some of the subset of the response bodies based on the request times.
14. The method of claim 13 , further comprising:
with the computing equipment, identifying an advertisement block included in the web page;
with the computing equipment, identifying an advertiser associated with the identified advertisement block; and
with data processing and storage equipment, storing ad block data for the identified advertisement block that includes a name of the advertiser.
15. The method of claim 14 , further comprising:
with the computing equipment, identifying an advertiser creative that is associated with the identified advertiser and the identified advertisement block; and
with the data processing and storage equipment, storing a file including the creative in the ad block data.
16. A method, comprising:
with a web browser, loading linked resources included in a web page;
obtaining, with the web browser for each linked resource, a response body and a request time;
with computing equipment that is associated with the web browser, identifying an advertisement that is included in the web page based on the response bodies;
with the computing equipment, determining a cause and effect relationship between at least some of the response bodies based on the request times; and
with the computing equipment, identifying a placement pathway by which the advertisement was included in the web page based on the determined cause and effect relationship.
17. The method of claim 16 , further comprising generating a site-specific data tree based at least on the identified placement pathway.
18. The method of claim 17 , further comprising:
identifying additional advertisements included in additional web pages;
identifying additional placement pathways for the identified additional advertisements;
generating an additional site-specific data tree based at least on the identified additional placement pathways; and
extracting data associated with an advertising network from the site-specific data tree and the additional site-specific data tree.
19. The method of claim 18 , further comprising:
with the computing equipment, determining, based on the additional placement pathways, which of a plurality of advertising channels has placed the additional advertisements into the additional web pages.
20. The method of claim 19 , further comprising:
with the computing equipment, generating aggregated advertisement data relating the additional advertisements and the plurality of advertising channels; and
providing a web-based user interface portal that includes a links to the aggregated advertisement data.