System and method for monitoring and analyzing internet traffic
Systems, methods and apparatus for analyzing Internet traffic. In an aspect, a method receives at a server from a client device a report request for a report related to web site traffic; in response to the report request, sends from the server web site traffic data and application code to the client device. The application code has instructions that cause the client device to: generate a report to display the web site traffic data, time the display of the web site traffic data, periodically request updated web site traffic data according to the time of the display, and update the report with the updated web site traffic data; and the method sends from the server to the client device the updated web site traffic data in response to the request for updated web site traffic data.
1. A computer-implemented method, comprising:
accessing visitor data from a log file that is separate from a persistent database storing visitor data content describing web traffic activity for a web server, the visitor data comprising log lines, each log line including data specifying a particular web page requested by a visitor, a time stamp associated with the request, and a visitor identifier;
for each log line:
determining, from the time stamp, whether the time stamp corresponds to a current period defining a first time period during which visitors visited the website and for which visitor data is stored in a buffer or a new period defining a second time period during which visitors visited the website and for which visitor data is not stored in the buffer, wherein the first time period is different from the second time period, and only in response to the time stamp corresponding to the new period, loading, from the persistent database, the buffer with all visitor data content for the new period;
accessing a visitor record that corresponds to the visitor identifier, the visitor record stored in the buffer; and
updating the visitor record corresponding to the visitor identifier with the data specifying the particular web page requested by a visitor, the time stamp associated with the request;
wherein the data specifying the particular web page requested by a visitor is inserted into the visitor record at an ordinal position based on the time stamp for the particular web page requested relative to time stamps of other web pages requested specified by other log lines for the visitor identifier.
2. The computer-implemented method of claim 1 , wherein the visitor record defines visitor footprint data related to the visitor, the visitor footprint data including one or more of browser data, referral data, domain data and visitor path data.
3. The computer-implemented method of claim 1 , further comprising:
for each log line, determining the visitor identifier; and
for each visitor identifier, generating a corresponding hash value in hash table for the visitor identifier; and
wherein accessing a visitor record that corresponds to the visitor identifier comprises accessing the hash table with a hash value corresponding to the visitor identifier.
4. The computer-implemented method of claim 3 , further comprising:
periodically determining whether the hash table meets a predetermined fullness threshold, the fullness threshold defining a partially empty measure for the hash table;
in response to determining the hash table meets the predetermined fullness threshold, increasing the size of the hash table.
5. The computer-implemented method of claim 1 , further comprising, in response to determining a completion of processing of log lines in the visitor data, persisting data stored in the buffer to a persistent database.
6. A system, comprising:
one or more server computers; and
a memory subsystem storing instructions executable by one or more server computers that upon such execution cause the one or more server computers to perform operations comprising:
accessing visitor data from a log file that is separate from a persistent database storing visitor data content describing web traffic activity for a web server, the visitor data comprising log lines, each log line including data specifying a particular web page requested by a visitor, a time stamp associated with the request, and a visitor identifier;
for each log line:
determining, from the time stamp, whether the time stamp corresponds to a current period defining a first time period during which visitors visited the website and for which visitor data is stored in a buffer or a new period defining a second time period during which visitors visited the website and for which visitor data is not stored in the buffer, wherein the first time period is different from the second time period, and only in response to the time stamp corresponding to the new period, loading, from the persistent database, the buffer with all visitor data content for the new period;
accessing a visitor record that corresponds to the visitor identifier, the visitor record stored in the buffer; and
updating the visitor record corresponding to the visitor identifier with the data specifying the particular web page requested by a visitor, the time stamp associated with the request;
wherein the data specifying the particular web page requested by a visitor is inserted into the visitor record at an ordinal position based on the time stamp for the particular web page requested relative to time stamps of other web pages requested specified by other log lines for the visitor identifier.
7. The system of claim 6 , wherein the visitor record defines visitor footprint data related to the visitor, the visitor footprint data including one or more of browser data, referral data, domain data and visitor path data.
8. The system of claim 6 , the operations further comprising:
for each log line, determining the visitor identifier; and
for each visitor identifier, generating a corresponding hash value in hash table for the visitor identifier; and
wherein accessing a visitor record that corresponds to the visitor identifier comprises accessing the hash table with a hash value corresponding to the visitor identifier.
9. The system of claim 8 , the operations further comprising:
periodically determining whether the hash table meets a predetermined fullness threshold, the fullness threshold defining a partially empty measure for the hash table;
in response to determining the hash table meets the predetermined fullness threshold, increasing the size of the hash table.
10. The system of claim 6 , the operations further comprising, in response to determining a completion of processing of log lines in the visitor data, persisting data stored in the buffer to a persistent database.
11. A non-transitory computer readable storage device and including instructions that cause a computer to perform operations comprising:
accessing visitor data from a log file that is separate from a persistent database storing visitor data content describing web traffic activity for a web server, the visitor data comprising log lines, each log line including data specifying a particular web page requested by a visitor, a time stamp associated with the request, and a visitor identifier;
for each log line:
determining, from the time stamp, whether the time stamp corresponds to a current period defining a first time period during which visitors visited the website and for which visitor data is stored in a buffer or a new period defining a second time period during which visitors visited the website and for which visitor data is not stored in the buffer, wherein the first time period is different from the second time period, and only in response to the time stamp corresponding to the new period, loading, from the persistent database, the buffer with all visitor data content for the new period;
accessing a visitor record that corresponds to the visitor identifier, the visitor record stored in the buffer; and
updating the visitor record corresponding to the visitor identifier with the data specifying the particular web page requested by a visitor, the time stamp associated with the request;
wherein the data specifying the particular web page requested by a visitor is inserted into the visitor record at an ordinal position based on the time stamp for the particular web page requested relative to time stamps of other web pages requested specified by other log lines for the visitor identifier.
12. The non-transitory computer readable storage device of claim 11 , wherein the visitor record defines visitor footprint data related to the visitor, the visitor footprint data including one or more of browser data, referral data, domain data and visitor path data.
13. The non-transitory computer readable storage device of claim 11 , the operations further comprising:
for each log line, determining the visitor identifier; and
for each visitor identifier, generating a corresponding hash value in hash table for the visitor identifier; and
wherein accessing a visitor record that corresponds to the visitor identifier comprises accessing the hash table with a hash value corresponding to the visitor identifier.
14. The non-transitory computer readable storage device of claim 13 , the operations further comprising:
periodically determining whether the hash table meets a predetermined fullness threshold, the fullness threshold defining a partially empty measure for the hash table;
in response to determining the hash table meets the predetermined fullness threshold, increasing the size of the hash table.
15. The non-transitory computer readable storage device of claim 11 , the operations further comprising, in response to determining a completion of processing of log lines in the visitor data, persisting data stored in the buffer to a persistent database.