System and method for hypertext transfer protocol layered reconstruction
View Patent ↗HTTP layered reconstruction is disclosed. A database is queried to identify a location of a previously reconstructed HTML artifact file or packet data of a HTML file in a repository that stores packet data captured from a network. The reconstructed HTML file is analyzed. Links to external files are identified and the database is queried to identify a location of previously reconstructed artifact files or packet data of associated external files. The external files are reconstructed, as needed. A web page is then reconstructed based on the reconstructed HTML file and reconstructed external files, presenting a view of the web page as it originally appeared to a user. A user may specify which external file types to include and/or not include. New versions of external files may be obtained and indicated in the reconstructed web page when associated artifact files or packet data are not stored within the repository.
1. A method of hypertext transfer protocol layered reconstruction comprising:
querying, utilizing at least one processing unit, at least one database to identify at least one of a location of packet data of a hypertext markup language file in a packet capture repository or a location of a previously reconstructed artifact of the hypertext markup language file, the packet capture repository storing network packet data, the database indexed by content analysis and inspection of the network packet data to point to at least one location of the network packet data in the packet capture repository according to at least one of artifact types of the network packet data, protocol types of the network packet data, or time stamps of the network packet data;
selecting a subset of the network packet data based on the time stamps of the network packet data and time stamps of at least one of the packet data of the hypertext markup language file or the previously reconstructed artifact of the hypertext markup language file;
identifying, utilizing the at least one processing unit, at least one external link in the hypertext markup language file;
identifying, utilizing the at least one processing unit, at least one additional external link in the hypertext markup language file;
querying, utilizing the at least one processing unit, the at least one database to determine that external file packet data of an additional external file associated with the at least one additional external link is not located in the packet capture repository and was not previously reconstructed;
querying, utilizing the at least one processing unit, the at least one database to identify at least one of a location of external file packet data of an external file in the packet capture repository associated with the at least one external link or a location of a previously reconstructed artifact of the external file; and
reconstructing, utilizing the at least one processing unit, a web page based on at least the hypertext markup language file and the external file, wherein reconstructing includes determining a first available external file type that the reconstructed web page can include and a second available external file type that the reconstructed web page cannot include to avoid potential network damage from malicious code, wherein a placeholder is used as a substitute for the second available external file type, wherein the reconstructed web page is based on a version of the additional external file obtained from the at least one additional external link instead of at least one of the packet capture repository or the previously reconstructed artifact of the external file.
2. The method of claim 1 , wherein the reconstructed web page includes an indication that the additional external file was obtained from the at least one additional external link instead of at least one of the packet capture repository or the previously reconstructed artifact of the external file.
3. The method of claim 1 , wherein the first available external file type and the second available external file type are determined based on input received from a user.
4. The method of claim 1 , wherein the first available external file type is at least one of an image file type, a script file type, a style sheet file type, and a video file type.
5. The method of claim 1 , wherein said querying, utilizing at least one processing unit, a database to identify a location of packet data of a hypertext markup language file in a packet capture repository or a location of a previously reconstructed artifact of the hypertext markup language file further comprises:
selecting a subset of the database based on a time window; and
performing the query on the subset of the database.
6. The method of claim 5 , wherein the time window is based on input received from a user.
7. The method of claim 1 , wherein said reconstructing, utilizing the at least one processing unit, a web page based on at least the hypertext markup language file and the external file further comprises:
storing at least one of the packet data of the hypertext markup file or the previously reconstructed artifact of the hypertext markup language file and at least one of the external file packet data of the external file or the previously reconstructed artifact of the external file to at least one non-transitory computer readable storage medium; and
altering at least one reference of the hypertext markup file and a reconstructed artifact of the external file to refer to the at least one non-transitory computer readable storage medium.
8. The method of claim 1 , wherein the at least one processing unit utilizes a document object model to reconstruct the web page and provide the reconstructed web page to a user.
9. The method of claim 1 , further comprising: creating an image of the reconstructed web page and providing the image of the reconstructed web page to a user.
10. A method of hypertext transfer protocol layered reconstruction comprising:
querying, utilizing at least one processing unit, at least one database to identify at least one of a location of packet data of a hypertext markup language file in a packet capture repository or a location of a previously reconstructed artifact of the hypertext markup language file, the packet capture repository storing network packet data, the database indexed by content analysis and inspection of the network packet data to point to at least one location of the network packet data in the packet capture repository according to at least one of artifact types of the network packet data, protocol types of the network packet data, or time stamps of the network packet data, wherein querying includes:
selecting a subset of the network packet data based on the time stamps of the network packet data and time stamps of at least one of the packet data of the hypertext markup language file or the previously reconstructed artifact of the hypertext markup language file;
creating an external file list by at least one of matching an absolutized version of the at least one external link with absolutized versions of addresses of the subset of the network packet data, wherein absolutized versions of addresses are absolute addresses for corresponding relative addresses, matching an external file type associated with the at least one external link with the subset of the network packet data, or matching a source or destination address of at least one of the packet data of the hypertext markup language file or the previously reconstructed artifact of the hypertext markup language file with a source or destination address of the external file packet data of the external file;
ranking the external file list based on the time stamps of the subset of the network packet data; and
selecting an entry from the ranked external file list based on the time stamps of the at least one of the packet data of the hypertext markup language file or the previously reconstructed artifact of the hypertext markup language file to obtain a best guess as to the file that was part of the web page as originally viewed by a user;
identifying, utilizing the at least one processing unit, at least one external link in the hypertext markup language file;
querying, utilizing the at least one processing unit, the at least one database to identify at least one of a location of external file packet data of an external file in the packet capture repository associated with the at least one external link or a location of a previously reconstructed artifact of the external file; and
reconstructing, utilizing the at least one processing unit, a web page based on at least the hypertext markup language file and the external file, wherein reconstructing includes determining a first available external file type that the reconstructed web page can include and a second available external file type that the reconstructed web page cannot include to avoid potential network damage from malicious code, wherein the first available external file type and the second available external file type are determined based on input received from a user.
11. The method of claim 10 , wherein said selecting an entry from the ranked external file list based on the time stamps of the at least one of the packet data of the hypertext markup language file or the previously reconstructed artifact of the hypertext markup language file further comprises selecting the entry with a time stamp soonest in time within a threshold after the time stamps of the at least one of the packet data of the hypertext markup language file or the previously reconstructed artifact of the hypertext markup language file.
12. The system of claim 10 , wherein said selecting an entry from the ranked external file list based on the time stamps of the at least one of the packet data of the hypertext markup language file or the previously reconstructed artifact of the hypertext markup language file further comprises:
determining the ranked external file list does not include entries with time stamps after the time stamps of the at least one of the packet data of the hypertext markup language file or the previously reconstructed artifact of the hypertext markup language file; and
selecting the entry with the time stamp soonest in time previous to the time stamps of the at least one of the packet data of the hypertext markup language file or the previously reconstructed artifact of the hypertext markup language file.