Techniques for web framework detection
Techniques are disclosed for analyzing documents to detect web components and the web frameworks in the documents. In at least one embodiment, a network analysis system is provided to passively detect web frameworks of documents. The network analysis system can render a document using a document object model to identify objects in the document that are defined as web components. A hash function may be applied to each of the objects to generate a hash signature for the object. Files defining web frameworks can be downloaded from a repository system. Each file may corresponding to a web component. A hash function is applied content in each file to generate a hash signature. The hash signatures of each file may be compared to the hash signatures of the objects in the document to identify a web component for each object. A web framework can be identified based on the web components.
1. A method, comprising:
processing a structure of a web page;
initiating requests for a set of web components in the web page for rendering the web page, for each request:
receiving a response to the request;
generating a digital signature of the response;
comparing the digital signature to a data store of digital signatures;
retrieving version information for the digital signature; and
grouping the version information with other version information for the web page;
determining version changes for web components in the set of web components using the grouped version information;
causing display of information about each version change for web components in the set of web components and a security score.
2. The method of claim 1 , wherein the information indicates a name of each displayed web component and a version of the web component.
3. The method of claim 2 , wherein the information indicates a first date when the web component was first identified and a second date when the web component was most recently identified.
4. The method of claim 1 , wherein the web page includes one or more web frameworks.
5. The method of claim 1 , further comprising:
determining one or more web frameworks in the web page using the grouped version information;
causing display of information about the one or more web frameworks that includes a name and a version identifier of each of the one or more web frameworks.
6. The method of claim 1 , further comprising:
determining a web framework in the web page using the grouped version information;
causing display of information about the web framework that includes a name, a previous version identifier, and a current version identifier of the web framework.
7. The method of claim 1 , further comprising:
determining a web framework in the web page having a best match among a set of web frameworks using the grouped version information.
8. The method of claim 1 , wherein the digital signature is generated using a cryptographic hash function.
9. The method of claim 8 , wherein the cryptographic hash function is based on an MD5 hashing algorithm.
10. The method of claim 1 , wherein the digital signature is generated using a non-cryptographic hash function based on a number hashing algorithm.
11. The method of claim 1 , wherein the web page is obtained from a server computer using a web crawler, and wherein the web page is one of a plurality of web page of a website.
12. The method of claim 1 , wherein the web page is obtained from a server computer using a uniform resource locator (URL), and wherein the web page is one of a plurality of web page of a website.
13. One or more non-transitory computer-readable storage media, storing one or more sequences of instructions, which when executed by one or more processors cause performance of:
processing a structure of a web page;
initiating requests for a set of web components in the web page for rendering the web page, for each request:
receiving a response to the request;
generating a digital signature of the response;
comparing the digital signature to a data store of digital signatures;
retrieving version information for the digital signature; and
grouping the version information with other version information for the web page;
determining version changes for web components in the set of web components using the grouped version information;
causing display of information about each version change for web components in the set of web components and a security score.
14. The one or more non-transitory computer-readable storage media of claim 13 , wherein the information indicates a name of each displayed web component and a version of the web component.
15. The one or more non-transitory computer-readable storage media of claim 14 , wherein the information indicates a first date when the web component was first identified and a second date when the web component was most recently identified.
16. The one or more non-transitory computer-readable storage media of claim 13 , wherein the web page includes one or more web frameworks.
17. The one or more non-transitory computer-readable storage media of claim 13 , further comprising:
determining one or more web frameworks in the web page using the grouped version information;
causing display of information about the one or more web frameworks that includes a name and a version identifier of each of the one or more web frameworks.
18. The one or more non-transitory computer-readable storage media of claim 13 , further comprising:
determining a web framework in the web page using the grouped version information;
causing display of information about the web framework that includes a name, a previous version identifier, and a current version identifier of the web framework.
19. The one or more non-transitory computer-readable storage media of claim 13 , further comprising:
determining a web framework in the web page having a best match among a set of web frameworks using the grouped version information.
20. The one or more non-transitory computer-readable storage media of claim 13 , wherein the digital signature is generated using a cryptographic hash function.