Static analysis of source code for improved web crawling
The processing logic (e.g., a web crawler program) receives a server output from a server hosting a web page and downloads content from the server output. The server output can include embedded resources, including source code files. The processing logic parses the source code and generates a plurality of elements into a data structure as elements. The data structure is made up of multiple elements, each element representing a language construct extracted from the source code. The processing logic can evaluate the elements of the data structure to identify and retrieve request components pertinent. Using the one or more request components retrieved from the data structure, the processing logic can generate server requests. The web crawler will send each server request to the server to receive a subsequent server output and continue the crawl.
1 . A method comprising:
receiving a server output from a server wherein the server output comprises source code of a web page;
parsing, without executing the source code, the source code into a data structure, the data structure comprising a plurality of elements, wherein each element of the plurality of elements corresponds to a respective language construct of the source code, wherein the plurality of elements represent one or more of function calls, variables, expressions, or code execution paths within the source code;
identifying one or more request components within the data structure, wherein the one or more request components comprise at least a request body, wherein the request body comprises a primary segment of an HTTP data transmission, and wherein the identifying involves evaluating the elements of the data structure to identify and retrieve the one or more request components by at least one of: locating function calls and parsing associated arguments, heuristically identifying variables, constructing request components from constituent variables or expressions, or statically evaluating divergent code-execution paths, and wherein identifying the one or more request components comprises extracting one or more uniform resource locators (URLs) from the one or more of function calls, variables, expressions, or code execution paths without executing the source code in a runtime environment, wherein the source code comprises client-side scripting code embedded in the web page, and wherein the extracted one or more URLs specify resources that would be dynamically loaded by the client-side scripting code upon execution in a browser environment;
generating, using the one or more request components including the extracted one or more URLs, one or more server requests to be provided to a web crawler, the web crawler to send the server request to the server to receive a subsequent server output, wherein the one or more server requests are generated without executing the source code or simulating user interaction with the web page;
populating, without having executed the source code in a browser environment, a crawler feeder with the one or more server requests, the crawler feeder embodying a queue from which the one or more server requests are provided to the web crawler; and
executing a server request of the one or more server requests from the crawler feeder to receive a subsequent server output.
2 . The method of claim 1 , wherein the web page comprises a single page application (SPA).
3 . The method of claim 1 , wherein the data structure comprises an abstract syntax tree (AST).
4 . The method of claim 1 , wherein the one or more server requests comprises a HyperText Transfer Protocol (HTTP) request.
5 . The method of claim 1 , wherein the source code comprises JavaScript code.
6 . A system comprising:
a memory storing instructions; and
a processor, coupled with the memory and to execute the instructions, the instructions when executed cause the processor to perform operations comprising:
receiving a server output from a server wherein the server output comprises source code of a web page;
parsing, without executing the source code, the source code into a data structure, the data structure comprising a plurality of elements, wherein each element of the plurality of elements corresponds to a respective language construct of the source code, wherein the plurality of elements represent one or more of function calls, variables, expressions, or code execution paths within the source code;
identifying one or more request components within the data structure, wherein the one or more request components comprise at least a request body, wherein the request body comprises a primary segment of an HTTP data transmission, and wherein the identifying involves evaluating the elements of the data structure to identify and retrieve the one or more request components by at least one of: locating function calls and parsing associated arguments, heuristically identifying variables, constructing request components from constituent variables or expressions, or statically evaluating divergent code-execution paths, and wherein identifying the one or more request components comprises extracting one or more uniform resource locators (URLs) from the one or more of function calls, variables, expressions, or code execution paths without executing the source code in a runtime environment, wherein the source code comprises client-side scripting code embedded in the web page, and wherein the extracted one or more URLs specify resources that would be dynamically loaded by the client-side scripting code upon execution in a browser environment;
generating, using the one or more request components including the extracted one or more URLs, one or more server requests to be provided to a web crawler, the web crawler to send the server request to the server to receive a subsequent server output, wherein the one or more server requests are generated without executing the source code or simulating user interaction with the web page;
populating, without having executed the source code in a browser environment, a crawler feeder with the one or more server requests, the crawler feeder embodying a queue from which the one or more server requests are provided to the web crawler; and
executing a server request of the one or more server requests from the crawler feeder to receive a subsequent server output.
7 . The system of claim 6 , wherein the web page comprises a single page application (SPA).
8 . The system of claim 6 , wherein the data structure comprises an abstract syntax tree (AST).
9 . The system of claim 6 , wherein the one or more server requests comprises a HyperText Transfer Protocol (HTTP) request.
10 . The system of claim 6 , wherein the source code comprises JavaScript code.
11 . A non-transitory computer-readable storage medium comprising stored instructions, which when executed by a processor, cause the processor to perform operations comprising:
receiving a server output from a server wherein the server output comprises source code of a web page;
parsing, without executing the source code, the source code into a data structure, the data structure comprising a plurality of elements, wherein each element of the plurality of elements corresponds to a respective language construct of the source code, wherein the plurality of elements represent one or more of function calls, variables, expressions, or code execution paths within the source code;
identifying one or more request components within the data structure, wherein the one or more request components comprise at least a request body, wherein the request body comprises a primary segment of an HTTP data transmission, and wherein the identifying involves evaluating the elements of the data structure to identify and retrieve the one or more request components by at least one of: locating function calls and parsing associated arguments, heuristically identifying variables, constructing request components from constituent variables or expressions, or statically evaluating divergent code-execution paths, and wherein identifying the one or more request components comprises extracting one or more uniform resource locators (URLs) from the one or more of function calls, variables, expressions, or code execution paths without executing the source code in a runtime environment, wherein the source code comprises client-side scripting code embedded in the web page, and wherein the extracted one or more URLs specify resources that would be dynamically loaded by the client-side scripting code upon execution in a browser environment;
generating, using the one or more request components including the extracted one or more URLs, one or more server requests to be provided to a web crawler, the web crawler to send the server request to the server to receive a subsequent server output, wherein the one or more server requests are generated without executing the source code or simulating user interaction with a web page;
populating, without having executed the source code in a browser environment, a crawler feeder with the one or more server requests, the crawler feeder embodying a queue from which the one or more server requests are provided to the web crawler; and
executing a server request of the one or more server requests from the crawler feeder to receive a subsequent server output.
12 . The non-transitory computer-readable storage medium of claim 11 , wherein the web page comprises a single page application (SPA).
13 . The non-transitory computer-readable storage medium of claim 11 , wherein the data structure comprises an abstract syntax tree (AST).
14 . The non-transitory computer-readable storage medium of claim 11 , wherein the source code comprises JavaScript code.