Integrated crawling and auditing of web applications and web content

1. A method for performing a vulnerability assessment of a target object, the method comprising the steps of:

identifying a target object to assess;

initiating a crawling process on the identified object;

storing the results of the crawling process into a queue in a memory storage device;

initiating an audit process while the crawling process is still in operation and additional results of the crawling process are being stored into the memory storage device;

extracting one or more stored results of the crawling process from the memory storage device and providing them to the audit process;

storing the results of the audit process into the memory storage device;

extracting the stored results of the audit process from the memory storage device and providing them to the crawling process, while the crawling process is still in operation, for further crawling; and

allocating processing power to the co-operating crawling process and the audit process based on the number of results existing in the queue.

2. The method of claim 1 , wherein the target object is a web site and the step of identifying the target object comprises identifying a URL for the web site and the step of initiating a crawling process further comprises providing the URL to the crawling process.

3. The method of claim 2 , wherein the step of initiating a crawling process further comprises examining the HTML and objects associated with the URL.

4. The method of claim 1 , wherein the target object is a web application accessible at a URL, and the step of initiating a crawling process involves examining the web application.

5. The method of claim 1 , wherein the step of initiating a crawling process comprises searching the target object for links, sessions and files.

6. The method of claim 1 , wherein the step of initiating an audit process is performed as soon as a threshold number of results are stored into the queue.

7. The method of claim 6 , further comprising the step of pausing the audit process when the number of results stored in the queue is less than a minimum number.

8. A system for assessing the vulnerability of a target object, the system comprising:

a memory device;

a crawler process, communicatively coupled to the memory device and operative to conduct a crawl of the target object and store the results of the crawl into the memory device;

a scanner process, communicatively coupled to the crawler process and operative to invoke the crawler process by providing the identity of the target object;

an audit process, communicatively coupled to the memory device and operative to conduct an audit simultaneously with the operation of the crawler process by extracting the results of the crawler process from the memory device while the crawler process continues to add new results to the memory device; and

a plurality of audit engines invoked by the audit process and operative to perform various attacks on the target object,

wherein the audit process is further operative to obtain attack results from the plurality of audit engines and provide at least a portion of the attack results as further input to the crawler process while the crawler process is still operating, and

wherein the audit process is further operative to store the attack results into the memory device and the scanner process is further operative to allocate processing power between the crawling process and the auditing process based at least in part on the number of crawling results and attack results stored in the memory device.

9. The system of claim 8 , wherein the at least a portion of the attack results provided to the crawler process are those attack results that reveal structure regarding the target object.

10. The system of claim 8 , further comprising a user interface, the scanner is communicatively coupled to the user interface and the audit process and is further operative to receive attack results from the crawler process and the audit process and provide these results to the user interface, whereby the results of the audit process are available before the crawling process is completed.

11. The system of claim 8 , wherein the scanner process is further operative to allocate processing power between the crawling process and the auditing process.

12. The system of claim 8 , wherein the audit process is operative to conduct the audit simultaneously with the operation of the crawler process only after the crawling process has stored a threshold number of crawling results into the memory device.

13. The system of claim 12 , wherein the audit process is further operative to pause when the number of crawling results stored in the memory device is less than a minimum number.

14. A method for performing a vulnerability assessment of a target object and providing results thereof to a user, the method comprising the steps of:

receiving input identifying a target object to assess;

conducting a crawling process on the identified object;

storing the results of the crawling process into a queue in a memory storage device;

conducting an audit process while the crawling process is still in operation and results are still being stored into the memory storage device;

extracting one or more of the stored results of the crawling process from the memory storage device and providing them to the audit process;

storing the results of the audit process into the memory storage device;

extracting the stored results of the audit process from the memory storage device and providing them to the crawling process, while the crawling process is still in operation, for further crawling; and

allocating processing power to the co-operating crawling process and the audit process based on the number of results existing in the queue.

15. The method of claim 14 , wherein the step of conducting an audit process is initiated once a threshold number of results from the crawling process have been stored in the memory storage device.