Systems and methods for prefetching objects for caching using QOS
The present solution provides a variety of techniques for accelerating and optimizing network traffic, such as HTTP based network traffic. The solution described herein provides techniques in the areas of proxy caching, protocol acceleration, domain name resolution acceleration as well as compression improvements. In some cases, the present solution provides various prefetching and/or prefreshening techniques to improve intermediary or proxy caching, such as HTTP proxy caching. In other cases, the present solution provides techniques for accelerating a protocol by improving the efficiency of obtaining and servicing data from an originating server to server to clients. In another cases, the present solution accelerates domain name resolution more quickly. As every HTTP access starts with a URL that includes a hostname that must be resolved via domain name resolution into an IP address, the present solution helps accelerate HTTP access. In some cases, the present solution improves compression techniques by prefetching non-cacheable and cacheable content to use for compressing network traffic, such as HTTP. The acceleration and optimization techniques described herein may be deployed on the client as a client agent or as part of a browser, as well as on any type and form of intermediary device, such as an appliance, proxying device or any type of interception caching and/or proxying device.
1. A method for speculatively prefetching an object using idle network bandwidth, the method comprising:
(a) receiving, by a device via a network, a communication identifying an object;
(b) generating, by the device, a request to an object server for the object, the request identified as a speculative request;
(c) determining, by the device, availability of idle network bandwidth to obtain the object from the object server; and
(d) transmitting, by the device in response to the determination of availability of idle network bandwidth, the generated request packet to the object server according to a transmission rate to maintain bandwidth usage from the speculative request within a predetermined level.
2. The method of claim 1 , forwarding, by the device, the communication to a requester.
3. The method of claim 1 , wherein step (d) comprises transmitting the generated request packet prior to a user requesting the object identified by the communication.
4. The method of claim 1 , wherein step (b) comprises generating the request packet prior to a user requesting the object identified by the communication.
5. The method of claim 1 , comprising receiving, by the device, the object in response to the transmitted request packet.
6. The method of claim 5 , comprising storing, by the device, the received object in a cache.
7. The method of claim 1 , wherein step (a) comprises receiving a page including a hyperlink identifying an object.
8. The method of claim 1 , comprising identifying, by the device, the generated request packet as a speculative request by encoding a field of one of a transport layer or internet protocol layer header option to a predetermined value.
9. The method of claim 1 , comprising identifying, by the device, the generated request packet as a speculative request by encoding a value of an application layer protocol option to a predetermined value.
10. The method of claim 1 , comprising identifying, by the device, the generated request packet as a speculative request by setting a value of one of a Type of Service (TOS) or a DiffServ Codepoint (DSCP) in a field of an Internet Protocol (IP) frame to a predetermined value.
11. The method of claim 1 , comprising identifying, by the device, the generated request packet as a speculative request by making an entry in a connection state table identifying the request as speculative.
12. The method of claim 1 , comprising identifying, by the device, the generated request packet as a lower priority for transmission than non-speculative requests for objects.
13. The method of claim 1 , wherein the device comprises one of a client, a server, or an appliance between the client and the server.
14. A system for speculatively prefetching an object using idle network bandwidth, the system comprising:
means for intercepting a communication transmitted via a network, the communication comprising an identifier of an object;
means for generating a request packet to an object server for the object identified, the request identified as a speculative request;
means for determining availability of idle network bandwidth to obtain the object from the object server; and
means for transmitting, in response to the determination of availability of idle network bandwidth, the generated request packet to the object server according to a transmission rate to maintain bandwidth usage from the speculative request within a predetermined level.
15. The system of claim 14 , wherein the system forwards the communication to a requester.
16. The system of claim 14 , wherein the system transmits the generated request packet prior to a user requesting the object identified by the page.
17. The system of claim 14 , wherein the system generates the request packet prior to a user requesting the object identified by the page.
18. The system of claim 14 , wherein the object in response to the transmitted request packet.
19. The system of claim 18 , wherein the system stores the received object in a cache.
20. The system of claim 14 , wherein the page includes a hyperlink identifying an object.
21. The system of claim 14 , wherein the system identifies the generated request packet as a speculative request by encoding a field of one of a transport layer or internet protocol layer header option to a predetermined value.
22. The system of claim 14 , wherein the system identifies the generated request packet as a speculative request by encoding a value of an application layer protocol option to a predetermined value.
23. The system of claim 14 , wherein the system identifies the generated request packet as a speculative request by setting a value of one of a Type of Service (TOS) or a DiffServ Codepoint (DSCP) in a field of an Internet Protocol (IP) frame to a predetermined value.
24. The system of claim 14 , wherein the system identifies the generated request packet as a speculative request by making an entry in a connection state table identifying the request as speculative.
25. The system of claim 14 , wherein the device identifies the generated request packet as a lower priority for transmission than non-speculative requests for objects.
26. A method of speculatively prefetching an object via multiple devices using idle network bandwidth, the method comprising:
(a) receiving, by a first device, a communication transmitted from a server to a client, the communication comprising an identifier of an object, the first appliance forwarding the communication to the requester;
(b) transmitting, by the first device, a request to the server for the object identified by the communication, the first device generating the request and identifying the request as speculative;
(c) identifying, by a second device, the request received from the first device as a speculative request;
(d) determining, by the second device, availability of idle network bandwidth to obtain the object from the server; and
(e) transmitting, by the second device in response to the determination of availability of idle network bandwidth, the request according to a transmission rate to maintain bandwidth usage from the speculative request within a predetermined level.
27. The method of claim 26 , wherein one of step (b) or step (e) comprises transmitting the generated request prior to a user requesting the object identified by the communication.
28. The method of claim 26 , wherein step (b) comprises generating the request prior to a user requesting the object identified by the communication.
29. The method of claim 26 , comprising receiving, by the second device, the object in response to the transmitted request.
30. The method of claim 26 , comprising storing, by the second device, the received object in a cache.
31. The method of claim 30 , comprising transmitting, by the second device, the object to the first device.
32. The method of claim 31 , comprising storing, by the first device, the received object in a cache.
33. The method of claim 26 , wherein step (a) comprises receiving, by the first device, a page including a hyperlink identifying the object.
34. The method of claim 26 , comprising identifying, by the first device, the generated request as a speculative request by encoding a field of one of a transport layer or internet protocol layer header option to a predetermined value.
35. The method of claim 26 , comprising identifying, by the first device, the generated request as a speculative request by encoding a value of an application layer protocol option to a predetermined value.
36. The method of claim 26 , comprising identifying, by the first device, the generated request as a speculative request by encoding a value of one of a Type of Service (TOS) or a DiffServ Codepoint (DSCP) in a field of an Internet Protocol (IP) frame to a predetermined value.
37. The method of claim 26 , comprising identifying, by the second device, the generated request is a speculative request by identifying an encoded value of an application layer protocol option comprises a predetermined value.
38. The method of claim 26 , comprising identifying, by the second device, the generated request is a speculative request by identifying an encoded value of one of a Type of Service (TOS) or a DiffServ Codepoint (DSCP) field of an Internet Protocol (IP) frame comprises a predetermined value.
39. The method of claim 26 , comprising identifying, by the second device, the generated request is a speculative request by identifying by an entry in a connection state table identifying the request as speculative.
40. The method of claim 26 , comprising transmitting, by one of the first device or the second device, the request at a lower priority of transmission than non-speculative requests for objects.
41. The method of claim 26 , wherein the first device comprises one of a client or an appliance.
42. The method of claim 26 , wherein the second device comprises an appliance.
43. The method of claim 26 , comprising transmitting, by one of the first device or the second device, the request at a lower priority of transmission than non-speculative requests for objects.