Methods and systems for advanced content cacheability determination
The embodiments provide systems and methods for efficiently and accurately differentiating requests directed to uncacheable content from requests directed to cacheable content based on identifiers from the requests. The differentiation occurs without analysis or retrieval of the content being requested. Some embodiments hash identifiers of prior requests that resulted in uncacheable content being served in order to set indices within a bloom filter. The bloom filter then tracks prior uncacheable requests without storing each of the identifiers so that subsequent requests for uncacheable requests can be easily identified based on a hash of the request identifier and set indices of the bloom filter. Some embodiments produce a predictive model identifying uncacheable content requests by tracking various characteristics found in identifiers of prior requests that resulted in uncacheable content being served. Subsequent requests with identifiers having similar characteristics to those of the predictive model can then be differentiated.
1. A request distribution device comprising:
a non-transitory computer-readable medium storing a set of processor-executable instructions; and
one or more processors configured to execute the set of processor-executable instructions, wherein executing the set of processor-executable instructions causes the one or more processors to:
generate a predictive model based on a set of characteristics repeating in (i) each of a first set of requests that resulted in a first type of content being returned, or (ii) each of a different second set of requests that resulted in a second type of content being returned, wherein each characteristic of the set of characteristics comprises a part that is less than all of a Uniform Resource Locator (“URL”), a regular expression, or a pattern repeating in the first set of requests or the second set of requests;
receive a particular request with at least one characteristic that is different than the set of characteristics and each request of the first set of requests and the second set of requests, and that is used to request different content than content returned in response to each request of the first set of requests and the second set of requests;
classify the particular request as a request directed to the first type of content or the second type of content based on a number or probability of characteristics of the particular request matching the set of characteristics of the predictive model;
distribute the particular request to one of a first set of servers based on a first request distribution scheme that prioritizes selection of the first set of servers over a second set of servers, and that is selected as a result of classifying the particular request as a request directed to the first type of content; and
distribute the particular request to one of the second set of servers based on a different second request distribution scheme that prioritizes selection of the second set of servers over the first set of servers, and that is selected as a result of classifying the particular request as a request directed to the second type of content.
2. The device of claim 1 ,
wherein the first type of content is common content that the first set of servers serve to two or more different users; and
wherein the second type of content is dynamic content that the second set of servers customize in some form for each requesting user.
3. The device of claim 1 , wherein the processor-executable instructions further include processor-executable instructions to:
detect a type for particular content that is returned in response to distributing the particular request; and
modify the predictive model based on the type of the particular content.
4. The device of claim 3 , wherein the processor-executable instructions to modify the predictive model include processor-executable instructions to:
add characteristics of the particular request to the set of characteristics used to generate the predictive model.
5. The device of claim 1 , wherein the processor-executable instructions to generate the predictive model include processor-executable instructions to:
determine, from the set of characteristics, a first subset of characteristics that have a higher probability of being included in the first set of requests, that resulted in the first type of content being returned, than being included in the second set of requests, that resulted in the second type of content being returned; and
determine, from the set of characteristics, a different second subset of characteristics that have a higher probability of being included in the second set of requests, that resulted in the second type of content being returned, than being included in the first set of requests, that resulted in the first type of content being returned.
6. The device of claim 5 , wherein the processor-executable instructions to classify the particular request include processor-executable instructions to:
identify the characteristics of the particular request from a URL of the particular request;
compute a first probability value based on one or more characteristics of the particular request matching to the first subset of characteristics;
compute a second probability value based on one or more characteristics of the particular request matching to the second subset of characteristics; and
predict the particular request as a request directed to the first type of content in response to the first probability value being greater than the second probability value.
7. The device of claim 1 , wherein the processor-executable instructions to classify the particular request include processor-executable instructions to:
produce a first probability value based on a number of the characteristics of the particular request matching to characteristics of the first set of requests that resulted in the first type of content being returned;
produce a second probability value based on a different number of the characteristics of the particular request matching to characteristics of the second set of requests that resulted in the second type of content being returned;
predict the particular request as a request directed to the first type of content in response to the first probability value being greater than the second probability value.
8. The device of claim 1 , wherein the processor-executable instructions to generate the predictive model include processor-executable instructions to:
define a first set of probabilities for the predictive model based on a frequency of different characteristics being present in different requests of the first set of requests; and
define a second set of probabilities for the predictive model based on a frequency of different characteristics being present in different requests of the second set of requests.
9. The device of claim 1 , wherein the processor-executable instructions to generate the predictive model include processor-executable instructions to:
compile a first set of characteristics from URLs of each request of the first set of requests; and
compile a different second set of characteristics from URLs of each request of the second set of requests.
10. The device of claim 9 , wherein the processor-executable instructions to classify the particular request include processor-executable instructions to:
classify the particular request as a request directed to the first type of content based on a greater number of the characteristics of the particular request matching to the first set of characteristics than to the second set of characteristics; and
classify the particular request as a request directed to the second type of content based on a greater number of the characteristics of the particular request matching to the second set of characteristics than to the first set of characteristics.
11. The device of claim 1 , wherein the processor-executable instructions to distribute the particular request based on the first request distribution scheme include processor-executable instructions to:
select the first set of servers from a plurality of servers based on filtering criteria of the first request distribution scheme, wherein the plurality of servers comprises the first set of servers and the second set of servers; and
select a particular server of the first set of servers based on a hash of a URL of the particular request.
12. The device of claim 1 , wherein the processor-executable instructions further include processor-executable instructions to:
receive a second request that matches a previous request in the first set of requests;
select the first request distribution scheme in response to matching the second request to the previous request in the first set of requests and without using the predictive model to classify a type of the content the second request is directed to; and
distribute the second request to a server in the first set of servers based on the first request distribution scheme.
13. A method comprising:
generating a predictive model based on a set of characteristics repeating in (i) each of a first set of requests that resulted in a first type of content being returned, or (ii) each of a different second set of requests that resulted in a second type of content being returned, wherein each characteristic of the set of characteristics comprises a part that is less than all of a Uniform Resource Locator (“URL”), a regular expression, or a pattern repeating in the first set of requests or the second set of requests;
receiving a particular request with at least one characteristic that is different than the set of characteristics and each request of the first set of requests and the second set of requests, and that is used to request different content than content returned in response to each request of the first set of requests and the second set of requests;
classifying the particular request as a request directed to the first type of content or the second type of content based on a number or probability of characteristics of the particular request matching the set of characteristics of the predictive model;
distributing the particular request to one of a first set of servers based on a first request distribution scheme that prioritizes selection of the first set of servers over a second set of servers, and that is selected as a result of classifying the particular request as a request directed to the first type of content; and
distributing the particular request to one of the second set of servers based on a different second request distribution scheme that prioritizes selection of the second set of servers over the first set of servers, and that is selected as a result of classifying the particular request as a request directed to the second type of content.
14. The method of claim 13 further comprising:
detecting a type for particular content that is returned in response to distributing the particular request; and
modifying the predictive model based on the type of the particular content.
15. The method of claim 13 , wherein generating the predictive model comprises:
determining, from the set of characteristics, a first subset of characteristics that have a higher probability of being included in the first set of requests, that resulted in the first type of content being returned, than being included in the second set of requests, that resulted in the second type of content being returned; and
determining, from the set of characteristics, a different second subset of characteristics that have a higher probability of being included in the second set of requests, that resulted in the second type of content being returned, than being included in the first set of requests, that resulted in the first type of content being returned.
16. The method of claim 15 , wherein classifying the particular request comprises:
identifying the characteristics of the particular request from a URL of the particular request;
computing a first probability value based on one or more characteristics of the particular request matching to the first subset of characteristics;
computing a second probability value based on one or more characteristics of the particular request matching to the second subset of characteristics; and
predicting the particular request as a request directed to the first type of content in response to the first probability value being greater than the second probability value.
17. The method of claim 13 , wherein classifying the particular request comprises:
producing a first probability value based on a number of the characteristics of the particular request matching to characteristics of the first set of requests that resulted in the first type of content being returned;
producing a second probability value based on a different number of the characteristics of the particular request matching to characteristics of the second set of requests that resulted in the second type of content being returned;
predicting the particular request as a request directed to the first type of content in response to the first probability value being greater than the second probability value.
18. The method of claim 13 , wherein generating the predictive model comprises:
defining a first set of probabilities for the predictive model based on a frequency of different characteristics being present in different requests of the first set of requests; and
defining a second set of probabilities for the predictive model based on a frequency of different characteristics being present in different requests of the second set of requests.
19. A non-transitory computer-readable medium, storing a set of processor-executable instructions, which, when executed by one or more processors, cause the one or more processors to:
generate a predictive model based on a set of characteristics repeating in (i) each of a first set of requests that resulted in a first type of content being returned, or (ii) each of a different second set of requests that resulted in a second type of content being returned, wherein each characteristic of the set of characteristics comprises a part that is less than all of a Uniform Resource Locator (“URL”), a regular expression, or a pattern repeating in the first set of requests or the second set of requests;
receive a particular request with at least one characteristic that is different than the set of characteristics and each request of the first set of requests and the second set of requests, and that is used to request different content than content returned in response to each request of the first set of requests and the second set of requests;
classify the particular request as a request directed to the first type of content or the second type of content based on a number or probability of characteristics of the particular request matching the set of characteristics of the predictive model;
distribute the particular request to one of a first set of servers based on a first request distribution scheme that prioritizes selection of the first set of servers over a second set of servers, and that is selected as a result of classifying the particular request as a request directed to the first type of content; and
distribute the particular request to one of the second set of servers based on a different second request distribution scheme that prioritizes selection of the second set of servers over the first set of servers, and that is selected as a result of classifying the particular request as a request directed to the second type of content.