Function based dynamic traffic management for network services
Technologies are disclosed for local and distributed function based dynamic traffic management for network services. A service host executes a network service and provides a service framework that includes one or more handlers. When a request is received for the service, one of the handlers assigns a classification to the request. The handler then provides the classification to a dynamic function based traffic controller. The controller determines whether the network service is to process the request based on the classification of the request, metrics associated with the network service, and a local traffic management policy. If the controller determines that the network service is not to process the request, the request is rejected. Otherwise, the request is passed to the network service for processing. Metrics can also be provided from the service host to a distributed performance monitoring system for use in managing network traffic at a fleet level.
1. A method comprising:
receiving, at a fleet of service host computers, a service request directed to a network service;
responsive to receiving the service request, determining a classification associated with the service request;
determining whether the service request is to be processed by the network service based at least in part on the classification, one or more real time or near real time metrics associated with the network service, and a distributed traffic management policy, wherein the distributed traffic management policy defines a traffic management function having an input comprising a value of at least one of the one or more real time or near real time metrics and an output defining a throttle rate at which at least one of the fleet of service host computers is to throttle service requests of the classification; and
in response to determining that the service request is not to be processed by the network service, rejecting the service request.
2. The method as recited in claim 1 , wherein the throttle rate applies to each of the fleet of service host computers.
3. The method as recited in claim 1 further comprising, in response to determining that the service request is to be processed by the network service, causing the service request to be processed by the network service.
4. The method as recited in claim 1 , wherein the one or more real time or near real time metrics are provided by a distributed performance monitoring system.
5. The method as recited in claim 4 , wherein the distributed performance monitoring system receives the one or more real time or near real time metrics from a component of each of the fleet of service host computers and provides the one or more real time or near real time metrics to a dynamic function based traffic controller configured to determine whether the service request is to be throttled based at least in part on the one or more real time or near real time metrics, the classification, and the distributed traffic management policy.
6. The method as recited in claim 5 , wherein the component is a local real time performance monitoring system.
7. The method as recited in claim 5 , wherein the distributed performance monitoring system receives the one or more real time or near real time metrics from the component of each of the fleet of service host computers via a distributed messaging system.
8. A computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by a service host computer of a fleet of service host computers, cause the service host computer to:
obtain a service request directed to a network service;
responsive to obtaining the service request, determine a classification associated with the service request;
determine one or more real time or near real time metrics associated with the network service;
determine a traffic management policy, wherein the traffic management policy defines a traffic management function having an input comprising a value of at least one of the one or more real time or near real time metrics and an output defining a throttle rate at which the fleet of service host computers is to throttle service requests of the classification
determine whether the service request is to be processed by the network service based at least in part on the classification, the one or more real time or near real time metrics, and the traffic management policy; and
in response to determining that the service request is not to be processed by the network service, reject the service request.
9. The computer-readable storage medium as recited in claim 8 , having further computer-executable instructions stored thereupon to, in response to determining that the service request is to be processed by the network service, cause the service request to be processed by the network service.
10. The computer-readable storage medium as recited in claim 8 , wherein an operator of the network service specifies the one or more real time or near real time metrics.
11. The computer-readable storage medium as recited in claim 8 , wherein determining the classification of the service request comprises determining the classification by examining metadata added to the service request by processing, by one or more handlers, the service request.
12. The computer-readable storage medium as recited in claim 8 , wherein determining the classification of the service request comprises:
converting the service request into a job, wherein the job comprises a protocol-agnostic data structure; and
determining the classification based at least in part on at least one of a size of the job, an amount of computing resources expected to be consumed during processing of the job, or a probability that the service request was generated by a bot.
13. The computer-readable storage medium as recited in claim 8 , wherein the component is a local real time performance monitoring system.
14. An apparatus, comprising:
one or more processors; and
at least one computer-readable storage medium having instructions stored thereupon which, when executed by the one or more processors, cause the processors to:
receive a service request directed to a network service;
determine a classification of the service request;
determine whether the service request is to be processed by the network service based at least in part on the classification, one or more real time or near real time metrics associated with the network service, and a distributed traffic management policy, wherein the distributed traffic management policy defines a traffic management function having an input comprising a value of at least one of the one or more real time or near real time metrics and an output defining a throttle rate at which a fleet of service host computers is to throttle service requests of the classification; and
in response to determining that the service request is to be processed by the network service, cause the service request to be processed by the network service.
15. The apparatus as recited in claim 14 , wherein the at least one computer-readable storage medium has further computer-executable instructions stored thereupon to, in response to determining that the service request is not to be processed by the network service, reject the service request.
16. The apparatus as recited in claim 14 , wherein determining the classification of the service request comprises:
converting the service request into a job, wherein the job comprises a protocol-agnostic data structure; and
determining the classification based at least in part on at least one of a size of the job, an amount of computing resources expected to be consumed during processing of the job, or a probability that the service request was generated by a bot.
17. The apparatus as recited in claim 14 , wherein the one or more real time or near real time metrics are provided by a distributed performance monitoring system.
18. The apparatus as recited in claim 17 , wherein the distributed performance monitoring system:
receives the one or more real time or near real time metrics from a component of each of the fleet of service host computers; and
provides the one or more real time or near real time metrics to a dynamic function based traffic controller configured to determine whether the service request is to be throttled based at least in part on the one or more real time or near real time metrics, the classification, and the distributed traffic management policy.
19. The apparatus as recited in claim 18 , wherein the component is a local real time performance monitoring system that generates the one or more real time or near real time metrics by periodically sampling a plurality of resources provided or utilized by at least one of the service host computer or network service.
20. The apparatus as recited in claim 18 , wherein the distributed performance monitoring system receives the one or more real time or near real time metrics from the component of each of the fleet of service host computers via a distributed messaging system.