Multi-factor request throttling techniques for streaming data analytics services

1 . A system, comprising:

one or more computing devices;

wherein the one or more computing devices include instructions that upon execution on or across the one or more computing devices:

determine, at a streaming data analysis service, a first set of account-specific request throttling parameters for requests from a first customer account, wherein the first set of account-specific request throttling parameters includes a first request rate acceleration threshold which differs from respective request rate acceleration thresholds of other sets of account-specific request throttling parameters for requests from other customer accounts;

in response to receiving, at the streaming data analysis service via one or more programmatic interfaces from a client associated with the first customer account, a first analysis request for a first stream,

retrieve, to a particular front-end node of a plurality of front-end nodes of the streaming data analysis service, from a consistent distributed cache accessible to the plurality of front-end nodes, a plurality of concurrency snapshot records of the first customer account, wherein an individual concurrency snapshot record of the plurality of concurrency snapshot records indicates at least a number of in-progress streams of the first customer account at a particular time;

compute, at the particular front-end node, using at least the plurality of concurrency snapshot records, a first request rate acceleration metric of the first customer account;

accept, by the particular front-end node, based at least in part on the first request rate acceleration metric and the first request rate acceleration threshold, the first analysis request; and

initiate processing of the first stream;

cause, prior to completion of processing of the first stream, a sequence of timestamped stream-in-progress records associated with the first stream to be stored in a repository other than the consistent distributed cache; and

in response to a subsequent cache miss in the consistent distributed cache, store a particular concurrency snapshot record of the first customer account in the consistent distributed cache, wherein content of the particular concurrency snapshot record is based at least in part on analysis of timestamps of at least a portion of the sequence of timestamped stream-in-progress records associated with the first stream.

2 . The system as recited in claim 1 , wherein to accept the first analysis request, the one or more computing devices include further instructions that upon execution on or across the one or more computing devices:

determine that a current number of in-progress streams of the first customer account is smaller than a first concurrency limit.

3 . The system as recited in claim 1 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more computing devices:

in response to receiving, at the streaming data analysis service via the one or more programmatic interfaces from a second client associated with a second customer account, a second analysis request for a second stream,

determine that acceptance of the second analysis request would not result in the number of in-progress streams of the second customer account exceeding a concurrency limit associated with the second customer account;

retrieve, from the consistent distributed cache, a second plurality of concurrency snapshot records of the second customer account;

compute, using at least the second plurality of concurrency snapshot records, a second request rate acceleration metric of the second customer account; and

reject the second analysis request based at least in part on the second request rate acceleration metric.

4 . The system as recited in claim 1 , wherein the first analysis request results from invocation of a first application programming interface (API) of a plurality of APIs of the streaming data analysis service, wherein the first request rate acceleration metric pertains to requests associated with the first API, wherein the first request rate acceleration threshold pertains to requests associated with the first API, and wherein the first set of account-specific request throttling parameters includes a second request rate acceleration threshold pertaining to a second API of the plurality of APIs.

5 . The system as recited in claim 1 , wherein the one or more computing devices include further instructions that upon execution on or across the one or more computing devices:

in response to receiving, at the streaming data analysis service via the one or more programmatic interfaces from a second client associated with a second customer account, a second analysis request for a second stream,

accept the second analysis request based at least in part on determining that the number of in-progress streams of the second customer account is smaller than a first concurrency level, wherein the second analysis request is accepted without taking a second request rate acceleration metric of the second customer account into consideration.

6 . A computer-implemented method, comprising:

in response to receiving, at a streaming data analysis service via one or more programmatic interfaces from a first client associated with a first customer account, a first analysis request for a first stream,

retrieving, from a first data store, a first set of concurrency snapshot records of the first customer account, wherein an individual concurrency snapshot record of the first set of concurrency snapshot records indicates at least a number of in-progress streams of the first customer account at a particular time;

computing, using at least the first set of concurrency snapshot records, a first request rate acceleration metric of the first customer account;

accepting, based at least in part on the first request rate acceleration metric, the first analysis request; and

initiating analysis of the first stream.

7 . The computer-implemented method as recited in claim 6 , further comprising:

causing, prior to completion of processing of the first stream, one or more timestamped stream-in-progress records associated with the first stream to be stored in a second data store; and

populating one or more concurrency snapshot records of the first data store based at least in part on analysis of the one or more timestamped stream-in-progress records.

8 . The computer-implemented method as recited in claim 6 , wherein said accepting the first analysis request is based at least in part on:

determining that a current number of in-progress streams of the first customer account is smaller than a first concurrency limit.

9 . The computer-implemented method as recited in claim 6 , further comprising:

in response to receiving, at the streaming data analysis service via the one or more programmatic interfaces from a second client associated with a second customer account, a second analysis request for a second stream,

determining that acceptance of the second analysis request would not result in the number of in-progress streams of the second customer account exceeding a second concurrency limit associated with the second customer account;

retrieving, from the first data store, a second set of concurrency snapshot records of the second customer account;

computing, using at least the second set of concurrency snapshot records, a second request rate acceleration metric of the second customer account; and

rejecting the second analysis request based on the second request rate acceleration metric.

10 . The computer-implemented method as recited in claim 6 , wherein the first analysis request results from invocation of a first application programming interface (API) of a plurality of APIs of the streaming data analysis service, wherein the first request rate acceleration metric pertains to requests associated with the first API, and wherein accepting the first analysis request comprises comparing the first request rate acceleration metric to a first acceleration threshold associated with the first API, the computer-implemented method further comprising:

in response to receiving, at the streaming data analysis service from the first client, a second analysis request for a second stream, wherein the second analysis request results from invocation of a second API of the plurality of APIs, determining whether to accept the second analysis request based at least in part on (a) a second request rate acceleration metric which pertains to requests which invoke the second API and (b) a second acceleration threshold associated with the second API.

11 . The computer-implemented method as recited in claim 6 , further comprising:

in response to receiving, at the streaming data analysis service via the one or more programmatic interfaces from a second client associated with a second customer account, a second analysis request for a second stream,

accepting the second analysis request based at least in part on determining that the number of in-progress streams of the second customer account is smaller than a first concurrency level, wherein the second analysis request is accepted without taking a second request rate acceleration metric of the second customer account into consideration.

12 . The computer-implemented method as recited in claim 6 , further comprising:

determining, based at least in part on a resource utilization metric of the streaming data analysis service, that the first request rate acceleration metric is to be computed.

13 . The computer-implemented method as recited in claim 6 , wherein the streaming data analysis service comprises one or more of: (a) a transcription service or (b) a chatbot service.

14 . The computer-implemented method as recited in claim 6 , wherein the first data store comprises a distributed in-memory cache of a cloud-based caching service.

15 . The computer-implemented method as recited in claim 6 , further comprising:

expiring individual ones of the concurrency snapshot records in the first data store based at least in part on a respective expiration parameter.

16 . One or more non-transitory computer-accessible storage media storing program instructions that when executed on or across one or more processors:

in response to receiving, at a streaming data analysis service via one or more programmatic interfaces from a first client associated with a first customer account, a first analysis request for a first stream,

retrieve, from a first data store, a first set of concurrency snapshot records of the first customer account, wherein an individual concurrency snapshot record of the first set of concurrency snapshot records indicates at least a number of in-progress streams of the first customer account at a particular time;

compute, using at least the first set of concurrency snapshot records, a first request rate acceleration metric of the first customer account;

accept, based at least in part on the first request rate acceleration metric, the first analysis request; and

initiate analysis of the first stream.

17 . The one or more non-transitory computer-accessible storage media as recited in claim 16 , storing further program instructions that when executed on or across the one or more processors:

cause, prior to completion of processing of the first stream, one or more timestamped stream-in-progress records associated with the first stream to be stored in a second data store; and

populate one or more concurrency snapshot records of the first data store based at least in part on analysis of the one or more timestamped stream-in-progress records.

18 . The one or more non-transitory computer-accessible storage media as recited in claim 17 , wherein the one or more concurrency snapshot records that are populated at the first data store include a particular concurrency snapshot record which is (a) associated with the first customer account and (b) populated based at least in part on analysis of respective stream-in-progress records of a plurality of streams associated with the first customer account.

19 . The one or more non-transitory computer-accessible storage media as recited in claim 16 , wherein to accept the first analysis request, the one or more non-transitory computer-accessible storage media store further program instructions that when executed on or across the one or more processors:

determine that a current number of in-progress streams of the first customer account is smaller than a first concurrency limit.

20 . The one or more non-transitory computer-accessible storage media as recited in claim 16 , storing further program instructions that when executed on or across the one or more processors:

in response to acceptance of the first analysis request, update a particular concurrency snapshot record in the first data store to increment a number of in-progress streams of the first customer account; and

in response to termination of analysis of the first stream, update another concurrency snapshot record in the first data store to decrement a number of in-progress streams of the first customer account.