Detecting an algorithmic attack against a hosted AI system based on inputs and outputs of the hosted AI system

1. A system comprising:

a memory; and

a processing system coupled to the memory, the processing system configured to:

derive features, which are associated with a known type of algorithmic attack, from numerical representations of queries that are received by a hosted artificial intelligence system and outputs that result from processing of the queries by the hosted artificial intelligence system;

use a feature-based classifier model to generate a classification score, which indicates a likelihood that at least a portion of the queries corresponds to the known type of algorithmic attack, by providing the derived features as inputs to the feature-based classifier model;

compare the classification score to a score threshold that is associated with the known type of algorithmic attack; and

detect an algorithmic attack based at least on the classification score being greater than or equal to the score threshold that is associated with the known type of algorithmic attack.

2. The system of claim 1 , wherein the numerical representations of the queries are locality-sensitive hashes of the queries.

3. The system of claim 1 , wherein the processing system is configured to:

determine that the outputs that result from processing of the queries by the hosted artificial intelligence system exhibit periodicity by spectrally analyzing the outputs; and

derive a first feature based at least on the outputs exhibiting periodicity; and

wherein the outputs exhibiting periodicity weighs in favor of increasing the classification score.

4. The system of claim 1 , wherein the processing system is configured to:

determine a distance between the numerical representations of first queries, which are include in the queries, and a numerical representation of a reference query;

determine differences between outputs that result from processing the first queries by the hosted artificial intelligence system and a reference output that results from processing the reference query by the hosted artificial intelligence system;

determine ratios for the first queries, a ratio being equal to the difference between the output that results from processing the first query by the hosted artificial intelligence system and the reference output divided by the distance between the numerical representation of the first query and the numerical representation of the reference query;

select a ratio from the ratios that is no less than each other ratio in the ratios to provide a selected ratio; and

derive a first feature based at least on the selected ratio;

wherein the selected ratio being relatively high weighs in favor of increasing the classification score; and

wherein the selected ratio being relatively low weighs against increasing the classification score.

5. The system of claim 1 , wherein the processing system is configured to:

determine distances between the numerical representations of the queries and a nearest-fitted line that is defined by the numerical representations of the queries;

determine a difference value based at least on the distances between the nearest-fitted line and the numerical representations of the queries; and

derive a first feature based at least on the difference value;

wherein the difference value being relatively low weighs in favor of increasing the classification score; and

wherein the difference value being relatively high weighs against increasing the classification score.

6. The system of claim 1 , wherein the processing system is configured to:

determine that the numerical representations of the queries define a Gaussian distribution; and

derive a first feature based at least on the Gaussian distribution; and

wherein the numerical representations of the queries defining the Gaussian distribution weighs in favor of increasing the classification score.

7. The system of claim 6 , wherein the processing system is configured to:

derive the first feature based at least on a standard deviation of the Gaussian distribution;

wherein the standard deviation being relatively low weighs in favor of increasing the classification score; and

wherein the standard deviation being relatively high weighs against increasing the classification score.

8. The system of claim 1 , wherein the processing system is configured to:

determine an entropy associated with the numerical representations of the queries; and

derive a first feature based at least on the entropy;

wherein a relatively low entropy weighs in favor of increasing the classification score; and

wherein a relative high entropy weighs against increasing the classification score.

9. The system of claim 1 , wherein the processing system is configured to:

determine confidences associated with the outputs that result from processing of the queries by the hosted artificial intelligence system;

determine an entropy associated with the confidences; and

derive a first feature based at least on the entropy;

wherein a relatively low entropy weighs in favor of increasing the classification score; and

wherein a relative high entropy weighs against increasing the classification score.

10. The system of claim 1 , wherein the processing system is configured to:

determine confidences associated with the outputs that result from processing of the queries by the hosted artificial intelligence system;

determine entropies at times that correspond to the outputs, wherein the entropies are based on the confidences associated with the outputs and confidences of preceding outputs that precede the outputs;

determine that a trend of the entropies decreases with reference to time; and

derive a first feature based at least on the trend of the entropies decreasing with reference to time; and

wherein the trend of the entropies decreasing with reference to time weighs in favor of increasing the classification score.

11. The system of claim 1 , wherein the processing system is configured to:

determine an entropy associated with the numerical representations of the queries; and

derive a first feature based at least on the entropy;

wherein a relatively high entropy weighs in favor of increasing the classification score; and

wherein a relative low entropy weighs against increasing the classification score.

12. The system of claim 1 , wherein the processing system is configured to:

determine confidences associated with the outputs that result from processing of the queries by the hosted artificial intelligence system;

determine an entropy associated with the confidences; and

derive a first feature based at least on the entropy;

wherein a relatively high entropy weighs in favor of increasing the classification score; and

wherein a relative low entropy weighs against increasing the classification score.

13. The system of claim 1 , wherein the processing system is configured to:

determine confidences associated with the outputs that result from processing of the queries by the hosted artificial intelligence system;

determine entropies at times that correspond to the outputs, wherein the entropies are based on the confidences associated with the outputs and confidence of preceding outputs that precede the outputs;

determine that a trend of the entropies increases with reference to time; and

derive a first feature based at least on the trend of the entropies increasing with reference to time; and

wherein the trend of the entropies increasing with reference to time weighs in favor of increasing the classification score.

14. The system of claim 1 , wherein the processing system is configured to:

determine confidences associated with the outputs that result from processing of the queries by the hosted artificial intelligence system;

determine entropies at times that correspond to the outputs, wherein the entropies are based on the confidences associated with the outputs and confidences of preceding outputs that precede the outputs;

determine that a trend of the entropies increases or decreases with reference to time;

determine that a rate at which the trend of the entropies increases or decreases with reference to time increases with reference to time; and

derive a first feature based at least on the trend of the entropies increasing or decreasing with reference to time and further based at least on the rate increasing with reference to time; and

wherein a combination of the trend of the entropies increasing or decreasing with reference to time and the rate increasing with reference to time weighs in favor of increasing the classification score.

15. A method implemented by a computing system, the method comprising:

deriving features, which are associated with a known type of algorithmic attack, from numerical representations of queries that are received by a hosted artificial intelligence system and outputs that result from processing of the queries by the hosted artificial intelligence system;

using a feature-based classifier model to generate a classification score, which indicates a likelihood that at least a portion of the queries corresponds to the known type of algorithmic attack, by providing the derived features as inputs to the feature-based classifier model;

comparing the classification score to a score threshold that is associated with the known type of algorithmic attack; and

detecting an algorithmic attack based at least on the classification score being greater than or equal to the score threshold that is associated with the known type of algorithmic attack.

16. The method of claim 15 , wherein deriving the features comprises:

determining that the outputs that result from processing of the queries by the hosted artificial intelligence system exhibit periodicity by spectrally analyzing the outputs; and

deriving a first feature based at least on the outputs exhibiting periodicity; and

wherein the outputs exhibiting periodicity weighs in favor of increasing the classification score.

17. The method of claim 15 , wherein deriving the features comprises:

determining a distance between the numerical representations of first queries, which are include in the queries, and a numerical representation of a reference query;

determining differences between outputs that result from processing the first queries by the hosted artificial intelligence system and a reference output that results from processing the reference query by the hosted artificial intelligence system;

determining ratios for the first queries, a ratio being equal to the difference between the output that results from processing the first query by the hosted artificial intelligence system and the reference output divided by the distance between the numerical representation of the first query and the numerical representation of the reference query;

selecting a ratio from the ratios that is no less than each other ratio in the ratios to provide a selected ratio; and

deriving a first feature based at least on the selected ratio;

wherein the selected ratio being relatively high weighs in favor of increasing the classification score; and

wherein the selected ratio being relatively low weighs against increasing the classification score.

18. The method of claim 15 , wherein deriving the features comprises:

determining distances between the numerical representations of the queries and a nearest-fitted line that is defined by the numerical representations of the queries;

determining a difference value based at least on the distances between the nearest-fitted line and the numerical representations of the queries; and

deriving a first feature based at least on the difference value;

wherein the difference value being relatively low weighs in favor of increasing the classification score; and

wherein the difference value being relatively high weighs against increasing the classification score.

19. The method of claim 15 , wherein deriving the features comprises:

determining that the numerical representations of the queries define a Gaussian distribution; and

deriving a first feature based at least on the Gaussian distribution; and

wherein the numerical representations of the queries defining the Gaussian distribution weighs in favor of increasing the classification score.

20. The method of claim 15 , wherein deriving the features comprises:

determining an entropy associated with the numerical representations of the queries; and

deriving a first feature based at least on the entropy; and

wherein:

a relatively low entropy weighs in favor of increasing the classification score, and a relative high entropy weighs against increasing the classification score; or

a relatively high entropy weighs in favor of increasing the classification score, and a relative low entropy weighs against increasing the classification score.

21. The method of claim 15 , wherein deriving the features comprises:

determining confidences associated with the outputs that result from processing of the queries by the hosted artificial intelligence system;

determining an entropy associated with the confidences; and

deriving a first feature based at least on the entropy; and

wherein:

a relatively low entropy weighs in favor of increasing the classification score, and a relative high entropy weighs against increasing the classification score; or

a relatively high entropy weighs in favor of increasing the classification score, and a relative low entropy weighs against increasing the classification score.

22. The method of claim 15 , wherein deriving the features comprises:

determining confidences associated with the outputs that result from processing of the queries by the hosted artificial intelligence system;

determining entropies at times that correspond to the outputs, wherein the entropies are based on the confidences associated with the outputs and confidence of preceding outputs that precede the outputs;

determining that a trend of the entropies increases with reference to time; and

deriving a first feature based at least on the trend of the entropies increasing with reference to time; and

wherein the trend of the entropies increasing with reference to time weighs in favor of increasing the classification score.

23. A computer program product comprising a computer-readable storage medium having instructions recorded thereon for enabling a processor-based system to perform operations, the operations comprising:

deriving features, which are associated with a known type of algorithmic attack, from numerical representations of queries that are received by a hosted artificial intelligence system and outputs that result from processing of the queries by the hosted artificial intelligence system;

using a feature-based classifier model to generate a classification score, which indicates a likelihood that at least a portion of the queries corresponds to the known type of algorithmic attack, by providing the derived features as inputs to the feature-based classifier model;

comparing the classification score to a score threshold that is associated with the known type of algorithmic attack; and

detecting an algorithmic attack based at least on the classification score being greater than or equal to the score threshold that is associated with the known type of algorithmic attack.