IP Library Granted Patent US 11,165,856
Granted Patent B2
US 11,165,856 · App. 15/496,675 · Granted Nov 2, 2021

Detecting uneven load balancing through multi-level outlier detection

Inventors: Nastaran Baradaran (San Jose, CA); Muraliraja Muniraju (Fremont, CA)
Assignee: Citrix Systems, Inc.
H04L67/1008H04L41/0816H04L41/0893H04L41/142H04L43/0876H04L43/10H04L43/14
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,165,856
App. No.
15/496,675
Granted
Nov 2, 2021
Kind
B2
Abstract

The present disclosure is directed towards systems and methods of detecting a cause of anomalous load balancing among a plurality of servers. A device intermediary to a plurality of clients and a plurality of servers collects values of a plurality of counters. The device identifies a server of the plurality of servers that is an outlier. The device can identify a counter of the plurality of counters that is an outlier based on at least a comparison of values of each of the plurality of counters for each of the plurality of servers. The device can provide, responsive to the determination, an indication that a value of the counter is a factor causing the server to have uneven load balancing during the time interval.

Claims (86)

1. A method of detecting a cause of anomalous load balancing among a plurality of servers, comprising:

collecting, by a device intermediary to a plurality of clients and a plurality of servers, values of a plurality of counters identifying statistics for each server of the plurality of servers and one or more virtual servers of the device load balancing the plurality of servers;

identifying, by an anomaly detector executed by the device, a server of the plurality of servers that is an outlier indicating uneven load balancing based on at least a comparison of a number of server hits for each of the plurality of servers during a time interval;

determining, by the anomaly detector, from the plurality of counters for the identified server, that a counter identifying a statistic is an outlier based on at least a comparison of values of a corresponding counter of the plurality of counters identifying the statistic for each of the plurality of servers; and

providing, by the anomaly detector responsive to the determination, an indication that a value of the counter is a factor causing the server to have uneven load balancing during the time interval.

2. The method of claim 1 , comprising:

establishing the plurality of counters from a predetermined set of counters that are correlated with causes of uneven load balancing.

3. The method of claim 1 , comprising:

determining, for each server of the plurality of servers, a value for each counter of the plurality of counters during the time interval;

determining that the counter is the outlier based on a statistical outlier detection algorithm applied to the value for each counter of the plurality of counters for each of the plurality of servers; and

adjusting a load balancer of the device balancing the load on the plurality of servers to even the load on the plurality of servers.

4. The method of claim 1 , comprising:

for each counter of the plurality of counters, identifying values associated with each of the plurality of servers during the time interval;

determining, based on the identified values and a statistical outlier detection algorithm, that the counter that is the outlier;

identifying a type of counter corresponding to the counter; and

determining, based on a policy and the type of counter, the cause of the anomaly.

5. The method of claim 1 , comprising:

for each counter of the plurality of counters, identifying values associated with each of the plurality of servers during the time interval;

determining, based on the identified values and a statistical outlier detection algorithm, that the counter that is the outlier;

identifying a type of configuration of the counter; and

determining, based on a policy and the type of configuration, the cause of the anomaly from a plurality of candidate causes of anomalies.

6. The method of claim 1 , comprising:

for each counter of the plurality of counters, identifying values associated with each of the plurality of servers during the time interval;

determining, based on the identified values and a statistical outlier detection algorithm, that the counter that is the outlier;

identifying, from a plurality of types of counters, that the counter is not defined by a configuration; and

determining, based on the counter not defined by the configuration, that the counter is the cause of the anomaly in the server that is the outlier during the time interval.

7. The method of claim 1 , comprising:

identifying a second server of the plurality of servers that is a second outlier of uneven load balancing during a second time interval;

for each counter of the plurality of counters, identifying values associated with each of the plurality of servers during the second time interval;

determining, based on the identified values and a statistical outlier detection algorithm, a second counter from the plurality of outlier counters that is a second outlier;

identifying, from a plurality of types of counters, that the second counter is defined by a configuration; and

determining, based on the second counter defined by the configuration, that the second counter is not a cause of an anomaly in the second server during the second time interval.

8. The method of claim 1 , comprising:

identifying a second server of the plurality of servers that is a second outlier of uneven load balancing during a second time interval;

for each counter of the plurality of counters, identifying values associated with each of the plurality of servers during the second time interval;

determining, based on the identified values and a statistical outlier detection algorithm, an absence of any counter among the plurality of counters that is an outlier during the second time interval; and

selecting, based on the absence of any counter that is an outlier, a second indication of a cause of an anomaly in the second server, the second indication different from the indication.

9. The method of claim 1 , comprising:

identifying loads on each of the plurality of servers based on server hits during the time interval.

10. The method of claim 1 , comprising:

forwarding, by the one or more virtual servers, the network traffic from the plurality of client devices to the plurality of servers using at least one of a least connection load balancing technique or a round robin load balancing technique;

determining, by the anomaly detector, the server that is the outlier using a first statistical outlier detection algorithm, the first statistical outlier detection algorithm comprising at least one of a box plot, a model, or a normal probability plot; and

determining, by the anomaly detector, the counter using a second statistical outlier detection algorithm, the second statistical outlier detection algorithm comprising at least one of the box plot, the model, or the normal probability plot.

11. A system to detect a cause of anomalous load balancing among a plurality of servers, comprising:

a monitor executed by a device intermediary to a plurality of clients and a plurality of servers configured to collect values of a plurality of counters identifying statistics for each server of the plurality of servers and one or more virtual servers of the device load balancing the plurality of servers;

an anomaly detector executed by the device configured to:

identify a server of the plurality of servers that is an outlier indicating uneven load balancing based on at least a comparison of a number of server hits for each of the plurality of servers during a time interval;

determine, from the plurality of counters for the identified server, that a counter identifying a statistic is an outlier based on at least a comparison of values of a corresponding counter of the plurality of counters identifying the statistic for each of the plurality of servers; and

provide, responsive to the determination, an indication that a value of the counter is a factor causing the server to have uneven load balancing during the time interval.

12. The system of claim 11 , wherein the device is further configured to:

establish the plurality of counters from a predetermined set of counters that are correlated with causes of uneven load balancing.

13. The system of claim 11 , wherein the device is further configured to:

determine, for each server of the plurality of servers, a value for each counter of the plurality of counters during the time interval;

determine that the counter is the outlier based on a statistical outlier detection algorithm applied to the value for each counter of the plurality of counters for each of the plurality of servers; and

adjust a load balancer of the device balancing the load on the plurality of servers to even the load on the plurality of servers.

14. The system of claim 11 , wherein the device is further configured to:

for each counter of the plurality of counters, identify values associated with each of the plurality of servers during the time interval;

determine, based on the identified values and a statistical outlier detection algorithm, that the counter is the outlier;

identify a type of counter corresponding to the counter; and determine, based on a policy and the type of counter, the cause of the anomaly.

15. The system of claim 11 , wherein the device is further configured to:

for each counter of the plurality of counters, identify values associated with each of the plurality of servers during the time interval;

determine, based on the identified values and a statistical outlier detection algorithm, that the counter is the outlier;

identify a type of configuration of the counter; and

determine, based on a policy and the type of configuration, the cause of the anomaly from a plurality of candidate causes of anomalies.

16. The system of claim 11 , wherein the device is further configured to:

for each counter of the plurality of counters, identify values associated with each of the plurality of servers during the time interval;

determine, based on the identified values and a statistical outlier detection algorithm, that the counter is the outlier;

identify, from a plurality of types of counters, that the counter is not defined by a configuration; and

determine, based on the counter not defined by the configuration, that the counter is the cause of the anomaly in the server that is the outlier during the time interval.

17. The system of claim 11 , wherein the device is further configured to:

identify a second server of the plurality of servers that is a second outlier of uneven load balancing during a second time interval;

for each counter of the plurality of counters, identify values associated with each of the plurality of servers during the second time interval;

determine, based on the identified values and a statistical outlier detection algorithm, a second counter from the plurality of outlier counters that is a second outlier;

identify, from a plurality of types of counters, that the second counter is defined by a configuration; and

determine, based on the second counter defined by the configuration, that the second counter is not a cause of an anomaly in the second server during the second time interval.

18. The system of claim 11 , wherein the device is further configured to:

identify a second server of the plurality of servers that is a second outlier of uneven load balancing during a second time interval;

for each counter of the plurality of counters, identify values associated with each of the plurality of servers during the second time interval;

determine, based on the identified values and a statistical outlier detection algorithm, an absence of any counter among the plurality of counters that is an outlier during the second time interval; and

select, based on the absence of any counter that is an outlier, a second indication of a cause of an anomaly in the second server, the second indication different from the indication.

19. The system of claim 11 , wherein the device is further configured to:

identify loads on each of the plurality of servers based on server hits during the time interval.

20. The system of claim 11 , wherein the device is further configured to:

forward, by the one or more virtual servers, the network traffic from the plurality of client devices to the plurality of servers using at least one of a least connection load balancing technique or a round robin load balancing technique;

determine, by the anomaly detector, the server that is the outlier using a first statistical outlier detection algorithm, the first statistical outlier detection algorithm comprising at least one of a box plot, a model, or a normal probability plot; and

determine, by the anomaly detector, the counter using a second statistical outlier detection algorithm, the second statistical outlier detection algorithm comprising at least one of the box plot, the model, or the normal probability plot.

Assignments (9)
PATENT SECURITY AGREEMENT Recorded Aug 15, 2025
From: CLOUD SOFTWARE GROUP, INC.; CITRIX SYSTEMS, INC.
To: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT
Reel/Frame 072488/0172 →
SECURITY INTEREST Recorded May 24, 2024
From: CLOUD SOFTWARE GROUP, INC. (F/K/A TIBCO SOFTWARE INC.); CITRIX SYSTEMS, INC.
To: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT
Reel/Frame 067662/0568 →
RELEASE AND REASSIGNMENT OF SECURITY INTEREST IN PATENT (REEL/FRAME 062113/0001) Recorded Apr 14, 2023
From: GOLDMAN SACHS BANK USA, AS COLLATERAL AGENT
To: CITRIX SYSTEMS, INC.; CLOUD SOFTWARE GROUP, INC. (F/K/A TIBCO SOFTWARE INC.)
Reel/Frame 063339/0525 →
PATENT SECURITY AGREEMENT Recorded Apr 14, 2023
From: CLOUD SOFTWARE GROUP, INC. (F/K/A TIBCO SOFTWARE INC.); CITRIX SYSTEMS, INC.
To: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT
Reel/Frame 063340/0164 →
PATENT SECURITY AGREEMENT Recorded Oct 7, 2022
From: TIBCO SOFTWARE INC.; CITRIX SYSTEMS, INC.
To: WILMINGTON TRUST, NATIONAL ASSOCIATION, AS NOTES COLLATERAL AGENT
Reel/Frame 062113/0470 →
PATENT SECURITY AGREEMENT Recorded Oct 7, 2022
From: TIBCO SOFTWARE INC.; CITRIX SYSTEMS, INC.
To: BANK OF AMERICA, N.A., AS COLLATERAL AGENT
Reel/Frame 062112/0262 →
SECOND LIEN PATENT SECURITY AGREEMENT Recorded Oct 7, 2022
From: TIBCO SOFTWARE INC.; CITRIX SYSTEMS, INC.
To: GOLDMAN SACHS BANK USA, AS COLLATERAL AGENT
Reel/Frame 062113/0001 →
SECURITY INTEREST Recorded Sep 30, 2022
From: CITRIX SYSTEMS, INC.
To: WILMINGTON TRUST, NATIONAL ASSOCIATION
Reel/Frame 062079/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 26, 2017
From: BARADARAN, NASTARAN; MUNIRAJU, MURALIRAJA
To: CITRIX SYSTEMS, INC.
Reel/Frame 042150/0415 →
Continuity (1)
Related Publication 20180309822A1 · Oct 25, 2018