HEALTH MONITORING ARCHITECTURE FOR MULTI-TENANT SYSTEM
Some embodiments provide a method for monitoring a system deployed in a cloud. The method deploys a first health monitoring service that monitors a first set of common services of the system deployed in the cloud by directly communicating with each service in the first set of services to determine whether a respective set of aspects of each respective service are properly operational. The first set of common services are accessed by multiple tenants of the system. Within each respective tenant-specific service instance of multiple tenant-specific service instances deployed in the cloud for the tenants, the method deploys a respective health monitoring service that monitors a respective group of microservices of the service instance by directly communicating with each microservice of the tenant-specific service instance to determine whether a respective set of aspects of each respective microservice are properly operational.
1 . A method for monitoring a system deployed in a cloud, the method comprising:
deploying a first health monitoring service that monitors a first set of common services of the system deployed in the cloud by directly communicating with each service in the first set of services to determine whether a respective set of aspects of each respective service are properly operational, the first set of common services accessed by a plurality of tenants of the system; and
within each respective tenant-specific service instance of a plurality of tenant-specific service instances deployed in the cloud for the plurality of tenants, deploying a respective health monitoring service that monitors a respective plurality of microservices of the service instance by directly communicating with each microservice of the tenant-specific service instance to determine whether a respective set of aspects of each respective microservice are properly operational.
2 . The method of claim 1 , wherein:
the system is a network management system that manages a plurality of groups of datacenters for a plurality of different tenants; and
each tenant-specific service instance performs a respective service of the network management system for a respective group of datacenters of a respective tenant.
3 . The method of claim 2 , wherein the first set of common services comprises at least a service for managing tenant subscriptions and a service for managing deployment for the tenant-specific service instances based on tenant requests.
4 . The method of claim 2 , wherein the tenant-specific service instances comprise at least a policy management service instance that manages network policy for a first group of datacenters and a network monitoring service instance that performs flow collection and monitoring for a second group of datacenters.
5 . The method of claim 2 , wherein the tenant-specific service instances comprise at least two service instances of a same type of service instance, each of the at least two service instances performing the same service for a different respective group of datacenters of a respective tenant.
6 . The method of claim 2 , wherein the tenant-specific service instances comprise at least two different types of service instances that perform different services for a same group of datacenters of a particular tenant.
7 . The method of claim 1 , wherein:
the first health monitoring service collects health monitoring data from each service in the first set of services; and
each respective health monitoring service deployed within a respective tenant-specific service instance collects health monitoring data from the microservices of the respective tenant-specific service instance.
8 . The method of claim 7 , wherein each health monitoring service deployed within a tenant-specific service instance provides the collected health monitoring data to the first health monitoring service.
9 . The method of claim 8 , wherein the first health monitoring service stores the health monitoring data collected from the first set of services and from the health monitoring services deployed within the tenant-specific service instances in a unified data store.
10 . The method of claim 1 , wherein a particular health monitoring service communicates with a particular microservice by accessing an internally exposed application programming interface (API) of the particular microservice at regular time intervals.
11 . The method of claim 1 , wherein the respective sets of aspects of at least two different respective microservices that are monitored by a particular health monitoring instance are different.
12 . The method of claim 11 , wherein the sets of aspects exposed by each different microservice to the particular health monitoring instance are determined by developers of the different microservices.
13 . The method of claim 1 , wherein:
the system is deployed within a Kubernetes cluster in a public cloud;
the respective tenant-specific service instances are each deployed in separate respective namespaces of the Kubernetes cluster; and
each respective health monitoring service is deployed in the respective namespace of the respective service instance monitored by the respective health monitoring service.
14 . A non-transitory machine-readable medium storing a program which when executed by at least one processing unit monitors a system deployed in a cloud, the program comprising sets of instructions for:
deploying a first health monitoring service that monitors a first set of common services of the system deployed in the cloud by directly communicating with each service in the first set of services to determine whether a respective set of aspects of each respective service are properly operational, the first set of common services accessed by a plurality of tenants of the system; and
within each respective tenant-specific service instance of a plurality of tenant-specific service instances deployed in the cloud for the plurality of tenants, deploying a respective health monitoring service that monitors a respective plurality of microservices of the service instance by directly communicating with each microservice of the tenant-specific service instance to determine whether a respective set of aspects of each respective microservice are properly operational.
15 . The non-transitory machine-readable medium of claim 14 , wherein:
the system is a network management system that manages a plurality of groups of datacenters for a plurality of different tenants; and
each tenant-specific service instance performs a respective service of the network management system for a respective group of datacenters of a respective tenant.
16 . The non-transitory machine-readable medium of claim 15 , wherein the first set of common services comprises at least a service for managing tenant subscriptions and a service for managing deployment for the tenant-specific service instances based on tenant requests.
17 . The non-transitory machine-readable medium of claim 15 , wherein the tenant-specific service instances comprise at least a policy management service instance that manages network policy for a first group of datacenters and a network monitoring service instance that performs flow collection and monitoring for a second group of datacenters.
18 . The non-transitory machine-readable medium of claim 15 , wherein the tenant-specific service instances comprise at least two service instances of a same type of service instance, each of the at least two service instances performing the same service for a different respective group of datacenters of a respective tenant.
19 . The non-transitory machine-readable medium of claim 15 , wherein the tenant-specific service instances comprise at least two different types of service instances that perform different services for a same group of datacenters of a particular tenant.
20 . The non-transitory machine-readable medium of claim 14 , wherein:
the first health monitoring service collects health monitoring data from each service in the first set of services; and
each respective health monitoring service deployed within a respective tenant-specific service instance collects health monitoring data from the microservices of the respective tenant-specific service instance.
21 . The non-transitory machine-readable medium of claim 20 , wherein:
each health monitoring service deployed within a tenant-specific service instance provides the collected health monitoring data to the first health monitoring service; and
the first health monitoring service stores the health monitoring data collected from the first set of services and from the health monitoring services deployed within the tenant-specific service instances in a unified data store.
22 . The non-transitory machine-readable medium of claim 14 , wherein the respective sets of aspects of at least two different respective microservices that are monitored by a particular health monitoring instance are different
23 . The non-transitory machine-readable medium of claim 14 , wherein:
the system is deployed within a Kubernetes cluster in a public cloud;
the respective tenant-specific service instances are each deployed in separate respective namespaces of the Kubernetes cluster; and
each respective health monitoring service is deployed in the respective namespace of the respective service instance monitored by the respective health monitoring service.