Constraint-based upgrade and deployment
One or more software products/services may be installed on a cloud deployment. Product versions of such services can be upgraded (or rolled back) based on a deployment plan comprising tasks adapted to reach a target deployment state. A central upgrade server can forward tasks to an upgrade agent for execution, the tasks being based on a current operational state of the cloud deployment (obtained by the upgrade agent) and one or more constraints. In multiple node deployments, some nodes may be upgraded to a new product version, while other nodes are kept at a current product version until stability of the new product version is determined. Traffic across nodes can be shaped to ensure a deployment is healthy before upgrading other nodes/deployments. If the health of a node/deployment does not meet specified criteria, an upgrade can be stopped, an alert can be triggered, and the node/deployment can be rolled back.
1 . A system comprising:
a central server configured to coordinate tasks across different deployments; and an agent to perform one or more tasks at a particular deployment, the one or more tasks comprising obtaining one or more snapshots indicating a level of health of the particular deployment, generating a report based on the obtained one or more snapshots, and transmitting the report to the central server, wherein the central server comprises: one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the central server to: receive a subscription from the particular deployment to a particular release channel, wherein the subscription indicates eligibilities of nodes to upgrade within the particular deployment of upgrading when an update in the particular release channel occurs, and the subscription indicates non-eligibilities of any nodes that are unsubscribed to the particular release channel; determine whether a different product release version compared to a currently deployed version of the product is available, within the particular release channel, that satisfies a particular level of health stability of the particular release channel, wherein the determining of whether the different product release version is available comprises: determining that the different product release version from the currently deployed version previously failed to satisfy the particular level of health and is stability and satisfied a lower level of stability; determining that the different product release version has been promoted from the lower level of stability to the particular level of stability based on levels of health associated with other deployments that have been upgraded with the different product release version;
in response to determining that the different product release version is available:
upgrade a first selected proportion of the subscribed nodes from the particular deployment; generate the report based on upgrade statuses of the deployment, wherein the upgrade statuses indicate whether each of the subscribed nodes have been upgraded and health levels of each of the subscribed nodes; and based on the report, selectively upgrade a second selected proportion of the subscribed nodes within the particular deployment according to the different product release version in response to determining that the different product release version compared to the currently deployed version is available, wherein based on the upgrade statuses of one or more other nodes from the particular deployment, wherein at least an unselected proportion of the subscribed nodes maintains the currently deployed version of the product.
2 . The system of claim 1 , wherein the particular release channel comprises a file that comprises protocol of upgrading to the product release version.
3 . The system of claim 1 , wherein the selectively upgrading is in response to determining one or more dependencies being satisfied between the particular deployment and a downstream deployment to which the particular deployment responds, the one or more dependencies comprising a compatibility between the particular deployment and the downstream deployment.
4 . The system of claim 3 , wherein the one or more dependencies comprises a matching schema version between the particular deployment and the downstream deployment.
5 . The system of claim 1 , wherein the selectively upgrading comprises selecting an upgrade mode that satisfies a downtime constraint of the particular deployment.
6 . The system of claim 1 , wherein the agent continuously transmits, to the central server, a snapshot of the particular deployment, and the central server, in response to receiving the snapshot of the particular deployment, transmits a series of tasks to the agent.
7 . The system of claim 6 , wherein the series of tasks comprises a stop task, a change version task, and a restart task.
8 . The system of claim 1 , wherein the selectively upgrading upgrade of the proportion of the subscribed nodes is based on a utilization level of the particular deployment.
9 . The system of claim 1 , wherein the selectively upgrading of the proportion of the subscribed nodes is based on a cascaded upgrade from a first version to a second version to a third version, and a direct upgrade from the first version to the third version is disqualified.
10 . The system of claim 1 , wherein a sum of the first selected proportion and the second selected proportion of the subscribed nodes comprises a subset of the subscribed nodes that is less than an entirety of the subscribed nodes within the particular deployment.
11 . The system of claim 1 , wherein the determining that the product release version satisfies the particular level of health stability of the particular release channel is based on one or more other deployments that received the product release version satisfying a level of health.
12 . The system of claim 1 , wherein selectively upgrading the second selected proportion comprises
the proportion comprises a first proportion, and the instructions further cause the agent to:
determine that the particular deployment, following the selectively upgrading of the first selected proportion, satisfies a level of health; and
in response to determining that the particular deployment following the upgrading satisfies the level of health, upgrading the second selected proportion of the subscribed nodes within the particular deployment in a subsequent iteration.
13 . The system of claim 1 , wherein the particular level of health is based on a level of utilization at the deployment and a response time.
14 . The system of claim 1 , wherein the determining of whether the newer product release version is available is in response to determining that the currently deployed version has been blacklisted.
15 . The system of claim 1 , wherein the snapshots are obtained from one or more logs.
16 . The system of claim 1 , wherein the instructions that, when executed by the one or more processors, cause the central server to:
build a distribution that comprises an outcome of the task, the distribution comprising a compressed version of configuration files corresponding to the task.
17 . The system of claim 1 , wherein the one or more snapshots comprise a plurality of continuously obtained snapshots, and the report is generated based on the continuously obtained snapshots.
18 . The system of claim 1 , wherein the selectively upgrading, by the central server, the second selected proportion of nodes comprises adjusting a target traffic ratio indicating the proportion of upgraded nodes, out of all nodes within the particular deployment, based on a monitored health within the particular deployment following the upgrading of the first selected proportion.
19 . A computer-implemented method, comprising:
coordinating tasks, by a central server, at different deployments;
performing one or more tasks, by an agent, at a particular deployment, the one or more tasks comprising obtaining one or more snapshots indicating a level of health of the particular deployment, generating a report based on the obtained one or more snapshots, and transmitting the report to the central server;
receiving, by the central server, a subscription from the particular deployment to a particular release channel, wherein the subscription indicates eligibilities of nodes to upgrade within the particular deployment of upgrading when an update in the particular release channel occurs, and the subscription indicates non-eligibilities of any nodes that are unsubscribed to the particular release channel;
determining, by the central server, whether a different product release version compared to a currently deployed version of the product is available, within the particular release channel, that satisfies a particular level of health stability of the particular release channel, wherein the determining of whether the different product release version is available comprises:
determining that the different product release version from the currently deployed version previously failed to satisfy the particular level of health and is stability and satisfied a lower level of stability; determining that the different product release version has been promoted from the lower level of stability to the particular level of stability based on levels of health associated with other deployments that have been upgraded with the different product release version;
in response to determining that the different product release version is available:
upgrade a first selected proportion of the subscribed nodes from the particular deployment;
generate the report based on upgrade statuses of the deployment, wherein the upgrade statuses indicate whether each of the subscribed nodes have been upgraded and health levels of each of the subscribed nodes; and
based on the report, selectively upgrading, by the central server, a second selected proportion of the subscribed nodes within the particular deployment according to the different product release version in response to determining that the different product release version compared to the currently deployed version is available, based on the upgrade statuses of one or more other nodes from the particular deployment.