IP Library Granted Patent US 12,360,760
Granted Patent B2
US 12,360,760 · App. 18/353,839 · Granted Jul 15, 2025

Cloud services release orchestration

Inventors: Sreeram Duvur (Fremont, CA); Vijayanth Devadhar (Fremont, CA); Matthew Gainsborough (Oakland, CA); Kiet Phong (Newcastle, WA); Sathish Santhanam (Fremont, CA); Lawrence Thomas Lopez (Cupertino, CA)
Assignee: Salesforce, Inc.
G06F8/65
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,360,760
App. No.
18/353,839
Granted
Jul 15, 2025
Kind
B2
Abstract

According to some implementations, while a proxy routes production traffic to a first application (app) version that runs in a plurality of container orchestration system (cos) pods having first app version containers, configuration information is received including an identification of a second app version container image for a second app version. The second app version is an updated version of the first app version. Cos pods having second app version containers are brought up based on the second app version container image identified in the configuration information. Test and/or warmup traffic is caused to be routed to the second app version containers. Responsive to an indication regarding the routing of the test and/or warmup traffic to the second app version, causing a transition to sending production traffic to the second app version containers instead of to the first app version.

Claims (43)

1. A non-transitory machine-readable storage medium that provides instructions that, if executed by a machine, are configurable to cause said machine to perform operations according to a pipeline to deploy, run acceptance tests, and release to production new versions of container applications, the operations comprising:

while ingress to a managed service built on Kubernetes is configured to route production traffic to an existing version of a container application (app) running in a container within a replica, receiving configuration information including an identification of a container image for an updated version of the container app;

causing the creation of a container within a replica to run the updated version of the container app based on the container image;

configuring, to enable testing of the replica running the updated version, the ingress to distinguish between traffic designated for the container within the replica running the updated version and production traffic, and to direct traffic designated for the container within the replica running the updated version while the production traffic continues to be routed to the container within the replica running the existing version;

responsive to different ones of a plurality of control signals, including responses to health probes of which at least one is a readiness probe, performing the following:

automatically configuring, according to a set of parameters in the configuration information, ingress to start routing production traffic to the container within the replica running the updated version instead of to the existing version; and

automatically starting a process to shut down the container within the replica running the existing version only after an affirmative response to the readiness probe.

2. The non-transitory machine-readable storage medium of claim 1 , wherein the traffic designated for the container within the replica running the updated version is test and/or warmup traffic.

3. The non-transitory machine-readable storage medium of claim 2 , wherein the operations further comprise:

causing generation and transmission of the test and/or warmup traffic.

4. The non-transitory machine-readable storage medium of claim 1 , wherein the process comprises:

starting a timer based on a time period and instructing the container within the replica running the existing version to shut down; and

when the timer expires, forcing the container within the replica running the existing version to shut down if it has not already.

5. The non-transitory machine-readable storage medium of claim 4 , wherein the time period is indicated in the configuration information.

6. The non-transitory machine-readable storage medium of claim 1 , wherein the traffic is external HTTP traffic.

7. The non-transitory machine-readable storage medium of claim 1 , wherein the operations are to perform container app versioning.

8. The non-transitory machine-readable storage medium of claim 1 , wherein the process is to allow for a graceful shutdown.

9. The non-transitory machine-readable storage medium of claim 1 , wherein the health probes are HTTP probes.

10. The non-transitory machine-readable storage medium of claim 1 , wherein the managed service is to run containerized applications on a serverless platform.

11. The non-transitory machine-readable storage medium of claim 1 , wherein the ingress of the managed service includes built-in support for external HTTP-based ingress.

12. The non-transitory machine-readable storage medium of claim 1 , wherein the ingress of the managed service is a built-in external ingress feature.

13. The non-transitory machine-readable storage medium of claim 1 , wherein the operations are to deploy application updates with zero downtime:

repeating the causing the creation of a container within a replica to run the updated version of the container app based on the container image until their number meets a number from the configuration information.

14. The non-transitory machine-readable storage medium of claim 1 , wherein the operations further comprise:

monitoring whether production traffic is being sent to the container within the replica running the updated version; and

responsive to requests as to whether the container within the replica running the updated version is ready, that container responding affirmatively after determining that production traffic is being sent to that container.

15. The non-transitory machine-readable storage medium of claim 14 , wherein the monitoring includes:

deriving a URL based on the configuration information; and

pinging that URL until it indicates that the container within the replica running the updated version is receiving production traffic.

16. The non-transitory machine-readable storage medium of claim 14 , wherein the requests inquiring whether the container within the replica running the updated version is ready are readiness probes.

17. The non-transitory machine-readable storage medium of claim 1 , wherein the existing version communicates with a database management system (DBMS) using a database name assigned to the existing version, wherein the configuration information includes DBMS connection information for the existing version and the updated version, and the operations further include:

preparing the DBMS by loading app version specific database release artifacts within the container image for the updated version using the DBMS connection information for the updated version.

18. The non-transitory machine-readable storage medium of claim 17 , wherein the operations further include the preliminary operations of:

preloading the DBMS connection information for the updated version into the DBMS; and

registering the container image for the updated version in a container registry.

19. The non-transitory machine-readable storage medium of claim 17 , wherein the DBMS connection information for the existing version and the updated version each include database credentials.

20. The non-transitory machine-readable storage medium of claim 19 , wherein the database credentials including database usernames and passwords.

21. The non-transitory machine-readable storage medium of claim 19 , wherein the DBMS connection information includes a database instance name.

22. The non-transitory machine-readable storage medium of claim 17 , wherein the app version specific database release artifacts include a PL/SQL package and a set or one or more query optimization hints.

23. The non-transitory machine-readable storage medium of claim 17 , wherein the operations further comprise:

causing a set of post release database cleanup activities related to the existing version.

24. The non-transitory machine-readable storage medium of claim 1 , wherein operations further comprise:

repeating the causing the creation of a container within a replica to run the updated version of the container app based on the container image until their number meets a number from the configuration information.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jul 18, 2023
From: DUVUR, SREERAM; DEVADHAR, VIJAYANTH; GAINSBOROUGH, MATTHEW; PHONG, KIET; SANTHANAM, SATHISH; LOPEZ, LAWRENCE THOMAS
To: SALESFORCE.COM, INC.
Reel/Frame 064290/0893 →
CHANGE OF NAME Recorded Jul 18, 2023
From: SALESFORCE.COM, INC.
To: SALESFORCE, INC.
Reel/Frame 064293/0732 →
Continuity (4)
Continuation 18049265 · Oct 24, 2022
Continuation 17225143 · Apr 8, 2021
Continuation 16261501 · Jan 29, 2019
Related Publication 20230385051A1 · Nov 30, 2023
References Cited (87)
US 6502102B1 · Haswell et al. · 2002 [cited by applicant]
US 6701514B1 · Haswell et al. · 2004 [cited by applicant]
US 7155462B1 · Singh et al. · 2006 [cited by applicant]
US 7730478B2 · Weissman · 2010 [cited by applicant]
US 8793691B2 · Devadhar · 2014 [cited by applicant]
US 9268605B2 · Wang et al. · 2016 [cited by applicant]
US 9367595B1 · Malks et al. · 2016 [cited by applicant]
US 9448790B2 · Collison et al. · 2016 [cited by applicant]
US 9519547B2 · Devadhar · 2016 [cited by applicant]
US 9858187B2 · Sundaravaradan et al. · 2018 [cited by applicant]
US 9984002B2 · Sundaravaradan et al. · 2018 [cited by applicant]
US 9990400B2 · Sundaravaradan et al. · 2018 [cited by applicant]
US 10013501B2 · Sundaravaradan et al. · 2018 [cited by applicant]
US 10019297B2 · Helmich et al. · 2018 [cited by applicant]
US 10089209B1 · Russell · 2018 [cited by applicant]
US 10162650B2 · Frank et al. · 2018 [cited by applicant]
US 10218813B2 · Peschansky et al. · 2019 [cited by applicant]
US 10303576B1 · Seymour et al. · 2019 [cited by applicant]
US 10452997B2 · Yang et al. · 2019 [cited by applicant]
US 10474559B2 · Moorthi et al. · 2019 [cited by applicant]
US 10491664B2 · Tan et al. · 2019 [cited by applicant]
US 10558457B1 · Kolb et al. · 2020 [cited by applicant]
US 10768923B2 · Phong et al. · 2020 [cited by applicant]
US 10776099B2 · Duvur et al. · 2020 [cited by applicant]
US 10810003B2 · Gainsborough et al. · 2020 [cited by applicant]
US 10817497B2 · Zaslavsky et al. · 2020 [cited by applicant]
US 10841152B1 · Humphreys · 2020 [cited by examiner]
US 10904122B2 · Wang et al. · 2021 [cited by applicant]
US 11108847B2 · Tan et al. · 2021 [cited by applicant]
US 11507364B2 · Duvur et al. · 2022 [cited by applicant]
US 20110264765A1 · Devadhar · 2011 [cited by applicant]
US 20120330913A1 · Devadhar · 2012 [cited by applicant]
US 20130311926A1 · Keegan et al. · 2013 [cited by applicant]
US 20140075445A1 · Wang et al. · 2014 [cited by applicant]
US 20140143401A1 · Carlen et al. · 2014 [cited by applicant]
US 20140304246A1 · Helmich et al. · 2014 [cited by applicant]
US 20150082374A1 · Dobson et al. · 2015 [cited by applicant]
US 20150128105A1 · Sethi et al. · 2015 [cited by applicant]
US 20150379579A1 · Chiang et al. · 2015 [cited by applicant]
US 20160119244A1 · Wang et al. · 2016 [cited by applicant]
US 20160134596A1 · Kovacs et al. · 2016 [cited by applicant]
US 20160239011A1 · Lo et al. · 2016 [cited by applicant]
US 20160316310A1 · Strom et al. · 2016 [cited by applicant]
US 20170054797A1 · Helmich et al. · 2017 [cited by applicant]
US 20170116130A1 · Sundaravaradan et al. · 2017 [cited by applicant]
US 20170116135A1 · Sundaravaradan et al. · 2017 [cited by applicant]
US 20170180249A1 · Shen et al. · 2017 [cited by applicant]
US 20170286614A1 · Morris et al. · 2017 [cited by applicant]
US 20180011699A1 · Manthiramoorthy · 2018 [cited by examiner]
US 20180146069A1 · Du et al. · 2018 [cited by applicant]
US 20180210814A1 · Mevorach et al. · 2018 [cited by applicant]
US 20180300220A1 · Agarwal · 2018 [cited by examiner]
US 20190020665A1 · Surcouf et al. · 2019 [cited by applicant]
US 20190058722A1 · Levin et al. · 2019 [cited by applicant]
US 20190171443A1 · Nagae · 2019 [cited by applicant]
US 20190212998A1 · Vallala · 2019 [cited by examiner]
US 20190289057A1 · Chen et al. · 2019 [cited by applicant]
US 20200099610A1 · Heron et al. · 2020 [cited by applicant]
US 20200241863A1 · Duvur et al. · 2020 [cited by applicant]
Adrian Holzer et al., Mobile App Development: Native or Web?, [Online], pp. 1-7, [Retrieved from Internet on Jul. 2, 2022], <https://d1wqtxts1xzle7.cloudfront.net/38245540/camera-ready-2012-WEB-Holzer-Ondrus-with-cover-… [cited by applicant]
Advisory Action for U.S. Appl. No. 16/261,501, dated Dec. 30, 2020, 7 pages. [cited by applicant]
Ali J. Fahs et al., “Proximity-Aware Traffic Routing in Distributed Fog Computing Platforms”, [Online], pp. 478-487, [Retrieve from Internet on Feb. 12, 2021], <https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8… [cited by applicant]
Amaral, et al., “Performance Evaluation of Microservices Architectures using Containers”, [Online], pp. 27-34, [Retrieved from Internet on Mar. 28, 2020], (Year: 2015). [cited by applicant]
Andrea Tosatto et al., “Container-based orchestration in cloud: state of the art and challenges”, [Online], p. 70-75, [Retrieved from Internet on Jul. 2, 2022], <https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=7185… [cited by applicant]
Carleto D., “Docker Enterprise 3.0 Brings Docker Kubernetes Services, New Lifecycle Automation Tools, and More,” InfoQ, retrieved from https://www.infoq.com/news/2019/07/docker-enterprise-3/, Jul. 29, 2019, 6 pages. [cited by applicant]
Cong-Phuoc Hoang et al., “An Extended Virtual Network Functions Manager Architecture to Support Container”, [Online], pp. 173-176, [Retrieved from Internet on Feb. 12, 2021], <https://dl.acm.org/doi/pdf/10.1145/3209914.… [cited by applicant]
Corrected Notice of Allowability for U.S. Appl. No. 16/261,495, dated Jul. 31, 2020, 9 pages. [cited by applicant]
Corrected Notice of Allowability for U.S. Appl. No. 16/261,504, dated Aug. 12, 2020, 6 pages. [cited by applicant]
Corrected Notice of Allowability for U.S. Appl. No. 16/261,504, dated Jul. 23, 2020, 6 pages. [cited by applicant]
Eduardo Saiz et al., “A Cloud Platform for QOE Evaluation: Qoxcloud”, [Online], pp. 1-7, [Retrieved from Internet on Jul. 2, 2022], <https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=6858471> (Year: 2014). [cited by applicant]
Final Office Action, U.S. Appl. No. 16/261,501, Oct. 16, 2020, 21 pages. [cited by applicant]
Hamdani D., “Announcing Docker Enterprise 3.0: Delivering High-Velocity Application Innovation,” Docker Blog, retrieved from https://blog.docker.com/2019/04/announcing-docker-enterprise-3-0/, Apr. 30, 2019, 5 pages. [cited by applicant]
Helgevold T., “Deploying Kubemetes with Zero Downtime,” retrieved from https://www.syntaxsuccess.com/viewarticle/deploying-kubemetes-with-zero-downtime, Aug. 3, 2019, 6 pages. [cited by applicant]
Kugathasan Janarthanan et al., “Policies Based Container Migration Using Cross-Cloud Management Platform”, [Online], pp. 1-5, [Retrieved from Internet on Feb. 12, 2021], <https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&… [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 16/261,501, dated Apr. 3, 2020, 23 pages. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 16/261,504, dated Mar. 12, 2020, 12 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 16/261,495, dated Apr. 3, 2020, 12 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 16/261,495, dated Jun. 16, 2020, 7 pages. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 16/261,504, dated Jun. 18, 2020, 15 pages. [cited by applicant]
Notice of Allowance, U.S. Appl. No. 16/261,501, Feb. 19, 2021, 10 pages. [cited by applicant]
Notice of Allowance, U.S. Appl. No. 17/225,143, Jul. 15, 2022, 11 pages. [cited by applicant]
Notice of Allowance, U.S. Appl. No. 18/049,265, Apr. 17, 2023, 14 pages. [cited by applicant]
Pahl, C., “Containerization and the PaaS Cloud”, [Online], pp. 24-31, [Retrieved from Internet on Mar. 28, 2020], (Year: 2015). [cited by applicant]
Pilon A., “Docker Enterprise 3.0 Introduces New Tools for Container-Based Applications,” TechGenix, retrieved from techgenix.com/docker-enterprise-3/, May 20, 2019, 6 pages. [cited by applicant]
Steven Van Rossem et al., “Introducing Development Features for Virtualized Network Services”, [Online], pp. 184-192, [ Retrieved from Interenton Feb. 12, 2021], <https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber… [cited by applicant]
Tosatto, et al., “Container-based orchestration in cloud: state of the art and challenges,” [Online], pp. 70-75, [Retrieved from Interent on Mar. 28, 2020],, (Year: 2015). [cited by applicant]
Townend, et al., “Improving Data Center Efficiency Through Holistic Scheduling in Kubernetes”, [Online], pp. 156-166, [Retrieved from Internet on Mar. 28, 2020], (Year: 2019). [cited by applicant]