Recovery from failure in a dynamic scalable services mesh

1. A method of improved recovery from failure of a service instance, in a service chain of services that perform at least services A and B, using service instance AA and service instances BA and BB to perform the services A and B, respectively, the method including:

the service instance BA receiving from the service instance AA a first packet in a stream for a subscriber, wherein the first packet includes an added header which includes a stream affinity code that is consistent for packets in the stream;

the service instance BA, in a primary role specified in a service map distributed to service instances, processing the first packet by performing service B;

the service instance BA identifying the service instance BB as having a secondary role for packets carrying the stream affinity code, which is specified in the service map distributed to service instances, and synchronizing service instance BA state information with the service instance BB after processing the first packet;

after failure of the service instance BA, the service instance AA receiving an updated service map, and preparing to forward a second packet, which includes the same stream affinity code as the first packet, to the service instance BA for performance of the service B, including:

determining from the updated service map that the service instance BA is no longer available;

determining from the updated service map that the service instance BB has the secondary role; and

forwarding the second packet to the service instance BB instead of the service instance BA.

2. The method of claim 1 , wherein the service chain is a security service chain for a subscriber and at least the service B is a security service.

3. The method of claim 1 , wherein the stream affinity code is included in an added header as an added IP header as IP source and destination.

4. The method of claim 1 , further including the packet carrying a service chain for a subscriber in an added packet header and the service B being among services specified in the service chain.

5. The method of claim 1 , wherein instances of the service A and the service B run in containers and the containers are hosted in pods.

6. The method of claim 1 , wherein instances of the service A and the service B are implemented on virtual machines, bare metal servers or custom hardware.

7. The method of claim 1 , wherein failure of the service instance BA is detected by a monitoring agent, including:

monitoring the service instance BA for packet processing activity; and

causing updating of the service map for the service B to remove the service instance BA from availability should it be inactive for a configurable predetermined amount of time.

8. The method of claim 1 , further including the service instance BB:

processing the second packet and based on the processing:

identifying a next service, among at least two additional services to which the subscriber has subscribed, that should next handle the packet; and

routing the processed second packet to the identified next service upon egress from the service instance BB.

9. The method of claim 1 , further including processing a plurality of packets in a stream through the service chain of services and directing the packets for processing, as a document, to a cloud access security broker (abbreviated CASB) that controls exfiltration of sensitive content in documents stored on cloud-based services in use by users of an organization, by monitoring manipulation of the documents.

10. A tangible non-transitory computer readable storage media, including program instructions loaded into memory that, when executed on processors, cause the processors to implement a method of improved recovery from failure of a service instance in a service chain of services that perform at least services A and B, using service instance AA and service instances BA and BB to perform the services A and B, respectively, the method including:

the service instance BA receiving from the service instance AA a first packet in a stream for a subscriber, wherein the first packet includes an added header which includes a stream affinity code that is consistent for packets in the stream;

the service instance BA, in a primary role specified in a service map distributed to service instances, processing the first packet by performing service B;

the service instance BA identifying the service instance BB as having a secondary role for packets carrying the stream affinity code, which is specified in the service map distributed to service instances, and synchronizing service instance BA state information with the service instance BB after processing the first packet;

after failure of the service instance BA, the service instance AA receiving an updated service map, and preparing to forward a second packet, which includes the same stream affinity code as the first packet, to the service instance BA for performance of the service B, including:

determining from the updated service map that the service instance BA is no longer available; and

determining from the updated service map that the service instance BB has the secondary role; and

forwarding the second packet to the service instance BB instead of the service instance BA.

11. The tangible non-transitory computer readable storage media of claim 10 , wherein the service chain is a security service chain for a subscriber and at least the service B is a security service.

12. The tangible non-transitory computer readable storage media of claim 10 , wherein the stream affinity code is included in an added header as an added IP header as IP source and destination.

13. The tangible non-transitory computer readable storage media of claim 10 , wherein instances of the service A and the service B run in containers and the containers are hosted in pods.

14. The tangible non-transitory computer readable storage media of claim 10 , wherein instances of the service A and the service B are implemented on virtual machines, bare metal servers or custom hardware.

15. The tangible non-transitory computer readable storage media of claim 10 , further including the service instance BB:

processing the second packet and based on the processing:

identifying a next service, among at least two additional services to which the subscriber has subscribed, that should next handle the packet; and

routing the processed second packet to the identified next service upon egress from the service instance BB.

16. A system for improved recovery from failure of a service instance in a service chain of services that perform at least services A and B, using service instance AA and service instances BA and BB to perform the services A and B, respectively, the system including a processor, memory coupled to the processor, and computer instructions from the non-transitory computer readable storage media of claim 10 loaded into the memory.

17. The system of claim 16 , wherein the stream affinity code is included in an added header as an added IP header as IP source and destination.

18. The system of claim 16 , further including the packet carrying a service chain for a subscriber in an added packet header and the service B being among services specified in the service chain.

19. The system of claim 16 , wherein failure of the service instance BA is detected by a monitoring agent, including:

monitoring the service instance BA, for packet processing activity; and

causing updating of the service map for the service B to remove the service instance BA from availability should it be inactive for a configurable predetermined amount of time.

20. The system of claim 16 , further including, processing a plurality of packets in a stream through the service chain of services and directing the packets for processing, as a document, to a cloud access security broker (abbreviated CASB) that controls exfiltration of sensitive content in documents stored on cloud-based services in use by users of an organization, by monitoring manipulation of documents.