Data masking in a microservice architecture
A method includes receiving an object from a client as part of a data ingestion process, directing the object to an object access microservice, providing a copy of the object to a masker worker microservice, masking the copy of the object to create a masked object, and the masking of the copy of the object is performed inline prior to storage of the object, deduplicating the object, and storing the masked object and the object in storage.
1. A method, comprising:
receiving an object from a client as part of a data ingestion process;
directing the object to an object access microservice;
providing a copy of the object to a masker worker microservice;
masking the copy of the object to create a masked object, wherein the masking of the copy of the object is performed inline prior to storage of the object;
deduplicating the object and the masked object; and
storing the masked object and the object in storage.
2. The method as recited in claim 1 , wherein the masking is performed at an object level.
3. The method as recited in claim 1 , wherein the object is directed to the object access microservice by a load balancer.
4. The method as recited in claim 1 , wherein a load balancer provides the copy of the object to the masker worker microservice.
5. The method as recited in claim 1 , wherein the masker worker microservice is provided by a masker worker that is a member of a scalable group of masker workers that are each capable of providing an instance of the masker worker microservice, and a load balancer apportions a masking workload among the masker workers in the group.
6. The method as recited in claim 1 , wherein at least part of the method is performed in a container orchestration system.
7. The method as recited in claim 1 , wherein the object access microservice creates the copy of the object.
8. The method as recited in claim 1 , wherein masking the copy of the object comprises masking one or more segments of the copy of the object to create masked segments, and the method further comprises:
creating respective fingerprints for each of the masked segments; and
creating copies of one or more unmasked segments of the object;
wherein deduplicating the object and the masked object comprises identifying any duplicate segments among the copies of one or more unmasked segments and omitting creation of fingerprints for any identified duplicate segments, and wherein storing the masked object and the object comprises storing any masked segments, and storing respective pointers to any duplicate segments.
9. The method as recited in claim 1 , further comprising receiving another object from the client, storing the another object in storage, and masking a copy of the another object after the another object has been stored.
10. The method as recited in claim 1 , further comprising calculating, and storing, a fingerprint of the masked object.
11. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
receiving an object from a client as part of a data ingestion process;
directing the object to an object access microservice;
providing a copy of the object to a masker worker microservice;
masking the copy of the object to create a masked object, wherein the masking of the copy of the object is performed inline prior to storage of the object;
deduplicating the object and the masked object; and
storing the masked object and the object in storage.
12. The non-transitory storage medium as recited in claim 11 , wherein the masking is performed at an object level.
13. The non-transitory storage medium as recited in claim 11 , wherein the object is directed to the object access microservice by a load balancer.
14. The non-transitory storage medium as recited in claim 11 , wherein a load balancer provides the copy of the object to the masker worker microservice.
15. The non-transitory storage medium as recited in claim 11 , wherein the masker worker microservice is provided by a masker worker that is a member of a scalable group of masker workers that are each capable of providing an instance of the masker worker microservice, and a load balancer apportions a masking workload among the masker workers in the group.
16. The non-transitory storage medium as recited in claim 11 , wherein at least some of the operations are performed in a container orchestration system.
17. The non-transitory storage medium as recited in claim 11 , wherein the object access microservice creates the copy of the object.
18. The non-transitory storage medium as recited in claim 11 , wherein masking the copy of the object comprises masking one or more segments of the copy of the object to create masked segments, and the operations further comprising:
creating respective fingerprints for each of the masked segments; and
creating copies of one or more unmasked segments of the object;
wherein deduplicating the object and the masked object comprises identifying any duplicate segments among the copies of one or more unmasked segments and omitting creation of fingerprints for any identified duplicate segments, and wherein storing the masked object and the object comprises storing any masked segments, and storing respective pointers to any duplicate segments.
19. The non-transitory storage medium as recited in claim 11 , wherein the operations further comprise receiving another object from the client, storing the another object in storage, and masking a copy of the another object after the another object has been stored.
20. The non-transitory storage medium as recited in claim 11 , wherein the operations further comprise calculating, and storing, a fingerprint of the masked object.