Data masking in a microservice architecture
A method includes receiving an object from a client as part of a data ingestion process, by a masker worker, generating a list that identifies one or more byte ranges, of the object, to be masked, providing the list to an object access microservice, by the object access microservice, creating an object recipe of the object, and the object recipe includes one or more slices, and each of the slices includes one or more segments of data, masking only those segments of a slice that are in a byte range identified in the list, creating a masked slice that comprises the masked segments, and creating a masked object recipe that contains a reference to the masked slice, and the masking is performed before the object is stored in storage, creating a masked object that includes the masked slice, and deduplicating the masked object.
1. A method, comprising:
receiving an object from a client as part of a data ingestion process;
by a masker worker, generating a list that identifies one or more byte ranges, of the object, to be masked;
providing the list to an object access microservice;
by the object access microservice, creating an object recipe of the object, and the object recipe includes one or more slices, and each of the slices includes one or more segments of data;
masking only those segments of a slice that are in a byte range identified in the list, creating a masked slice that comprises the masked segments, and creating a masked object recipe that contains a reference to the masked slice, wherein the masking is performed before the object is stored in storage;
creating a masked object that includes the masked slice and a reference to an unmasked segment of data in storage;
deduplicating the masked object; and
persistently storing the masked object and unmasked object in storage such that the persistently stored masked object and unmasked object are useable to restore the received object.
2. The method as recited in claim 1 , wherein the object is provided to the masker worker by a load balancer.
3. The method as recited in claim 1 , wherein the masked slice further comprises one or more unmasked segments.
4. The method as recited in claim 1 , wherein deduplication of the masked object is performed at the segment level and, for the unmasked segment, the object access microservice forms a segment of the unaltered data, and the segment of unaltered data is referenced by, and contained in, both the object and the masked object.
5. The method as recited in claim 1 , wherein information identifying the byte ranges is provided to the masker worker by another entity.
6. The method as recited in claim 1 , wherein at least part of the method is performed in a container orchestration system.
7. The method as recited in claim 1 , further comprising storing the deduplicated masked object in storage.
8. The method as recited in claim 1 , further comprising generating respective fingerprints only for masked segments, but not for unmasked segments.
9. The method as recited in claim 1 , further comprising checking, with an object access microservice that receives the object at least indirectly from the client, whether or not masking is enabled.
10. A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
receiving an object from a client as part of a data ingestion process;
by a masker worker, generating a list that identifies one or more byte ranges, of the object, to be masked;
providing the list to an object access microservice;
by the object access microservice, creating an object recipe of the object, and the object recipe includes one or more slices, and each of the slices includes one or more segments of data;
masking only those segments of a slice that are in a byte range identified in the list, creating a masked slice that comprises the masked segments, and creating a masked object recipe that contains a reference to the masked slice, wherein the masking is performed before the object is stored in storage;
creating a masked object that includes the masked slice and a reference to an unmasked segment of data in storage;
deduplicating the masked object; and
persistently storing the masked object and unmasked object in storage such that the persistently stored masked object and unmasked object are useable to restore the received object.
11. The non-transitory storage medium as recited in claim 10 , wherein the object is provided to the masker worker by a load balancer.
12. The non-transitory storage medium as recited in claim 10 , wherein the masked slice further comprises one or more unmasked segments.
13. The non-transitory storage medium as recited in claim 10 , wherein deduplication of the masked object is performed at the segment level and, for the unmasked segment, the object access microservice forms a segment of the unaltered data, and the segment of unaltered data is referenced by, and contained in, both the object and the masked object.
14. The non-transitory storage medium as recited in claim 10 , wherein information identifying the byte ranges is provided to the masker worker by another entity.
15. The non-transitory storage medium as recited in claim 10 , wherein at least part of the non-transitory storage medium is performed in a container orchestration system.
16. The non-transitory storage medium as recited in claim 10 , wherein the operations further comprise storing the deduplicated masked object in storage.
17. The non-transitory storage medium as recited in claim 10 , wherein the operations further comprise generating respective fingerprints only for masked segments, but not for unmasked segments.
18. The non-transitory storage medium as recited in claim 10 , wherein the operations further comprise checking, with an object access microservice that receives the object at least indirectly from the client, whether or not masking is enabled.