Global namespace in a cloud-based data storage system
Techniques and mechanisms described herein provide for verifying data across cloud-based and on-premises data storage systems. According to various embodiments, data can be received by a gateway from a client machine and stored in a file data repository accessible via the internet. The stored data can have a common master namespace. Indications of updates to the master namespace can be received. Updated namespace data and metadata can be transmitted to the gateway.
1. A system comprising:
a plurality of gateways each configured to receive data from a client machine and store the data in a file data repository accessible via the internet, each of the gateways including a respective local namespace and metadata repository characterizing the data;
a central namespace and file data repository storing data received from the plurality of gateways, the stored data sharing a common master namespace; and
a messaging system configured to receive an indication of updated central namespace data and metadata from the central namespace and file data repository and transmit the central namespace data and metadata to the plurality of gateways, wherein the messaging system is further configured to identify a first one of the plurality of gateways that subscribe to updates related to the namespace data and metadata repository and identify a second one of the plurality of gateways that subscribe to updates related to the namespace data and metadata, the updated central namespace data and metadata being transmitted to the identified gateways, wherein the first gateway is configured to update a first local namespace and metadata repository associated with the first gateway based on the updated central namespace data and metadata and wherein the second gateway is configured to update a second local namespace and metadata repository associated with the second gateway based on updated central namespace data and metadata.
2. The system of claim 1 , wherein a first one of the gateways is configured to receive a request to add a file to the central namespace and file data repository, and responsive to receiving the request to add the file to the central namespace and file data repository, updating the common master namespace according based on the file added to the central namespace and file data repository.
3. The system of claim 2 , wherein the file is parsed into a plurality of file segments and a first one of the file segments is provided to the file data repository storing data using a segment map, the segment map including a segment identifier and a corresponding hash value.
4. The system of claim 1 , the system further comprising a metadata change system configured to communicate with to the messaging system, the metadata change system configured to provide indications of updates associated with the common master namespace.
5. The system of claim 1 , wherein a first one of the gateways is configured to:
determine that a first local namespace is different from the common master namespace; and
responsive to determining that the first local namespace is different from the common master namespace, requesting updated namespace data.
6. A method comprising:
receiving, by a plurality of gateways, data from a client machine and storing the data in a file data repository accessible via the internet, each of the gateways including a respective local namespace and metadata repository characterizing the data;
receiving, by a central namespace and file data repository, data from the plurality of gateways, the stored data sharing a common master namespace;
receiving, by a messaging system an indication of updated central namespace data and metadata from the central namespace and file data repository;
transmitting the updated central namespace data and metadata to the plurality of gateways;
identifying a first one of the plurality of gateways that subscribe to updates related to the namespace data and metadata repository and identifying a second one of the plurality of gateways that subscribe to updates related to the namespace data and metadata, the namespace data and metadata being transmitted to the identified gateways; and
updating a first local namespace and metadata repository associated with the first gateway based on the updated central namespace data and metadata and updating a second local namespace and metadata repository associated with the second gateway based on the updated central namespace data and metadata.
7. The method of claim 6 , the method further comprising receiving a request to add a file to the central namespace and file data repository and responsive to receiving the request to add the file to the central namespace and file data repository, updating the common master namespace according based on the file added to the central namespace and file data repository.
8. The method of claim 7 , wherein the file is parsed into a plurality of file segments and a first one of the file segments is provided to the file data repository storing data using a segment map, the segment map including a segment identifier and a corresponding hash value.
9. The method of claim 6 , the method further comprising a metadata change system configured to communicate with to the messaging system, the metadata change system configured to provide indications of updates associated with the common master namespace.
10. The method of claim 6 , wherein a first one of the gateways determines that a first local namespace is different from the common master namespace and responsive to determining that the first local namespace is different from the common master namespace, requesting updated namespace data.
11. A computer program product comprising one or more non-transitory computer readable media having instructions stored thereon for performing a method, the method comprising:
receiving, by a plurality of gateways, data from a client machine and storing the data in a file data repository accessible via the internet, each of the gateways including a respective local namespace and metadata repository characterizing the data;
receiving, by a central namespace and file data repository, data from the plurality of gateways, the stored data sharing a common master namespace;
receiving, by a messaging system an indication of updated central namespace data and metadata from the central namespace and file data repository;
transmitting the updated central namespace data and metadata to the plurality of gateways;
identifying a first one of the plurality of gateways that subscribe to updates related to the namespace data and metadata repository and identifying a second one of the plurality of gateways that subscribe to updates related to the namespace data and metadata, the namespace data and metadata being transmitted to the identified gateways; and
updating a first local namespace and metadata repository associated with the first gateway based on the updated namespace data and metadata and updating a second local namespace and metadata repository associated with the second gateway based on updated namespace data and metadata.
12. The computer program product of claim 11 , the method further comprising receiving a request to add a file to the central namespace and file data repository and responsive to receiving the request to add the file to the central namespace and file data repository, updating the common master namespace according based on the file added to the central namespace and file data repository.
13. The computer program product of claim 12 , wherein the file is parsed into a plurality of file segments and a first one of the file segments is provided to the file data repository storing data using a segment map, the segment map including a segment identifier and a corresponding hash value.
14. The computer program product of claim 11 , the method further comprising a metadata change system configured to communicate with to the messaging system, the metadata change system configured to provide indications of updates associated with the common master namespace.