Scalable cloud hosted metadata service
Systems and methods for protecting and deduplicating streams of data in a cloud based platform. A platform is configured with multiple services and can communicate with multiple clients. The platform receives all requests at an endpoint and distributes the requests to workers using multiple queues. The platform may be stateless and is scalable. The meta-data is handled separately from the data in one example.
1. A system for protecting data in a cloud environment, the system comprising: one or more hardware processors; and a plurality of services comprising computer-executable instructions that, when executed by one or more hardware processors, protect the data, the services including: a gateway service configured to receive a request from a client; a plurality of queues, the plurality of queues including a global request queue and a session request queue; a stream service configured to receive the request from the gateway service, wherein the stream service is configured to evaluate headers included in the request and place the request in one of the plurality of queues based on the header; and a plurality of workers, wherein any of the plurality of workers can service the request when the request is placed in the global request queue and wherein only a particular worker associated with a session associated with the session request queue can service the request when the request is in the session request queue.
2. The system of claim 1 , wherein the plurality of queues comprises a plurality of session request queues and each of the plurality of workers is associated with a different session request queue.
3. The system of claim 1 , wherein each of the plurality of workers is associated with one or more session request queues.
4. The system of claim 1 , the plurality of queues includes a notification queue configured to notify the plurality of workers about events and a response queue configured to deliver responses to the client.
5. The system of claim 1 , wherein the stream services passes a payload included in the request to the queues.
6. The system of claim 1 , wherein the worker performs the request and places a response in a response queue that is configured to be accessed by the client.
7. The system of claim 1 , wherein data associated with the request is handled separately from metadata associated with the request.
8. The system of claim 1 , wherein the worker is associated with a distributed store configured to store metadata associated with the data.
9. The system of claim 1 , wherein the request includes one of a create operation, an open operation, a flush operation, a close operation, a cancel operation, a delete operation, a write chunk data operation, a write chunk map operation, a read chunk map operation, a set attributes operation, a get attributes operation, and a queue operation.
10. A method for protecting data in a cloud environment, the system comprising:
receiving a request from a client at a gateway service;
receiving the request at a stream service from the gateway service;
evaluating headers included in the request and placing the request in one of a plurality of queues based on the header, the plurality of queues including a global request queue and a session request queue; and
servicing the request, wherein any of a plurality of workers can service the request when the request is placed in the global request queue and wherein only a particular worker associated with a session associated with the session request queue can service the request when the request is in the session request queue.
11. The method of claim 10 , wherein the plurality of queues comprises a plurality of session request queues and each of the plurality of workers is associated with a different session request queue.
12. The method of claim 11 , further comprising associating each of the plurality of workers with one or more session request queues.
13. The method of claim 10 , further comprising notifying the plurality of workers of events by placing notifications in a notification queue and responding to the request by placing a response in a response queue.
14. The method of claim 10 , further comprising passing, by the stream service, a payload included in the request to the plurality of queues.
15. The method of claim 10 , further comprising performing the request and placing a response in a response queue that is configured to be accessed by the client.
16. The method of claim 10 , further comprising handling data associated with the request separately from metadata associated with the request.
17. The method of claim 10 , further comprising associating the plurality of workers with a distributed store configured to store metadata associated with the data.
18. The method of claim 10 , wherein the request includes one of a create operation, an open operation, a flush operation, a close operation, a cancel operation, a delete operation, a write chunk data operation, a write chunk map operation, a read chunk map operation, a set attributes operation, a get attributes operation, and a queue operation.