Data object store and server for a cloud storage environment, including data deduplication and data management across multiple cloud storage sites
View Patent ↗Data storage operations, including content-indexing, containerized deduplication, and policy-driven storage, are performed within a cloud environment. The systems support a variety of clients and cloud storage sites that may connect to the system in a cloud environment that requires data transfer over wide area networks, such as the Internet, which may have appreciable latency and/or packet loss, using various network protocols, including HTTP and FTP. Methods are disclosed for content indexing data stored within a cloud environment to facilitate later searching, including collaborative searching. Methods are also disclosed for performing containerized deduplication to reduce the strain on a system namespace, effectuate cost savings, etc. Methods are disclosed for identifying suitable storage locations, including suitable cloud storage sites, for data files subject to a storage policy. Further, systems and methods for providing a cloud gateway and a scalable data object store within a cloud environment are disclosed, along with other features.
1. A method for scheduling storage operations on a cloud storage site, comprising:
receiving multiple new requests for cloud storage from one or more clients,
wherein the multiple new requests each include a request for data storage, and
wherein the multiple new requests each include information associated with the data storage requested;
determining a current capacity of the cloud storage site, wherein the current capacity of the cloud storage site is determined based at least in part on:
(i) a capacity policy,
wherein the capacity policy specifies preferences and criteria associated with allocating system resources for the cloud storage site, and
(ii) at least one of:
a quotation policy,
wherein the quotation policy includes a set of preferences and criteria associated with generating a quote in response to received client requests, and
a scheduled job,
wherein the scheduled job is associated with a quote for cloud storage accepted by a client,
a quoted job,
wherein the quoted job is associated with a quote for cloud storage provided to a client, and
queued requests,
wherein queued requests include requests by clients for cloud storage for which the respective clients have not been provided a quote;
identifying one or more approved requests,
wherein the one or more approved requests are identified from pending requests based at least in part on preferences and criteria specified in the accessed quotation policy and the current capacity,
wherein pending requests comprise the received multiple new requests and queued requests;
generating a responsive quote for each approved request,
wherein the responsive quotes are generated based at least in part on preferences and criteria specified in the accessed quotation policy, and
wherein each responsive quote includes one or more pricing values;
sending a generated responsive quote to a client associated with an approved request that the responsive quote was generated for; and
receiving from the client that was sent the generated responsive quote an indication of acceptance of the generated responsive quote.
2. The method of claim 1 , wherein, the capacity policy specifies system resources available during specified periods, scheduled maintenance windows and current storage capacity available on servers.
3. The method of claim 1 , further comprising determining system resources required for storage operations already scheduled or quoted.
4. The method of claim 1 , wherein the quotation policy specifies at least three of:
a revenue function;
a pricing function;
a pricing rate table;
information associated with marketing promotions;
a list of preferred clients;
a list of disfavored clients;
classes of storage;
retention policies;
upload time periods;
data characteristics;
compression or encryption requirements; and
estimated or historic cost of storage, including a cost of power.
5. The method of claim 1 , wherein the quotation policy specifies a revenue function that describes a method for numerically evaluating a projected revenue generated by the received requests.
6. The method of claim 1 , wherein the quotation policy specifies a pricing function that describes a method for generating various pricing values for a responsive quote.
7. The method of claim 1 , wherein identifying one or more approved requests further comprises excluding out of the received requests those that either do not satisfy minimum requirements specified by the quotation policy or cannot be accommodated due to a lack of system resources.
8. The method of claim 1 , further comprising at least one of the following:
generating a responsive quote, to be sent to a client, having at least one term that is different from a term in a request received from the client; and
queuing a received request for later evaluation.
9. The method of claim 1 , wherein identifying one or more approved requests further comprises identifying a set of requests that result in a maximum combined value of a revenue function.
10. The method of claim 1 , wherein identifying one or more approved requests further comprises identifying a set of requests that result in a combined value of a revenue function that is sufficient to satisfy the quotation policy.
11. A non-transitory computer-readable storage medium whose contents cause a computer system to perform a method for scheduling storage operations on a cloud storage site, the method comprising:
receiving multiple new requests for cloud storage from one or more clients,
wherein the multiple new requests each include a request for data storage, and
wherein the multiple new requests each include information associated with the data storage requested;
determining a current capacity of the cloud storage site, wherein the current capacity of the cloud storage site is determined based at least in part on:
(i) a capacity policy,
wherein the capacity policy specifies preferences and criteria associated with allocating system resources for the cloud storage site, and
(ii) at least one of:
a quotation policy,
wherein the quotation policy includes a set of preferences and criteria associated with generating a quote in response to received client requests, and
a scheduled job,
wherein the scheduled job is associated with a quote for cloud storage accepted by a client,
a quoted job,
wherein the quoted job is associated with a quote for cloud storage provided to a client, and
queued requests,
wherein queued requests include requests by clients for cloud storage for which the respective clients have not been provided a quote;
identifying one or more approved requests,
wherein the one or more approved requests are identified from pending requests based at least in part on preferences and criteria specified in the accessed quotation policy and the current capacity,
wherein pending requests comprise the received multiple new requests and queued requests;
generating a responsive quote for each approved request,
wherein the responsive quotes are generated based at least in part on preferences and criteria specified in the accessed quotation policy, and,
wherein each responsive quote includes one or more pricing values;
sending a generated responsive quote to a client associated with an approved request that the responsive quote was generated for; and
receiving from the client that was sent the generated responsive quote an indication of acceptance of the generated responsive quote.
12. The non-transitory computer-readable storage medium of claim 11 , wherein the capacity policy specifies system resources available during specified periods, scheduled maintenance windows and current storage capacity available on servers.
13. The non-transitory computer-readable storage medium of claim 11 , further comprising determining system resources required for storage operations already scheduled or quoted.
14. The non-transitory computer-readable storage medium of claim 11 , wherein the quotation policy specifies at least three of:
a revenue function;
a pricing function;
a pricing rate table;
information associated with marketing promotions;
a list of preferred clients;
a list of disfavored clients;
classes of storage;
retention policies;
upload time periods;
data characteristics;
compression or encryption requirements; and
estimated or historic cost of storage, including a cost of power.
15. The non-transitory computer-readable storage medium of claim 11 , wherein the quotation policy specifies a revenue function that describes a method for numerically evaluating a projected revenue generated by the received requests.
16. The non-transitory computer-readable storage medium of claim 11 , wherein the quotation policy specifies a pricing function that describes a method for generating various pricing values for a responsive quote.
17. The non-transitory computer-readable storage medium of claim 11 , further comprising at least one of the following:
generating a responsive quote, to be sent to a client, having at least one term that is different from a term in a request received from the client; and
queuing a received request for later evaluation.
18. The non-transitory computer-readable storage medium of claim 11 , wherein identifying one or more approved requests further comprises identifying a set of requests that result in a maximum combined value of a revenue function.
19. The non-transitory computer-readable storage medium of claim 11 , wherein identifying one or more approved requests further comprises identifying a set of requests that result in a combined value of a revenue function that is sufficient to satisfy the quotation policy.
20. A system for scheduling storage operations on a cloud storage site, the system comprising:
means for receiving multiple new requests for cloud storage from one or more clients,
wherein the multiple new requests each include a request for data storage, and
wherein the multiple new requests each include information associated with the data storage requested;
means for determining a current capacity of the cloud storage site, wherein the current capacity of the cloud storage site is determined based at least in part on:
(i) a capacity policy,
wherein the capacity policy specifies preferences and criteria associated with allocating system resources for the cloud storage site, and
(ii) at least one of:
a quotation policy,
wherein the quotation policy includes a set of preferences and criteria associated with generating a quote in response to received client requests, and
a scheduled job,
wherein the scheduled job is associated with a quote for cloud storage accepted by a client,
a quoted job,
wherein the quoted job is associated with a quote for cloud storage provided to a client, and
queued requests,
wherein queued requests include requests by clients for cloud storage for which the respective clients have not been provided a quote;
means for identifying one or more approved requests,
wherein the one or more approved requests are identified from pending requests based at least in part on preferences and criteria specified in the accessed quotation policy and the current capacity,
wherein pending requests comprise the received multiple new requests and queued requests;
means for generating a responsive quote for each approved request,
wherein the responsive quotes are generated based at least in part on preferences and criteria specified in the accessed quotation policy, and
wherein each responsive quote includes one or more pricing values;
means for sending a generated responsive quote to a client associated with an approved request that the responsive quote was generated for; and
means for receiving from the client that was sent the generated responsive quote an indication of acceptance of the generated responsive quote.