System and method of implementing an object storage infrastructure for cloud-based services
A method for storing objects in an object storage system includes the steps of establishing a network connection with a client over an inter-network, receiving an upload request indicating an object to be uploaded by the client, selecting at least two storage nodes on which the object will be stored, receiving the object from the client via the network connection, and streaming the object to each of the selected storage nodes such that the object is stored on each of the selected storage nodes. The method can also include writing an object record associating the object and the selected storage nodes to a shard of an object database and generating a Universally Unique Identifier (UUID). The UUID indicates the shard and the object ID of the object record, such that the object record can be quickly retrieved. Object storage infrastructures are also disclosed.
1. In an object storage system, a method for storing digital objects of a client, said method comprising:
establishing a network connection with a client system over an inter-network;
receiving an upload request via said network connection, said upload request requesting that a digital object be uploaded from said client system;
selecting at least one storage node from a set of storage nodes, said digital object to be stored on said selected storage node;
receiving said digital object from said client system via said network connection;
determining whether said digital object was previously stored on said selected storage node; and
preventing retention of multiple copies of said digital object on said selected storage node; and wherein
said step of receiving said digital object includes storing said digital object on said selected storage node;
said step of determining whether said digital object was previously stored on said selected storage node includes analyzing said digital object as it is stored on said selected storage node; and
said step of analyzing said digital object includes generating a hash value associated with said digital object and storing said hash value in an object record indicative of said digital object.
2. The method of claim 1 , wherein:
said set of storage nodes includes a plurality of storage nodes;
said step of selecting at least one storage node from a set of storage nodes includes selecting multiple storage nodes from said set of storage nodes;
said step of determining whether said digital object was previously stored on said selected storage node includes determining whether said digital object was previously stored on any of said selected storage nodes; and
said step of preventing retention of multiple copies of said digital object on said selected storage node includes preventing retention of multiple copies of said digital object on any of said selected storage nodes.
3. The method of claim 2 , further comprising:
generating a new object record associated with said digital object; and
modifying said new object record according to an existing object record associated with an existing digital object that is identical to said digital object and was previously stored on at least one of said selected storage nodes.
4. The method of claim 3 , wherein said step of generating a new object record associated with said digital object includes associating said new object record with said existing object record.
5. The method of claim 2 , wherein said step of selecting multiple storage nodes from said set of storage nodes includes selecting at least one storage node hosted on another Internet cloud system other than said object storage system.
6. The method of claim 1 , wherein:
said step of receiving said digital object includes storing said digital object on said selected storage node; and
said step of preventing retention of multiple copies of said digital object on said selected storage node includes, if said digital object was previously stored on said selected storage node, removing said digital object or said previously stored copy of said digital object.
7. The method of claim 1 , wherein:
said step of generating a hash value associated with said digital object includes generating a checksum associated with said digital object; and
said step of storing said hash value in an object record includes storing said checksum in an object record indicative of said digital object.
8. The method of claim 7 , wherein said step of determining if said digital object is already stored on any of said selected storage nodes includes searching object records associated with objects stored on said selected storage node for a second checksum associated with a second digital object, said second checksum being identical to said checksum.
9. The method of claim 8 , wherein said step of generating a checksum includes generating the checksum utilizing a hash function.
10. The method of claim 1 , further comprising:
generating an encrypted object from said digital object after said step of analyzing said digital object; and wherein
said step of storing said digital object on said selected storage node includes storing said encrypted object.
11. The method of claim 1 , wherein said step of analyzing said digital object includes generating information from said digital object.
12. An object storage system, comprising:
a set of storage nodes for storing digital objects associated with clients; and
at least one client application server being operative to
establish a network connection with a client system over an inter-network,
receive an upload request via said network connection, said upload request requesting that a digital object be uploaded from said client system,
select at least one storage node from said set of storage nodes, said digital object to be stored on said selected storage node,
receive said digital object from said client system via said network connection,
determine whether said digital object was previously stored on said selected storage node,
prevent retention of multiple copies of said digital object on said selected storage node,
store said digital object on said selected storage node,
determine whether said digital object was previously stored on said selected storage node by analyzing said digital object as it is stored on said selected storage node,
generate a hash value associated with said digital object, and
store said hash value in an object record indicative of said digital object.
13. The system of claim 12 , wherein said set of storage nodes includes a plurality of storage nodes and said client application server is further operative to:
select multiple storage nodes from said set of storage nodes, said digital object to be stored on said selected storage nodes;
determine whether said digital object was previously stored on any of said selected storage nodes; and
prevent retention of multiple copies of said digital object on any of said selected storage nodes.
14. The system of claim 13 , wherein said client application server is further operative to:
generate a new object record associated with said digital object; and
modify said new object record according to an existing object record associated with an existing digital object that is identical to said digital object and was previously stored on at least one of said selected storage nodes.
15. The system of claim 14 , wherein said client application server is operative to modify said new object record by associating said new object record with said existing object record.
16. The system of claim 13 , wherein said client application server is configured to select at least one storage node hosted on another Internet cloud system other than said object storage system.
17. The system of claim 12 , wherein said client application server is further operative to:
store said digital object on said selected storage node; and
if said digital object was previously stored on said selected storage node, remove said digital object or said previously stored copy of said digital object.
18. The system of claim 12 , wherein
said hash value is a checksum.
19. The system of claim 18 , wherein said client application server searches object records associated with objects stored on said selected storage node for a second checksum associated with a second digital object, said second checksum being identical to said checksum.
20. The system of claim 19 , wherein said client application server generates the checksum utilizing a SHA512 hash function.
21. The system of claim 12 , wherein said client application server is further operative to:
generate an encrypted object from said digital object after analyzing said digital object; and wherein
storing said digital object on said selected storage node includes storing said encrypted object.
22. The system of claim 12 , wherein analyzing said digital object includes generating information from said digital object.
23. A non-transitory, electronically-readable storage medium having code embodied therein for causing an electronic device to:
establish a network connection with a client system over an inter-network;
receive an upload request via said network connection, said upload request requesting that a digital object be uploaded from said client system;
select at least one storage node from a set of storage nodes, said digital object to be stored on said selected storage node;
receive said digital object from said client system via said network connection;
determine whether said digital object was previously stored on said selected storage node;
prevent retention of multiple copies of said digital object on said selected storage node;
store said digital object on said selected storage node;
determine whether said digital object was previously stored on said selected storage node by analyzing said digital object as it is stored on said selected storage node;
generate a hash value associated with said digital object; and
store said hash value in an object record indicative of said digital object.