Methods, apparatuses, computer programs and computer program products for data storage
A data storage system in which different copies of a data object (e.g., a file) can be compressed using different compression processes (e.g. different compression algorithms/processes and/or compression parameters), with some favoring faster decompression, while others favoring storage space savings. When a data object needs to be accessed, the copy of the data object that can be decompressed using minimal resource (computing and/or time) can be located and retrieved.
1. A method for redundant storage of a data object, the method being performed by a data storage system comprising a storage controller and a plurality of storage nodes and the method comprising:
obtaining the data object; and
in response to obtaining the data object performing a method comprising:
determining a first storage node for storing a first copy of the data object;
determining a second storage node for storing a second copy of the data object;
providing the first copy of the data object to the first storage node; and
providing the second copy of the data object to the second storage node, wherein the method further comprises:
the first storage node determining a first compression process for use in compressing the first copy of the data object;
the second storage node determining a second compression process for use in compressing the second copy of the data object, wherein the second compression process is different than the first compression process;
the first storage node using the first compression process to compress the first copy of the data object, thereby producing a first compressed version of the data object;
the second storage node using the second compression process to compress the second copy of the data object, thereby producing a second compressed version of the data object, wherein the second compression process is computationally simpler than the first compression process;
storing the first compressed version of the data object in a first non-transitory computer readable medium; and
storing the second compressed version of the data object in a second non-transitory computer readable medium, wherein
the method further comprises:
providing to the first storage node a storage container identifier identifying a storage container, wherein (1) the first storage node uses the storage container identifier to obtain from a first database a first compression process identifier associated with the storage container identifier and (2) the first storage node uses the first compression process identifier to determine the first compression process;
providing to the second storage node the storage container identifier, wherein (1) the second storage node uses the storage container identifier to obtain from a second database a second compression process identifier associated with the storage container identifier and (2) the second storage node uses the second compression process identifier to determine the second compression process.
2. The method of claim 1 , wherein
the storage container identifier and the first copy of the data object are provided to the first storage node at substantially the same time, and
the storage container identifier and the second copy of the data object are provided to the second storage node at substantially the same time.
3. The method of claim 1 , further comprising, prior to obtaining the data object, configuring the first storage node and the second storage node, wherein
configuring the first storage node comprises causing the first storage node to associate the storage container identifier with the first compression process identifier; and
configuring the second storage node comprises causing the second storage node to associate the storage container identifier with the second compression process identifier.
4. The method of claim 1 , wherein
the method further comprises, prior to providing the first copy of the data object to the first storage node, selecting the first compression process from a set of two or more available compression processes,
providing the first copy of the data object to the first storage node comprises transmitting to the first storage node the first copy of the data object and a first compression process identifier identifying the first compression process, and
the first storage node determines the first compression process based on the first compression process identifier.
5. The method of claim 4 , wherein
selecting the first compression process comprises selecting the first compression process based on one or more of: an attribute of the data object, current system resource availability, and predicted system resource availability.
6. The method of claim 1 , further comprising:
prior to the first storage node using the first compression process to compress the first copy of the data object, the first storage node determines whether it has sufficient resources to compress the first copy of the data object using the first compression process.
7. The method of claim 1 , further comprising:
scanning at least one of the first storage node and the second storage node to identify files requiring compression; and
for each identified file requiring compression, compressing the identified file.
8. The method of claim 1 , further comprising:
scanning at least one of the first storage node and the second storage node to identify files that have not been accessed since a predetermined prior point in time; and
for each identified file that has not been accessed since the predetermined prior point in time, compressing the identified file.
9. The method of claim 1 , further comprising:
determining that a storage level of the first storage node has exceeded a storage level threshold, and
as a result of determining that the storage level of the first storage node has exceeded the storage level threshold, compressing a file stored in the first storage node to a higher compression level.
10. A computer program product comprising a non-transitory computer readable medium storing a computer program comprising instructions which, when executed on at least one processor, cause the at least one processor to carry out the method of claim 1 .
11. A system, system comprising:
a data storage system; and
a data processing system coupled to the data storage system, the data processing system comprising one or more processors, wherein the system is configured to perform the method of claim 1 .
12. A method for redundant storage of a data object, the method being performed by a data storage system comprising a storage controller and a plurality of storage nodes and the method comprising:
obtaining the data object; and
in response to obtaining the data object performing a method comprising:
determining a first storage node for storing a first copy of the data object;
determining a second storage node for storing a second copy of the data object;
providing the first copy of the data object to the first storage node; and
providing the second copy of the data object to the second storage node, wherein the method further comprises:
the first storage node determining a first compression process for use in compressing the first copy of the data object;
the second storage node determining a second compression process for use in compressing the second copy of the data object, wherein the second compression process is different than the first compression process;
the first storage node using the first compression process to compress the first copy of the data object, thereby producing a first compressed version of the data object;
the second storage node using the second compression process to compress the second copy of the data object, thereby producing a second compressed version of the data object, wherein the second compression process is computationally simpler than the first compression process;
storing the first compressed version of the data object in a first non-transitory computer readable medium;
storing the second compressed version of the data object in a second non-transitory computer readable medium;
prior to the first storage node using the first compression process to compress the first copy of the data object, the first storage node determines whether it has sufficient resources to compress the first copy of the data object using the first compression process; and
as a result of determining that the first storage node does not have sufficient resources to compress the first copy of the data object using the first compression process, the first storage node stores the first copy of the data object in the first non-transitory computer readable medium and, after the first storage node stores the first copy of the data object in the first non-transitory computer readable medium, the first storage node uses the first compression process to compress the first copy of the data object, thereby producing the first compressed version of the data object.
13. A method for redundant storage of a data object, the method being performed by a data storage system comprising a storage controller and a plurality of storage nodes and the method comprising:
obtaining the data object; and
in response to obtaining the data object performing a method comprising:
determining a first storage node for storing a first copy of the data object;
determining a second storage node for storing a second copy of the data object;
providing the first copy of the data object to the first storage node; and
providing the second copy of the data object to the second storage node, wherein the method further comprises:
the first storage node determining a first compression process for use in compressing the first copy of the data object;
the second storage node determining a second compression process for use in compressing the second copy of the data object, wherein the second compression process is different than the first compression process;
the first storage node using the first compression process to compress the first copy of the data object, thereby producing a first compressed version of the data object;
the second storage node using the second compression process to compress the second copy of the data object, thereby producing a second compressed version of the data object, wherein the second compression process is computationally simpler than the first compression process;
storing the first compressed version of the data object in a first non-transitory computer readable medium;
storing the second compressed version of the data object in a second non-transitory computer readable medium;
receiving a request for the data object;
in response to receiving the request for the data object, selecting a storage node from a set of storage nodes, wherein each storage node included in the set of storage nodes stores a copy of the data object; and
retrieving the data object from the selected storage node, wherein
the storage node is selected based on one or more of: the compression level at which the data object is stored on the selected storage node and a network access time for accessing the selected storage node.