Automated firmware update with rollback in a data storage system
Systems and methods for automated firmware update with rollback are described herein. The systems include a plurality of storage zones, each storage zone including a plurality of storage nodes, each storage node including a plurality of storage media. The method includes monitoring storage system activity and parameters and maintaining a data storage system usage and parameter database containing system activity information. When a firmware update is available, data storage system activity is evaluated. Storage nodes needing the firmware update are identified. The firmware update is run on available storage nodes identified as needing the firmware update. The impact of the firmware update is evaluated and a rollback of the firmware update is initiated on all firmware updated storage nodes when parameter variations are significant and/or result in degraded performance.
1. A computing device to manage a data storage system, the data storage system including a plurality of storage zones, each storage zone including a plurality of storage nodes, each storage nodes including a plurality of storage media, the computing device comprising:
a processor;
a memory coupled with the processor;
a storage medium having instructions stored thereon which when executed cause the computing device to perform actions comprising:
monitoring storage system activity and parameters
maintaining a data storage system usage and parameter database including system activity information
checking whether a firmware update is available
when the firmware update is available,
evaluating data storage system activity from the data storage system usage and parameter database
identifying storage nodes needing the firmware update
checking whether storage nodes needing the firmware update are available
running the firmware update on available storage nodes identified as needing the firmware update
evaluating the impact of the firmware update on firmware updated storage nodes to determine parameter variations, including comparing at one or more time periods before the firmware update and one or more time periods after the firmware update at least three system parameters selected from the group including node operation latency; node to node latency; file writes per second; file reads per second; file deletes per second; file reservations per second; put and get throughput; put, get and delete latencies; put, get, delete and update failures; put, get and delete backlog; multipart construction rate; and object update rate
initiating a rollback of the firmware update on all firmware updated storage nodes when the parameter variations of the at least three system parameters each exceed a system threshold.
2. The computing device of claim 1 wherein the storage media are one or more selected from the group including hard disk drives, silicon storage devices and magnetic tape devices.
3. The computing device of claim 1 wherein the computing device is a storage server.
4. The computing device of claim 1 wherein the system activity information includes information about node performance comprising at least three selected from the group including: File Writes Per Second (FWPS); File Reads Per Second (FRPS); File Deletes Per Second (FDPS); File Reservations Per Second (FResPS); put and get throughput; put, get and delete latencies; put, get, delete and update failures for a time period; CPU usage; Memory usage; Network backlog; put, get and delete backlog; multipart rate; object update time; drive failure rate; start times and end times of inactive periods.
5. The computing device of claim 4 wherein checking whether storage nodes needing the firmware update are available includes:
evaluating the start times and end times of inactive periods for the storage nodes by referring to the system activity information
scheduling the firmware update for an anticipated inactive time period for the storage nodes.
6. The computing device of claim 1 wherein the storage medium has further instructions stored thereon which when executed cause the computing device to perform further actions comprising:
optimizing updated storage nodes.
7. The computing device of claim 1 :
wherein maintaining a data storage system usage and parameter database further includes maintaining priority information for data stored in the data storage system
wherein checking whether storage nodes needing the firmware update are available includes considering the priority of the data stored on the storage nodes.
8. The computing device of claim 1 wherein optimizing updated storage nodes includes adjusting parameters of updated storage nodes based on the firmware update.
9. The computing device of claim 1 wherein initiating a rollback of the firmware update includes:
sending a rollback alert to a system operator
receiving a rollback approval from the system operator.
10. A data storage system comprising:
a plurality of storage zones, each storage zone including a plurality of storage nodes, each storage nodes including a plurality of storage media
a storage server communicatively coupled with the storage zones, the storage server including a storage medium having instructions stored thereon which when executed cause the storage server to perform actions comprising:
monitoring storage system activity and parameters
maintaining a data storage system usage and parameter database including system activity information
checking whether a firmware update is available
when the firmware update is available,
evaluating data storage system activity from the data storage system usage and parameter database
identifying storage nodes needing the firmware update
checking whether storage nodes needing the firmware update are available
running the firmware update on available storage nodes identified as needing the firmware update
evaluating the impact of the firmware update on firmware updated storage nodes to determine parameter variations, including comparing at one or more time periods before the firmware update and one or more time periods after the firmware update at least three system parameters selected from the group including node operation latency; node to node latency; file writes per second; file reads per second; file deletes per second; file reservations per second; put and get throughput; put, get and delete latencies; put, get, delete and update failures; put, get and delete backlog; multipart construction rate; and object update rate
initiating a rollback of the firmware update on all firmware updated storage nodes when the parameter variations of the at least three system parameters each exceed a system threshold.
11. The data storage system of claim 10 wherein the storage media are one or more selected from the group including hard disk drives, silicon storage devices and magnetic tape devices.
12. The data storage system of claim 10 wherein the computing device is a storage server.
13. The data storage system of claim 10 wherein the system activity information includes information about node performance comprising at least three selected from the group including: File Writes Per Second (FWPS); File Reads Per Second (FRPS); File Deletes Per Second (FDPS); File Reservations Per Second (FResPS); put and get throughput; put, get and delete latencies; put, get, delete and update failures for a time period; CPU usage; Memory usage; Network backlog; put, get and delete backlog; multipart rate; object update time; drive failure rate; start times and end times of inactive periods.
14. The data storage system of claim 13 wherein checking whether storage nodes needing the firmware update are available includes:
evaluating the start times and end times of inactive periods for the storage nodes by referring to the system activity information
scheduling the firmware update for an anticipated inactive time period for the storage nodes.
15. The data storage system of claim 10 wherein the storage medium has further instructions stored thereon which when executed cause the storage server to perform further actions comprising:
optimizing updated storage nodes.
16. The data storage system of claim 10 :
wherein maintaining a data storage system usage and parameter database further includes maintaining priority information for data stored in the data storage system
wherein checking whether storage nodes needing the firmware update are available includes considering the priority of the data stored on the storage nodes.
17. The data storage system of claim 10 wherein optimizing updated storage nodes includes adjusting parameters of updated storage nodes based on the firmware update.
18. The data storage system of claim 10 wherein initiating a rollback of the firmware update includes:
sending a rollback alert to a system operator
receiving a rollback approval from the system operator.