US Patent 11182252 High availability state machine and recovery

Embodiments of the present invention provide systems and methods for recovering a high availability storage system. The storage system includes a first layer and a second layer, each layer including a controller board, a router board, and storage elements. When a component of a layer fails, the storage system continues to function in the presence of a single failure of any component, up to two storage element failures in either layer, or a single power supply failure. While a component is down, the storage system will run in a degraded mode. The passive zone is not serving input/output requests, but is continuously updating its state in dynamic random access memory to enable failover within a short period of time using the layer that is fully operational. When the issue with the failed zone is corrected, a failback procedure brings the system back to a normal operating state.