Context
A degraded software RAID array can continue serving data while having less redundancy than intended. That is an operational warning, not permission to remove the first disk with a suspicious device name. Linux device names can change across boots, and several layers may sit between an array member and a physical drive bay.
Prepare the array design, RAID level, current backup evidence and hardware ownership map. Confirm that the operator understands the remaining failure tolerance. Replacement and rebuild are storage changes with data-loss risk; the following initial steps focus on read-only identification before a specific recovery plan is executed.
Establish array and device identity
- Read
cat /proc/mdstatand record the array names, member indicators and any active recovery or resynchronization. A progress line means work is underway; it does not establish completion. Compare the observed member count with the intended design. If the array is inactive or its identity is uncertain, stop speculative assembly attempts and involve the storage owner. - Run
sudo mdadm --detail /dev/mdXfor the verified array. Replace the placeholder only after identifying it. Record array UUID, RAID level, active devices, failed devices and member roles. Distinguish a missing role from a present member marked faulty. Keep a timestamped copy with the incident record before changing membership. - Map members using
lsblk -o NAME,SIZE,TYPE,MODEL,SERIAL,WWNand the hardware or hypervisor inventory. Some virtual environments do not expose useful serials, requiring provider-level mapping. Review stable identifiers and partition membership instead of assuming/dev/sdbalways represents the same drive bay. - Inspect a suspected member with a supported read-only
mdadm --examineoperation where appropriate. Compare array identity and metadata with the recorded design. Do not create a new array, zero metadata or add an arbitrary device as a trial; those commands change the recovery problem and can overwrite useful evidence.
An illustrative dangerous assumption
In an illustrative storage server, a replacement boot changes device enumeration. A previous ticket called the failed drive /dev/sdc, but that name now belongs to a healthy member. The team's serial-to-bay mapping prevents the wrong disk from being removed. The useful runbook identifies the array role and hardware identity together, rather than preserving only a transient device name.
| Observation | Evidence required | Decision boundary |
|---|---|---|
| Member missing | Array role and hardware mapping | Locate the actual device path |
| Member marked faulty | Error history and stable identity | Plan supported replacement |
| Recovery progressing | Progress, storage health and workload | Monitor before further removal |
| Metadata disagrees | Array UUID and event history | Escalate recovery design |
Recover with a reviewed plan
Choose the distribution-supported replacement procedure for the exact RAID level and state. Confirm the replacement capacity and partition layout, current recoverability and remaining tolerance. Avoid running multiple independent replacement experiments. A mirrored or parity array is not a backup, and a rebuild cannot repair every form of application or filesystem corruption.
Verify the array returns to the intended member state and that recovery finishes without new device errors. Check filesystem access, application transactions and backup behavior separately. Preserve the failed-device evidence needed for hardware follow-up.
Maintain redundancy monitoring
Monitor degraded state, recovery progress and member health, with a named response owner. Periodically reconcile stable drive identity with physical or virtual inventory. Record replacement steps, acceptance checks and escalation conditions in the SOP. Clouvenix Linux administration can help prepare storage procedures before a degraded-array incident.