Infrastructure best practices: the controls that decide whether an estate stays up

Eliminate single points of failure deliberately

Redundancy is only useful where the failure would actually hurt, and only real if it has been tested. A second power supply on a server behind one switch, one internet link and one hypervisor host is decoration.

Work outwards through the dependency chain — power, connectivity, network hardware, compute, storage, and the identity and DNS services everything else silently depends on.

• Map the chain end to end and mark each element as redundant, degraded-tolerant or single.
• Test failover by causing it in a maintenance window, not by trusting the datasheet.
• Remember DNS, DHCP and directory services — they cause outages disproportionate to their size.

Segment the network

A flat network converts one compromised workstation into a full-estate incident. Segmentation limits lateral movement and, as a side effect, makes performance problems far easier to diagnose.

The practical order is to isolate management interfaces first, then separate server, user, voice and guest traffic, then apply policy between segments.

• Management plane — hypervisors, switches, PBX and iDRAC/iLO — on its own restricted network.
• Separate VLANs for servers, users, voice and guests, with QoS for voice.
• Default-deny between segments, with documented exceptions.
• Multi-factor authentication on every administrative entry point.

Patch on a cadence, not on an incident

Unpatched infrastructure is the most common root cause in post-incident reviews, and the reason is rarely ignorance — it is the absence of an agreed maintenance window that the business will actually honour.

Fix the cadence first, then automate within it. Stage changes: test environment, low-risk production, then the rest, with a rollback point at each stage.

• A standing monthly window for routine patching, agreed with the business in advance.
• An out-of-cycle process for critical vulnerabilities with a defined severity threshold.
• Snapshots or configuration backups taken before every change.
• Firmware — switches, hypervisors, storage, handsets — included, not just operating systems.

Treat backup as a restore capability

A backup job that reports success proves that data was written somewhere. It does not prove that a service can be recovered inside the time the business can tolerate. That is only established by restoring.

Define recovery point and recovery time objectives per service, then design the backup to meet them and test against them on a schedule.

• Multiple copies, at least one offsite and at least one immutable or offline.
• Backup credentials separated from production administrative credentials.
• Scheduled restore tests with the result recorded — including how long it actually took.
• Alerting on job failure that reaches a person, not just a log file.

Monitor what users experience, and write it down

Host-level metrics tell you a server is alive. They do not tell you that the application is unusable. Add service-level checks that mirror what a user does, and alert on the conditions that precede failure — disk trending full, certificate expiry, backup age, trunk saturation.

Finally, documentation is an availability control. Topology, addressing, credentials in a managed vault, escalation contacts and runbooks are what let a second engineer resolve an incident at 03:00 without the person who built the system.

• Synthetic checks for the transactions that matter to the business.
• Predictive alerts: capacity trends, certificate and licence expiry, backup age.
• Current network diagram, IP plan and change log kept with the environment.
• Runbooks for the top failure scenarios, reviewed after every real incident.