Cloud, server &
infrastructure guides.
Choose the problem you need to investigate. Each field note connects the technical boundary with diagnostic evidence, a practical exercise and primary references.
All engineering insights ↗Subscribe to updates via Atom ↗
AWS
EC2 CPU Credits: Diagnose Throttling and Unlimited Charges ↗
Diagnose EC2 CPU credit exhaustion, Standard-mode throttling and Unlimited surplus usage with workload evidence, sustained tests and reviewed capacity choices.
EBS gp3 Performance: Separate IOPS, Throughput, and EC2 Limits ↗
Separate gp3 IOPS and throughput limits from aggregate EC2 EBS capacity. Measure request size, queueing and application completion before resizing storage.
S3 Gateway Endpoints: Verify Routes Before Tightening Bucket Policy ↗
Trace S3 gateway endpoint route associations and caller policies. Test sourceVpce restrictions, object access and recovery paths before tightening a bucket.
ALB 502 vs 504: Find the Failed Connection Boundary ↗
Distinguish ALB-generated 502 and 504 errors from target failures. Correlate resets, idle deadlines, backend timing and deployment drain evidence.
CloudTrail Integrity Validation: Verify the Digest Chain and Scope ↗
Validate CloudTrail digest chains over an explicit account, Region and interval. Separate file integrity from delivery coverage and audit selector scope.
AWS Network Firewall: Diagnose Asymmetric Inspection Paths ↗
Trace forward and return routes through AWS Network Firewall. Diagnose endpoint asymmetry, NAT placement and centralized inspection before widening rules.
Route 53 Resolver Outbound DNS: Test the Rule and Return Path ↗
Troubleshoot Route 53 Resolver outbound DNS with rule associations, target consistency, TCP and UDP reachability, and explicit hybrid lookup tests.
EC2 ENA Allowance Drops: Explain Loss Despite Low Average Traffic ↗
Use EC2 ENA counter deltas to separate bandwidth, packet-rate and tracked-connection pressure. Verify application impact beyond low average byte graphs.
Azure
Azure Accelerated Networking: Validate the Guest and Failover Path ↗
Validate Azure Accelerated Networking eligibility, guest drivers and synthetic-interface binding. Test workload performance and virtual-function continuity.
Azure Storage Failover: Use Last Sync Time to Plan Data Reconciliation ↗
Plan Azure Storage failover using Last Sync Time, operation identities and data reconciliation. Verify client recovery and post-failover protection.
Azure Load Balancer Idle Timeouts: Coordinate TCP Reset and Keepalive ↗
Diagnose stale connections behind Azure Load Balancer. Coordinate idle timeouts, TCP reset, keepalive, connection pools and safe write reconciliation.
Azure Zones vs Regions: Design for the Failure You Need to Survive ↗
Compare Azure zone and regional failure coverage across the whole application. Validate dependency placement, remaining capacity and recovery objectives.
Azure Files SMB Multichannel: Find the Actual Throughput Limit ↗
Investigate Azure Files throughput using share support, SMB channel evidence, metadata operations, client resources and end-to-end latency comparisons.
Azure Firewall FQDN Rules: Align Client DNS with the Firewall Proxy ↗
Troubleshoot Azure Firewall FQDN network rules by aligning client DNS, proxy settings, upstream resolution and effective policy inheritance.
Google Cloud
Persistent Disk Performance: Check Disk Size and VM Limits Together ↗
Plan Google Persistent Disk performance around disk type, size, VM series and vCPUs. Measure real I/O and aggregate limits before changing capacity.
E2 Shared-Core CPU: Explain Why a Short Benchmark Looks Faster ↗
Understand E2 shared-core CPU entitlement and temporary bursting. Test sustained latency, background overlap and useful throughput before sizing a VM.
Google Cloud Firewall Policies: Trace the Effective Rule Hierarchy ↗
Trace Google Cloud hierarchical firewall decisions through effective policies. Distinguish inherited denials, goto_next delegation and local rule scope.
Cloud SQL Maintenance: Reconnect Without Repeating a Committed Action ↗
Prepare Cloud SQL clients for maintenance interruptions with bounded reconnects, pool headroom and transaction-outcome reconciliation before retries.
GKE Autopilot Requests: Inspect Admission Changes Before Estimating Cost ↗
Inspect GKE Autopilot admitted resource requests and compute-class rules. Reconcile Pod resources, rollout overlap and the applicable billing model.
Compute Engine Snapshot Schedules: Test Retention After Source Deletion ↗
Verify Compute Engine snapshot schedule associations, retention and source-disk deletion behavior. Test recovery points, encryption access and restoration.
Kubernetes
Kubernetes Ephemeral Storage Evictions: Account for Logs and emptyDir ↗
Diagnose Kubernetes ephemeral-storage evictions by measuring logs, writable layers and emptyDir. Separate Pod limits from node-wide DiskPressure.
Kubernetes Termination: Coordinate EndpointSlice State and Connection Drain ↗
Coordinate Kubernetes EndpointSlice conditions, process signals and shutdown deadlines. Validate active requests and connection draining during rollouts.
Kubernetes Multi-Attach Errors: Fence the Old Writer Before Recovery ↗
Investigate Kubernetes Multi-Attach errors with CSI and backend evidence. Fence the old writer and validate exclusive ownership before replacement.
cert-manager DNS01 Pending: Trace Public DNS and the Self-Check Resolver ↗
Trace cert-manager DNS01 challenges through solver permissions, public authority, split DNS and self-check resolution. Verify the served certificate.
Kubernetes Image Garbage Collection: Recover Disk Without Breaking containerd ↗
Recover Kubernetes image storage through supported kubelet and runtime behavior. Diagnose reclaimable images, inodes and rollout pull headroom.
Kubernetes CPU Throttling: Investigate Latency Below Average CPU Limits ↗
Correlate Kubernetes CPU quota enforcement with tail latency, workload bursts and node capacity. Review requests, limits and autoscaling together.
VPA Recommendation Mode: Turn Observations into Reviewed Resource Requests ↗
Use VPA recommendation-only mode to review resource history, policy bounds and platform constraints before validating a controlled sizing change.
Admission Webhook Failures: Keep Policy Enforcement Recoverable ↗
Diagnose admission webhook denials, timeouts and certificate failures. Review matching scope, failure policy and an independently recoverable policy path.
Servers and storage
Linux CPU Steal Time: Separate Hypervisor Contention from Guest Work ↗
Investigate Linux VM latency using steal time, run queues, task states and host evidence. Separate hypervisor delay from guest CPU and I/O pressure.
NFS Hard vs Soft Mounts: Investigate Hangs Without Trading Away Data Integrity ↗
Understand NFS hard and soft mount failure semantics. Trace server and network evidence, then verify application recovery and complete file publication.
ZFS Scrub vs Resilver: Interpret Checksum Errors and Recovery Evidence ↗
Interpret ZFS scrubs, resilvers and checksum errors using pool topology and valid repair copies. Test independent backup restoration separately.
NVMe SMART Health: Read Wear, Media Errors, and Thermal Trends Together ↗
Read NVMe health trends across wear, critical warnings, media errors and temperature. Correlate stable device identity with storage and application evidence.
DM Multipath All Paths Down: Define the Application Queueing Policy ↗
Diagnose all-paths-down multipath events with LUN identity, path state and effective queueing policy. Rehearse application recovery and fencing.
Windows Nonpaged Pool Growth: Use PoolMon to Identify the Driver Path ↗
Use repeated PoolMon samples to investigate Windows kernel-memory growth. Correlate allocation tags, driver versions and a tested pilot correction.
Windows DNS Scavenging: Validate Record Refresh Before Removing Stale Names ↗
Validate Windows DNS timestamp refresh, aging and scavenging eligibility. Inventory registration ownership and rehearse cleanup without losing active names.
Hyper-V Dynamic Memory: Plan Startup and Runtime Headroom Separately ↗
Plan Hyper-V Dynamic Memory with measured guest demand, host reserve and startup requirements. Rehearse restart waves and failure placement.
Linux NUMA Locality: Correlate Remote Memory with the Actual Workload ↗
Correlate Linux NUMA memory allocation and thread placement with application latency. Test supported locality changes against representative workloads.
Linux Filesystem Suddenly Read-Only: Preserve Evidence Before Repair ↗
Diagnose an unexpected read-only Linux filesystem using mount state, kernel errors and storage health. Plan supported offline repair and restore validation.
Databases and messaging
PostgreSQL Transaction ID Age: Prevent Wraparound Before Emergency Recovery ↗
Monitor PostgreSQL transaction ID age, limiting relations and vacuum progress. Resolve maintenance obstacles before the freeze horizon becomes an emergency.
PgBouncer Prepared Statements: Validate the Driver and Transaction Pooling Contract ↗
Validate PgBouncer prepared-statement tracking with the real driver and transaction pool. Test backend reuse, reconnects and session assumptions.
MySQL Redo Capacity: Investigate Checkpoint Pressure Before Changing Durability ↗
Investigate MySQL redo generation, checkpoint progress and storage capacity. Test supported redo sizing without silently weakening durability.
MySQL Buffer Pool Sizing: Measure the Working Set and Memory Headroom ↗
Size the InnoDB buffer pool using interval read misses, query work and total memory headroom. Validate warmup, peak connections and batch overlap.
MongoDB WiredTiger Cache: Separate Useful Occupancy from Eviction Pressure ↗
Separate normal WiredTiger occupancy from harmful eviction pressure. Review query patterns, filesystem cache and the deployment memory boundary.
Redis Big Keys: Reduce Blocking Work and Verify Deferred Memory Reclamation ↗
Diagnose Redis large-key latency and compare DEL with UNLINK. Measure deferred cleanup, memory headroom and a bounded key lifecycle.
Kafka acks=all and min.insync.replicas: Define the Failure Contract ↗
Understand Kafka acks=all and minimum ISR as a failure contract. Test replica loss, producer outcomes and restored protection with identified records.
Reliability and cost
OpenTelemetry Tail Sampling: Budget Trace Affinity, Decision Time, and Memory ↗
Plan OpenTelemetry tail sampling around trace-ID affinity, span arrival and memory. Test delayed spans, scaling and retained trace completeness.
Prometheus Remote Write Backlogs: Track Lag and the Recoverable Window ↗
Diagnose Prometheus remote-write lag using queues, WAL behavior and receiver acceptance. Verify catch-up capacity and actual sample coverage.
SLO Burn-Rate Alerts: Calculate Multiwindow Thresholds from the Budget ↗
Calculate SLO burn rates from the allowed bad fraction. Test multiwindow thresholds, recovery, low traffic and missing telemetry with explicit assumptions.
Ransomware-Resilient Backups: Test Identity Isolation and Clean Recovery ↗
Design protected backup copies around identity isolation, keys and recovery points. Rehearse an isolated restore and verify clean application recovery.
Cloud Data Transfer Costs: Map Egress, NAT, and Cross-Zone Paths ↗
Map cloud traffic paths and billing meters before changing architecture. Compare egress, NAT, endpoints and cross-zone costs within resilience requirements.