Failures and antipatterns · Intermediate

Disk exhaustion and write failures

Identify mount, inode and quota constraints, preserve database recovery history, and verify both restored writes and recurrence causes.

Ask Logeia about this articleTry the three questionsReview your answers

Updated

On this page
  1. Free bytes are only part of the picture
  2. Separate the destination from the retention cause
  3. Do not destroy recovery while freeing space
  4. Verify successful writes after capacity returns
  5. Ask Logeia about this articleCheck your understanding

Free bytes are only part of the picture

A disk-full failure can involve free bytes, inode exhaustion, a user or container quota, the actual target mount, or reserved capacity. Space on a different volume does not help a write directed at the exhausted one. A deleted file may also remain allocated while a process keeps it open. Record the error and destination path before cleaning up at random. The first goal is to identify which resource limit was reached, because different limits require different remedies and can produce similar application symptoms.

Separate the destination from the retention cause

Suppose application logs and database WAL occupy separate volumes. Removing old application logs cannot necessarily relieve the WAL volume. In PostgreSQL, failed archiving or delayed replication-slot consumption can retain WAL that is still needed by the configured recovery pipeline. Do not infer that a rapidly growing file is disposable simply because it looks old. Compare capacity over time with archive and replication state. That links a growth pattern to its retention cause and helps distinguish a temporary burst from a backlog that will continue consuming every newly available byte.

Do not destroy recovery while freeing space

Do not manually delete database-managed data or WAL files to make an error disappear. Removing recovery information or storage structures can turn an availability incident into data loss. Follow an established operational procedure for disposable logs, failed temporary exports or capacity expansion instead. Releasing deleted-but-open files may require cooperation from the owning process; evaluate the impact of restarting it. Also avoid assuming that a space-reclaiming operation is free: VACUUM FULL can require substantial extra workspace and strong locks, making it a poor improvised first response on an already exhausted system.

Verify successful writes after capacity returns

After capacity is restored, verify that actual writes succeed, transactions recover appropriately, and archiving or replication resumes where relevant. Before replaying failed requests, establish their execution outcome and protect against duplicate side effects. Capacity monitoring should consider growth rate, time to exhaustion, inodes and retained WAL, not just a percentage threshold. Separate the immediate mitigation that creates headroom from the repair that stops unbounded accumulation. A successful cleanup is not durable recovery if the same backlog refills the filesystem shortly afterward or the application remains in a failed transaction state.

Reading and practice are recorded separately in this browser, for each language.

Learn this in order

Understand the systems behind the web · STEP 13

Choose a learning path