The Database That Was Silently Read-Only for 3 Months: SELinux, Docker, and a UID You Forgot

Meta description: A production database went read-only for months due to a forgotten UID mapping in Docker with SELinux. Learn how to prevent this costly infrastructure mistake.

The Silent Killer of Database Performance

Your monitoring dashboards look green. Your application appears healthy. Users aren't complaining about downtime. Yet for three months, your production database has been silently rejecting every write operation, forcing your application into an unexpected read-only mode that nobody noticed.

This isn't hypothetical. It's a real-world disaster that happens when three technologies collide in ways most engineering teams never anticipate: SELinux security policies, Docker containerization, and Unix user ID mappings.

How a Simple UID Becomes Your Worst Enemy

The problem starts innocently enough. You're running a containerized database—PostgreSQL, MySQL, or MongoDB—inside Docker on a RHEL, CentOS, or Fedora system with SELinux enabled. Everything works perfectly in development and staging. Then production breaks in the most insidious way possible.

Here's what actually happens behind the scenes:

Your database container runs as a specific user ID inside the container namespace. Docker maps this internal UID to a corresponding UID on the host system. When SELinux enforces security policies, it checks whether this mapped UID has permission to write to the database files on the host filesystem.

If the UID mapping is wrong—and it often is—SELinux blocks all write operations while allowing reads to continue normally. Your database appears functional but can't persist any changes.

The Three-Month Blind Spot

Why does this problem persist for months without detection? Because most monitoring focuses on uptime, not write capability. Your database responds to queries. Connection pools stay healthy. Response times remain normal for read operations.

Meanwhile, your application logic might gracefully handle write failures by falling back to cached data or read-only modes. Users see slightly stale information but don't experience obvious breakage. Critical business data—new user registrations, transaction records, inventory updates—disappears into the void.

The financial impact compounds daily. Lost customer data means lost revenue. Compliance violations accumulate. When you finally discover the issue, you're facing data recovery challenges that could have been prevented with proper configuration.

SELinux: Security That Bites Back

SELinux operates on mandatory access control. Unlike traditional Unix permissions that rely on discretionary access control, SELinux enforces policies regardless of what the file owner wants to allow.

When Docker creates containers, it doesn't automatically configure SELinux contexts for your specific use case. The default policies often conflict with database operations, especially when volume mounts are involved. Your database process might have the correct Unix permissions but lack the proper SELinux security context to write to mounted volumes.

This creates a perfect storm: the container starts successfully, the database initializes, connections work, but writes fail silently because SELinux is doing exactly what it's designed to do—block potentially dangerous operations.

Docker's UID Mapping Maze

Docker's user namespace feature adds another layer of complexity. When you run a container with user namespace remapping enabled, the UID inside the container doesn't match the UID on the host system. A process running as UID 999 inside the container might actually be UID 100999 on the host.

Database containers often expect to run as specific UIDs—postgres typically uses UID 999, mysql uses UID 999 or 27, mongodb uses UID 999. If your Docker daemon is configured with user namespace remapping, these UIDs get shifted in ways that break file ownership assumptions.

The database starts normally because it can read its configuration files and initial data. But when it tries to write transaction logs, create new tables, or update existing records, the host filesystem rejects these operations due to permission mismatches.

Detection Strategies That Actually Work

Standard monitoring won't catch this problem. You need specific checks that verify write capability, not just read availability:

Monitor database transaction logs for write operation failures. Most databases log permission errors, but these logs often get buried in verbose output that nobody reads regularly.

Implement synthetic write tests that attempt small, non-critical database operations at regular intervals. A simple health check that inserts and deletes a timestamp record will immediately reveal write permission issues.

Track application-level metrics that indicate write success rates. If your application normally processes 1000 write operations per hour but suddenly drops to zero, investigate immediately.

Set up alerts for SELinux denial messages in system logs. The ausearch command can help you identify when SELinux is blocking database operations:

ausearch -m avc -ts recent

Prevention Through Proper Configuration

The most effective prevention involves getting the configuration right from the start:

Configure SELinux policies specifically for your database containers. Use setsebool to enable container-specific policies:

setsebool -P container_manage_cgroup on

Set proper SELinux contexts on database volume mounts. Use the Z flag when mounting volumes in Docker:

docker run -v /host/data:/container/data:Z

Disable user namespace remapping for database containers if you don't need the additional isolation. Add --userns=host to your Docker run command.

Test write operations explicitly during deployment. Don't assume that a successful container start means everything works correctly.

The Real Cost of Infrastructure Assumptions

This type of failure highlights a broader problem in modern infrastructure: the complexity of layered security and containerization creates failure modes that traditional testing doesn't catch.

When you're evaluating infrastructure decisions, consider not just the happy path but the subtle ways that security policies, container runtimes, and application assumptions can interact. The most dangerous failures are the ones that look like success until you need the functionality that's been silently broken.

Your database isn't just a storage layer—it's a critical component that needs explicit verification of both read and write capabilities. Don't let a forgotten UID mapping cost you three months of data and customer trust.

The next time you deploy a containerized database with SELinux enabled, remember: green dashboards don't guarantee working writes. Test explicitly, monitor comprehensively, and never assume that silence means success.