45-5 · Chapter 45 · 5 min read

Why Your App Behaves Differently in Staging and Production

You deploy the same code to staging, run your tests, everything passes. Then you deploy to production, and the app crashes. The code is identical. The

Why Your App Behaves Differently in Staging and Production

You deploy the same code to staging, run your tests, everything passes. Then you deploy to production, and the app crashes. The code is identical. The difference is a single configuration value you forgot to update.

This scenario plays out in teams every week. The code is fine. The configuration is not. And when the problem is a secret like a database password or an API token, the stakes get even higher. A leaked secret can cause more damage than most code bugs.

What Configuration and Secrets Actually Are

Configuration is any value that changes how your application behaves without changing its code. Server addresses, maximum connection counts, logging levels, feature flags - these are all configuration. The same codebase runs differently in development, staging, and production because the configuration changes.

Secrets are a special category of configuration. They are values that must stay confidential: passwords, tokens, encryption keys, certificates. Secrets need different handling because their exposure is a security incident, not just a configuration mistake.

The key insight is that both configuration and secrets need to be managed separately from your application code. They live in a different layer of your delivery system.

Start With a Configuration Template

Before you can manage configuration properly, you need to know what configuration your application actually needs. A configuration template is simply a complete list of every configuration value your application expects, organized by environment.

For each configuration item, record:

Here is a concrete example of what such a template might look like in YAML:

# config-template.yaml
# Application configuration template
# Override values per environment in environment-specific files

app:
  name: my-app
  version: 1.0.0
  log_level: info  # override: dev=debug, staging=info, prod=warn
  max_retry: 3

database:
  host: localhost  # override: staging=db-staging.example.com, prod=db-prod.example.com
  port: 5432
  name: myapp_db
  pool_size: 10  # override: prod=50

cache:
  host: localhost  # override: staging=redis-staging.example.com, prod=redis-prod.example.com
  port: 6379
  ttl_seconds: 3600

feature_flags:
  new_checkout: false  # override: staging=true, prod=false
  dark_mode: true

The variable name (like DB_HOST or MAX_RETRY)
The type of value (string, number, boolean)
Which environments use it
Whether the value differs between environments

Some values will be the same everywhere. MAX_RETRY might be 3 in development, staging, and production. Other values must differ. DB_HOST will point to your local database in development and your production database cluster in production.

The template should also note who can change each configuration value and how those changes are tracked. Not everyone on the team should be able to change production database connection strings.

Secrets Need Their Own Template

Secrets follow the same idea but with stricter rules. A secret template records:

The secret name
Where it comes from (vault, parameter store, encrypted file)
Which environments need access to it
When it was last rotated
How long it is valid
The exact steps to rotate it
How to verify the application still works after rotation

Rotation is the process of replacing an old secret with a new one on a regular schedule. Many teams skip this until a breach forces them to act. A template makes rotation a routine operation instead of an emergency.

The verification step is critical. After rotating a database password, you need to confirm that your application can still connect and query data. Otherwise you might rotate the secret and break production without realizing it until users start reporting errors.

Audit Everything

Configuration and secrets need audit trails. An audit log records who accessed or changed a configuration value or secret, when they did it, and why.

The template for audit logs should capture:

Timestamp of the action
Who performed it
What action was taken (view, modify, delete)
Which configuration or secret was affected

This log is essential when something goes wrong. If a secret leaks, you need to know who accessed it and when. If a configuration change breaks production, you need to trace back to who made the change and what the previous value was.

Without audit logs, you are debugging blind. You know something changed, but you have no way to find out what or who caused it.

Test Your Configuration and Secrets

Here is a step that most teams overlook: configuration and secrets need to be tested. You test your code. You test your infrastructure. But how often do you test that your application can actually read its configuration and secrets correctly in each environment?

A configuration test verifies that:

All required configuration values exist
They have the correct type
They are within expected ranges
The application starts successfully with these values

A secret test verifies that:

The secret exists and is accessible
The application can authenticate using the secret
The application can perform its basic operations after authentication

These tests catch problems before deployment. A missing secret or a misformatted configuration value will cause your application to crash at startup. Better to catch that in a test than in production.

A Practical Checklist

Here is a short checklist you can use when setting up configuration and secret management for a new application or service:

List all configuration values the application needs, organized by environment
Note which values differ between environments and which stay the same
Identify which values are secrets and need special handling
Document the source of each secret (vault, parameter store, etc.)
Create a rotation schedule for each secret
Set up audit logging for configuration and secret access
Write tests that verify configuration and secret loading in each environment
Document who can change each configuration value and secret

The Takeaway

Configuration and secrets are not an afterthought. They are a separate concern that needs its own templates, its own testing, and its own audit trail. The same code that works perfectly in staging will fail in production if a single configuration value is wrong or a single secret is missing. Treat configuration and secrets with the same rigor you treat your application code, and you will eliminate a whole category of deployment failures.