When a Green Pipeline Doesn't Mean a Healthy Deployment

You just finished a deployment. The pipeline shows green. Every build step passed, every test ran clean, and the deploy script completed without a single error. You close your laptop feeling good.

Then the first user report comes in. The app is loading slowly. Some requests are failing. A few minutes later, the database connection pool is exhausted, and the app is practically down.

What happened? The pipeline said everything was fine.

This scenario is more common than most teams want to admit. A successful build only proves that your code can compile or package. It does not prove that your application can actually receive requests, connect to its database, or return correct responses. There is a gap between "the deploy finished" and "the application works." That gap is where health signals come in.

What a Health Signal Actually Tells You

A health signal is how your application reports its own condition. The most common form is a health endpoint: a dedicated URL that your pipeline, load balancer, or operations team can call to check whether the app is functioning. You have probably seen endpoints named /health or /healthz. When called, a healthy application returns HTTP 200. An unhealthy one returns 500 or another error code.

But the real value is not in the endpoint itself. It is in what the endpoint actually checks. A health endpoint that always returns 200 regardless of the application's real state is worse than having no health check at all. It gives false confidence. Your pipeline thinks the deployment succeeded, but your users are already hitting errors.

Readiness vs Liveness: Two Different Questions

There are two distinct kinds of health checks, and they answer different questions.

The following flowchart illustrates the difference between liveness and readiness checks and how they guide deployment decisions:

flowchart TD A[Health Check Request] --> B{Is the app process running?} B -->|No| C[Liveness fails: Restart container] B -->|Yes| D{Is the app ready to serve traffic?} D -->|No| E[Readiness fails: Remove from load balancer] D -->|Yes| F[Readiness passes: Send traffic] C --> G[Automatic recovery attempt] E --> H[Wait and retry readiness] F --> I[App serves users normally] H --> D G --> B

Readiness tells the system whether the application is ready to accept traffic. When an application just started, it might need a few seconds to connect to the database, load cache, or warm up connections. During that time, the app should report itself as "not ready." The load balancer or orchestrator will then hold off sending requests until the app signals it is ready. Without a proper readiness check, users might hit a half-initialized application that returns errors or timeouts.

Liveness tells the system whether the application is still alive and processing. If the application gets stuck in a deadlock, runs out of memory, or enters a state where it cannot handle any work, the liveness check detects that. In containerized environments like Kubernetes, a failing liveness check typically triggers an automatic restart.

Both checks serve different purposes in your deployment pipeline. When you run a deployment, the pipeline usually calls the readiness check first. If the new version does not become ready within a reasonable time, the pipeline can decide to roll back immediately. That is far better than letting a half-baked version serve users. Liveness checks, on the other hand, are more useful after the deployment is done. They keep the application running healthily over time.

Here is a practical YAML example that configures both probes for a Kubernetes deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
      - name: app
        image: my-app:1.0.0
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 10
          failureThreshold: 3
        livenessProbe:
          httpGet:
            path: /live
            port: 8080
          initialDelaySeconds: 15
          periodSeconds: 20
          failureThreshold: 3

In this example, the readiness probe checks /ready every 10 seconds after a 5-second delay. If it fails three times, Kubernetes stops sending traffic to the pod. The liveness probe checks /live less frequently and restarts the container if it fails.

What a Good Health Check Actually Checks

A health check that just returns 200 without verifying anything is a decoration, not a signal. A meaningful health check should test the components that matter for your application to function:

  • Database connectivity: Can the app reach the database and run a simple query?
  • Critical external APIs: Are the services your app depends on available?
  • Internal resources: Is the thread pool healthy? Is memory usage within limits?

But there is a trade-off. A health check that tests every dependency on every call can become a burden on the system. If your pipeline or load balancer calls the endpoint every few seconds, a heavy check can degrade performance or even cause cascading failures.

A common pattern is to keep liveness checks lightweight: just confirm the process is running. Readiness checks can be deeper, because they are called less frequently and their result directly affects whether traffic is routed to the instance.

Health Signals in Progressive Deployment Strategies

Health signals become even more important when you use deployment strategies like canary or blue-green. Imagine you roll out a new version to a small subset of users. Your pipeline can monitor the health endpoint of that new version. If the health check starts showing errors or the response time increases, the pipeline can automatically shift traffic back to the old version.

Without health signals, you rely on user reports. User reports are valuable, but they arrive late. By the time someone files a bug report, many users may have already been affected. A health check that runs every few seconds can catch problems within seconds, not minutes or hours.

What Happens When Health Checks Are Missing or Weak

Teams that skip proper health checks often discover problems the hard way. A deployment succeeds, but the new version cannot connect to the database because a credential changed. The pipeline reports success. Users see errors. Someone has to manually investigate, realize the issue, and trigger a rollback. That process takes time, and during that time, the application is degraded.

The same applies to dependencies. If your application depends on an external API and that API goes down, a good health check will report the application as unhealthy. A weak health check will report healthy, and your users will experience failures while your monitoring dashboard shows green.

A Quick Practical Checklist

If you are setting up health checks for your application, here is a short list of things to verify:

  • Does your health endpoint actually check something meaningful, or just return 200?
  • Do you have separate readiness and liveness checks with different depths?
  • Does your pipeline use the readiness check to decide whether to proceed with deployment or roll back?
  • Are your health checks lightweight enough that they do not degrade performance when called frequently?
  • Do your progressive deployment strategies use health signals to detect problems early?

The Real Test of a Deployment

A green pipeline is not proof that your application works. It is proof that your build and deploy process ran without errors. The real test comes after the deploy, when traffic hits the new version. Health signals are the bridge between "the deploy finished" and "the application is actually serving users correctly."

When your pipeline can detect that a new version is not healthy within seconds and automatically roll back, you have moved from hoping deployments go well to knowing they are safe. That is the difference between a deployment that looks good on paper and one that actually works in production.