Not All Backend Services Deploy the Same Way

A team is preparing to deploy a new feature. The API service update goes smoothly - a few seconds of rolling replacement, zero downtime. Then comes the worker update. Someone hits deploy, and suddenly all queued image-processing jobs disappear. The team stares at the logs, confused. "It worked for the API. Why did the worker break?"

This scenario plays out in engineering teams every week. The root cause is almost never a bug in the deployment tool. It is a misunderstanding about what kind of backend service they are dealing with.

Backend services look similar on the surface. They all run on servers, they all have code, they all need updates. But how they receive work, how they hold state, and how they handle interruptions are fundamentally different. A CI/CD pipeline that works perfectly for one type can silently destroy data for another.

API Services: Always Listening, Always Available

The most familiar backend type is the API service. It listens on a port, waits for incoming requests from mobile apps, websites, or other services, and returns a response. Users feel it immediately when an API service goes down - the app stops loading, the page shows an error, the transaction fails.

The following flowchart classifies backend services by how they receive work and maps each type to its recommended deployment strategy.

flowchart TD A[How does the service receive work?] --> B[Request] A --> C[Queue] A --> D[Schedule] A --> E[Stream] B --> F[API Service] F --> G[Rolling / Blue-Green / Canary] C --> H[Worker] H --> I[Graceful shutdown after job completes] D --> J[Scheduler] J --> K[Prevent overlap, use distributed locks] E --> L[Consumer] L --> M[Commit offset on graceful shutdown]

API services need to be always ready. They must handle traffic spikes by scaling instances up and down. They must accept new connections while old ones are still being processed. And critically, they must be updated without dropping in-flight requests.

For CI/CD, this means the deployment strategy matters. A simple stop-and-start deployment will cut off active users. Rolling updates, blue-green deployments, or canary releases become necessary. The pipeline needs to verify that new instances pass health checks before old ones are removed.

Workers: Process Jobs, Not Requests

Workers do not talk directly to users. They pick up tasks from a queue, process them one by one, and store the results. Image resizing, email sending, report generation - these are worker jobs. The user uploads a photo and gets a response immediately, but the resizing happens behind the scenes.

Because workers do not serve live requests, they can be updated more gently. The key concern is not dropping active jobs. A good worker pipeline waits for the current job to finish before shutting down the old process. If the queue system supports it, the worker can signal that it is stopping, finish its current task, then exit. New instances start, pick up the next jobs from the queue, and the system continues without losing work.

The mistake teams make is treating workers like API services. They kill the process immediately during deployment, and all in-progress jobs are lost. The queue does not know the job failed halfway - it just never sees a completion signal. The job disappears into a black hole.

A Kubernetes Deployment expects to run forever and uses readiness probes to manage traffic. A Job, by contrast, runs to completion and uses restartPolicy: Never:

# API service - runs forever, needs readiness probe
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-service
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
      - name: api
        image: myapp/api:latest
        ports:
        - containerPort: 8080
        readinessProbe:
          httpGet:
            path: /health
            port: 8080
---
# Worker - runs a single job, then exits
apiVersion: batch/v1
kind: Job
metadata:
  name: image-resize-job
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: worker
        image: myapp/worker:latest
        command: ["process-queue"]

Schedulers: Timing Is Everything

Schedulers run tasks on a schedule. Clean up old data at midnight. Sync with an external system every hour. Generate daily reports at 6 AM.

The challenge with schedulers is avoiding duplicate execution. If a scheduler runs a cleanup job, and the deployment happens to restart the scheduler right when the job starts, there is a risk that the new instance also triggers the same job. Now the cleanup runs twice, potentially causing conflicts or data issues.

A good scheduler pipeline ensures that the scheduler does not start a new instance while the old one is still running a scheduled task. It also needs a mechanism to detect and skip overlapping runs. Some teams use distributed locks or database markers to prevent double execution.

Consumers: Keeping Up With the Stream

Consumers are similar to workers, but they process a continuous stream of data. They read from message brokers like Kafka or RabbitMQ, process messages in real time, and often need to maintain their position in the stream.

The critical difference is speed and offset tracking. If a consumer falls behind, it needs to catch up without overwhelming the system. If a consumer crashes, it must resume from where it left off, not from the beginning.

For CI/CD, this means the deployment must handle offset commits carefully. Shutting down a consumer without committing the current offset can cause reprocessing of already-handled messages. Restarting from the wrong offset can skip messages or process them twice. The pipeline needs to coordinate with the message broker to ensure graceful shutdown and resume.

Internal Services: The Hidden Dependencies

Internal services are not accessed by frontend applications. They serve other services within the system - authentication, configuration, logging, feature flags. Multiple services depend on them, often synchronously.

The danger with internal services is cascading failures. A small change in an authentication service can break every service that calls it. A slow response from a configuration service can cause timeouts across the entire system.

These services need stricter testing and more careful deployment. The pipeline should run integration tests that simulate real caller behavior. Deployment should be gradual, with monitoring on dependent services. Rollback must be fast and reliable because the blast radius is wide.

A Practical Checklist for Backend Service Pipelines

Before designing a pipeline for any backend service, ask these questions:

  • How does this service receive work? (request, queue, schedule, stream)
  • Can it be stopped safely mid-task? If yes, what happens to the task?
  • Does it hold any in-memory state that cannot be recovered?
  • What happens if two instances run the same task simultaneously?
  • Which other services break if this service goes down?
  • How long can this service be unavailable before users or systems notice?

The answers will tell you whether you need rolling updates, graceful shutdown hooks, queue coordination, or dependency-aware deployment ordering.

The Takeaway

A CI/CD pipeline is not a one-size-fits-all script. It is a reflection of how your service actually works. API services need connection awareness. Workers need job completion guarantees. Schedulers need overlap prevention. Consumers need offset management. Internal services need dependency coordination.

When you understand what kind of backend service you are dealing with, you stop copying pipeline templates from blog posts and start building pipelines that protect your data, your users, and your team's sleep.