Spring Boot Observability and Production Operations - Complete Notes

A practical and interview-focused guide to Actuator, health probes, Micrometer metrics, structured logs, correlation, distributed tracing, secure configuration, graceful shutdown, JVM diagnostics, containers, deployment safety, and evidence-based incident troubleshooting.

00. The production mental model

Production operations is the discipline of making a running system understandable, safe to change, and recoverable when reality differs from expectations.

A program can be logically correct and still fail as a service. It can run out of database connections, wait forever on a downstream dependency, exhaust heap, produce too much logging, reject traffic during a rollout, or return successful HTTP responses while silently losing business work. Production readiness therefore includes runtime signals and operating procedures, not only application code.

01. Spring Boot Actuator

Actuator adds production-oriented endpoints and auto-configuration that expose selected runtime information through HTTP or JMX.

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
  <scope>runtime</scope>
</dependency>

An endpoint must be both enabled and exposed. Most endpoints are enabled for in-process use, but only health is exposed over HTTP by default. Exposure is an attack-surface decision. Endpoints such as environment, configuration properties, heap dump, loggers, mappings, scheduled tasks, and thread dump can reveal secrets, application structure, customer data, or operational controls.

management:
  endpoints:
    web:
      exposure:
        include: health,info,prometheus
  endpoint:
    health:
      show-details: when-authorized
      roles: OPS
  info:
    env:
      enabled: false

Prefer an explicit allowlist over *. Protect management traffic with Spring Security, network policy, a private listener, or several layers together. A separate management port can reduce exposure, but a probe on that port can succeed while the application connector is broken. For traffic decisions, expose a minimal health group on the main server port.

Securing Actuator with an explicit filter chain

@Bean
@Order(1)
SecurityFilterChain actuatorSecurity(HttpSecurity http) throws Exception {
    http.securityMatcher(EndpointRequest.toAnyEndpoint())
        .authorizeHttpRequests(authorize -> authorize
            .requestMatchers(EndpointRequest.to(HealthEndpoint.class)).permitAll()
            .anyRequest().hasRole("OPS"))
        .httpBasic(Customizer.withDefaults());
    return http.build();
}

Defining a custom SecurityFilterChain makes Spring Boot back off from its Actuator security auto-configuration, so the application must cover every intended management rule. Network restrictions remain useful because application authentication can itself be misconfigured.

02. Health indicators and health groups

Health is a small operational contract, not a detailed status page and not a replacement for metrics.

Spring Boot assembles HealthContributor beans into a tree. A HealthIndicator returns a status such as UP, DOWN, OUT_OF_SERVICE, or UNKNOWN, optionally with details. A CompositeHealthContributor groups contributors. A status aggregator chooses the overall status, and an HTTP status mapper turns it into a response code.

@Component
final class CatalogHealthIndicator implements HealthIndicator {
    private final CatalogClient catalog;

    CatalogHealthIndicator(CatalogClient catalog) {
        this.catalog = catalog;
    }

    @Override
    public Health health() {
        try {
            CatalogStatus status = catalog.status(Duration.ofMillis(300));
            return status.acceptingOrders()
                    ? Health.up().withDetail("mode", "accepting-orders").build()
                    : Health.outOfService().withDetail("mode", status.mode()).build();
        } catch (Exception failure) {
            return Health.down(failure).build();
        }
    }
}

Keep checks fast, bounded, and cheap. Never perform a full table scan, mutate state, or wait on an unbounded remote call. If health collection becomes slow, monitoring can amplify an outage by creating more load. A component that can be observed through metrics but does not control routing often should not be in readiness.

03. Liveness, readiness, and startup probes

Each probe must answer one narrow question because the platform takes a different action for each answer.

management:
  endpoint:
    health:
      probes:
        enabled: true
      group:
        liveness:
          additional-path: "server:/livez"
        readiness:
          additional-path: "server:/readyz"

startupProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 5
  failureThreshold: 30
livenessProbe:
  httpGet:
    path: /livez
    port: 8080
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2

Tune thresholds from real startup and response distributions. A startup probe is clearer than a huge liveness initial delay because it protects slow initialization without weakening later detection. Avoid probe periods so aggressive that temporary CPU pressure triggers a restart loop. Test failure actions in a staging environment, including rolling deployment and termination.

04. Metrics and Micrometer

Metrics summarize behavior as numeric time series and are best for trends, alerts, capacity, and service-level objectives.

Micrometer is an instrumentation facade. Application and library code use a common meter model, while a registry publishes to Prometheus, OTLP, Datadog, or another backend. Spring Boot auto-configures a MeterRegistry, built-in JVM and system binders, web request metrics, and integration metrics when the matching technology is present.

@Component
final class CheckoutMetrics {
    private final MeterRegistry registry;

    CheckoutMetrics(MeterRegistry registry) {
        this.registry = registry;
    }

    void recordRejected(String reason) {
        registry.counter(
                "checkout.rejected",
                "reason", normalizeReason(reason))
            .increment();
    }

    <T> T timeAuthorization(Supplier<T> action) {
        return registry.timer("checkout.authorization")
                .record(action);
    }

    private String normalizeReason(String reason) {
        return switch (reason) {
            case "OUT_OF_STOCK", "PAYMENT_DECLINED" -> reason;
            default -> "OTHER";
        };
    }
}

Cardinality is a production constraint

Every unique combination of meter name and tag values creates a time series. Tags such as customer ID, order ID, URL with identifiers, exception message, email address, or raw SQL can create millions of series. That increases memory, ingestion cost, query time, and incident risk. Use bounded dimensions such as route template, status class, operation, region, or enumerated outcome. Put request-specific values in traces and logs.

Configure histograms and percentiles deliberately. Client-side percentiles are difficult to aggregate across instances. Histogram buckets support aggregate percentile estimation but add series. Choose buckets around user-facing latency objectives, and verify backend semantics for counters, steps, resets, and exemplars.

05. Logs, structured logging, and correlation

A useful log is a bounded, structured event that explains something an operator may need to know.

Spring Boot uses Commons Logging internally and normally provides Logback through its starters. Containerized applications usually write to standard output and let the platform collect, rotate, retain, and ship logs. Avoid application-managed files unless the environment specifically requires them.

spring:
  application:
    name: orders
logging:
  structured:
    format:
      console: ecs
  level:
    root: INFO
    com.example.orders: INFO

log.atInfo()
    .addKeyValue("order.id", orderId)
    .addKeyValue("outcome", "accepted")
    .log("Order accepted");

log.atWarn()
    .addKeyValue("payment.provider", provider)
    .addKeyValue("attempt", attempt)
    .setCause(failure)
    .log("Payment authorization failed");

Define event names and field meanings as an internal contract. Include service, environment, version, trace and span IDs, operation, stable outcome, and safe business identifiers only when policy permits. Never log passwords, access or refresh tokens, session cookies, API keys, secret configuration, full payment data, or unnecessary personal information. Masking after ingestion is weaker than never emitting the value.

Correlation IDs join events for one logical operation. Distributed tracing usually supplies trace and span IDs and propagates trace context over supported clients. If accepting a client-provided request ID, validate its length and character set before logging it. Do not assume MDC values move to worker threads automatically. Thread changes require explicit context propagation, and every manually installed MDC value must be cleared in a finally block.

06. Observations and distributed tracing

A trace follows one operation through nested spans, while Micrometer Observation can drive both metrics and tracing from one instrumentation lifecycle.

An observation has a name, a context, lifecycle events, and key values. Low-cardinality values can become metric tags and trace attributes. High-cardinality values belong only in traces. Registered handlers turn the lifecycle into timers, spans, context propagation, or other behavior.

@Component
final class InventoryReservation {
    private final ObservationRegistry observations;

    InventoryReservation(ObservationRegistry observations) {
        this.observations = observations;
    }

    Reservation reserve(String warehouse, UUID orderId) {
        return Observation.createNotStarted("inventory.reserve", observations)
            .lowCardinalityKeyValue("warehouse.region", regionOf(warehouse))
            .highCardinalityKeyValue("order.id", orderId.toString())
            .observe(() -> performReservation(warehouse, orderId));
    }
}

Auto-instrumented HTTP clients are important because they inject trace headers. Constructing a client outside the auto-configured builders can lose propagation. Context can also disappear at @Async, executor, scheduler, callback, or reactive boundaries. Use Spring Boot context propagation support or a ContextPropagatingTaskDecorator for a custom executor, then test that child work retains the expected trace.

Sampling controls volume and cost. Head sampling decides near the start and can miss rare failures. Tail sampling decides after more of the trace is known but requires capable collector infrastructure. A production strategy may keep all errors and slow operations while sampling a smaller share of ordinary successes. Trace data is not a durable audit record.

07. Configuration and secret management

Build one artifact and supply environment-specific configuration at deployment time.

Spring Boot can read properties, YAML, environment variables, command-line arguments, imported files, and configuration trees. Bind related settings into validated @ConfigurationProperties objects. Fail startup for missing or invalid required values rather than discovering them under traffic.

@ConfigurationProperties("payments")
@Validated
public record PaymentProperties(
        @NotNull URI baseUrl,
        @NotNull Duration connectTimeout,
        @NotNull Duration readTimeout,
        @Min(1) int maxAttempts) {}

spring:
  config:
    import: "optional:configtree:/run/secrets/"

Store secrets in a dedicated secret manager or platform secret facility, grant access by workload identity, encrypt in transit and at rest, rotate them, and audit reads. Environment variables are convenient but may be visible through process inspection, diagnostics, deployment metadata, or accidental logging. Mounted files also need strict permissions and rotation behavior. Never bake secrets into source, images, default configuration, test fixtures, or command history.

08. Graceful shutdown and draining

Graceful shutdown stops new work, gives accepted work a bounded chance to finish, and then releases resources.

Current Spring Boot versions enable graceful shutdown for supported embedded servers. Closing the application context begins server shutdown early in the SmartLifecycle stop phases. Existing requests receive a grace period while new requests are rejected according to server behavior.

spring:
  lifecycle:
    timeout-per-shutdown-phase: 25s

The platform termination grace period must be longer than application draining, with time for network propagation and final process termination. For example, if Spring has 25 seconds, a Kubernetes pod might need 35 or 40 seconds. On SIGTERM, readiness should turn false before or while the server stops accepting traffic. Load balancer state does not update instantaneously, so allow a measured drain interval where needed.

HTTP is only one kind of work. Stop message consumption, scheduler claims, polling loops, and new async submissions. Let in-flight work finish or persist a safe retry state. A request that exceeds the grace period needs idempotency because the client may retry without knowing whether the first attempt committed. Do not rely only on JVM shutdown hooks for critical durable work.

09. Database connection pool monitoring

A connection pool is a concurrency gate around a scarce database resource, not a speed setting to maximize.

Spring Boot commonly uses HikariCP with JDBC and JPA starters. Monitor active, idle, total, pending, acquisition time, usage time, timeout count, maximum size, and database-side connections. The important incident pattern is sustained pending demand plus acquisition latency and timeouts, especially when active connections stay near the configured maximum.

Increasing the pool can move contention into PostgreSQL and make every query slower. Size the total across all instances, workers, administrative tools, and failover headroom. Keep transactions short, set bounded acquisition and statement timeouts, diagnose leaks, and apply backpressure before requests pile up without limit.

10. Thread dumps, heap diagnostics, and JFR

Runtime diagnostics are snapshots and recordings that test a hypothesis about CPU, blocking, deadlock, allocation, garbage collection, or memory retention.

Thread dumps

A thread dump records thread names, states, stack traces, and owned or awaited monitors. Capture several dumps a few seconds apart. One snapshot can show a normal moment; repeated identical stacks reveal stuck work, while moving stacks reveal progress.

jcmd <pid> Thread.print -l
jcmd <pid> GC.class_histogram
jcmd <pid> GC.heap_dump filename=/secure/diagnostics/app.hprof
jcmd <pid> JFR.start name=incident settings=profile duration=5m \
  filename=/secure/diagnostics/incident.jfr

Heap evidence

Heap usage rising after full garbage collections suggests retained live objects, but a single sawtooth is normal. Start with memory pool, allocation rate, pause, and post-GC trend metrics. A class histogram is smaller and faster than a full heap dump. A heap dump can pause the process, consume disk roughly comparable to live heap, and contain secrets or personal data. Store it in a restricted location, transfer it securely, set retention, and delete it according to policy.

In analysis, distinguish shallow size from retained size. Find dominator paths and GC roots that keep unexpected objects alive. Common causes include unbounded caches, static collections, thread-local values, listeners that were never removed, buffered requests, oversized sessions, class-loader retention, and queues whose consumers cannot keep up.

Java Flight Recorder

JFR records time-based evidence such as CPU samples, allocation, locks, garbage collection, file and socket activity with production-oriented overhead. It is often more useful than a heap dump for a broad latency or CPU incident. Use a bounded recording, preserve the exact deployment version, and correlate its time window with service metrics and traces.

11. Error reporting and failure taxonomy

Error reporting must group useful failures without leaking data or turning expected outcomes into noise.

Separate programmer defects, transient infrastructure failures, invalid client input, expected business rejection, and security events. They have different severity, ownership, retry policy, and alert behavior. A declined card is not an application exception; a NullPointerException in every checkout is.

• Capture the application version, environment, operation, safe request context, and trace ID.
• Preserve exception type, message after redaction, stack, cause, and suppressed exceptions.
• Group by stable fingerprint, not raw message containing identifiers.
• Rate-limit duplicate reports while retaining occurrence counts.
• Attach release and deployment markers to spot regressions.
• Link alerts to a dashboard, runbook, owner, and rollback procedure.

Report an exception once at the boundary that can add operational context or choose recovery. Logging and rethrowing at every layer creates duplicates. If a lower layer handles and recovers from a failure, log at an appropriate level only when the event is still operationally meaningful.

12. Containerization and deployment concerns

A container image is a versioned runtime artifact; orchestration supplies identity, configuration, resources, rollout, and recovery.

FROM eclipse-temurin:21-jre
RUN addgroup --system spring && adduser --system --ingroup spring spring
WORKDIR /app
COPY --chown=spring:spring target/orders.jar app.jar
USER spring
ENTRYPOINT ["java", "-jar", "/app/app.jar"]

Use a trusted minimal base, run as a non-root user, scan application and operating-system dependencies, rebuild for security updates, and pin an auditable image digest in deployment. Multi-stage builds can keep compilers and source out of the runtime image. Never put secrets in image layers or build arguments that become metadata.

Resource limits and JVM ergonomics

Container memory includes Java heap, metaspace, code cache, direct buffers, thread stacks, native libraries, memory-mapped regions, and monitoring overhead. Setting maximum heap equal to the container limit invites an operating-system kill. Leave measured native headroom. CPU limits can affect parallelism, garbage collection, timeouts, and startup. Load-test with the same limits used in production.

Safe rollouts

• Use immutable versions and expose the version in telemetry.
• Require readiness before routing and set a realistic progress deadline.
• Keep enough healthy capacity during rolling replacement.
• Use backward-compatible database migrations across old and new application versions.
• Watch error, latency, saturation, and business outcomes during canary or staged rollout.
• Define an automated or quick manual rollback, but remember data migrations may not roll back.
• Test termination, node loss, dependency outage, and telemetry backend failure.

13. SLOs, dashboards, and actionable alerts

Monitoring becomes operationally useful when it measures a promise, identifies risk, and routes a concrete action to an owner.

A service-level indicator is a measured ratio or distribution, such as successful eligible requests divided by total eligible requests, or the share of requests below a latency threshold. A service-level objective is the target over a window. The error budget is the allowed unreliability. Define eligibility and success from the user view, including which status codes, cancellations, maintenance, and client errors count.

Page on sustained user impact or imminent exhaustion that requires timely human action. Ticket slower trends such as disk growth or certificate expiry. Avoid alerts for every exception, CPU spike, or single failed probe. Use multi-window burn-rate alerts for SLOs so severe fast burns and slower persistent burns are both detected without constant noise.

14. A practical incident workflow

Troubleshooting should narrow uncertainty with time-correlated evidence, not begin with random restarts or configuration changes.

1. Protect users. Roll back, disable a risky feature, shed optional work, fail over, or scale only when the action is safe and understood.
2. State the symptom precisely. Name the operation, affected population, start time, severity, and evidence of recovery or continued impact.
3. Check recent changes. Compare deployment, configuration, traffic, dependency, certificate, and infrastructure events to the symptom time.
4. Follow the golden signals. Determine whether the problem is errors, latency, traffic shape, saturation, or several together.
5. Localize. Compare versions, zones, instances, routes, tenants, and dependencies. Use traces to find where time or failure appears.
6. Form one falsifiable hypothesis. Say what evidence would confirm or reject it, then run the smallest safe check.
7. Mitigate and verify. Watch the original user signal, not only the component that was changed.
8. Preserve evidence and learn. Record the timeline, contributing conditions, detection gap, safe corrective actions, owners, and deadlines.

Symptom-driven playbooks

15. Testing production behavior

Production controls deserve automated tests and failure drills because their mistakes often appear only during deployment or outage.

• Test endpoint exposure and authorization, including anonymous denial of sensitive endpoints.
• Test custom health indicators for timeout, exception, degraded state, and redacted details.
• Test readiness transitions during startup and graceful shutdown.
• Use a test registry to assert meter name, bounded tags, count, and timing behavior.
• Capture logs to verify event level, required fields, correlation, and secret redaction.
• Verify trace propagation through HTTP clients, executors, async work, and message boundaries.
• Run the application with container limits and exercise startup, load, termination, and restart.
• Simulate slow, unavailable, and partially failing dependencies with bounded timeouts.
• Validate dashboards and alert queries against synthetic incidents or replayed data.

@Test
void recordsABoundedRejectionReason() {
    SimpleMeterRegistry registry = new SimpleMeterRegistry();
    CheckoutMetrics metrics = new CheckoutMetrics(registry);

    metrics.recordRejected("unexpected-provider-text-123");

    Counter counter = registry.get("checkout.rejected")
        .tag("reason", "OTHER")
        .counter();
    assertThat(counter.count()).isEqualTo(1.0);
}

16. Security and performance implications

17. Common mistakes and corrections

18. Practice exercises

1. Design Actuator exposure for a public API. State which endpoints are public, which are private, which are disabled, and how each layer is protected.
2. A Redis outage makes liveness fail and Kubernetes restarts 100 pods. Explain the feedback loop and redesign liveness, readiness, and dependency alerting.
3. Define availability and latency SLIs for checkout. Specify eligible events, success, latency boundary, labels, and how retries count.
4. Create Micrometer meters for a file import. Include completion outcome, item count, duration, and active imports without unbounded tags.
5. Trace an HTTP request that enters Spring MVC, calls an @Async method, and invokes a payment service. Identify every context propagation boundary.
6. Write a termination timeline for an API plus Kafka consumer. Fit readiness, listener stop, in-flight work, telemetry flush, and forced termination into one budget.
7. Diagnose rising latency with low CPU, a full Hikari pool, and many threads waiting for connections. List evidence that separates slow SQL, lock waits, and remote calls in transactions.
8. Plan collection of a heap dump from a production container. Include disk capacity, security, expected pause, transfer, analysis, retention, and deletion.

19. Interview questions and answers

What is the difference between monitoring and observability?

Monitoring checks known conditions through predefined signals and alerts. Observability is the ability to infer internal state from outputs, including investigating failures that were not predicted. Good systems need both: clear known alerts and enough correlated telemetry for new questions.

What does Spring Boot Actuator provide?

It provides production endpoints, health contributor infrastructure, Micrometer integration, application availability state, auditing and management support, and auto-configuration around runtime features. Endpoint enablement, exposure, and authorization remain explicit operational decisions.

How do liveness and readiness differ?

Liveness means the process cannot recover and should be restarted. Readiness means the instance should or should not receive new traffic. A dependency outage normally affects capability or readiness, not liveness, because restarting the process does not repair the dependency.

Why should liveness exclude external dependencies?

A shared dependency failure would cause every application instance to restart simultaneously. That removes otherwise responsive capacity, increases startup and dependency load, and can create a restart loop. Liveness should detect internal unrecoverable process states.

What is Micrometer?

Micrometer is a facade for application metrics and an observation API used by Spring Boot for metrics and tracing integration. Code uses common meter types and a registry publishes them to a chosen backend.

Why is metric tag cardinality important?

A unique tag-value combination creates a distinct time series. Unbounded values such as request IDs or customer IDs can exhaust memory, increase monitoring cost, and make queries unusable. Metrics need stable bounded dimensions.

How are logs, metrics, and traces used together?

Metrics detect that a population is unhealthy. Traces localize latency and errors for sampled operations across components. Correlated structured logs explain detailed decisions and events. Time, service version, trace IDs, and stable operation names connect them.

What is an Observation in Micrometer?

It is a named lifecycle around an operation with context and key values. Observation handlers can produce metrics, spans, context propagation, and other effects. Low-cardinality values may feed metrics and traces; high-cardinality values should remain trace-only.

How do you investigate a Spring application with high latency and low CPU?

Check latency by route and dependency, thread dumps, database pool pending and acquisition time, executor queues, database locks and sessions, downstream spans, timeouts, and connection state. Low CPU suggests waiting or contention more than computation.

Why can a full connection pool be a symptom rather than a root cause?

Connections stay active because transactions or queries take too long. The cause may be slow SQL, lock waits, network calls inside a transaction, database saturation, or leaked resources. Increasing pool size can only increase database concurrency and worsen the condition.

What does graceful shutdown require besides enabling a property?

It requires traffic draining, aligned platform and application timeouts, stopping every work admission source, bounded completion, retry-safe interrupted work, correct resource close order, and deployment tests using real termination signals.

When would you take a thread dump, heap dump, or JFR recording?

Use repeated thread dumps for blocking, deadlock, and current stack evidence. Use heap histograms and dumps for retained-object or leak analysis. Use JFR for time-based CPU, allocation, GC, lock, and I/O evidence. Heap dumps have the greatest storage and data-sensitivity risk.

How should secrets reach a Spring Boot application?

Prefer a secret manager or platform-mounted secret with workload identity, least privilege, encryption, rotation, and audited access. Import or bind it at runtime, prevent it from entering images and logs, and define how rotation reaches live instances.

What makes an alert actionable?

It identifies real or imminent user impact, has an owner and urgency, contains evidence and useful scope, links to a dashboard and runbook, and offers a safe first action. If no timely human action is required, it is usually a ticket or dashboard signal rather than a page.

20. Concise cheat sheet

21. Official references

• Spring Boot production-ready features
• Spring Boot Actuator endpoints and health
• Spring Boot metrics
• Spring Boot observability and context propagation
• Spring Boot tracing
• Spring Boot logging and structured logging
• Spring Boot externalized configuration
• Spring Boot graceful shutdown
• Micrometer reference documentation
• Micrometer Tracing reference
• Kubernetes liveness, readiness, and startup probes
• Docker build best practices
• Java troubleshooting guide