Instrumenting Spring Boot Apps with OpenTelemetry

In modern microservices architectures, observability is key to maintaining performance and reliability. OpenTelemetry (OTel) has emerged as the standard for distributed logs, tracing and metrics collection. In this article, we’ll explore how to instrument Spring Boot applications and microservices using OpenTelemetry with three approaches:

• Zero-Code Instrumentation using an Agent
• Annotations for Low-Code Instrumentation
• Manual Instrumentation using the OTel API

We’ll also discuss the best practices for managing over-instrumentation and controlling the volume of telemetry data.

Zero-Code Instrumentation with the OpenTelemetry Java Agent

The simplest way to add observability to your Spring Boot application is by using the OpenTelemetry Java agent. This approach requires no code changes, just attach the agent at startup.

Steps to Use the OTel Agent

• Download the Agent
  Get the latest OpenTelemetry Java agent JAR.
• Configure Environment Variables (Optional)
  Set environment variables to control the exporter, sampling, and other settings. For example, to send traces to an OTLP endpoint:

export OTEL_EXPORTER_OTLP_ENDPOINT="http://otel-collector:4317"
export OTEL_RESOURCE_ATTRIBUTES="service.name=my-spring-boot-app"

• Run Your Application with the Agent
  Launch your Spring Boot application with the -javaagent flag:

java -javaagent:/path/to/opentelemetry-javaagent.jar \
     -jar my-spring-boot-app.jar

The agent will automatically instrument supported libraries (e.g., Spring MVC, JDBC, Kafka, etc.) without requiring code modifications.

• Pros:
  • Zero-code: No changes to application code.
  • Automatic: Provides out-of-the-box instrumentation for many popular libraries.

• Cons:
  • Limited Customization: Fine-grained control over spans or attributes might require manual instrumentation.

Low-Code Instrumentation with Annotations

Sometimes you need to instrument specific parts of your codebase or add custom context to traces. OpenTelemetry supports annotations—such as @WithSpan—to simplify this process.

Required Dependencies

To use the @WithSpan annotation in your Spring Boot application, include the following dependency. (Be sure to check for the latest version when adding to your project.)

Maven:

<dependency>
   <groupId>io.opentelemetry.instrumentation</groupId>
   <artifactId>opentelemetry-instrumentation-annotations</artifactId>
   <version>2.8.0</version>
</dependency>

Gradle:

implementation 'io.opentelemetry.instrumentation:opentelemetry-instrumentation-annotations:2.8.0'

Note: If you’re also using the Java agent for auto-instrumentation, you typically don’t need to include additional dependencies beyond the annotation support for your custom spans.

Using the @WithSpan Annotation

Annotating methods with @WithSpan automatically creates spans around method execution. Each time the application invokes the annotated method, it creates a span that denotes its duration and provides any thrown exceptions. By default, the span name will be method name, unless a name is provided as an argument to the annotation. If the return type of the method annotated by @WithSpan is one of the future- or promise-like types, then the span will not be ended until the future completes.

Example:

import io.opentelemetry.instrumentation.annotations.WithSpan;
import org.springframework.stereotype.Service;

@Service
public class OrderService {

    @WithSpan
    public void processOrder(String orderId) {
        // Business logic to process the order.
        // The span is automatically created around this method execution.
    }
}

Adding attributes to the span with @SpanAttribute

When a span is created for an annotated method, the values of the arguments to the method invocation can be automatically added as attributes to the created span. Simply annotate the method parameters with the @SpanAttribute annotation:

Example:

@WithSpan
    public void processOrder(@SpanAttribute("orderId") String orderId) {
      // Business logic to process the order.
      // The span is automatically created with attribute orderId around this method execution.
   }

Suppressing @WithSpan instrumentation

Suppressing @WithSpan is useful if you have code that is over-instrumented using @WithSpan and you want to suppress some of them without modifying the code.

System property: 

otel.instrumentation.opentelemetry-instrumentation-annotations.exclude-methods

Description: Suppress @WithSpan instrumentation for specific methods. Format is: 

my.package.MyClass1[method1,method2];my.package.MyClass2[method3]

Creating spans around methods with otel.instrumentation.methods.include

In cases where you are unable to modify the code, you can still configure the Java agent to capture spans around specific methods.

System property: 

otel.instrumentation.methods.include

Description: Add instrumentation for specific methods in lieu of @WithSpan. Format is:

my.package.MyClass1[method1,method2];my.package.MyClass2[method3]

If a method is overloaded (appears more than once on the same class with the same name but different parameters), all versions of the method will be instrumented.

Customizing Spans with Annotations

You can mix the low-code annotation approach with some manual operations if you need to add attributes or events. For example, combining @WithSpan with manual attribute setting:

import io.opentelemetry.instrumentation.annotations.WithSpan;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.api.GlobalOpenTelemetry;
import org.springframework.stereotype.Service;

@Service
public class PaymentService {

    @WithSpan
    public void processPayment(String paymentId) {
        var span = Span.current();
        try {
            // Add custom attribute to the span
            span.setAttribute("payment.id", paymentId);
            // Business logic for processing the payment.
        } catch (Exception e) {
            span.recordException(e);
            throw e;
        }
    }
}

Manual Instrumentation Using the OpenTelemetry API

For complete control over your telemetry data, manual instrumentation with the OpenTelemetry API is the way to go. This approach is useful when you need to instrument non-standard flows, handle complex error scenarios, or conditionally emit spans and events.

Setting Up a Tracer

Before creating spans manually, obtain a tracer instance:

import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.api.GlobalOpenTelemetry;

public class TracingConfig {
    public static final Tracer tracer = GlobalOpenTelemetry.getTracer("com.example.myapp");
}

Creating Spans Manually and Emitting Events

Wrap your code in spans to capture execution details. You can add events to record significant moments in the execution flow.

Example: Emitting Events in a Span

import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.context.Scope;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.common.AttributeKey;

public class InventoryService {

    public void updateInventory(String productId, int quantity) {
        // Start a span for the updateInventory method
        Span span = TracingConfig.tracer.spanBuilder("updateInventory").startSpan();
        try (Scope scope = span.makeCurrent()) {
            // Emit an event indicating the start of inventory update
            span.addEvent("Inventory update started",
		 Attributes.of(AttributeKey.stringKey("product.id"), productId));
            
            // Business logic for updating the inventory
            // ...

            // Emit another event upon successful update
            span.addEvent("Inventory update completed successfully", 
                          Attributes.of(AttributeKey.longKey("quantity.change"), 
			quantity));
        } catch (Exception e) {
            // Emit an event for error handling with additional context
            span.addEvent("Inventory update error", 
                          Attributes.of(AttributeKey.stringKey("error.message"), 
			e.getMessage()));
            span.recordException(e);
            span.setStatus(StatusCode.ERROR, "Inventory update failed");
            throw e;
        } finally {
            span.end();  // End the span to complete the trace
        }
    }
}

In the above example:

Starting the Span: A span starts before business logic execution.

Emitting Events:

An event named “Inventory update started” is emitted at the beginning with an attribute.

A completion event is added with details about the quantity change.

In the event of an exception, an error event is emitted before recording the exception.

Scope Management: Using a try-with-resources block ensures the span’s context is correctly propagated.

Ending the Span: The span is ended in the finally block, ensuring proper closure even if an error occurs.

Adding Custom Metrics

Metrics provide insights into application performance and health. OpenTelemetry allows you to define and collect custom metrics.

Required Dependencies

Maven:

<dependency>
   <groupId>io.opentelemetry</groupId>
   <artifactId>opentelemetry-api-metrics</artifactId>
   <version>1.18.0</version>
</dependency>

Creating a Custom Metric

Metric types:

• A counter tracks the number of occurrences of an event.
• Gauge metric records values that change over time, like memory usage or queue size.
• A histogram tracks distributions of values, such as request durations.

Example:

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.metrics.DoubleHistogram;
import io.opentelemetry.api.metrics.LongCounter;
import io.opentelemetry.api.metrics.LongGauge;
import io.opentelemetry.api.metrics.Meter;
import io.opentelemetry.api.metrics.MeterProvider;
import java.util.concurrent.atomic.AtomicLong;

public class MetricsService {
    private static final Meter meter = GlobalOpenTelemetry.getMeter("my-app");
    // counter
    private static final LongCounter orderCounter = meter
            .counterBuilder("order.processed.count")
            .setDescription("Counts the number of processed orders")
            .setUnit("orders")
            .build();
    private static final AtomicLong queueSize = new AtomicLong(0);
    // gauge
    private static final LongGauge gauge = meter
            .gaugeBuilder("queue.size")
            .setDescription("Tracks the current size of the queue")
            .setUnit("items")
            .buildWithCallback(observable -> observable.observe(queueSize.get()));
    // histogram
    private static final DoubleHistogram requestLatency = meter
            .histogramBuilder("http.request.duration")
            .setDescription("Tracks the latency of HTTP requests")
            .setUnit("ms")
            .build();

    public void processOrder(String orderId) {
        orderCounter.add(1);
        queueSize.addAndGet(1);
        // Business logic for processing an order
        // ...
        requestLatency.record(duration);
    }
}

Managing Over-Instrumentation

While instrumentation is essential for observability, excessive instrumentation (over-instrumentation) can lead to performance degradation, increased noise in your traces, and higher storage costs.

Disable Specific Instrumentation via Environment Variables

You can disable instrumentation for a specific library by setting:

export OTEL_INSTRUMENTATION_<LIBRARY>_ENABLED=false

For example, to disable JDBC instrumentation:

export OTEL_INSTRUMENTATION_JDBC_ENABLED=false

Common OpenTelemetry Java Instrumentations

Disable All Instrumentation

To disable all automatic instrumentation via the Java agent:

export OTEL_INSTRUMENTATION_ENABLED=false

Use otel.instrumentation.exclude for Namespace-Level Exclusions

To disable all instrumentation for a specific package or namespace, use:

export OTEL_INSTRUMENTATION_EXCLUDE=com.example.myapp.*

This will prevent OpenTelemetry from instrumenting any class in the com.example.myapp package.

Sampling

Sampling allows you to control the amount of telemetry data collected. This is crucial in high-throughput systems where collecting every span could overwhelm your monitoring backend.

Configuring Sampling via Environment Variables

For the agent and many OTel SDKs, you can set a sampling rate using environment variables:

export OTEL_TRACES_SAMPLER=traceidratio
export OTEL_TRACES_SAMPLER_ARG=0.1  # Sample 10% of all trace

The OTEL_TRACES_SAMPLER environment variable controls the sampling strategy for tracing. The possible values are:

• always_on – Captures 100% of traces (default).
• always_off – Captures 0% of traces (disables tracing).
• traceidratio – Samples a percentage of traces based on a probability (controlled by OTEL_TRACES_SAMPLER_ARG, e.g., 0.25 for 25%).
• parentbased_always_on – Follows the parent trace’s sampling decision, defaulting to always_on if there’s no parent.
• parentbased_always_off – Follows the parent trace’s sampling decision, defaulting to always_off if there’s no parent.
• parentbased_traceidratio – Uses traceidratio sampling but respects the parent’s sampling decision.
• jaeger_remote – Uses Jaeger’s remote sampling strategy.
• xray – Uses AWS X-Ray’s sampling strategy.

Configuring Sampling with TailSampler Processor

The tail sampling processor samples traces based on a set of defined policies. All spans for a given trace MUST be received by the same collector instance for effective sampling decisions. Before performing sampling, spans will be grouped by trace_id.

processors:
  batch:
  tail_sampling:
    policies: [
      # This policy defines that traces that include spans that contain errors should be kept.
      {
        name: sample-error-traces,              # Name of the policy.
        type: status_code,                      # The type must match the type of policy to be used.
        status_code: { status_codes: [ERROR] }  # Only sample traces which have a span containing an error.
      },
      # This policy defines that traces that are over 200ms should be sampled.
      {
        name: sample-long-traces,               # Name of the policy.
        type: latency,                          # The type must match the type of policy to be used.
        latency: { threshold_ms: 200 },         # Only sample traces which are longer than 200ms in duration.
      },
    ]


service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlphttp/tempo, spanmetrics]

Conclusion

Instrumenting your Spring Boot microservices with OpenTelemetry can be as simple or as granular as your requirements demand. Whether you prefer the zero-code approach with an agent, the low-code approach using annotations (supported via dependencies like opentelemetry-instrumentation-annotations), or the manual approach using the OpenTelemetry API, you have the tools to gain deep insights into your application’s performance and behavior.

Remember:

• Start simple with the agent for quick insights.
• Use annotations to add context where needed without heavy lifting.
• Employ manual instrumentation when you need complete control over trace creation and want to emit events that highlight key moments in your code.
• Avoid over-instrumentation by focusing on critical code paths.
• Leverage sampling to balance observability with performance and cost.

Contact us to explore how OpenTelemetry can enhance your Spring Boot applications. Whether you need zero-code instrumentation, fine-tuned annotations, or full manual control, we can help you optimize observability and performance. Let’s build reliable and efficient microservices together!