Atlassian is replacing the internal components of its metrics platform with OpenTelemetry while keeping the existing StatsD interface available to application teams. The platform receives metrics from about 100,000 hosts across 14 regions and operates with a 99.95% service-level objective.

The migration is structured so application teams do not have to replace their existing instrumentation before the underlying telemetry platform changes. Atlassian kept the StatsD interface in place while rebuilding the collection, routing, aggregation, and forwarding layers behind it using OpenTelemetry Collector distributions.

The company had used gostatsd, the open-source StatsD implementation it maintains, for much of the past decade. Gostatsd handled metrics collection as a sidecar on individual hosts and aggregation further along the pipeline.

Atlassian said gostatsd’s UDP-based architecture did not support traces or logs, while more of its internal telemetry was moving towards OpenTelemetry. The company kept the existing application-facing interface and moved the platform first, avoiding a fleet-wide reinstrumentation project at the start of the migration.

OpenTelemetry became a graduated CNCF project in May this year, with metrics, traces, and logs established as core telemetry signals.

Keeping application interfaces unchanged

Atlassian divided the new metrics pipeline into four stages covering collection, ingest, aggregation, and forwarding. Each stage uses a purpose-built distribution of the OpenTelemetry Collector, allowing teams to replace components independently.

The collection layer accepts both StatsD and OpenTelemetry Protocol, or OTLP, data. Existing applications can therefore continue sending StatsD metrics over UDP while newer instrumentation sends OpenTelemetry metrics directly.

This lets existing services continue emitting StatsD metrics while Atlassian replaces the backend pipeline, with application reinstrumentation left for a later phase.

Atlassian also merged its metrics and tracing sidecars. Its tracing team had already used the OpenTelemetry Collector as the core of its tracing pipeline and for host metrics before the broader metrics migration began.

The company then consolidated StatsD metrics and tracing collection into the same Collector-based sidecar while retaining support for existing StatsD traffic alongside OTLP data.

Removing the separate gostatsd sidecar reduced CPU usage by about 3.9% per service on average across Atlassian’s most expensive Micros services. The company said this represented about a 30% reduction in sidecar cost across the fleet.

OpenTelemetry provides common conventions and context across metrics, traces, and logs, including resource information and mechanisms for correlating telemetry through trace context.

Reworking routing and aggregation at scale

The ingest layer required a different approach because metric aggregation is stateful. Datapoints belonging to the same time series must reach the same aggregator so that the platform can calculate the correct output.

Atlassian previously used an internal proxy called nomad, which assigned traffic to shards by hashing the service and environment. Because metric volumes were uneven across services, shards responsible for larger services could receive substantially more traffic than others.

Service-level routing could therefore concentrate more work on a single aggregator when a large service generated a high volume of metrics.

The company replaced this routing method with the OpenTelemetry Collector’s load-balancing exporter. Instead of routing by service, the exporter can hash using a stream ID that identifies an individual time series, allowing traffic from larger services to be spread across more aggregators while keeping each series on a consistent shard.

OpenTelemetry’s load-balancing exporter can derive a stream ID from the metric and its associated attributes and resource information. Datapoints from the same metric stream stay on one aggregator, while different streams from the same service can be distributed across multiple shards.

Atlassian said the change produced a more even distribution of CPU usage between shards. It also allowed the aggregation pool to scale down further during periods of lower activity.

Aggregation also reduces the amount of telemetry that reaches storage. Atlassian’s pipeline receives about 4.8 billion datapoints per minute but stores around 220 million after aggregation, a reduction of roughly 96%.

Most incoming datapoints are combined before being written downstream, reducing the amount of data that reaches storage.

Much of Atlassian’s metrics data uses delta temporality, where each datapoint represents a change since the previous measurement. The company said existing upstream components did not aggregate these deltas in the way its services required, so its engineers developed a custom delta aggregation processor and released it through Atlassian Labs.

With delta temporality, successive datapoints describe individual collection intervals rather than accumulating from a fixed starting point. Atlassian’s custom processor preserved the aggregation behaviour required by its existing metrics system.

The new aggregation tier now uses about half the CPU required by the previous system under the same traffic, according to Atlassian. The company attributed the reduction to removing gostatsd parsing from the aggregators, distributing load more evenly, and using OpenTelemetry Collector components.

The forwarding stage was also replaced with a stateless Collector distribution called metrics-gateway. It sends telemetry to destinations including SignalFx and Amazon S3 while using OpenTelemetry Collector features for retries, queuing, and backpressure.

Atlassian built a separate OpenTelemetry extension for AWS Lambda workloads, where a conventional sidecar cannot run. The extension retains the same StatsD address and environment variables used by the previous implementation, avoiding application code changes during the infrastructure migration.

That approach lets Lambda workloads keep the same configuration and endpoint while Atlassian changes the telemetry implementation behind the interface.

Testing the new pipeline in production

Atlassian also found that benchmark results alone did not show the full runtime cost of the new pipeline. The engineering team said some component behaviour and resource use only became visible under production traffic, so it used continuous profiling during the migration to identify where optimisation work was needed.

The production environment exposed component costs and runtime behaviour that smaller tests and benchmarks did not reproduce.

Atlassian did not replace the metrics platform in one deployment. Engineers first tested the new components in development and staging environments and with less critical workloads before expanding the rollout in stages.

Production deployment then increased from about 1% to 10%, 50%, and finally full coverage. Atlassian said the old and new systems operated side by side during much of the migration, making operational parity important as the rollout expanded.

The platform supports monitoring across thousands of services and operates against a 99.95% service-level objective. Atlassian therefore introduced the replacement pipeline progressively rather than combining the backend migration with a fleet-wide application change.

Atlassian said gostatsd aggregators and its nomad routing service account for about 38% of CPU requests across its metrics clusters. Nomad alone accounts for around 13% of total resources, according to the company.

The migration has so far concentrated on replacing the infrastructure behind existing application interfaces. Atlassian’s next step is to move application-side instrumentation away from StatsD, DogStatsD, and other vendor or internal clients towards OpenTelemetry SDKs.

Application reinstrumentation therefore comes after the backend migration. Services can move to OpenTelemetry SDKs separately rather than making that change a prerequisite for replacing the infrastructure.

(Photo by Mohammad Rahmani)

See also: MongoDB Atlas Agent Engine takes AI agents out of the sandbox

Want to dive deeper into the tools and frameworks shaping modern development? Check out the AI & Big Data Expo, taking place in Amsterdam, California, and London. Explore cutting-edge sessions on machine learning, data pipelines, and next-gen AI applications. The event is part of TechEx and co-located with other leading technology events. Click here for more information.

Developer Tech News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.