How Rail moved from Datadog to OpenTelemetry and stopped sampling

Rail’s Datadog bill grew as its telemetry volumes grew. To keep it under control, the team had to sample aggressively in lower environments, which meant developers were debugging with part of the picture missing. With Randoli, they were saving 50-60% on their existing observability bill.

The problem: cost that scaled with data

With per-volume pricing, every new service and every extra log line pushed the bill up. Sampling was the lever the team had, so they used it, hardest in lower environments, where cost seemed least justified. The trade-off was visibility: the traces and logs a developer needed were often among the ones that had been dropped.

Moving to OpenTelemetry, without a big-bang cutover

Randoli convinced the team to adopt OpenTelemetry for instrumentation. Rather than switch overnight, they ran Randoli and Datadog side by side. That let them compare what each platform showed on the same workloads and move over when they were confident, with no gap in coverage.

Real-time logs, with nothing sampled away

Developers could see their telemetry without sampling, and the feature they liked most was real-time log monitoring. Instead of searching through partial data after something went wrong, they could watch logs as they came in, across every environment, with nothing dropped to keep the bill down.

Cost and visibility had been pulling in opposite directions. Here they stopped.

What’s next: Raiya for incident response

Rail was in the process of adopting Raiya, Randoli’s SRE Agent, to help respond to incidents.