Tail Sampling: New Policy For Low-Frequency Attributes

Hey everyone, let's dive into a cool idea for improving how we handle tail sampling in OpenTelemetry. The goal is to make it easier to catch those rare but important events, especially when dealing with heterogeneous trace data. The current policies can sometimes miss the mark, and we're looking for a smarter way to do things.

The Problem: Missing the Low-Frequency Signals

So, here's the deal, guys. When we're using tail sampling, we often set up policies to grab those interesting, low-frequency events. Think about it: you want to snag those traces with HTTP errors because they're worth a closer look. But, when you've got services with drastically different throughput – some spitting out tons of spans, and others barely whispering – these policies can get wonky. They might oversample the high-volume stuff and completely miss the low-volume, critical events. Right now, manually tagging or classifying each service is a real pain, and using rate-limiting is kinda reactive. You can't easily measure and sample based on the frequency of these events. This is where a new policy comes in.

Imagine a scenario. Let's say you've got a service that's rarely called, but when it is, it's super important to monitor. Existing policies might not catch those instances because they're drowned out by the noise of more frequent events. Or, consider specific HTTP routes or custom attributes. Currently, there's no easy way to ensure that you're capturing events that are, say, less than 1% of your total traffic. This gap in functionality can lead to blind spots in your observability setup, preventing you from identifying and resolving critical issues that are infrequent but impactful. The need for a dynamic, frequency-aware sampling mechanism is clear, especially in complex, distributed environments.

In essence, the existing tools don't let you easily target and sample events based on how often they occur. This is the core issue, and the proposed solution aims to address it head-on. It's about making sure you're not just looking at the loudest voices in the room but also paying attention to the whispers that might reveal important insights.

The Solution: Introducing the Frequency Policy

Here’s the bright idea: a new component/policy called the frequency policy. This policy would track how often a specific attribute pops up and then sample the trace if that attribute is under or over a certain percentage of the total instances of that attribute. Pretty neat, huh?

Let's break down a real-world example. Picture this: you want to focus on low-rate services. You'd configure the frequency policy to keep an eye on the service.name attribute. It would then match values that show up in less than, say, 10% of the total span volume. Now, for those services that pump out tons of spans, the policy wouldn't match. But if a span comes from a rare service – one that represents less than 10% of your total spans – boom, the policy kicks in, and that span gets sampled. The cool part? You can tweak which attribute the policy watches. So, you could also track http.route, http.request.header.<key>, or any custom attribute you've got. This flexibility is key.

By making the comparison threshold (over/under) configurable, this policy gets even more powerful. You could use it to sample high-frequency events, too. Plus, you could combine it with invert, drop, or the probability policy to build super complex sampling rules. The possibilities are pretty exciting.

Under the Hood: Implementation Details

To make this work efficiently, we'd use a probabilistic data structure. Think of it like a smart, memory-efficient way to keep tabs on these attributes. Specifically, we'd use a Count Min Sketch (CMS) with a FNV hash. The CMS is designed to track high-cardinality values with a known error rate. This means you get a good balance between accuracy and memory usage. The best part is that CMS has blazing-fast insert and lookup performance. This makes it perfect for the demands of this policy.

The configuration for the frequency policy would look something like this:

• attribute: The attribute you want to measure (e.g., service.name).
• threshold_min / threshold_max: The range for comparison (e.g., match if the attribute appears in less than 10% of spans).
• error_rate or confidence: This controls the accuracy of the CMS. A lower error rate means more accuracy, but it also means using more memory.

This frequency policy would keep its state in a similar way to the rate_limiting policy. This approach ensures that the policy is able to remember and react to the frequency of attributes over time.

Alternatives Considered: A Dedicated Processor

We also kicked around the idea of building this as a dedicated processor component. In this setup, the processor would evaluate the frequency of the attribute and then add a numeric attribute to the telemetry (span, resource, or context). This frequency value would then be passed on to the next processor. Using this processor before the tailsamplingprocessor would allow an OTTL policy in the sampler to evaluate this additional attribute and make sampling decisions.

However, compared to the preferred solution, a dedicated processor has some downsides:

• Higher cost: Creating and maintaining a dedicated processor is more work.
• Performance impact: Adding and potentially removing the attribute adds overhead.
• Limited use: It's mainly useful for sampling, not for other observability tasks.

So, while the processor approach is a valid alternative, the frequency policy offers a simpler, more efficient solution that fits better with the existing tail sampling framework. The policy-based approach is more focused and less prone to feature creep.

The Goal: Enhancing Observability

This frequency policy is all about giving us more control and insight into our trace data. It's about ensuring that we don't miss those critical, low-frequency events that could be indicators of major problems. By using this policy, we'd be able to:

• Identify and diagnose issues faster: By capturing rare events, we can quickly pinpoint the source of problems.
• Optimize resource allocation: Sampling only what's needed helps us save on storage and processing costs.
• Improve overall system health: By keeping an eye on the outliers, we can proactively address potential issues.

This feature is designed to seamlessly integrate into the existing OpenTelemetry collector-contrib ecosystem. It's about enhancing, not replacing, the current sampling capabilities.