Telemetry in distributed systems

For a distributed system like microservices, the telemetry can be generated by the applications running on a VM machine, baremetal machine, container, serverless function potentially in different parts of the world. This can be a challenge from the telemetry perspective - how to trace the flow of information for example. In your book, do you mention some best practices when it comes to these potential issues?

Distributed systems have gotten a lot of attention lately, and it sometimes feels like parts of the community consider distributed systems the only way to do anything. Any microservice or distributed system has a lot of component parts, each with their own highly local problems and constraints. This is why the tracing telemetry type has exploded in the last five or so years, as it's best suited to providing telemetry for an entire ecosystem rather than a specific system.

This is where OpenTelemetry (OTel) comes into play. OTel is a telemetry format, a Shipping Stage technology, that relies on a service that can consume OTel formatted data to provide goodness. Jaeger is the most prevalent open source system supporting OTel, but SaaS providers like Honeycomb, New Relic, and Datadog all can consume it as well. In fact, many newer, microservcice-from-the-start companies don't bother with centralized logging, or even metrics and rely entirely on tracing!

That said, tracing is a rather heavy service to provide, so you're not going to get datastores with a year plus of telemetry in it without DIYing it or paying stupendous amounts of money to a SaaS provider. When companies start feeling this pressure, they start looking at dedicated metrics systems, since those handle long timelines far better.

Tracing is designed from the start to handle high cardinality. In fact, tracing as we know it today grew out of frustration over existing metrics systems and how bad they were at handling cardinality. Metrics systems are great for telling you how the system as a whole is behaving, and don't have the cardinality room to answer the "how did this one user session perform" or "what happened to this account at 1932 last Thursday" kind of questions. Tracing does, and goes even further to loop in centralized logging concepts to give the troubleshooter everything that happened in that workflow, regardless of the services it passed through.

But what if you need to provide metrics and centralized logging for a distributed system?

An intentionally designed centralized logging system can begin to provide the "everything that happened in that workflow" concept, but won't give you that wonderful waterfall graphic that shows you the timeline of execution split out by service. You can do this by agreeing as a product to pass around correlation identifiers as part of execution. Each step in a workflow accepts identifiers and passes those identifiers on to the next service, where those identifiers are used as fields in logging. Some common identifiers:

account_id - the account doing the action

session_id - the browser-session doing the action

request_id - the specific request that triggered the action, this will correlate all actions in the workflow

execution_id - the specific execution of a service

These all allow you to bring up all the logging for a specific series of events. Agree to this as a product of services, and you can make an existing centralized logging system far more useful than it would be otherwise. Strongly recommended.

These identifiers would be great in a metrics system, but these IDs tend to be quite high cardinality. If your metrics system can handle it, perhaps you're using a newer time-series system backed by a columnar database, use them there too. For metrics systems, you mostly have to give up the idea of providing metrics for individual executions or accounts and focus on whole-system telemetry. Focus instead on providing a common schema for capturing the key concepts used by all of your microservices. Having a common schema allows you to get around the sheer flexibility of English that can wreak havoc in a time-series database. Consider the following attributes, they all capture the same thing but are phrased differently:

upload_size

file_size

size

size_upd

transfer_size

This is a classic example of asking five engineers to name the same thing and get five different answers. By having that agreed to schema you unify many of these concepts, which reduces cardinality burden on the database and makes it perform faster. Also, it enables engineers working on different services to have a better idea of how metrics for your service are shaped, which improves the maintainability of the overall system.