Are you ready to start your journey to collecting telemetry data from your applications? Great observability starts with great instrumentation!
In this series, you will learn how to implement OpenTelemetry (OTel) and how to instrument your application to collect trace telemetry. You will learn how to leverage basic automated instrumentation tools and understand when to explore more advanced manual instrumentation for your application. By the end of this series, you will understand how telemetry flows from your application to the OpenTelemetry Collector and be ready to implement OpenTelemetry in your future projects. Everything discussed here is supported by a hands-on, self-paced workshop written by Paige Cruz.
In the previous article, we programmatically instrumented and configured an application that sent all telemetry data to a Jaeger instance for visual insight. In this article, we will take a closer look at how Jaeger can be used to help developers visually explore telemetry data.
It is assumed that you have set up OpenTelemetry and an example Python application project following the previous documentation, but if not, please go back and refer to that documentation as it is not covered here.
Jaeger Exploration
In the previous article, we installed Jaeger, but did not go into the background or details. Jaeger is an open source distributed tracing system and an incremental CNCF project. It supports multiple telemetry storage backends and provides a web-based frontend. As used in the example using the in-memory storage (default), it supports receiving telemetry in OTLP format by default.
We picked up where we left off in the previous article and created a Python application. Hello Hotel, It is running and the developer has programmed it in the trace instrument. The trace export has been set up and used. OTLPSpanExporter Pushes telemetry data in OTLP (OpenTelemetry Protocol) format.
Next, in our pod configuration we added: Jaeger All-in-One An instance designed for testing in a development environment. It contains a collector, query, and UI components, and I spun it up to explore. At that point, I opened a browser and saw the Jaeger UI. http://localhost:16686Gopher Detective appears as shown below.
We then access the application to generate telemetry data (tracking period) and perform several requests. http://localhost:8001/dogo terminal.
Search through traces
Back in the Jaeger UI, you can choose: Hello Hotel at service Please note the drop down menu and this evidence The landing page provides a search panel where you can filter your tracking by specific services, specific actions, or relevant tags (e.g. status). http.status-code=500), period, or a combination of the above:
Traces matching the search query above will populate: Scatter plot and table. The scatter plot is a quick way to visually identify unusual traces, and clicking on a bubble will take you directly to that trace waterfall. The table view is useful for sorting to find specific traces by period, span amount, or recency.
Since a trace can contain hundreds of services and thousands of spans, the default view is to collapse all span details to give you an overview of the request results along with quick statistics about the trace (duration and number of services). This view allows you to see which spans are taking up the most time compared to the overall request duration.
The search box here allows you to search within a trace for spans with the following properties: GET, 200Etc. Clicking on a target will open the matching spans as shown.
Let’s take a closer look at the visual options for exploring our trail.
Scatterplots and graphs
A scatter plot is a way to visually identify unusual traces and quickly review them by clicking on them. bubbleYou will be taken directly to that page. Tracking waterfall. The table view is useful for sorting to find specific traces. continue, The amount of span, or recent:
Click Tracking name Takes you to it Tracking waterfall By opening a page and checking two or more checkboxes, you can compare the selected traces against each other.
Once you have decided on an interesting range, click Next. name Opens more details like related ones. Tags and Process Information. This is your place Manually measured metadata It is a powerful way to examine system behavior.
Special visualization
Now you’ll notice that there’s a dropdown next to the Tracking search bar. Clicking on it will bring up a specific list of available visualizations, some of which we’ll explain in more detail below.
- Trace Graph
- Tracking range table
- Tracking Flame Graph
that Trace Graph The view shows spans grouped into node blocks, with the option to color the nodes uniquely. serviceBy hour Highlight the critical path or Self time Shows the longest span of time without waiting for a child.
that Tracking range table Show the table continue, work, and Service Name Span in tracking. Search options service or work You can amplify specific interactions. Span ID I take you Tracking Details View with the range highlighted:
no way Trace Flame Graph Views are another way to visualize your trace waterfall. When exploring a span, you can right-click to collapse unnecessary details, copy function names for use in other queries, or highlight similar spans within the trace.
Now let’s look at a more advanced use case where we want to compare multiple traces to see how they change over time.
Advanced Search
When a new bug is discovered or unexpected behavior occurs in a system, investigations typically attempt to answer the following questions:
- What has changed?
- How does this compare to the norm?
This is the place Tracking comparison Shine. Use this to compare two or more traces to quickly identify which spans exist or occur more frequently in one or the other. Below we start asking ourselves, why did one ask us? /doggo Why do some applications take 685ms while others only take 281ms?
Select two traces and click Compare Traces to see a color-coded visual graph representation modeling the differences in code.
- gray – Nodes of both trace A and trace B
- Red – Nodes that are only in trace A
- Green – Applies only to nodes in trace B
As we discovered, trace data provides both high-level and low-level insights into the relationships between services. It would be overwhelming to display all the information in one view, and this is where the ability to jump between Tracking comparison, span query, Individual properties About method calls Topology Map Makes tracking data flexible and powerful.
These examples use code from Python applications that you can explore in the provided hands-on workshops.
What’s next?
In this article, we’ve detailed how developers can use Jaeger to visually explore telemetry data. In the next article, we’ll dive deeper into how to manually instrument metadata from the tracing scope of your application.