Introduction to OpenTelemetry | Kubernetes Observability Series Part 1

This particular series is all about open telemetry, right? And because open telemetry is a topic that has been discussed for quite some time now and it is one of the most hot topics in the uh recent cubecon and the whole cloner ecosystem. So we thought I think it would be great to have a series where uh we help people understand what are the different use cases of open telemetry and how you can actually start implementing open telemetry into your existing infrastructure to get the benefits of vendor lock in and so on and so forth. So this is something that we are going to cover today. So what we're going to do is as this is the first webinar for open telemetry right uh we are going to go through all the code and the basic concepts that anyone needs to understand in order to implement open telemetry understand why it is uh why it is necessary why it came into being right and how does it actually solve the existing challenges in observability right so this is a sort of a basic overview of what we are going to cover but just to quickly start with this This is a glimpse of open telemetry adoption and the massive increase uh that we have been seeing in the cloud native ecosystem for a while now. So this is these both statistics are from a recent CNCF report and you can see uh the project started in 2019. It joined the CNCF ecosystem in 2019 and throughout these years we have seen numerous number of companies, numerous number of contributors from across the globe contributing to this project and you can see this particular graph. These are basically the contributions done by the companies right so you can see the number of contributions that have been increased throughout these years and Rajit I think you also went to CubeCon this time so what are the kind of discussions that you saw people were having related to open telemetry adoption uh did you saw an increase in the discussion or what was happening there uh yeah open telemetry was one of the uh more popular I guess topics uh there was a lot of buzz about it people asking about Open telemetry people trying to learn more about open telemetry uh and this time at CubeCon um you know generally you find they have the CNCF project both uh and that's where all the CNCF projects are but open telemetry had its own both a really large both so that tells you the sort of like the interest and the adoption what's really um interesting about open telemetry is that it has a very vibrant ecosystem like you mentioned uh lots of contributors not just from observability vendors but also from other ecosystem um contributors as well right like AWS etc. So it's it's really good to see the kind of like the the interest and and more importantly the contributions from sort of across the spectrum. Um and you have um interest from different industry verticals as well like you know people from telecom talking about it, people from you know uh automotive interested talking about it. I'm looking at it as a very vibrant healthy community uh that's continuing to grow. So I am not surprised to see the adoption and you know being probably the most popular CNCF project I guess at this point. Yeah. So Rajit I think you can start us off by discussing the why part right why does open telemetry exist and what are the current observability challenges due to which open telemetry later was born. Uh yeah, I think that's a great way to start. Um you know why this kind of explains why open telemetry uh came into place and then you know like why was it created in the first place and then why is it so popular as the complexity of the uh applications um started to grow like we move from you know monoliths to microservices u so did the observability challenges that grew alongside it. If you look at the observability market that has been growing at a steady pace and uh various uh you know vendors uh came into the picture to solve these observability challenges. Uh but everybody had a a slightly different approach. Um to get more deeper visibility um people had to instrument their code. So there was proprietary SDKs, different libraries, different tools. Uh so what that also meant was you locked into a particular vendor uh because you know you you were using the SDKs uh you know you were using um you know the data formats were proprietary etc. So there's no interoperability uh kind of very locked into uh you know whatever that vendor provided uh and and then you know alongside that there are other operational challenges that that came about as well right like the setups were complicated uh you know I mean there was kubernetes there was you know people are moving to the cloud you know the microservices all kinds of you know things started to happen as well and and you know in that particular kind of environment as things the complexity group, how do you optimize all of these things? So, you know, there was uh several other attempts as well. There was open tracing before, you know, open telemetry came into being. Uh so, people were trying to figure out this challenge of how do we have a um sort of like a vendor neutral open standard when it comes to telemetry and and and that's how open telemetry started. uh these challenges were the sort of like the the reasons behind uh the existence of open telemetry and I think because these are so fundamental challenges that's why I believe that the adoption of open telemetry has uh thrived in these recent years because it solves these fundamental challenges for a lot of organizations right yes I mean it's it's not a um you know I mean it's not a perfect standard like any any other standard out there but it's certainly a step in the right traction and it's uh you know by the number of uh like the the number of vendors that are participating, the number of ecosystem partners that are participating and then the wide adoption is proving that um you know people are willing to work with it and continuing to improve and evolve it and it's also an extensible sort of ecosystem where um you know people are looking at using open telemetry in some other interesting use cases as well. It's, you know, today's probably not the day to talk about those things, but um it's fair to say that it's a growing vibrant ecosystem and um you know, people should be paying attention to it. Yeah. Amazing. Yeah. Thank you so much for sharing that Rajat. So let's talk about the solution now, right? We we have been mentioning about open telemetry and why it uh exists now. So what exactly is open telemetry? It is an open-source observability framework and it's important to understand that it's a framework. It's not an observability tool or an observability UI because I feel a lot of people confuse uh with that. It's a framework which gives us the rules which gives us the specification on how the telemetry data is collected and exported. Right? So it standardizes how the telemetry data logs metrics and traces which we'll talk about uh later as well is first collected and then exported to different backends. Right. um open telemetry the framework it works with any backend which supports open telemetry right any supported back end for example Jerger Prometheus or if you are using let's say a custom solution for example randoli or any other solution out there if that supports if it is open telemetry native you'll be able to view all of your telemetry data uh in in that particular solution right um it supports multiple programming languages and multiple environments you can run this on your kubernetes infrastructure We'll talk more about that as well. And it supports multiple programming languages so that you can instrument your code in any language of your choice based on the application that you're using and it should be good to go. Right. And lastly, because we have been talking about the adoption, it's a CNCF incubating project and has quickly become the de facto uh standard of observability. Just like uh Kubernetes became the de facto at one point for containerized applications. I think open telemetry is heading that way for observability. Right. So all in all this is what open telemetry is. And just to summarize that a bit the main goal that open telemetry is trying to solve is how you can enable easy instrumentation of your application or system. And regardless of the observability vendor that you're using, regardless of the programming language that you have used in your application, the infrastructure that you're using or the runtime environment which you're using, right? So this is I hope this gives you a good gist of what open telemetry is all about and the problem that it's trying to solve. So if we go into the core concepts right uh that you need to know in order to understand how open telemetry is working underneath the first concept is basically the signals and signals are essentially the telemetry data uh which your applications which you are exporting via your applications and then you are viewing using the observability backends. Uh the current currently open telemetry supports traces, metrics and logs. That means you can easily instrument traces, metrics and logs from your application. And through the open telemetry collector which we'll talk about a little bit later as well. You can view these into your custom back end or any observability backend that you wish. The signals which are under development right now are the events and the profiles. Right? So these are under development and the community is working on uh including these signals and trying to get these running in open telemetry too. But this is uh this is something that this is the first concept that you need to understand in order to see how open telemetry is working in the back end. But Raj do you want to take from here and tell everyone about what do we mean by distributed tracing? Because if you search on Google about open telemetry distributed tracing is something that will come up right. So I think these two are very much related. So if we can take this topic ahead. Uh yeah for sure. Um I think today we're going to focus a little bit more on the tracing side. Uh I think most people understand metrics, most people understand logs. Uh distributed tracing. Um you know it started to get more popular um alongside microservices. Uh you took one monolith and you broke it into multiple microservices. And then along with that came the challenge how are I going to debug this right? uh because these microservices are you know completely different runtimes and sometimes written in different languages as well. Uh but how do you you know uh trace a particular transaction across the microservices, right? Um I mean what I have here is a a fictitious order processing system. You know maybe from the front end someone is trying to create an order, you know, they click um purchase on the or or buy on the on the you know on the the shopping cart application. It hits an order um endpoint. Uh that that could be modeled as its own microser. But now in order to fulfill that order, he may have to look up certain information for that uh for those items. So it goes talks to a product catalog service. It needs to arrange for for shipping. Uh so it needs to talk to like a shipping service. Uh maybe need to coordinate with an inventory service. Maybe need to coordinate with a payment gateway which I haven't even put anything over here. Uh so you know you need to um to just to fulfill that order where you get a single perhaps a single endpoint to create that order multiple operations has to happen in the back end and any one of these operations can fail any one of these operations can slow down any one of these operations can have sort of you know subtle issues and when you're debugging um performance issues or you're debugging an error now you need to trace that first call, the the order that got placed in across all of these microservices. And that's challenging because if you don't have a um um a way to co correlate your logs, your metrics and traces, then it's going to be very difficult to from a debugging perspective, right? You're going to look at hours and hours of logs um and you know look at different metrics, try to correlate them. That's going to be very difficult. So distributed tracing uh was a way to trace a particular context propagating through these microservices and and trying to figure out how how much each of these uh stages um you know took to complete right like you know what's the processing time look like you know what are some of the additional data uh you know that we can look at from each of these steps um you know was there any other error was there any errors in in this process was there any other complexity ities that are alongside that how do we capture all of that right so that requires some sort of coordination and some sort of uh what I would call context propagation and that's what the distributed tracing um is trying to trying to solve right and with uh open telemetry we talked about that being a signal uh this context propagation um is is a key sort of concept what it what it allows is to um trace exactly what I talked about, right? You know, the a transaction uh going through order service to the product catalog service, going to shipping service, going to uh the inventory service which is actually completely decoupled as well. The other ones uh in that diagram was synchronous but going to the inventory service was a Kafka message. So how do you really debug all of this together, right? And that's what this visible tracing is about. So in the context propagation case what what's happening here is here you can see a simplified diagram where you know from front end it goes to a another service and from there onwards to the checkout service so they're trying to capture all of those steps and preserve that context uh for that particular transaction right and these are modeled as if you go to the next slide I think it's easier to explain um these are modeled as spans u so each of those steps within a particular process or a service. It could be a lambda function for all you know but all of those steps are you can think of it as a a series of uh operations u that are modeled as span right and and then now when you move from one service to another so in this particular case let's say from order service to the shipping service not only you have to capture that those spans but you have to propagate that context for that particular transaction to the next um service in the call chain which is the in in this case, you know, the the shipping service or um this one is an HTTP request, but let's say in the case of um you know, talking to the inventory service, in this case, it's an it's a it's a Kafka message. So if you have a if you figure out a way to uh propagate this context uh across uh different communication mediums in a standard way then you can continue to add you know uh using that context you can continue to you know record those different stages right within a service uh so that you when it comes to debugging it's easier to see you know what are the different steps it took uh to fulfill that order request. um you know what are some of the attributes that that were attached to this. So, so you can get more insight into the the the process you know um and and then um tie them all together using using that context. So you can see a entire call chain end to end regardless of how that process was executed or orchestrated right could be HTTP could be you know Kafka uh could be some sort of a JMS provider um you should be able to tie in all together. So that's that whole uh context propagation I guess concept of context propagation where you are able to capture um certain information that allows you to tie in various different execution flows that are all related to a particular uh business context like a like a transaction. It could be like a like an order. It could be um um some sort of a financial transaction. It could be uh like a payment action etc. it it helps you to track a process as it moves through the system because we are going to show this in the demo as well today. So you'll I think have a better idea of uh the concept of context propagation. But just to iterate back on this particular diagram right. So here you can see that when order service was hit you can see the entire flow of the request and the why behind this is because open telemetry supports contract propagation. each and every service which is getting hit. Even if you hit the product catalog service, you'll be able to see all the related service that were hit which are connected to the product catalog service. Why? Because each and every request from each different services has that context with that and it travels with the request. At the end of the day, you'll be able to track the request from end to end, right? And you'll have a better picture when we'll uh show you all the demo that we have today at the end of the day. So you'll have a better idea of how this thing looks in real life. So these are the two fundamental concepts that you need to know in order to like understand how open telemetry is working underneath it. One is signals that we covered and other is context propagation. Let's move on to the how part. Right? So we covered the why, we covered the what. Now let's cover how does open telemetry work and how does it do the things that it does. Right? So these are the main components, the key components and the building blocks on which open telemetry is built and this is how it enables us to be vendor neutral and it helps us to standardize the export and the collection of telemetry data. Uh the number one component is the open telemetry specification which basically gives you all the rules the data models and it sort of has explained how the telemetry data would be collected and would be exported to different kinds of back end. So this is the hotel spec. The next thing is the language specific API and NCDKs and these are essentially what you use to instrument your application code or this is what you use to instrument your applications on a Kubernetes infra which we are going to cover in the later section as well. So instead of you using a vendor specific library let us say if you want to export matrix to Prometheus you would have to use promp client right which is specific to prometheus but now what you can do is you can instrument using the open telemetry uh SDK export the matrix and you can send that matrix to any of the observity back end which supports open telemetry right so this is an advantage of using these language specific APIs and SDKs the next component in the list is OTLP P and this is what provides you that vendor neutrality right this is basically the protocol in which the telemetry data is collected from your applications and gets exported to the relevant back end that you have selected right uh it's available in gRPC and HTTP so you can configure in which uh protocol that you want to use this but if you ask the question of how does open telemetry provides that vendor neutrality and uh how it solved the challenge of different tools having different data formats. Uh this is how it does it. It's it uses OTLP a single protocol for exporting and collecting telemetry data. The next important component in the list is the open telemetry collector. Uh we are going to cover the collector a bit more because that's uh the key aspect of open telemetry. But open telemetry collector is basically the central service of uh open telemetry. You can consider this as a CPU which takes in the data from all the different data sources and handles the processing and exporting of that data to the relevant back end. We'll look into this a little bit later in the next slide. The lastly is the Kubernetes operator. Again, this is a fairly uh important component which enables you to use open telemetry in your existing Kubernetes infrastructure. Again, this is because this is so important, we are going to cover this in the next section. So let's dive into the open telemetry collector because this is important for you to understand. If you are new to the concepts of open telemetry, open telemetry collector is the vendor agnostic proxy or the service which actually receives processes and then exports the telemetry data. Right? So it has got three components. The number one is the receiver which you can see here. the receiver pulls uh the data or you can push the data from your application to the receiver. It will take the data. It will then send it to some processes. You can set up some uh some built-in processes. We're going to not talk about the processes in this. You can definitely check out the documentation to know more because open telemetry also helps you to process the data to clean the data so that you are only sending what's relevant to the observability back end and at the end of the day that will enhance uh your telemetry data and your it will enhance your visualizations at the end of the day right uh after the processes it then sends it to the exporters which will export the data to your selected back end. So I think one of the things to highlight here is that um the open telemetry collector and or the open telemetry like the pipeline it's a very um key component and a very powerful component because this is where users can take ownership of their telemetry data take actions on you know how to send what to send what information to prioritize etc because you know I mean we we all talk about the cost of observability these days um it's pretty massive and one of the factors that increases the cost of observability is all the data being sent. Um and sometimes the the the customers or the end users don't have control over that pipeline of how the data is in ingested and and exported and open telemetry can give you the ability to actually um before you push it to an observability vendor to take that data curate it and in some cases keep the data uh for your own usage as well and then send it to the the observability vendor. hotel collector, right? Because you can also chain these collectors as well. So, this this in my opinion um hands the power back to the to the customers to the to the end users. Uh and there's a lot of things you can do with these uh these processes, right? I mean this is not the uh webinar to kind of go deep into it but things like uh sampling um you know whether it is um some random sampling or you can do tail sampling head sampling these kind of different sampling strategies enriching data uh where you can grab data from multiple uh sources to enrich uh the data that you collect because you know that can help you improve the quality of the telemetry data uh and get better deeper insights. So you can do a lot more of this closer to the source where you collect uh than taking all of that data and trying to process it later on which can increase the cost and the complexity. Right? So this hotel like if there's one area to focus on um it's it's it's here um you know there's a lot of power in this and we'll talk more about you know hacking your open telemetry pipelines in one of the future webinars. um understand these concepts and uh uh you know you can do a lot more to uh improve the quality of the insights you get and have take control more important to me is taking control of your own telemetry data. One take away from that is the open telemetry pipelines. How you configure that that uh really is essential a component and I think the other component is collector patterns which defines how you are deploying your collector in your infrastructure. Right? So these are the two aspects that you need to take care of if you are thinking of using open telemetry collector and use it to its full extent. Just to give all of you an idea, this is basically the hotel collector configuration that you write in order to configure your uh open telemetry collector. The first part is the receivers as I mentioned and in the receivers we have you can define the protocols. So we have the HTTP protocol and the gRPC protocol and you can see that the default port for the HTTP is 4318 and for the GRPC is 4317. So you can configure that. Uh then here we define the processes right and as Rajat mentioned there are a lot of different kinds of built-in processes that open telemetry supports. You have the batch processor where you can batch the telemetry data and send that in batches. Uh right you can decide the time for those batches. You can do uh there are various techniques for do sampling as well. Just to give you a good picture. This is how we enable that. This is how we define the processors. And then here you can see we are defining the exporters. Right. So the number one exporter here we are sending for example to Jerger. So we are sending traces to Jerger right and then we are sending the metrics to Prometheus and we have defined the uh addresses to both of these right. So let us say in the future right now you're using and prometheus right let us say in the future you wish to change your observability back end you don't have to change anything in your code you just change you just add your exporter here and open telemetry will take care uh of everything for you right so that's the kind of advantage that you get when you're using the open telemetry collector right and this is the third important part of the collector config which is the actual pipeline so as I mentioned all of these different components the receiver was the processors and the exporters work in a continuous manner as a pipeline. Here you're defining the pipeline, right? So the first collect the traces. Here we're defining what receivers to use, what processes to use and what exporters to use. Similarly for metrics and similarly we can do uh for for logs as well. I believe this is a good way for you to understand how you define open telemetry and what are the configurations that you need to define this. So if you bring everything together right uh I think this diagram sums up everything really well. So you have all of your data sources on one hand which is basically all your different services which are built using different programming languages because you want to use the advantages of different languages for different operations that you are doing right you will instrument your telemetry data using the APIs and SDKs you'll send the data to the hotel collector via uh you know via the OTLP protocol it will receive the data it will process the data you know it will patch batch processing, sampling, metadata, so on and so forth and at the end of the day it will export the data to the relevant back end which you configure in the OTL configuration file. Right? So this is how things are working and either you can use separate backends or you can use a custom backend which helps you to visualize everything at one single place. Right? So you have that kind of a flexibility if you have instrumented using the APIs and SDKs of open telemetry. Before we move on to the next section about using open telemetry in Kubernetes, right, I would want to touch about what are the different ways you can instrument using open telemetry, right? The number one is using manual instrumentation, right? Instrumentation is basically enabling your application to export the relevant telemetry data, right? Which is the first step uh in the entire pipeline. So the first one is manual instrumentation and here we are sort of defining the traces and the spans manually. Uh we are defining the data points manually in our code. We are defining what we want to see uh in terms of traces and spans in the observability back end of our choice. Right? The advantages of this is uh it gives you full control. Right? It gives you the full control of what kind of telemetry data do you want to expose from your applications. It gives you deep visibility because you can define custom traces, custom spans, custom metrics as well. The pros are you get full control. The con is in this case it takes a lot of time, right? You'll have to go through the entire application code. You'll have to figure out what are the data points that I want to set up, what kind of metrics uh I care about, right? So, it gives you the control. But there are the cons as well that it takes a lot of time. Next approach which uh a lot of people use and it's a great way to start is auto instrumentation right and it requires very minimal to no code changes what happens in auto instrumentation is that you use the predefined SDKs and the instrumentation libraries that open telemetry provide based on the language that you're using right for example in this particular case we are auto instrumenting components of NodeJS application which is express uh HTTP P, MongoDB and Mongus and these are basically the common instrumentation libraries that we're using from open telemetry and in the back end they will set what are the different data points that will be configured in the back end right so we don't have to define all of the different trace points all of the custom metrics we can but we don't have to just to get started it will do everything for us right so minimal to no code changes needed it's great for quick start it's ideal for common frameworks and common libraries for a particular language. But one thing to understand here is that auto instrumentation is different from the perspective of who you are. Right? If you are a developer, you'll do auto instrumentation using the libraries uh using the common libraries, right? You'll change to the code, you'll add these libraries and that's the way for you to do auto instrumentation of your application. But let us say if you're a DevOps engineer, if you don't want to touch the code at all, what are the options for you to do auto instrumentation? So in that particular case, you'll use the open telemetry operator, right? And on your Kubernetes infrastructure, you'll set up the instrumentation custom resources and you'll have to configure the uh language specific annotations for your application. So in that case you're not touching the application code itself as a CIS administrator as a devops engineer but you're still instrumenting the telemetry data and you're still exporting that to the relevant observability back end. Right? So in my opinion this particular difference is really important for understand if you're a developer and if you're an operations uh if you're a devops or an S sur engineer like you said in a future webinar we'll we'll touch on the uh you know the auto instrumentation uh not from a framework library perspective but hotel operator perspective right like how you can leverage um you know ebpf or you know the java engine to kind of get get stuff going but uh libraries is a great place to start I think nodejs like you said has really good support. Uh if you're using Spring Boot, Spring Boot has amazing uh out of the box support to the point that it just by changing a few things in your pom file uh by adding the open telemetry um starter, it provides a lot of auto instrumentation as well uh and and also the ability to add uh custom spans and stuff like that. So there's a lot of options available uh for the popular um languages out there. Okay. Yep. Totally agree. And if you're a developer who's watching this, you can check out the open telemetry documentation. I'll maybe link it in the chat right now and see the supported languages and you can explore what are the different uh instrumentation libraries based on the language which you are using, right? So you can explore it through the documentation as well. Just to quickly touch upon the uh open telemetry operator as well, right? Because we want to touch that aspect as well. uh just to give you a good overview of how that would work if you are an uh if you're a DevOps and an S sur person. So how does that work is uh you'll set up you'll install the open telemetry kubernetes operator. You can use helm to install that or you can check out the repository to know more about how you can install this. But the most popular option is helm right. So you install the operator, you set up the custom resource definition for the hotel collector. Basically you configure the receivers, the exporters, the processors and the entire pipeline of your collector. And then when it comes to auto instrumentation, right, you'll use the instrumentation custom resource definition to instrument your particular app. And for your existing deployments you'll add the annotations. So just to give you all a clear picture of how would that look like here you can see this is an example of the open telemetry collector custom resource that we have created and here you can see we are defining the entire open telemetry config right so right from the receivers the processes and the pipelines basically the same thing but this is basically a custom resource that you're defining which is via the open telemetry operator then what we do is we define the instrument instrumentation custom resource which is specific to the language that you're using. Again uh I would recommend to check uh to check out the documentation because every language specific instrumentation has different configuration options that you can explore using the documentation. This is specific to NodeJS. So you can see that we are defining the common instrumentation libraries that you want to use. So in the first case we were defining the libraries right now we are just giving the names and the values here. We want to configure HTTP, Express, MongoDB, Mongus. These are the common ones. You can uh explore a bit more. Similarly, this would be for Python. Similarly, this would be for any other languages that you're using, right? And lastly, for your existing deployment, you'll have to add this particular annotation here of uh inject NodeJS. is specific to NodeJS but again there are a lot of configuration options and there are a lot of options that you'll get on how you can add these annotations and what are the different options which are available I would recommend you to check out the documentation I hope you got an overview of how you'll use the open telemetry collector this is like the most popular way of using open telemetry in your Kubernetes infrastructure yeah right one thing to add quickly here and it's a mistake that even I I did add at the when I was trying these things out, you have to choose one approach over the other. If you end up doing trying to do both, uh you will start to get subtle bugs uh because you have the auto instrumentation also working on top of the libraries that you have and that can cause some subtle bugs. So be careful just use one approach over the other and that's something uh I learned the hard way. Cool. Uh who's ready for a demo? What could go wrong? So what we are doing is we have a simple m application which is u mongodv express react and nodejs. Uh we have a simple front end which is using react which is you know running on the browser. And then in the back end this is what is going on. Whenever you click something on the front end I'll show you the application right now. Whenever you click something on the front end, it will make an API call to the backend service to the back end ingress and the back end will make the call to the MongoDB database which is running on MongoDB Atlas which is the cloud version of MongoDB. Right? And it will save the data there. While you're doing this because the NodeJS code is instrumented. I'm going to show you how I have done that. The back end will export the matrix and the traces to the open telemetry collector. Right? And there are two approaches now. One approach is you can send your traces and metrics to separate backends. In this particular case, you can see that the Jerger traces the traces are be sent to Jerger, right? So you can configure that or you can configure Prometheus to send metrics to Prometheus and visualize them using Grafana, right? So this is basically the open source way of doing things. But today what we are going to do is the setup is the same but at the end the open telemetry collector is sending the metrics and the traces to randoli agent and we'll be able to view the metrics and the traces on the on a unified view on the randoli dashboard itself. Right? We're going to focus on traces today uh just to show you all how does that look because I would want to explain the concept of context propagation as well while we doing that. So I'll just quickly show you what is happening in the code itself. So this is how you'll define your backend uh tracing file. First we're defining the node SDK as I mentioned. So because this is a back end for node. Similarly you'll do this for Python. You'll you use the Python SDK and similarly you'll do this for Golang as well. And then we are getting the node instrumentation from there right node auto instrumentation. The two important components to look here are the OTLP trace exporter and the OTLP metrics exporter. Right? And this is how we are exporting the metrics and the traces via OTLP to our open telemetry collector which is deployed on our cluster. If you're using let us say a local open telemetry collector using docker, you'll configure that particular endpoint to send traces and to send the metrics to that particular collector. Cool. The last thing we are configuring here which is again important to understand is the trace context which we talked about right. So I'm going to show you how things are working here. So here we are configuring the open telemetry uh sorry the OTLP endpoint which is basically connected to our hotel collector deployed on the cluster and then we are configuring the trace exporter which will send the traces. Here we are configuring the metrics exporter which will send metrics every 15 minutes to the open telemetry collector. There are two configurations that uh can come into play. Either your application can push the metrics to the collector or your collector can pull the metrics as well. Right? Uh so these are the two ways that you can use. In production you would want your uh application to push the metrics to the collector because that's more optimal. But in a development setup you can configure Prometheus or you can configure the hotel collector to pull the metrics as well. Right? In the interval that you configure. Uh here we are defining the trace context propagator. I'll not go into the details of this but as Rajit explained we basically uh with each of the trace spans we send the trace ID so that you can see end to end entire request flow. When we'll show the dashboard you'll have a much clearer picture of how this looks in real world. But this is how the instrumentation is looking like right. So we are using the uh most common libraries which is the express HTTP MongoDB. This will enable us to trace the different data points based on these libraries that we have selected. Right? The express uh HTTP and MongoDB right then we are starting the SDK and this is basically the configuration for the back end. Similarly what we are doing is for the front end right? So front end is based on React. So if you want to use open telemetry SDK to trace the front-end request, you use the web tracer provider which is from the SDK trace web which will help you to capture the traces from the UI itself from the front end. And then we are using some more libraries for example documentation load instrument which will track the load on your front end right fetch instrumentation which will track the API calls that your front end uh makes with the back end. Right? So this is something that you may want to configure as well. And the rest of the configuration remains the same. For the auto instrumentation on the front end side, we are using this get web auto instrumentation right here. And if I go and if I show you what is happening here is so it's capturing the HTTP request. It's show it's capturing the load as well and it's capturing the user interaction. User interaction means uh when you click on the front end itself. So this is enough for the code. I'll show you how things are working in the UI itself. So this is what the front end looks like. So if you click here and you know it increases the increases the counter in the back end it's updating the MongoDB database. So you can see the data is getting saved in MongoDB. So you have the database right but let's see how does the traces looks like. Okay. So this is the randoli UI which is we are using as a custom back end. This is open telemetry native. So you can instrumentate your code using open telemetry and you don't have to worry about uh touching that. Again if we head over to the observability snapshot which will generate you the observability data on demand at that particular time for your entire application for that particular deployment. Basically if we hit fetch observability data and you can see firstly we can see the metrics of that particular deployment. We can see the memory being used the CPU millores being used. uh these are the span metrics. Now I'm not going to go into the depth of what are span metrics but the important point to note here is that this is the section we care about right now which is the distributed tracing part right so here you can see that we've got these uh huge number of traces because we've been testing this out uh for for a few days now but here you can see the individual traces basically which are getting tracked and if I open one of them you'll be able to see the entire trace view from the front end till the database call. Right? So you can see that when we hit when we clicked on the front end when we incremented the counter this particular function was called in our front end which is the increment counter which made a post request via engineext to our back end which is the counter item and then we had some middlewares uh in our code which we had configured then the request went to the MongoDB it updated MongoDB and that's how things worked in the back end. So because we configured context propagation on the front end and on the back end side that's how you are able to view the entire request flow end to end right because with each span with each trace that particular context is transferring through each request right I hope this gives you a clearer picture of how things are working like overall you could see from the front end to the back end so for example let's say um um you know Um this could be like an order system or anything like you your users complain saying uh hey it you know I I'm seeing some issues it's um either you know I'm getting errors or uh it's taking too low too long to load right now we want to know where is the slowdown happening right and this gives you from you see the front end then you see the uh so in the front end we're talking about what's running on your browser it hits your engine X from engine X it goes to you know in this case only a simple backend service but it could be multiple microservices in that chain. So at a glance you can see the entire call chain. uh if you yes you see the like so for example let's say to increase the counter it takes argument let's say 10 milliseconds right it takes 10 milliseconds to update the counters now what you want to find out is how long did it take for the backend to process it right um and you know I mean you can also see the latency between your browser into engine X into the backend service and then inside the backend service there are a couple of things that happen and you can see the different spans in In a more complex scenario, maybe you're you're doing some other queries talking to other microservices. So you can get a breakdown of those individual steps to identify where the time is coming from. So in this particular case, you can even see the the call around the database. So it breaks down all those individual steps. It gives you a really good understanding of uh where the problem might be. So you know where to look for when it comes to optimizing. Assume you had an error in this call chain, right? uh then the trace will hopefully capture some of that information. So now you can understand where are the arrays coming from very quickly and then get some um uh details around it like for one of the things that I don't think we have enabled it but one of the things that what you can easily do is click on the logs I don't think we've enabled it here uh so that you can quickly go to the logs for these traces right so you can do this with other means as well the the the real star here is like open telemetry because there are ways to annotate it like for example you can instrument your logs with the same trace ID. Um you know in NodeJS you can do it in in in like in log forj you can add it as a field. So now what you can do is you can correlate your trace with a particular log entry. So let's say you found an error here. Now you can look at the logs to get further information right or you know you if you have um you know certain metrics correlated using exam plus you can then go and look at those to see like you know get a breakdown of like you know what may have caused these slowness right maybe click on one of these spans um around the database if you don't mind um sure yeah um show me the tags if you don't mind okay now it's hard for me to see what it is but if you can see it tells you the span type is database Right? It's a database call. It tells you the collection name if I'm not mistake. So it's a find one operation, right? It says the collection name. Going down a little bit more. In some cases, you may actually, for example, if you do SQL, you can sometimes even see the SQL query. So if you're an operations person like an SR, DevOps, you're trying to debug, you have a lot of information that you can look at and communicate to the developers. Or let's say you're debugging a production issue together, all looking at the same kind of data, right? uh and you can see exactly uh you know which you know database operations were done, how long it took, was there any errors etc. It gives you that like level of visibility that is otherwise going to take a lot more time if you try to collect all this information manually. Right? So this gives that great amount of visibility uh from you know from your browser all the way to the database and and back right. So the entire round trip you can instrument it. So this um I guess hopefully shows the sort of like the benefits of open telemetry where you can get this kind of visibility instrument your applications in a vendor neutral way. So assuming tomorrow uh you you switch to a different observability vendor. You shouldn't have to change your applications. You shouldn't have to change how you instrumented. This kind of information should be visible should be able to present it in the other observability vendors solution or you switch to some sort of open source solution like using Jerger or Prometheus you should be able to see it. So this gives the customers the the end users more power gives them more options without being vendor locked into a particular vendor system. Quickly touching back to the point of you know if you want to change your observability window let us say right it's really easy for you because you instrument your code once and then you don't touch that for for for a while right because you only have to change your configuration and point that particular exporter where you want to export your telemetry data so you instrument once and you can do this configuration again and again so you don't have to change this again and again and I think the same point that you mentioned Rajit that's how open telemetry is able to enable vendor neutrality right and because all of your telemetry data is following the same protocol which is OTLP that's how we are solving the problem of different formats right of that being followed by different tools right and if anybody's wondering like where is the vendor adding value in this chain is you know most vendors help uh by providing um you know hotel pipelines a standard hotel collector to kind of get going you know some customers you prefer to have their own hotel pipelines before. Uh so you can chain them together as well. If you want to take ownership of that uh and then where they really add value is in uh you know taking this data and processing it at scale and providing deeper insights into it right uh so that's where like so the vendors now have to uh in in my opinion we all work together I guess on the collection phase and making it neutral and then they'll have to sort of work hard on the u on the visualization and insights and other aspect aspects to provide value to the customer. So it's a it's a great I think that's how it should be. Uh this is how it can sort of you know like improve observability for everyone. I can totally agree with that. Right. Uh thank you so much for sharing that information, Raj.