Observing Kafka at Scale with OpenTelemetry

[music] [music] All righty, we are live. Hello everyone. Welcome to this particular live webinar on observing Kafka at scale with open telemetry. My name is Konal. I'm working as a dev engineer at Randoli and I'll be your host for today and I'm joined by Rajat and Shriml from the Randoli team. Let's go over through a quick round of introductions. Rajat would like to go first. >> Yeah, sure. Um I'm Rajat. I'm the founder and CTO for Randoli. Very excited for for today's webinar. >> Shrimal. >> Hi. Hi everyone. So I'm stream like I've been working uh in Mandoli for like uh almost 3 years and like uh we are building like uh some exciting features here. So yeah very excited to like present what we are doing here today. >> Amazing. Yeah. So uh today we're going to talk about observing Kafka at scale and just a couple of housekeeping things before we start. Uh because we are live streaming on LinkedIn and YouTube. If you want to chat with us which I would definitely uh recommend that you do you can ask your questions. This is basically the link for the YouTube live stream because LinkedIn sometimes lags when you chat. So I wouldn't want you to have that kind of an experience. And feel free to ask all of your questions throughout the entire session. We'll definitely try our best to take your questions, your thoughts and your experiences after the end of this particular session. But with that being said, Rajat and Shrimal, uh, take it away. >> Sure. All right. Awesome. Um, Kunal, you want to um, sort of share the screen? Yeah. All right. So um you know um Kafka is a is a remarkable piece of software. Um um it's um not a stretch to say that a lot of teams depend on um Kafka um to be the backbone of of their production systems, right? It is arguably the the sort of the go-to choice for um event- driven architectures and generally considered the backbone of most production systems out there. And like I said, it's a it's a remarkable piece of software and um generally holds up really really well under adverse conditions, but it's the situations around it that makes it a little bit tricky to monitor. So um let's kind of go through a bit of a story u t um let's say you you know in this particular case um um you know customer deployed an application uh they made a they made a change u um and the deployment was done around you know 11:00 a.m. Uh what was different this time was uh due to a slight uh change in code uh there was some extra processing time uh that was added um in the code in the form of a a faulty retry. Uh but all was well you you know you you did the deployment and u um you know your team is monitoring Kafka, your team is monitoring the applications. No problem. Um by the afternoon uh those milliseconds have slowly caught up um you know um every every every message there was a couple milliseconds and then towards the evening when the evening rush hour came uh there was more messages that are now coming into the queues slowly um the processing has um you know um slowed down and uh by around uh you know 10 10 p.m. um the there's considerable amount of messages that are piling up in in in the topic now because the consumers are starting to slow down there is downstream effects. Um and it's you know it's it's affecting the rest of the system and by 2:00 a.m. you have a full-blown production incident. Even at that time if you look at it your brokers are all showing healthy green no problems. your applications are um looking okay. There wasn't a much movement in CPU. Perhaps a little increase in memory, but not a lot to look when you look at the uh when you observe the applications in the broker, you can't really say much, but your system has ground to a halt and um you know, you have a full-blown production outage. And that's the kind of um um stories that we want to talk about today. How can you identify scenarios like that? How can you um you know build guard rails uh in your monitoring observability to handle situations like this? All right, so let's get started. Um shall we kind of look at the agenda today? All right. So, we'll we'll start by um looking at the critical metrics we want to monitor for the broker, the consumer and the producer. Uh we'll talk a little bit about what the traditional monitoring looks like and compare and contrast that with the alternative approach and then um we attempting to do a real um failure scenario. a couple of failure scenarios. Um, you know, live demos are always fun. Um, let's hope the the demo gods are with us. All right, so let's dive into the um, you know, the the broker metrics, right? So, um, can we uh, let's roll forward. All right, good. Okay, so let's talk about the broker side metrics here. Um you know Primal you know you have been involved in um dealing with actual production systems that were very very heavily uh reliant on on Kafka right and um and often um you know there are some what I would call leading indicators of trouble right um so let's talk about offline partitions why is offline partitions important what is the effect on you know um uh the consumer producing applications and the broker itself. Let's start there. >> Yeah. Yeah. So uh so open functions like one of the like uh very critical you know like uh piece of metric that we need to keep an eye on uh like uh sometimes you know like uh like uh Kafka is a system you know like which allows parallelism through partitions. So like we can create topics and you know we can create multiple multiple partitions and uh that's how it distributes the load. Uh so like consumers can consume from you know like those partitions and that allows us to run multiple uh consume applications. Uh so uh like uh partitions has uh a config called uh replication factor. So like if you don't have replication factor configured properly let's say we create a partition and we with a few top uh few we create a topic with a few partitions and the replication fact is one that means we only keep one replica of that partition. Uh so whenever the broker which holds that partition goes down that partitions becomes offline that means uh producers are not able to produce messages and consumers are not able to you know like consume u uh anything from that partition. So that's why offline partitions uh become a very like critical metric whenever partition a partition goes offline that means that for sure that is going to make an impact on the system. uh so because of that we have to you know like treat this metric is a kind of like very critical one >> uh so great explanation um and uh so you brought up a good point I know this is purely about monitoring um um you know Kafka but um I think it's worth explaining a little bit about uh the partitions and the replication factor right so you you talked about a situation where if the replication factor is one and um you know If the broker that happens to have that partition goes goes down then that's completely un um you know unavailable for the producers and consumers. U now in general most people run minimum of three brokers in production and teams often think you know what I can tolerate one um one broker failure um but may or may not pay attention to the replication factor. So it's probably like it's it's for some you know for some situations maybe you are okay with having a replication factor of one but for all your uh critical topics uh you should at least have a replication factor of two or three and the the the tradeoff there is the higher the replication factor the the more the broker needs to work to to keep everything in sync right it's a it's a fine balance between high avail availability and uh uh you know performance. Uh but if you are um you know if you have a couple with replication factor one then you want to make sure that um you know any time a broker goes down uh you might be able to tolerate a couple seconds or minutes of sort of outage but your team is fully aware and you attend to the issue immediately. Yes. All right. Awesome. Uh if you by the way if you have any questions please um you know feel free to to ask and then Kunal just alert us um I'm not keeping an eye on it so let us know if there's questions that we're happy to answer as we go. All right so Sim so what about active control account um can you explain why that's important and you know to monitor? Yeah. So, uh, Kapka cluster should have at least like like there should be one active uh controller in the cluster. So, uh, active control is the one like who takes care of like uh partial lead assignment you know like it it is the one you like who manage the cluster. So, uh at a given time we should only have one active controller. If you see you know like more than one active controller in the cluster that means uh there's a script D condition you know like column is not maintained properly. Uh and if you see you know like zero active controller that means cluster is not uh functional because there's no active controller. So uh that's why active control account is super critical like we'll have to keep an eye on it because FT controller count is you know like zero or more than one that means we have a nonfunctional cluster that for sure that's going to make you know like impact on all of your consumer applications producer applications and data pipelines yeah so that's going to be super critical um I we didn't mention it here uh from an availability point of view we may have to think about zookeeper too because not everybody is on kof mode yet. Um can you talk to a little to us a little bit about uh the zookeeper side uh as well? >> Yeah. So I think uh yeah kaka recently introduced kab that means it maintains the corum internally. So it uh it has the ability of you know like keep all those state changes uh uh internally in the kapka cluster. So we need to have at least you like three brokers running if you're using K app but uh earlier versions you know like uh I think that's you know like most widely used versions of Kupka still uses sookeeper uh as uh you know like uh state you know state storage and sookeep is the one you know like who maintains a corum you know like we need to have at least you know three three zookeeper instances to you know like maintain the corum and uh cluster you know like just uses that so keep clustered you know like uh keep all those metadata like if you take active controller like the uh election process like uh whenever we start the cluster one broker is going to create a key in zookeeper uh and it will keep you know like refreshing that key that's how it becomes the active controller soap is super critical in that case because that's you know like heart of kafka you know like that's what you know like keeps all those uh like keep the uh state of the cluster and uh yeah all the like uh uh topic information uh sometimes uh the uh broker meta data everything uh is kept in keeper so that's going to be super critical so we'll have to keep an eye on keeper as well in that case but in recent versions you know like uh it's just going to be Kafka brokers if you have at least you know like three Kafka brokers cluster will mostly be functional uh yeah I That that sums it up. >> Gotcha. Um you also mentioned about like so with with keraf though like you you need the um you know for cororum you need a minimum of three right so that means we have to also monitor uh a minimum number of brokers in production if you're using kaf mode right yeah so that's actually pretty important as well to kind of keep in mind yes very good okay all right um any questions please keep come uh please keep it coming happy to answer. All right. So now we talked about the availability of of the the broker side. Uh let's you know go on to the u to the sort of like the durability side of it. So um why should we think about underreplicated partition? What is underreplicated partitions and why does it matter? >> Yes. So we talked earlier like um a partition can have multiple replicas and in a production systems we need to have at least three replicas. So that's the standard. >> Uh so if a partition is not able to you know like uh fetch updates uh in time from the uh partition leader like we have like in Kapka we have uh like even if you have multiple partitions there's a leader elected for each partition. So that acts as the partition leader that is the partition uh who is you know like uh catering uh all consumer request and uh produce request. So uh followers are like uh replicas are like just going to be you know like uh stay in the passive mode. So uh if one of those replicas are not able to you know like keep up with the uh leader leader replica leader partition the then that partition becomes uh an underreplicated partition and uh like uh with this we have to think about uh insync replicas as well because like uh for each Kafka topic uh like we can configure uh number of instinct replicas that means uh how many of those partitions has to be in sync with the leader at all the time. So uh like uh if you have unreplicated partitions you know like that that is you know like exceeding the number of insect insync replicas uh like minimum number of insync replicas that's going to be uh like that partition is going to be uh underminr that means uh the the the the partition is not able to keep uh the minimum number of instant replicas active. So in that situation if uh a broker goes down we will be surely end up with you know like having a data loss because uh the uh new leader has to be elected and new leader is not in sync with the uh uh partition leader. So we are going to lo some messages in that case. So that's why we need to you know like keep an eye on this metric especially under replicated part partitions and like we need to configure mean ISR count properly like uh at least you know like two mean ISR >> yeah uh so that would help you know like whenever goes down we can you know make sure we have at least one replica available to you know like take over the load. >> Gotcha. Okay good. So this is so so it's very critical um if you're looking at a um well I mean I'm not a big fan of just staring at dashboards but if you're doing u any kind of monitoring uh it's very important to keep keep an eye on uh these two these two metrics and get on top of it right away right good okay uh great explanation um FMA um let's uh now switch track uh towards the uh performance side of things Right. Um generally uh you know like I [clears throat] said it's Kafka has a pretty good reputation um on um how to um you know like to handle you know millions and millions of messages um you know it's a very well battle tested proven piece of software but things can go wrong um and um not to mention uh Kafka is written in Java so there's uh there's the JVM aspect of it as well. Uh you know poorly configured Kafka clusters can cause a lot of grief. Uh so we haven't sort of mentioned those metrics here. We'll we'll talk about it when we actually describe those in um in the um when we talk about the dashboard. Uh but let's talk about the performance related aspects of it. um uh you know apart from the resource usage because everybody I think most people know that uh they need to look at memory CPU especially keep an eye on JVM JVM metrics and stuff like that but what what what are the other things that you should keep an eye on so let's start with the sort of like the maybe um the the disk aspect first oh sorry we'll start with the actually the performance of uh you know like the messages coming in and going out kind of scenario. Um what what about the the the request latency and and is there anything else that you feel is important that's not um mentioned here? Let's chat about that as well. Yeah, go for it. >> Okay. Yes. >> Yes. So like if you take a closer look at the Kafka protocol, it's a branch of requests and responses. So if you take a look at the light documentation, they define, you know, like these are the requests that a broker supports and these are the responses that it provides. So even if you publishes a message that's going to be a request. So it has an API key and uh it has a binary payload. So and it holds a batch of messages. So uh when it comes to Kapka broker site what it sees see is this requests and responses and the duration it takes to you know like uh fulfill a request or like generate response. So that's what we see from Kafka broker perspective. So uh it's important you know like keep an eye on uh these metrics. I mean uh fetch and produce is just you know like uh uh API keys you know they have API key for fetch API key for produce you know that's how they identify different message types. So we'll have to you know like keep an eye on the latency of uh each of these message types. So if you see uh an increase in uh fetch latency that means uh the broker is not able to fetch messages uh maybe from the disk or maybe you know like there's something happening in the uh processor level you know like it's not being able to uh fetch these messages in time. So like uh that's why we need to uh keep an eye on these metrics. If you see uh a spike or a growth of these metrics that's going to be you know like uh we'll have to pay attention and uh uh you know like uh see why it is happening and this PV usage uh like that's also like I have seen you know like uh like we configure Kafka and you know like it runs without uh no problem but uh as you said you know like in the midnight you know the system stops working because we are running out of disk. So like uh monitoring disk usage is super critical in that case because these are the metrics that you know like we usually uh >> don't uh don't care much right. Yeah. >> Yes. Exactly. >> Everybody's looking at CPU memory and stuff like that which is also why we didn't include it here because those are super common things but these are things that people neglect like you said right and comes back to bite you. Yeah. and lo flash time also like related to disk PV performance. Uh usually uh yeah so uh if we see an increase in lo flash time most probably it would be caused by caused by a kind of like uh slow disk or like uh the broker is getting you know like u like uh heavy traffic but you know we don't have uh a disk with you know like capable IO attached to the the the brokers. So like uh those are the things that we need to tune. Uh so if you see you know like lo time is increasing or you know like uh and like going above the uh number that uh going above the standard number. So then um we'll have to think about you know upgrading the the infrastructure you know like update like either we have to you know like reduce the uh incoming load usually that cannot be done because we have to carry the traffic. So most probably it's going to be an upgrade in the infrastructure. >> Okay. Gotcha. So you talked about log flash time. So one of the things that I remember when when when we were u when I was working on message brokers not cafka but you know like the the traditional message brokers uh flushing to the disk was very very critical um because that until it's on the disk that means there is a chance of losing messages. I think in Kafka Kafka mitigates that problem by u replicating uh you know it provides HA by u you know or the fall tolerance by replicating and then um it batches the uh messages and and and and it hits the disk um you know ever so often that's how it can keep that performance right so if you write to disk more frequently that means u your performance can um suffer but uh If you you know increase that uh you know the batch that you actually write to the disk uh if you make it larger they the the inout window is higher now if so if the brokers have a catastrophic failure you may actually end up losing messages. So this log flush time um and and so how how would some someone look at so disk is pretty clear. So like for example for me the disk is very clear I can look at the usage but the log flush time what should I look at uh log flush time is it you know like if I'm looking at for some sort of you know leading indicator of trouble how how would I monitor this uh log flush time? Yeah, I think like uh it's going to be like what I would recommend is you know like uh having you know like alerts monitors configured you know like uh uh because we don't have to you know like keep an eye on this manually so we can configure some alert in uh just you know like uh alert or like trigger a monitor u whenever this increases so we can configure with the threshold >> so uh yeah I mean our platform you know like provides that functionality you know like we built some features to uh u configure these monitors so I mean yeah that capabilities are there so >> awesome so we have some questions um and um yeah I think from Sidant um so this one's pretty easy so I'll answer this myself and I'll keep the hard one for [laughter] for Simal that's the next one so yeah you should look for at least one controller uh Right? That's what you should monitor for. You need a minimum of one control. Um but I mean you could also alert on you know a reduction in in in as well but like at the least there should be one controller. Um how about the next one Kunal? Let's bring that up. Simal you want to take this. Oh, I mean like uh yeah, so this question asked about uh how do you correlate ISR trink expand events with the request latence pipe. Yeah. So if you see uh very noisy ISR shrink expand uh metric that means there's something wrong most probably you know like uh with the network or like broker resources because uh uh instant replicas you know like um the replicas are able to sometimes uh keep up with the uh leader sometimes uh it falls you know like uh uh falls behind. So uh if you see a noise in ISR string expand metric we should definitely you know like uh check it and uh see what is going on with the with the broker but like we'll have to correlate it it with the broker you know like uh in which broker those ISR sync expand events happening because uh that helps us to you know like minimize the problem and you know like uh check that exact broker uh But if this metric is noisy, we'll have to you know like definitely uh uh check uh what is going on uh with those brokers. Uh most probably it is uh network saturation but sometimes you know it could be uh due to JVM errors. It could be due to uh uh like uh resource restrictions uh yeah it could be anything but uh most probably it falls under network side when it comes to heavy loads. Gotcha. >> And request latency spikes. Uh if there's no change in partition leader, uh there won't be any impact on uh request latencies. Uh because uh partial leader is still up and running. Uh yeah. >> Awesome. Um SAL um shall we sort of um look at how to visualize some of these things and we'll talk a little bit more about the broker side before we move on to the consumer side. Um if you like I can share the screen and um you could talk through and I'm just let me know and I will uh I can um scroll. >> Let's do it. >> Yeah, let's do this. Uh share my screen. >> Yeah, I just want to say thank you for the question folks. Keep them coming. Uh we are happy to take questions related to what we talked about and what we are going to talk about. So keep the questions coming in. Love this. >> Right. Okay. Give me one second here. Um I now let me quickly share the screen. All right. All right. So I have a Kafka broker [clears throat] over here. Um Simal, you want to um take it away and I I can navigate um if you explain. Um >> yeah. >> Yeah. So uh yeah, I mean uh this is a dashboard we built uh you know like considering all of this stuff we discussed. Uh so like it starts with you know like some like very static metrics like number of brokers and number of total partitions uh we have in the cluster um and uh active controller should be one uh all the time and uh offline partitions uh if you see at least one offline partition and if that partition belongs to you know like any of your critical topics you should you know like uh uh take that as uh a critical metric uh crit critical indicator. Um unreplicated partitions uh like if you have you know like any partitions uh uh which are not uh being able to uh fetch uh updates from the leader uh like uh that partition becomes uh unreplicated. So we'll have to keep an eye on this as well. And unclean unclean lead elections like we talked already like we need to have at least one instinct replica ready to take over the leadership. uh whenever the uh broker which holds the partial leader goes down. So uh but if you have a situation where we don't have any insync replicas and the partial leader goes down that broker broker goes down. So a leader elected without uh lead elected from uh non sync replica I mean u because we don't have any uh insync replicas available to take over the leadership. So uh controller had to you know like elect a leader from uh one of those replicas which are not in sync. So then we treat it as unclean lead election. So that's why this uh metric is super important. If you see you know like uh this metric is greater than zero that means uh uh you will most probably uh experience a data loss >> and uh those you know like uh instance metrics these are like very you know like regular metrics that we have in any system CPU memory >> and the important one is JVM because K cup runs on JVM and uh JVM is you know like uh like uh most of the time you know like memory issues are more common in JVM because of mismpulations. Yeah. So uh we'll have to keep an eye on JVM memory utilization as well. And uh since we need to keep uh an eye on network ingress and it does we kept those metrics here as well. So that allows you to see network andress for each broker. Yeah. So we provide you know like uh a detailed view as well like you can you know target a broker and uh see these metrics and PV utilization as well. So we can >> you can even go down to the level of a PV volume claim and then just visualize in case you're like worried about uh you know one of you know some of those. So that's a great one. I think this kind of visibility like you said um like a lot of people monitor CPU memory um not and and JVM for sure because it's it's a known you know pain point uh but often forget about this so it's something to keep an eye on because it can creep up slowly and suddenly cause uh you know some a lot of grief due to an outage and now you have to you know scramble to to get things sorted out right all right so uh let's start here um message consume Um shall I expand probably it might be easier. Okay. All right. So >> uh provides uh a detailed view you know like uh for each topic uh it uh shows you know like how much of messages coming in. uh messages out is not here because Kaka broker doesn't track it you know exactly because uh um Kapka is designed in a way that you know like it uh pushes all that consume heavy lifting into the consumer side. So broker doesn't care you know like uh uh of that aspect much. So that's why we don't see uh message out count here. But message in count is a provided metric in Kapka. So uh that MBN is there. So yeah this view just provides uh a detail you know like uh we can apply filters and see uh for each instance for each topic how much of messages uh uh we receive. >> All right good. Okay. Um let me uh so that's that's actually a good example and then you can actually drill deeper into the to the metrics to see sort of what's going on. Right? So um there's a reason why we ordered these things in in an in a such a way so that you can see the um you know like the important metrics kind of on the top. uh generally like you know like Simal said you want to pay attention to you know some of these uh you know highly valuable signals and the utilization we talked about consumer message consumer count um remember that production scenario that I explained right at the beginning here's a hint uh those things can be avoided if you pay attention to consumer life so when we talk about u uh the produce and consume applications. We'll dive deeper into it but um why don't we quickly explain this part because this is a critical critical metric um um and then this is probably where most team most most issues happen because like I said um Kafka once you configure it properly for the most part it's a very well battle tested um you know super performant piece of software so uh generally where I find um you know when I when I worked in messaging for a long time it's it's the consumer um consumer side is often uh the culprit where you know where you have a lot of incidents revolving around uh you want to talk to us through the consumer lag and we already have something steadily climbing uh [gasps] >> yeah so one of the like most important metrics when it comes to Kafka consumer groups you know like one of like uh most popular features of Kapka is that you know the consumer group API. So it allows us to you know like have a group of cons group coordinated uh uh applications you know like consuming from a topic and applications you know like uh uh will be assigned to uh partitions and applications uh can consume from those assigned partitions. So uh uh like if an application instance is not able to you know like uh consume messages uh consume and process messages in time consumer group goes up that means uh the broker keep receiving messages and consumer is not u able to uh process those those messages in time. So uh that causes consumer group l to you know like uh go up uh yeah so uh one of the hard parts of consumer group lag is like from the broker side we know there's a consumer group lag but it's hard to correlate it to the application side because uh what broker provides is uh this is the consumer group name this is the topic this is a partition and this is the uh host address of the application. So most of the time what I have experienced is that whenever we see a consumer group plan we have to go through those tooling to you know like figure out which instance is that we get the IP and we have to log into some other tool or we'll we'll have to use uh Kapka consumer C like Kapka admin CLI uh Kapa consumer groups uhsh like that is provided by Kafka to get details of that consumer group get the IP and then we'll have to find out we have to log into you know like uh cloud platform and find out the VM which holds that IPO like we'll have to go through Kubernetes port to find out which consume instance is that you know that takes time and most of the production uh issues what I have experienced is that like we are not able to handle the pressure you know with pressure makes you you know like uh doing you know like unexpected things so even like uh little pieces of information like that misses. So uh like that's why we try to you know like correlate this consumer group lag with applications so no one has to uh you know like uh follow those IPs and you know like find find out which instance is that we do that correlation automatically uh through our hotel pipeline. So uh that allows us to create a monitor which uh may trigger a monitor uh in that exact application whenever a consumer lag grows up. Rajit if you can like uh show one of those monitors triggered. Uh >> oh for sure. So actually we'll do better yet we can try to actually there's a simulation running. So when we talk about um consumer applications uh when we explain that we can we can definitely show. So I I completely agree with you like my experience is the same that um generally um you know consumer applications are the major pain point of any messaging system. And then the second thing that you highlighted which is um it's easy to see from the broker that there is a consumer lag but how do you correlate to the workload? So we are going to show that uh that'll be the the final part of the demo. So stay tuned for that. Uh we can show you how easy it is to add a monitor to correlate that. Um, okay. Shamemer, do you want to um um uh generate some monitors so I I can explain some stuff here and uh um you know, we will attempt to bring one of the brokers down and um you know and cause um uh some some monitors to trigger. Uh the idea be the idea here is that um we want to um you know like this um you know try to actually do something uh like you know live and and and you know simulate this uh bear in mind in this particular environment uh we are sharing a Kafka broker um that's uh that's supposed to also process some of these uh signals that are coming through. It's a um you know it's a closed loop environment. Um so there might be some unforeseen um um I guess issues as well but you know nothing like a live demo. So we are going to Fremont's going to deal with it and I I'll probably I'll explain the u the monitors here. So Freel talked about a couple of monitors. Um meanwhile Siml can um you know bring down one of the brokers. We we provide um a catalog of monitors to make it very easy to do uh monitors. So that's what I was talking about. So we from the broker side to the consumer side. Um these are just uh cookie cutter stuff. You can also write your own custom monitors as well which is possible. But um you know I've added a couple monitors um to kind of you know to show you how it is uh how it can be done. So I'll So for example um you can add stuff for underreplicated partitions. Uh so what this what this what uh this triggers if there's at least one under replicated cuff partition. Um if this is a more frequent issue that you're not too worried about you can write a custom metric and give a um you know a higher threshold if you want to but for most folks um you know I would like to know if there's even one under replicated partnership because I want to get on top of it. So that's the monitor. Uh let's look at um uh minimum brokers. So this is an important one. We talked about um you know u oh we accidentally added two two of those but uh it's fine. Uh what what this does is uh if the number of brokers go below three you you should get a monitor. So um you know all of our monitors issues u um you know log analysis and stuff like this those are all connected to alerts um so you can assume uh if you get a monitor you can configure an alert to go to the right team all of that is so I'm not going to go into that because that's that's a separate topic but you can assume that you will get notified right so here what the condition is uh especially if you're running keraf right you want to maintain cororum uh and then the other case trim mentioned about um uh you know if replication factor is one if you have a couple of those those topics uh some of your you know um partitions could be offline so it's important to have uh this uh you know a monitor like this um obviously uh you can also add one for JVM as well this is for pod usage but you could add one for JVM as well we have one for JVM uh pretty easy um you you know you can uh um um you know if you can also do a percentage as well. So let's say I don't know um you know 80% 90 generally for Java by the time you reach 90% 95% it's trouble. Uh so you can add a percentage uh value as well uh for usage and um we already talked about offline partitions. So here's a way if there's at least one offline partition uh you know you will you will get uh a monitor will be triggered. uh cluster degraded again um this is mo mostly simal you want to explain this about this is um if this is more about if let's say you have a five node cuffka cluster and then suddenly one or two has gone down so it's still maintained cororum so it's not the minimum number of brokers but something has um you know shifted in the the the so it's now running at a degraded state so you know I've seen people running about five cup brokers in production ction for you know so this is a great one to figure out that uh you're still not yet in trouble but you are probably trending towards that so keeping an eye on this so uh this is where you know that's like a bit of an you know you don't have to configure any threshold we'll just look if you're running five now we're running four problem the the difference between this and the minimum is the minimum is that's absolute um minimum that you willing to tolerate so you see like different sort of metrics, sorry, monitors available in the, you know, in the built-in catalog and you can also write your own. Um, did you uh did anything trigger FMLA? Did you uh shut down the broker? >> Yeah, I I uh put one broker down. >> Okay, so hopefully soon we should see something um over here. Um see, give it a few minutes. Um like I said it's something sometimes things works in live demo sometimes it doesn't. Um >> okay >> so there's a quick question related to this Rajit sorry to interject maybe you want to take that >> that's a real quick one. >> Yeah. >> Ah yes so so okay so someone that's a great question it's a bit of a blast from the past. So uh when we worked in traditional message brokers we were obsessed with Q uh QEP ex um more or less the it's actually um a little bit different in the sense that um um in the in the traditional message brokers QEP meant massive trouble for the broker. So the more messages backed up on the cues the broker generally had more um like you know issues. So we we monitored QEP more from a broker perspective and that was like a you know like a a pretty tricky situation and you had to like quickly get on it. It's because your consumers are u um you know the consumers are slowing down but um in Kafka generally um you know uh the the consumer lag you it's it's similar if you can think of it that way but um it doesn't affect Kafka the same way at least that's my experience doesn't affect the same way as it did for the traditional message brokers but the more messengers backing up um you know is generally more of a problem uh on on the consumer side. Um um Simal is there an issue on the broker itself if if you know the you know uh if there's more messages sort of piling up >> what is your >> I don't like I don't expect any impact on the broker side because it's just the disk which will be you know like clean up >> uh because Kafka is designed in a way that you know like uh like the difference in between Kafka and traditional message cues is that those message cues are designed in a way that all All the heavy lifting happens in the uh server side in that you know like broker implementation but in Kafka all the consumer heavy lifting happens in the consumer side consumer application side. What it provides is a rich API to consume and coordinate. So we can have a consumer group. It allows us to consume messages but it doesn't take you know like uh that you know like expensive uh operations to the broker. It all you know like happens in the consumer side itself. So if you see consumer group lag in Kapka like uh it usually you know like fills up the disk uh but again you know like uh even if you don't see a consumer group lag if you continue receiving messages it fills up the disk right. So uh until it reaches that uh retention policy. So that's the difference in between you know like uh those traditional cues and Kafka. So consumer group lag would not make any impact on a broker. Yeah. >> Oh okay. Gotcha. rupture right so so so my understand is correct then you know um because I haven't seen that kind of like pressure on the broker it's mostly the pressure is on the on the consumer applications but in the traditional messaging side QEP is uh uh can be fatal to the brokers and that's generally where the brokers topple yes >> so I think this is a continuation of the question yeah Yeah. So actually that's a good good way to sort of look at it because the the when the cons when the consumer side is slowing down you're going to have upstream and downstream impacts on on your uh business uh flow right u generally you're you're doing some sort of transaction and um this puts pressure on your um this puts pressure on your uh ability to process uh those business transactions. options, right? And this will manifest most likely in in other um you know um areas as well, right? So for example, if it's an order processing system, suddenly you have a bit of a backlog of orders being uh processed and people are not able to either place orders or they're going to see a significant delay in getting u their responses and stuff like that. So if you're doing some sort of SLO monitoring on the number of orders processed or you know the average response the average time it takes to process an order those things will start to trend up. U so you can definitely tie into it. I I hope I explained it to you or um or did you have a different sort of um uh ask here? I'm just trying to understand it. I I'm hoping I understood your question correctly. So feel free to mention if I didn't explain it properly u uh sorry I I can't see your name uh um but um you know if you want um ask to clarify or if you have followup oh that is fine okay thanks appreciate it good um oh I looks like I stopped sharing my screen uh let me uh let me share it again uh where is Yeah. So >> I also got a comment about uh the monitor templates that you were showing previously. >> Okay. Yeah. If you can ask me. So there is a so this is the Kafka one the minimum um broker thing um you know there is you know alerted value. So we we wanted a minimum three but you got u only two. So we were able to trigger it up. Um maybe you might want to scale it back up so it doesn't affect the rest of the start of the demo. Yeah. Um what was the question again? Uh Kuna. >> Oh, it wasn't a question. Uh this person loved the built-in monitors that you showed earlier. >> Oh, thank you. I appreciate it. Yeah, we figured that you know I mean um we used Kafka heavily in production and um when when we u you know I used to do consulting before and then and and it was really tough sometimes to write these monitors why not you know these are things that uh can help people so that's the kind of idea behind it I appreciate uh your comment uh thank you all right so um may before we go onto the um um consumer um uh producer and consumer side. One other thing I want to talk about something that's pretty powerful in the platform we provide which is uh log analyzers. We didn't uh we didn't um you know recreate that but um you know I want to talk to you can we talk a little bit about the the log analyzers. So this is real time log monitoring. Generally there are lots of uh things that you might be um you might want to keep an eye on from a Kafka logs perspective right I'm you know instead of people looking through the logs uh or you know this is a way for you to monitor logs by the way this real-time log monitoring is you know it's for any application that we monitor um and and that's one of our u key features where we can provide real-time log monitoring without the ingestion costs So can you talk to a little bit about this um Rema like why is this this kind of important and and this is an example? >> Yeah. So I mean like uh one of the missing parts is that like we deploy Kafka and you know it runs without any issues because we don't see some stuff in metrics but you know something is happening in the background which may you know like end up uh you know like triggering something bigger you know uh some cascading failure. So that's why we need to you know like keep an eye on Kafka logs as well. Uh so like uh this is one example. So a broker complains about uh uh there's a you know like partition that has you know replication factor uh let's say uh we have uh replication factor tree and it is not been able to you know like uh catch up with that you know like replication factor it's not been able to create those you know like required replicas. So like these kind of exceptions like can trigger time to time and can get resolved automatically as well. But it's important to keep an eye on these logs. That is not something we can do manually, right? I mean we can't do like we can't uh go through these logs every day. Uh so um that's not something uh we should do. So instead of doing that we can just configure a log analyzer uh capturing those critical log entries in Kapka and uh whenever uh that that log entry emits uh we will you know like get a notification. So that's the idea of this feature and uh we can you know like add more and more log analyzers targeting those critical uh cups. This is a generic pager that we provide uh for all the applications that we monitor but you know uh will help in Kafka monitoring as well. >> Yeah, that's that's actually a um a great point. Um and and generally like the way we use it so when when we encounter issues in production and we you know u whether it's cuff or anything else we we try to add that as a log analyzer. So most most teams um you know like every every situation is different and and you might um you know be experiencing some issues that others don't. So you know your setup very well. So what you need to do is um you know when you get these kind of um issues um and you know what you have seen in the logs you can use that as a as a hint to uh you know do more proactive monitoring for the next time it happens right or if you already know some of the gotchas you can configure it. So what we do is we generally look for bleeding indicators of trouble in the logs. We we we we create log analyzer for those patterns and when production incidents happens um you know one of the key learnings is we want to we look for um log patterns that we can monitor going forward. Right? So this is a great way of for kind of like you know learning from outages and incidents and then capturing that knowledge uh and and you know like doing adding that to you know automatic monitoring uh going forward right so good. All right so we talked about that as well um and then um how about we switch tracks and go back to the to talk about producer and consumer application. So I'm going to stop sharing my screen. Kunal, you wanna um uh you want to start uh the slide that we'll go to the other one? Okay, good. All right. Okay. Uh Sri um you know um let's talk a little bit about the producer side uh metrics and consumer side metrics. I think you know keeping an eye on the time as well. I think let's go through the slide deck and then finally go for the for the first for the demo. Um so on on uh generally when when you're producing and consum consuming there are a couple things that you want to keep an eye on. Uh talk to us about the you know why these are important. Arguably I've rarely seen producers having trouble but it's mostly consumers but you know humor me and explain this and then we'll jump onto the consumer side. >> Yeah. So request rate is you know kind of like uh very common metric that we keep uh an eye on. Uh basically you know like uh it shows you know like how much of messages it produces per second. Uh that's a very basic metric and request latency like uh how much of time Kafka broker you know takes to uh respond to that request because a request can you know like contain one message or a batch of messages. So uh so request latency like uh if you you know like if you're not using batch producer then request latency is going to be you know like uh very small number but if you're using batch producer it's going to be you know like somewhat high that's natural uh and again like error rates you know like most of the Kafka producer libraries rappers like uh uh have uh inbuilt retry mechanisms as well. So if it has a retry mechanism, we'll have to keep an eye on that as well because more retries means something is wrong in the you know like maybe in the network level or the broker level. So uh we'll have to keep an eye on the retry rate as well. So these are like very basic metrics that we need to keep an eye when it comes to uh producer side uh of cup applications. Uh yeah, I think I think that sums it up. >> Okay, awesome. Uh let's move on to the consumer side. Um and let's talk about this. Um Simal, go for it. I mean we already talked about the consumer group lag. Um and then one of the more important parts is to how you can correlate consumer group L to a particular consumer. We'll show you that in a second. Um you know maybe take a a second or two to explain the the other stuff as well. Yeah, I think like we talked about consumer group black the consumer is not uh being able to uh consume and process messages in time that you know like may lead to uh you know like pile up messages in the broker side. So uh and consumption rate you know like uh the messages consumed per second uh like uh so if you have like you know like uh multiple consumers running and uh if you take a closer look at you know each of those consumers in the group and if you see a kind of imbalance in consumption rates that means you have you have something to fix in the you know like uh message producer side because uh messages are evenly distributed across those partitions. So if you see uh consumption rate uh you know like uh not even in the uh consumer group that's something you know like which signals us like take a look uh at the uh load distribution. So most probably it's going to be the uh producer site which produces the message. So it has to you know like uh uh distribute the load distribute the message load across all those partitions. So that's how consumers uh get uh an even lot uh processing latency. Uh this is a kind of like uh custom metric uh that we see in most of the consumer applications which which you know like uh tells us how much of time uh the application takes to process a single message. Uh so it may contain you know like uh the latencies caused by databases caused by you know like other operations. Uh so but it basically says how much of time the uh application takes to consume a single message. Uh processing errors uh again you know like uh a metric exposed by applications usually uh to you know like uh signal us about uh processing errors happen in you know like uh when it processes those incoming messages. Commit errors are more related to consumer groups because uh consumer groups uh will have to you know like um commit the commit the uh group offset time to time. So yeah, we don't have to you know like uh do it for each and every message because that may you know like uh create an additional overhead uh in terms of you know like uh networking and you know like uh uh get creates a pressure on Kapka broker side as well because uh whenever we commit a group uh offset it has to you know like publish a new message to uh uh to an internal topic called consumer groups. So that's how Kafka keeps the uh group box offet. So we don't have to you know like consumers doesn't have to you know like do not have to uh commit it for each and every message but what usually happens is that it processes a batch and commit that batch. So in that way we can reduce you know that that overhead. Uh yeah I mean uh these are the important metrics uh when it comes to consumer side metrics. Raj. >> Awesome. All right. But um before we jump onto the demo, let's uh in the interest of time, shall we tackle the you know the how we monitor stuff. Um and that that's probably important to tackle. So yeah. Um right. So that's the sort of like the traditional monitoring. Um I I'll speak to this part and then Simon can uh talk to the other one. uh generally people deploy JMX agent and then um grab the uh script them into Prometheus and then uh build dashboards or consume from there or you deploy some sort of proprietary agent uh to grab these uh metrics right but um you know we have open telemetry now so uh let's go to the next slide and simal you can explain to us about the open telemetry way of uh doing these things yes >> yeah so So instead of you know like having uh the JMX agent running uh uh inside Kafka. So uh uh this is the uh this is how we can do it uh using open telemetry collectors. You know we can build an open telemetry pipeline and we can uh configure receiver. So uh we have our own uh open telemetry collector distribution. So that has some you know like additional uh metric scrapers which consumes uh some extra APIs of Kafka. So it's just not you know like the JMX metrics published by uh brokers. Uh we also consume the Kafka admin API to you know like collect some some metrics. So uh that's why we have our own uh hotel collector distribution. So we can like you can use that distribution and configure the tail collector to you know like scrape all those metrics uh from Kaka that means we are going we are collecting JMX metrics plus the metrics coming through the management API. >> Awesome. Um we have a generic article on um that talks about monitoring Kafka with open telemetry if anyone kind of wants to get started with the open source approach. Uh Kunal you can link that um you know provide that link over here. Um I you know let's jump into the um demo here. Uh Shrimal um you know we've already kind of done the scenarios uh on the broker side. Uh we just you know in the interest of time we we brought down on broker and then we showed that um uh we we were able to um you know simulate the minimum number of brokers. uh but if you create enough chaos you can um you know simulate the rest as well. Um so you know in the interest of time we're not going to go through that. Uh but on the application side um let me walk you through um there is we've simulated a delay in consumer lag and then we can show how that is you know affect how how you can correlate the consumer lag to a specific consumer application uh as well as um you know show you how you can look at produce and consumer latency for for an for an application itself. So, let me share my screen and uh you know uh wrap this up. I think I'm showing the right screen. Let me see. I have a couple. Oh, so it's the wrong screen. So, let me go back to the other one. Uh yeah. All right. So for this particular one um what you're seeing here is that um um there's definitely error logs as well. So something is going wrong in this service. This is uh using Kafka um and um you know it's consuming from this particular um you know auto processing service. So first of all you are able to look at the the processing pipeline um latency like scream talked about um you know uh and there is some error. So the error is actually happening when it's trying to you know do the corresponding application. It's uh making a put like making an HTTP so processing the message taking an um and actually creating uh invoking an HTTP endpoint to do the second action. So what is happening is uh that is failing um in this case almost 100%. So because of that uh situation um I I think we ran the simulation a little too much [gasps] uh because it's failing um you know and and it's holding things up suddenly your consumer lag is building uh right things are going south. So here you are able to kind of um look at the uh inbound um so the the the the produce uh the consumer latency the consumer throughput as well as um the errors associated with it in this case the bearer manifested in sort of like the the outbound call when you process the message so you can get insight into that uh aspect and then um we talked about monitors in in over here I mean it's also reflected in the in the traces here as well and and the logs. Uh so if you go to the monitors on the application side, remember I talked about there's multiple things that you can do. So here I've added you know um something for the error rates itself but I added it to the to the wrong side. So that's why you're not you will not see anything there. But um in the latency the autoprocessing latency of fraud score. So here um um you know I've added something on the on the on on the on on the um u um message processing because this correctly identifies that this service is um consuming from the order process um um you know topic and um you know if it exceeds this particular latency I want I wanted an alert um and then the consumer group like that talked about. So here I've given a very small threshold. Uh it's actually pretty high right now. Um so you can say if my lag is I don't know 10 [clears throat] 50 20 messages. This depends from team to team. So for example for some some applications um you can tolerate you know lag of 100. But other critical applications you probably don't want to tolerate more than a couple you know messages because it's super critical to uh consume it as fast as possible. So um you know you can you can tune it uh to to your requirements to set up a consumer group plan. So what happens is when this consumer group plug gets triggered we are able to this is the the order latency uh one um you know we've simulated too much so it's now in seconds instead of milliseconds. Uh there should be one for the um you know the the the consumer group like over here. See there you go. Um that's just that one. So what what what Shrimal talked about is generally it's hard to pinpoint it back to the actual application that is uh causing it even um pinpoints it to the to the pod that is actually so if you have multiple pods this is very important right we should have probably run a couple more pods streamer so it's very important if you're having let's say five replicas processing it the problem could be in a single pod because of some other issue Right. So we are able to pinpoint the lag for that uh particular partition to a particular pod or if you run in a VM in the future for to a particular uh instance running in a VM. So that is important because normally teams correlate this manually right you you know look at some other tools from somewhere put things together uh and then arrive at this. Here you will get an alert right away because you can alert on monitors. It will have the information right away to say like u you know which application and which specific instance of that application is uh is is experiencing the lag. All right. So um I think we've kind of gone through the um sort of the situations here. Um please if you have any kind of questions we're happy to answer them. Uh you can also uh reach out to us anytime. happy to have a chat and uh you know um we can uh we can definitely um uh you know we continue this conversation offline on you know on LinkedIn or uh through the comments as well. >> Amazing. Yeah. Uh thank you so much Rajat and Shrimal for sharing uh this information. I know we've been going for a while but I feel that there was a lot of valuable information shared today and I've been sharing all of the links that we mentioned for the folks who have been uh staying till now. We have all right now got around 10 to 12 folks watching. So thank you so much for staying this late um through the throughout the entire session. you can access all the links that I've been sharing via YouTube because last time we got a uh we got a suggestion that they are not able to view people are not able to view the links on LinkedIn that's something on the LinkedIn side I'm sorry about that but if you wish to learn more about what we have been sharing uh you can access all the links on YouTube and I'll make sure that uh once the recording is uploaded I'll uh add the links to the description box as well but again thank you so much folks and if you have got any questions feel free to drop them as well and I'll So make sure to ping either Rajat and Shrimal so that they are able to answer your questions. But do we have any closing remarks from either of you? Rajat any closing remarks from your end? >> Um nothing much. I again I just want to thank everyone for tuning in and um you know we hope you enjoyed it. U thank you very much. Uh I really appreciate that that comment. Um and uh you know we we look forward to doing more of these webinars in in in the future. um you know talking about real production uh scenarios and how to uh monitor them. Yeah. >> Amazing. Yeah. Shimal, what about you? Any closing remarks for your end? >> No, I mean like uh we've been you know like spending time on building this for some time you know and we have you know like lots of plans in our mind as well. So like we'll be implementing those in future as well. So yeah we'll see uh you all in future sessions as well with all those you know like new exciting features. Awesome. >> Amazing. Yeah. Thank you so much everyone for attending and we'll see you in the next session. Bye-bye. >> All right. See you guys. Bye-bye. Thank you again. Bye-bye.