3 Challenges When Implementing Logging & How to Overcome Them

Hey folks, Rajat here. I want to talk about three challenges when implementing logging and how to overcome them. Logging is a key pillar when it comes to observability. And more often than not, sometimes it's the only way to debug certain issues in production. So, it's really important to get logging right. The first challenge, too many logs. uh when when I was helping a customer uh they had a postal tracking application the first implementation of it had copious amounts of logs that was useless and uh and and overburden the logging stack not to mention it increased the cost as well. So we took a step back and looked at logging more holistically and figured out some key things uh to improve the overall situation. High volume low value telemetry will just increase the cost and it won't provide you any benefits. So you have to really think about the quality of the logs and proper categorization of the logs to get the benefits while keeping the cost low. Here are three things to think about when you add a log message. First of all, what is the purpose of this log message? Second, how does it fit into the overall picture? How is it adding to the overall story? How is it tying into the previous step and to the and the one after? So that when you're debugging it helps you connect the dots from one service to another, one method to another, one function to another. So you get can build uh a picture of how the system is behaving. Number three, what is the severity or the importance of this log message? Not every log message is equally important. Not every log message is critical. So we need to think hard about which ones should be uh critical, which one should be error and and sometimes it's not that easy. Uh especially um I've seen a lot of um inexperienced developers struggling with this aspect. If you think about these three things, it will drastically improve the quality of the log message. So my previous example, we we took a hard look at the logs. We removed a lot of unnecessary logs and then we ended up um you know adding some logs back step by step following those three criterias and it really improved drastically improved the operational the debugging experience for the ops team. Next challenge, lack of context. And I talked about a little bit in the in the previous uh you know challenge. Some of these log messages were literally useless because it lacked any didn't have any context. Which transaction did this log belong to? Uh which service did it originate from? Which function? Which component? How does it relate to the overall picture? If you don't have contextual information to help you uh piece together the logs from different service at the least and better yet if you can um you know connect them with other observability um um you know other telemetry data like metrics and trace even better. But at the least you should be able to correlate the logs um log entries to other log entries from other systems, other services that are participating uh in a transaction. So how can you how can you implement this? Right? More often than not, there's always going to be some sort of a correlation ID. Um and in my opinion, there are two types of correlation ids that I would like to put in a log message. One is u a system generated log message uh sorry a system generated um trace ID and the other one some sort of a business related ID for example uh like an order ID. Imagine you know you have an order processing system and and a customer complains um you know something went wrong with my order. Now you may have some um steps recorded that you can quickly see where it failed but more often than not when things go wrong they don't go exactly they don't go wrong exactly the way we want. So you have to sometimes go into the logs to look into um the the log messages to see where it where it failed or at least get some clues to understand where it failed. So in in a situation like this imagine now we have to narrow down by approximately what the customer is telling you. Uh, and if you don't have any contextual information, it's going to be really hard, almost um sometimes impossible. It's like finding a a needle in a haststack. But imagine if you had order ID as an indexed field. Now you can search by that order ID and you get all the log messages that are related to that order. um even they even though they're across multiple microservices now you can get to those log messages within minutes or even seconds. Uh similar to that if you have some sort of a systemdri trace ID you may be able to trace that from uh if you can correlate a a business ID to that particular trace ID then you might be able to figure out okay using this trace ID you can filter through the the different services to to get to that. So always think about contextual information. How do I um help the person who is looking at the log messages to tie it into to the rest of the system to the rest rest of the telemetry data. And the final challenge is again building upon the previous um issue I talked about is siloed views. You know, lack of correlation means your ops team is looking at a bunch of disjointed dashboards and uh different um sets of log messages. Sometimes in complicated in complex systems um log messages can span multiple u production environments uh can span multiple clusters can span multiple services. So think about how do you you know correlate across the board and how can you build more um you know connected views instead of disjointed views to help um the ops teams to to better understand how to put um the how to piece things together. And this starts again by having um proper logging practices what we talked about the first challenge and then um adding proper contextual information so you can correlate better. And once you have those things in place, now you're able to take those log messages and connect the dots across systems, across um clusters, across services, across systems and and able to also correlate your operational data to business metrics as well. So think about these things the next time you u are implementing logging. I would put the same thought and care that I would put when you know designing a system or writing a system. Logging should not be an afterthought and it should be something that you have to think through when you are building the system and when you're getting it ready to deploy into production. Hope this video is useful. I'll be posting more videos like this in the future. See you. Bye.