Gaining Visibility into Kubernetes Costs w/OpenCost | Ep1 - Behind the Build

All right. Either you are an organization running Kubernetes at scale or you are an engineer maintaining a production cluster. You've probably come across these two scenarios. On one hand, the sheer complexity of your infrastructure is becoming unmanageable each day. Or you often find yourself wondering where exactly is the money going. How much are you spending on your Kubernetes infrastructure on a per month basis? And you can't really seem to justify the dollar amount at the end of each month. I think we all agree that Kubernetes cost can spiral out of control really quickly. Teams often struggle with gaining that initial visibility trying to understand where exactly is the money going and how they can optimize the resource usage to first save cost and second it does not impact the performance as well. If this is something that sounds familiar, you're in the right place. Hi, I'm Kunal, Devril engineer at Randoli and I'm excited to have you in the very first episode of Behind the Build a series where we bring in engineers from our team to talk about shared challenges, problems and potential solutions to topics related to observability and cost. Joining me in this episode is Jean Pimemental, a lead engineer working on the cost team at Randoli and together we'll try to break down the topic of Kubernetes cost management. We'll be discussing the challenges. We'll also be discussing why this topic becomes important now more than ever and how you can get started to gain that initial visibility into your Kubernetes infrastructure. If this is something that sounds interesting, watch along and I'll see you inside. So, hey, how you doing today and good morning. Thanks so much for joining me in this particular episode. Hey, thank you Kunal for inviting me. I'm I'm very happy to be here and let's do it. Amazing. Yeah, this is pretty exciting because we are going to talk about Kubernetes cost management and I couldn't have asked for a better guest than you because I know the amount of experience you have in this particular domain specifically because you've been leading the cost team at Randoli, right? So, you know what are the challenges and what are the things that people talk a lot about when it comes to managing Kubernetes cost and the cost topic in general in the observability space. So we have got a lot of things to cover and a lot of exciting things to talk about but before that if you could tell us a little bit more about yourself a bit of your background and what are you doing at Randoli we can start from there okay so I'm Jean Mattel nice to meet you all and yeah I'm a software engineer had here at rand I work from Brazil my home country and I've been a programmer not professionally but I have been programming since a kid because I love programming. For me, it was a toy that I could play around. So, I had that opportunity as a kid. So, I'm very grateful for that. And ever since then, I never stopped programming. But, funnily enough, I didn't start my career as a programmer. I started as a professor. I was a full-time professor here in Brazil. But then after a few years I realized okay so I professor I teach computer science I do research on software engineering but I don't do a lot of programming right so okay I need to pivot my career so then I decided to become a full-time software engineer so that's when I joined Randoli a few years ago amazing that's really incredible journey to hear thank you so much for sharing and at randoli what specifically do you work at and what specific specifically are you working on right now? Can you maybe share that bit more about that as well? Yeah, I work everywhere here and there on product design on management on developing front end back end agent side as well. So a little bit of everything literally sometimes I even help with the devops part of the job because you know you are startup you need to wear multiple hats but lately I have been working a lot on the costing as I mentioned before. So right now we are working on some very interesting features related to chargeback enable users to to do some kind of chargeback model within their companies. Amazing. Yeah, I definitely agree to the point that in a startup you wear multiple hats when I'm a de engineer. So sometimes I'm designing, sometimes I'm writing a blog, sometimes I'm editing a video and sometimes I'm also coding. So there are a lot of different things that I tend to do. But I think this gives experience as well, right? The more you do, the more you make yourself uncomfortable, you learn more. But uh thank you so much again for joining me here today. Um we're going to talk about cost management, right? And the way I see it, there are two segments that I have seen people think about cost management. On one hand, we have engineers and organizations who realize the importance of cost management when they see unpredictable bills at the end of the month, right? And they're not able to decide and they're not able to figure out like where did this dollar amount come from? They're not able to figure that out. So either they are too late to uh realize the importance or even if they realize and if they try to take action on if they try to implement some strategies those strategies aren't very effective enough to save actual money right so these are the two things that I have seen people think about cost management but I'd like to ask you from your experiences from having conversations with other other engineers in the community why do you think just to set the stage of Why do you think this topic has become important to talk about right now more than ever? And why do you think people need to talk about cost management in Kubernetes in general? Well, that's very easy. It's money, right? Cost is all about money and we don't want to spend money, right? We work really hard to earn it. We want to save it as much as possible, right? We don't want to waste it away. And when it comes to the clouds, it's very easy to spend huge amount of money because the clouds made to scale up, right? So you suddenly can have uh your classes get bigger, your workloads scale up. So if you don't uh have limits in place, you don't have budgets in place, that's going to become a problem. Even with those controls in place, there is that thing you mentioned. So we need to understand why the cost is increasing right. So in order to do that we need to do the work beforehand. We need to be prepared so that if that happens we are able to understand why and then mitigate the the reasons. Yeah. Yes. I I absolutely agree with that and I think uh to follow up on this particular point, I think that the topic of cost right has become a major point of discussion in the ecosystem since the past two years specifically and I feel because it was one of the highlights of CubeCon India that happened last year in 2024. I was fortunate to attend that in person and while I was having a conversation with people cost and saving cost how we can save money as you mentioned right people wants to save money because money is involved right uh so that was one of the highlights of cubecoin as well and I think part of the reason this topic has become fairly important is because of cubernetes adoption as well we have seen the adoption of kubernetes increased year on year it has increased one of the reports I was reading of CNCF adoption has increased like 84% in 2024 right that's a huge number and I feel that when the adoption increases it brings more complexity and complexity is directly related to having inefficiencies in your infrastructure because at the end of the day the engineers are humans right we could make mistakes you couldn't figure out because the infrastructure is so complex and because of those inefficiencies you're not able to track the costs you're not able to see where you are spending your money is it really necessary to spend that much right I think everything is related and that's why this topic becomes more and more important to talk about and I think cost management is not just a financial concern it's not just about saving money I mean is it it's it plays a big part to that but I think it's also about efficiency as well right because if you know that where you are spending your money if you know that Okay, this is something that we are not using but we are being charged for that efficiency is getting decreased and over the period of time you're also compromising your users. What do you think about this particular point? Would you agree? Would you disagree? Wow, a lot to unpack here. So, let me try to go step by step. You you mentioned companies adoption and you are completely right because the cloud vendors if you go to their portals they already have a lot of stuff in place to help you have visibility on the costs and have controls but that's at the cloud level but when use corpetis you have something inside the cloud I mean the cluster itself is run inside of virtual machines that they wanted to in some networks that are part of the cloud. But uh what the vendor does don't provide at least yet is good visibility inside the copy cluster. So uh we found that visibility is hard to control, it's hard to limit, it's hard to understand and optimize. You also mentioned that it's not only a concern for cost itself but as a performance factor, right? in our computer right because when we talk about cost we always need to consider the tradeoff cost and performance because you can very easily decrease the cost and affect performance or you can improve the performance and also increase the cost dramatically. So it's a trade-off. You also need to to search for a good balance between those two factors. Oh yes, I absolutely agree and I really love that point that can increase performance but you can compromise on cost but sometimes it's like the other way around like you are if you want more performance you are you end up paying more than what you actually need right love that point. Yeah I definitely agree on that. So talking about cost management right I feel that managing cost in Kubernetes is more difficult as compared to traditional infrastructure and if we go back in the days when we had virtual machines when we had bare metal servers I believe it was easier to do cost management at that particular time if we compared with Kubernetes and part of the reason I feel is the amount of abstractions that Kubernetes has specifically because if we talk about in manage Kubernetes Right? Because at the end of the day, the aim for Kubernetes as a tool was to simplify things for people to simplify large scale deployments for people and due to that it has a lot of abstractions and due to those abstractions it makes it difficult to understand which resources are actually driving the costs. Right? The second reason I feel as I mentioned previously you know adoption has increased complexity. So a lot of dynamic and changing environments, lot of dynamic scaling of workloads is happening when we are talking about running Kubernetes at scale which again makes it harder to predict and track the costs. Right? Building on top of this point according to you when you talk to engineers what are some of the common cost related issues that you have seen people struggle with a lot from an engineer's perspective and from an organization's perspective as well. I think we can have a discussion on this now. Yeah. From a developer perspective, one challenge we see people face is that they don't know the impact of what they do, right? Sometimes people develop the code, they test locally, they testing cluster or development cluster, but then goes to production and they don't have any visibility how it is behaving. Is it scaling up a lot? is scaling down or not? Is it crashing? Is it having performance issues? The visibility that is easy to get is users complain, right? Users complain. Okay, it's too slow. We need to check that and they sometimes they need to interact with the DevOps team or SR team in order to get that data. But when developers have access to that data or even on prod is easier for them to take ownership of that right and take uh keep both performance and cost under control. Interesting. Okay. So the first point that I definitely agree with and understood is lack of clear visibility. Right. Because you can't really optimize what you can't really see. So you first need to have that clear visibility and people don't have a clue of where the cost is coming from. Yeah. And to further that point, if I don't have visibility, I I'm going to be safe, right? So I'm going to to set the request as high as possible to make sure nothing's bad is going to happen with that workload. But if I have visibility, I can h keep an eye and keep adjusting. Yeah, definitely. Uh I think this is something that I've also seen you know just to play it safe you can allocate more resources so that you don't have to worry about any performance issues but it might be the case that that particular workload is actually not using those many resources and you're paying for more than what you are actually using. I think this is a this is a major concern that I have also you know seen a lot of people talk about one is lack of proper visibility overprovisioning is something that a lot of people you talk about we talked about right now just to play it safe you give more amount of CPU and memory to it that kind of mentality I think that also is a major concern and I think which is something interesting you mentioned about chargeback right so lack of accountability is I think one of the other challenges to cost management management because in Kubernetes you have shared cluster resources. two teams might be using a shared cluster and you can't really figure out you can't really attribute which particular team is incurring more cost right so the price has to be paid by both of the teams even if one of the teams is using it yeah historically I think that's has always been the case so before using cloud people have onrem service all right so the company was going to pay for the machines and pay to maintain and keep it running, right? And then we could have both situations. A team is going to allocate the budget for those machines. So, it's going to be their machines or the company may decide to split the cost across entire company or multiple teams and they're going to share those resources. This could happen before the cloud. It happens now in the cloud. It also happens now in Kubernetes. We even have a interest case that one of our customers one more team had a cluster only for them. They wanted the cluster only for them, right? But then using our analytics, our cluster analytics features, the management was able to provoke a discussion and see, okay, we have this cluster all by yourself. You are using it. Okay, but look at how much money we are wasting because that cluster is only used by you. So if instead we allow other teams to also use that cluster, we're going to optimize, right? We're going to uh save money because that cluster is not going to be used by multiple teams, multiple workloads. And then since they are also using our chargeback model they are able to show that and also now that are going to have that cluster shared they'll be able to allocate the cost of running that cluster to the multiple teams that are going to use it. really glad that you shared that real experience with the customer and we are definitely going to talk about what are we doing at Randoli and building the product app insights and what are we doing in the cost section as well what are the features that we are working on because J has been directly involved in making all those particular features but coming to talk about again the challenges right one of the challenge that I have seen a lot of people talk about and because I have been studying a lot about the this particular topic in the past 2 3 months is Kubernetes is a complex infrastructure right if you're using Kubernetes in production it becomes it can become really complex I mean if you're using a managed Kubernetes service the typical model that you follow is you pay for what you are actually using right but sometimes you may end up paying for what you don't use right so that is something which is called as unused resources or idle resources you are not paying for those resources. They're just sitting there taking up resources and you're paying for them but you're not actually using them. So do you think that this becomes a significant problem for especially companies who are using Kubernetes at scale? How significant is the problem of ideal resources when it comes to you know cost management? Oh yeah, that's for sure a big issue sometimes. And the good news is that once you have visibility that that is easy to address, right? You mentioned before, right? The case of overprovisioned workloads, but we also have overprovisioned nodes, right? And overprovisioned clusters. So if you have visibility, if you have alerting tools, you are able to optimize for that. You can optimize your nodes. You can right size your nodes. You can change node types and you can set limits on your cluster autoscaler. Definitely agree with you that is a big concern. Okay. So we talked about you know these different challenges and when we are trying to implement an effective cost management in our organization or you know what are the different challenges engineers are facing right now. But Jean, let's talk about solutions. There are a lot of strategies that organizations have been implementing. If you go on to the internet and if you read a couple of blog post, by the way, we also have a few blog posts you can definitely check out on this particular topic of Kubernetes cost management. You'll find the link to them in the description below as well. There are a lot of strategies that you can follow to tackle problems of cost management. You can do right sizing, you can do idle workload detection. We can definitely have a session in the future to go in a bit more detail into these particular strategies. But I think the most important starting point is gaining proper visibility into your cluster, right? Because if you don't have that clear visibility, you can't really optimize your costs, right? One of the tools out there and it's completely open source and a lot of organizations have been using to implement Kubernetes cost management is open cost right and uh it's it's a completely 100% open source tool for used by companies to manage and to monitor the Kubernetes costs. It has been in the ecosystem for a while now and it has been used by a lot of companies. I think this is a good starting point to gain that initial visibility and I'll definitely come back to the point that why I said it's a good starting point because you need something to go beyond it as well but Jean I think we can talk a little bit about open cost right because I'm definitely interested to know more how it works so if you can share your screen and show us what exactly is open cost why it was created in the first place and how engineers can use to track their Kubernetes cost using open cost. Yeah, of course. Let's take a look at it. Hope you can see it already. But yes, as I mention mentioned, open cost is a great open source tool and it's the first point in giving visibility, right? Because as I mentioned before, the cloud vendors already provides controls and visibility at the cloud level, but within the cluster, they don't go there, right? So that's where open cost comes in. It runs in your cluster. It is a port running in your cluster and with that we can have this kind of data that you see here. I'm showing here the name space breakdown which shows how much I have been spending on each of these name spaces. And as we also discussed this this could be the starting point for having a chargeback model if your teams are separated by name spaces. And how does it get to this point? So what open cost does is it gets the cost of running your nodes. It downloads the pricing sheets from the cloud vendors and then keeps track of your workloads, the ports that are running in your cluster. So by matching the cost of the nodes and the execution of the ports, how much memory and CPU and GPU they are requesting and actually using. We can have this data here of uh how much money each workload cost in each name space. And what is really interesting you can see here the first item is idle. What is idle? I do mean resources that are not being used by anything any workload in this case. So we can see here that we can clearly save some money not a lot of money. This is isol that we have just created but you can see here that we could save this money because you are not using these resources. And then I can go in space and I can start seeing. Oh, okay. So, oh, Flink is using a lot. Should we keep using Flink or should we consider another alternative? And once you have visibility, you can start uh analyzing the data and think about how we can improve on that. But that's just uh one of the breakdowns that are available. You can get the different breakdowns like I really like the controller breakdown that gets the port cost but aggregates up to the controller level which are deployments, replica sets, stateful sets and demo sets. But yes, again we can see the idle cost and we can see here workload by workload. We have the state for set for quick clock it cost this much. We have the breakdown by source type, CPU, RAM and volumes as well. The GPU data is not shown here in the UI but it is available in the responses. You can go workload by workload and start to analyze their cost and also this is a very good site from the open cost community. We have a efficiency value. Okay, here it is a overall efficiency. I myself I prefer to look into it separately. CPU efficiency and memory efficiency. But what is efficiency in this context? Efficiency indicates how much of the requested resources you are using. So if you request one gigabyte of memory but you only use 500 megabytes then your efficiency is 50%. If you request one gigabyte again and uses 1 GB your efficiency is 100%. And interestingly it can go beyond 100%. Why? Because you can use more no matter request. Right? In Kubernetes we have requests and limits. So it can go anywhere from below the request up to the limit. Again, if you request 1 GB, but you use 1.5 GB in practice, it's going to be 150% efficiency. This is really cool hearing about this and learning and seeing the amount of visibility that you get just from this single page just by connecting your cluster. I think this is a game changer uh especially for teams starting with thinking about cost management because here you are actually able to see which workload is costing how much right and make the necessary decisions according to that. But I think one misconception that a lot of people have is okay this particular workload is costing this much. So maybe if I reduce this workload or maybe if I delete this particular workload that would save me some cost. But that's not the case at all. The cost would get saved when you scale down your node, right? Or when you basically do cluster right sizing, right? So maybe if you can touch on this particular topic like how is workload resource allocation directly impacts cluster right sizing and what actually is important to save cost at the end of the day you are completely right Canal in practice we don't pay to run workloads right we don't pay our cloud vendor to run workloads we pay our cloud vendors to use their machines right the virtual machines. So even if I delete all of my workloads, if the nodes which are the virtual machines, if they're still there, I'm still paying for that. I pay for what I request from the cloud vendor, not from what I use. If you want to save money so you need to do both intended, you have to work on the workload right sizing and potentially that is going to give you briefing space to do the right sizing on your cluster on your nodes. And we can even see here I really like this breakdown from open cost is the nodes breakdown. We can see each node here and analyze the efficiency of each node individually. So you can see here that this node almost has capacity it's 81% but some other nodes have a lot of run for optimization. So then I can start thinking about have a combination of larger nodes and smaller nodes or depending on my environment I can think about switching node types but always as I mentioned before what save us money is changing the nodes if I look only at the workloads I'm not going to save any money at all and of course if you have autoscaler a cluster autoscaler in case if you reduce the workloads end off it's going to scale down but that's not always the case so you need to think about that as well yeah I definitely agree with that and I think this whole topic of you know autoscaling right sizing this is definitely an important and it certainly requires a session of its own because there are a lot of nuances when it comes to right sizing and there are lot of different levels of right sizing you have cluster right sizing ing workload right sizing namespace level right sizing container right sizing so there are a lot of things we can talk about only about right sizing in one which is one of the strategies right I think we can definitely have another session on this but all in all what I understood is open cost definitely helps to gain that initial visibility into your cluster how much a workload is actually costing you and eventually it helps you to make that decision of scaling on your node which eventually will help you to save your cost right so you are able to gain that initial visibility now that we know what open cost can do it's only fair to ask this question that is open cost enough because let's say I'm an engineer from a company we have used open cost to gain that initial visibility once you have done that that's really good you are you are already a step ahead of a lot of companies but there is still work to be done right because lack of proper visibility. That was the first challenge. We tackled that. Now we have proper visibility. But the next step is to apply have some actionable insights to actually save those costs. Right? Once you have that visibility, you need to do some optimizations. You need to perform some actions and I believe open cost in its own way has some limitations when it comes to giving you those actionable insights. Right? So Jean I would like to ask you if an engineer let us say or if an organization decide to install open cost today itself after hearing this particular podcast what are some of the key limitations do you think that they should be aware of from day one when they're using open cost okay so if you're interested in analyzer closer and potentially save some money for your company for sure please go ahead Now pause this video, come back later, but go ahead and start open cost is really helpful. That's the starting point and maybe that's going to be enough for you for your needs for your organization. So definitely start there. If at some point I start stumbling upon the limitations of open cost then you can start think about all the solutions as well. But definitely definitely this a great starting point. And one of the limitations we have is that in order to analyze cost we usually need a longer time frame. Most times one week, two weeks not enough. We need to analyze a few months in order to get a proper sense of how is evolving. Is the cost increasing? Is the cost decreasing? Why is it increasing or decreasing? But uh since open cost relies on Prometheus the amount of time the data is available it's going to depend on the rotation policy that you have on your promeus which is usually one week or a few weeks not months right you can start addressing that also using the the opensource tools around the open cost itself that are part of the open cost community. So for instance there you can find exporters but I can export the open cost data and store it somewhere else for you to analyze later on. Got it. Yeah, and definitely we'll be linking some resources of the different concepts that we are talking about in the description. But I definitely agree with John. If you are listening to this particular podcast, you can just go ahead and install open cost is a great tool for you to get started and gain that initial visibility. And once you decide to scale, right, then these certain limitations that Jean right now mentioned, those are important to cater as well, right? One of the few other things that I can also think about Jean right now is you only have a single cluster view and I think this is something that you may need to think when you scale and additional to this if you are thinking of doing cost management at scale right you need to also think about different other functionalities like setting up cost alerts having some right sizing recommendations because as we mentioned right sizing is again one of the most important and essential strategies to implement, right? So, you may need to have a solution that could give you these automated right sizing recommendations. Of course, you can always use some external tools to set that up, right? While you're using open cost, but again, it becomes pretty difficult for you when your infrastructure scales up, right? U one of the other examples I can think of is cost alerts. I've seen a lot of organizations work with Prometheus and alert manager, right? But I think there are some limitations to setting up Prometheus and alert manager as well. These are some of the things that I think you need to be aware of when you are getting started with open cost. The TLDDR version is open cost is a good start but it may not be the complete solution for cost visibility at scale. So the question is like what can we do and how can we go beyond open cost right? How can we think of cost management at scale? That's where we'd like to definitely talk about how we can build on top of open cost. I remember you mentioned that this is exactly what we are doing at App Insights, right? uh we are we are building on top of open cost and part of the aim of this particular series which is called as behind the build is to help you all understand what we are trying to do to solve this particular problem to solve the challenges of troubleshooting cost and observability and what we are trying to do in that particular space. So Joan if you could tell everyone how app insights takes open cost extends its functionality and just go beyond simple cost visibility if you can touch upon that particular point. Okay. If you're using open cost and start facing it limitations you can build on top of it or you can use another tool. There are multiple options available in the market. You can another tool that works to complement open cost and our tool app insights is one of those tools. As you can see here in my screen you can think hey this is exactly what open cost does right we have the breakdowns have the date range but we start to have some more features here. So for instance we have longterm storage you can go and analyze the data from multiple months ago. You can have the different breakdowns. You can go use filters. You can use uh grouping and you can look inside the workloads. So this is the workloads breakdown. We have the different types of resources here. If I go, you know, of those uh workloads, I can click here and I can see a whole lot more data that's going to be helpful for you to analyze and optimize your workloads. You can have the efficiency of CPU and run separately. H the current cost of running that workload and the right size impact because we provide the right sizing recommendations as well which open cost by itself is not able to provide. So you can see here the current request and limits for CPU and memory and the recommendation that we have for that. And we also show you how much money you are going to save if apply that recommendation. We show a range of cost because as you know we have the request which is the baseline but we have the limits as well. So the cost is going to depend on where in that range your workload stays during the time. So we provide this range in order to be transparent to you. And we also provide here the data that shows why this recommendation makes sense. We can see here there is a huge gap between limit and usage. And you can also see usage is below the request which is why we also recommend to reduce the requests for CPU. We have on top we have CPU and below here we have memory. So that's already a huge benefit from using appsides instead of relying only on open cost. But as you also mentioned before now open course I can only look closer by closer. If I have two clusters I need to run open cost on one cluster look at that data of that cluster and I need to run open cost on the other cluster and look at data of that cluster. In app insight we provide a single pane of glass in our overview and analytics pages where you can go aggregating the data across multiple classes and analyze that together. We also have the notion of cost alerts, right? Again, cloud vendors already provides alerts and limits at the cloud level. But here, what is the difference here? You if you have an increase like this one we had six days ago, we can go and understand where the cost increase comes from. which resources are driving the increase and which workloads are driving the increase. And again, you can just click in the workloads and see the details for that workloads and try to understand what's happening there. And moving on to the chargeback, we can provide you the data at multiple breakdowns. But most interesting, we can add additional data on top of the workload data. So for instance, if I have a breakdown by name space uh you can see here that some entries are classified as overhead and we have three kinds of overhead costs tracked in sites. We have the idle cost which indicates resources that are not used which comes from open cost directly. We also have idea of overhead name spaces. So if you have name spaces like cube system is a clear case right cube system is there in order to have the cluster up and running right but who is going to pay for cube system which team in your company is going to pay for that which team is going to pay for a CD which team is going to pay for the authentication that is used by all teams if that's the case so we enable users to specify by which name spaces in the cluster can be considered overhead name space. So I can quickly go here. I going to see all my name spaces and I can say okay so let's say this Argo CD name spaces cafa and also cube system they're all of head spaces I can add it here save then when I go to my reports again I'm going to see those name spaces here you see that markers overhead head and now it comes the interesting piece. I can now go and spread that cost within that cluster spread that cost of those name spaces across the different name spaces. There are different spread options. So we can see here that the crunchy database name space it the workloads run there only cost 120. But if I consider the overheads of running the closer and if I spread that cost equally across all spaces, I have additionally $3 for each name space. H and I can do that proportionally as well based on CPU or memory usage in each name space. But what this provides you is the ability to share the common cost across multiple name spaces for your individual teams. Like I said, name space like cube system. Uh in your case, Kafka is a shared name. Yeah, shared name space in sense that is used by all things that work in that cluster. I showed you already the idle cost which is a kind of overhead then overhead name spaces but you may also have external cost uh for instance the control plane that you need to pay in order to run a manage kubernetes cluster in your cloud vendor or you could have some tool that will run on your cluster that you need to pay a fee for using that tool. So you can add that as external cost here. You can specify the name. It can be a fixed month course or it can be a variable cost based on node count or CPU course. Yeah. So you can keep all that together within the same visualization those three overhead cost kinds are going to be shown in the cost reports. I can relate to that particular point because we talked about one of the challenges being lack of accountability. You know, teams use shared cluster resources and you can't really attribute that dollar value at the end of the month to certain teams and I think a breakdown like this can really help you to do that at the end of the day, right? And where is the cost exactly coming from and which team is using more than a certain team, right? So we can definitely make those kind of decisions if we have a breakdown like this. This is pretty interesting show. So I would love to know from you you know uh because we have we are talking about app insights. What's that one feature in app insights that you have seen most engineers find very useful in their day-to-day workflow when it comes to cost management. answering your question. I think that is the thing that make our customers eyes brighten the most right because we can right get some visibility from open cost. You can build your own tools to aggregate that and have some visibility across multiple cluster. But this flexibility we provide on chargeback model in order to see different breakdowns and provide the different visualizations of the overhead cost and how to share that or not share that depends on how you want to do it. Of course, you can always download it in order to analyze it with an Excel sheet or wherever you want to use. And we also provide an API. So we have our own cost API that you can integrate with and from that you can do whatever analysis and processing that you want in order to attend your cost management and phops needs. Amazing. Yeah, this is really interesting John. Thank you so much for sharing uh about app insights and you know I definitely agree that the more visibility you have then only you'll be able to implement the actionable strategies and at app insights that's what we are trying to do we are trying to give you the initial visibility that you need and then we are also helping you to actually do something with that visibility actually perform some actionable things to you know make the decisions which will actually save you cost. Thanks again for sharing about app insights and I think we are nearing at the end of this particular session. We have talked a lot about Kubernetes cost management today. A lot of insights have been shared but I believe one thing is definitely clear that engineers and companies should really start thinking about gaining more visibility first when it comes to thinking about Kubernetes cost management. And as we discussed open cost is a great starting point for everyone if you need that initial set of visibility and then you can think about implementing actionable insights right and then you can think about moving towards you know optimizing your workload and optimizing your infrastructure but gaining visibility is the first step there right so I think open cost is a great start and of course if you want to scale things up you can definitely try out app insights we have a 30-day free trial I'll give the link in the description as well we love your feed feedback when you try out app insights. But Joe, thank you so much for sharing all the insights uh you gave us today. Before I let you go, uh what's the one piece of suggestion and advice that you would give any engineer or a phops person if they are thinking about getting started with Kubernetes cost management? You already said it kal go ahead install install open cost. It is free is very useful. You start it today. You keep it running for a few days so that it can collect the data and maybe a week later you go back and you're going to start analyzing the data and it's going to be a great start point to use I mean to optimize your closes and save some money and yeah it was my pleasure. Thank you so much for inviting me. Amazing. Yeah, and we'll definitely be seeing you again in another episode of this particular series. And yeah, thanks everyone for joining and we'll see you in the next one. Bye-bye. Thank you. Bye-bye.