Understanding Kubernetes Cost Fundamentals | Cost Management Series Part 1

Why communities cost management matters now more than ever? What are the different nuances which are involved? What are the things that you need to understand? Where does the money come from? Right? Uh how is your communities cluster divided in such a way that is incurring cost for you. So this is something that we'll be talking about in this particular webinar which is going to set a strong foundation for other webinars that we are going to do. But just to quickly start with something interesting, I would love to show you all this particular interesting statistics. Okay. So when we talk about Kubernetes cost management, right? This is a pretty interesting statistic by sysdic which shows that out of all the resources that you have provisioned 69% of the CPU resources go unused. Right? That's a that's a really huge number. So you can think it that way that out of all the resources that you have provisioned 69% of the resources are you are not actually using those but you are paying for that at the end right so this is a good way to get started to understand why this topic is so important right but apart from this what I feel is increase in the adoption increase in because Kubernetes adoption has been increased a lot in the recent years right according to one of the reports of CNCF 84% of people of organizations using Kubernetes have increased throughout throughout these past three years which is a really huge number and I feel that when adoption increases infrastructure becomes more and more complex and it becomes more and more complex to manage that infrastructure and because of that because we are all humans because devops engineers platform engineers are all humans there are certain inefficiencies in the infrastructure which costs more and you end up seeing that huge bill at at the end of the day right so Rajit I think to quickly start off with the discussion the question I would like to ask you is according to you what's causing this surge in Kubernetes cost recently and why do you think it has become difficult to manage cost at scale I think we can talk about one from the perspective of an organization itself right who are seeing the bills at the end of the at the end of each month or at the end of each here but I think it's also important to talk about from the perspective of a DevOps engineer as well because those are the ones who are actually managing the infrastructure right so let's touch upon this point first okay awesome uh I think you've already highlighted uh one statistic and uh you know I want to you know go back to that same statistic which is a cloud in a secure report to to explain uh what the sort of potential for costsaving uh this actually um jives with what we've We've seen during consulting engagements and and when people started use our product the the data that we get from that there's a lot of unused resources uh and these are costing a lot of money and uh the reason why we are highlighting this it's like you said platform engineers um you know DevOps engineers uh they are the custodians of you know the Kubernetes clusters they provision clusters they manage clusters they take care of clusters and um so I kind of wanted to you know highlight to them the potential for them to kind of help their organizations to um you know save cost uh and and and then we'll we can chat about like you know some of the pain points and challenges and how we can get there right uh because at the end of the day so I actually sympathize with the platform engineers because on one end they have to kind of keep the uh applications running keep the developers happy uh in lower environments and at the same time there's a lot of pressure from the management to save cost right so what I really want to work here today is to empower and enable our platform engineers, our DevOps engineers to really understand you know what are the potential for saving cost but at the same time what are the fundamentals that help them gain that visibility so they can start providing you know better cost management strategies to their organization. That's interesting. So in the stat that you have shown here right as I mentioned 69% of the resources CPU resources have gone unused right and a couple of more statistics. So can you go over a little bit in depth and help us understand like why does this matter if you're a DevOps engineer and why does this matter looking it from the organizational perspective right yeah so if you look at most applications um they are more memory bound and less CPUbound but uh when you ask people they will you know if you ask developers hey how much CPU memory you need people would say that you know I need x amount uh but really the the the actual usage even in our own organization let's be honest CPU usage is very low most applications are memory bound and CPU is an expensive resource actually it's more expensive than memory uh so that's why this is an important sort of statistic to understand um I've seen a lot of wasted CPU and then we beginning to see a lot of wasted GPUs um you know when you look at the the aggregated metrics from our product across the board uh that's another one GPU also very very expensive resource. So those are things that people need to care about and also I think the the cost savings per number of nodes from smaller organizations to larger organizations actually especially in very large organizations there's lots of potential for um you know cost savings because um so many uh different clusters different teams and without having that overall visibility it's very difficult to figure out you know where your costs are going which clusters are inefficient which workloads are inefficient. So um this should kind of you know give the platform engineers a bit of a wakeup call to say hey this is what you're potentially dealing with at your organizations uh and then let's jump into the next section to kind of break down this and and figure out how we can start uh improving costs. You talked about inefficiencies right and I definitely agree with that point. So from your experiences in talking with organizations and talking with different teams, what are the different real pain points for engineers that you have seen that could cost you more from an organizational perspective? Maybe you can touch upon that point. All right. So let me walk you through a couple. Um so um let's look at these pain points. Um so first one is overprovisioning. Um I mean you know if you're a developer you will relate to this and if you're a platform engine you'll certainly relate to this right if you ask any developer uh worth their salt they'll like if you ask them they're like how much CPU memory I I can as much as I can get right uh and and that's also because um people are afraid of uh performance issues people are afraid of u you know um the applications u not performing so they error on the side of caution and end up giving more the there's another aspect to it. So a lot of organizations have right sizing guidelines um mostly t-shirt sizing right and and so people tend to air on the side of caution and pick something a little bit oversized and if there's no feedback loop and nobody revisits them now you have a whole bunch of applications taking up resources that they're not necessarily using it right um so that's important to kind of understand uh you know the developer mentality we have to sympathize with them but at the same time we have to empower them right and sometimes also I've noticed I mean you know we deal with a lot of observability right and when there are production issues sometimes the standard answer is what about the CPU usage what about the memory usage have you increased it uh but if you have that data then you can go back the platform engineers can go back and say listen this is what you're seeing right now that's not the real cause because I have seen any number of times that um people think that is the the answer to the question right so they'll increase the number of pods or increase the size uh but that's really not the root cause. So very important to understand overprovisioning. This is a number one area that you can see. Um underutilized clusters um we've seen this uh over and over. um we'll when you break down the cluster cost fundamentals we'll talk a little bit more uh in depth in this but generally what I'm talking about here is uh you have a cluster with x amount of capacity but your workloads are only using a certain percentage and the idle capacity of that cluster sometimes I've seen 30 40 50%. And you're paying for it your cl you know I mean it's reflected in your cloud bill. So uh reducing that idle capacity is um you know uh very important. So I would look at probably I will start here look at what you know underutilized clusters start uh improving the idle capacity there then look at the the workload cost. Um this is uh another problem. I think we already talked about this. Um you know without adequate guardrails if you don't have some sort of guardrails platform engineering practices um you know developers may be able to provision more than they should. Um and typically organizations handle this with you know kas and limit ranges and things like that. We're going to go deep into it in in a in one of the webinars in the future. Uh but this is a this is also a key challenge the platform engineers face um you know to kind of get the developers to uh follow the best practices because that has a direct impact on cost right and then finally um doant workloads we've seen this a lot in larger organizations where teams are now platform engineering has empowered people to sometimes spin up clusters you know they get their own name spaces where you can do whatever you want uh and and even if you have limits enforced uh like resource kota sometimes people do test workloads um you know they and then they forget uh they they don't shut them down and so now even if you have um you know whether it's carpent or cluster autoscale your trust is not going to scale back down because some workloads are hugging there but it's really not doing anything right so do workloads are a big problem for some very large organizations without that visibility without that oversight that's just eating up cost uh for no reason right so those are kind of like the main challenges that I've seen very interesting you mentioned about overprovision workloads right I think at the same time there's also problem of underprovisioning right you on one hand you end up uh you you end up provisioning a lot more than what you actually need but on the other end you may end up provisioning too little right that's also a problem so I think in that scenario datadriven right sizing is one of the strategies that a lot of organizations are now trying to implement. We covered all the real pain points according to you. What's the one most prominent one that you have seen that a lot of organizations is a real pain point for a lot of organizations. What's the most prominent one from this? I think across the board from startups to very large organization I think overprovisioning of workloads is the biggest problem. It is um usually it's the fear of what if something happens uh you know um so people end up overprovisioning just to get around uh you know in production. So I would say that's if you you know if the if you want to remember one thing that would be the one thing but also think about the idle capacity in the clusters as well and in in that case it's generally not the developers it's the platform engineers worried about even if you have autoscalings uh it takes time to autoscale whether it's carp and it's faster but still you need a little bit of burstable capacity so always try to you know like play on the side of uh caution so those are I would say two things from a platform engineer's perspective, think about idle capacity. From a a developer, DevOps engineer perspective, think about overprovisioning. Yes, absolutely. Yeah. And when we talked about underutilized clusters, right, it's not about what you are actually paying and what you're using, right? Because that's how the model works. The problem is that you're paying for what you are not using. So those unused resources, that's the main pain point. And we'll definitely talk about more strategies on how you can optimize so you can reduce that number of unused resources. So I feel that now you would have a good idea of why cost management is important now more than ever to talk about and we also discuss some of the real pain points that engineers are facing currently. I think Raj it's good to transition now towards talking about cost fundamentals and in this particular one we are going to help you understand where exactly is the money coming from. Can you help us understand like if I see from a cluster's point of view how can we break down the cost so that exactly know where the cost is going. Right. So if you take um a Kubernetes cluster uh you can broadly categorize into two parts. One is the cluster asset cost and one is the cluster overhead cost. Um the asset cost is the uh the resources you use whether it is um you know allocated before or or usage based and then the overhead is typically the operational cost. So this could include your you know uh control plane cost the cost of running the team etc. So let's dive a little bit deeper. Um now here if you take the the the asset costs there are like you know there's the the resource allocation cost. So this is where uh your your compute comes in storage compute um other things like load balances etc. Now very what is important to understand is once this is allocated to your cluster once that node is added it starts to incur cost whether you use it or not right so that's that's why you know kind of highlighted it in in um you know in red right the second part is usage bas so this is u network cost right egress cross zone uh cross region um some for some customers this can be for some users this can be a significant um you know portion of their bill as well. But this is based on usage. So the more you use it, that's only when you get charged. Then at the bottom is the cluster overhead cost. This is your control plane cost. Um you know if you're paying any licensing cost, right? Like uh this is important to understand because when you're thinking about um you know we'll talk about the you know number of nodes etc. This comes into play right? like should I use um you know four large nodes or I don't know eight smaller nodes right um and and you know the blast radius is higher if you have less number of nodes but you may be paying um a higher licensing cost because let's say your observability or something is charged on a per host basis right so it's kind of important to understand um your overhead cost as well and then balance that with performance and high availability and other things other indirect costs like your you know the cost of running the platform Right. Um and then the and it you can also in some ways uh in uh add the cost of common name spaces. So for example um let's say you're running um like a monitoring workspace that is shared across the rest of the um you know workloads right that could be typically uh added into your overhead as you think of that as well right so there's these are the sort of like the three things and you have to understand how your costs are going into these three buckets building that awareness will help you to now take measures to monitor and then take measures to um you know optimize that. So that's the cluster cost breakdown. Um shall we? Now the next thing is once you understand the cluster cost, let's dive a little bit deeper into the resource allocation cost. We'll go into the other ones as well. So when you take the resource allocation cost um now this is where it's further divided into two buckets. One is how much it is actually actively used by workloads and how much it is idle. So if you're a platform engineer, you got to keep an eye on uh this idle cost u because your idle capacity every cluster will need some sort of idle capacity right so you can't have 100% utilized cluster right you need a little bit of breathing space because even if you have autoscalers carpenter it still takes a little bit of time so you need to have that breathing space uh but if you have idle capacities like 20% 30% 40% 50% then it's something to think about. Right? So that's one area that you can uh make a big impact right away uh without compromising u you know performance if you see a very idle high idle capacity right and and if you haven't done any kind of autoscaling uh at the least get cluster autoscaler so that you can take some concrete steps to reduce idle cost. Let's look at actively used by workloads. That portion here also doesn't even though it's green like this doesn't necessarily mean it's fully optimized. This is where your overprovision workloads will make a difference. The the more you right size the actively used by workloads will start to decrease right and that means you can now your idle capacity will grow. So you can start shrinking the the cost. Remember unless you remove the number of nodes or resize the the nodes you're not saving on the cloud bill. So it's that's very important to understand. So the two things that will help that is a reducing the idle cost and right sizing your workload. So those are the two things to kind of remember in this segment. A quick question here I have for you Raj. We talked about idle capacity right and I feel that this can be regarded as one of the hidden costs which are sort of hidden from the teams who are working. So according to you from your experiences what are the common mistakes or surprises that you have seen teams face when we talk about these hidden cost? Uh yes so I it's I think mostly comes from a ei the lack of awareness or lack of visibility um or both. So understanding what your idle capacity is kind of important you know I mean at the least use Prometheus to see what's what's going on right like a you know so scrape those metrics see what what is going on so I think that can help you you know build that awareness about the efficiency and and idle capacity yes so we now know that how are the costs broken at a cluster level but I think let's take a deep dive and talk about the container cost breakdown on it. How is the container comprised of and how is it costing you and how it adds to the overall build of a particular cluster. Right? So when we are trying to right size uh and you know reduce the actively used by workloads portion of the big cluster thing it's good to understand uh kind of what's happening at the at the container level so that we can rightsize. Um so if you take a a typical container you giving it some resources CPU GPU uh in some cases the memory disk uh and that's your um allocation uh cost and then usage based on actual usage um you know egress cross zone cross region um now the reason why I'm kind of highlighting the network um cost I'll give you a quick story um I actually did a LinkedIn post on that. It was a smaller amount but compared to what we were paying for that cluster the network cost was more than 15%. uh because we had a Kafka cluster across three availability zones and um the majority of the cost came from just cross zone traffic because there is a cost associated with cross traffic right uh so it's important to understand that measure that uh and keep that in mind because most people understand CPU GPU and memory right um right you have to rightsize the container that's the most fundamental piece if you rightsize the container um then you start to um write you know the the trickle effect will help rightsize the cluster. Now um there's two sort of dimensions and we'll talk a little bit about that as well. You know you rightize the container and then you right size the number of replicas for it as well. That is very important. Uh and once you um understand this u the measuring the uh uh the we'll actually come to this part first and then we'll go back. uh once you understand how much you are giving in terms of requests and limits um from from a container perspective that will really help you to uh rightsize this container. So let's um you know everybody understands about the the limits and requests part. So um you know the limit is u what the um you know container or the pod you know sorry the container can go up to and the request is what the kubernetes system will guarantee you right if it can fulfill the request the workload will not get scheduled right now what we often see is that the usage is actually lower than the request right and and that's where the efficiency calculation comes in if your efficiency is low um you know then there's some like say 30% 40% 50% 60% something to think about right um if if your workload is constantly breaching the limit I mean getting close to the limit then it's a different story you you need to uh you know upsize it but majority of the workloads that we've seen based on the numbers that we you know see from our cost management solution um usage is less than request the consequence here is remember I said Kubernetes will try to um get you that right otherwise you your workload will not your pod will not get scheduled. So what that means is if you don't have capacity in the cluster and you have idle uh sorry cluster autoscaler setup or carpenter setup then the cluster will scale up now you're actually incurring cost um and your workload is not using that or you're actually taking up space from some other application that actually may need extra resources. Right? So uh very important to keep this metric in mind. If there's one thing you're going to measure at the at the workload level, look at how efficient your workload is, right? And you would find um a lot of the times people fairly okay with memory but pretty poor when it comes to CPU. I've seen 5% 10% uh 1% CPU um you know efficiency because people give very high requests. So it's important to understand. Now coming back here let's a little bit explain a little bit further here. CPU is a compressible resource right and memory is non-compressible. Meaning that with CPU um you can overs subscribe but what's going to happen is you'll start to get throttled right. Uh with memory um you know if you don't have uh memory and your application starts to consume beyond the limits you're going to get killed. So that's also kind of important to understand. One thing to remember, look at the efficiency. If your efficiency is low, take action and rightsize it. Um, and you know, I mean, if you're worried about uh variable uh traffic patterns, you could potentially handle that with limits. Um, because otherwise you're just and then that also helps the cluster to autoscale properly, right? Um otherwise you're going to just pay for um you know unnecessarily for for the resources that you're hogging. Yeah, one quick takeaway that I got is at the container level, you need to measure your efficiency, right? That's the one of the most important metrics you need you need to track in order to see how much is the container actually using and compare that with how much you are actually have provisioned, right? How much you have allocated, right? I think that's the most important metric to track. Right sizing again fits into this this picture as well. You already have a slide of number of replicas which is one of the ways you can right size. If we can quickly touch on this topic as well, how does the number of pod replicas matter too? Right. So the second dimension of sort of workload level right sizing is once you right size your container um you have to think about the number of replicas. Um if your container is inefficient you're going to double like you know I mean you're going to multiply that by the number of replicas right so that's the impact it has. um for high availability you have to run at least two replicas kind of like especially in production that's kind of like the minimum recommended um you know best practice but sometimes you will need to have more and uh I've seen sometimes people go with a static pole at the least consider using HPA um with Kada you have a lot more control on how you scale so always have to think about the number of uh replicas as well because if you can that way you can reduce cost but at the same time balance performance because the cluster uh will autoscale based on the um you know the um the number of replicas it will start to autoscale when the demand rises right so think about not just right sizing the container try to figure out what might be the best way for you to figure out the optimal pod count range right if you're going beyond the two pod limit think about HPA or kada because otherwise you will air on the right you know side of caution and give more than you need. So this is an important one to think about as well. Rajit I think we are at a good point now because we now have understood why cost management matters now more than ever. We have also understood how the cost is distributed throughout a Kubernetes cluster from a cluster perspective and from a container perspective as well. Right? I think we're at a good point to have more visibility. So what are the major cost drivers right in Kubernetes that engineers need to understand and because I think we are talking about mostly about managed Kubernetes I think we can talk from that perspective so maybe you can touch upon that point uh broadly the three cost drivers are compute storage network right so let's talk about compute cost which is what this question is about right there's a direct impact based on the number of nodes the size of the nodes and the type of the nodes that you use right uh and and there are different ways to think about it too. So one is exactly what this gentleman asked can we use spot instances or preemptable instances right there is significant cost savings in using those um but there are tradeoffs right um some workload so your mission critical workloads you probably don't want to use it right so we use some spot instances so they are great for you know jobs that like could be like a you know specific job that can recover um you know so schedule your um you know jobs uh chron job type of uh stuff your event driven type of applications that are not super mission critical that can tolerate um you know disruptions because what happens is a spot instance uh can get cleaned up any time right so if you're running like some important u you know banking type of transaction you probably don't want to use spot instances right u but there are plenty of other workloads that can tolerate disruptions those are great and you save a a good amount of money on that. The other thing to think about is the type of node, right? Um so there are every cloud vendor offers multiple classes of virtual machines. Um you know balanced nodes, memory optimized nodes, CPU optimized nodes, you know nodes that do well with GPUs etc. So having different node pools can help. So in that particular case using node poles and using ts and tolerations you can do it. So for example for what this gentleman asked now in the you know you can use ts and tolerates. making sure that the um the workloads that can tolerate disruptions get scheduled in the uh the preemptable nodes or spot instances and the others get on the on demand nodes or or if you have very memory intensive ones they will get scheduled on the memory intensive nodes and the others on the uh you know like your balance nodes or whatever sometimes there's also uh unexpected cost that comes so we had one instance from our own experience we can say we in We used a group of memory um intensive or sorry memory optimized nodes but there was CPU was starting to get throttled right because they had low CPU so you may have hidden performance issues there's also cost implication because in one of the set of nodes that we use there was burstable CPU capacity suddenly there was a there's a line item for extra CPU in IWS bill why because it it it you know the CPU was not enough so it it used that burst. I forgot the the exact type of instance we use. look at your cloud bill also to understand um you know the impact of the type of node you use right um so we talked about the type uh we talked about the sort of like the size number of nodes uh this is again it's a trade-off between performance high availability so I already mentioned let's say you can run four large nodes or eight smaller nodes from a high availability perspective if one node crashes your blast is small right because only that goes away But maybe from a licensing cost perspective it could be higher. So you have to like you know because if you're paying per host right some monitoring tools including our monitoring tool uh it's charged on a per host basis um that's kind of like some of the standard practice out there. So it have to balance between cost and performance and high energy. So all of those are trade-offs. You can't really optimize for every variable. There's a cost of running your business. So you have to think about those aspects as well. The other thing to kind of think about is using variable node sizes. So if you just use cluster autoscale, you will always notice the last node that gets scaled up is always underutilized because it's not um you know because the the node you can't shrink the node size, right? But with carpent you can have variable node pool sizes. So u it can do better bin packing and we'll we'll dive into these strategies a little bit more deeper in the the the follow-up webinar. But those are the ways that you can optimize your uh compute cost because that is a significant cost uh you know in your in your Kubernetes cluster storage cost. What we normally see is two sort of two things. one is um people don't pay attention to the class or the type of um persistent storage and the second is um massive volumes um people just you know get large volumes and it's using like I don't know 10% 15% of it most cloud vendors now allow to dynamically resize um I mean you probably don't want to like you know keep it very close because your puzzles run out of disk if you you know like database let's say you're running a database or Kafka cluster things like that but um I mean don't um you know oversize it there are diff like you know you can definitely resize in most cases um and then also um you know uh in some other cases you probably need to think about SSD and other things a lot of people are running data um uh data science workloads these days on Kubernetes um and it can add significant cost if you have very massive persistent volumes provision but not cleaned up after. So unused persistent volumes could be another source of um uh increased cost. So then you need to understand the often often volumes unused volumes and and clean up uh because once the workload goes away right there are other things you can set to like um for certain workloads people don't want to uh set the value to to delete it when it's no longer being used because you you need to retain the data but pay attention to those settings. Some workloads you don't really care once it's done you need to clean up. So unused volumes can be a significant source. network cost. Uh I already mentioned that this can be a significant source uh of uh cost for you and understanding this breakdown, right? Like for example, if you have a Kafka cluster deployed on a three availability zone setup and you put your brokers on the three availability zones, amazing for a high availability. But if the cluster is very busy, the cost so will the cost start to ramp up, right? because that um the the cross uh zone traffic is charged. Um I'm sorry I forgot the the exact cost if you remember let me know. I believe it's one um uh probably 1 cent per gigabyte. I could be wrong but just double check and let me know. U the cost is also um we we'll write an article on this as well. Uh it'll be charged from going out of the zone and coming into the zone. You get charged twice per gigabyte. uh pay attention to that. If you're running um you know higher and then if there's a lot of chatter in your cluster um and you have multiple microservices try to group your more frequently um you know like the microservices that are that are talking to each other try to peg them into a a zone and I know it'll reduce some high availability but it might um reduce your cross zone chatter that can help you with the cost because it can be significant depending on the amount of traffic um that you're generating, right? Egress cost is what you're serving to your customers outside and that um hopefully it's the um you know, if that goes up, it's good sign for your business because people are using it and you have adequate mechanisms to kind of tie that to revenue. But egress cost is also something you need to understand and keep an eye on because u you you're getting charged for that, right? Cross zone um cross region traffic is usually when you're backing up things uh to a different region like let's say you have a database replication that's for high available backing up to a region but understanding that these are the cost drivers is very important when you look at the cloud bill sometimes a little hard to understand and and that's where some of the vendor products are helping having this awareness is important because if you get a big network cost bill you need to figure out is it because of egress is it because of cross zone is it because of cross region then figuring out okay which workloads are actually contributing to this so then you can go we were able to track it down and this is coming from the the Kafka PL then you can take adequate you know mechanisms to reduce that right otherwise um it's a bit of a um you know blind spot a bit of a black box and you you can't um optimize the cost right yeah I We talked a lot about we started from understanding why cost management is important to talk about today. We then talked about how the cluster is divided and what's the cluster cost breakdown. We understood because that initial visibility is really important. You can't again coming back to the code you can't optimize what you can't see. Right? So we saw it from the cluster perspective. We then saw it from the container perspective. We understood that one of the most important metrics to see from a container level is efficiency. Right? You got to measure efficiency. You need to see how much you're actually using and how much you have requested. Then we moved on to the major cost drivers, right? Network cost, compute cost and the storage cost. According to you, if you could give us three major takeaways from this particular session that someone can take this knowledge and implement something in their organization, what would those be? Yes. So, first one is um you know, understand your cost drivers uh for your kubern uh cluster. Think about the three buckets. Uh build visibility awareness into these three buckets. So that would be number one because that will help you build build that awareness. Understand um and promote this awareness. Uh so platform engineers can also promote this awareness within their teams giving people you know the their developers the engineers understanding of how the container cost you know container level cost breakdown because right sizing starts here right and then efficiency right because the the number of replicas will somewhat like you can minimize the impact of that if the container is right size. So think about the breakdown and then understand how efficient it is. So if if you even if I break it down just two things under on the cluster level understand the the breakdown and the idle capacity and then here and try to figure out how efficient your workloads are. If I'm giving request, I need to at least get very very close to that request if not exceed it if you can, right? Um otherwise you're unnecessarily paying and and and and if you see inefficiencies here, this is an important um place to keep an eye on that will help you reduce your cloud bill. So those are the sort of like the the three things that I would uh leave you with. Understand your cluster cost breakdown. Understand your container cost breakdown. Understanding the efficiency of your uh workloads, your containers. Closing thoughts. Yeah, you can optimize what you don't see. Uh so build that awareness. Uh understand the cost fundamentals and then at the least use Prometheus. Uh there are tools like open cost in this vendor products. Um but you should be able to start at least with Prometheus to understand these things that I build that awareness. I you'll be surprised how many teams don't even have that basic visibility. Build that awareness, build that visibility. You will that's a great step compared to most teams out there. Then you can from there onwards you can build your cost optimization cost management practice.