The eBPF Shift in Observability

Alrighty, I think we are live. Hello everyone. U welcome to the seventh episode of observability unplugged. Super excited to have you all here. My name is Kunal. I'm working as a devil engineer at Randoli and I'm joined by Rajat, founder and CTO at Randoli and this is very exciting because this is the first episode of the new year. So happy new year to all of you. I know we are in February right now but still we've been doing these episodes consistently. We started last year and we've already had six episodes where in each episode we talk about a certain topic related to observability AI because there's just so much happening in the AI and observability ecosystem. So our main aim with starting the series was to come on and uh you know bring a guest from who's active in the ecosystem and just share the information with all of you because um it's always good to share value and it's always good to keep you all updated right and this one is special because we are going to talk about ABPF and something we are calling if you have read the title we are calling this as the ABPF shift in observability because uh what it means in practice we are going to talk about that in the recent years there's been a lot of uh discussions around EBPF's use cases in a lot of different domains right it started with networking then we talked about security but today specifically we are going to talk about how EPF is being used in the observability ecosystem and what value it actually is delivering and we've got a very special guest Bill uh thank you so much for joining us in in today's podcast would you like to tell everyone a bit more about yourself and what you're doing these Sure. Thanks for having me. Um, I'm Bill Mulligan. I work at Isovalant which is part of Cisco and I'm a maintainer of the Selium project which is a graduated project in the cloudnative computing foundation actually the second fastest moving project in the whole CNCF and I'm also a maintainer of the ebpf.io website and really involved in the EBPF ecosystem through the EBPF foundation, sitting on the governing board there and uh working a lot with the people to make EVPF um kind of the strategic infrastructure technology for the future. >> Amazing. Yeah. Uh super excited to have you Bill. Thank you so much for taking out your time. Just before we start, I see a lot of people are starting to join in. We are still early in the session. So, a few housekeeping things for all of you. We're live streaming this on YouTube and LinkedIn. Um, so either you can join from either of the platforms, but if you want to chat with us, uh, I would prefer if you can also join the YouTube stream as well because sometimes LinkedIn lags while you chat. So I wouldn't want you to have that kind of an experience. Um, there this is the link to the YouTube live stream. Feel free to join this one. Um, and yeah, feel free to ask all of your questions if you have related to this particular topic. We'd love to take out questions and we'd love to uh take out some time after the after the initial questions that we both have uh to take your questions related to EPF, how are using it today and if you want to get if you want to get started with EPF, those sorts of related topic. All right, I think we can start right. Um so before we get into EVPF itself, Bill, um I think it's worth taking a step back for a second because this quote unquote shift that we are talking about, right? It it didn't just happen in a snap. It it sort of uh it's years of change we have seen in how we are building and running systems. Right. So when you look at cloudnative systems being built today, what what has actually changed according to you that has sort of made the old and traditional ways of doing monitoring just stop working the way they used to previously? Yeah, I think there's a lot of different changes when we're kind of moving to a cloudnative infrastructure. I think the biggest one is just like the speed that they're you're moving at. Instead of, you know, having really long live VMs or bare metal machines, suddenly you have containers. I think there was some stat statistic from Google they're like changing a billion containers a second or something. So there's a lot more churn. Things are a lot more ephemerrable ephemeral. You can't just kind of like pick thing and say it's going to be the same the next year, the next month, the next day, even the next hour. So, it's how can we deal with a lot of change and churn in the system in a scalable and repeatable way without completely losing our mind. I think that's kind of like the key thing in cloud native is that the pace at which things change is increasing. And so with that comes a whole new set of challenges. And so I think that's where a lot of the cloud native tooling came from is things are moving faster. How can we keep up with this pace of change? >> Yeah, I totally agree with that and I feel that you know a really good example for this is the CNCF landscape itself right from the day it started to now what it is. Uh I mean I still remember whenever I present about the CNCF landscape I've given a couple of talks on this topic. it takes hell of a lot time to download it now and just to put it on the slide itself right so it's so huge right and it definitely makes sense systems have become more dynamic more distributed and that's why I feel because we're talking about observability right we are now starting to see EBPF added in that conversation so before we talk about specifically in the observability lens because that is what we're going to discuss I think it will help to do a quick primer uh for people because there might be some people who might not know what EVPF is uh who might have just started working with EPF but they would like to understand it a bit more right so there is a lot of different use cases about EBPF now but at its core what was it originally trying to solve when the project came into being if you let's say had to explain EBPF in very simple terms uh what problem was it designed to address as a project in the first place >> yeah so eBPF as a technology is about 11 years old now. And the problem I was trying to address at the time is Linux at the time was 25 years old. And as you kind of progress in technology, you have a lot of things and change becomes harder um to do things. And so the Linux kernel kind of slowed down. You know, it's deployed on more than a billion devices worldwide and everything from a satellite to a phone to the a lot of servers in your data center. And so the question is when we're deployed on that like how can we make sure there's stability across all these different systems we need to run it and with that kind of the the pace of innovation kind of slows down. And on top of that like a lot of people aren't running the latest kernel in their environment. And so you're not able to kind of change the fundamental operating system that you're working with as the pace of technology is increasing. And so you kind of have these two things pulling in opposite directions. things need to move faster, but we can't innovate as fast as we used to anymore. And that's kind of where EVPF stepped in. It brought innovation back into the kernel. And the way that it does that is it allows you to reprogram the kernel to add new functionality that you need. And so at the time, some of the top use cases are right were this is kind of like the advent of streaming, more data coming through. Actually, one of the early adopters was um Netflix. It's like how can we process more packets and how can we get observability into all the data that we're sending across the internet. Another one Facebook, you know, we're getting a lot of memes, a lot of cat pictures. How can we, you know, get the raw throughput um increasing uh in our system? And another one was selium. It's like how can we make networking native to the cloudnative world? And so looking at kind of all these, what EVPF allows us to do is to insert programs into the Linux kernel to change functionality on the fly. And the example that a lot of people usually use um is EVPF is to the kernel what JavaScript is to the browser. So if I take you back a couple more decades back to, you know, a classic 1990s website, what you have is this static page where you're just reading off information. And that's kind of like what the kernel was a decade ago, you know, like here's the functionality and this is what you get. And now suddenly with JavaScript, you're able to add functionality. So as a user interacts with the web page, you can do things like click the button, zoom. You know, it's interactive and it changes based on input. And so that's exactly kind of what EVPF does in the kernel 2. As new inputs come in, programs are run and functionality can change. You know, another example I like to give is like the the app store, right? Before you used to get your mobile phone, uh you'd flip it open, type some numbers, and it just had the functionality that it had, right? But when the app store came along, it suddenly you could add new programs to do things that you want to do. Be like, say now I want to get from A to B. Well, I can open install Google Maps, open it and look at the route and then I can install Uber and call a taxi. You can so you can add new functionality to an already existing piece of hardware. And that's really cool because rather than waiting for these long innovation cycles in the kernel that can take a couple years to come to consensus and then a couple more years until you actually deploy that uh agent of kernel into production. You can have an idea, you can write a program and you can deploy it into production uh all in the same day. In fact, there's even a talk from Brendan Greg who's one of the kind of early innovators um of EVPF and he was talking about how customers would ask him questions and by the end of the meeting he could come up with an EVPF program and answer their questions about the systems that they had, right? So instead of years long innovation cycles, it brings it down to, you know, weeks, months, hours, even the the span of a meeting. And so that's what makes EbF exciting is it's bringing innovation back in the kernel, which is kind of like the fundamental thing that we're all running on. And when we're able to do that, we're able to add new functionality. And what that unlocks is creating, I would say, like a new era or new generation of infrastructure software. We can add the functionality that we need on the fly in exactly the way we need. Um, and I I think that's what makes EBPF so exciting for me and why a lot of companies are starting to bet their whole kind of strategic um choice of platform of the stack and technologies um that they're building, the projects, the products, the services, the tools that they use on eB. >> Yeah, that totally makes sense and I think that's why uh because EPF is so foundational, it works at the kernel level. It's we are seeing use cases in so many different domains, right? We are seeing networking use cases, observability use cases which we're going to talk about today. We are also seeing use cases of it in the security aspect as well. Right? And there's a very interesting comment by someone saying kernel man-in-the-middle observability agent to avoid application instrumentation. It's interesting. Yeah, we are going to talk about instrumentation of VBPF as well. But for people who are new to EBPF and would like to know more, there is a very interesting film that was released by the community on EBPF and this sort of takes you through the entire journey of uh you know how EBPF was formed and what was the idea behind it. So we'll you know make sure to check it out. It's on YouTube. It's free to watch by the way. All right. So now that we I hope the audience now is aligned a bit on the idea of EVPF and why it came into being. Now let's talk about a bit from the observability perspective. Right? Because that's that's what we are going with today. Right? So in conversations around EBPF being used in observability right the first angle that I've seen come up in the discussion is around EVPF based instrumentation because instrumentation is sort of the first step towards gaining observability. Right? you have to instrument applications. So what actually has changed here if we compare to the traditional way of instrumenting applications according to you how is EVPFbased instrumentation a bit different because everything starts from here. >> Yeah. So I think this is a little bit what the LinkedIn user was commenting about. Uh in the traditional way we need to add instrumentation to our applications that may be requiring changes to the application code. But with EVPF, you're actually looking at it from a totally different way. Um, so instead of having to instrument the application code, you're actually instrumenting the kernel itself. And this is really cool because it doesn't require any changes to the application code. And the other thing since kind of like the the kernel is the thing that all the software on top is relying on to be able to actually access the hardware underneath. Um, what you're able to do is to see everything that's happening on the machine. you're not there isn't like oh this part of our application we couldn't instrument it's like no you've instrumented the whole kernel so everything that you want to see you can see and I think that's kind of the exciting part it allows you really deep visibility it allows you to have complete coverage of whatever you want and it also allows you to have very granular visibility too >> yeah that makes sense yeah and I I think the because ebpf is we are talking about low-level signals right with epf because so close to the kernel. It gives you that what you mentioned, it gives you the deep visibility uh without entirely depending on the application instrumentation itself, right? And it gives you a bit more than that. So from a practical standpoint, right, when we are talking about moving things to the kernel level, what kind of things are there that actually makes the know the teams visible? What what kind of visibility do we get that we couldn't really see before? if you can give some examples here. >> Yeah. So I would say this is um both the power and the peril of ebpf is that you're actually hooking into for example the CIS calls that the application is making to the kernel. um you're looking at very specific points like open this file uh connect to this socket um read this file right so it's you can pinpoint things specifically but that's a huge amount of data that you can have and actually looking at the comments right now there's one from Derek and he's talking about I'm curious if evf has improved at all on the performance front is a bit of a bottleneck to leverage at scale years ago and so right really powerful you can see everything you can have complete coverage you can have really fine grained ability to look into that, but that's also dangerous because you get a lot of data, a lot of information back from that. So, it's a superpower technology, but you need to use that power wisely. Um, I don't know exactly what Derek's kind of like use case is um or like in what kind of sense he was looking at it, but if I had to guess um where he ran into performance issues is that he was trying to collect too much data, right? And so, you can have this really granular thing. you can see every single system call that's happening on your host, right? But in a lot of cases, you don't care about a lot of that data. It's I think this is where people need to EVPF is a very low-level tool. What you actually need to do is you don't need to have data, you need to have insights. And EVPF gives you data. You need to have the ability to turn that into insight. Whether that's through a project that you're using like psyllium like this is how we should do routing in our network. Whether that's observability is like only we're only looking for this specific things. Whether it's security being like okay we only want to monitor these specific files rather than every single file access on the system. Like I think this is what people probably misunderstand about ebpf. They think it's like oh wow it's like this like perfect solution that solves all problems because it it's super granular. But data is very different than insights. Um, especially in observability, right? Like I I think we're kind of past the point where like more data is better in observability. I don't think anybody's really trying to make that argument. What people really want is like what's going wrong on my system? Why is it consuming too many resources? Why are we having cross zone traffic? They want to have the actual insights into their system. They don't really care about the data. EVPF is just a tool like a means to that end. It allows you to really dive into this granular data, but then you need to be able to use that data to draw out the information that you actually need. And so I think EVPF is this powerful tool, but you need to use it wisely. >> Yeah, 100% agree with that. I think um that's that's one thing I'm very sort of, you know, adamant about that u there's no shortage of data like you said. In fact, I think we have too much data uh all around uh within observability itself. It's uh you know gathering those insights and presenting it uh and finding that right balance even with data collection too right uh it has made everything easy to collect data but I think we're sometimes collecting too much data to the point that it's detrimental to the the applications that we are running etc. So that's that's a great point. I I love the way you explained it. So thank you. Yeah, definitely like more data doesn't mean better data. There is like better types of data, right? That's EVPF. You can really do fine grain instrumentation of just the things you're looking for, but just because you have more of it doesn't mean that you're going to find the solution to your problems. >> Amazing. Yeah, thank you for sharing your perspective and we are getting some interesting questions as well. Uh I think we can spare a few minutes after the discussion to answer a few questions but keep those questions coming folks. It's really interesting to see >> on those questions. I think especially when we hit a particular section as well instead of waiting till the end. So um I I I'll I'll help out and I'll point out if u you know some of these things are actually probably good when we talk about observ itself so we can get Bill's perspective on those. Yeah, >> absolutely. Yeah. Thank you so much for sharing your question folks. Okay. So we talked about we have talked about you know how ebpf can uh you know how epf can help in the observability aspect right u how it changes conceptually especially once visibility moves into the kernel level right I think you gave some really nice examples there Bill I think things become a bit more real once we look at kubernetes specifically because today we see that's where most teams are running their entire infrastructure right it's either Kubernetes or it's a combination of let's say Kubernetes and VMs and but the center component is always Kubernetes right and that's where most of the companies and most of the teams are looking to build their observability solution on so Rajit I think this may be a good place for you to jump in and you know take us on the Kubernetes side of the discussion >> sure yeah um I think when Bill started you mentioned that um especially in the Kubernetes world um there's there's a lot of things right so Um in fact I saw one of the CNCF service that said um the average lifespan of a container was um you know a couple minutes maybe probably less. Uh so there's a lot of things that come and go and especially from a security perspective uh and a traceability perspective for for other performance reasons or debugging troubleshooting perspective you need to be able to track these in a in a sort of a scalable way and uh whatever we had up until that point wasn't great. So do you can you elaborate a little bit more um about the the security aspect and sort of like the network tracing aspect? uh because I thought those were the for me when I talk about observability though I consider you know security and sort of like the visibility um as part of it right I mean the observability has sort of you know the scope has kind of grown right it's not just you thinking about performance or monitoring so could you elaborate a little bit more about the um you know the security aspect of it and the traceability aspect of it because I think that was one of the first use cases that I saw people getting super excited excited about uh when it came to EVPF. >> Yeah. So, EVPF um I think observability, security, and traceability, they're all kind of like interrelated and one doesn't always lead to the other. Um with EVPF, right, you're able to on the security side like right like the interesting thing is you're able to very finely tailor kind of what you're looking for. So rather than being like I want to look at every single file open um I want to look at just these specific security relevant files. So is somebody trying to ask for access like the passwords file right I probably want to like know about that especially if it's like you know in unknown application. So with with EVPF, what's really cool is you can like really tailor down the observability information that that you're doing and also being able to be in the kernel, you can see which process is trying to do which thing. And so by having all that like kind of host level information, you have really detailed information. It's this process is trying to open this file, right? That that's a lot lower level than like okay th this application is calling to this other application. You can actually trace it back to actually the parent process. you can do things like process ancestry trees and then actually seeing what each of those processes processes are actually trying to do. Um, and then yeah, like how EVPF actually functions. I guess this is going back to what we were talking about before. It's it's a tool like being able to use it for higher level use cases like observability, like security, traceability. That's when you need to turn this tool into an actual project or product. Like I think very few people in this world are going to be writing their own EVPF programs and deploying them in into production. The people that are going to do that are companies like Meta, like Netflix, like Google that have their own kernel teams that are actually probably contributing patches upstream. Um, most people in this world are going to be consumers of either projects or products that are leveraging EVPF as the underlying tool that allows them to achieve the the outcomes that they're actually trying to do, whether it's around observability or tracing or networking, security, scheduling. And if you're doing that, you really need to evaluate how the tool is actually using EVPF, right? Is it trying to do everything or is it trying to do just the specific things that you want to? And I think that speaks a little bit to like Derek's comment too is the actual implementation, right? So here's the outcome that I want. I want better observability. I want better security. And how do we implement that? Well, if you use EVPF as this powerful tool, we can look at everything. Yeah, that's going to cause a lot of performance overhead. Same thing with observability. It's like I want to observe everything on the system that's going to cause a lot of performance overhead. Um but if you implement it like I care in my environment is extremely IO intensive right so you're just looking at those specific things what are the few things that we can monitor to understand that you can use EVPF to really dig down to the granular detail for that but uh if you're trying to like I'm trying to look at what every single process on my system is doing yeah it's going to cause a lot of over overhead so a lot of it is about the actual implementation that the project or their product is using ebpf is just a tool Right. And um yeah, that's that's a great great point. So I I mean if anybody who's looking to kind of you know leverage some of the capabilities that EbF provides then they should be looking at things like Psyllium or Cubescape that is um you know that's using uh from a security perspective. Uh there's a ton of other projects that uh ends up using you know ABPF as a as an underlying technology. uh but people should be looking at these projects that are you know making it more I guess end end user friendly that's built on top of EBF right yeah >> yeah absolutely >> yeah um so my my my next question is actually it's a bit of a sort of counterpoint to what what Derek's talking about right so you know you can't get observability for free right so that's one of the sort of like my one of my pet peeves um any kind of instrument strumentation is going to impose some level of overhead. But what I would argue is that the overhead that comes from EVPF might actually be less than what some of the uh you know uh agents that are running you know in in in user space right so whether it's a JVM instrumentation agent or um some other code level type of agent um I would argue that you know um ebpf would u an ebpf approach would have less overhead is that always the case or am I off base? I would love to get your thoughts on that. >> You want me to dive you want me to dive into it? >> Yes. Yes. Let's let's let's >> always the best solution. Uh I don't know if I'll be able to answer that definitively, but I think maybe the part that we did skip over earlier is talking a little bit about the architecture of EVPF and why so many people kind of love it for use cases like observability. So what ebpf is doing, so you're writing your ebpf program in C, in Rust, in Go, in Python, and you're loading it into the kernel. So there's kind of two steps here. One, there's a verifier that makes sure that the program is safe to run in the kernel, um that it won't crash, otherwise harm the system. This is really good so we don't have something like the CrowdStrike incident, if people are familiar, where they took a lot of the world's it offline. Um and then it goes through a JIT compiler. And what this compiler does is basically turn it into machine code that's going to run basically as fast as natively compiled kernel code. And so if people aren't kind of familiar with dealing with kernel space and user space and what the difference is basically the closer you can get to the hardware like in general the faster your your code is going to run. And so by loading things into the kernel um you're going to your program will in general run faster. And the other thing that it's saving is what applications in user space have to do. So things that aren't running in the kernel, they have to transition. They have to make sis calls from user space into kernel space being like, "Hey kernel, can you go do this?" Whereas the difference with ebpf, it's actually running in the kernel itself. So it's pulling the information directly from the kernel. All that's happening kernel space and you don't have the overhead of being like, okay, I need to this is happening in the kernel. I need to pull that information from kernel space into user space. So this whole like transition and the overhead of that you're also skipping two. So based on the architecture of EVPF um like one through the JIT compiler to kind of running natively in the kernel you're going to get a lot of performance benefits from that. And then the other interesting thing is uh like once again going back to the implementation it's not just do you have EVPF check it's like how are you actually using EVPF and I I think this is one of the cool things that we're doing right now in silium and also in Tetragonon is you can be really smart about how you're actually using VPF right and I think that's the cool part so it's not just ah it's like this natively compiled like kernel code that runs really fast it's like you can also do other things like so some things that we do is For instance, for security data um in Tetraon, you need to pull like relevant alerts from kernel space into user space, but you maybe you don't need all the alerts or you can um you can aggregate them together. So you can look over time. So rather than always pulling information immediately, you can group them together and send it to user space. So it's not just this kind of continuously running process. And by that we're able to lower the overhead that was already extremely small even more. So it's implementing not just implementing ebpf but being smart about how you can implement ebpf too. And so I think the cool thing about running in the kernel is that there's a lot of optimizations that you can make once you're running in kernel space. You can do smart things with the code too. Um, so I can't answer it's always faster, but the cool thing about kind of instrumenting the kernel is it gives you a lot of flexibility that things that are running in user space don't have access to one in terms of like how fast they can run the code, but then actually just fundamentally can we discard some of this data? Can we filter it? Can we aggregate it? And do those things in the kernel before you even have to make this transition from kernel space into user space too. >> Awesome. You know that that that's actually that's actually a great point. And I just um just wanted to kind of bring that up because that my my own experience shows that um the EVPF stuff um like you know the EVPF approach is generally faster for the for the very reasons you kind of outlined. I just don't want to make a general statement like you said but um certainly that's what I've seen. >> Yes, I have too. >> All right. Um, I I've got a follow-up question, but I'll wait until we get to sort of like the challenges part. Um, it's very related to this. Uh, but I would let Kunal take over and, you know, talk a little bit more about, you know, um, uh, how EBPF is now being leveraged in observability especially with, you know, with hotel in the mix. So, Kuna, why don't you take take it away from here? >> Sure. Yeah. I I just want to say I was loving the discussion and love the way Bill answered the last question very very grounded way but uh because we were talking about traceability right there is just one question which is related to this I think we can quickly answer it uh this person is asking about context propagation support and eBPF >> oh that's a good question did you >> No no no gohead go ahead I want you >> yeah um I think this is kind of like what context are you are you trying to propagate. Um and this is once again like eBPF is like the underlying tool and you need to usually enrich that information. So like in psyllium I can tell you about like the context propagation that we're trying to do is like what is the actual identity of this pod. So as we're connecting pod a like front end to back end we can say like this is front end this is back end and we need to actually like pull that information from the the kubernetes API and we pull we give everything selium identities and through those we're able to like propagate the context of this is front end this is back end this is they should be able to connect or like front end a can't talk to database B and we can do that through things like network policy but like the thing about ebpf it's running in the kernel. So what the context that it has is things like process ID like like file. It doesn't have right the the kernel actually doesn't even know what a container is. It has things like croups and namespace and that's the things that the the kernel knows. So, EPF is going to know, okay, this is a process running in this like namespace, right? And if you want to tie that up into Kubernetes related context, well, you need to give that context back into EVPF. And you can do that through things like ebpf maps. Um, and that's why you're actually usually relying not on eB itself, but you're going to have a project like selium for instance in between. um that's going to be able to provide or open telemetry that's going to be able to provide that context to the EVPF programs and then take what the EVPF programs and then bring that back up into the Kubernetes con uh um context. Uh the thing is it's it's a very low-level tool and you need to give it that context and then take the information and put it back into context because EVPF itself doesn't have that. >> You explained it really well. Thank you. That's actually um a good question and a great answer. Thank you very much. >> Thank you so much Bill. Uh and I'm sorry for people who are some of the people we are not able to see your names. Uh so we we can't give you a shout out but thank you so much for asking your question. Keep keep those questions coming in folks and we are seeing I'm seeing a lot of interesting questions and we'll be taking them uh in the in the middle of the conversation. So okay uh we've talked about Kubernetes observability right? uh Raj dwelled into that and we we we sort of went into that route but I want to zoom in a bit more here because and talk about implementation right because we talked about instrumentation we touched upon that topic uh uh in the earlier part of it conversation as well and when we talk about instrumentation as Rajat mentioned there is a discussion that happens uh a question that commonly pops up is ebpf or open telemetry right uh both cater to different use cases And this in my opinion confuses teams as they see you know multiple ways of approaching it right. So because there are a lot of options and teams sometimes wonder which one to choose versus the another. I want to ask you Bill like is this even the right way to think about it? First of all >> I mean the first problem is open telemetry is using EVPF. So you're you're not choosing one or the other. If you're if you're using open telemetry for like some of the things obviously it's a very big project then you're already using EPF so it's not an either or it's you're actually already getting both >> you're very complimentary yes >> yeah um and so I guess tying it back into like our the earlier parts of our conversation too is like it's like looking at two different things so you can think of right open telemetry is more of the application running in user space providing the context and ebpf is this underlying tool collecting the information And so you're getting all this observability data and then you need to provide the context and draw the actual insights out of that. And so that's where something like open telemetry comes in. And so an example of this like the open telemetry profiler um is using EVPF to profile applications and then it pulls all that information and gives you the actual application profile. I don't know if they do icol or flame graphs but actually looking at like okay which part of the application is consuming the most CPU cycles. Um and so that's one example where open telemetry is using ebpf to collect data from the kernel and then bring that into the context of kubernetes or your application and then actually providing you the insight. Okay, if we can change this part of our code we can make our application run faster. And so in a lot of ways they're going to be working hand in hand. Um I think it depends on how you want to do it. like some people if you're instrumenting there's also open telemetry you can use open telemetry to just instrument your application um right and then then it's you're instrumenting your application code um then you may not use ebpf for for that part of it but there's many different things you're trying to do there's also right open telemetry is a like standardized way to ship observability information right and so ebpf isn't doing that what it's doing is collecting the information ebpf is very good that collecting information that you want to know from the kernel and then it's up to you how you actually do that and that's where something like open telemetry can come in. So I don't see them as competing. I actually see them as really complimentary. >> Yeah, >> that's actually the best way to put that. Yeah, thank you so much. I >> completely agree with that. I think like the the EDP profiler is one of the great examples of how um you know they're working together and I've certainly seen other approaches where um like for tracing uh instead of relying on you know uh application level agents uh where there are some projects even if it's CNCF that have attempted to u do application tracing using EVPF. There are some challenges around that which we will we'll talk about you know soon. Uh but those are some of the other areas that I have seen um around trying to use eBPF uh for for things like you know tracing and profiling. Yeah. Uh sorry Kuna go ahead. I didn't mean to cut you off. >> No worries. No worries at all. Yeah. Uh I just shared a link as well in the chat for everyone to look out. So this is basically the open telemetry eBPF instrumentation. In short it's called as OBI. So you can definitely check it out on as Bill mentioned right. It's it's not like one versus other. It's how you can use it in a complimentary manner and that's where you get you know the the most insights that you want according to your use case of course. So um yeah I mean we we I think this was an important topic to touch on because I see a lot of people have this confusion and I think Bill you answered it really well uh because these are complimentary approaches. So because we are talking about instrumentation right now right I think it's fair we talk about the challenges as well. I know we have touched a lot about what teams should consider, what what teams should keep in mind, what they should be aware of as well, but I think we can have uh a few specific use cases in mind to talk about these particular aspects. So, Rajat just to start this off because we have implemented an EVPFbased solution in the past as well, right? So, do you want to I think it's a good time for you to kick us kick us off and uh tell us a bit more about that and what challenges do you think we faced? Sure. Yeah. I think one of the first things that we were interested apart from sort of the security aspect of it uh was was tracing and you know how we could do um you know u tracing with zero code instrumentation or I mean that's a bit of a uh tricky term but you know without them having to change anything implement any kind of libraries or any of those things how can we enable tracing and there was some interesting projects so Pixie was one one such project that that tried to do that. One of the challenges that we we we faced was um you know when it came to layer 7 visibility. So layer 4 was amazing. It's just uh you know you just turn it off and you you get a nice little network map right off the bat. Uh a lot of things you know um work really well but when it comes to layer 7 you start to get into challenges. So anytime a library changes because the method signatures have changed changed now uh it's no longer working. So for example we had um uh you know when we move from one Kafka version to another all the Kaf Kafka related uh uh information we were getting uh suddenly disappeared or started acting out weird and giving sort of wrong information. But when it worked though it was beautiful because you you literally um you know just point at it and and it picks up everything and it's very rich sometimes the data you get. We didn't have this kind of challenge with HTTP because a very mature protocol the the libraries are fairly stable but uh you know fastm moving uh you know protocol or or something that's you know con constantly being developed. So for example when you move from one Kafka version to another we ran into those challenges um I don't know if you know that's the right approach when it comes to uh layer 7 related stuff but there is no shortage of uh people trying to sort of get to that level and there are um open source projects and also like you know observable vendors who are sort of going after this bit of a holy grail in my opinion But it's what are your thoughts on that? Um you know is that even advisable? U should people not think of it at the layer 7 level? Because you need extra information to correlate like you said YouTube is very low level. You can't it doesn't know anything at the at the application level certainly not things are running on the user space. So you need some sort of correlation and coordination. Uh what are your thoughts on that? Yeah, this is a little bit about how EVPF is innovating itself. So, I know we went very quickly over kind of like what EVPF is and the history of it, but kind of like another po important part to kind of bring up here is that ebpf because you know it has this verifier. It's like checking to make sure it's safe to run. There's a lot of limitations on what you can actually do with ebpf to ensure that the code is safe to run in the kernel because you obviously don't want to crash the kernel. And so one of those is actually like a limit on the number of instructions. So when ebpf was first merged into the kernel, there was a limit of 4k instructions per program, right? So pretty small program. Um now they've able been able to increase that to a million instructions. Um and there's other kind of like limitations around loops and other things. It's changing as the EVPF ecosystem continues to evolve and we can verify programs to make sure that that they're safe while still kind of working in existing limitations or kind of pushing those limitations out. You know, going from 4K instructions to a million instructions, but there are some limitations in terms of what EVPF can do. And this is I think kind of like where it started is like, oh, EVPF can't do this, EVPF can't do that. Um, but that has actually like changed a lot in EVPF. Um there's actually uh Liz Rice my boss she actually recently did a talk and a blog post about how ebpf is now turning complete right you can write anything that you want to you could write doom in ebpf um the question is whether you actually want to and what what's the benefit that you get from that right so ebpf is still a complex way to write a program running things in the kernel is like more complicated has less flexibility than running something in user space um so I think when you're thinking about Can I do this with EVPF? The question is or the answer is yes. The real question is should I do this with EPF? And the the answer for that isn't always yes. Um I think there's a lot of things that'll go back and forth. I think it's been interesting kind of seeing the transition of moving more things into the kernel especially on the networking side because like there it's really concerned about like throughput performance how we're actually doing these things. And so we actually like um I like said no we wrote an HTTP parser that works in EBPF in the kernel. And if you would have said that like five years ago people would have been like no you'll never be able to do that. But because of kind of these innovations in the EVPF ecosystem itself being smart about how you actually do it and implement it and program it. It's like yes you can do L7 parsing in the kernel with EPF. Um so that's something that we do know. So then the question is, yeah, like what are you actually trying to achieve? What are your goals? What are the trade-offs that you're willing to make? And I I think that's a question for actually actual implementation. If you're writing your own EVPF programs, you need to evaluate that. And if you're looking at different vendors or different projects, you're going to be like, what use cases are most important to me? What solutions am I trying to do? Am I looking for speed, performance, complete flexibility? Um, and looking at those trade-offs. And that'll help you understand is this something I should do with EVPF? Because the question of whether you can do it with EVPF is yes, but that doesn't always mean you should. >> And know great, great answer. Definitely. Yeah, this protocol passes are one of those use cases that we were we were after and when it works, it works beautiful. Uh it's just uh >> it's like magic. You turn on and you can see everything like Kafka was one of those things where you can see from the consumer side like everything that kind of goes through the wire. Um yeah, but to your point exactly like you know I mean just because you can do it should you do it? Um and there are lots of trade-offs to sort of consider but it but we can what we can safely say is that though EBPF has sort of come a long way from like hype a while ago to very concrete use cases where people are continuously pushing the the envelope to see where it can be applicable. Um so what what do you think would be next? Like where where are you seeing it's kind of going right now? Um you know where it started with like just networking and and security side to you know people trying to do things like what I talked about like protocol passes and try to you know do more high level tracing. What are some of the other use cases you've seen uh people are you know tapping into and where do you think it's heading? Yeah. So, one of my favorite lines from the EVPF documentary um at the be that we posted a link to at the beginning is from Thomas um and he's the founder of ISAN or one of the founders and the line that sticks with me is suddenly we realize we could write rewrite everything better. What should we rewrite first? >> And their answer to that question is networking. That's how selium came in. That's how isalent came to be. That's kind of like the whole history history of our company. And then we saw, you know, Brendan Greg tackle observability, tracing, profiling at Netflix. We saw um other people tackling security with EVPF. Um and it's kind of like you can replace a lot of different parts of the Linux kernel with EVPF. I mean, I've even heard ideas of like writing a whole kernel just in EVPF and providing just the functionality that you need. Then you can do things like micro and uni kernels with ebpf. I mean there's lots of wild ideas um actually but the one that I actually really like right now um is once again getting back to this like innovation cycle right innovation slowed down in a lot of parts of the kernel because it's like okay we have a standard implementation of this it works it works well for most people like we stick with that unless you can actually prove something and so actually one of the areas that is probably like not changed that much um is actually Linux kernel scheduling there's a completely fairuler in the Linux kernel and it works great for a lot of use cases, but you know there's always some edge cases, some very specialized things. For example, like gaming, you know, you care about different things than like an e-commerce website. Um, if you think about like what the application actually looks like and fundamentally they're using the same scheduler in the kernel, right? And so what's cool with EVPF now is you can write your own custom scheduler, right? And I think that's really cool. It's like once again like how do we bring innovation back into the kernel? That's one way. It's like before the idea of writing your own um kernel was like something maybe you could do a computer science class and it's like oh now you can actually write one like the two funniest ones that that I've seen is like somebody um wrote auler that was based on them singing. So it's like as audio inputs coming in that's how it schedules the programs. Um, and the other one that was really funny is they literally made an astrological um, and based on the star signs, you know, like if like Venus is in retrograde and like um, it's yeah, it's like Sagittarius right now, it's like this is how your programs are scheduled. So, um, you can do a lot of things. Like obviously that's not one to put into production, but it kind of gives you the idea. You can okay, we we have a different type of workload. We have different requirements. We have different problems that we're trying to solve than this specific thing is. And suddenly we're able to we have the power to do things differently. And that's what EPF enables. That's that that that's that's actually amazing to kind of hear these um um you know wild sort of ideas. Uh that's you know the the important part is that uh you know we only limited by our creativity. So we can you know we can come up with all of these different things. One of the other things that if I may quickly touch upon um at least in the past you needed sort of specialized knowledge at least some idea to to your point you said like you know most people should be consuming um you know applications like psyllium or you know cubescape or pixie that is written on top of ebpf and you needed some knowledge to like this write these ebpf probes in the past and you know the logic companies had access to that. But with um you know generative AI and other things that are making things a little bit more um I guess approachable you know covering the knowledge gap. Um you think they will see a bit bit of innovation where people who were limited by uh I guess knowledge before skill gaps in certain areas are now perhaps able to kind of you know tackle into this. I mean it's impacting everything around us but I was curious to know about this area. So this is how much do you believe in AI coding agents? And I think there's a lot of theories and opinions out on that right now about how good they are. And I I do think like there are some cool use cases about if you understand one area being able to rele your existing knowledge and the computers existing knowledge to expand into a new area. Like I I know some of our our kernel team is like obviously very smart people but they don't know about um for for instance we're actually looking at how do we do um bring essentially the power of ebpf to virtualized networking um and containers running VMs running in containers and kubernetes and like we don't know as much about like you know some of the KVM QM um type of things and so it's like okay we know a lot about kernel but not this specific part because we were focusing on other parts. Can we take this learn a little bit from AI and do things like that? Um so but that's how much do you believe in AI coding agents? I think maybe the better answer to this question is like you don't have to know about ebpf to actually benefit from it. I think that's the coolest thing. My favorite example is the majority of the world's actually already using ebpf and it's sitting right here in your phone. So any Android operating system is using EDPF for the network statistics. So if you wanted to know how much data did I use this month, how much do I have left? Well, you can thank EVPF for that. And so if you think about the number of Android devices out there in the world, >> I mean you're already looking at most of the world using EVPF, right? And so this is once again even if you're not using the super low-level technology, there's the products and the projects on top of it that are using that that are allowing you to still benefit from it too. Totally agree. Yeah, I love that line. You know, you don't need to know it to reap the results of, you know, what it can provide it to you. Yeah, love that point. All righty. I think in the interest of time, we are at the end of this one. I was just scrolling through the chat to see if I haven't missed any particular question. U there is this interesting one by Swapnil. uh he's asking about what could be some business related use cases of uh ebpf. >> Yeah. So, I saw one of the comments later was about um uh blocking domain requests based on business requirements. And actually, I don't remember the exact name of the GitHub repo, but if you type in something like Spotify ads and ebpf, um there's actually a project where somebody used ebpf to block the ads on Spotify based on um like the IP addresses where the different information was coming or something like that. But if you look it up on GitHub, uh there's a project out there that does that. So yes, depending on your business requirements, whether it's blocking ads or something else, you can do that with EPF too. >> That's interesting. Yeah. How much uh like people are creative and if you put an innovative tool like EPF in their hands and anything could happen, right? >> Yeah. I think that's the fun thing. It's like it allows you to innovate again. It allows you to ideulate again and you never know what human creativity will do. >> Amazing. Yeah. Uh yeah, I think we have answered most of the questions here. Uh folks, if you have been uh watching the session and if you have more questions, feel free to comment them down below. And I saw one comment by Rafael, I hope I'm pronouncing your name correctly. Sorry. Uh that they don't see the links. So this is a bug from LinkedIn folks. So I'm really sorry about that. If you want to access the links that we shared throughout the session, you can head over to the YouTube live stream. I have shared the link for that here. Uh you can go there and you'll be able to access it from the recording as well. once you want to, you know, if you want to access those because you've shared a bunch of interesting links as well. All right. Um, thank you so much, Bill. It's been real. Uh, we've had an interesting conversation. EVPF, I think, is is not something that you can just cover in a 40-minute chat. There's a lot more to it, right? And we'd love to have you in another conversation as well sometimes to talk about the internals of EVPF if someone would be interested in that. But yeah, uh, do you have any closing remarks for people who are watching till now? Yeah, I think it's just kind of how I talked about it last. Um, EBPF allows you to innovate again and now it's up to you to use your creativity to figure out how to use that. Amazing. Yeah, it's it's been real. Thank you so much, Bill, for taking out your time. And thanks everyone for watching. You can access the recording on LinkedIn and YouTube as well. And feel free to reach out to Bill. Feel free to follow him on all the platforms. He's doing some amazing work. And we'll see you in the next episode of Observability Unplugged. >> Yes. Bye-bye. >> Thank you, Bel. Appreciate it.