# The Anatomy of Kubernetes ListWatch(): Prologue When I was in high school, I took languages (German, English) as my major classes. Reading, interpreting and writing about literature, i.e. books, became a daunting task. And to be honest with you, I don't recall much from the Shakespeare's and other legends of past centuries. But there was one book which I still remember. Perhaps you have heard about Johann Wolfgang von Goethe and one of the most famous books in German literature: Faust: A Tragedy. In this book, Goethe describes *the restless pursuit of knowledge* and the never-satisfied desires of a person who is dissatisfied with his life. In order to satisfy his need for knowledge and pleasure, Faust dedicated himself to the devil and to the life (and love) of a person. {{< image src="faust-1.jpg" caption="Dr. Faust (Source: Britannica)" width=300 >}} With the ever increasing rate of innovation in information technology, many of us have a desire to continuously learn and grow, seeking for answers to small and big questions, like Dr. Faust. Reflecting on myself, this part of a monologue by Faust became a theme throughout my whole career (and still is). > *βTo enlighten me more, What holds the world together at its innermost coreβ* > > (Faust, J. W. Goethe) Whenever I am confronted with a complex problem space or piece of technology, I try to *distill it to its essence*, its core. What are the building blocks (atomic units)? How are they assembled to form a more complex structure? What are the decisions and tradeoffs the architects had to make? How did these influence the overall design? Not only does this help me to better understand whether and how to optimally use a certain technology. I also became *a better software engineer*, standing on the many shoulders of the giants in our industry. Since my early days with [Kubernetes](https://kubernetes.io/), my inner Dr. Faust pushed me to learn more about the core concepts, e.g. how the (many) *autonomous and stateless* controllers (*control loops*) react to state changes (*events*) without a central orchestrator. I wrote a couple of blog posts[^1] [^2] and gave presentations[^3], to answer the same questions many of you also had, doing my best to educate and grow our great community. Even though over time I developed a good understanding and mental model on the Kubernetes architecture, I still had the feeling to not fully grok certain details how these events are generated, propagated and reliably consumed throughout the various Kubernetes components and actors. The `ListerWatcher` interface in the `client-go` SDK, used in all Go-based Kubernetes controllers, plays a critical role here: ```go // ListerWatcher is any object that knows how to perform an // initial list and start a watch on a resource. type ListerWatcher interface { Lister Watcher } ``` It all started with a simple question: how does `ListerWatcher` work? As you see from this series, it turned out to be another journey into a fascinating and deep rabbit hole. For example, is there a canonical definition of a state change *event*[^4] in Kubernetes? How is the *level-triggered* state change notification implemented? Why do I always get `ADDED` events from an initial `kubectl` or controller `WATCH`, when the object's last state is actually `MODIFIED`? How is the `resourceVersion`, which is critical for *optimistic concurrency control*[^5] in such an asynchronous system, generated? And if it's so important, why is `resourceVersion` not explicitly persisted in the persistence layer `etcd` then? Speaking of `etcd`, how is the Kubernetes object registry physically represented? What is a ["flat binary key space"](https://etcd.io/docs/v3.4.0/learning/data_model/) (to quote from the docs) anyways? And what's the difference between a compaction and defragmentation operation related to `resourceVersions`? Lastly, how does an end-to-end event notification `WATCH` stream work under the covers, e.g. when using `client-go` or `controller-runtime` (kubebuilder)? How can a *collection*[^6] ("LIST") have a `resourceVersion`, if the latter is based on individual object-level changes? What are all these `Indexers` and `Queues` used for in the controller SDKs? And why don't we see `ADD`/`UPDATE`/`DELETE` events anymore in the `Reconcile()` handler of controller-runtime? The list goes on...in fact, it looks like I am not the only one with these questions π