Event-Driven Framework

Event-Driven Framework#

Design#

Everyone sees Envoy as a proxy, mainly implementing request forwarding with customizable logic. That’s not wrong. But like other middleware that require high throughput and low latency, the design must take into account load scheduling and flow control. A good scheduling mechanism should balance throughput, response time, and resource footprint.

Figure: Event-Driven Framework Design

Figure: Event-Driven Framework Design#

Open in Draw.io

  1. Dispatcher thread event loop: The Dispatcher Thread waits for events (epoll wait), and processes them when timeout occurs or events are triggered.

  2. The following events can wake up epoll wait:

    • Receiving inter-thread post callback messages. Mainly used for updating Thread Local Storage (TLS) data, such as cluster/stats updates.

      • Dispatcher handles inter-thread events.

    • Timer timeout events

    • File/socket/inotify events

    • Internal active events. Events triggered explicitly by other threads or the dispatcher thread itself.

  3. Event handling

One full loop of event processing includes the above three steps. This full cycle is called an event loop, or sometimes an event loop iteration.

Implementation#

The above describes how event processing works at the kernel syscall level. Now we’ll look at how it is abstracted and encapsulated in Envoy’s codebase.

Envoy uses libevent, a C-based event library, and adds further abstraction with C++ OOP wrappers.

Figure: Envoy Event Abstraction Model

Figure: Envoy Event Abstraction Model#

Open in Draw.io

How do you quickly understand the core logic of a project that makes heavy (even excessive) use of OOP encapsulation and design patterns, without getting lost in the sea of source code? The answer: follow the main thread. For Envoy’s event handling, that thread starts with the core libevent objects:

  • libevent::event_base

  • libevent::event

If you’re unfamiliar with libevent, check the section Core Concepts of libevent in this book.

  • libevent::event is wrapped in ImplBase objects.

  • libevent::event_base is included in LibeventScheduler ← DispatcherImpl ← WorkerImpl ← ThreadImplPosix

Different types of libevent::event are further wrapped into different ImplBase subclasses:

  • TimerImpl – used for timer-based functions like connection timeouts or idle timeouts.

  • SchedulableCallbackImpl – Under heavy load, Envoy needs to balance event responsiveness with throughput. To prevent a single event loop from doing too much work and delaying subsequent event handling, certain internal or timed processes can be scheduled to complete at the end of the current event loop or be deferred to the next one. SchedulableCallbackImpl encapsulates this kind of schedulable task. Use cases include thread callback posts and retry logic.

  • FileEventImpl – handles file/socket events.

Additional details are already well explained in the diagram above, so we won’t elaborate further.

Extended reading#

If you are interested in studying the implementation details, I recommend checking out the articles on my Blog:

And last but not least: Envoy author Matt Klein: Envoy threading model

Contents