Getting Started with Kafka's Cluster Architecture
In our previous chapter, we discussed the usage and need for Kafka in various architectures, examining Kafka externally. Now, let's take an internal look at the framework and become familiar with what actually goes on within the ecosystem. We will cover several concepts, starting with a holistic view of them, and then exploring each one by one, building on the intuition. There's a lot of terminology here, and if you haven't worked with distributed systems or Kafka before, you might not be familiar with some terms. We'll set up reminders from time to time to reinforce the terminology. If something doesn't make sense right away, it might help to finish the rest of the chapter and review the concept from a different perspective once it's discussed in relation to other components.
In the preceding chapter, we used terms such as "scalability", "fault-tolerance", "resiliency", and "high availability". While these can be used as buzzwords to make a framework/library sound advanced and useful, they're ultimately the effects of good architecture design. These same adjectives can be applied to various other systems, including regular web applications, if their architecture allows them to scale (horizontally or vertically) without suffering performance loss, recover from exceptional states (not just exception handling, but full recovery), etc. These are the effects of good design, which is not easy to achieve, as any software engineer has painfully learned while trying to create a complex system. In this chapter, we'll revisit the terminology, but this time, we'll explore the mechanisms at play that allow Kafka to use these buzzwords in their documentation and marketing copy.