2.4.1. Distributed Systems

Distributed Systems

2.4.1. Distributed Systems & Cloud Native Architectures: Let’s Build BIG and SMART!
- 2.4.1.1. Event-driven architectures, CQRS, Event Sourcing using message queues: Talking with Events, Not Just Requests!
GPT Prompts for Further Exploration
Future Reading Links

2.4.1. Distributed Systems & Cloud Native Architectures: Let’s Build BIG and SMART!

Okay, so we’re now in the Advanced Level! That means we’re going from building solid houses to designing entire cities of back-end systems. We’re talking about systems that are not just powerful, but also incredibly smart, adaptable, and can handle tons of users without breaking a sweat. This is where Distributed Systems and Cloud Native Architectures come into play.

Think of a single back-end server like a really good restaurant. It can serve many customers, but there’s a limit. What happens when your restaurant becomes super popular? People are queuing down the block! To handle this growth, you don’t just make your kitchen faster (though that helps – performance!). You might:

Open more restaurants! (Horizontal Scaling - more servers)
Specialize roles! Maybe one restaurant focuses on taking orders, another on cooking, another on delivery (Microservices - specialized services)
Make sure if one restaurant has a problem, the others keep serving customers! (Resilience - fault tolerance)
Have a central system to guide customers to the best restaurant and manage all of them! (Orchestration & Management - Kubernetes, API Gateways)

That’s the spirit of Distributed Systems and Cloud Native Architectures. We’re building back-ends that are made of many parts working together, designed to be incredibly scalable, reliable, and easy to manage, especially in the cloud (like AWS, Google Cloud, Azure).

Let’s start with the first building block:

2.4.1.1. Event-driven architectures, CQRS, Event Sourcing using message queues: Talking with Events, Not Just Requests!

Imagine you’re in a busy office. Traditionally, if someone needs something from another department, they might walk over, interrupt them, ask their question, and wait for an answer. This is like synchronous communication – you have to wait for a direct response right now.

But what if the office used a better system? What if they used memos or an internal messaging system? You could write a memo (“Order Placed!”, “New User Signed Up!”, “Inventory Changed!”), put it in the internal mail, and then get back to your own work. The other department (service) can pick up the memo (event) when they are ready and react to it. This is more like asynchronous communication – you send a message and don’t need to wait for an immediate reply.

Event-Driven Architectures (EDA) are all about building systems that communicate using these “memos” – we call them events.

What is an Event? Think of it as a “Notification” of Something Important

In our back-end world, an event is simply a notification that something significant has happened. Think of these examples:
- “A new customer just signed up!”
- “An order was placed – order number 123!”
- “The price of a product (Product ID: 456) was updated to $29.99!”
- “Payment received for Order 123!”
- “Inventory for Product 456 is now down to 5 units!”
These are all events. They are things that have already happened. They are facts.
Event Producers and Event Consumers: Who Sends and Who Listens?

In an EDA system, we have services that produce events and services that consume (or listen to) events.
- Event Producers: These are services that generate and send out events when something important happens in their part of the system. For example:
  - An Order Service might produce an “Order Placed” event when a new order is created.
  - A Payment Service might produce a “Payment Received” event after a successful payment.
  - An Inventory Service might produce an “Inventory Updated” event when stock levels change.
- Event Consumers: These are services that are interested in certain types of events. They “subscribe” to events and react when they receive them. For example:
  - A Shipping Service might consume “Order Placed” events to start preparing shipments.
  - An Analytics Service might consume “Order Placed,” “Payment Received,” and “User Signed Up” events to track sales and user trends.
  - A Notification Service might consume “Payment Received” events to send confirmation emails to customers.
Message Queues: The “Internal Mail System” for Events

How do these services actually send and receive these “event memos”? That’s where message queues come in. Think of a message queue like a very efficient and reliable internal mail system for your back-end. Popular message queues include:
- RabbitMQ: A widely used, open-source message broker.
- Kafka: A distributed streaming platform, excellent for high-throughput event streams.
- Cloud-based Message Queues: Services like AWS SQS, Google Cloud Pub/Sub, Azure Queue Storage offered by cloud providers.
How it works:
1. Event Producer Sends Event to Queue: When a service (producer) wants to send an event, it doesn’t send it directly to another service. Instead, it sends the event to a specific message queue. Think of putting a memo in the “outgoing mail” basket for a specific topic.
2. Message Queue Holds the Event: The message queue stores the event temporarily. It makes sure the event is not lost, even if the receiving service is temporarily busy or offline. It’s like the central mailroom holding onto the memo safely.
3. Event Consumer Reads Event from Queue: Services (consumers) that are interested in a certain type of event “subscribe” to the relevant message queue. When they are ready, they read (or “pick up”) events from the queue. It’s like departments checking their “incoming mail” basket.
4. Asynchronous Processing: The producer service can send the event and immediately continue with its other tasks. It doesn’t need to wait for a response from the consumer. The consumer processes the event asynchronously – at its own pace, whenever it’s ready.
Why Event-Driven Architectures are Powerful:
- Decoupling: Services become much more decoupled. Producers don’t need to know anything about consumers. They just send events to the queue. Consumers don’t need to know where events come from, they just listen to the queue. This makes services more independent and easier to change or update without affecting others.
- Scalability: Each service can scale independently. If the Shipping Service is getting overwhelmed by orders, you can scale up just the Shipping Service without needing to scale up the Order Service or Payment Service. Message queues also help handle surges in traffic smoothly.
- Resilience: If one consumer service fails temporarily, the system can still function. Producers can still send events, and the message queue will hold onto them. When the consumer service comes back online, it can pick up and process the events it missed. This makes the system more resilient to failures.
- Real-time and Reactive: EDA is great for building real-time applications because services can react to events immediately as they happen. Think of live dashboards, real-time updates, etc.
CQRS (Command Query Responsibility Segregation): Separate Roads for Reading and Writing!

Imagine a library. Think about how you interact with it. You do two main things:
1. Borrow books (Write/Change): When you borrow a book, you are changing the state of the library’s collection. You are taking a book out.
2. Look up books (Read/View): When you search for a book, you are just reading information. You’re not changing anything in the library.
In a traditional system, we might use the same library catalog (database) for both borrowing and searching. But what if searching became super slow because so many people were borrowing and returning books at the same time?

CQRS (Command Query Responsibility Segregation) is a pattern that says: “Let’s use separate systems for handling changes (commands) and for handling lookups (queries)!” It’s like having separate roads for “write” traffic (changes) and “read” traffic (lookups).
- Command Side (Write Side): The “Change Makers”
  
  This side is responsible for commands – actions that change the data in your system. Think of “place an order,” “update user profile,” “add product to cart.”
  - Focus: Handling business logic, validation, and making sure data is consistently written.
  - Database: Might use a database optimized for writing data reliably and consistently (like a traditional relational database).
- Query Side (Read Side): The “Information Providers”
  
  This side is responsible for queries – actions that read data to show information to users. Think of “show product details,” “list orders,” “get user profile.”
  - Focus: Optimized for fast and efficient data retrieval for reading.
  - Database: Might use a different type of database optimized for reading and searching quickly. This could be a NoSQL database, a read-optimized database, or even just a simplified, “flattened” version of the data.
- How do they stay in sync? (Eventual Consistency)
  
  If we have two databases, how does the “read” database get updated when something changes in the “write” database? This is often done using events!
  1. When a command changes data in the “write” database, it also publishes an event (like “Product Price Updated”).
  2. The “query side” (read side) listens for these events.
  3. When it receives an event, it updates its own database to reflect the change, but it might not happen instantly.
  This is called eventual consistency. The “read” database might be slightly behind the “write” database for a very short time, but eventually it will catch up and be consistent. For many read-heavy applications, this slight delay is acceptable, and the performance benefits of CQRS are worth it!
- Why CQRS is Great for Scalability and Performance:
  - Independent Scaling: You can scale the “read side” and “write side” independently. If your application is read-heavy (like an e-commerce site browsing products), you can scale up the “query side” to handle tons of read requests without affecting the “write side.”
  - Optimized for Reads and Writes: You can use different database technologies and data models that are best suited for reads and writes separately. This leads to better performance overall.
  - Simplified Query Logic: The “query side” database can be designed specifically for the types of queries you need to perform, making read queries faster and simpler.
Event Sourcing: The “Historical Record” of Everything!

Imagine you are keeping track of your bank account. Traditionally, you might just see your current balance. If you want to know how you got to that balance, you might look at a transaction history.

Event Sourcing takes this “transaction history” idea to the extreme. Instead of just storing the current state of things, you store every single event that ever happened that changed the state! Think of it as keeping a complete, unchangeable journal of all changes.
- The Event Store: Your “Journal”
  
  In Event Sourcing, your primary database becomes an event store. This event store only does one thing really well: store events in the order they happened, and never change or delete them. It’s like an append-only journal.
  - Example Events in an Event Store (for an Order):
    1. “Order Placed” (Order ID: 123, Customer: John, Items: …) - First event
    2. “Item Added to Order” (Order ID: 123, Product: Product A, Quantity: 2)
    3. “Shipping Address Updated” (Order ID: 123, New Address: …)
    4. “Payment Received” (Order ID: 123, Amount: …)
    5. “Order Shipped” (Order ID: 123, Tracking Number: …) - Last event
- Reconstructing the Current State: “Reading the Journal”
  
  If you want to know the current state of an order (e.g., “What’s the status of Order 123?”), you don’t just read a “current order record.” Instead, you replay all the events in the event store for that order in the order they happened, from the very first event to the latest event.
  
  By replaying these events, you can reconstruct the current state of the order step-by-step.
- Why Event Sourcing is Amazing:
  - Complete Audit Log: You have a full, immutable history of everything that ever happened in your system. This is fantastic for auditing, compliance, and understanding exactly how your system reached its current state.
  - “Time Travel”: Because you have the full event history, you can easily “go back in time.” You can replay events up to a specific point to see what the system looked like at any moment in the past. This is incredibly powerful for debugging and analysis.
  - Resilience and Replayability: If something goes wrong or you need to change how you process data, you can simply replay the event stream through a new version of your application. You can “reprocess history” if needed!
  - Natural Fit with EDA and CQRS: Event Sourcing works beautifully with Event-Driven Architectures and CQRS. Events become the central way to represent changes and communicate them between systems. Events from the Command side in CQRS can be the events stored in your Event Store!
- Things to Consider with Event Sourcing:
  - Complexity: It’s more complex than traditional data storage. You need to think in terms of events, and reconstructing state can be different.
  - Event Schema Evolution: Changing the structure of events over time needs careful planning.
  - Read Performance: Reconstructing state from events can be slower than directly reading a current state record, especially if you have long event histories. You often use techniques like snapshots (periodically saving the current state) and read models (like the “query side” in CQRS) to improve read performance.

In a Nutshell:

Event-Driven Architectures: Services talk to each other using asynchronous events via message queues for decoupling, scalability, and resilience.
CQRS: Separate models and databases for commands (writes) and queries (reads) to optimize performance for each type of operation.
Event Sourcing: Store application state as a sequence of events in an append-only event store, providing a complete audit log, time-travel capabilities, and enhanced resilience.

These are powerful, advanced patterns used in complex, modern back-end systems. They might seem a bit abstract now, but as you delve deeper into distributed systems and cloud-native architectures, you’ll see how incredibly valuable they are for building robust and scalable applications!

GPT Prompts for Further Exploration

- "Explain the benefits and challenges of microservices architecture."
- "How do microservices communicate with each other in a distributed system?"
- "What are the core components of Kubernetes and their roles?"
- "How does Kubernetes handle scaling and load balancing?"
- "What is serverless computing and how does it differ from traditional cloud computing?"
- "Discuss the pros and cons of using serverless architectures for building applications."
- "How can event-driven architectures improve system resilience and scalability?"
- "What are the best practices for designing event-driven systems?"
- "What are the key considerations when implementing CQRS in a distributed system?"
- "How does event sourcing help in maintaining data consistency and integrity?"

Future Reading Links

These resources will provide you with a deeper understanding and practical insights into distributed systems, cloud-native architectures, and related advanced topics.