Load Balancing: How Backend Systems Handle Millions of Requests

Load Balancing: How Backend Systems Handle Millions of Requests

Tópico: Load Balancing: How Backend Systems Handle Millions of Requests
Categoria: Tutoriais | Programação & Tecnologia
Idioma Principal: Português (Conteúdo de Tecnologia)

Descrição do Conteúdo / Informações:
-------------------------------------------------------------------------
Imagine you have built a backend API that works perfectly.

You deploy it to a server, connect your database, and everything is running smoothly.

Then your application becomes popular.

Instead of 100 requests per minute, you're suddenly handling 10,000. Then 100,000.

Your single server now has too much work to handle.

It becomes slow.

Eventually, it crashes.

So what do you do?

One solution is to make the server more powerful. But there is a limit to how much you can scale a single machine.

A more practical approach is to run multiple backend servers and distribute incoming traffic between them.

This is where load balancing comes in.

In this article, we'll explore what load balancing is, why it matters, how different algorithms work, the difference between Layer 4 and Layer 7 load balancing, health checks, sticky sessions, and how load balancing fits into a real-world backend architecture.

What Is Load Balancing?

Load balancing is the process of distributing incoming network traffic across multiple servers.

The system responsible for doing this is called a load balancer.

Instead of clients communicating directly with one backend server, they communicate with the load balancer.

The load balancer then decides which backend server should handle each request.

A simplified architecture looks like this:

Clients
│
▼
┌──────────────┐
│ Load Balancer│
└──────┬───────┘
│
┌────────┼────────┐
│ │ │
▼ ▼ ▼
┌──────┐ ┌──────┐ ┌──────┐
│ API 1│ │ API 2│ │ API 3│
└──────┘ └──────┘ └──────┘

The clients don't necessarily need to know that three backend servers exist.

From their perspective, they're communicating with one application.

The load balancer handles the distribution behind the scenes.

Why Do We Need Load Balancing?

The biggest reason is scalability.

Suppose your application initially has one server:

Users
│
▼
┌─────────┐
│ Server 1│
└─────────┘

This might work perfectly when you have a small number of users.

But as traffic increases:

Thousands of Users
│
▼
┌─────────┐
│ Server 1│
└─────────┘
│
💥

The server eventually becomes a bottleneck.

You could upgrade the server:

2 CPU → 8 CPU
8 GB RAM → 32 GB RAM

This is known as vertical scaling.

But vertical scaling has physical and financial limits.

Eventually, you may need a different approach:

Users
│
▼
Load Balancer
│
┌──────────┼──────────┐
▼ ▼ ▼
Server 1 Server 2 Server 3

This is horizontal scaling.

Instead of making one machine significantly more powerful, you add more machines.

Vertical Scaling vs Horizontal Scaling

There are two common ways to scale a system.

Vertical Scaling

Vertical scaling means making an existing server more powerful.

For example:

Before:

4 CPU
8 GB RAM

↓

After:

16 CPU
64 GB RAM

The advantage is simplicity.

You don't necessarily need to change your application architecture.

However, the machine has a physical limit.

There is also another problem.

If that server goes down, your entire application goes down.

Horizontal Scaling

Horizontal scaling means adding more servers.

For example:

Before:

Server
│
▼
API App

After:

Load Balancer
│
┌─────────┼─────────┐
▼ ▼ ▼
API 1 API 2 API 3

Now traffic can be distributed across multiple machines.

If one server fails, the others can continue serving requests.

This gives us both scalability and availability.

How Does a Load Balancer Work?

At a high level, the process looks like this:

1. Client sends request
│
▼
2. Load balancer receives request
│
▼
3. Load balancer selects backend
│
▼
4. Backend processes request
│
▼
5. Response goes back to client

Suppose a user requests:

GET /api/users

Instead of going directly to:

api-server-1

the request first reaches the load balancer.

The load balancer might decide:

Server 1 already has many active connections. I'll send this request to Server 2.

So the request becomes:

Client
│
│ GET /api/users
▼
Load Balancer
│
│ forwards request
▼
API Server 2

The load balancer is essentially acting as a traffic manager.

Load Balancing Algorithms

The load balancer needs a way to decide:

Which server should receive this request?

There are several strategies for answering that question.

1. Round Robin

Round Robin is one of the simplest approaches.

Requests are distributed sequentially.

Suppose we have three servers:

Server A
Server B
Server C

Requests might be distributed like this:

Request 1 → Server A
Request 2 → Server B
Request 3 → Server C
Request 4 → Server A
Request 5 → Server B
Request 6 → Server C

It basically goes around in a circle.

Advantages

• Simple

• Easy to implement

• Works well when servers have similar capacity

Disadvantages

It doesn't consider how busy a server currently is.

For example:

Server A → 10 active requests
Server B → 2 active requests
Server C → 1 active request

Round Robin might still send the next request to Server A.

2. Weighted Round Robin

Sometimes servers don't have equal capacity.

For example:

Server A → 16 CPU
Server B → 8 CPU
Server C → 4 CPU

Giving each server the same amount of traffic wouldn't necessarily be ideal.

Weighted Round Robin allows us to assign different weights.

For example:

Server A → weight 3
Server B → weight 2
Server C → weight 1

Traffic could approximately look like:

A → A → A
B → B
C

The more powerful server receives more traffic.

This is useful when backend instances have different capacities.

3. Least Connections

Instead of simply counting requests sequentially, the load balancer looks at the number of active connections.

Suppose:

Server A → 20 connections
Server B → 8 connections
Server C → 3 connections

The next request would probably go to Server C.

New Request
│
▼
Least Connections
│
▼
Server C

This can be useful when requests take different amounts of time to complete.

4. IP Hash

With IP Hash, the load balancer uses the client's IP address to determine which backend server receives the request.

Conceptually:

Client IP
│
▼
Hash Function
│
▼
Backend Server

For example:

192.168.1.10 → Server A
192.168.1.20 → Server B
192.168.1.30 → Server C

The goal is often to make requests from the same client consistently reach the same server.

This can be useful in some session-based architectures.

However, it also has limitations.

If the distribution of client IPs is uneven, traffic may become unevenly distributed.

5. Random

Another simple strategy is to randomly select a backend server.

For example:

Request 1 → Server B
Request 2 → Server A
Request 3 → Server A
Request 4 → Server C

Random selection can work surprisingly well with a large number of requests, although more sophisticated algorithms are often preferable when the system needs better control.

Layer 4 vs Layer 7 Load Balancing

This is one of the most important concepts when learning load balancing.

Load balancers can operate at different layers of the network stack.

Two common approaches are:

• Layer 4

• Layer 7

Layer 4 Load Balancing

Layer 4 operates at the transport layer.

It primarily works with protocols such as:

• TCP

• UDP

The load balancer doesn't necessarily need to understand the contents of an HTTP request.

It can make decisions based on information such as:

Source IP
Destination IP
Source Port
Destination Port
Protocol

For example:

Client
│
│ TCP connection
▼
Layer 4 Load Balancer
│
├──────► Server A
│
└──────► Server B

Because it operates at a lower level, Layer 4 load balancing can be fast and efficient.

Layer 7 Load Balancing

Layer 7 operates at the application layer.

For web applications, this usually means understanding HTTP or HTTPS.

Now the load balancer can inspect things such as:

HTTP Method
URL
Headers
Cookies
Host

For example:

GET /api/users

could be routed to one group of servers:

/api/* → API Servers

while:

GET /images/logo.png

could be routed somewhere else:

/images/* → Static Content Servers

This gives Layer 7 load balancers much more control over routing.

Layer 4 vs Layer 7: Simple Comparison

Feature
Layer 4
Layer 7

Network layer
Transport
Application

Common protocols
TCP, UDP
HTTP, HTTPS

Understands HTTP
No
Yes

Can inspect URL
No
Yes

Can inspect headers
No
Yes

Routing flexibility
Lower
Higher

Typical use
Network-level traffic
Application-aware routing

Neither is universally better.

The right choice depends on the architecture and requirements of the system.

Health Checks

Imagine you have three servers:

Server A → Healthy
Server B → Healthy
Server C → Crashed

What happens if the load balancer continues sending requests to Server C?

Users will receive errors.

This is why load balancers commonly use health checks.

The load balancer periodically checks whether backend servers are healthy.

For example:

GET /health

The server might respond:

HTTP/1.1 200 OK

The load balancer interprets that as:

This server is healthy.

But if the server repeatedly fails:

Connection refused
Timeout
HTTP 500

the load balancer can temporarily remove it from the pool.

Now traffic becomes:

Load Balancer
│
┌───────┴───────┐
▼ ▼
Server A Server B

Server C
❌ Unhealthy

This is one of the mechanisms that makes load balancing useful for fault tolerance.

Active vs Passive Health Checks

There are different approaches to detecting failures.

Active Health Checks

The load balancer actively sends requests to the server.

For example:

GET /health

If the server responds correctly, it remains available.

If it repeatedly fails, the load balancer removes it.

Passive Health Checks

The load balancer observes actual traffic.

If a backend repeatedly produces failures or connection errors, the load balancer can mark it as unhealthy.

In practice, systems can use a combination of health-check mechanisms.

Sticky Sessions

Here's an interesting problem.

Suppose your application stores user session information directly in server memory.

For example:

Server A
└── Session for User 123

The user's next request might go to Server B.

Request 1 → Server A
Request 2 → Server B

Server B doesn't know about the session stored inside Server A.

The user might suddenly appear logged out.

One solution is sticky sessions.

The load balancer attempts to keep the user connected to the same backend server.

User 123
│
├── Request 1 → Server A
├── Request 2 → Server A
├── Request 3 → Server A
└── Request 4 → Server A

This can solve some problems, but it introduces another dependency.

If Server A fails, the user's session may disappear.

Stateless Applications

A more scalable approach is often to make backend servers stateless.

Instead of storing important session state inside one server:

Server A
└── User Session

we can store shared state somewhere accessible to all servers.

For example:

Load Balancer
│
┌─────────┼─────────┐
▼ ▼ ▼
Server A Server B Server C
│ │ │
└─────────┼─────────┘
▼
Shared Storage

This could be a database, cache, or another shared state-management system.

Now any backend server can handle the request.

This makes horizontal scaling much easier.

Load Balancer vs Reverse Proxy

These concepts are closely related but aren't exactly the same.

A reverse proxy sits between clients and backend servers.

For example:

Client
│
▼
Reverse Proxy
│
▼
Backend

A reverse proxy can perform tasks such as:

• TLS termination

• Request routing

• Compression

• Caching

• Security filtering

• Header manipulation

A load balancer focuses specifically on distributing traffic across multiple backend instances.

However, one piece of software can perform both roles.

For example, Nginx can act as a reverse proxy and load balancer.

A Real-World Backend Architecture

Let's put everything together.

A production application might look something like this:

Internet
│
▼
┌─────────────┐
│ DNS │
└──────┬──────┘
│
▼
┌─────────────┐
│Load Balancer│
└──────┬──────┘
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
┌────────┐ ┌────────┐ ┌────────┐
│ API 1 │ │ API 2 │ │ API 3 │
└───┬────┘ └───┬────┘ └───┬────┘
│ │ │
└──────────────┼──────────────┘
│
┌─────────┴─────────┐
▼ ▼
┌─────────┐ ┌─────────┐
│ Database│ │ Redis │
└─────────┘ └─────────┘

A request might travel through the system like this:

User
│
│ HTTPS request
▼
DNS
│
▼
Load Balancer
│
▼
API Server
│
├──► Redis
│
└──► Database
│
▼
Response
│
▼
User

Each component has a specific responsibility.

DNS helps the client find the service.

The load balancer distributes traffic.

The backend handles business logic.

Redis can provide fast access to cached or shared data.

The database provides persistent storage.

This separation allows each part of the system to scale independently.

What Happens When a Server Fails?

Let's say we have:

Server A → Healthy
Server B → Healthy
Server C → Healthy

Then Server B crashes.

Without load balancing:

Users
│
▼
Server B
❌

Requests fail.

With a load balancer:

Load Balancer
/ \
▼ ▼
Server A Server C
Healthy Healthy

Server B
❌

The load balancer detects that Server B is unhealthy and stops sending new traffic to it.

The application can continue operating using the remaining servers.

This is the difference between:

"One server is down."

and:

"The entire application is down."

That distinction is extremely important in highly available systems.

Does a Load Balancer Eliminate Downtime?

No.

A load balancer improves availability, but it doesn't magically eliminate every failure.

For example, the load balancer itself can become a single point of failure.

If your architecture looks like:

Users
│
▼
One Load Balancer
│
├── Server A
├── Server B
└── Server C

what happens if the load balancer crashes?

Everything behind it becomes unreachable.

This is why production systems often use redundant load balancers or managed load-balancing services.

The architecture might look more like:

Users
│
┌────────┴────────┐
▼ ▼
Load Balancer 1 Load Balancer 2
│ │
└────────┬────────┘
│
┌────────┼────────┐
▼ ▼ ▼
Server A Server B Server C

Now the load-balancing layer itself has redundancy.

Common Load Balancing Mistakes

Understanding load balancing also means understanding what it doesn't solve.

1. Adding More Servers Doesn't Fix Everything

If your database is the bottleneck:

Load Balancer
│
┌───┼───┐
▼ ▼ ▼
API API API
\ | /
\ | /
Database
💥

Adding more API servers won't necessarily solve the problem.

The database might still be overloaded.

2. Ignoring Health Checks

If unhealthy servers continue receiving traffic, the load balancer becomes part of the problem rather than the solution.

3. Poor Session Management

Sticky sessions can hide architectural problems.

If your application depends heavily on one specific backend server, scaling becomes more difficult.

4. Ignoring Connection Limits

Every backend server has limits.

CPU, memory, network connections, database connections, file descriptors, and other resources can become bottlenecks.

A load balancer doesn't remove those limits.

It simply distributes traffic.

Load Balancing and Backend Engineering

As a backend developer, you don't necessarily need to build a load balancer from scratch.

But you should understand what happens around your API.

For example, suppose you build a Go API:

GET /api/users
POST /api/orders
GET /api/products

Initially, you might deploy one instance:

Client
│
▼
Go API

Later, your application grows:

Client
│
▼
Load Balancer
│
├── Go API #1
├── Go API #2
└── Go API #3

Now your Go application needs to behave correctly when multiple instances are running.

This means thinking about:

• Shared state

• Database connections

• Authentication

• Sessions

• Caching

• Idempotency

• Concurrency

• Timeouts

• Retries

• Graceful shutdown

• Health endpoints

This is where load balancing becomes more than just a networking concept.

It starts influencing how you design backend applications.

A Simple Mental Model

When learning load balancing, remember this:

MANY CLIENTS
│
▼
LOAD BALANCER
│
┌──────────┼──────────┐
▼ ▼ ▼
SERVER A SERVER B SERVER C
│ │ │
└──────────┼──────────┘
│
▼
SHARED DATA

The load balancer answers one fundamental question:

"Which backend should handle this request?"

The answer can depend on:

• Round Robin

• Server weight

• Active connections

• Client identity

• Request information

• Server health

Once you understand that, the rest of the topic becomes much easier.

Final Thoughts

Load balancing is one of the fundamental building blocks of scalable backend systems.

When an application is small, one server may be enough.

But as traffic grows, relying on a single server creates bottlenecks and increases the impact of failures.

Load balancing allows us to distribute traffic across multiple backend instances while improving scalability, availability, and fault tolerance.

The important thing isn't just memorizing algorithms like Round Robin or Least Connections.

It's understanding why load balancing exists in the first place.

A scalable backend is rarely just:

Client → Server

Instead, it increasingly becomes:

Client
│
▼
DNS
│
▼
Load Balancer
│
├──────► Backend 1
├──────► Backend 2
└──────► Backend 3
│
▼
Database / Cache

And this is an important shift in thinking for backend engineers:

You're no longer just writing code that works on one machine. You're designing systems that continue working when traffic, users, and failures increase.

That is where concepts like load balancing start to matter.