Every time you open a webpage, send a message, or stream a video, you're using TCP. It's the backbone of the internet—reliable, ordered, connection-oriented data delivery. And yet, most programmers never touch the socket layer directly.
I built a TCP server from scratch to understand what happens beneath the abstractions. This article walks through the process, from raw sockets to handling multiple clients. The code is in C because that's where the system calls live.
What We're Building
We'll build a TCP echo server—it receives data from clients and sends it back. Simple, but it covers all the fundamentals: socket creation, binding, listening, accepting connections, and I/O.
By the end, we'll have a server that:
- Handles multiple concurrent clients
- Uses non-blocking I/O with epoll
- Handles errors gracefully
- Cleans up resources properly
The Basics: Socket Creation
A socket is an endpoint for communication. Creating one is the first step:
#include <sys/socket.h>
#include <netinet/in.h>
int main() {
// Create a socket
// AF_INET = IPv4, SOCK_STREAM = TCP
int server_fd = socket(AF_INET, SOCK_STREAM, 0);
if (server_fd == -1) {
perror("socket failed");
return 1;
}
// Allow address reuse (helpful during development)
int opt = 1;
setsockopt(server_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
}
socket() returns a file descriptor—an integer representing the socket. Everything in Unix is a file, including network connections.
Binding to an Address
A server needs an address to listen on. We bind the socket to a port:
struct sockaddr_in address;
address.sin_family = AF_INET;
address.sin_addr.s_addr = INADDR_ANY; // Accept connections on any interface
address.sin_port = htons(8080); // Port 8080, network byte order
if (bind(server_fd, (struct sockaddr *)&address, sizeof(address)) < 0) {
perror("bind failed");
return 1;
}
Note htons()—it converts the port number to network byte order (big-endian). Network protocols use big-endian; your machine might use little-endian. Always convert.
Listening and Accepting
Once bound, the socket can listen for incoming connections:
// Start listening, with a backlog of 10 pending connections
if (listen(server_fd, 10) < 0) {
perror("listen failed");
return 1;
}
printf("Server listening on port 8080\n");
// Accept a connection
struct sockaddr_in client_addr;
socklen_t client_len = sizeof(client_addr);
int client_fd = accept(server_fd, (struct sockaddr *)&client_addr, &client_len);
if (client_fd < 0) {
perror("accept failed");
return 1;
}
printf("Client connected\n");
accept() blocks until a client connects. It returns a new file descriptor for the client connection. The original server_fd continues listening for more clients.
Reading and Writing
With a connected client, we can read and write data. For an echo server:
char buffer[1024];
while (1) {
// Read from client
ssize_t bytes_read = read(client_fd, buffer, sizeof(buffer) - 1);
if (bytes_read <= 0) {
if (bytes_read == 0) {
printf("Client disconnected\n");
} else {
perror("read failed");
}
break;
}
buffer[bytes_read] = '\0';
printf("Received: %s", buffer);
// Echo back to client
write(client_fd, buffer, bytes_read);
}
close(client_fd);This works but has a problem: it handles only one client. While talking to one client, other clients wait.
The Concurrency Problem
A real server handles many clients simultaneously. There are several approaches:
- Fork per connection — Simple but expensive (process overhead)
- Thread per connection — Better, but still limited (thread overhead)
- Event-driven I/O — Efficient, handles thousands of connections
We'll use epoll, Linux's event notification mechanism for scalable I/O.
Non-Blocking Sockets and epoll
First, we make our sockets non-blocking. A blocking socket waits until data is available. A non-blocking socket returns immediately with EAGAIN if no data is ready.
#include <fcntl.h>
void set_nonblocking(int fd) {
int flags = fcntl(fd, F_GETFL, 0);
fcntl(fd, F_SETFL, flags | O_NONBLOCK);
}Now, epoll. It lets us wait on multiple file descriptors efficiently:
#include <sys/epoll.h>
#define MAX_EVENTS 64
int epoll_fd = epoll_create1(0);
if (epoll_fd == -1) {
perror("epoll_create1");
return 1;
}
// Add the server socket to epoll
struct epoll_event ev;
ev.events = EPOLLIN; // Notify on readable
ev.data.fd = server_fd;
epoll_ctl(epoll_fd, EPOLL_CTL_ADD, server_fd, &ev);The event loop waits for activity on any registered socket:
struct epoll_event events[MAX_EVENTS];
while (1) {
int nfds = epoll_wait(epoll_fd, events, MAX_EVENTS, -1);
for (int i = 0; i < nfds; i++) {
if (events[i].data.fd == server_fd) {
// New connection on server socket
int client_fd = accept(server_fd, NULL, NULL);
set_nonblocking(client_fd);
// Add client to epoll
ev.events = EPOLLIN | EPOLLET; // Edge-triggered
ev.data.fd = client_fd;
epoll_ctl(epoll_fd, EPOLL_CTL_ADD, client_fd, &ev);
} else {
// Data from existing client
handle_client(events[i].data.fd);
}
}
}Edge-Triggered vs Level-Triggered
Notice EPOLLET? That's edge-triggered mode. The difference matters:
- Level-triggered: epoll notifies as long as data is available
- Edge-triggered: epoll notifies only when new data arrives
Edge-triggered is more efficient but requires you to read all available data in one go:
void handle_client(int fd) {
char buffer[1024];
while (1) {
ssize_t bytes = read(fd, buffer, sizeof(buffer));
if (bytes == -1) {
if (errno == EAGAIN || errno == EWOULDBLOCK) {
// No more data available right now
break;
}
perror("read");
close(fd);
return;
}
if (bytes == 0) {
// Connection closed
close(fd);
return;
}
// Echo back
write(fd, buffer, bytes);
}
}Graceful Shutdown
A proper server handles SIGINT (Ctrl+C) gracefully, closing all connections:
#include <signal.h>
volatile sig_atomic_t running = 1;
void handle_signal(int sig) {
running = 0;
}
int main() {
signal(SIGINT, handle_signal);
signal(SIGTERM, handle_signal);
// ... setup code ...
while (running) {
int nfds = epoll_wait(epoll_fd, events, MAX_EVENTS, 1000); // 1s timeout
if (nfds == -1) {
if (errno == EINTR) continue; // Interrupted by signal
break;
}
// ... handle events ...
}
// Cleanup
close(server_fd);
close(epoll_fd);
printf("Server shut down\n");
}Putting It All Together
The complete server structure:
int main() {
// 1. Create server socket
int server_fd = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0);
// 2. Set socket options
int opt = 1;
setsockopt(server_fd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
// 3. Bind to address
struct sockaddr_in addr = {
.sin_family = AF_INET,
.sin_addr.s_addr = INADDR_ANY,
.sin_port = htons(8080)
};
bind(server_fd, (struct sockaddr *)&addr, sizeof(addr));
// 4. Listen
listen(server_fd, SOMAXCONN);
// 5. Create epoll instance
int epoll_fd = epoll_create1(0);
// 6. Add server to epoll
struct epoll_event ev = {.events = EPOLLIN, .data.fd = server_fd};
epoll_ctl(epoll_fd, EPOLL_CTL_ADD, server_fd, &ev);
// 7. Event loop
struct epoll_event events[MAX_EVENTS];
while (running) {
int n = epoll_wait(epoll_fd, events, MAX_EVENTS, -1);
for (int i = 0; i < n; i++) {
if (events[i].data.fd == server_fd) {
accept_connection(server_fd, epoll_fd);
} else {
handle_client(events[i].data.fd);
}
}
}
// 8. Cleanup
close(server_fd);
close(epoll_fd);
}What I Learned
Building this taught me things no tutorial covered:
1. Error handling is half the code
Network code fails in countless ways. Clients disconnect unexpectedly. Sockets become invalid. Signals interrupt system calls. Robust error handling isn't optional.
2. Buffering is complicated
TCP is a stream, not a message protocol. You might receive half a message, or multiple messages at once. Real servers need proper buffer management and message framing.
3. Resource limits matter
File descriptors are limited. Memory is limited. Connections per second are limited. Production servers need to handle these limits gracefully.
4. The kernel does a lot
TCP retransmission, flow control, congestion control—the kernel handles all of this. Understanding what happens at the socket layer gives you appreciation for what you don't have to implement.
Going Further
This echo server is a foundation. From here, you could:
- Implement a protocol (HTTP, Redis, etc.)
- Add TLS encryption
- Benchmark with tools like wrk or netperf
- Compare with io_uring (Linux's newer async I/O)
- Port to other systems (kqueue on macOS/BSD)
The full source code is on GitHub as part of my tcp-server project, including a C++ version that uses modern features while keeping the same architecture.
Network programming is where software meets the physical world—cables, packets, latency. It's humbling and fascinating in equal measure.