Microservices and Service Discovery

A monolith is a single deployable unit containing all application logic — web layer, business logic, and data access — compiled and deployed together. Early-stage products almost always start as monoliths: they are simple to develop, test, deploy, and debug. A single process means no network hops between components, no distributed tracing overhead, and easy transactions across the entire data model.

Microservices decompose a monolith into small, independently deployable services, each responsible for a single business domain. A social platform might split into: users service, posts service, notifications service, search service, and feed generation service — each with its own database, deployment pipeline, and scaling policy.

The key question is not "monolith or microservices" but "when does the monolith's coordination cost exceed the microservices' operational overhead?" For most teams, the answer is: start with a monolith structured as a modular codebase (clear boundaries between domains, no circular dependencies), then extract services when a specific module needs independent scaling, a different tech stack, or causes deployment bottlenecks.

Team scale: Conway's Law states that organizations design systems that mirror their communication structure. With 5 engineers, a monolith is almost always right. With 50 engineers across 8 teams, microservices with team ownership boundaries allow each team to move independently without coordinating with everyone else on every deployment.

Independent scaling: if your image processing service needs 100 GPU instances but your user profile service needs 3 CPU instances, splitting them lets you scale each appropriately. In a monolith, you scale everything together.

Technology heterogeneity: if one service needs Python for ML, another needs Go for low-latency networking, and another needs Java for existing library support, microservices allow each to use the best tool for the job.

Fault isolation: a bug in one microservice crashes that service, not the entire system. Bulkhead patterns, circuit breakers, and timeouts prevent cascading failures across service boundaries.

Deployment velocity: each service can be deployed independently, enabling multiple teams to release multiple times per day without coordination. In a monolith, every deployment is a full system deployment.

In a monolith, function calls between components are in-process. In a microservice architecture, every call is a network request — and each service needs to know where to find the others. This is the service discovery problem.

The challenge: service instances come and go as containers are scheduled, scaled up/down, or replaced after failures. IP addresses are ephemeral. Hardcoding IPs in configuration is not viable.

Client-side discovery: each service client queries a service registry (e.g., Consul, etcd, Eureka) to get a list of available instances of the target service, then uses a client-side load balancing algorithm to pick one. The client holds service registry logic.

Server-side discovery: the client sends requests to a load balancer or proxy (e.g., NGINX, AWS ALB, Kubernetes kube-proxy). The proxy queries the service registry and forwards the request to an available instance. The client just knows one stable address (the proxy). Simpler for clients but adds a network hop.

DNS-based discovery: service instances register their IP under a DNS name (e.g., users-service.internal). Clients resolve the DNS name to get an IP. Simple and works with any language without SDK dependencies. Used by Kubernetes (CoreDNS) and Consul. Downside: DNS TTLs mean changes propagate slowly; not ideal for services that scale aggressively.

A service registry is a database of all live service instances: their service name, IP, port, and health status. The registry receives heartbeats from instances and marks instances as unhealthy if heartbeats stop.

Consul: the most widely used service registry outside of Kubernetes. Provides service registration, health checking, key-value store, and DNS-based discovery. Supports multiple data centers.

etcd: a distributed key-value store used as the backing store for Kubernetes. Strong consistency via the Raft consensus protocol. Kubernetes controllers watch etcd for changes (e.g., pod IP assignments).

Zookeeper: an Apache coordination service used for leader election, configuration management, and service discovery. More complex to operate than Consul; less common for new projects.

Kubernetes: in a Kubernetes cluster, the control plane (etcd + API server + kube-proxy + CoreDNS) handles service discovery automatically. Services are given stable DNS names (my-service.my-namespace.svc.cluster.local) that resolve to a cluster-internal virtual IP, load-balanced across all healthy pods by kube-proxy.

Operational complexity: instead of one process to monitor, you have dozens (or hundreds). You need distributed tracing (Jaeger, Zipkin), centralized logging (Elasticsearch, Loki), metrics dashboards (Prometheus, Grafana), and alerting per service. The infrastructure investment is significant.

Network unreliability: in a monolith, a function call never fails due to a network error. In microservices, every inter-service call can fail, time out, or be slow. Services must implement retries with exponential backoff and jitter, circuit breakers (stop sending requests to a failing service), timeouts (never wait indefinitely), and bulkheads (isolate failures in thread pools or connection pools so one slow dependency does not block the entire service).

Distributed transactions: ACID transactions across multiple services are not natively supported. Patterns like Sagas (a sequence of local transactions with compensating transactions for rollback) replace database transactions. Sagas are complex to implement correctly.

Data consistency: each microservice owns its data store. Cross-service queries require either API calls (slow, fragile) or event-driven denormalization (copy data you need via events into your own store). Keeping multiple stores in sync with eventual consistency is a significant design challenge.