Operating Systems
Processes, threads, scheduling, memory, file systems, and synchronization — the OS concepts every software engineer needs to understand, from first principles to interview depth.
1What is an Operating System
An Operating System (OS) is system software that acts as an intermediary between users/applications and hardware. It manages hardware resources (CPU, memory, disk, I/O devices), provides abstractions (files, processes, sockets), and enforces protection between programs.
Core OS responsibilities:
- Process management — create, schedule, and terminate processes and threads
- Memory management — allocate and free memory, implement virtual memory
- File system management — organize data on disk into files and directories
- I/O management — abstract device differences, provide uniform interface
- Security and protection — prevent processes from interfering with each other
- Networking — provide socket API, manage network stack
Kernel is the core of the OS — the part that runs in privileged (kernel) mode with direct hardware access. Everything else is user space. Types: Monolithic kernel (Linux, traditional Unix — all OS services in kernel space, fast but large), Microkernel (Mach, L4 — minimal kernel, most services in user space, more modular but slower IPC), Hybrid kernel (Windows NT, macOS XNU — blend of both).
System calls are the interface between user space and kernel. When a process needs to read a file, create a socket, or allocate memory, it invokes a system call. This triggers a mode switch from user mode (ring 3 on x86) to kernel mode (ring 0). The kernel validates the request, performs the operation, and returns to user mode. Examples: open(), read(), write(), fork(), exec(), mmap(), socket(), accept().
Interrupts are signals that tell the CPU to stop what it's doing and handle an event. Hardware interrupts come from devices (keyboard, NIC, timer). Software interrupts (traps) are triggered intentionally by programs (system calls, divide-by-zero). The interrupt handler (ISR) runs in kernel mode, services the interrupt, and control returns.
2Processes
A process is a running instance of a program. It's the OS's unit of resource allocation. Each process has its own independent address space containing: text segment (code), data segment (global/static variables), heap (dynamic allocation), stack (function calls, local variables), and metadata maintained by the OS.
Process Control Block (PCB) is the OS's data structure representing a process. Contains: PID (Process ID), process state, program counter (next instruction), CPU registers, memory management info (page tables), open file descriptors, scheduling priority, parent PID, signal masks. The OS switches between processes by saving/restoring PCBs.
Process states:
- New — process being created
- Ready — waiting to be assigned to a CPU (in the ready queue)
- Running — instructions being executed on a CPU
- Blocked/Waiting — waiting for an event (I/O completion, signal, lock)
- Terminated/Zombie — finished but PCB not yet reaped by parent
Context switching is the OS saving the state of the currently running process and loading the state of the next process. The overhead includes: saving/restoring registers, flushing TLB (unless tagged), cache cold-start. Modern CPUs can context switch in ~1–10 microseconds but the true cost is cache pollution. Context switches happen due to: timer interrupt (time slice expired), I/O blocking, higher-priority process becoming ready, system call yielding CPU.
fork() and exec(): On Unix/Linux, fork() creates a child process as an exact copy of the parent (copy-on-write optimization — pages are only actually copied when written). fork() returns 0 to the child and the child's PID to the parent. exec() replaces the process's memory with a new program. Together, fork() + exec() = spawn a new program. wait() lets the parent collect the child's exit status (preventing zombies).
Process vs Program: A program is passive (code on disk). A process is active (program in execution with resources). One program can have multiple processes (e.g., Chrome spawns a process per tab).
3Threads
A thread is a lightweight unit of execution within a process. Threads within the same process share: address space (code, heap, global data), open file descriptors, and signal handlers. Each thread has its own: stack, register set (including program counter), and thread ID.
Why threads? Parallelism — multiple threads can run on different CPU cores simultaneously. Concurrency — one thread can continue while another waits for I/O. Better resource sharing than multiple processes (no IPC overhead). Faster to create/destroy than processes.
User-level threads vs Kernel-level threads:
- User-level threads — managed by a user-space library (green threads, goroutines, Python coroutines). Kernel sees one thread. Fast create/context-switch. Problem: if one thread blocks on a system call, all threads block (unless the library intercepts syscalls).
- Kernel-level threads — managed by the OS kernel. True parallelism on multi-core. Each thread can independently block. Overhead: kernel context switch for each switch.
Threading models (many-to-many mappings):
- Many-to-One — N user threads → 1 kernel thread. No parallelism, one block blocks all.
- One-to-One — 1 user thread → 1 kernel thread. True parallelism. Used by Linux (pthreads), Windows threads. Overhead: each thread is a kernel thread.
- Many-to-Many — N user threads → M kernel threads (M ≤ N). Best of both. Used by Solaris, Go's goroutine scheduler (M:N scheduling with GOMAXPROCS kernel threads).
Thread pools pre-create a fixed number of threads waiting for work. Tasks are submitted to a queue; idle threads pick them up. Avoids the overhead of thread creation/destruction for each task. Used by web servers (thread-per-request model), database connection pools, Java's ExecutorService, Python's ThreadPoolExecutor.
Thread safety means code behaves correctly when multiple threads execute it concurrently. Problems arise when multiple threads access shared mutable state without synchronization. Race condition: output depends on non-deterministic timing of threads.
4CPU Scheduling
The CPU scheduler decides which ready process/thread runs next. Scheduling goals: maximize CPU utilization, maximize throughput (jobs/sec), minimize turnaround time (submit to complete), minimize waiting time (time in ready queue), minimize response time (for interactive systems).
Key metrics: Turnaround time = completion − arrival. Waiting time = turnaround − burst time. Response time = first response − arrival.
| Algorithm | Preemptive? | Key Idea | Pros | Cons |
|---|---|---|---|---|
| FCFS (First Come First Served) | No | Run in arrival order | Simple, fair | Convoy effect: short jobs wait behind long jobs |
| SJF (Shortest Job First) | No | Run shortest burst next | Optimal average waiting time | Requires knowing burst time; starvation of long jobs |
| SRTF (Shortest Remaining Time First) | Yes | Preempt if new job is shorter | Optimal avg waiting time (preemptive) | Starvation, overhead, burst time unknown |
| Round Robin | Yes | Fixed time quantum, cycle through | Fair, good response time | High context-switch overhead if quantum too small; poor throughput if large |
| Priority Scheduling | Both | Highest priority runs first | Important tasks get CPU | Starvation of low-priority jobs; fixed by aging (increase priority over time) |
| MLFQ (Multi-Level Feedback Queue) | Yes | Multiple queues with different priorities and time quantums; process moves between queues based on behavior | Approximates SJF without knowing burst time, good for interactive + batch mix | Complex to tune; starvation possible without aging |
MLFQ rules (Ousterhout): If priority(A) > priority(B), A runs. If priority(A) = priority(B), round-robin. A job starts at the highest priority. If it uses its time slice without blocking, it drops a priority level. If it gives up the CPU before the slice expires (I/O bound), it stays at the same priority. Periodically boost all jobs to top priority to prevent starvation.
Linux CFS (Completely Fair Scheduler): The default Linux scheduler since 2.6.23. Uses a red-black tree ordered by virtual runtime (vruntime). Each process accumulates vruntime while running. The scheduler always picks the process with the smallest vruntime. Nice values adjust the rate at which vruntime increases (lower nice = slower vruntime growth = more CPU time).
5Synchronization
When multiple threads access shared mutable state concurrently, correctness requires synchronization. Without it, race conditions produce non-deterministic, incorrect behavior.
Critical section: A piece of code that accesses shared resources and must not be executed by more than one thread at a time. Requirements for a correct solution: Mutual exclusion (only one thread in the critical section), Progress (if no thread is in the critical section, one that wants to enter must eventually), Bounded waiting (a thread can't wait forever).
Mutex (Mutual Exclusion Lock): Binary lock — locked or unlocked. lock() acquires the mutex (blocks if already locked). unlock() releases it. The thread that locks must be the one to unlock (ownership). Used to protect critical sections.
Semaphore: Integer counter with two atomic operations. wait() (P, down): decrement; block if value becomes negative. signal() (V, up): increment; wake a blocked thread if any. Binary semaphore (init=1) ≈ mutex but no ownership. Counting semaphore (init=N): limits concurrent access to N (connection pool of size N).
Spinlock: Instead of blocking, the thread loops ("spins") checking if the lock is free. No context switch overhead. Only beneficial if the wait is very short (lock held for nanoseconds) and on multi-core systems. Wastes CPU if held long. Used in OS kernels for very short critical sections.
Monitor: High-level synchronization construct — a class with synchronized methods and condition variables. Only one thread can execute inside the monitor at a time. Condition variables (wait, signal, broadcast) let threads wait for a condition without holding the lock. Java's synchronized keyword and Object.wait()/notify() implement monitors.
Common concurrency bugs:
- Race condition — outcome depends on timing. Fix: use locks or atomic operations.
- Deadlock — threads wait for each other's locks forever. Fix: lock ordering, timeouts, or deadlock detection.
- Livelock — threads keep responding to each other without making progress (like two people in a hallway stepping the same direction). Fix: randomized backoff.
- Priority inversion — low-priority thread holds a lock needed by high-priority thread. Fix: priority inheritance (temporarily raise the lock-holding thread's priority).
- ABA problem — in lock-free programming, a value changes from A→B→A; CAS (compare-and-swap) succeeds thinking nothing changed. Fix: version counter or hazard pointers.
6Deadlocks
A deadlock occurs when a set of processes are each waiting for a resource held by another process in the set — a circular wait from which no process can proceed.
Four necessary conditions (Coffman conditions): All four must hold simultaneously for deadlock to occur. Breaking any one prevents deadlock.
- Mutual Exclusion — at least one resource is non-shareable (only one process can use it at a time)
- Hold and Wait — a process holds at least one resource while waiting to acquire additional resources held by others
- No Preemption — resources cannot be forcibly taken; a process must release them voluntarily
- Circular Wait — there exists a cycle P1 → P2 → … → Pn → P1, each waiting for the next
Deadlock handling strategies:
- Prevention — structurally break one of the four conditions. Example: require all locks to be acquired in a fixed global order (breaks circular wait). Or: require a process to request all resources at once at startup (breaks hold-and-wait).
- Avoidance — use knowledge of future resource requests to avoid unsafe states. Banker's Algorithm: before granting a resource, simulate the allocation and check if the system remains in a "safe state" (there exists a sequence where all processes can finish). Expensive — requires declaring max resource needs upfront.
- Detection and Recovery — allow deadlocks to occur, periodically detect them (resource allocation graph, cycle detection), then recover by: killing one or more processes, preempting and rolling back resources.
- Ostrich Algorithm — ignore the problem. Used in most general-purpose OSes (Windows, Linux) — deadlocks are rare enough that the cost of prevention/avoidance outweighs the benefit. Let the user kill the program if it hangs.
Resource Allocation Graph: Directed graph where nodes are processes and resources. Request edge: P → R (process wants resource). Assignment edge: R → P (resource assigned to process). A cycle in the graph (with single-instance resources) indicates a deadlock.
7Memory Management
Memory management is the OS component responsible for allocating, tracking, and reclaiming memory. Goals: maximize memory utilization, provide each process with an isolated address space, support sharing where needed.
Contiguous allocation (early systems): Each process occupies a single contiguous region of physical memory. Simple but suffers from external fragmentation (free memory scattered in small pieces) and fixed partition sizes.
Paging: Divides physical memory into fixed-size frames (typically 4KB). Divides each process's virtual address space into same-size pages. A page table maps virtual page numbers (VPN) to physical frame numbers (PFN). Eliminates external fragmentation. Internal fragmentation: last page may not be fully used.
Address translation: Virtual address = VPN | Offset. MMU (Memory Management Unit) looks up VPN in page table → gets PFN → physical address = PFN | Offset. On a 64-bit system with 4KB pages, VPN is 52 bits and offset is 12 bits. Multi-level page tables (2, 3, or 4 levels — x86-64 uses 4-level) reduce memory needed for page tables by only allocating entries that are used.
TLB (Translation Lookaside Buffer): On-chip cache of recent VPN→PFN translations. Without TLB, every memory access requires a page table walk (multiple memory accesses). With TLB, most translations take one cycle. TLB hit rate is typically >99% for most programs. On context switch, TLB must be flushed (or tagged with ASID — Address Space ID — to avoid flushing).
Segmentation: Divides address space into variable-size logical segments (code, data, stack, heap, library). Each segment has a base address and limit. Provides protection (can't read another segment's data), but causes external fragmentation. x86 originally used segmentation; modern OSes mostly use paging (or paging with segmentation disabled/flat model).
Memory protection: Each page table entry has permission bits: present (is the page in RAM?), read, write, execute. Accessing a page without the right permission → page fault → OS handles (may be a segfault or a legitimate demand-paging case).
8Virtual Memory
Virtual memory allows processes to use more memory than physically available by using disk as an extension of RAM. Each process gets a large virtual address space (e.g., 128TB on x86-64), even if physical RAM is 16GB. Pages not currently needed are stored on disk (swap space / page file).
Demand paging: Pages are loaded into RAM only when accessed (on demand), not at program start. First access to a page that's not in RAM triggers a page fault. The OS: pauses the faulting process, finds a free frame (possibly evicting another page), loads the page from disk, updates the page table, resumes the process.
Page replacement algorithms choose which page to evict when RAM is full:
- OPT (Optimal) — evict the page that will be used farthest in the future. Theoretically best but requires knowing the future. Used as a benchmark.
- FIFO — evict the oldest page in memory. Simple but can evict heavily used pages. Suffers Belady's anomaly (more frames can cause more faults).
- LRU (Least Recently Used) — evict the page accessed least recently. Close to optimal in practice. Expensive to implement exactly (need to track access times). Approximated with a reference bit and clock algorithm.
- CLOCK (Second Chance) — circular list of pages with a reference bit. Hand sweeps around; if ref bit=1, clear it and move on; if ref bit=0, evict. Approximates LRU with O(1) overhead. Used by Linux.
- LFU (Least Frequently Used) — evict the page accessed fewest times. Can be slow to forget old frequently-used pages. Redis uses an approximation.
Thrashing: When a process is spending more time paging than executing. Happens when working set size exceeds available RAM. Too many page faults → OS spending all time loading/evicting pages. Fix: add more RAM, reduce multiprogramming degree, use working set model to suspend some processes.
Copy-on-Write (COW): After fork(), parent and child share the same physical pages, marked read-only. When either writes to a page, a fault occurs and the OS copies only that page. Makes fork() O(1) instead of O(address space size).
mmap(): Maps a file or anonymous memory region directly into the process's virtual address space. Reading/writing the memory region reads/writes the file. Used for: memory-mapped files, shared memory between processes (MAP_SHARED), large allocations (malloc uses mmap for large chunks).
9File Systems
A file system organizes data on storage devices into files and directories, manages free space, and provides metadata. It bridges the gap between the byte-level block device and the filename-based interface programs expect.
Inode (Unix/Linux): A data structure that stores file metadata — everything except the filename and file data. Contains: file type (regular, directory, symlink, device), permissions (rwxrwxrwx), owner UID/GID, size, timestamps (atime, mtime, ctime), link count, and pointers to data blocks. A directory is a file that maps filenames to inode numbers. Soft links (symlinks) point to a path. Hard links point to the same inode.
Block allocation: Inodes point to data blocks. Traditional Unix inodes use: direct pointers (12 blocks), single indirect pointer (block of pointers), double indirect, triple indirect. ext4 uses extents (contiguous range of blocks) instead, reducing fragmentation and improving sequential I/O.
Common file systems:
- ext4 — default Linux FS. Journaling, extents, large file support (16TB), backward compatible with ext2/3. Checksums on journal. Good general-purpose choice.
- XFS — high-performance, excellent for large files and parallel I/O. Default on RHEL. B+tree directory indexing.
- Btrfs — modern Linux FS. Copy-on-write, snapshots, built-in RAID, checksums on data and metadata. Still maturing.
- NTFS — Windows default. Journaling, file permissions, compression, encryption (EFS), alternate data streams. MFT (Master File Table) is central metadata store.
- FAT32 — simple, highly compatible. No permissions, no journaling, 4GB file size limit. Used for USB drives, SD cards for cross-OS compatibility.
- APFS — macOS/iOS default. Copy-on-write, native encryption, snapshots, space sharing between volumes. Optimized for SSDs and flash.
- ZFS — combined FS + volume manager. Checksums everything, 128-bit, snapshots, RAID-Z, excellent data integrity. Popular in storage servers and FreeBSD.
Journaling: Writes changes to a journal (log) before applying them to the filesystem. On crash, journal is replayed. Prevents filesystem corruption. Modes: write-back (only metadata journaled, fastest), ordered (data written before metadata is committed, default in ext4), data journaling (data and metadata journaled, safest but slowest).
VFS (Virtual File System): Linux abstraction layer that allows different file systems to be accessed through the same system call interface. open("/tmp/file") works whether /tmp is ext4, tmpfs, NFS, or a FUSE filesystem. VFS defines a standard set of objects (superblock, inode, dentry, file) that each filesystem driver implements.
10I/O Management
I/O management abstracts diverse hardware devices and provides uniform interfaces. I/O is one of the biggest performance bottlenecks — CPU is orders of magnitude faster than disk or network.
I/O techniques:
- Programmed I/O (Polling) — CPU continuously checks if device is ready. Wastes CPU cycles. Only used for fast devices or when simplicity matters.
- Interrupt-driven I/O — CPU initiates I/O and continues other work. Device interrupts CPU when operation completes. CPU handles interrupt, resumes original task. Good for slow devices. Overhead: interrupt handling, context switch.
- DMA (Direct Memory Access) — DMA controller transfers data directly between device and memory without CPU involvement. CPU sets up the transfer (source, destination, size), DMA does the work, interrupts CPU when done. Essential for high-bandwidth devices (NIC, SSD, GPU). CPU is free for useful work during the transfer.
Buffering: Temporary storage to smooth speed mismatch between producer and consumer. Single buffer: producer fills while consumer drains — stalls when buffer is full/empty. Double buffering: two buffers alternating roles — producer fills one while consumer drains the other. Circular buffer (ring buffer): fixed-size buffer with head and tail pointers, used in OS kernel I/O paths and network drivers.
Caching: OS maintains a page cache (buffer cache) — recently read disk blocks are kept in RAM. Reads check cache first (cache hit = fast). Writes go to cache (write-back) and are flushed to disk asynchronously by the pdflush/writeback daemon. This is why "sync" or "fsync()" is needed to guarantee data is on disk.
Spooling (Simultaneous Peripheral Operations Online): Queuing I/O jobs for a device that can only serve one job at a time (classic example: printer). Jobs are written to a spool directory; a daemon feeds them to the printer. Allows multiple processes to "print" concurrently without conflicts.
Device drivers: Software that translates generic OS I/O requests into device-specific commands. Runs in kernel space. A bad driver can crash the kernel. Modern approach: driver signing, microkernel user-space drivers.
I/O scheduling (disk): SSDs have near-uniform access time — NOOP scheduler (FIFO) is often best. HDDs have seek time — schedulers like CFQ (Completely Fair Queuing), Deadline, and BFQ reorder requests to minimize head movement (elevator algorithm). Linux default for NVMe SSDs: mq-deadline or none.
11Inter-Process Communication
Since processes have isolated address spaces, they need explicit mechanisms to communicate. IPC (Inter-Process Communication) mechanisms:
- Pipes — unidirectional byte stream between related processes (parent-child). Anonymous pipes exist only while both ends are open. Named pipes (FIFOs) have filesystem names, allowing unrelated processes to communicate. The shell "pipe" (cmd1 | cmd2) is an anonymous pipe — stdout of cmd1 is connected to stdin of cmd2.
- Message Queues — OS-maintained queue of discrete messages. Sender puts message in queue; receiver reads. Messages can have priority. Asynchronous — sender doesn't wait for receiver. Accessible by name (POSIX MQs) or key (System V MQs). Used when you need structured, priority-ordered communication.
- Shared Memory — fastest IPC. Two or more processes map the same physical memory region into their address spaces. One writes, another reads with no copying. Requires explicit synchronization (semaphores, mutexes) to avoid race conditions. Used by databases (PostgreSQL shared buffers), high-performance message buses.
- Signals — asynchronous notifications sent to a process. Predefined signal types: SIGKILL (kill, can't be caught), SIGTERM (terminate, can be caught for cleanup), SIGSEGV (segfault), SIGINT (Ctrl+C), SIGCHLD (child terminated), SIGUSR1/2 (user-defined). Signal handlers run asynchronously, interrupting normal execution — must use only async-signal-safe functions.
- Sockets — bidirectional communication. Unix domain sockets (AF_UNIX) for same-machine IPC via filesystem path (much faster than TCP loopback — no network stack). TCP/UDP sockets for network communication. Used by: Docker daemon, Redis, most microservices.
- Memory-mapped files (mmap) — with MAP_SHARED, two processes mapping the same file share the same physical pages. Writes are immediately visible to both. Backed by filesystem — changes are persistent. Used for: log tailing, shared configuration, database buffer pools.
- D-Bus — high-level IPC system used in Linux desktop environments. Message-passing system with object model and service discovery. Used by systemd, NetworkManager, BlueZ.
12OS Security
Privilege rings (x86): Intel CPUs define 4 privilege levels (rings 0–3). Ring 0 (kernel mode) has full hardware access. Ring 3 (user mode) is restricted — can't access hardware directly, can't modify page tables, can't disable interrupts. Modern OSes use only rings 0 and 3. Hypervisors add ring -1 (VMX root mode). Switching from ring 3 to ring 0 requires a system call (SYSCALL instruction) or interrupt.
Capabilities (Linux): Traditional Unix: root (UID 0) has all privileges, others have none. Linux capabilities break root privileges into discrete units: CAP_NET_BIND_SERVICE (bind to ports < 1024), CAP_SYS_PTRACE (trace other processes), CAP_DAC_OVERRIDE (bypass file permissions), CAP_SYS_ADMIN (many admin operations). Containers (Docker) drop most capabilities by default for security.
Mandatory Access Control (MAC): Traditional Unix uses DAC (Discretionary Access Control) — file owner sets permissions. MAC adds system-wide policy enforced by the kernel regardless of owner settings. SELinux (Red Hat, Android): labels on files and processes, policy defines which labels can interact. AppArmor (Ubuntu): profiles per executable, simpler than SELinux. Both are implemented as Linux Security Modules (LSMs).
Buffer overflow: Writing past the end of a stack buffer can overwrite the return address, redirecting execution to attacker-controlled code (shellcode). Classic attack on C programs. Mitigations:
- Stack canaries — OS places a random value (canary) before the return address. Buffer overflow overwrites the canary. On function return, kernel checks if canary changed; if so, terminate. Enabled by -fstack-protector in GCC.
- ASLR (Address Space Layout Randomization) — randomizes where stack, heap, and libraries are loaded. Makes it harder to predict target addresses for exploits. Enabled by default on Linux, macOS, Windows.
- NX/DEP (No-Execute / Data Execution Prevention) — marks stack and heap pages as non-executable. Shellcode injected there can't run. Bypassed by ROP (Return-Oriented Programming).
- PIE (Position Independent Executable) — code compiled to run at any address, enabling ASLR to randomize the code segment too.
- Safe languages — use Rust, Go, Java, Python instead of C/C++. Memory-safe languages prevent buffer overflows by design.
namespaces and cgroups (Linux containers): Linux namespaces isolate process views of the system: PID namespace (isolated PID numbering), network namespace (separate network stack), mount namespace (separate filesystem view), user namespace (separate UID mapping). cgroups limit resource usage: CPU, memory, disk I/O, network bandwidth. Together, these are what Docker and Kubernetes pods use for isolation — not separate VMs, just isolated process groups.
13Common Interview Questions
These are the operating systems questions most frequently asked at Google, Amazon, Meta, Microsoft, and other top tech companies. Each answer is written at interview depth — enough to demonstrate understanding without over-explaining.
Q1: What is the difference between a process and a thread?
A process is an independent program in execution with its own address space, file descriptors, and system resources. A thread is a unit of execution within a process — threads share the address space, heap, and file descriptors of their parent process but each has its own stack and register set. Creating a thread is faster than creating a process (~10x) because no new address space is needed. Use multiple threads for parallelism within one program (e.g., a web server handling concurrent requests). Use multiple processes for isolation — a crash in one process doesn't affect others (e.g., Chrome's process-per-tab architecture).
Q2: What is a context switch and what triggers it?
A context switch is the OS saving the state of the currently running process (registers, program counter, memory map) to its PCB and loading the state of the next scheduled process. Triggers: (1) timer interrupt — the current process's time quantum expired, (2) I/O blocking — the process issued a blocking system call and must wait, (3) a higher-priority process became runnable, (4) the process voluntarily yielded the CPU. Cost: register save/restore (~100ns) + TLB flush (switching address spaces — expensive, ~1–10µs effective cost from cache cold start). Context switches between threads of the same process are cheaper because the address space is shared — no TLB flush needed.
Q3: What are the four necessary conditions for deadlock?
All four must hold simultaneously for deadlock to occur — remove any one to prevent it: (1) Mutual Exclusion — at least one resource is held in non-shareable mode. (2) Hold and Wait — a process holds at least one resource while waiting to acquire additional resources held by others. (3) No Preemption — resources cannot be forcibly taken from a process; they must be released voluntarily. (4) Circular Wait — a circular chain of processes each waiting for a resource held by the next. Prevention strategies: always acquire locks in a global fixed order (breaks circular wait), use timeout-based lock acquisition (simulates preemption), allocate all resources upfront (breaks hold-and-wait).
Q4: Explain virtual memory. What is a page fault?
Virtual memory gives each process the illusion of a large, contiguous private address space, regardless of physical RAM size. The OS and hardware (MMU) map virtual pages to physical frames on demand. Not all pages need to be in RAM simultaneously — pages are swapped to disk (swap space) when RAM is full. A page fault occurs when a process accesses a virtual page not currently mapped to a physical frame. The MMU raises an exception, the OS page fault handler runs: if the page is on disk, it is loaded into a free frame (possibly evicting another page using LRU or Clock), the page table is updated, and the faulting instruction is restarted. Minor page fault: page is in memory but not in the page table (zero-fill on demand). Major page fault: page must be read from disk (~10ms — 100,000x more expensive than a RAM access).
Q5: What is the difference between a mutex and a semaphore?
A mutex (mutual exclusion lock) has two states: locked and unlocked. Only the thread that locked it can unlock it. It protects a critical section — one thread enters at a time. A semaphore is a counter. A binary semaphore (counter 0 or 1) behaves like a mutex but any thread can signal it (release it), not just the one that waited. A counting semaphore (counter N) allows up to N threads to access a resource concurrently. Key differences: mutex has ownership (only the locker can unlock — prevents accidental unlock bugs). Semaphores are used for signaling between threads (producer signals consumer) and managing pools of identical resources (connection pool of N connections). In interviews, always clarify whether you need mutual exclusion (mutex) or resource counting/signaling (semaphore).
Q6: What is a race condition? How do you detect and prevent it?
A race condition occurs when the behavior of a program depends on the relative timing of events (thread scheduling, I/O) in a way that produces incorrect results. Example: two threads both read balance = 100, both add 50, and both write 150 — instead of the correct 200. Detection: thread sanitizers (ThreadSanitizer in GCC/Clang), stress testing with high concurrency, code review for shared mutable state. Prevention: protect shared mutable state with mutexes or atomic operations. Use immutable data where possible. Design with message passing (Go channels, actor model) instead of shared memory. The rule: if two threads access the same variable and at least one writes, synchronize them.
Q7: What is Copy-on-Write and how does fork() use it?
Copy-on-Write (CoW) is a lazy copying optimization. When fork() creates a child process, instead of immediately copying the entire parent's address space, the OS marks all pages as read-only and shared between parent and child. When either process tries to write to a shared page, a page fault triggers the OS to create a private copy of that page for the writing process — only then is the page actually copied. Result: if the child immediately calls exec() (the common pattern), almost no actual copying happens — fork()+exec() is very fast. CoW also applies to copy-on-write file systems (ZFS, Btrfs) and data structures (Redis fork for background save, Python's multiprocessing).
Q8: What is thrashing and how do you fix it?
Thrashing occurs when a system spends more time paging (moving pages between RAM and disk) than executing useful work. Cause: the total working set of active processes exceeds available physical RAM. The OS continuously evicts pages needed by one process to make room for another, causing a cascade of page faults. Symptoms: near-100% disk I/O, near-0% CPU utilization, system appears frozen. Fixes: (1) reduce the degree of multiprogramming — suspend some processes, (2) increase physical RAM, (3) use a working set model — only run processes whose full working set fits in RAM, (4) apply page replacement policies that respect locality (LRU approximations). Modern OS kernels use techniques like KSM (Kernel Same-page Merging) and memory cgroups to prevent thrashing in virtualized environments.
Q9: What is the difference between a spinlock and a mutex?
A spinlock busy-waits — the thread continuously polls in a loop until the lock is available. A mutex blocks — the thread is put to sleep by the OS and woken when the lock is released. Use spinlocks when: the critical section is very short (microseconds), the overhead of an OS context switch exceeds the wait time, or you're in kernel code where sleeping isn't allowed. Use mutexes when: the critical section may take milliseconds or the lock is frequently contended — spinning wastes CPU cycles that other threads could use. Modern mutexes often use a hybrid approach: spin briefly (adaptive spinning), then fall back to OS blocking if the lock isn't released quickly.
Q10: What is the TLB and why does it matter?
The TLB (Translation Lookaside Buffer) is a hardware cache in the MMU that stores recent virtual-to-physical address translations. Without TLB: every memory access requires 2–4 memory reads to walk the page table (one per page table level). With TLB: most translations are resolved in 1 CPU cycle. TLB hit rate is typically 99%+ due to locality of reference. TLB miss: the MMU walks the page table (hardware page table walker on x86) or traps to the OS (software-managed TLB on MIPS). TLB flush on context switch: when switching between processes with different address spaces, the TLB must be flushed (or tagged with address space IDs — ASIDs — to allow coexistence). TLB flushes are a significant hidden cost of context switching and are why kernel threads are faster to context-switch than processes.
Q11: What is a system call? What happens when you call read()?
A system call is the interface between user-space programs and the OS kernel. When a process needs OS services (file I/O, network, memory allocation), it invokes a system call which causes a controlled transition from user mode (ring 3) to kernel mode (ring 0). What happens when read() is called: (1) user program calls the C library wrapper which loads the syscall number into a register, (2) executes a special instruction (syscall on x86-64, svc on ARM), triggering a trap into the kernel, (3) kernel validates the file descriptor, checks permissions, (4) if data is in the page cache, copies it to the user buffer and returns — all in kernel mode, (5) if not cached, blocks the process, issues I/O to disk, and context-switches to another process, (6) when I/O completes, the process is woken, data copied, return value set, kernel mode exits back to user mode.
Q12: How does LRU page replacement work?
LRU (Least Recently Used) evicts the page that was least recently accessed, based on the principle of temporal locality — recently used pages are likely to be used again soon. Ideal LRU requires tracking the exact access time of every page — impractical in hardware. Approximations: Clock algorithm (second-chance) — pages in a circular list each have a reference bit. When a page is accessed, its bit is set. On eviction, scan: if bit is 0, evict; if bit is 1, clear it and move on (give it a second chance). NFU (Not Frequently Used) — maintain counters. Aging — shift counters right each interval and OR with the reference bit. Linux uses a variant of the Clock algorithm with active and inactive lists. In interviews, describe exact LRU, explain why it's impractical for hardware, then describe the Clock approximation.
Sources & Further Reading
This guide follows the curriculum of the most widely used OS courses and textbooks:
- Operating Systems: Three Easy Pieces (Arpaci-Dusseau & Arpaci-Dusseau), the free, peer-reviewed textbook used by university OS courses worldwide.
- MIT 6.S081 / xv6 book, a hands-on walkthrough of a real Unix-like kernel covering processes, interrupts, and file systems.
- The Linux Kernel documentation, the authoritative reference for scheduler, memory management, and locking internals.
- Operating System Concepts (Silberschatz, Galvin & Gagne), the classic "dinosaur book" this guide's topic order follows.
Related Topics