Binary Exploitation

Binary exploitation is finding and abusing vulnerabilities in compiled programs. Buffer overflows, format string bugs, heap corruption, return-oriented programming — you are operating at the level of memory and assembly.

This is the deepest technical track in security. It rewards patience and a genuine interest in how computers actually work.

Prerequisites

You need a working understanding of:

  • C — most vulnerable programs are written in C
  • x86/x64 assembly — you will be reading disassembled code
  • How the stack and heap work — memory layout, function calls, return addresses
  • Linux process memory — segments, permissions, ASLR, NX bit

If these are unfamiliar, start there before touching exploitation. CS:APP (free online) covers the fundamentals.

Core Concepts

Concept What it is
Stack buffer overflow Writing past the end of a stack buffer to overwrite the return address
Heap overflow Similar, but on the heap — more complex memory management
Format string bug Passing user input directly to printf — leaks memory or writes arbitrary values
Use-after-free Accessing memory after it has been freed — can corrupt allocator metadata
ROP (Return-Oriented Programming) Chain existing code gadgets to bypass NX (non-executable stack)
ASLR bypass Address Space Layout Randomization makes exploitation harder — leaking an address defeats it

How to Learn

pwn.college is the best structured curriculum for binary exploitation. Free, with interactive challenges.

Start here → pwn.college

LiveOverflow on YouTube explains binary exploitation concepts with real examples.

YouTube → LiveOverflow

Tools

Tool What it does
GDB The debugger. Add pwndbg or peda for better UI.
pwntools Python library for writing exploits — handles connections, packing, ROP
Ghidra Decompiler and disassembler. Free, made by the NSA.
IDA Free Industry standard disassembler. Free version has limits.
checksec Check what protections a binary has (ASLR, NX, stack canaries, PIE)
ROPgadget Find ROP gadgets in a binary

Start with GDB and pwntools. You will live in those two.

In CTF Environments

Pwn challenges give you a binary and often remote access to a running instance. Your goal is to exploit the binary to get a shell and read the flag.

Typical workflow:

# 1. Identify the binary
file challenge
checksec challenge      # what protections are enabled?

# 2. Find strings and function names
strings challenge
nm challenge | grep -i "func\|main\|win"   # look for a "win" function

# 3. Run it and see what it does
./challenge

# 4. Open in Ghidra or gdb
gdb ./challenge

Checksec output tells you what you are working with:

RELRO:    Partial RELRO
Stack:    No canary found    ← buffer overflow is viable
NX:       NX enabled         ← can't execute shellcode on stack, need ROP
PIE:      No PIE             ← fixed binary addresses, easier ROP

Basic buffer overflow in GDB:

gdb ./challenge
(gdb) run $(python3 -c "print('A'*200)")   # find crash point
(gdb) info registers                         # check RIP/EIP
(gdb) x/20x $rsp                            # inspect stack

pwntools skeleton for a remote pwn:

from pwn import *

elf = ELF("./challenge")
conn = remote("target.htb", 4444)   # or process("./challenge") for local

# Find offset with cyclic pattern
pattern = cyclic(200)
conn.sendline(pattern)
conn.wait()
# In gdb: examine crash, get offset from cyclic_find(value_at_rip)

offset = 72   # example
payload = b"A" * offset + p64(elf.symbols["win"])   # jump to win function

conn.sendline(payload)
conn.interactive()

Using AI for Binary Exploitation

Where it helps:

  • Reading assembly: Paste a disassembled function from Ghidra. Ask “what does this do and is there a vulnerability?” AI reads x86/x64 assembly well enough to identify buffer handling, format strings, and logic bugs.
  • pwntools syntax: The API is large. “How do I set up a ROP chain with pwntools to call system(’/bin/sh’)?” AI gives working code. The offsets are still yours to find.
  • Checksec interpretation: Paste the output. Ask “what does this mean for exploitation strategy?” AI explains what each protection blocks and what attacks remain viable.
  • GDB commands: gdb/pwndbg commands are not obvious. Ask for the command to do what you want — set a breakpoint, examine memory at an offset, print the stack.
  • Understanding exploit primitives: “What is a format string vulnerability and how do you get arbitrary write from it?” AI explains clearly with examples.

Where it fails:

  • The actual offsets: you have to run the binary and measure.
  • Custom heap allocator behavior: AI knows glibc heap internals but not custom ones.
  • Novel techniques: recent research (2023+) it may not have seen.

Real-World Context

Binary exploitation is central to:

  • Offensive security research and CVE discovery
  • Malware analysis (understanding how shellcode works)
  • Kernel and browser exploitation
  • Embedded systems and IoT security

It is niche but highly valued. People who can reliably exploit memory corruption bugs are rare.

How the Club Uses This

TODO: Add pwn challenges the club has solved, resources members recommend, and any binary exploitation workshop sessions.

References

Next ForensicsWhat digital forensics involves, from real incident response to CTF file analysis.