SnailSploit/Claude-Red / Skills/exploit-dev/offensive-crash-analysis
offensive-crash-analysis
UnverifiedSKILL: Week 4: Crash Analysis and Exploitability Assessment
Metadata
- Skill Name: crash-analysis
- Folder: offensive-crash-analysis
- Source: https://github.com/SnailSploit/offensive-checklist/blob/main/4-crash-analysis.md
Description
Week 4 exploit development curriculum. Crash triage and analysis methodology: WinDbg/GDB analysis, ASAN/MSAN output interpretation, exploitability assessment, register/stack trace reading, root cause identification. Use when analyzing crash dumps, assessing exploitability, or understanding fuzzer-generated crashes.
Trigger Phrases
Use this skill when the conversation involves any of:
crash analysis, crash triage, WinDbg, GDB, ASAN, MSAN, exploitability, stack trace, register dump, segfault, null deref, access violation, week 4
Instructions for Claude
When this skill is active:
- Load and apply the full methodology below as your operational checklist
- Follow steps in order unless the user specifies otherwise
- For each technique, consider applicability to the current target/context
- Track which checklist items have been completed
- Suggest next steps based on findings
Full Methodology
Week 4: Crash Analysis and Exploitability Assessment
Overview
created by AnotherOne from @Pwn3rzs Telegram channel.
After finding potential vulnerabilities through fuzzing (Week 2) or patch diffing (Week 3), the next critical step is analyzing crashes to determine if they're exploitable. This week focuses on crash triage, debugger mastery, and techniques for identifying how to reach vulnerable code paths from attacker-controlled input.
Once you've confirmed a crash is exploitable and built a PoC, you'll be ready for Basic Exploitation in Week 5.
Prerequisites
Before starting this week, ensure you have:
- A Windows VM (for WinDbg labs) and a Linux VM (for GDB/ASAN/CASR labs).
- Completed Week 2 fuzzing labs, including running AFL++ or libFuzzer against at least one C/C++ target
- Completed (or skimmed) Week 3 patch diffing labs:
- Familiar with Ghidriff/Diaphora diff reports and how to interpret changed functions
- Understand how to extract Windows updates and Linux kernel patches
- Reviewed at least one case study (CVE-2022-34718 EvilESP, CVE-2024-1086 nf_tables, or 7-Zip symlink bugs)
- Comfortable understanding from Week 1 of basic vulnerability classes (buffer overflow, UAF, integer bugs, info leaks) and their exploit primitives
Crash Analysis Decision Tree
Use this decision tree to select the appropriate tools and workflow for any crash you encounter:
┌─────────────────────────────────────────────────────────────────────┐
│ CRASH RECEIVED │
└─────────────────────────────────────────────────────────────────────┘
│
▼
┌───────────────────────┐
│ Source code available?│
└───────────────────────┘
│ │
Yes No
│ │
▼ ▼
┌─────────────────────┐ ┌──────────────────────────┐
│ Recompile with │ │ What platform? │
│ ASAN + UBSAN │ └──────────────────────────┘
│ (Day 2) │ │ │ │
└─────────────────────┘ │ │ │
│ Windows Linux Mobile
│ │ │ │
▼ ▼ ▼ ▼
┌─────────────────────┐ ┌───────┐ ┌───────┐ ┌───────────┐
│ Run crash input │ │WinDbg │ │Pwndbg │ │ Tombstone │
│ Get detailed report │ │+ TTD │ │+ rr │ │ + Frida │
└─────────────────────┘ │(Day 1)│ │(Day 1)│ │ (Future) │
│ └───────┘ └───────┘ └───────────┘
│ │ │ │
└─────────────┴────┬────┴─────────┘
│
▼
┌─────────────────────────────────────┐
│ Crash requires special environment? │
└─────────────────────────────────────┘
│ │
Yes No
│ │
▼ │
┌─────────────────────────────┐ │
│ Setup reproduction env: │ │
│ - Network (tcpdump, proxy) │ │
│ - Files (strace, procmon) │ │
│ - Services (docker, VM) │ │
└─────────────────────────────┘ │
│ │
└──────────────┬───────────────┘
│
▼
┌─────────────────────┐
│ Crash type known? │
└─────────────────────┘
│ │
Yes No
│ │
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ Run CASR for │ │ Manual analysis: │
│ classification │ │ - Examine registers │
│ (Day 3) │ │ - Check memory │
└─────────────────────┘ │ - Disassemble │
│ │ (Day 3) │
│ └─────────────────────┘
│ │
└────────┬────────┘
│
▼
┌─────────────────────────┐
│ EXPLOITABILITY ASSESS │
│ - Check mitigations │
│ - Control analysis │
│ - Reachability (Day 4) │
└─────────────────────────┘
│
▼
┌─────────────────────────┐
│ Multiple crashes? │
└─────────────────────────┘
│ │
Yes No
│ │
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ Deduplicate (Day 5) │ │ Minimize (Day 5) │
│ - CASR cluster │ │ - afl-tmin │
│ - Stack hash │ │ - Manual reduction │
└─────────────────────┘ └─────────────────────┘
│ │
└────────┬───────────┘
│
▼
┌─────────────────────────┐
│ Create PoC (Day 6) │
│ - Python + pwntools │
│ - Verify reliability │
│ - Document findings │
└─────────────────────────┘
Quick Reference - Tool Selection by Scenario:
| Scenario | Primary Tool | Secondary Tool | Sanitizer |
|---|---|---|---|
| Linux binary, have source | GDB + Pwndbg | rr | ASAN + UBSAN |
| Linux binary, no source | GDB + Pwndbg | Ghidra | N/A |
| Windows binary, have source | WinDbg + TTD | Visual Studio | ASAN |
| Windows binary, no source | WinDbg + TTD | IDA/Ghidra | N/A |
| Fuzzer crash corpus | CASR | afl-tmin | ASAN |
| Non-deterministic crash | rr (Linux) / TTD (Windows) | Chaos mode | TSAN |
| Kernel crash (Linux) | crash utility | GDB + KASAN | KASAN |
| Kernel crash (Windows) | WinDbg kernel | Driver Verifier | N/A |
| Android app crash | Tombstone + ndk-stack | Frida | HWASan |
| Rust/Go crash | Native debugger | Sanitizer output | Built-in |
Day 1: Debugger Fundamentals and Crash Dump Analysis
- Goal: Learn Windows Debugger (WinDbg) and Linux debugger (GDB + Pwndbg) for analyzing application crashes.
- Activities:
- Reading:
- "Practical Malware Analysis" by Michael Sikorski - Chapter 9 and 10
- WinDbg Official Documentation
- Pwndbg Documentation
- Online Resources:
- Tool Setup:
- Windows: Install WinDbg Preview from Microsoft Store
- Linux: Install GDB with Pwndbg enhancement
- Install Windows SDK for symbol support
- Exercise:
- Analyze 5 pre-generated crash dumps (Windows and Linux)
- Identify crash type and root cause for each
- Reading:
Reproduction Fidelity
[!IMPORTANT] Before any crash analysis, ensure you can reproduce the crash reliably. A crash that only happens "sometimes" or "on the fuzzer's machine" is nearly impossible to analyze or exploit. This section establishes the mandatory checklist for achieving reproduction fidelity.
Reproduction Fidelity Checklist
Before analyzing any crash, verify these match between discovery and analysis environments:
┌─────────────────────────────────────────────────────────────────┐
│ REPRODUCTION FIDELITY CHECKLIST │
├─────────────────────────────────────────────────────────────────┤
│ System Environment │
│ [ ] OS/Kernel version : ________________________________ │
│ [ ] libc version : ________________________________ │
│ [ ] CPU architecture : [ ] x86 [ ] x86_64 [ ] ARM64 │
│ [ ] Container/VM : [ ] Native [ ] Docker [ ] VM │
│ [ ] ASLR state : [ ] Enabled [ ] Disabled │
├─────────────────────────────────────────────────────────────────┤
│ Process Environment │
│ [ ] argv (command-line) : ________________________________ │
│ [ ] Environment variables : ________________________________ │
│ [ ] Working directory : ________________________________ │
│ [ ] Locale (LC_ALL, LANG) : ________________________________ │
│ [ ] umask / permissions : ________________________________ │
├─────────────────────────────────────────────────────────────────┤
│ Input Path │
│ [ ] Input source : [ ] stdin [ ] file [ ] network │
│ [ ] Input file path : ________________________________ │
│ [ ] Network port/protocol : ________________________________ │
├─────────────────────────────────────────────────────────────────┤
│ Build Configuration │
│ [ ] Compiler version : ________________________________ │
│ [ ] Optimization level : [ ] -O0 [ ] -O1 [ ] -O2 [ ] -O3 │
│ [ ] Sanitizers : [ ] ASAN [ ] UBSAN [ ] TSAN [ ] None│
│ [ ] Debug symbols : [ ] Yes [ ] No │
│ [ ] Mitigations : [ ] PIE [ ] Canary [ ] RELRO │
└─────────────────────────────────────────────────────────────────┘
Essential Environment Knobs
ASAN/UBSAN Options (Linux/macOS):
# Full ASAN options for crash analysis
export ASAN_OPTIONS="\
abort_on_error=1:\
symbolize=1:\
detect_leaks=1:\
disable_coredump=0:\
halt_on_error=1:\
print_stats=1:\
check_initialization_order=1:\
detect_stack_use_after_return=1:\
quarantine_size_mb=256"
# UBSAN options
export UBSAN_OPTIONS="\
print_stacktrace=1:\
halt_on_error=1:\
suppressions=ubsan_suppressions.txt"
# Symbolizer path (required for readable stack traces)
export ASAN_SYMBOLIZER_PATH=$(command -v llvm-symbolizer)
glibc Allocator Tuning (Linux):
# Enable glibc heap consistency checks (catch corruption early)
export MALLOC_CHECK_=3
# Modern glibc tunable interface (glibc 2.26+)
export GLIBC_TUNABLES="\
glibc.malloc.check=3:\
glibc.malloc.perturb=165"
# What these do:
# MALLOC_CHECK_=3: Abort on heap corruption detection
# glibc.malloc.perturb=165: Fill freed memory with 0xA5 (helps detect UAF)
Core Dump Configuration (Linux):
# Enable unlimited core dumps
ulimit -c unlimited
# Verify core pattern (where dumps go)
cat /proc/sys/kernel/core_pattern
# For local dumps in CWD (temporary, affects system):
# echo 'core.%e.%p' | sudo tee /proc/sys/kernel/core_pattern
ASLR Control (Linux - for deterministic analysis):
# Check current ASLR state
cat /proc/sys/kernel/randomize_va_space
# 0 = disabled, 1 = conservative, 2 = full
# Disable ASLR for current shell (temporary, per-process)
setarch $(uname -m) -R ./target < crash_input
# Or system-wide (DANGEROUS - only for isolated VMs):
# echo 0 | sudo tee /proc/sys/kernel/randomize_va_space
Input Path Matching
The crash may behave differently depending on HOW input reaches the target:
# If fuzzer used stdin:
./target < crash_input
# If fuzzer used file argument:
./target crash_input
# If fuzzer used network:
cat crash_input | nc localhost 8080
# WRONG: Mixing input paths can change behavior!
# Fuzzer: ./target @@ (file)
# You: ./target < crash (stdin) # May not reproduce!
Example: stdin vs file difference:
// Some programs behave differently:
// - stdin may be line-buffered
// - File may be memory-mapped
// - Network may have different read chunk sizes
// This can affect:
// - Buffer contents at crash time
// - Heap layout (different allocation patterns)
// - Race conditions (timing changes)
Quick Reproduction Test Script
#!/bin/bash
# repro_test.sh - Verify crash reproduction
CRASH_INPUT="$1"
TARGET="$2"
EXPECTED_SIGNAL="${3:-11}" # Default: SIGSEGV (11)
echo "[*] Testing reproduction of $(basename $CRASH_INPUT)"
echo "[*] Target: $TARGET"
echo "[*] Expected signal: $EXPECTED_SIGNAL"
# Set up environment
ulimit -c unlimited
export ASAN_OPTIONS="abort_on_error=1:symbolize=1"
# Run 10 times
CRASHES=0
for i in {1..10}; do
timeout 5s $TARGET < "$CRASH_INPUT" 2>/dev/null
EXIT_CODE=$?
# Check for crash signal (128 + signal number)
if [ $EXIT_CODE -gt 128 ]; then
SIGNAL=$((EXIT_CODE - 128))
if [ $SIGNAL -eq $EXPECTED_SIGNAL ] || [ $SIGNAL -eq 6 ]; then
((CRASHES++))
fi
fi
done
echo "[*] Crash rate: $CRASHES/10"
if [ $CRASHES -ge 9 ]; then
echo "[+] Reproduction: RELIABLE"
elif [ $CRASHES -ge 5 ]; then
echo "[!] Reproduction: FLAKY - investigate environment"
else
echo "[-] Reproduction: FAILED - check environment checklist"
fi
Installing WinDbg and Symbol Support
WinDbg Preview (recommended - modern UI):
winget install Microsoft.WinDbg
Windows SDK Debugging Tools (includes cdb.exe for command-line/batch analysis):
# Option 1: Install via winget (Windows SDK)
winget install --source winget --exact --id Microsoft.WindowsSDK.10.0.26100
# Option 2: Download from Microsoft
# https://developer.microsoft.com/en-us/windows/downloads/windows-sdk/
# During installation, select "Debugging Tools for Windows"
# After installation, cdb.exe is located at:
# C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe
# Add to PATH for convenience (run as Administrator):
setx PATH "%PATH%;C:\Program Files (x86)\Windows Kits\10\Debuggers\x64" /M
# Or use full path in scripts:
"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe" -z dump.dmp -c "!analyze -v; q"
Configure Symbol Path:
# In WinDbg Settings -> Default Symbol Path, or:
# In WinDbg command window:
.sympath SRV*C:\Symbols*https://msdl.microsoft.com/download/symbols
# Or set environment variable permanently (recommended):
setx _NT_SYMBOL_PATH "SRV*C:\Symbols*https://msdl.microsoft.com/download/symbols"
# Create symbols cache directory
mkdir C:\Symbols
# Reload symbols (in debugger)
.reload /f
Linux Crash Dump Generation and Pwndbg Setup
[!HINT] While Windows uses WinDbg, Linux crash analysis uses GDB enhanced with Pwndbg. This section covers parallel Linux setup.
Installing Pwndbg:
# Install GDB
sudo apt install gdb
# Install Pwndbg (recommended for crash analysis)
cd ~/tools
git clone --depth 1 https://github.com/pwndbg/pwndbg
cd pwndbg
./setup.sh
# Verify installation
gdb -q -ex "quit" 2>&1 | grep -q "pwndbg" && echo "pwndbg installed successfully"
[!WARNING] Pwndbg is installed per-user in
~/.gdbinit. If you runsudo gdb, it uses root's home directory and won't find your pwndbg config. Solutions: For crash analysis of your own compiled test programs, you typically don't need sudo. Only use sudo when attaching to system processes or analyzing setuid binaries.
# Option 1: Use gdb as regular user (recommended for most analysis)
cd ~/crash_analysis_lab
gdb ./vuln_no_protect -c core.dump
# Option 2: If you MUST use sudo (e.g., attaching to privileged process)
sudo -E gdb ./program # -E preserves your environment including HOME
# Option 3: Install pwndbg for root as well
sudo su -
cd /root
git clone https://github.com/pwndbg/pwndbg
cd pwndbg && ./setup.sh
exit
# Option 4: Explicitly source pwndbg in sudo gdb session
sudo gdb -ex "source /home/<YOUR_USER>/tools/pwndbg/gdbinit.py" ./program
Configuring Core Dumps on Linux:
# Check current core dump configuration
cat /proc/sys/kernel/core_pattern
# Enable core dumps for current shell (recommended for learning)
ulimit -c unlimited
[!TIP] For the exercises in this course, you typically only need:
ulimit -c unlimited # In your current shellOn modern Ubuntu/Debian with systemd, cores are handled by
systemd-coredumpeven if you setulimit. Usecoredumpctlto list and debug them.
[!WARNING] Optional: Local core files in CWD (modifies system-wide settings)
If you specifically need core files in your working directory instead of systemd-coredump:
# This is SYSTEM-WIDE and may interfere with other tooling echo 'core.%e.%p' | sudo tee /proc/sys/kernel/core_patternAdditional kernel settings that affect core dumps:
kernel.core_uses_pid: Append PID to core filenamefs.suid_dumpable: Controls dumps for setuid binaries (0=disabled, 1=enabled, 2=suidsafe)
Building a Vulnerable Test Suite for Linux
Create these vulnerable C programs to generate real crashes:
# Create a directory for crash analysis practice
mkdir -p ~/crash_analysis_lab/{src,crashes,cores}
cd ~/crash_analysis_lab/src
vulnerable_suite.c - Save this file for testing multiple vulnerability types:
// ~/crash_analysis_lab/src/vulnerable_suite.c - Compile with different flags for different exercises
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
// 1. Stack Buffer Overflow
void stack_overflow(char *input) {
char buffer[64];
printf("[*] Copying input to 64-byte buffer...\n");
strcpy(buffer, input); // No bounds check!
printf("[*] Buffer: %s\n", buffer);
}
// 2. Heap Buffer Overflow
void heap_overflow(char *input) {
char *buf = malloc(32);
printf("[*] Allocated 32 bytes at %p\n", buf);
strcpy(buf, input); // Overflow heap buffer
printf("[*] Buffer: %s\n", buf);
free(buf);
}
// 3. Use-After-Free
void use_after_free() {
char *ptr = malloc(64);
strcpy(ptr, "Hello, World!");
printf("[*] Allocated at %p: %s\n", ptr, ptr);
free(ptr);
printf("[*] Freed, now accessing...\n");
printf("[*] UAF read: %s\n", ptr); // UAF read - may print stale data
ptr[0] = 'X'; // UAF write - may corrupt allocator state
}
// 4. Double Free
void double_free() {
char *ptr = malloc(64);
printf("[*] Allocated at %p\n", ptr);
free(ptr);
printf("[*] First free done\n");
free(ptr); // Double free!
}
// 5. NULL Pointer Dereference
void null_deref(int trigger) {
char *ptr = trigger ? malloc(10) : NULL;
printf("[*] ptr = %p\n", ptr);
*ptr = 'A'; // NULL deref if trigger is 0
}
void print_usage(char *prog) {
printf("Usage: %s <test_num> [input]\n", prog);
printf("Tests:\n");
printf(" 1 <input> - Stack overflow (need ~100+ chars)\n");
printf(" 2 <input> - Heap overflow (need ~50+ chars)\n");
printf(" 3 - Use-after-free\n");
printf(" 4 - Double free\n");
printf(" 5 <0|1> - NULL deref (0=crash)\n");
printf("\nExample: %s 1 $(python3 -c \"print('A'*100)\")\n", prog);
}
int main(int argc, char **argv) {
if (argc < 2) { print_usage(argv[0]); return 1; }
int test = atoi(argv[1]);
switch(test) {
case 1: if (argc<3) return 1; stack_overflow(argv[2]); break;
case 2: if (argc<3) return 1; heap_overflow(argv[2]); break;
case 3: use_after_free(); break;
case 4: double_free(); break;
case 5: if (argc<3) return 1; null_deref(atoi(argv[2])); break;
default: print_usage(argv[0]); return 1;
}
return 0;
}
Build the test suite:
cd ~/crash_analysis_lab/src
# 1. Build WITHOUT mitigations (for basic crash analysis)
gcc -g -fno-stack-protector -no-pie -z execstack \
vulnerable_suite.c -o ../vuln_no_protect
# 2. Build WITH ASAN (for detailed memory error reports)
gcc -g -O1 -fsanitize=address -fno-omit-frame-pointer \
vulnerable_suite.c -o ../vuln_asan
# 3. Build with standard protections (see how mitigations affect crashes)
gcc -g vulnerable_suite.c -o ../vuln_protected
Generate your first crashes:
cd ~/crash_analysis_lab
# Enable core dumps
ulimit -c unlimited
# Test 1: Stack overflow - generates a core dump
./vuln_no_protect 1 $(python3 -c "print('A'*200)")
# You should see: Segmentation fault (core dumped)
# Check for core file: ls -la core* (if core_pattern writes to CWD) or use coredumpctl (systemd systems) or look at output of `cat /proc/sys/kernel/core_pattern`
# Test 2: Stack overflow with ASAN - detailed report
./vuln_asan 1 $(python3 -c "print('A'*200)") 2>&1 | tee crashes/stack_asan.txt
# ASAN prints detailed overflow information
# Test 3: Use-after-free with ASAN
./vuln_asan 3 2>&1 | tee crashes/uaf_asan.txt
# Test 4: NULL dereference - generates core dump
./vuln_no_protect 5 0
Using coredumpctl (systemd systems):
sudo apt install systemd-coredump
# List recent core dumps
coredumpctl list
# Show details of most recent crash
coredumpctl info
# Debug most recent crash with GDB
coredumpctl debug
# Debug specific crash by PID
coredumpctl debug 12345
# Extract core dump to file for offline analysis
coredumpctl dump -o crash.core
# View where cores are stored
cat /etc/systemd/coredump.conf
# [Coredump]
# Storage=external # 'external' = /var/lib/systemd/coredump/
# Compress=yes
# MaxUse=1G # Max disk space for cores
Configuring systemd-coredump (/etc/systemd/coredump.conf):
[Coredump]
# Where to store cores: external (disk), journal, or none
Storage=external
# Compress with zstd/lz4
Compress=yes
# Maximum size for stored cores
ProcessSizeMax=2G
# Maximum total disk usage
MaxUse=5G
# Keep cores for this long
KeepFree=1G
After editing, reload: sudo systemctl daemon-reload
ASAN and Core Dumps
[!NOTE] ASAN often exits via SIGABRT, not SIGSEGV. This can be confusing when trying to capture core dumps.
# ASAN default: aborts on error (SIGABRT = signal 6)
# Core dumps may not be generated by default for SIGABRT
# Method 1: Configure ASAN to allow core dumps
export ASAN_OPTIONS="abort_on_error=1:disable_coredump=0"
# Method 2: Check that coredumpctl captures SIGABRT
# coredumpctl list
# Should show crashes with signal=6 (SIGABRT)
# Method 3: Use gdb to catch ASAN abort
echo "1 $(python3 -c "print('A'*200)")" > crash_input
gdb ./vuln_asan
(gdb) run < crash_input
# ASAN prints report, then GDB catches SIGABRT
(gdb) bt full # Get full backtrace
# What "success" looks like with ASAN + core dump:
# 1. ASAN prints detailed error report (allocation/free stacks)
# 2. Program aborts with SIGABRT
# 3. coredumpctl captures the core
# 4. coredumpctl debug lets you examine state at abort
Building Vulnerable Test Suite for Windows
Prerequisites:
- Visual Studio 2022 (Community edition is free) or Build Tools for Visual Studio
- Open "x64 Native Tools Command Prompt for VS 2022" for compilation
vulnerable_suite_win.c - Save this file for Windows crash analysis practice:
// C:\CrashAnalysisLab\src\vulnerable_suite_win.c
#include <windows.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
void stack_overflow(char *input) {
char buffer[64];
printf("[*] Copying input to 64-byte buffer...\n");
strcpy(buffer, input);
printf("[*] Buffer: %s\n", buffer);
}
void heap_overflow(char *input) {
char *buf = (char*)HeapAlloc(GetProcessHeap(), 0, 32);
printf("[*] Allocated 32 bytes at %p\n", buf);
strcpy(buf, input);
printf("[*] Buffer: %s\n", buf);
HeapFree(GetProcessHeap(), 0, buf);
}
void use_after_free() {
char *ptr = (char*)HeapAlloc(GetProcessHeap(), 0, 64);
strcpy(ptr, "Hello, World!");
printf("[*] Allocated at %p: %s\n", ptr, ptr);
HeapFree(GetProcessHeap(), 0, ptr);
printf("[*] Freed, now accessing...\n");
printf("[*] UAF read: %s\n", ptr);
ptr[0] = 'X';
}
void double_free() {
char *ptr = (char*)HeapAlloc(GetProcessHeap(), 0, 64);
printf("[*] Allocated at %p\n", ptr);
HeapFree(GetProcessHeap(), 0, ptr);
printf("[*] First free done\n");
HeapFree(GetProcessHeap(), 0, ptr);
}
void null_deref(int trigger) {
char *ptr = trigger ? (char*)HeapAlloc(GetProcessHeap(), 0, 10) : NULL;
printf("[*] ptr = %p\n", ptr);
*ptr = 'A';
}
void integer_overflow(unsigned int size) {
unsigned int alloc_size = size + 16;
if (alloc_size < size) {
printf("[*] Integer overflow detected! alloc_size=%u\n", alloc_size);
}
char *buf = (char*)HeapAlloc(GetProcessHeap(), 0, alloc_size);
printf("[*] Allocated %u bytes at %p\n", alloc_size, buf);
memset(buf, 'A', size);
HeapFree(GetProcessHeap(), 0, buf);
}
void print_usage(char *prog) {
printf("Windows Vulnerable Test Suite\n");
printf("==============================\n");
printf("Usage: %s <test_num> [input]\n\n", prog);
printf("Tests:\n");
printf(" 1 <input> - Stack overflow (need ~100+ chars)\n");
printf(" 2 <input> - Heap overflow (need ~50+ chars)\n");
printf(" 3 - Use-after-free\n");
printf(" 4 - Double free\n");
printf(" 5 <0|1> - NULL deref (0=crash)\n");
printf(" 6 <size> - Integer overflow (try 4294967280)\n");
printf("\nExamples:\n");
printf(" %s 1 ", prog);
printf("AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\n");
printf(" %s 5 0\n", prog);
}
int main(int argc, char **argv) {
if (argc < 2) { print_usage(argv[0]); return 1; }
int test = atoi(argv[1]);
switch(test) {
case 1: if (argc<3) return 1; stack_overflow(argv[2]); break;
case 2: if (argc<3) return 1; heap_overflow(argv[2]); break;
case 3: use_after_free(); break;
case 4: double_free(); break;
case 5: if (argc<3) return 1; null_deref(atoi(argv[2])); break;
case 6: if (argc<3) return 1; integer_overflow((unsigned int)strtoul(argv[2], NULL, 10)); break;
default: print_usage(argv[0]); return 1;
}
printf("[*] Test completed without crash\n");
return 0;
}
Build the Windows test suite:
# install visual studio community
# Open "x64 Native Tools Command Prompt for VS 2022"
# Create lab directory
mkdir C:\CrashAnalysisLab\src
mkdir C:\CrashAnalysisLab\dumps
cd C:\CrashAnalysisLab\src
# Save the source code above as vulnerable_suite_win.c, then:
# 1. Build WITHOUT mitigations (for basic crash analysis)
# /GS- disables stack cookies, /DYNAMICBASE:NO disables ASLR
cl /Zi /Od /GS- vulnerable_suite_win.c /Fe:..\vuln_win.exe /link /DYNAMICBASE:NO /NXCOMPAT:NO
# 2. Build WITH ASAN (Visual Studio 2019 16.9+ or VS 2022)
cl /Zi /Od /fsanitize=address vulnerable_suite_win.c /Fe:..\vuln_asan.exe
# 3. Build with standard protections (default mitigations)
cl /Zi /Od vulnerable_suite_win.c /Fe:..\vuln_protected.exe
Generate your first Windows crashes:
cd C:\CrashAnalysisLab
# Test 1: Stack overflow
vuln_win.exe 1 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
# Should crash with access violation
# crash will be at C:\CrashDumps\
# Test 2: Stack overflow with ASAN - detailed report
vuln_asan.exe 1 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
# ASAN prints detailed overflow information
# Test 3: Use-after-free with ASAN
vuln_asan.exe 3
# Test 4: NULL dereference
vuln_win.exe 5 0
# Test 5: Double free (may not crash immediately without PageHeap)
vuln_win.exe 4
# Directory of C:\CrashDumps
# 01/05/2026 03:11 PM <DIR> .
# 01/05/2026 03:10 PM 9,879,181 vuln_win.exe.7452.dmp
# 01/05/2026 03:11 PM 9,866,817 vuln_win.exe.7756.dmp
# 01/05/2026 03:09 PM 10,543,599 vuln_win.exe.984.dmp
Using PowerShell to generate long strings:
# PowerShell equivalent of Python one-liners
cd C:\CrashAnalysisLab
# Generate 200 'A' characters
$payload = "A" * 200
# Test stack overflow
.\vuln_win.exe 1 $payload
# Test with ASAN
.\vuln_asan.exe 1 $payload 2>&1 | Tee-Object -FilePath C:\CrashDumps\stack_asan.txt
# Test UAF with ASAN
.\vuln_asan.exe 3 2>&1 | Tee-Object -FilePath C:\CrashDumps\uaf_asan.txt
Verify crashes are captured:
# If WER LocalDumps is configured (see next section), check:
dir C:\CrashDumps\
# Or use Event Viewer:
# Windows Logs -> Application -> Look for "Application Error" events
WER/ProcDump Dump Collection
Windows Error Reporting (WER) LocalDumps
WER is Windows' built-in crash reporting. Configure it to save dumps locally:
Enable LocalDumps via Registry:
# Create LocalDumps key for ALL applications
reg add "HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps" /v DumpFolder /t REG_EXPAND_SZ /d "C:\CrashDumps" /f
reg add "HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps" /v DumpType /t REG_DWORD /d 2 /f
reg add "HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps" /v DumpCount /t REG_DWORD /d 10 /f
# DumpType values:
# 0 = Custom (use CustomDumpFlags)
# 1 = Mini dump
# 2 = Full dump (recommended for crash analysis)
# Create dump directory
mkdir C:\CrashDumps
Per-Application LocalDumps (configure for our test binary):
# Configure for our vulnerable test binary
reg add "HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\vuln_win.exe" /v DumpFolder /t REG_EXPAND_SZ /d "C:\CrashAnalysisLab\dumps" /f
reg add "HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\vuln_win.exe" /v DumpType /t REG_DWORD /d 2 /f
# Or for any application
reg add "HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\target.exe" /v DumpFolder /t REG_EXPAND_SZ /d "C:\CrashDumps\target" /f
reg add "HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps\target.exe" /v DumpType /t REG_DWORD /d 2 /f
Verify WER is Enabled:
# Check WER service status
Get-Service WerSvc
# Check LocalDumps configuration
Get-ItemProperty "HKLM:\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumps"
Sysinternals ProcDump
ProcDump provides more control than WER and catches crashes in real-time:
Basic Crash Capture (using our test binary):
winget install Microsoft.Sysinternals.Suite
# First, ensure you've built the test suite (see "Building a Windows Vulnerable Test Suite" above)
cd C:\CrashAnalysisLab
# Options:
# -ma : Full memory dump (recommended)
# -e : Write dump on unhandled exception
# -x : Launch and monitor (below)
# Launch and monitor for crashes
procdump -ma -e -x dumps vuln_win.exe 1 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
# Monitor already-running process
procdump -ma -e -p <PID>
Advanced ProcDump Usage:
cd C:\CrashAnalysisLab
# Capture on first-chance exceptions (catches more bugs)
procdump -ma -e 1 -x dumps vuln_win.exe 1 AAAA...
# Capture on specific exception codes
procdump -ma -e 1 -f C0000005 -x dumps vuln_win.exe 5 0 # Access violation (NULL deref)
# Capture multiple dumps (for intermittent crashes)
procdump -ma -e -n 5 -x dumps vuln_win.exe 3 # UAF - capture up to 5 dumps
# Monitor service (generic example)
# procdump -ma -e -x C:\Dumps -w ServiceName.exe
ProcDump + Fuzzing Integration:
# Monitor fuzzing target (generic example)
# procdump -ma -e -x C:\FuzzDumps -accepteula target.exe @@
# Batch process dumps from fuzzing run
# for %d in (C:\CrashAnalysisLab\dumps\*.dmp) do cdb -z "%d" -c "!analyze -v; q" >> analysis.txt
Batch Dump Triage with CDB
Analyze multiple dumps automatically:
# Set CDB path (adjust version number as needed)
set CDB="C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe"
# Single dump analysis (use actual dump from ProcDump)
%CDB% -z C:\CrashAnalysisLab\dumps\vuln_win.exe_XXXXXX.dmp
# Or if cdb is in PATH:
cdb -z C:\CrashAnalysisLab\dumps\vuln_win.exe_XXXXXX.dmp -c "!analyze -v; q"
Batch triage script (batch_triage.cmd):
@echo off
# Set path to cdb.exe (adjust if needed)
set CDB="C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe"
for %%f in (C:\CrashAnalysisLab\dumps\*.dmp) do (
echo ======================================== >> triage_report.txt
echo Analyzing: %%f >> triage_report.txt
echo ======================================== >> triage_report.txt
%CDB% -z "%%f" -c ".symfix; .reload; !analyze -v; q" >> triage_report.txt 2>&1
)
echo Done! Results in triage_report.txt
PowerShell Batch Analysis:
# batch_analyze.ps1
# Path to cdb.exe - adjust if your Windows SDK version differs
$cdb = "C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe"
# Verify cdb exists
if (-not (Test-Path $cdb)) {
Write-Error "cdb.exe not found at $cdb. Install Windows SDK Debugging Tools."
exit 1
}
$dumps = Get-ChildItem "C:\CrashAnalysisLab\dumps\*.dmp"
$results = @()
foreach ($dump in $dumps) {
Write-Host "Analyzing $($dump.Name)..."
$output = & $cdb -z $dump.FullName -c "!analyze -v; !exploitable; q" 2>&1 | Out-String
# Extract key info
$exploitable = if ($output -match "Exploitability Classification: (\w+)") { $Matches[1] } else { "Unknown" }
$bugcheck = if ($output -match "EXCEPTION_CODE: \(NTSTATUS\) (0x[0-9a-f]+)") { $Matches[1] } else { "Unknown" }
$results += [PSCustomObject]@{
DumpFile = $dump.Name
Exploitability = $exploitable
ExceptionCode = $bugcheck
}
}
$results | Export-Csv "triage_results.csv" -NoTypeInformation
$results | Format-Table -AutoSize
Symbols and Symbolization (Linux Quick Reference)
Meaningful backtraces (GDB, CASR, ASAN reports) require symbols.
1. Build with debug info (preferred for labs):
cd ~/crash_analysis_lab/src
sudo apt install -y clang-18 clang-18-dbgsym
clang -g -O1 -fno-omit-frame-pointer vulnerable_suite.c -o ../target
2. Install debug symbols for system libraries (real-world targets):
# Ubuntu/Debian: prefer -dbg packages when available (example: libc6-dbg).
# Some packages ship -dbgsym via Ubuntu's ddebs repository.
# Fedora/RHEL:
# sudo dnf debuginfo-install glibc
3. Use debuginfod for "fetch symbols on demand" (when local symbols unavailable):
# Set URL for your distribution (GDB/LLDB will auto-fetch symbols)
# Ubuntu:
export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com"
# Fedora:
# export DEBUGINFOD_URLS="https://debuginfod.fedoraproject.org"
# Generic fallback:
# export DEBUGINFOD_URLS="https://debuginfod.elfutils.org/"
# Note: If you install -dbgsym packages locally (recommended),
# GDB uses those directly without needing debuginfod.
4. Symbolize raw addresses when you only have PCs:
sudo apt install -y elfutils binutils
cd ~/crash_analysis_lab
# IMPORTANT: Full source info requires debug symbols (-g flag at compile time)
# Verify with: file ./target (look for "with debug_info, not stripped")
# Find function addresses in your binary
nm ./target | grep -E " T " | head -5
# Example output:
# 00000000000012b0 T double_free
# 0000000000001624 T _fini
# 00000000000011e0 T heap_overflow
# 0000000000001000 T _init
# 00000000000013c0 T main
# Symbolize using an address from nm output or a crash backtrace
# (PIE binaries show low addresses; add the runtime base for live processes)
addr2line -e ./target -f -C 0x12b0
# With debug info (-g at compile time):
# double_free
# /home/dev/crash_analysis_lab/src/vulnerable_suite.c:37
#
# Without debug info, you only get the function name:
# double_free
# ??:0
# NOTE: eu-addr2line (from elfutils) may show ??:0 even with debug info
# due to DWARF5 compatibility issues. Prefer addr2line (from binutils).
# eu-addr2line -e ./target -f -C 0x12b0 # May not resolve line numbers
# Dynamic lookup example:
addr2line -e ./target -f -C $(nm ./target | grep " T main" | awk '{print $1}')
Symbol Hygiene Best Practices
- Symbols make or break crash analysis.
- Without them, you're staring at hex addresses instead of function names.
- This section provides best practices for both Windows and Linux.
Linux Symbol Management
1. debuginfod (Automatic Symbol Fetching):
debuginfod can automatically fetch debug symbols on-demand from public servers when you don't have them installed locally.
# Install debuginfod client
sudo apt install debuginfod
# Configure debuginfod URL for your distribution
# Ubuntu:
export DEBUGINFOD_URLS="https://debuginfod.ubuntu.com"
# Fedora:
# export DEBUGINFOD_URLS="https://debuginfod.fedoraproject.org"
# Arch:
# export DEBUGINFOD_URLS="https://debuginfod.archlinux.org"
# For GDB, enable automatic fetching
echo "set debuginfod enabled on" >> ~/.gdbinit
# For LLDB
export LLDB_DEBUGINFOD_URLS="https://debuginfod.elfutils.org/"
[!IMPORTANT] debuginfod vs local debug packages: debuginfod queries remote servers for symbols you don't have locally. If you install debug symbol packages (e.g.,
coreutils-dbgsym), the symbols are stored locally at/usr/lib/debug/and GDB uses them directly without needing debuginfod.
Verification: Don't use debuginfod-find to verify your setup—it only queries remote servers. Instead, verify GDB can find symbols:
# Install local debug symbols (recommended for common packages)
sudo apt install coreutils-dbgsym
# Verify GDB finds the symbols
gdb -q -ex "file /usr/bin/ls" -ex "info sources" -ex "quit" 2>&1 | head -5
# Expected output (with pwndbg you'll see its banner first, then):
# Reading symbols from /usr/bin/ls...
# Reading symbols from /usr/lib/debug/.build-id/xx/xxxxx.debug...
# ... followed by source file paths like ls.c, hash.c, etc.
When to use debuginfod: debuginfod is useful when you're analyzing crashes in binaries where you haven't installed the -dbgsym package.
GDB will automatically fetch symbols from the configured server.
2. Installing Debug Symbol Packages:
sudo apt install libc6-dbgsym # Common libraries
sudo apt install libssl3t64-dbgsym # OpenSSL (Ubuntu 24.04+)
sudo apt install zlib1g-dbgsym # zlib
# For -dbgsym packages (automatically generated):
# Enable ddebs repository first:
#echo "deb http://ddebs.ubuntu.com $(lsb_release -cs) main restricted universe multiverse" | \
# sudo tee /etc/apt/sources.list.d/ddebs.list
#sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys F2EDC64DC5AEE1F6B9C621F0C8CAB6595FDFF622
#sudo apt update
#sudo apt install package-dbgsym
3. Symbolizing Addresses with addr2line:
# addr2line (from binutils) is preferred for symbolization
# NOTE: eu-addr2line (from elfutils) may show ??:0 even with debug info
# due to DWARF5 compatibility issues. Prefer addr2line.
# IMPORTANT: ASAN reports are ALREADY SYMBOLIZED!
# If your ASAN output shows:
# #0 0x59cc1877a53e in use_after_free src/vulnerable_suite.c:33
# The file:line info (src/vulnerable_suite.c:33) is already there!
# You do NOT need to run addr2line on ASAN output.
#
# addr2line is only needed for:
# - Raw core dumps without ASAN
# - Stripped binaries with separate debug info
# - Non-ASAN crash logs that only show addresses
#
# If ASAN output shows "??:0" instead of file:line, fix symbolization:
# sudo apt install llvm
# export ASAN_SYMBOLIZER_PATH=$(which llvm-symbolizer)
# # Then re-run the crash
# For non-ASAN crashes, use STATIC addresses from nm (not runtime addresses):
# Runtime addresses like 0x59cc1877a53e include PIE base and won't work!
nm ./vuln_asan | grep "T use_after_free"
# Output: 00000000000014a3 T use_after_free
# Use the static address with addr2line:
addr2line -e ./vuln_asan -f -C 0x14a3
# Output:
# use_after_free
# /home/dev/crash_analysis_lab/src/vulnerable_suite.c:27
# addr2line options:
# -f: Show function names
# -C: Demangle C++ symbols
# -i: Show inlined functions
# Example: Look up a function by name and symbolize it
addr2line -e ./vuln_asan -f -C $(nm ./vuln_asan | grep "T print_usage" | awk '{print $1}')
# To convert runtime address to static (for PIE binaries):
# 1. Get the binary's load base from /proc/<pid>/maps or ASAN output
# 2. Subtract base from runtime address
# Example: If base is 0x59cc18779000 and crash addr is 0x59cc1877a53e:
# Static offset = 0x59cc1877a53e - 0x59cc18779000 = 0x153e
# addr2line -e ./vuln_asan -f -C 0x153e
# With debuginfod (for system binaries without local debug packages):
DEBUGINFOD_URLS="https://debuginfod.ubuntu.com" \
addr2line -e /usr/bin/crashed_binary -f -C 0x12345
4. Verifying Symbol Quality:
# Check if binary has debug symbols
file target
# Look for: "with debug_info, not stripped"
# Check symbol table size
nm target | wc -l
# Check DWARF info presence
readelf --debug-dump=info target | head -50
# Verify specific function is symbolized
nm target | grep stack_overflow
Windows Symbol Management
1. Configuring _NT_SYMBOL_PATH:
# Set symbol path permanently (user environment)
setx _NT_SYMBOL_PATH "srv*C:\Symbols*https://msdl.microsoft.com/download/symbols"
# Or in current session
set _NT_SYMBOL_PATH=srv*C:\Symbols*https://msdl.microsoft.com/download/symbols
# Multiple symbol sources (local + Microsoft + custom server)
set _NT_SYMBOL_PATH=C:\MySymbols;srv*C:\Symbols*https://msdl.microsoft.com/download/symbols;srv*C:\ThirdParty*https://symbols.example.com/
2. WinDbg Symbol Commands:
# open C:\CrashAnalysisLab\vuln_win.exe in windbg
# Quick setup for Microsoft symbols
.symfix C:\Symbols
# Add additional symbol path
.sympath+ C:\CrashAnalysisLab
.reload
# Show current symbol path
.sympath
# Force reload all symbols
.reload /f
# Reload specific module
# .reload /f ntdll.dll
# Enable verbose symbol loading (debugging symbol issues)
!sym noisy
.reload /f
# Disable noisy mode when done
!sym quiet
# Check symbol status for module
lm m ntdll
# Look for: "pdb symbols" vs "export symbols" vs "no symbols"
# Verify specific symbol loads
x ntdll!Rtl* # List all Rtl* functions - only works with symbols
3. Troubleshooting Symbol Issues:
# Symbol loading failed? Check these:
!sym noisy
.reload /f vuln_win.exe
# Common issues:
# 1. Symbol server timeout → Use local cache
# 2. PDB mismatch → Check build matches binary
# 3. Private symbols missing → Request from vendor
# Verify PDB matches binary
!lmi target
# Check: "Checksum" matches between .exe and .pdb
# Force load unverified symbols (use with caution)
.symopt+ 0x40 # SYMOPT_LOAD_ANYTHING
.reload /f
.symopt- 0x40 # Disable after
Cross-Platform Symbol Checklist
- Linux
- debuginfod URL configured
- Debug packages installed for target libraries
- Binary built with -g flag
- eu-addr2line available for batch symbolization
- Windows
-
_NT_SYMBOL_PATHenvironment variable set - Symbol cache directory exists and writable
- Microsoft symbol server accessible
- PDB files match target binaries (same build)
-
- Both Platforms
- Third-party library symbols obtained
- Symbol server accessible (or offline cache populated)
- Test symbolization: verify backtrace shows function names
Analyzing Crash in Pwndbg
# Load core dump
# If you have a local core file (e.g., from core_pattern writing to CWD):
cd ~/crash_analysis_lab
# run it against Test 1 in line 564
gdb ./vuln_no_protect -c /var/crash/core.vuln_no_protect.1184.1766232655
#Reading symbols from ./vuln_no_protect...
#[New LWP 1184]
#[Thread debugging using libthread_db enabled]
#Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
#Core was generated by `./vuln_no_protect 1 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'.
#Program terminated with signal SIGSEGV, Segmentation fault.
##0 0x4141414141414141 in ?? ()
#------- tip of the day (disable with set show-tips off) -------
#GDB and Pwndbg parameters can be shown or set with show <param> and set <param> <value> GDB commands
#LEGEND: STACK | HEAP | CODE | DATA | WX | RODATA
#──────────────────────────────────────────────────────────────────────────────────────────────────────────────────[ REGISTERS / show-flags off / show-compact-regs off ]#───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
# RAX 0xd5
# RBX 0x7ffcc3436ea8 —▸ 0x7ffcc343748f ◂— './vuln_no_protect'
# RCX 0
# RDX 0
# RDI 0x7ffcc3436b20 —▸ 0x7ffcc3436b50 ◂— 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA\nAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAH'
# RSI 0xee2e2a0 ◂— 0x66667542205d2a5b ('[*] Buff')
# R8 0
# R9 0
# R10 0xffffffff
# R11 0x202
# R12 3
# R13 0
# R14 0x403e00 (__do_global_dtors_aux_fini_array_entry) —▸ 0x4011a0 (__do_global_dtors_aux) ◂— endbr64
# R15 0x74e4537e6000 (_rtld_global) —▸ 0x74e4537e72e0 ◂— 0
# RBP 0x4141414141414141 ('AAAAAAAA')
# RSP 0x7ffcc3436d60 ◂— 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
# RIP 0x4141414141414141 ('AAAAAAAA')
#───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────[ DISASM / x86-64 / set emulate on ]#────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#Invalid address 0x4141414141414141
#─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────[ STACK ]#─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#00:0000│ rsp 0x7ffcc3436d60 ◂— 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
#... ↓ 7 skipped
#───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────[ BACKTRACE ]#───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
# ► 0 0x4141414141414141 None
# 1 0x4141414141414141 None
# 2 0x4141414141414141 None
# 3 0x4141414141414141 None
# 4 0x4141414141414141 None
# 5 0x4141414141414141 None
# 6 0x4141414141414141 None
# 7 0x4141414141414141 None
pwndbg> print $_siginfo
#$1 = {
# si_signo = 11,
# si_errno = 0,
# si_code = 1,
# _sifields = {
# _pad = {1094795585, 1094795585, 0 <repeats 26 times>},
# _kill = {
# si_pid = 1094795585,
# si_uid = 1094795585
# },
# _timer = {
# si_tid = 1094795585,
# si_overrun = 1094795585,
# si_sigval = {
# sival_int = 0,
# sival_ptr = 0x0
# }
# },
# ...
#
# Key fields:
# - si_signo = 11 → SIGSEGV
# - si_code = 1 → SEGV_MAPERR (address not mapped to object)
# - si_code = 2 → SEGV_ACCERR (invalid permissions, e.g., NX violation)
# - _sigfault.si_addr → The address that caused the fault
#
# This confirms: Control flow hijack - CPU tried to execute at invalid address 0x4141...
# Check stack for overflow pattern
pwndbg> telescope $rsp 30
#00:0000│ rsp 0x7ffcc3436d60 ◂— 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
#... ↓ 14 skipped
#0f:0078│ 0x7ffcc3436dd8 —▸ 0x74e4537e6000 (_rtld_global) —▸ 0x74e4537e72e0 ◂— 0
#10:0080│ 0x7ffcc3436de0 ◂— 0xdd1f52d83d3c0cb0
#11:0088│ 0x7ffcc3436de8 ◂— 0xcb2e72dba51e0cb0
#12:0090│ 0x7ffcc3436df0 ◂— 0x7ffc00000000
#13:0098│ 0x7ffcc3436df8 ◂— 0
#14:00a0│ 0x7ffcc3436e00 ◂— 0
#15:00a8│ 0x7ffcc3436e08 ◂— 3
#16:00b0│ 0x7ffcc3436e10 —▸ 0x7ffcc3436ea0 ◂— 3
#17:00b8│ 0x7ffcc3436e18 ◂— 0xa20d707b5eb54d00
#18:00c0│ 0x7ffcc3436e20 —▸ 0x7ffcc3436e80 ◂— 0
#19:00c8│ 0x7ffcc3436e28 —▸ 0x74e45342a28b (__libc_start_main+139) ◂— mov r15, qword ptr [rip + 0x1d8cf6]
#1a:00d0│ 0x7ffcc3436e30 —▸ 0x7ffcc3436ec8 —▸ 0x7ffcc343756c ◂— 'SHELL=/bin/bash'
#1b:00d8│ 0x7ffcc3436e38 —▸ 0x403e00 (__do_global_dtors_aux_fini_array_entry) —▸ 0x4011a0 (__do_global_dtors_aux) ◂— endbr64
#1c:00e0│ 0x7ffcc3436e40 —▸ 0x7ffcc3436ec8 —▸ 0x7ffcc343756c ◂— 'SHELL=/bin/bash'
#1d:00e8│ 0x7ffcc3436e48 —▸ 0x401485 (main) ◂— endbr64
WinDbg User Interface Overview
Command Window: Type commands here Registers Window: View CPU register state Disassembly Window: View assembly code at current IP Memory Window: Inspect memory contents Call Stack Window: View function call hierarchy Locals/Watch Window: Inspect variables
Essential Keyboard Shortcuts:
F5: Go (continue execution)F10: Step overF11: Step intoShift+F9: Set/remove breakpointShift+F11: Step outCtrl+Break: Break into debugger
Analyzing Stack Buffer Overflow Crashes
Crash Scenario: Stack buffer overflow in vulnerable application
Load Crash Dump:
# Open crash dump file
File → Open Dump file → select C:\CrashAnalysisLab\dumps\xxx.dmp (or one of the crashes from linux)
# Or from command line
cd C:\CrashAnalysisLab\dumps
windbg -z xxx.dmp
# Verify dump loaded
!analyze -v
#FILE_IN_CAB: vuln_win.exe_260105_151715.dmp
#COMMENT:
#*** procdump -ma -e -x dumps vuln_win.exe 1 #AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA#AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA#AAAAAAAAAA
#*** Unhandled exception: C0000005.ACCESS_VIOLATION
#NTGLOBALFLAG: 70
#APPLICATION_VERIFIER_FLAGS: 0
#CONTEXT: (.ecxr)
#rax=00000000000000d7 rbx=000000000052e6b0 rcx=0000000000000000
#rdx=0000000000010000 rsi=0000000000000000 rdi=00000000005342d0
#rip=000000014000744a rsp=000000000014fed8 rbp=0000000000000000
# r8=7ffffffffffffffc r9=0000000000000000 r10=0000000000000000
#r11=000000000014fcd0 r12=0000000000000000 r13=0000000000000000
#r14=0000000000000000 r15=0000000000000000
#iopl=0 nv up ei pl nz na po nc
#cs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00010204
#vuln_win!stack_overflow+0x3a:
#00000001`4000744a c3 ret
#Resetting default scope
#EXCEPTION_RECORD: (.exr -1)
#ExceptionAddress: 000000014000744a (vuln_win!stack_overflow+0x000000000000003a)
# ExceptionCode: c0000005 (Access violation)
# ExceptionFlags: 00000000
#NumberParameters: 2
# Parameter[0]: 0000000000000000
# Parameter[1]: ffffffffffffffff
#Attempt to read from address ffffffffffffffff
#PROCESS_NAME: vuln_win.exe
#READ_ADDRESS: ffffffffffffffff
#ERROR_CODE: (NTSTATUS) 0xc0000005 - The instruction at 0x%p referenced memory at 0x%p. The memory could not be %s.
#EXCEPTION_CODE_STR: c0000005
#EXCEPTION_PARAMETER1: 0000000000000000
#EXCEPTION_PARAMETER2: ffffffffffffffff
#IP_ON_HEAP: 4141414141414141
#The fault address in not in any loaded module, please check your build's rebase
#log at <releasedir>\bin\build_logs\timebuild\ntrebase.log for module which may
#contain the address if it were loaded.
Initial Analysis Commands:
# Show registers at crash
r
# Display call stack
k
kv # Verbose with frame pointer
kP # With full source paths (if symbols loaded)
kn # With frame numbers
# Show current instruction
u @rip
u @rip L10 # Disassemble 10 instructions
# Examine stack
dps @rsp
dps @rsp L50 # Display 50 pointer-sized values
Analyzing Heap Corruption Crashes
Using the vuln_win.exe test suite from the "Building a Windows Vulnerable Test Suite" section, generate heap-related crashes:
# Generate heap overflow crash (Test 2)
cd C:\CrashAnalysisLab
"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\gflags.exe" /p /enable vuln_win.exe /full
vuln_win.exe 2 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA#AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA#AAAAAAAAAA
# Crash dump saved to C:\CrashDumps\vuln_win.exe.<PID>.dmp
# Generate use-after-free crash (Test 3)
vuln_win.exe 3
# May not crash without PageHeap - see PageHeap lab below
# Generate double-free crash (Test 4)
vuln_win.exe 4
# May not crash without PageHeap - see PageHeap lab below
Load and Analyze Heap Overflow Dump:
# open dump! via GUI: File → Open Crash Dump → select the .dmp file
# Initial analysis
0:000> !analyze -v
# Look for: EXCEPTION_CODE: c0000005 (Access violation)
# Look for: heap_overflow or HeapFree in the stack
Heap Metadata Corruption Pattern (typical output):
# Crash often occurs in HeapFree or subsequent allocation
0:000> k
# ntdll!RtlUserThreadStart$filt$0+0x3f
# ntdll!_C_specific_handler+0x93
# ntdll!RtlpExecuteHandlerForException+0xf
# ntdll!RtlDispatchException+0x437
# ntdll!KiUserExceptionDispatch+0x2e
# vuln_win!__entry_from_strcat_in_strcpy+0x1f
# vuln_win!heap_overflow+0x45
# vuln_win!main+0xdb
# Check heap state
0:000> !heap -s # Summary of all heaps
0:000> !heap -a 0 # Analyze default process heap
# Check what was the destination buffer
0:000> dq @rdx L8
# See how far past the buffer you wrote(WRITE_ADDRESS from !analyze -v)
0:000> !address 0x01fac000
# Examine the vulnerable function
0:000> uf vuln_win!heap_overflow
# Check source if symbols are good
0:000> lsa vuln_win!heap_overflow
Identifying UAF with vuln_win.exe:
# First, enable PageHeap for better UAF detection (run as Administrator)
#cd C:\CrashAnalysisLab
#"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\gflags.exe" /p /enable /full vuln_win.exe
# Now run the UAF test (Test 3)
vuln_win.exe 3
# With PageHeap, this will crash immediately on UAF access
# Load the crash dump
windbg -z C:\CrashDumps\vuln_win.exe.<PID>.dmp
0:000> !analyze -v
# Typical UAF crash pattern
0:000> k
# ChildEBP RetAddr
# 00 ntdll!RtlpLowFragHeapFree+0x42
# 01 vuln_win!use_after_free+0x15
# 02 vuln_win!main+0x89
# Check if address was recently freed (requires PageHeap) -(READ_ADDRESS)
0:000> !heap -p -a 0x01fabfc0
address 0000000001fabfc0 found in
_DPH_HEAP_ROOT @ 1c01000
in free-ed allocation ( DPH_HEAP_BLOCK: VirtAddr VirtSize)
1c0c820: 1fab000 2000
00007ffd1074b2d3 ntdll!RtlDebugFreeHeap+0x0000000000000037
00007ffd106e370c ntdll!RtlpFreeHeap+0x000000000000178c
00007ffd10739300 ntdll!RtlFreeHeap+0x0000000000000620
000000014000753d vuln_win!use_after_free+0x000000000000005d
# Don't forget to disable PageHeap after analysis
#"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\gflags.exe" /p /disable vuln_win.exe
Classification: Use-After-Free - object accessed after being freed.
Common Crash Patterns and Identification
1. Null Pointer Dereference:
0:000> r rax
rax=0000000000000000
0:000> u @rip
mov qword ptr [rax], rcx # Writing to NULL
# Usually not exploitable unless kernel-mode
2. Access Violation (Invalid Address):
0:000> r rax
rax=deadbeefdeadbeef # Invalid address
# Could be:
# - Uninitialized pointer
# - Freed memory
# - Corrupted pointer
3. Stack Cookie Violation:
0:000> k
ntdll!RtlReportCriticalFailure
ntdll!RtlpReportHeapFailure
<Application>!__security_check_cookie
<Application>!function_with_stack_cookie
# Stack overflow detected, but mitigated by /GS
4. Heap Corruption Detected:
0:000> k
ntdll!RtlReportCriticalFailure
ntdll!RtlpHeapHandleError
ntdll!RtlpLogHeapFailure
# Heap allocator detected corruption
# Check nearby allocations for overflow source
Essential WinDbg Commands Reference
Memory Examination:
db <address> # Display bytes
dw <address> # Display words (2 bytes)
dd <address> # Display dwords (4 bytes)
dq <address> # Display qwords (8 bytes)
da <address> # Display ASCII string
du <address> # Display Unicode string
dps <address> # Display pointer-sized values with symbols
Disassembly:
u <address> # Unassemble at address
u <address> L<count> # Unassemble count instructions
ub <address> # Unassemble backward
uf <function> # Unassemble entire function
Breakpoints:
bp <address> # Set breakpoint
bp <module>!<function> # Set breakpoint on function
ba r 1 <address> # Hardware breakpoint on read
ba w 4 <address> # Hardware breakpoint on write (4 bytes)
bl # List breakpoints
bc * # Clear all breakpoints
Execution Control:
g # Go (continue)
p # Step over
t # Step into (trace)
pt # Step to next return
pc # Step to next call
gu # Go up (step out)
Searching Memory:
s -a 0 L?80000000 "string" # Search for ASCII string
s -u 0 L?80000000 "string" # Search for Unicode string
s -b 0 L?80000000 41 41 41 41 # Search for bytes (hex)
Modules and Symbols:
lm # List loaded modules
lm m <module> # Show specific module
x <module>!<symbol> # Examine symbols
dt <structure> # Display type (struct definition)
dt <structure> <addr> # Display structure at address
Heap Commands:
!heap # List all heaps
!heap -s # Heap summary
!heap -a <address> # Analyze heap at address
!heap -p -a <address> # Page heap info for allocation
!heap -x <address> # Search heaps for address
Linux (Pwndbg Equivalents):
| WinDbg Command | Pwndbg Equivalent | Description |
| -------------- | --------------------------------- | --------------------- |
| `db/dd/dq` | `x/b`, `x/w`, `x/g` or `hexdump` | Memory display |
| `dps` | `telescope` | Smart pointer display |
| `u` | `x/i` or `disassemble` | Disassembly |
| `bp` | `break` or `b` | Set breakpoint |
| `ba w` | `watch` or `rwatch` | Hardware watchpoint |
| `g` | `continue` or `c` | Continue execution |
| `p` | `next` or `n` | Step over |
| `t` | `step` or `s` | Step into |
| `s -a` | `search "string"` | Search memory |
| `lm` | `info shared` or `vmmap` | List modules |
| `!heap` | `heap`, `bins`, `arena` | Heap analysis |
| `!analyze -v` | `bt`, `info registers`, `context` | Crash analysis |
Pwndbg Crash Analysis Commands
Essential Pwndbg Commands for Crash Analysis:
# Start GDB with crash dump
gdb ./target -c core.dump
# Or attach to process
gdb -p <pid>
# Load crash core with pwndbg
pwndbg> # Pwndbg automatically shows context on stop
# Display full context (registers, stack, code, backtrace)
pwndbg> context
# Examine registers
pwndbg> regs
pwndbg> info registers
# Backtrace
pwndbg> bt
pwndbg> bt full
# Memory examination (smart pointer display)
pwndbg> telescope $rsp 20
pwndbg> telescope $rsp 50
# Hexdump
pwndbg> hexdump $rax 64
pwndbg> hexdump 0x7fffffff0000 128
# Memory map
pwndbg> vmmap
pwndbg> vmmap libc
# Check binary protections
pwndbg> checksec
# Search memory
pwndbg> search "AAAA"
pwndbg> search -t qword 0x4141414141414141
pwndbg> search -x "deadbeef"
# Heap analysis (critical for heap bugs)
pwndbg> heap
pwndbg> bins
pwndbg> fastbins
pwndbg> tcache
pwndbg> vis_heap_chunks
# Disassembly
pwndbg> disassemble $rip
pwndbg> nearpc 20
# Find ROP gadgets
pwndbg> rop --grep "pop rdi"
# Cyclic pattern (for offset finding)
pwndbg> cyclic 200
pwndbg> cyclic -l 0x61616174
Stack Overflow Offset Mini-Lab
This mini-lab teaches you to find the exact offset needed to control RIP:
# Step 1: Generate a cyclic pattern (de Bruijn sequence)
cd ~/crash_analysis_lab
python3 -m venv .venv
source .venv/bin/activate
pip install pwntools
python3 -c 'from pwn import *; print(cyclic(200).decode())' > pattern.txt
cat pattern.txt
# aaaabaaacaaadaaaeaaafaaagaaahaaaiaaajaaakaaalaaa...
# Step 2: Crash the program with the pattern
./vuln_no_protect 1 "$(cat pattern.txt)"
# Segmentation fault (core dumped)
# Step 3: Analyze the crash in GDB/Pwndbg (use the correct crash file- cwd or proper location)
gdb ./vuln_no_protect -c /var/crash/core.vuln_no_protect.3441.1766236363
pwndbg> info reg rip rbp
# rip 0x6161617461616173 0x6161617461616173
# rbp 0x6161617261616171 0x6161617261616171
# Step 4: Find the offset using the pattern in RIP
pwndbg> cyclic -n 4 -l 0x61616173
# Finding cyclic pattern of 4 bytes: b'saaa' (hex: 0x73616161)
# Found at offset 72
# Or using pwntools directly:
python3 -c "from pwn import *; print(cyclic_find(0x61616173))"
# 72
# Step 5: Verify control - overwrite RIP with a known value
python3 << 'EOF'
from pwn import *
p = process(["./vuln_no_protect", "1", b"A"*72 + p64(0xdeadbeefcafebabe)])
p.wait()
EOF
# In GDB, confirm RIP = 0xdeadbeefcafebabe (use the correct crash file)
gdb ./vuln_no_protect -c /var/crash/core.vuln_no_protect.3552.1766237600
pwndbg> info reg rip
# rip 0xdeadbeefcafebabe <-- We control RIP!
[!NOTE] The offset (72 in this example) is the number of bytes from the start of your input to the saved return address. In Week 5, you'll replace
0xdeadbeefcafebabewith actual exploit targets (ROP gadgets, shellcode addresses, etc.).
Time Travel Debugging (TTD)
What Is TTD?:
- Time Travel Debugging (TTD) is Microsoft's revolutionary debugging technology that records program execution and allows stepping backward in time.
- Unlike traditional debugging where you can only step forward, TTD captures the entire execution trace, enabling you to navigate to any point in the program's history.
Why TTD Matters for Crash Analysis:
- No More "Oops, I stepped too far": Step backward to inspect the exact state before a crash
- Perfect Reproducibility: Recorded traces can be replayed indefinitely with identical behavior
- Non-deterministic Bug Analysis: Catches race conditions, timing issues, and heisenbug patterns
- Offline Analysis: Record on one machine, analyze on another
- Root Cause Discovery: Trace backward from crash to find where corruption originated
Example TTD Workflow with vuln_win.exe:
This example uses the stack overflow crash from our test suite:
# Record the crash (if not already done)
# In WinDbg Preview: File → Start debugging → Launch executable (advanced)
# Executable: C:\CrashAnalysisLab\vuln_win.exe
# Arguments: 1 AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
# Check "Record with Time Travel Debugging"
# Click "Record"
# Program crashes, trace saved automatically
# Note: After recording completes, WinDbg loads the trace at position A:0
# (the beginning), NOT at the crash point. You'll see ntdll!LdrInitializeThunk
# in the call stack - this is normal. Use 'g' or '!tt 100' to reach the crash.
0:000> g
# Initial analysis - we're at the crash point (Access violation at 0x4141414141414141)
0:000> k
# Shows call stack at crash - completely corrupted with 0x41414141`41414141
# 00 ntdll!NtRaiseException+0x14
# 01 ntdll!KiUserExceptionDispatch+0x53
# 02 0x41414141`41414141 <- Attempted to execute here!
# 03 0x41414141`41414141 <- Stack smashed with 'AAAAAAAA'
# ... (more 0x41414141`41414141 entries)
0:000> r
# Note: RIP won't show 0x4141414141414141 directly - it points to the
# exception handler (ntdll!NtRaiseException). The crash happened when
# the CPU tried to execute at the corrupted address. Evidence is in the
# call stack above showing the attempted return to 0x41414141`41414141.
# Jump to beginning of trace
0:000> !tt 0
# Now at program start
# Set breakpoint at vulnerable function
0:000> bp vuln_win!stack_overflow
0:000> g
# Breakpoint hit at start of stack_overflow()
# Examine state before overflow
0:000> r
0:000> dps @rsp L10
# Stack looks normal, return address intact
# Step through the function
0:000> p
0:000> p
0:000> p
0:000> p
# At 'add rsp,68h' - about to return
0:000> p
# Crash! Now at 0x41414141`41414141
# Step 8: Use TTD to examine the crash point
# Note: p- steps back to previous "step boundary" (breakpoints, calls),
# not single instructions. To examine state just before crash, use !tt
# with the position shown before the crash:
0:000> !tt 5C:110
# Now at 'add rsp,68h' just before the corrupted ret
# Step 9: Examine the corrupted stack before ret executes
0:000> dps @rsp L10
# Return address at rsp now contains 0x4141414141414141!
# Compare to earlier - the strcpy overwrote the saved return address
# Alternative: p- goes back to step boundaries, not single instructions
0:000> p-
# Goes back to breakpoint at stack_overflow entry (clean stack state)
# Continue to crash
0:000> g
# Crash occurs when function returns to 0x4141414141414141
# Go backward from crash to find corruption point
0:000> g-
# Stops at previous breakpoint - we can examine state just before crash
TTD Data Model Queries:
TTD integrates with WinDbg's data model, enabling powerful queries:
Memory Access Queries:
# Find all memory writes to the return address location
# First, get RSP at function entry to know where return address is stored
0:000> !tt 0
0:000> bp vuln_win!stack_overflow
0:000> g
0:000> r rsp
# rsp=000000000014fed8 # Return address stored here
# Find all writes to this address range
0:000> dx @$cursession.TTD.Memory(0x14fed8, 0x14fee0, "w")
# Returns many entries - each write to this memory region
# Get details of the LAST write (the one that corrupted return address)
0:000> dx @$cursession.TTD.Memory(0x14fed8, 0x14fee0, "w").Last()
# EventType : 0x1
# TimeStart : 59:1A7 [Time Travel]
# AccessType : Write
# IP : 0x140083412
# Address : 0x14fedd
# Size : 0x8
# Value : 0x4141414141414141 <- The overflow!
# OverwrittenValue : 0xa3d5d3000000 <- Original value destroyed
# Navigate to the exact instruction that corrupted the return address
0:000> dx @$cursession.TTD.Memory(0x14fed8, 0x14fee0, "w").Last().TimeStart.SeekTo()
0:000> u @rip L3
# vuln_win!__entry_from_strcat_in_strcpy+0x1f:
# 00000001`40083412 4889040a mov qword ptr [rdx+rcx],rax <- strcpy writing 'AAAAAAAA'
Call Queries:
# Find all calls to strcpy
0:000> dx @$cursession.TTD.Calls("vuln_win!strcpy")
# [0x0] <- One call found
# Find all strcpy-related functions (includes internal helpers)
0:000> dx @$cursession.TTD.Calls("vuln_win!*strcpy*")
# Returns multiple entries for strcpy and its internal routines
# Find calls to stack_overflow with full details
0:000> dx @$cursession.TTD.Calls("vuln_win!stack_overflow")[0]
# EventType : 0x0
# TimeStart : 55:5AA [Time Travel]
# TimeEnd : Max Position [Time Travel] <- Never returned (crashed)
# Function : vuln_win!stack_overflow
# ReturnAddress : 0x14000789d
# Parameters : [expand to see function arguments]
# View function parameters - shows the malicious input!
0:000> dx @$cursession.TTD.Calls("vuln_win!stack_overflow")[0].Parameters
# input : 0xa3d5d3 : "AAAAAAAAAA..." [Type: char *]
# Navigate to specific call
0:000> dx @$cursession.TTD.Calls("vuln_win!stack_overflow")[0].TimeStart.SeekTo()
# Now at the start of stack_overflow() - can step through
Example: Finding Where Return Address Was Overwritten:
# The key insight: use TTD.Memory() to find who wrote to the return address
# Step 1: Find where return address is stored
0:000> !tt 0
0:000> bp vuln_win!stack_overflow
0:000> g
0:000> r rsp
# rsp=000000000014fed8 # Return address at this location
# Step 2: Query all writes to return address location
0:000> dx @$cursession.TTD.Memory(0x14fed8, 0x14fee0, "w").Last()
# Value: 0x4141414141414141 - confirms overflow wrote here
# IP: 0x140083412 - instruction that did the write
# Step 3: Navigate to the corruption point
0:000> dx @$cursession.TTD.Memory(0x14fed8, 0x14fee0, "w").Last().TimeStart.SeekTo()
# Step 4: Examine the guilty instruction
0:000> u @rip L1
# vuln_win!__entry_from_strcat_in_strcpy+0x1f:
# mov qword ptr [rdx+rcx],rax # strcpy's copy loop overwrote return address!
# Step 5: Check registers to see the overflow in action
0:000> r rax
# rax=4141414141414141 # Source data being copied
Example: Tracing User Input Through vuln_win.exe:
# Goal: Trace how command-line input flows to the crash
# Step 1: Find and navigate to main()
0:000> dx @$cursession.TTD.Calls("vuln_win!main")[0]
# TimeStart : 55:4D6 [Time Travel]
# ReturnValue : 0 [Type: int]
# Parameters : [contains argc, argv]
0:000> dx @$cursession.TTD.Calls("vuln_win!main")[0].TimeStart.SeekTo()
# Step 2: Examine argv (RDX = argv in Windows x64 calling convention)
0:000> dps @rdx L4
# 00000000`00a3d590 00000000`00a3d5b0 # argv[0] - program name
# 00000000`00a3d598 00000000`00a3d5d1 # argv[1] - "1" (test number)
# 00000000`00a3d5a0 00000000`00a3d5d3 # argv[2] - overflow input
# 00000000`00a3d5a8 00000000`00000000 # NULL terminator
# Step 3: View the malicious input
0:000> da poi(@rdx+0x10)
# 00000000`00a3d5d3 "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
# 00000000`00a3d5f3 "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
...
# Step 4: See how input reaches vulnerable function
0:000> dx @$cursession.TTD.Calls("vuln_win!stack_overflow")[0].Parameters
# input : 0xa3d5d3 : "AAAAAAAAAA..." [Type: char *]
# Same address as argv[2] - input passed directly to vulnerable function!
# Step 5: Find all reads from the input buffer to trace data flow
0:000> dx @$cursession.TTD.Memory(0xa3d5d3, 0xa3d5d3+0x100, "r")
# Shows every instruction that read from the malicious input
Practical TTD Crash Analysis: Use-After-Free in vuln_win.exe:
This example demonstrates TTD's power for analyzing UAF bugs:
# Step 1: Enable PageHeap for reliable UAF detection (run as admin)
"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\gflags.exe" /p /enable vuln_win.exe /full
# Step 2: Record UAF crash with TTD
# In WinDbg Preview: File → Start debugging → Launch executable (advanced)
# Executable: C:\CrashAnalysisLab\vuln_win.exe
# Arguments: 3
# Check "Record with Time Travel Debugging"
# Click "Record"
# Step 3: Run to the crash
0:000> g
# (24c4.2178): Access violation - code c0000005 (first/second chance not available)
# Time Travel Position: 379:0
# vuln_win!strnlen+0x84:
# 00000001`4005c464 vpcmpeqb ymm1,ymm1,ymmword ptr [rdx] ds:00000000`02393fc0=48
# Step 4: Analyze the crash
0:000> k
# Call stack shows:
# vuln_win!strnlen+0x84 <- Crash here, reading freed memory
# vuln_win!printf+0x41 <- printf trying to print the string
# vuln_win!use_after_free+0x7a <- Our vulnerable function
# vuln_win!main+0xe6
0:000> !analyze -v
# Key findings:
# READ_ADDRESS: 0000000002393fc0 <- Attempting to read freed memory
# Failure.Bucket: INVALID_POINTER_READ_AVRF_c0000005_vuln_win.exe!strnlen
# Step 5: Find all heap frees and identify the one matching crash address
0:000> dx @$cursession.TTD.Calls("ntdll!RtlFreeHeap")
# [0x0], [0x1], [0x2] <- Three frees in the trace
0:000> dx @$cursession.TTD.Calls("ntdll!RtlFreeHeap")[2].Parameters
# [0x0] : 0x2080000 <- HeapHandle
# [0x1] : 0x0 <- Flags
# [0x2] : 0x2393fc0 <- BaseAddress - MATCHES CRASH ADDRESS!
# Step 6: Get details on the free and navigate to it
0:000> dx @$cursession.TTD.Calls("ntdll!RtlFreeHeap")[2]
# TimeStart : 373:118 [Time Travel]
# ReturnAddress : 0x14000753d
# ReturnValue : 0x1 <- Free succeeded
0:000> dx @$cursession.TTD.Calls("ntdll!RtlFreeHeap")[2].TimeStart.SeekTo()
0:000> k
# 00 ntdll!RtlFreeHeap
# 01 vuln_win!use_after_free+0x5d <- free() called here (line 27)
# 02 vuln_win!main+0xe6
# Step 7: Navigate to use_after_free function entry
0:000> dx @$cursession.TTD.Calls("vuln_win!use_after_free")[0]
# TimeStart : 366:1218 <- Function entry
# TimeEnd : Max Position <- Never returned (crashed)
0:000> dx @$cursession.TTD.Calls("vuln_win!use_after_free")[0].TimeStart.SeekTo()
0:000> k
# Now at the start of use_after_free()
# Step 8: Examine the freed memory at crash point
0:000> !tt 379:0
0:000> !address 0x2393fc0
# "Address could not be mapped" - PageHeap unmapped the page after free!
0:000> dc 0x2393fc0 L10
# 02393fc0 6c6c6548 57202c6f 646c726f c0c00021 Hello, World!...
# 02393fd0 c0c0c0c0 c0c0c0c0 c0c0c0c0 c0c0c0c0 ................
# The string data is still there, but 0xc0 fill pattern shows it's freed!
# Timeline Summary:
# Position 366:1218 - use_after_free() called
# Position 373:118 - free(ptr) called, memory freed
# Position 379:0 - printf(ptr) crashes trying to read freed memory
# Step 9: Don't forget to disable PageHeap after analysis
# "C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\gflags.exe" /p /disable vuln_win.exe
TTD Best Practices:
- Record Minimal Scope: Only record the crashing process to keep traces manageable
- Use Breakpoints Wisely: Set breakpoints before recording to stop at interesting points
- Leverage Data Model: TTD queries are more powerful than manual navigation
- Save Interesting Positions: Use
!positionsto bookmark important execution points - Combine with Memory Analysis: Use TTD to find when corruption occurred, traditional commands to analyze it
- Enable PageHeap for Heap Bugs: TTD + PageHeap gives you allocation/free stacks AND time travel
TTD Limitations:
- Trace Size: Long-running processes create large trace files (GBs)
- Performance: Recording adds ~10-20x slowdown
- Windows Only: No Linux equivalent (use rr instead - see Day 4)
- No Kernel Mode: TTD is user-mode only
- x64 Only: No 32-bit support in modern versions
- WinDbg Preview Required: Classic WinDbg from Windows SDK doesn't include TTD
Black-Box Crash Analysis
[!IMPORTANT] In real-world vulnerability research, especially on Windows, you rarely have source code. The sanitizer-based techniques in Day 2 require recompilation. This section covers black-box techniques for when you can't recompile.
When to Use Black-Box Analysis:
- Analyzing crashes in closed-source software (Microsoft, Adobe, etc.)
- Third-party libraries shipped as binaries
- Malware analysis
- CTF challenges without source
- Production crash dumps from customers
Setup: Creating a Symbol-less Binary for Practice:
# Compile without debug symbols to simulate closed-source binary
cl /O2 /GS- src\vulnerable_suite_win.c /Fe:vuln_win_nosym.exe
# Record crash with TTD
# WinDbg Preview: File → Start debugging → Launch executable (advanced)
# Executable: C:\CrashAnalysisLab\vuln_win_nosym.exe
# Arguments: 1 AAAA...(200+ chars)
# Check "Record with Time Travel Debugging"
Manual Crash State Analysis
Initial Crash Assessment:
# After crash, examine the state
0:000> g
# (2194.20e0): Access violation - code c0000005
# Time Travel Position: 61:0
# 41414141`41414141 ?? ???
# RIP is completely controlled - classic stack overflow!
0:000> r
# rip=4141414141414141 # Controlled!
# rbx=4141414141414141 # Also controlled
# rdi=4141414141414141 # Also controlled
# rsp=000000398bb0fe30
# Call stack is destroyed - all 0x41414141
0:000> k
# 00 0x41414141`41414141
# 01 0x41414141`41414141
# 02 0x41414141`41414141
# ...
When RIP is Invalid - Use TTD to Go Back:
# Can't disassemble at invalid RIP
0:000> u @rip-20 L30
# ^ Memory access error # Expected - RIP points to garbage
# Use TTD to find last valid state (position before crash)
0:000> !tt 60:0
0:000> k
# Now we see the real call stack with module offsets:
# 00 ntdll!NtWriteFile+0x14
# 01 KERNELBASE!WriteFile+0x8d
# 02 vuln_win_nosym+0xf186 <- CRT printf internals
# ...
# 0a vuln_win_nosym+0x146b <- Caller
# 0b vuln_win_nosym+0x1098 <- Vulnerable function (returns to 0x41414141)
Module and Section Analysis:
# List loaded modules
0:000> lm
# start end module name
# 00007ff7`73770000 00007ff7`73798000 vuln_win_nosym (no symbols)
# 00007ff9`179e0000 00007ff9`17c47000 ntdll (pdb symbols)
# Get PE header info - entry point, sections
0:000> !dh vuln_win_nosym
# 8664 machine (X64)
# 1000 base of code
# 16D4 address of entry point <- Entry point offset
# 15200 size of code
# 17000 [ 240] address [size] of Import Address Table
# Check exception directory for function boundaries
0:000> .fnent vuln_win_nosym+0x1010
# BeginAddress = 00000000`00001010
# EndAddress = 00000000`000012a3 <- Function spans 0x1010-0x12a3
# UnwindInfoAddress = 00000000`0002003c
Reverse Engineering the Vulnerable Function:
# Disassemble the function that crashed (identified from call stack)
0:000> u vuln_win_nosym+0x1010 L50
# Look for function prologue
vuln_win_nosym+0x1010:
# mov qword ptr [rsp+10h],rbx # Save rbx
# push rdi # Save rdi
# sub rsp,60h # Allocate 0x60 bytes stack frame
# Find the vulnerable call - look for string copy patterns
# vuln_win_nosym+0x1071:
# lea rcx,[rsp+20h] # Destination: stack buffer at rsp+0x20
# vuln_win_nosym+0x1076:
# call vuln_win_nosym+0x15b90 # <- This is strcpy!
# Function epilogue shows where crash happens
# vuln_win_nosym+0x1098:
# xor eax,eax
# mov rbx,qword ptr [rsp+78h]
# add rsp,60h
# pop rdi
# ret # <- Returns to corrupted address!
Identifying Library Functions Without Symbols:
# Dump Import Address Table to identify API calls
0:000> dps vuln_win_nosym+0x17000 L20
# 00007ff7`73787000 ntdll!RtlAllocateHeap
# 00007ff7`73787008 KERNEL32!HeapFreeStub
# 00007ff7`73787010 KERNEL32!GetProcessHeap
# 00007ff7`737870d8 KERNEL32!GetStdHandleStub
# 00007ff7`737870e0 KERNEL32!WriteFile
# Identify strcpy by its implementation pattern
0:000> u vuln_win_nosym+0x15b90 L15
# Byte-by-byte copy loop with null check = strcpy
# mov r11,rcx # Save dest
# sub rcx,rdx # Calculate offset
# mov al,byte ptr [rdx] # Load source byte
# mov byte ptr [rdx+rcx],al # Store to dest
# test al,al # Check for null
# je <end> # Exit if null
# inc rdx # Next byte
# ...
String Search for Context Clues:
# Search for interesting strings in the binary
0:000> s -a vuln_win_nosym L28000 "overflow"
# 00007ff7`7378741c "overflow detected! alloc_size=%u."
# 00007ff7`737874dd "overflow (need ~100+ chars)."
# View the strings
0:000> da 00007ff7`7378741c
# "overflow detected! alloc_size=%u."
# Search for function names, error messages
0:000> s -a vuln_win_nosym L28000 "Test"
# 00007ff7`73787473 "Test Suite."
0:000> s -a vuln_win_nosym L28000 "free"
# 00007ff7`737873f2 "free done."
Pattern Recognition Without Symbols:
# 1. Stack Overflow Pattern (what we found):
# - RIP contains controlled data (0x41414141...)
# - Stack filled with repeating pattern
# - Function epilogue (add rsp, XX / ret) leads to crash
# 2. Heap Corruption Pattern:
# - Crash in ntdll!Rtl*Heap* functions
# - Invalid forward/backward pointers
# - Corrupted heap metadata
# 3. Use-After-Free Pattern:
# - Crash reading/writing freed memory
# - PageHeap shows 0xc0c0c0c0 fill pattern
# - !address shows "could not be mapped"
# 4. Type Confusion Pattern:
# - Valid object pointer
# - Wrong vtable being used
# - Field access at unexpected offset
WinDbg Scripting for Black-Box Analysis
Automated Crash Classification Script:
// crash_classify.js - Save to C:\CrashAnalysisLab\crash_classify.js
// Run with: .scriptrun C:\CrashAnalysisLab\crash_classify.js
"use strict";
function initializeScript() {
return [new host.apiVersionSupport(1, 7)];
}
function invokeScript() {
var dbgControl = host.namespace.Debugger.Utility.Control;
var regs = host.currentThread.Registers.User;
host.diagnostics.debugLog("=== BLACK-BOX CRASH ANALYSIS ===\n\n");
// Get exception record
host.diagnostics.debugLog("[*] Exception Record:\n");
try {
var exrOutput = dbgControl.ExecuteCommand(".exr -1");
for (var line of exrOutput) {
host.diagnostics.debugLog(" " + line + "\n");
}
} catch (e) {
host.diagnostics.debugLog(" Could not get exception record\n");
}
// Check RIP validity and controlled input patterns
host.diagnostics.debugLog("\n[*] Register Analysis:\n");
var patterns = {
41414141: "ASCII 'AAAA' - controlled input!",
42424242: "ASCII 'BBBB' - controlled input!",
43434343: "ASCII 'CCCC' - controlled input!",
cccccccc: "Uninitialized stack (MSVC debug)",
cdcdcdcd: "Uninitialized heap (MSVC debug)",
c0c0c0c0: "PageHeap freed memory",
feeefeee: "Freed heap memory (MSVC debug)",
baadf00d: "Uninitialized heap (LocalAlloc)",
deadbeef: "Marker value (test/exploit)",
};
var criticalRegs = ["Rip", "Rax", "Rbx", "Rcx", "Rdx", "Rsi", "Rdi", "Rsp"];
var ripControlled = false;
for (var i = 0; i < criticalRegs.length; i++) {
var regName = criticalRegs[i];
try {
var regVal = regs[regName];
var val = regVal.toString(16);
// Pad to 16 chars
while (val.length < 16) {
val = "0" + val;
}
var analysis = "";
for (var pattern in patterns) {
if (val.toLowerCase().indexOf(pattern) !== -1) {
analysis = " <- " + patterns[pattern];
if (regName === "Rip") {
ripControlled = true;
}
break;
}
}
host.diagnostics.debugLog(
" " + regName + ": 0x" + val + analysis + "\n",
);
} catch (e) {
host.diagnostics.debugLog(" " + regName + ": <error reading>\n");
}
}
// Exploitability assessment
host.diagnostics.debugLog("\n[*] Exploitability Assessment:\n");
if (ripControlled) {
host.diagnostics.debugLog(
" [CRITICAL] RIP contains controlled pattern - EXPLOITABLE!\n",
);
host.diagnostics.debugLog(
" Stack overflow with RIP control detected.\n",
);
} else {
// Try to disassemble at RIP
try {
var uOutput = dbgControl.ExecuteCommand("u @rip L1");
var instruction = "";
for (var line of uOutput) {
instruction += line + " ";
}
if (instruction.indexOf("???") !== -1) {
host.diagnostics.debugLog(
" [HIGH] Invalid instruction at RIP - likely controlled\n",
);
} else if (
instruction.indexOf("mov") !== -1 &&
instruction.indexOf("[") !== -1
) {
host.diagnostics.debugLog(
" [HIGH] Crash on memory access - potential read/write primitive\n",
);
} else if (
instruction.indexOf("call") !== -1 &&
instruction.indexOf("[") !== -1
) {
host.diagnostics.debugLog(
" [HIGH] Crash on indirect call - potential code execution\n",
);
} else {
host.diagnostics.debugLog(
" [MEDIUM] Examine crash context for exploitability\n",
);
}
} catch (e) {
host.diagnostics.debugLog(
" [HIGH] Cannot disassemble at RIP - address likely controlled\n",
);
}
}
// Stack analysis for return addresses
host.diagnostics.debugLog("\n[*] Stack Analysis (valid return addresses):\n");
try {
var stackOutput = dbgControl.ExecuteCommand("dps @rsp L20");
var validAddrs = 0;
var controlledAddrs = 0;
for (var line of stackOutput) {
var lineStr = line.toString();
if (
lineStr.indexOf("41414141") !== -1 ||
lineStr.indexOf("42424242") !== -1
) {
controlledAddrs++;
}
if (lineStr.indexOf("!") !== -1) {
validAddrs++;
host.diagnostics.debugLog(" " + lineStr + "\n");
}
}
host.diagnostics.debugLog(
"\n Valid return addresses: " + validAddrs + "\n",
);
host.diagnostics.debugLog(
" Controlled values on stack: " + controlledAddrs + "\n",
);
} catch (e) {
host.diagnostics.debugLog(" <error reading stack>\n");
}
host.diagnostics.debugLog("\n=== END ANALYSIS ===\n");
}
Usage:
# First go to the crash point
0:000> g
# Or for TTD traces, go to crash position
# 0:000> !tt 61:0
# Run the analysis script (uses invokeScript automatically)
0:000> .scriptrun C:\CrashAnalysisLab\crash_classify.js
#JavaScript script successfully loaded from 'C:\CrashAnalysisLab\crash_classify.js'
#=== BLACK-BOX CRASH ANALYSIS ===
#
#[*] Exception Record:
# ExceptionAddress: 4141414141414141
# ExceptionCode: c0000005 (Access violation)
# ExceptionFlags: 00000000
# NumberParameters: 2
# Parameter[0]: 0000000000000000
# Parameter[1]: 0000414141414141
# Attempt to read from address 0000414141414141
#[*] Register Analysis:
# Rip: <error reading>
# Rax: <error reading>
# Rbx: <error reading>
# Rcx: <error reading>
# Rdx: <error reading>
# Rsi: <error reading>
# Rdi: <error reading>
# Rsp: <error reading>
#[*] Exploitability Assessment:
# [HIGH] Cannot disassemble at RIP - address likely controlled
#[*] Stack Analysis (valid return addresses):
# 00000039`8bb0feb8 00007ff9`17a6c510 ntdll!RtlUserThreadStart
# Valid return addresses: 1
# Controlled values on stack: 16
#=== END ANALYSIS ===
Quick Black-Box Analysis Commands:
# List modules (identify target binary without symbols)
0:000> lm
# start end module name
# 00007ff7`73770000 00007ff7`73798000 vuln_win_nosym (no symbols)
# 00007ff9`179e0000 00007ff9`17c47000 ntdll (pdb symbols)
# Get exception details
0:000> .exr -1
# ExceptionAddress: 4141414141414141
# ExceptionCode: c0000005 (Access violation)
# Check all registers
0:000> r
# Short call stack - shows controlled return addresses
0:000> k 5
# 00 0x41414141`41414141
# 01 0x41414141`41414141
# ... (all corrupted)
# Check stack for controlled values - classic overflow pattern
0:000> dps @rsp L30
# 00000039`8bb0fe30 41414141`41414141 <- Controlled!
# 00000039`8bb0fe38 41414141`41414141
# ... (16 entries of 0x41414141)
# 00000039`8bb0feb8 00007ff9`17a6c510 ntdll!RtlUserThreadStart <- Only valid addr
# Search binary for strings (clues about functionality)
0:000> s -a vuln_win_nosym L28000 "overflow"
# 00007ff7`7378741c "overflow detected..."
# 00007ff7`737874dd "overflow (need ~100+ chars)..."
GDB/Pwndbg Black-Box Script:
# blackbox_analyze.py - Source this in GDB: source blackbox_analyze.py
import gdb
class BlackBoxAnalyze(gdb.Command):
"""Analyze crash without symbols"""
def __init__(self):
super(BlackBoxAnalyze, self).__init__("bb-analyze", gdb.COMMAND_USER)
def invoke(self, arg, from_tty):
print("=== BLACK-BOX CRASH ANALYSIS ===\n")
# Get exception record
try:
pc = int(gdb.parse_and_eval("$pc"))
print(f"[*] Crash at: {hex(pc)}")
except:
print("[-] Could not get program counter")
return
# Check instruction at crash
print("\n[*] Crash Instruction:")
try:
gdb.execute(f"x/10i {pc-20}")
except gdb.MemoryError:
print(f" Cannot disassemble at {hex(pc)} - address not mapped")
print(" (PC likely contains attacker-controlled value)")
# Register analysis
print("\n[*] Register Analysis:")
controlled_patterns = [0x41414141, 0x42424242, 0x61616161]
for reg in ["rax", "rbx", "rcx", "rdx", "rsi", "rdi", "r8", "r9"]:
try:
val = int(gdb.parse_and_eval(f"${reg}"))
analysis = ""
# Check for controlled input
for pattern in controlled_patterns:
if (val & 0xffffffff) == pattern or (val >> 32) == pattern:
analysis = " <- CONTROLLED INPUT!"
break
# Check for null
if val == 0:
analysis = " <- NULL"
# Check for heap-like address
if 0x10000 < val < 0x800000000000:
analysis = analysis or " <- possible heap/data"
print(f" {reg}: {hex(val)}{analysis}")
except:
pass
# Stack analysis
print("\n[*] Stack Contents (potential return addresses):")
try:
gdb.execute("x/20gx $rsp")
except:
gdb.execute("x/20wx $esp")
# Exploitability hints
print("\n[*] Exploitability Assessment:")
# Check if PC is controlled
pc_controlled = False
for pattern in controlled_patterns:
if (pc & 0xffffffff) == pattern or (pc >> 32) == pattern:
print(" [CRITICAL] Program counter contains controlled input!")
pc_controlled = True
break
# Check for common exploit marker patterns
marker_patterns = {
0xdeadbeef: "DEADBEEF marker",
0xcafebabe: "CAFEBABE marker",
0xdeadc0de: "DEADC0DE marker",
0xfeedface: "FEEDFACE marker",
}
if not pc_controlled:
pc_lower = pc & 0xffffffff
pc_upper = (pc >> 32) & 0xffffffff
for pattern, name in marker_patterns.items():
if pc_lower == pattern or pc_upper == pattern:
print(f" [CRITICAL] PC contains {name} - likely controlled!")
pc_controlled = True
break
# Check if PC is in non-executable region (indicates control)
if not pc_controlled and pc > 0x7f0000000000:
print(" [WARNING] PC in high memory - possible stack/heap address")
elif not pc_controlled and pc < 0x10000:
print(" [WARNING] PC near NULL - possible partial overwrite")
elif not pc_controlled:
print(" [INFO] PC not directly controlled - check for indirect paths")
BlackBoxAnalyze()
print("Black-box analysis command loaded. Use: bb-analyze")
Lab: Root Cause ≠ Crash Site
The Problem:
- Heap corruption crashes often occur in
malloc()/free()consistency checks - The actual overflow/UAF happened earlier—sometimes thousands of instructions before
- Without understanding this, you'll waste hours staring at allocator internals
Lab Setup: The Delayed Corruption Bug
vulnerable_delayed.c - A bug where corruption and crash are separated:
// ~/crash_analysis_lab/src/vulnerable_delayed.c
// The bug is in process_data(), but the crash is in cleanup()
// This version uses HEAP allocations to demonstrate delayed corruption
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
struct metadata {
size_t size;
char* data;
struct metadata* next;
};
struct metadata* head = NULL;
void add_entry(const char* input) {
struct metadata* entry = malloc(sizeof(struct metadata));
entry->size = strlen(input);
entry->data = malloc(entry->size + 1);
strcpy(entry->data, input);
entry->next = head;
head = entry;
printf("[+] Added entry at %p (data=%p, next=%p)\n", entry, entry->data, entry->next);
}
void process_data(const char* input) {
if (head == NULL) return;
char* buffer = malloc(16);
printf("[*] Allocated 16-byte buffer at %p\n", buffer);
printf("[*] About to copy %zu bytes into 16-byte buffer...\n", strlen(input));
strcpy(buffer, input); // OVERFLOW if input > 16 bytes!
printf("[*] Copy complete (overflow occurred if input > 16 bytes)\n");
// Note: we intentionally don't free buffer here to keep corruption intact
}
void cleanup() {
printf("[*] Starting cleanup - traversing linked list...\n");
struct metadata* current = head;
int i = 0;
while (current) {
printf("[*] Entry %d: current=%p, data=%p, next=%p\n",
i++, current, current->data, current->next);
struct metadata* next = current->next;
free(current->data);
free(current);
current = next;
}
printf("[*] Cleanup complete\n");
}
int main(int argc, char** argv) {
if (argc < 2) {
printf("Usage: %s <input>\n", argv[0]);
printf("Example: %s $(python3 -c \"print('A'*200)\")\n", argv[0]);
return 1;
}
printf("[*] Creating linked list entries...\n");
add_entry("normal entry 1");
add_entry("normal entry 2");
printf("\n[*] Processing user input (%zu bytes)...\n", strlen(argv[1]));
process_data(argv[1]);
printf("\n[*] Adding more entries after overflow...\n");
add_entry("post-overflow entry");
printf("\n[*] Starting cleanup (CRASH likely here, not in process_data!)...\n");
cleanup();
printf("[*] Program completed successfully\n");
return 0;
}
Exercise Part 1: Observe the Problem (Without ASAN)
# Build WITHOUT sanitizers
cd ~/crash_analysis_lab
gcc -g -fno-stack-protector -o delayed_vuln src/vulnerable_delayed.c
source .venv/bin/activate
# Trigger the bug with a LARGE overflow (200+ bytes needed to corrupt heap structures)
./delayed_vuln $(python3 -c "print('A'*200)")
# Analyze with GDB
gdb ./delayed_vuln
(gdb) run $(python3 -c "print('A'*200)")
# CRASH in free() or during list traversal
(gdb) bt
# Backtrace shows crash in add_entry() NOT in process_data() where the bug actually is!
What You'll See:
- Crash occurs in
add_entry()orcleanup()- NOT inprocess_data()! - The error message is
malloc(): corrupted top size- heap corruption detected - Backtrace shows allocator functions (
_int_malloc,malloc_printerr, etc.) - The actual vulnerable
strcpy()inprocess_data()is NOT visible in the backtrace - Signal is SIGABRT (from allocator detecting corruption)
Example backtrace (notice process_data is NOT shown):
#0 __pthread_kill_implementation at ./nptl/pthread_kill.c:44
#1-4 ... (signal handling) ...
#5 __libc_message_impl at ../sysdeps/posix/libc_fatal.c:134
#6 malloc_printerr (str="malloc(): corrupted top size")
#7 _int_malloc at ./malloc/malloc.c:4447
#8 __GI___libc_malloc
#9 add_entry (input="post-overflow entry") at vulnerable_delayed.c:17 <-- CRASH HERE
#10 main at vulnerable_delayed.c:78
The crash is in add_entry() during a malloc() call - the allocator detected that heap metadata was corrupted. But the actual bug is in process_data() which overwrote heap structures with 'A's.
Exercise Part 2: Reproduce with ASAN
# Build WITH ASAN
gcc -g -O0 -fsanitize=address -fno-omit-frame-pointer \
-U_FORTIFY_SOURCE -o delayed_vuln_asan src/vulnerable_delayed.c
# Now ASAN catches the overflow AT THE SOURCE (even with small overflow!)
./delayed_vuln_asan $(python3 -c "print('A'*20)")
ASAN Output (shows TRUE root cause):
[*] Creating linked list entries...
[+] Added entry at 0x503000000040 (data=0x502000000010, next=(nil))
[+] Added entry at 0x503000000070 (data=0x502000000030, next=0x503000000040)
[*] Processing user input (20 bytes)...
[*] Allocated 16-byte buffer at 0x502000000050
[*] About to copy 20 bytes into 16-byte buffer...
=================================================================
==3871==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x502000000060 at pc 0x7174a0ca7923 bp 0x7ffee2ef7d60 sp 0x7ffee2ef7508
WRITE of size 21 at 0x502000000060 thread T0
#0 0x7174a0ca7922 in strcpy ../../../../src/libsanitizer/asan/asan_interceptors.cpp:563
#1 0x6273088c443c in process_data src/vulnerable_delayed.c:36
#2 0x6273088c46e8 in main src/vulnerable_delayed.c:75
#3 0x7174a082a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#4 0x7174a082a28a in __libc_start_main_impl ../csu/libc-start.c:360
#5 0x6273088c41e4 in _start (/home/dev/crash_analysis_lab/delayed_vuln_asan+0x11e4) (BuildId: 5ba4175df72d24b28ce5932020c5be09d8b70064)
0x502000000060 is located 0 bytes after 16-byte region [0x502000000050,0x502000000060)
allocated by thread T0 here:
#0 0x7174a0cfd9c7 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
#1 0x6273088c43e7 in process_data src/vulnerable_delayed.c:31
#2 0x6273088c46e8 in main src/vulnerable_delayed.c:75
#3 0x7174a082a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#4 0x7174a082a28a in __libc_start_main_impl ../csu/libc-start.c:360
#5 0x6273088c41e4 in _start (/home/dev/crash_analysis_lab/delayed_vuln_asan+0x11e4) (BuildId: 5ba4175df72d24b28ce5932020c5be09d8b70064)
SUMMARY: AddressSanitizer: heap-buffer-overflow ../../../../src/libsanitizer/asan/asan_interceptors.cpp:563 in strcpy
Exercise Part 3: Find Root Cause with Watchpoints/rr
When you can't use ASAN (closed-source binary, can't recompile):
# Method A: Hardware watchpoints in GDB
gdb ./delayed_vuln
# Break in process_data first to skip add_entry's strcpy calls
(gdb) break process_data
(gdb) run $(python3 -c "print('A'*200)")
# Hits breakpoint in process_data
# Now set breakpoint on strcpy and continue - next hit is the vulnerable one
(gdb) break strcpy
(gdb) continue
# Stops at strcpy inside process_data
# Get buffer address from RDI register (destination argument)
(gdb) print/x $rdi
# Output: $1 = 0x555555559730
# Set watchpoint on top chunk size field (buffer + 0x18 for 16-byte alloc)
(gdb) set $buf = $rdi
(gdb) watch *(long*)($buf + 0x18)
(gdb) continue
# Watchpoint triggers during strcpy - showing EXACT instruction causing corruption
# Output:
# Hardware watchpoint 3: *(long*)($buf + 0x18)
# Old value = 133313
# New value = 133185
# __strcpy_sse2 () at ../sysdeps/x86_64/multiarch/strcpy-sse2.S:110
#
# Backtrace shows the corruption path:
# __strcpy_sse2+163 <- overflow happens HERE
# process_data+123 <- vulnerable function
# main+201
Exercise Part 4: Document the Difference
Create a comparison table of what you observed:
| Aspect | Without ASAN | With ASAN |
| -------------------- | ------------------------ | ----------------------- |
| Crash Location | add_entry() or cleanup() | process_data():strcpy() |
| Signal | SIGABRT (allocator) | SIGABRT (ASAN) |
| Backtrace shows bug? | NO | YES |
| Root cause visible? | NO | YES |
| Time to identify | 30+ minutes | 5 seconds |
Lab Deliverables
- Screenshot/log of non-ASAN crash (showing misleading backtrace)
- Screenshot/log of ASAN crash (showing true root cause)
- GDB transcript showing watchpoint catching the overflow
- Written explanation (2-3 sentences) of why the crash and bug are in different locations
Success Criteria:
- Understand that crash site ≠ bug site for heap corruption
- Can use ASAN to find true root cause
- Can use watchpoints/rr to trace corruption without ASAN
- Can explain the delayed corruption phenomenon
Identifying Vulnerability Types Without Source
1. Recognizing Heap UAF in Closed-Source:
# Step 1: Check if crash is on object method call
0:000> u @rip
# Look for: call qword ptr [rax+XX] <- vtable dispatch
# Step 2: Check the object pointer
0:000> dq @rcx L8 # Dump supposed object
# If first qword looks like valid vtable, but other fields look wrong → UAF
# Step 3: Check heap state (requires PageHeap enabled)
0:000> !heap -p -a @rcx
# Look for "free" status or "freed and reallocated"
# Step 4: Back-trace with TTD (if available)
0:000> r rcx
# rcx=000001efe2393fc0 <- The freed pointer
# Find all heap frees and check parameters for matching address
0:000> dx @$cursession.TTD.Calls("ntdll!RtlFreeHeap")
# Examine each one:
0:000> dx @$cursession.TTD.Calls("ntdll!RtlFreeHeap")[0].Parameters
0:000> dx @$cursession.TTD.Calls("ntdll!RtlFreeHeap")[1].Parameters
# Look for Parameters[2] matching your crash address
# Navigate to the free that matches
0:000> dx @$cursession.TTD.Calls("ntdll!RtlFreeHeap")[2].TimeStart.SeekTo()
# Now at the free - examine call stack
0:000> k
2. Recognizing Type Confusion:
# Pattern: Valid object, wrong type being assumed
# - Object pointer is valid
# - Vtable is valid but for WRONG class
# - Crash when accessing field at wrong offset
# Check: Compare vtable to known vtables
0:000> dps poi(@rcx) L10 # Dump vtable methods
# Cross-reference with known class vtables in the binary
# Use TTD to find where wrong type was assumed
0:000> !tt 0
0:000> ba r 8 @rcx # Break on reads of this object (can be noisy)
0:000> g # Observe the code that reads/uses the object
3. Recognizing Logic Bugs:
# Logic bugs often don't crash in memory functions
# Instead: crashes in application-specific code
# Signs of logic bug:
# - Crash NOT in heap/string functions
# - Values are valid but unexpected
# - Race condition patterns (varies between runs)
# - File/network state inconsistency
# Example: Race condition in file handling
0:000> k
# Call stack shows file operation, but state is inconsistent
# Use TTD to check for interleaved operations
0:000> !tt 0
0:000> bp kernelbase!CreateFileW
0:000> bp kernelbase!CloseHandle
0:000> g
# Watch for close-then-use patterns
Practical Exercise
[!NOTE] You should have already built the vulnerable test suite earlier in this section. If not, scroll up to "Building a Vulnerable Test Suite (Do This First!)" and complete that setup before continuing.
Alternative: Pre-built Vulnerable Targets
If you want additional crash samples beyond the test suite:
# CASR includes test cases with sample crash reports
git clone --depth 1 https://github.com/ispras/casr.git ~/casr-tests
ls ~/casr-tests/casr/tests/casr_tests/casrep/
# Fuzzing101 has vulnerable targets with known bugs
git clone --depth 1 https://github.com/antonio-morales/Fuzzing101.git ~/Fuzzing101
# Follow Exercise1 to build xpdf with bugs
Tasks
Task: Analyze 5 different crash types and classify each
Using the test suite you built above (or crashes from your Week 2 fuzzing), analyze each crash type.
Crash Types to Generate and Analyze (Linux):
stack_overflow- Run:./vuln_no_protect 1 $(python3 -c "print('A'*200)")heap_overflow- Run:./vuln_asan 2 $(python3 -c "print('A'*100)")use_after_free- Run:./vuln_asan 3double_free- Run:./vuln_asan 4null_deref- Run:./vuln_no_protect 5 0
Crash Types to Generate and Analyze (Windows with TTD):
stack_overflow- Record with TTD:vuln_win.exe 1 AAAA...(200+ chars)heap_overflow- Enable PageHeap first, then:vuln_win.exe 2 AAAA...(100+ chars)use_after_free- Enable PageHeap first, then:vuln_win.exe 3double_free- Run:vuln_win.exe 4null_deref- Run:vuln_win.exe 5 0
For Each Crash (WinDbg):
-
Load and Get Overview:
# For dump files: windbg -z <dump_file> !analyze -v # For TTD traces: # File → Open trace file → Select .run file 0:000> g # Run to crash 0:000> !analyze -v -
Examine Crash State:
0:000> k # Call stack 0:000> r # Registers 0:000> u @rip # Current instruction 0:000> dps @rsp L20 # Stack contents -
For TTD Traces - Find Root Cause:
0:000> !tt 0 # Go to start 0:000> dx @$cursession.TTD.Calls("ntdll!RtlFreeHeap") # Find heap frees 0:000> dx @$cursession.TTD.Memory(<addr>, <addr>+8, "w") # Find writes
For Each Crash (GDB/Linux):
-
Load and Get Overview:
gdb ./vuln_no_protect core bt # Backtrace info registers # Registers -
Examine Crash State:
x/10i $rip # Disassemble at crash x/20gx $rsp # Stack contents
Classify Bug Type:
- What register/memory caused crash?
- What operation was attempted?
- What's the root cause?
Assess Exploitability:
- Can attacker control crash address?
- Is value being written controllable?
- Are there mitigations active?
Document Findings:
## Crash: stack_overflow
- **Type**: Stack Buffer Overflow
- **Location**: vulnerable_function+0x42
- **Cause**: strcpy without bounds checking
- **Controlled**: Return address, saved registers
- **Exploitability**: High (if DEP/ASLR bypassed)
Success Criteria:
- All 5 dumps analyzed
- Correct crash type identified for each
- Root cause understood
- Exploitability assessment provided
- Findings documented clearly
Lab: PageHeap/AppVerifier for Windows
[!IMPORTANT] PageHeap is the Windows equivalent of ASAN for heap bugs—it surrounds allocations with guard pages and tracks allocation/free stacks.
What PageHeap Does
PageHeap (part of Application Verifier / gflags) modifies the Windows heap to:
- Place each allocation on its own page boundary
- Add inaccessible guard pages after allocations
- Keep freed memory inaccessible (catches UAF immediately)
- Record allocation and free stack traces
Normal Heap: PageHeap (Full):
┌──────────────────────┐ ┌──────────────────────┐
│ alloc1 │ alloc2 │ ...│ │ alloc1 │ GUARD PAGE │
└──────────────────────┘ ├──────────────────────┤
│ alloc2 │ GUARD PAGE │
Overflow goes undetected └──────────────────────┘
Overflow hits guard → CRASH
Lab Setup
[!TIP] You can also use
vuln_win.exefrom the "Building a Windows Vulnerable Test Suite" section earlier in Day 1. The dedicatedheap_vuln.cbelow is simpler and focused specifically on heap bugs for this lab.
1. Create Vulnerable Windows Program:
// c:\CrashAnalysisLab/src/heap_vuln.c - Compile with: cl /Zi src/heap_vuln.c
#include <windows.h>
#include <stdio.h>
#include <string.h>
void heap_overflow(char* input) {
char* buf = (char*)HeapAlloc(GetProcessHeap(), 0, 32);
printf("[*] Allocated 32 bytes at %p\n", buf);
// OVERFLOW: strcpy has no bounds check
strcpy(buf, input);
printf("[*] Copied: %s\n", buf);
HeapFree(GetProcessHeap(), 0, buf);
}
void use_after_free() {
char* buf = (char*)HeapAlloc(GetProcessHeap(), 0, 64);
printf("[*] Allocated at %p\n", buf);
strcpy(buf, "Hello World");
HeapFree(GetProcessHeap(), 0, buf);
printf("[*] Freed\n");
// UAF: Access after free
printf("[*] UAF read: %s\n", buf);
buf[0] = 'X'; // UAF write
}
int main(int argc, char** argv) {
if (argc < 2) {
printf("Usage: %s <1|2> [input]\n", argv[0]);
printf(" 1 <input> - Heap overflow\n");
printf(" 2 - Use-after-free\n");
return 1;
}
switch(atoi(argv[1])) {
case 1:
if (argc < 3) return 1;
heap_overflow(argv[2]);
break;
case 2:
use_after_free();
break;
}
printf("[*] Done\n");
return 0;
}
2. Compile the Test Program:
cd c:\CrashAnalysisLab
# Open "x64 Native Tools Command Prompt for VS 2022"
cl /Zi /Od src/heap_vuln.c /link /DEBUG
Step-by-Step PageHeap Lab
Step 1: Run WITHOUT PageHeap (observe the problem):
# Without PageHeap, many heap bugs don't crash immediately
heap_vuln.exe 1 "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
# May print "Done" without crashing - corruption went undetected!
heap_vuln.exe 2
# May print stale data without crashing - UAF went undetected!
Step 2: Enable PageHeap:
# Enable FULL page heap for target.exe
# Run as Administrator
"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\gflags.exe" /p /enable heap_vuln.exe /full
# Verify it's enabled
"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\gflags.exe" /p
# Should show: heap_vuln.exe: page heap enabled
# Alternative: Using Application Verifier GUI
#appverif.exe
# Add heap_vuln.exe → Check "Heaps" under "Basics"
Step 3: Reproduce with PageHeap (crashes immediately):
# Now the overflow crashes immediately
heap_vuln.exe 1 "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
# UAF also crashes immediately
heap_vuln.exe 2
Step 4: Analyze in WinDbg:
# Start WinDbg with the target executable
# File -> Open Executable -> heap_vuln.exe
# Then set arguments in the dialog:
# 1 "AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
# In WinDbg, continue past loader breakpoints until crash:
0:000> g
# You may hit multiple breakpoints - keep pressing 'g' until you see:
# (xxxx.xxx): Access violation - code c0000005 (first chance)
# Example crash output:
# heap_vuln!__entry_from_strcat_in_strcpy+0x1f:
# 00007ff7`6a3e08b2 4889040a mov qword ptr [rdx+rcx],rax ds:0000015b`52cf6ffc=???
# View the call stack - shows exact crash location:
0:000> kb
# RetAddr : Call Site
# 00 00007ff7`6a367295 : heap_vuln!__entry_from_strcat_in_strcpy+0x1f
# 01 00007ff7`6a367411 : heap_vuln!heap_overflow+0x45 [heap_vuln.c @ 11] <-- strcpy line!
# 02 00007ff7`6a3677c8 : heap_vuln!main+0xa1 [heap_vuln.c @ 40]
# 03 (Inline Function) : heap_vuln!invoke_main+0x22
# 04 00007ffd`23dbe8d7 : heap_vuln!__scrt_common_main_seh+0x10c
# 05 00007ffd`24f6c53c : KERNEL32!BaseThreadInitThunk+0x17
# Check registers - reveals the overflow data:
0:000> r
# rax=4141414141414141 <-- "AAAAAAAA" being written (0x41 = 'A')
# rdx=0000015b52c86fe0 <-- Buffer base address
# rcx=000000000007001c <-- Offset into buffer (way past 32 bytes!)
# rdx+rcx = target address in guard page
# Get detailed heap information for the buffer address:
# Use the address from r11 (guard page area) or the buffer start
0:000> !heap -p -a 0000015b52cf6fe0
address 0000015b52cf6fe0 found in
_DPH_HEAP_ROOT @ 15b529e1000
in busy allocation (DPH_HEAP_BLOCK: UserAddr UserSize - VirtAddr VirtSize)
15b529ea618: 15b52cf6fe0 20 - 15b52cf6000 2000
# UserSize: 0x20 = 32 bytes (your HeapAlloc request)
# VirtSize: 0x2000 = 8KB page allocated by PageHeap for protection
# ALLOCATION STACK TRACE (shows where memory was allocated):
00007ffd24f30727 ntdll!RtlDebugAllocateHeap+0x387
00007ffd24f32f3a ntdll!RtlpAllocateHeap+0x246a
00007ffd24efd0d1 ntdll!RtlpAllocateNTHeapInternal+0x3d1
00007ffd24efcca4 ntdll!RtlAllocateHeap+0xad4
00007ff76a367270 heap_vuln!heap_overflow+0x20 [heap_vuln.c @ 6] <-- HeapAlloc call
00007ff76a367411 heap_vuln!main+0xa1 [heap_vuln.c @ 40]
00007ff76a3677c8 heap_vuln!__scrt_common_main_seh+0x10c
# Full automated analysis:
0:000> !analyze -v
# Shows: HEAP_CORRUPTION, faulting module, and root cause analysis
For UAF (Use-After-Free) Analysis:
# Run with UAF test case:
windbg heap_vuln.exe 2
0:000> g
# Keep pressing 'g' past loader breakpoints until crash:
# (xxxx.xxxx): Access violation - code c0000005 (first chance)
# heap_vuln!strnlen+0x84: <-- Crash in printf trying to read freed string
# View call stack - shows UAF access path:
0:000> kb
0f heap_vuln!use_after_free+0x75 [heap_vuln.c @ 26] <-- printf("%s", ptr) after free
10 heap_vuln!main+0xa9 [heap_vuln.c @ 43]
# Get heap info - NOTE: use !ext.heap on newer WinDbg versions
0:000> !ext.heap -p -a 0000022bc3fa6fc0
address 0000022bc3fa6fc0 found in
_DPH_HEAP_ROOT @ 22bc3c91000
in free-ed allocation <-- PageHeap knows this was FREED!
# FREE STACK TRACE (shows where memory was freed):
00007ffd24f6b2d3 ntdll!RtlDebugFreeHeap+0x37
00007ffd24f0370c ntdll!RtlpFreeHeap+0x178c
00007ffd24f59300 ntdll!RtlFreeHeap+0x620
00007ff76a367328 heap_vuln!use_after_free+0x58 [heap_vuln.c @ 22] <-- HeapFree call!
00007ff76a367419 heap_vuln!main+0xa9 [heap_vuln.c @ 43]
# This tells you:
# 1. Memory WAS freed (line 22: HeapFree)
# 2. Then accessed (line 26: printf with freed ptr)
# 3. PageHeap protected the freed memory, causing immediate crash
Step 5: Check Mitigations with PowerShell:
# Check mitigations for a running process
# First, run heap_vuln.exe under WinDbg (paused), then in another terminal:
Get-Process heap_vuln | Get-ProcessMitigation
# Example output for heap_vuln.exe:
ProcessName: heap_vuln
Source : Running Process
Id : 10468
DEP:
Enable : ON # Can't execute code on stack/heap
EmulateAtlThunks : ON
ASLR:
BottomUp : ON # Address randomization active
HighEntropy : ON # 64-bit high entropy ASLR
ForceRelocateImages : OFF
CFG:
Enable : OFF # Not compiled with /guard:cf
SEHOP:
Enable : ON # SEH overwrite protection
# Key mitigations for exploitability assessment:
# - DEP ON = need ROP chain, can't just jump to shellcode
# - ASLR ON = need info leak to find gadgets/addresses
# - CFG OFF = indirect calls not protected (easier to exploit)
# - SEHOP ON = can't easily overwrite SEH handlers
# Check system-wide defaults:
Get-ProcessMitigation -System
# Check PE header mitigations in WinDbg:
0:000> !dh -f heap_vuln
# 8160 DLL characteristics
# High Entropy Virtual Addresses
# Dynamic base <-- ASLR
# NX compatible <-- DEP
Step 6: Disable PageHeap After Analysis:
# IMPORTANT: Always disable PageHeap after debugging!
# PageHeap has significant performance/memory overhead
# If you used gflags:
"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\gflags.exe" /p /disable heap_vuln.exe
# Verify it's disabled:
"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\gflags.exe" /p
# Should NOT show heap_vuln.exe in the list
# If you used Application Verifier (appverif.exe):
# 1. Open appverif.exe
# 2. Select heap_vuln.exe from the list
# 3. Uncheck all tests or click "Delete Application"
# 4. Click Save
# Alternative: Clear all gflags for the executable
"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\gflags.exe" /i heap_vuln.exe -ust -hpa
Lab Deliverables
- Screenshot: gflags showing PageHeap enabled
- WinDbg log:
!heap -p -aoutput showing allocation stack - Comparison: Document behavior with/without PageHeap
- PowerShell output:
Get-ProcessMitigationresults
Key Takeaways
- WinDbg is essential: Primary tool for Windows crash analysis
- Symbols are crucial: Without symbols, analysis is much harder
- Crash patterns are recognizable: Common patterns indicate specific bug types
- Context matters: Same crash can have different exploitability based on mitigations
- Practice builds speed: Analyzing many crashes makes patterns obvious
- Pattern recognition is essential: Learn to recognize crash signatures without symbols
- Registers tell the story: Systematic register analysis reveals control
- Scripts accelerate triage: Automate repetitive analysis tasks
- TTD is powerful: Time-travel debugging helps even without symbols
- Document methodology: Structured reports help track analysis
- PageHeap is essential: Windows heap bug detection requires it
Discussion Questions
- How do stack cookies change the exploitability of stack overflows?
- What information can be gained from a crash even if it's not directly exploitable?
- How does Page Heap help identify heap corruption root causes?
- How does Time Travel Debugging (TTD) change your approach to finding where memory corruption originated, compared to traditional forward-only debugging?
Day 2: AddressSanitizer and Memory Error Classification
- Goal: Use AddressSanitizer (ASAN) to detect and classify memory errors with detailed diagnostics.
- Activities:
- Reading:
- Online Resources:
- Tool Setup:
- Clang compiler with ASAN support
- Visual Studio 2022+ (for Windows ASAN)
- Exercise:
- Compile test programs with ASAN
- Trigger and classify 10 different memory error types
Understanding AddressSanitizer
[!TIP] Ubuntu Quick Setup - Copy this environment block before running ASAN-compiled binaries:
# Recommended ASAN/UBSAN environment for Ubuntu export ASAN_SYMBOLIZER_PATH=$(command -v llvm-symbolizer) export ASAN_OPTIONS="abort_on_error=1:symbolize=1:detect_leaks=1:disable_coredump=0" export UBSAN_OPTIONS="print_stacktrace=1:halt_on_error=1"Key options explained:
abort_on_error=1: Abort on first error (generates signal for debugging)disable_coredump=0: Allow core dump generation even with ASANdetect_leaks=1: Enable LeakSanitizer (LSan)symbolize=1: Show source file/line in reportsNote on ASAN + core dumps: ASAN often calls
abort()on errors, which generates SIGABRT (-6), not SIGSEGV (-11). Setdisable_coredump=0if you need core dumps for post-mortem analysis.
What is ASAN?:
- Compiler instrumentation tool for detecting memory errors
- Inserts runtime checks around memory operations
- Uses "shadow memory" to track allocation state
- Detects: buffer overflows, UAF, double-free, memory leaks, and more
How It Works:
- Shadow Memory: 1 shadow byte tracks 8 bytes of application memory
- Red Zones: Poisoned memory surrounding allocations
- Quarantine: Freed memory held before reuse to catch UAF
- Stack Instrumentation: Red zones around stack variables
Installing and Using ASAN (Linux)
With Clang:
# Install clang
sudo apt install clang llvm
# Navigate to lab directory (created in Day 1)
cd ~/crash_analysis_lab
# Compile with ASAN (using vulnerable_suite.c from Day 1)
clang -g -O1 -fsanitize=address -fno-omit-frame-pointer src/vulnerable_suite.c -o vuln_asan
# Enable symbolization
export ASAN_SYMBOLIZER_PATH=$(command -v llvm-symbolizer)
export ASAN_OPTIONS="abort_on_error=1:symbolize=1:detect_leaks=1:disable_coredump=0"
export UBSAN_OPTIONS="print_stacktrace=1:halt_on_error=1"
# Run and observe detailed error report (test case 3 = UAF)
./vuln_asan 3
With GCC:
# GCC also supports ASAN
cd ~/crash_analysis_lab
gcc -g -O1 -fsanitize=address -fno-omit-frame-pointer -D_FORTIFY_SOURCE=0 src/vulnerable_suite.c -o vuln_asan1
# Run with same environment variables (test case 1 = stack overflow)
./vuln_asan1 1 $(python3 -c "print('A'*200)")
ASAN Error Types and Reports
1. Heap Buffer Overflow:
Vulnerable Code:
// ~/crash_analysis_lab/src/heap.c
#include <stdlib.h>
#include <string.h>
int main() {
char *buf = malloc(10);
strcpy(buf, "This is too long!"); // Overflow!
free(buf);
return 0;
}
cd ~/crash_analysis_lab
gcc -g -O0 -fsanitize=address -fno-omit-frame-pointer -D_FORTIFY_SOURCE=0 src/heap.c -o heap
./heap
ASAN Report:
==1330==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x50200000001a at pc 0x773226afb303 bp 0x7ffdf29dd780 sp 0x7ffdf29dcf28
WRITE of size 18 at 0x50200000001a thread T0
#0 0x773226afb302 in memcpy ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors_memintrinsics.inc:115
#1 0x5ac25166523d in main src/heap.c:6
#2 0x77322662a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#3 0x77322662a28a in __libc_start_main_impl ../csu/libc-start.c:360
#4 0x5ac251665144 in _start (/home/dev/crash_analysis_lab/heap+0x1144) (BuildId: 060cf895aa12e860df15a930f5880bac28c424b2)
0x50200000001a is located 0 bytes after 10-byte region [0x502000000010,0x50200000001a)
allocated by thread T0 here:
#1 0x5ac25166521e in main src/heap.c:5
#2 0x77322662a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#3 0x77322662a28a in __libc_start_main_impl ../csu/libc-start.c:360
#4 0x5ac251665144 in _start (/home/dev/crash_analysis_lab/heap+0x1144) (BuildId: 060cf895aa12e860df15a930f5880bac28c424b2)
SUMMARY: AddressSanitizer: heap-buffer-overflow ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors_memintrinsics.inc:115 in memcpy
Shadow bytes around the buggy address:
0x501ffffffd80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x501ffffffe00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x501ffffffe80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x501fffffff00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x501fffffff80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x502000000000: fa fa 00[02]fa fa fa fa fa fa fa fa fa fa fa fa
0x502000000080: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x502000000100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x502000000180: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x502000000200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x502000000280: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
Shadow Memory Interpretation:
fa= heap redzone (poison bytes around allocations)00= 8 fully addressable bytes02= 2 more addressable bytes (totaling the 10-byte allocation)[02]bracket shows exactly where the overflow was detected
Analysis:
- Error: heap-buffer-overflow
- Operation: WRITE of size 18 (string "This is too long!" + null terminator)
- Location: heap.c:6 (strcpy transformed to memcpy)
- Allocation: 10-byte buffer allocated at line 5
- Overflow: 8 bytes past end of allocation (detected at byte 10)
2. Stack Buffer Overflow:
Vulnerable Code:
// ~/crash_analysis_lab/src/stack.c
#include <string.h>
void vulnerable_function(char *input) {
char buffer[16];
strcpy(buffer, input); // No bounds check!
}
int main() {
vulnerable_function("AAAAAAAAAAAAAAAAAAAAAAAAAAAA");
return 0;
}
gcc -g -O0 -fsanitize=address -fno-omit-frame-pointer -D_FORTIFY_SOURCE=0 src/stack.c -o stack
./stack
ASAN Report:
==1349==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x732093f00030 at pc 0x7320964a7923 bp 0x7ffd05f3a950 sp 0x7ffd05f3a0f8
WRITE of size 29 at 0x732093f00030 thread T0
#0 0x7320964a7922 in strcpy ../../../../src/libsanitizer/asan/asan_interceptors.cpp:563
#1 0x5a7f7e0e52aa in vulnerable_function src/stack.c:5
#2 0x5a7f7e0e5314 in main src/stack.c:9
#3 0x73209602a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#4 0x73209602a28a in __libc_start_main_impl ../csu/libc-start.c:360
#5 0x5a7f7e0e5144 in _start (/home/dev/crash_analysis_lab/stack+0x1144) (BuildId: 03503cc1bce726df73220dfdcbbb15bc88eceb61)
Address 0x732093f00030 is located in stack of thread T0 at offset 48 in frame
#0 0x5a7f7e0e5218 in vulnerable_function src/stack.c:3
This frame has 1 object(s):
[32, 48) 'buffer' (line 4) <== Memory access at offset 48 overflows this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
(longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-buffer-overflow ../../../../src/libsanitizer/asan/asan_interceptors.cpp:563 in strcpy
Shadow bytes around the buggy address:
0x732093effd80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x732093effe00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x732093effe80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x732093efff00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x732093efff80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x732093f00000: f1 f1 f1 f1 00 00[f3]f3 00 00 00 00 00 00 00 00
Analysis:
- Error: stack-buffer-overflow
- Operation: WRITE of size 29 (28 'A' characters + null terminator)
- Location: stack.c:5 (strcpy in vulnerable_function)
- Buffer: 16-byte buffer 'buffer' at stack frame offset [32, 48)
- Overflow: 13 bytes past end of allocation (access at offset 48, buffer ends at 48)
- Shadow byte
f1: Stack left redzone - Shadow byte
f3: Stack right redzone (where overflow was detected)
3. Use-After-Free:
Vulnerable Code:
// ~/crash_analysis_lab/src/uaf.c
#include <stdlib.h>
int main() {
int *ptr = malloc(sizeof(int));
*ptr = 42;
free(ptr);
*ptr = 43; // UAF!
return 0;
}
gcc -g -O0 -fsanitize=address -fno-omit-frame-pointer -D_FORTIFY_SOURCE=0 src/uaf.c -o uaf
./uaf
ASAN Report:
==1371==ERROR: AddressSanitizer: heap-use-after-free on address 0x502000000010 at pc 0x59df62e93267 bp 0x7ffe0df212f0 sp 0x7ffe0df212e0
WRITE of size 4 at 0x502000000010 thread T0
#0 0x59df62e93266 in main src/uaf.c:8
#1 0x7c6892c2a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#2 0x7c6892c2a28a in __libc_start_main_impl ../csu/libc-start.c:360
#3 0x59df62e93104 in _start (/home/dev/crash_analysis_lab/uaf+0x1104) (BuildId: c4ef3acea8680ee4593d16ce8307652cb859190c)
0x502000000010 is located 0 bytes inside of 4-byte region [0x502000000010,0x502000000014)
freed by thread T0 here:
#0 0x7c68930fc4d8 in free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52
#1 0x59df62e9322f in main src/uaf.c:7
#2 0x7c6892c2a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#3 0x7c6892c2a28a in __libc_start_main_impl ../csu/libc-start.c:360
#4 0x59df62e93104 in _start (/home/dev/crash_analysis_lab/uaf+0x1104) (BuildId: c4ef3acea8680ee4593d16ce8307652cb859190c)
previously allocated by thread T0 here:
#0 0x7c68930fd9c7 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
#1 0x59df62e931de in main src/uaf.c:5
#2 0x7c6892c2a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#3 0x7c6892c2a28a in __libc_start_main_impl ../csu/libc-start.c:360
#4 0x59df62e93104 in _start (/home/dev/crash_analysis_lab/uaf+0x1104) (BuildId: c4ef3acea8680ee4593d16ce8307652cb859190c)
SUMMARY: AddressSanitizer: heap-use-after-free src/uaf.c:8 in main
Shadow bytes around the buggy address:
0x501ffffffd80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x501ffffffe00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x501ffffffe80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x501fffffff00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x501fffffff80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x502000000000: fa fa[fd]fa fa fa fa fa fa fa fa fa fa fa fa fa
Analysis:
- Error: heap-use-after-free
- Operation: WRITE of size 4 (writing int value 43)
- Location: uaf.c:8 (assignment
*ptr = 43) - Allocation: 4-byte region allocated at line 5
- Free: Memory freed at line 7
- Use: Dangling pointer write at line 8
- Shadow byte
fd: Freed heap memory (quarantined by ASAN)
4. Double-Free:
Vulnerable Code:
// ~/crash_analysis_lab/src/df.c
#include <stdlib.h>
int main() {
char *ptr = malloc(10);
free(ptr);
free(ptr); // Double-free!
return 0;
}
gcc -g -O0 -fsanitize=address -fno-omit-frame-pointer -D_FORTIFY_SOURCE=0 src/df.c -o df
./df
ASAN Report:
=================================================================
==1388==ERROR: AddressSanitizer: attempting double-free on 0x502000000010 in thread T0:
#0 0x71e78a6fc4d8 in free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52
#1 0x651975eaa1da in main src/df.c:7
#2 0x71e78a22a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#3 0x71e78a22a28a in __libc_start_main_impl ../csu/libc-start.c:360
#4 0x651975eaa0e4 in _start (/home/dev/crash_analysis_lab/df+0x10e4) (BuildId: 9e41cb0cfeda12d633976b0ec4789b8bbcf76d11)
0x502000000010 is located 0 bytes inside of 10-byte region [0x502000000010,0x50200000001a)
freed by thread T0 here:
#0 0x71e78a6fc4d8 in free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52
#1 0x651975eaa1ce in main src/df.c:6
#2 0x71e78a22a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#3 0x71e78a22a28a in __libc_start_main_impl ../csu/libc-start.c:360
#4 0x651975eaa0e4 in _start (/home/dev/crash_analysis_lab/df+0x10e4) (BuildId: 9e41cb0cfeda12d633976b0ec4789b8bbcf76d11)
previously allocated by thread T0 here:
#0 0x71e78a6fd9c7 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
#1 0x651975eaa1be in main src/df.c:5
#2 0x71e78a22a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#3 0x71e78a22a28a in __libc_start_main_impl ../csu/libc-start.c:360
#4 0x651975eaa0e4 in _start (/home/dev/crash_analysis_lab/df+0x10e4) (BuildId: 9e41cb0cfeda12d633976b0ec4789b8bbcf76d11)
SUMMARY: AddressSanitizer: double-free ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:52 in free
==1388==ABORTING
Aborted
Analysis:
- Error: double-free (attempting to free already-freed memory)
- Operation: Second free() call on same pointer
- Location: df.c:7 (second
free(ptr)) - Allocation: 10-byte region allocated at line 5
- First free: Memory freed at line 6
- Second free: Invalid free attempt at line 7
- Impact: Can corrupt heap metadata, potentially exploitable
5. Memory Leak:
Vulnerable Code:
// ~/crash_analysis_lab/src/ml.c
#include <stdlib.h>
int main() {
char *leak = malloc(100);
// No free! Program exits.
return 0;
}
gcc -g -O0 -fsanitize=address -fno-omit-frame-pointer -D_FORTIFY_SOURCE=0 src/ml.c -o ml
./ml
ASAN Report (with leak detection enabled):
=================================================================
==1404==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 100 byte(s) in 1 object(s) allocated from:
#0 0x7536e1efd9c7 in malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:69
#1 0x5b8260bb219e in main src/ml.c:5
#2 0x7536e1a2a1c9 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
#3 0x7536e1a2a28a in __libc_start_main_impl ../csu/libc-start.c:360
#4 0x5b8260bb20c4 in _start (/home/dev/crash_analysis_lab/ml+0x10c4) (BuildId: a852beeb6801e117a58cb487aa280c8fb55a3964)
SUMMARY: AddressSanitizer: 100 byte(s) leaked in 1 allocation(s).
Aborted
Analysis:
- Error: Memory leak detected by LeakSanitizer (part of ASAN)
- Type: Direct leak (pointer lost, not reachable)
- Size: 100 bytes in 1 allocation
- Location: ml.c:5 (malloc call)
- Cause: Program exits without freeing allocated memory
- Note: LeakSanitizer runs at program exit to detect unreachable allocations
ASAN Options and Configuration
Key Options:
# Common ASAN options
export ASAN_OPTIONS="symbolize=1:abort_on_error=1:detect_leaks=1:detect_stack_use_after_return=1:check_initialization_order=1:strict_init_order=1:allocator_may_return_null=1"
# Break into debugger on error
export ASAN_OPTIONS="symbolize=1:abort_on_error=0:halt_on_error=1"
# Generate detailed logs
export ASAN_OPTIONS="symbolize=1:log_path=asan.log:log_exe_name=1"
# Suppress specific errors
export ASAN_OPTIONS="suppressions=asan_suppressions.txt"
Suppression File Example (asan_suppressions.txt):
# Suppress known false positives
leak:known_leak_function
heap-buffer-overflow:third_party_library
Comparing ASAN with Traditional Debugging
ASAN Advantages:
- Detects errors at point of occurrence (not later crash)
- Provides exact allocation/free stack traces
- Catches leaks without explicit testing
- Red zones catch off-by-one errors
- Quarantine catches some UAF that might not crash
Limitations:
- Performance overhead limits production use
- Doesn't catch all logic bugs
- Can miss non-deterministic races
- Requires recompilation
When to Use Each:
- ASAN: During development and fuzzing for comprehensive testing
- Traditional debugging: Production crashes, reverse engineering binaries
- Both: Reproduce ASAN-found bug in debugger for detailed analysis
When ASAN Changes Behavior
[!WARNING] ASAN modifies heap layout and timing. A bug that crashes reliably under ASAN may behave completely differently (or not manifest at all) in a non-ASAN build. Always reproduce important bugs in both configurations.
Why ASAN Changes Crash Behavior:
-
Heap Layout Changes:
- ASAN adds red zones (padding) around allocations
- Allocation sizes are rounded up
- Heap addresses are completely different
- Adjacent allocations that would overlap in normal builds are separated
-
Quarantine Effects:
- Freed memory is held in quarantine before reuse
- UAF bugs may "disappear" because memory isn't immediately reallocated
- Without ASAN, freed memory may be immediately reused
-
Timing Differences:
- ASAN instrumentation adds overhead
- Race conditions may hide or manifest differently
- Callback timing changes
Mini-Lab: Same Bug, Different Manifestation
uaf_timing.c - Demonstrates how UAF behavior differs with/without ASAN:
// ~/crash_analysis_lab/src/uaf_timing.c - UAF that behaves differently with ASAN
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
int main() {
// Allocate object
char* victim = malloc(32);
strcpy(victim, "ORIGINAL_DATA");
printf("[1] Allocated victim at %p: %s\n", victim, victim);
// Free it
free(victim);
printf("[2] Freed victim\n");
// Allocate something else (may reuse the slot without ASAN)
char* other = malloc(32);
strcpy(other, "REPLACED!!!!!");
printf("[3] Allocated other at %p: %s\n", other, other);
// USE AFTER FREE - read victim
printf("[4] UAF read of victim: %s\n", victim);
// The output differs dramatically:
// Without ASAN: May print "REPLACED!!!!!" (memory reused)
// With ASAN: Crashes immediately at the UAF read
free(other);
return 0;
}
Exercise:
cd ~/crash_analysis_lab
# set the asan envs(from start of day 2)
# Build without ASAN
gcc -g -O0 -fno-omit-frame-pointer src/uaf_timing.c -o uaf_normal
# Build with ASAN
gcc -g -O0 -fsanitize=address -fno-omit-frame-pointer -D_FORTIFY_SOURCE=0 src/uaf_timing.c -o uaf_asan
# Run without ASAN - observe behavior
./uaf_normal
# [1] Allocated victim at 0x5a9d113fa2a0: ORIGINAL_DATA
# [2] Freed victim
# [3] Allocated other at 0x5a9d113fa2a0: REPLACED!!!!!
# [4] UAF read of victim: REPLACED!!!!! <-- No crash! Memory reused.
# Run with ASAN - immediate crash
./uaf_asan
# =================================================================
# [1] Allocated victim at 0x503000000040: ORIGINAL_DATA
# [2] Freed victim
# [3] Allocated other at 0x503000000070: REPLACED!!!!! <-- Different address!
# =================================================================
# ==1443==ERROR: AddressSanitizer: heap-use-after-free on address 0x503000000040 at pc 0x746dd12a1a6a # bp 0x7fff339b5190 sp 0x7fff339b4908
# ... ASAN report with allocation/free stacks ...
Key Observations:
- Without ASAN:
malloc()immediately reused the freed slot - With ASAN: Quarantine prevents reuse; UAF is detected
- The "bug" exists in both builds, but only ASAN catches it
Quarantine Tuning
Control ASAN's quarantine to understand timing effects:
# Disable quarantine entirely (behaves more like non-ASAN)
export ASAN_OPTIONS="quarantine_size_mb=0"
./uaf_asan
# May now behave more like non-ASAN build (memory reused faster)
# Increase quarantine (hold freed memory longer)
export ASAN_OPTIONS="quarantine_size_mb=256"
./uaf_asan
# UAF detection more reliable, but uses more memory
# Default is usually 256MB - check with:
export ASAN_OPTIONS="verbosity=1"
./uaf_asan 2>&1 | grep quarantine
Reproduction Best Practice
For any bug found with ASAN:
# 1. Document ASAN detection
./target_asan < crash_input 2>&1 | tee asan_report.txt
# 2. ALWAYS reproduce without ASAN
./target_normal < crash_input 2>&1 | tee normal_report.txt
# 3. Compare behaviors
echo "=== ASAN Behavior ===" && head -20 asan_report.txt
echo "=== Normal Behavior ===" && head -20 normal_report.txt
# 4. If normal build doesn't crash:
# - Bug is still real, but harder to exploit
# - May need heap grooming for reliable exploitation
# - Document both behaviors in your report
Other Sanitizers
- While AddressSanitizer (ASAN) is the most widely-used sanitizer for spatial memory safety, the LLVM sanitizer family includes several complementary tools that detect different bug classes.
- Understanding when to use each sanitizer—and which ones can be combined—is essential for comprehensive testing.
MemorySanitizer (MSAN): Detecting Uninitialized Memory
What MSAN Detects:
- Use of uninitialized memory
- Uninitialized variables passed to functions
- Uninitialized memory in conditionals
- Propagation of uninitialized data
Compilation:
# Compile with MSAN
clang -fsanitize=memory -fPIE -pie -fno-omit-frame-pointer -g -O0 program.c -o program_msan
# MSAN requires instrumented standard library for best results
# On Ubuntu with custom-built libc++:
clang -fsanitize=memory -stdlib=libc++ -fPIE -pie -g -O0 program.c -o program_msan
Installing libc++ for MSAN from apt.llvm.org (Optional but recommended):
MSAN works best with an instrumented libc++. Without it, you may get false positives from uninstrumented stdlib calls. The LLVM project provides pre-built libc++ packages via apt.llvm.org.
sudo apt-get update
sudo apt-get install -y wget lsb-release software-properties-common gnupg
# install llvm if you haven't already
sudo apt-get install -y \
libc++-19-dev \
libc++abi-19-dev
Example MSAN Detection:
// ~/crash_analysis_lab/src/msan.c
#include <stdio.h>
int main() {
int x; // Uninitialized!
if (x > 10) { // Reading uninitialized memory
printf("x is large\n");
}
return 0;
}
cd ~/crash_analysis_lab
clang++-19 -fsanitize=memory -stdlib=libc++ -o msan src/msan.c
./msan
MSAN Report:
==2329==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x555555621d01 in main (/home/dev/crash_analysis_lab/msan+0xcdd01) (BuildId: a1bfcfbc905803f4547f0977c2e647e8f076e8a8)
#1 0x7ffff7a2a1c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#2 0x7ffff7a2a28a in __libc_start_main csu/../csu/libc-start.c:360:3
#3 0x5555555862f4 in _start (/home/dev/crash_analysis_lab/msan+0x322f4) (BuildId: a1bfcfbc905803f4547f0977c2e647e8f076e8a8)
SUMMARY: MemorySanitizer: use-of-uninitialized-value (/home/dev/crash_analysis_lab/msan+0xcdd01) (BuildId: a1bfcfbc905803f4547f0977c2e647e8f076e8a8) in main
Exiting
When to Use MSAN:
- Logic errors from uninitialized variables
- Information leaks via uninitialized stack/heap data
- Parser bugs that rely on uninitialized state
- Kernel-style code sensitive to info leaks
ThreadSanitizer (TSAN): Detecting Data Races
What TSAN Detects:
- Data races between threads
- Unsynchronized memory accesses
- Use-after-free in multithreaded contexts
- Deadlocks
- Lock order violations
Example TSAN Detection:
// ~/crash_analysis_lab/src/tsan.c
#include <pthread.h>
#include <stdio.h>
int shared_variable = 0;
void* thread_func(void* arg) {
shared_variable++; // Race condition!
return NULL;
}
int main() {
pthread_t t1, t2;
pthread_create(&t1, NULL, thread_func, NULL);
pthread_create(&t2, NULL, thread_func, NULL);
pthread_join(t1, NULL);
pthread_join(t2, NULL);
printf("Result: %d\n", shared_variable);
return 0;
}
Compilation:
gcc -fsanitize=thread -g -O0 -fno-omit-frame-pointer src/tsan.c -o tsan -lpthread
setarch $(uname -m) -R ./tsan
TSAN Report:
==================
WARNING: ThreadSanitizer: data race (pid=10025)
Read of size 4 at 0x555555558014 by thread T2:
#0 thread_func src/tsan.c:7 (tsan+0x1294) (BuildId: 44799b6c3e78781b5904ab4054a54211be4ffe7d)
Previous write of size 4 at 0x555555558014 by thread T1:
#0 thread_func src/tsan.c:7 (tsan+0x12ac) (BuildId: 44799b6c3e78781b5904ab4054a54211be4ffe7d)
Location is global 'shared_variable' of size 4 at 0x555555558014 (tsan+0x4014)
Thread T2 (tid=10028, running) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5ac1a) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
#1 main src/tsan.c:14 (tsan+0x1327) (BuildId: 44799b6c3e78781b5904ab4054a54211be4ffe7d)
Thread T1 (tid=10027, finished) created by main thread at:
#0 pthread_create ../../../../src/libsanitizer/tsan/tsan_interceptors_posix.cpp:1022 (libtsan.so.2+0x5ac1a) (BuildId: 38097064631f7912bd33117a9c83d08b42e15571)
#1 main src/tsan.c:13 (tsan+0x130a) (BuildId: 44799b6c3e78781b5904ab4054a54211be4ffe7d)
SUMMARY: ThreadSanitizer: data race src/tsan.c:7 in thread_func
==================
Result: 2
ThreadSanitizer: reported 1 warnings
When to Use TSAN:
- Multithreaded applications
- Server software with concurrent request handling
- Race condition vulnerabilities
- Non-deterministic crashes
- Lock-free data structures
Lab: Race Condition Analysis with TSAN and valgrind
Lab Target: Multithreaded UAF
race_uaf.c - A race condition leading to use-after-free:
// ~/crash_analysis_lab/src/race_uaf.c - Thread race causes UAF
#include <pthread.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
typedef struct {
char* data;
int active;
} Resource;
Resource* global_resource = NULL;
void* writer_thread(void* arg) {
for (int i = 0; i < 1000; i++) {
if (global_resource && global_resource->active) {
// RACE: Resource may be freed between check and use
strcpy(global_resource->data, "Updated by writer");
}
usleep(100);
}
return NULL;
}
void* destroyer_thread(void* arg) {
for (int i = 0; i < 100; i++) {
usleep(1000);
if (global_resource) {
// RACE: Writer may be using data when we free it
global_resource->active = 0;
free(global_resource->data); // UAF source!
global_resource->data = NULL;
// Reallocate
global_resource->data = malloc(64);
global_resource->active = 1;
}
}
return NULL;
}
int main() {
// Initialize resource
global_resource = malloc(sizeof(Resource));
global_resource->data = malloc(64);
global_resource->active = 1;
strcpy(global_resource->data, "Initial data");
pthread_t writer, destroyer;
pthread_create(&writer, NULL, writer_thread, NULL);
pthread_create(&destroyer, NULL, destroyer_thread, NULL);
pthread_join(writer, NULL);
pthread_join(destroyer, NULL);
free(global_resource->data);
free(global_resource);
return 0;
}
Exercise Part 1: Reproduce with TSAN
gcc -fsanitize=thread -g -O0 -fno-omit-frame-pointer src/race_uaf.c -o race_uaf -lpthread
setarch $(uname -m) -R ./race_uaf
Exercise Part 2: Detect Races with Helgrind
TSAN detects the race, but Helgrind (part of Valgrind) provides more detailed analysis and works in VMs without hardware PMU support:
cd ~/crash_analysis_lab
# Build normally (without TSAN - for Helgrind analysis)
clang -g -O0 -fno-omit-frame-pointer src/race_uaf.c -o race_normal -lpthread
# Normal run - may or may not crash
./race_normal # Often "works" due to lucky timing
# Install Valgrind if needed
sudo apt install valgrind
# Run with Helgrind - detects races without needing a crash
valgrind --tool=helgrind ./race_normal
# For more detailed history (slower but more accurate)
valgrind --tool=helgrind --history-level=full ./race_normal
# Alternative: DRD (another Valgrind thread checker, sometimes catches different issues)
valgrind --tool=drd ./race_normal
Sample Helgrind Output:
==1124== Possible data race during write of size 4 at 0x4A8B048 by thread #3
==1124== Locks held: none
==1124== at 0x10924C: destroyer_thread (src/race_uaf.c:32)
==1124==
==1124== This conflicts with a previous read of size 4 by thread #2
==1124== Locks held: none
==1124== at 0x1091C5: writer_thread (src/race_uaf.c:17)
==1124== Address 0x4a8b048 is 8 bytes inside a block of size 16 alloc'd
==1124== at 0x48488A8: malloc
==1124== by 0x1092C8: main (src/race_uaf.c:46)
Helgrind shows:
- Which threads are racing (thread #2 vs #3)
- Exact source locations (line 32 vs line 17)
- The memory address and allocation origin
- That no locks were held during access
Exercise Part 3: Analyze the Race Conditions
Use Helgrind output to answer these questions:
-
What data is being raced on?
Look for "Possible data race" messages - they show the address and what allocated it:
Address 0x4a8b048 is 8 bytes inside a block of size 16 alloc'd by main (src/race_uaf.c:46) -
Which threads are involved?
Helgrind announces threads and shows their creation stack:
Thread #3 was created at pthread_create by main (src/race_uaf.c:53) -
What's the UAF pattern?
Look for races where one thread writes/frees while another reads:
# Thread 3 (destroyer) writes to data->active destroyer_thread (src/race_uaf.c:32) # Thread 2 (writer) reads data->active writer_thread (src/race_uaf.c:17) -
Identify the strcpy UAF:
Possible data race during write of size 1 at 0x4A8B090 by thread #2 at strcpy by writer_thread (src/race_uaf.c:19) Address 0x4a8b090 is 0 bytes inside a block of size 64 alloc'd by destroyer_thread (src/race_uaf.c:37) # <-- reallocated after free!
Lab Deliverables
- TSAN report showing the detected race
- valgrind helgrind command that reproduces the crash
- Interleaving description: Which thread did what, in what order
- Root cause: One paragraph explaining the bug
Success Criteria:
- Can detect race with TSAN
- Can reproduce race with valgrind
- Can explain the thread interleaving that causes the bug
- Understand why normal runs often don't crash
UndefinedBehaviorSanitizer (UBSAN): Catching Undefined Behavior
What UBSAN Detects:
- Integer overflow (signed)
- Division by zero
- Null pointer dereference
- Misaligned pointer access
- Array bounds violations (with bounds checking)
- Type confusion (via vptr checks)
- Shifts by invalid amounts
Example UBSAN Detection:
// ~/crash_analysis_lab/src/ubsan.c
#include <stdio.h>
#include <limits.h>
int main() {
int x = INT_MAX;
x++; // Signed integer overflow
printf("x = %d\n", x);
int y = 5;
int z = y / 0; // Division by zero
return 0;
}
Compilation:
# Compile with UBSAN (all checks)
clang -fsanitize=undefined -g -O0 -fno-omit-frame-pointer src/ubsan.c -o ubsan
# Compile with specific checks
clang -fsanitize=signed-integer-overflow,bounds -g -O0 -fno-omit-frame-pointer src/ubsan.c -o ubsan1
# Abort on first error (no recovery)
clang -fsanitize=undefined -fno-sanitize-recover=undefined -g -O0 -fno-omit-frame-pointer src/ubsan.c -o ubsan2
Compiler Warning (at compile time):
src/ubsan.c:11:15: warning: division by zero is undefined [-Wdivision-by-zero]
11 | int z = y / 0; // Division by zero
| ^ ~
1 warning generated.
UBSAN Runtime Report:
# Run with halt_on_error=0 to see all errors (otherwise aborts on first)
$ UBSAN_OPTIONS=halt_on_error=0 ./ubsan
src/ubsan.c:7:6: runtime error: signed integer overflow: 2147483647 + 1 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior src/ubsan.c:7:6
x = -2147483648
src/ubsan.c:11:15: runtime error: division by zero
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior src/ubsan.c:11:15
UndefinedBehaviorSanitizer:DEADLYSIGNAL
==1189==ERROR: UndefinedBehaviorSanitizer: FPE on unknown address 0x5555b6f99873 (pc 0x5555b6f99873 bp 0x7ffe6cee6130 sp 0x7ffe6cee6110 T1189)
#0 0x5555b6f99873 in main /home/dev/crash_analysis_lab/src/ubsan.c:11:15
#1 0x73ccf2e2a1c9 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#2 0x73ccf2e2a28a in __libc_start_main csu/../csu/libc-start.c:360:3
#3 0x5555b6f6f3e4 in _start (/home/dev/crash_analysis_lab/ubsan+0x53e4)
UndefinedBehaviorSanitizer can not provide additional info.
SUMMARY: UndefinedBehaviorSanitizer: FPE /home/dev/crash_analysis_lab/src/ubsan.c:11:15 in main
==1189==ABORTING
Key Observations:
- Integer overflow (line 7): Detected and recoverable — execution continues, showing wrapped value
-2147483648 - Division by zero (line 11): Detected but fatal — CPU raises
SIGFPE(Floating Point Exception), program aborts regardless ofhalt_on_errorsetting - Without
halt_on_error=0, UBSAN aborts on the first error (integer overflow)
When to Use UBSAN:
- Integer overflow vulnerabilities
- Arithmetic bugs in parsers
- Type confusion detection
- Undefined behavior that doesn't crash immediately
- Hardening development builds
Sanitizer Combinations
Compatible Combinations:
cd ~/crash_analysis_lab
# ASAN + UBSAN (Recommended for general fuzzing)
clang -fsanitize=address,undefined -g -O0 -fno-omit-frame-pointer -D_FORTIFY_SOURCE=0 src/ubsan.c -o asan_ubsan
# ASAN + UBSAN + leak detection
clang -fsanitize=address,undefined -g -O0 -fno-omit-frame-pointer -D_FORTIFY_SOURCE=0 src/ubsan.c -o asan_ubsan_leak
export ASAN_OPTIONS=detect_leaks=1
# MSAN + UBSAN (for uninitialized memory + undefined behavior)
# Note: MSAN requires instrumented libc++, use clang++ with -stdlib=libc++
clang++-19 -fsanitize=memory,undefined -stdlib=libc++ -fPIE -pie -g -O0 -fno-omit-frame-pointer src/ubsan.c -o msan_ubsan
Running Combined Sanitizers:
# ASAN + UBSAN catches both memory errors and undefined behavior
$ UBSAN_OPTIONS=halt_on_error=0 ./asan_ubsan
src/ubsan.c:7:6: runtime error: signed integer overflow: 2147483647 + 1 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior src/ubsan.c:7:6
x = -2147483648
src/ubsan.c:11:15: runtime error: division by zero
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior src/ubsan.c:11:15
UndefinedBehaviorSanitizer:DEADLYSIGNAL
...
Incompatible Combinations (Cannot Use Together):
| Combination | Reason |
|---|---|
| ASAN + MSAN | Both use shadow memory with conflicting layouts |
| ASAN + TSAN | Conflicting instrumentation and memory tracking |
| MSAN + TSAN | Conflicting instrumentation |
Combination Best Practices:
-
Default Fuzzing Setup: ASAN + UBSAN
- Catches most memory corruption + arithmetic errors
- Good performance trade-off (~2x slowdown)
- Use:
clang -fsanitize=address,undefined ...
-
Dedicated MSAN Run: Separate build with MSAN + UBSAN
- Run periodically to catch uninitialized memory
- Requires instrumented libc++ (
clang++ -stdlib=libc++) - Cannot combine with ASAN
-
Dedicated TSAN Run: For multithreaded targets
- Run separate TSAN build (cannot combine with ASAN/MSAN)
- Higher overhead (~5-15x slowdown)
- Use:
gcc -fsanitize=thread -lpthread ...
Performance Comparison
| Sanitizer | CPU Overhead | Memory Overhead | Use Case |
|---|---|---|---|
| ASAN | ~2x | 2-3x | Spatial memory safety (overflow, UAF) |
| MSAN | ~3x | 2-3x | Uninitialized memory reads |
| TSAN | 5-15x | 5-10x | Data races in multithreaded code |
| UBSAN | ~1.2x | Minimal | Undefined behavior (overflow, div-by-zero) |
| ASAN+UBSAN | ~2.2x | 2-3x | Combined memory + arithmetic bugs |
Performance Notes:
- ASAN overhead is predictable and acceptable for fuzzing
- TSAN overhead makes it impractical for long fuzzing campaigns
- UBSAN adds minimal overhead—almost always worth enabling
- MSAN requires instrumented standard library for full effectiveness
Advanced Sanitizers (Brief Overview)
Several newer sanitizer technologies address ASAN's limitations. These are covered in depth in later weeks but are important to know about for crash analysis:
HWASan (Hardware-assisted AddressSanitizer):
- Uses ARM64 Top Byte Ignore (TBI) feature for memory tagging
- ~2x overhead vs ASAN's ~2x (similar), but uses only ~15% more memory vs ASAN's 2-3x
- Essential for Android/ARM64 crash analysis
- Detects same bug classes as ASAN with better memory efficiency
MTE (Memory Tagging Extension):
- ARM hardware feature (ARMv8.5+, e.g., Pixel 8, server ARM64)
- Near-zero overhead memory safety in production
- Crashes from MTE-enabled binaries require understanding tag mismatch errors
- Increasingly important as ARM64 adoption grows
GWP-ASan (Google-Wide Performance ASan):
- Sampling-based allocator for production use
- Catches ~1% of heap bugs with minimal overhead
- Deployed in Chrome/Chromium and Android (platform- and version-specific), and available via allocator integrations (e.g., LLVM Scudo)
- Useful for analyzing crashes from production telemetry
Frida for Dynamic Analysis:
- Runtime instrumentation without recompilation
- Essential for closed-source binary crash analysis
- Can trace memory operations, hook functions, and dump state
- Covered in detail in later weeks for mobile/binary analysis
These tools become relevant when analyzing crashes from production systems, mobile platforms, or closed-source binaries where traditional ASAN isn't available.
GWP-ASan: Production Crash Analysis
GWP-ASan (originally "Google-Wide Performance ASan") is a sampling-based heap error detector designed for production use.
Where GWP-ASan Runs:
- Chrome/Chromium: Deployed in production (often via feature flags/field trials); used for crash telemetry
- Android: Integrated into the platform allocator on many devices; configuration is platform-specific
- LLVM/Scudo allocator: Includes GWP-ASan; the easiest way to try it locally is building with
-fsanitize=scudo - Other allocators: Some allocators implement guarded sampling / GWP-ASan-style mechanisms
How GWP-ASan Works:
Traditional ASAN: Every allocation → Shadow memory → Every access checked
GWP-ASan: Random sample → Guard pages → Only sampled allocs checked
┌─────────────────────────────────────────────────────────────┐
│ Normal Allocations (99.9%) │ GWP-ASan Sampled (0.1%) │
│ ┌─────┬─────┬─────┬─────┐ │ ┌─────┬─────┬─────┐ │
│ │alloc│alloc│alloc│alloc│ │ │GUARD│alloc│GUARD│ │
│ └─────┴─────┴─────┴─────┘ │ └─────┴─────┴─────┘ │
│ No overhead │ Guard pages catch OOB │
└─────────────────────────────────────────────────────────────┘
Analyzing GWP-ASan Crash Reports:
GWP-ASan reports look similar to ASAN but with sampling context:
*** GWP-ASan detected a memory error ***
Use-after-free at 0x7f1234567890
Allocation:
#0 0x7f111 in malloc
#1 0x7f222 in create_object (object.c:45)
#2 0x7f333 in main (main.c:123)
Deallocation:
#0 0x7f444 in free
#1 0x7f555 in destroy_object (object.c:89)
#2 0x7f666 in cleanup (main.c:150)
Use-after-free access:
#0 0x7f777 in use_object (object.c:67)
#1 0x7f888 in process (main.c:175)
GWP-ASan sampling rate: 1/1000 allocations
Enabling GWP-ASan:
# IMPORTANT: GWP-ASan is allocator-integrated. There is no generic "enable it in glibc" switch.
# The most practical way to experiment locally is via LLVM Scudo:
clang -fsanitize=scudo -g program.c -o program_scudo
# Adjust sampling via Scudo (example). Lower SampleRate => more sampling.
# SampleRate=1 means "always sample" (development only).
export SCUDO_OPTIONS=GWP_ASAN_SampleRate=1
./program_scudo < crash_input
# Android - check app eligibility
adb shell getprop | grep gwp
# persist.device_config.runtime_native.gwp_asan.* properties
# Chrome/Chromium - see current docs (flags/config changes over time)
# https://chromium.googlesource.com/chromium/src/+/HEAD/docs/gwp_asan.md
Reproducing GWP-ASan Crashes:
GWP-ASan crashes are non-deterministic (sampled). To reproduce:
# Option 1: Use full ASAN to reproduce deterministically
clang -fsanitize=address -g program.c -o program_asan
./program_asan < crash_input
# Option 2: If you can rebuild with Scudo, reproduce under its GWP-ASan integration
clang -fsanitize=scudo -g program.c -o program_scudo
SCUDO_OPTIONS=GWP_ASAN_SampleRate=1 ./program_scudo < crash_input
# Option 3: If you only have a production binary, run repeatedly until it gets sampled
for i in {1..1000}; do
./program < crash_input 2>&1 | grep -q "GWP-ASan" && break
done
GWP-ASan vs ASAN for Crash Analysis:
| Aspect | GWP-ASan | ASAN |
|---|---|---|
| Overhead | ~0.1% | ~200% |
| Memory | Minimal | 2-3x |
| Detection rate | ~1% of bugs | 100% of bugs |
| Use case | Production | Development/fuzzing |
| Reproducibility | Low (sampling) | 100% |
| Deployment | Safe for prod | Never in prod |
Workflow: GWP-ASan Crash → Full Analysis:
# 1. Receive GWP-ASan crash from production telemetry
# 2. Extract crash details (allocation stack, free stack, access stack)
# 3. Create reproducer from crash input
echo "$crash_input" > repro.bin
# 4. Build with full ASAN for deterministic reproduction
clang -fsanitize=address -g program.c -o program_asan
# 5. Run ASAN build to get complete analysis
./program_asan < repro.bin
# Now get full ASAN report with 100% detection
# 6. If can't reproduce, the allocation pattern matters
# GWP-ASan only caught it because specific allocation was sampled
# May need to create targeted test case based on stacks
Key Points for GWP-ASan Analysis:
- Sampling means incomplete view: The bug exists, but you only caught it by luck
- Allocation context is crucial: The allocation stack tells you what was sampled
- Use full ASAN to reproduce: Convert GWP-ASan report to ASAN-reproducible test
- Production-only bugs are real: Some bugs only manifest under real workloads
- Check telemetry frequency: Multiple GWP-ASan hits = higher severity bug
Practical Workflow
Step 1: Initial Fuzzing (ASAN + UBSAN):
# Compile with recommended combination
clang -fsanitize=address,undefined -g -O0 -fno-omit-frame-pointer -D_FORTIFY_SOURCE=0 target.c -o target_asan_ubsan
# Fuzz with AFL++
afl-fuzz -i seeds/ -o out/ -m none -- ./target_asan_ubsan @@
Step 2: Periodic MSAN Check:
# Compile with MSAN (requires instrumented libc++)
clang++-19 -fsanitize=memory -stdlib=libc++ -fPIE -pie -g -O0 -fno-omit-frame-pointer target.c -o target_msan
# Run corpus through MSAN build
for testcase in out/queue/*; do
./target_msan < $testcase
done
Step 3: Multithreaded Target TSAN Check:
# Compile with TSAN (use gcc or clang)
gcc -fsanitize=thread -g -O0 -fno-omit-frame-pointer target.c -o target_tsan -lpthread
# Run with diverse inputs
for testcase in out/queue/*; do
./target_tsan < $testcase
done
Sanitizer Selection Guide:
┌─────────────────────────────────────────────────────────────────────┐
│ What are you testing? │
└─────────────────────────────────────────────────────────────────────┘
│
├─ Single-threaded parser/server
│ └─> ASAN + UBSAN (default choice)
│ clang -fsanitize=address,undefined ...
│
├─ Multithreaded application
│ └─> Separate runs: ASAN+UBSAN, then TSAN
│ gcc -fsanitize=thread ... -lpthread
│
├─ Kernel/crypto code with info leaks
│ └─> MSAN (separate run, requires instrumented libc++)
│ clang++ -fsanitize=memory -stdlib=libc++ ...
│
└─ Arithmetic-heavy code
└─> UBSAN (minimal overhead, always enable)
clang -fsanitize=undefined ...
Example: Combining Sanitizers
Scenario: Fuzzing a multithreaded HTTP server
Phase 1: ASAN + UBSAN fuzzing (24 hours)
afl-fuzz -i seeds/ -o findings_asan/ -m none -- ./httpd_asan_ubsan @@
# Found: 3 heap overflows, 2 integer overflows
Phase 2: MSAN validation (4 hours)
# Run interesting inputs through MSAN
for crash in findings_asan/crashes/*; do
./httpd_msan < $crash
done
# Found: 1 uninitialized variable leading to info leak
Phase 3: TSAN validation (4 hours)
# Run corpus through TSAN
for input in findings_asan/queue/*; do
./httpd_tsan < $input
done
# Found: 2 data races in request handling
Result: 8 unique bugs across 3 bug classes
Practical Exercise
Task: Identify and classify 10 ASAN-detected bugs
If you built and fuzzed real targets in Week 2 (for example, libWebP, GStreamer, or your own small parser/HTTP server), consider recompiling one of those exact targets with ASAN and running this workflow on the crashes you already found. The synthetic exercises below are fine to start with, but applying the same process to a familiar Week 2 target will make the connection between fuzzing and crash analysis very concrete.
Provided Test Programs (compile each with ASAN):
heap_overflow.c- Heap buffer overflowstack_overflow.c- Stack buffer overflowuaf_read.c- Use-after-free (read)uaf_write.c- Use-after-free (write)double_free.c- Double-freememory_leak.c- Memory leakglobal_overflow.c- Global buffer overflowstack_use_after_return.c- Stack use-after-returninitialization_order.c- Initialization order bugalloc_dealloc_mismatch.c- new/delete mismatch
For Each Program:
-
Compile with ASAN:
clang -g -O1 -fsanitize=address -fno-omit-frame-pointer program.c -o program_asan -
Run and Capture Output:
./program_asan 2>&1 | tee program_output.txt -
Analyze Report:
- What type of error was detected?
- What line triggered it?
- What was the allocation/free stack trace?
- How many bytes were involved?
-
Classify Exploitability:
- Read vs Write access?
- Controlled by attacker input?
- How many bytes overflow?
- What mitigations apply?
-
Document:
## Bug: heap_overflow.c
- **ASAN Type**: heap-buffer-overflow
- **Operation**: WRITE of size 18
- **Overflow**: 8 bytes past 10-byte allocation
- **Exploitability**: High - Write overflow with large controlled data
Success Criteria:
- All 10 programs analyzed
- ASAN error types correctly identified
- Stack traces interpreted
- Exploitability assessed
- Clear documentation of findings
Key Takeaways
- ASAN is powerful: Catches bugs at source, not just symptoms
- Detailed reports: Allocation and free stacks make root cause obvious
- Multiple error types: Different bugs have different ASAN signatures
- Essential for fuzzing: Turns crashes into actionable vulnerability reports
- Combine with debugging: ASAN finds bug, debugger analyzes exploit primitive
Discussion Questions
- Why does ASAN have lower false positive rate than traditional memory checkers like Valgrind?
- How does the quarantine mechanism help catch use-after-free bugs?
- When would you use MSAN vs ASAN vs TSAN for a multi-threaded program with suspected memory issues?
- Why can't ASAN and MSAN be combined in the same build, and how do you work around this limitation?
Day 3: Exploitability Assessment with Automated Tools
- Goal: Use automated tools to assess crash exploitability and prioritize vulnerabilities.
- Activities:
- Reading:
- Online Resources:
- Tool Setup:
- CASR (Rust-based crash analyzer - primary tool)
- AFL++ utilities (afl-tmin, afl-cmin)
- Exercise:
- Triage 20 AFL++ crashes
- Bucket by exploitability and uniqueness
Quick Triage Checklist
Before diving into detailed analysis, run through this checklist for every crash:
# Crash Triage Checklist
## What crashed?
- Instruction: (e.g., mov [rax], rcx)
- Signal: (e.g., SIGSEGV, SIGABRT)
## What register/memory was accessed?
- Faulting address: (e.g., 0x4141414141414141)
- Access type: [ ] Read [ ] Write [ ] Execute
## Is that value attacker-controlled?
- Pattern visible: [ ] Yes [ ] No
- Input correlation: [ ] Direct [ ] Indirect [ ] Unknown
## What mitigations are active?
- Stack canary: [ ] Yes [ ] No
- NX/DEP: [ ] Yes [ ] No
- ASLR/PIE: [ ] Yes [ ] No
- RELRO: [ ] None [ ] Partial [ ] Full
- CFG/CFI: [ ] Yes [ ] No
- CET: [ ] Yes [ ] No
## Initial classification
- Type: [ ] Stack overflow [ ] Heap overflow [ ] UAF [ ] Format string [ ] Other
- Severity: [ ] EXPLOITABLE [ ] PROBABLY_EXPLOITABLE [ ] NOT_EXPLOITABLE
Interactive Analysis and Mitigation Checks
Checking Binary Mitigations First
Always check mitigations before deep analysis - they determine exploitability:
Using checksec (pwntools):
# Install if needed
cd crash_analysis_lab/
source .venv/bin/activate
# pip install pwntools
checksec --file=./vuln_no_protect
#[*] '/home/dev/crash_analysis_lab/vuln_no_protect'
# Arch: amd64-64-little
# RELRO: Partial RELRO
# Stack: No canary found
# NX: NX unknown - GNU_STACK missing
# PIE: No PIE (0x400000)
# Stack: Executable
# RWX: Has RWX segments
# SHSTK: Enabled
# IBT: Enabled
# Stripped: No
# Debuginfo: Yes
checksec --file=./vuln_asan
# [*] '/home/dev/crash_analysis_lab/vuln_asan'
# Arch: amd64-64-little
# RELRO: Full RELRO
# Stack: Canary found
# NX: NX enabled
# PIE: PIE enabled
# FORTIFY: Enabled
# ASAN: Enabled
# SHSTK: Enabled
# IBT: Enabled
# Stripped: No
# Debuginfo: Yes
Checking for CET (Control-flow Enforcement Technology):
# Check if binary has CET enabled
readelf -n ./vuln_protected
# Properties: x86 feature: IBT, SHSTK
# GNU_PROPERTY_X86_FEATURE_1_SHSTK (Shadow Stack)
# GNU_PROPERTY_X86_FEATURE_1_IBT (Indirect Branch Tracking)
Checking System-Wide Protections:
# ASLR status
cat /proc/sys/kernel/randomize_va_space
# 0 = disabled, 1 = conservative, 2 = full
# Kernel protection features
cat /sys/devices/system/cpu/vulnerabilities/*
Enhanced GDB with Pwndbg
- Modern crash analysis on Linux uses enhanced GDB plugins that provide significantly better crash context than vanilla GDB.
- Pwndbg is the current standard for exploit development and crash analysis, replacing older tools like the now-unmaintained GDB exploitable plugin.
What Pwndbg Provides:
- Automatic context display on every stop (registers, stack, code, backtrace)
- Heap visualization and analysis (
heap,bins,arena) - Memory search and pattern finding (
search,telescope) - Exploit development helpers (
cyclic,rop,checksec) - Enhanced memory display with smart dereferencing
Crash Analysis with Pwndbg:
# Navigate to lab directory and load crashing program (using Day 1 binaries)
cd ~/crash_analysis_lab
gdb ./vuln_no_protect
# Run with crashing input (test case 1 = stack overflow)
pwndbg> run 1 $(python3 -c "print('A'*200)")
# Pwndbg automatically shows context on crash:
#LEGEND: STACK | HEAP | CODE | DATA | WX | RODATA
#──────────────────────────────────────────────────────────────────────────────────────────────────────────────────[ REGISTERS / show-flags off / show-compact-regs off ]───────────────────────────────────────────────────────────────────────────────────────────────────────────────────
# RAX 0xd5
# RBX 0x7fffffffe128 —▸ 0x7fffffffe3db ◂— '/home/dev/crash_analysis_lab/vuln_no_protect'
# RCX 0
# RDX 0
# RDI 0x7fffffffdda0 —▸ 0x7fffffffddd0 ◂— 0x4141414141414141 ('AAAAAAAA')
# RSI 0x4052a0 ◂— 0x66667542205d2a5b ('[*] Buff')
# R8 0x73
# R9 0
# R10 0xffffffff
# R11 0x202
# R12 3
# R13 0
# R14 0x403e00 (__do_global_dtors_aux_fini_array_entry) —▸ 0x4011a0 (__do_global_dtors_aux) ◂— endbr64
# R15 0x7ffff7ffd000 (_rtld_global) —▸ 0x7ffff7ffe2e0 ◂— 0
# RBP 0x4141414141414141 ('AAAAAAAA')
# RSP 0x7fffffffdfd8 ◂— 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'
# RIP 0x401225 (stack_overflow+79) ◂— ret
#───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────[ DISASM / x86-64 / set emulate on ]────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
# ► 0x401225 <stack_overflow+79> ret <0x4141414141414141>
# ↓
#─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────[ SOURCE (CODE) ]─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
#In file: /home/dev/crash_analysis_lab/src/vulnerable_suite.c:11
# 6 void stack_overflow(char *input) {
# 7 char buffer[64];
# 8 printf("[*] Copying input to 64-byte buffer...\n");
# 9 strcpy(buffer, input); // No bounds check!
# 10 printf("[*] Buffer: %s\n", buffer);
# ► 11 }
Key Pwndbg Commands for Crash Analysis:
# Memory examination
pwndbg> telescope $rsp 20 # Smart stack display (shows 20 qwords with dereferencing)
pwndbg> hexdump $rdi 64 # Hex dump memory (use register containing valid pointer)
pwndbg> vmmap # Memory map with permissions (STACK/HEAP/CODE highlighting)
# Heap analysis (critical for heap bugs)
pwndbg> heap # Heap overview (shows allocated chunks with addr/size)
pwndbg> bins # Show all bin states (tcache, fastbins, unsorted, small, large)
pwndbg> arena # Display main arena (top chunk, bins, fastbinsY)
# Search for input patterns (finds pattern across all memory regions)
pwndbg> search "AAAA" # Find pattern in memory (shows [heap], [stack], libc, etc.)
pwndbg> search -t qword 0x4141414141414141 # Search for specific qword value
# Security checks
pwndbg> checksec # Show binary mitigations
# Exploit helpers - finding offset to control RIP
pwndbg> run 1 $(cyclic 200) # Run with de Bruijn pattern
# After crash, RSP points to pattern (e.g., "saaataaa...")
pwndbg> x/s $rsp # View pattern string at RSP
pwndbg> cyclic -l saaa -n 4 # Find offset using first 4 chars → offset 72
# Context control
pwndbg> context # Redisplay context
pwndbg> context reg stack code # Custom context
Automated Batch Analysis with Pwndbg:
#!/bin/bash
# analyze_crashes.sh
# Run from ~/crash_analysis_lab directory
cd ~/crash_analysis_lab
for crash in crashes/*; do
echo "=== Analyzing $(basename $crash) ==="
gdb -batch \
-ex "run < $crash" \
-ex "bt" \
-ex "info registers" \
-ex "checksec" \
-ex "quit" \
./vuln_no_protect 2>&1 | tee analysis_$(basename $crash).txt
done
Exploitability Assessment with Pwndbg:
# At crash, assess exploitability (using vuln_no_protect from Day 1):
cd ~/crash_analysis_lab
gdb ./vuln_no_protect
pwndbg> run 1 $(python3 -c "print('A'*200)")
# ... crash occurs ...
pwndbg> checksec
# File: /home/dev/crash_analysis_lab/vuln_no_protect
# Arch: amd64
# RELRO: Partial RELRO
# Stack: No canary found
# NX: NX unknown - GNU_STACK missing
# PIE: No PIE (0x400000)
# Stack: Executable
# RWX: Has RWX segments
# SHSTK: Enabled
# IBT: Enabled
# Stripped: No
# Debuginfo: Yes
#
# Key indicators for exploitation:
# - "Stack: Executable" + "Has RWX segments" = shellcode can run on stack
# - "No canary found" = no stack smashing protection
# - "No PIE" = fixed addresses, no ASLR for binary
# Check if RIP/RAX controlled
pwndbg> p/x $rip
# $1 = 0x401225
# RIP points to valid code (ret instruction), not yet hijacked
# Check what instruction we're at
pwndbg> x/i $rip
# => 0x401225 <stack_overflow+79>: ret
# About to return - the ret will pop 0x4141414141414141 into RIP
# Examine the backtrace - this reveals the overflow
pwndbg> bt full
# #0 0x0000000000401225 in stack_overflow (input=0x7fffffffe40a 'A' <repeats 200 times>) at vulnerable_suite.c:11
# buffer = 'A' <repeats 64 times>
# #1 0x4141414141414141 in ?? () <-- EXPLOITABLE! Return address overwritten
# #2 0x4141414141414141 in ?? () <-- Stack completely corrupted with our input
# ... (more 0x41's)
#
# Key indicators:
# - Return addresses show 0x4141414141414141 = "AAAAAAAA" (our input)
# - This means we control where execution goes after ret
# - VERDICT: EXPLOITABLE - classic stack buffer overflow with RIP control
CASR - Modern Crash Analyzer
What Is CASR?:
CASR (Crash Analysis and Severity Reporter) is a modern, Rust-based crash analysis framework developed by ISP RAS.
Key Features (v2.13+ / Latest: v2.14):
- Multi-language support: C/C++, Rust, Go, Python, Java, JavaScript, C#
- Multiple analysis backends: ASAN, UBSAN, TSAN, MSAN, GDB, core dumps
- Fuzzer integration: AFL++, libFuzzer, Atheris (Python), honggfuzz
- CI/CD ready: SARIF reports, DefectDojo integration, GitHub Actions support
- 23+ severity classes: Precise exploitability assessment with modern patterns
- Clustering: Automatic deduplication using stack trace similarity
- TUI interface: Interactive crash browsing with filtering
- LibAFL integration: Native support for Rust-based fuzzing (v2.14+)
Installation:
# Install via cargo
cargo install casr
# Or from source for latest features
git clone https://github.com/ispras/casr
cd casr
cargo build --release
sudo cp target/release/casr-* /usr/local/bin/
# Verify installation
casr-san --version
casr-gdb --version
casr-cluster --version
[!IMPORTANT] CASR severity is heuristic-based: CASR is a triage assistant, not an oracle. Its classifications (EXPLOITABLE, PROBABLY_EXPLOITABLE, NOT_EXPLOITABLE) are based on crash patterns and may not reflect actual exploitability. Always perform manual analysis on high-priority crashes. For example:
- A "NOT_EXPLOITABLE" null deref might become exploitable with heap manipulation
- An "EXPLOITABLE" crash might be blocked by mitigations CASR doesn't detect
- Use CASR for prioritization, not final verdicts
CASR Tool Suite
casr-san: Analyze sanitizer output (ASAN/UBSAN/MSAN/TSAN)
# Navigate to lab directory (created in Day 1)
cd ~/crash_analysis_lab
# Create output directory for CASR reports
mkdir -p casrep
# Compile with ASAN (if not already done in Day 1)
clang -g -O1 -fsanitize=address -fno-omit-frame-pointer src/vulnerable_suite.c -o vuln_asan
# Analyze crash (test case 3 = UAF)
casr-san -o casrep/uaf.casrep -- ./vuln_asan 3
# Analyze stack overflow (test case 1 - needs ~100+ chars to overflow 64-byte buffer)
casr-san -o casrep/stack_overflow.casrep -- ./vuln_asan 1 $(python3 -c "print('A'*200)")
# Analyze heap overflow (test case 2)
casr-san -o casrep/heap_overflow.casrep -- ./vuln_asan 2 $(python3 -c "print('A'*100)")
# Analyze double free (test case 4)
casr-san -o casrep/double_free.casrep -- ./vuln_asan 4
# Analyze NULL dereference (test case 5 with trigger=0)
casr-san -o casrep/null_deref.casrep -- ./vuln_asan 5 0
casr-gdb: Analyze crashes via GDB (no sanitizer needed)
# Analyze crash using GDB (using vuln_no_protect from Day 1)
cd ~/crash_analysis_lab
# Stack overflow (test case 1) - crashes due to return address overwrite
casr-gdb -o casrep/stack_overflow_gdb.casrep -- ./vuln_no_protect 1 $(python3 -c "print('A'*200)")
# Double free (test case 4) - crashes due to glibc allocator detection
casr-gdb -o casrep/double_free_gdb.casrep -- ./vuln_no_protect 4
# NULL dereference (test case 5) - crashes on NULL pointer access
casr-gdb -o casrep/null_deref_gdb.casrep -- ./vuln_no_protect 5 0
# NOTE: Heap overflow (test 2) and UAF (test 3) typically don't crash without
# sanitizers - they corrupt memory silently. Use ASAN builds (casr-san) to detect these.
# For file-input binaries (not vulnerable_suite.c), use @@ placeholder:
# casr-gdb -o casrep/crash.casrep -- ./file_based_target @@
# With custom GDB path
casr-gdb --gdb-path /usr/local/bin/gdb -o casrep/crash.casrep -- ./vuln_no_protect 1 $(python3 -c "print('A'*200)")
casr-core: Analyze core dumps
# Navigate to lab directory (created in Day 1)
cd ~/crash_analysis_lab
mkdir -p casrep cores
# Enable core dumps
ulimit -c unlimited
# Generate crashes for different test cases
./vuln_no_protect 1 $(python3 -c "print('A'*200)") # Stack overflow
./vuln_no_protect 3 # Use-after-free
./vuln_no_protect 4 # Double free
./vuln_no_protect 5 0 # NULL dereference
# Analyze core dump
# On systemd systems, extract core first using coredumpctl(you might to look at cwd or /var/crash as well):
coredumpctl dump -o cores/vuln_no_protect.core
casr-core -o casrep/crash.casrep -e ./vuln_no_protect -c cores/vuln_no_protect.core
# Alternative: If core_pattern writes to CWD (core.%e.%p):
# casr-core -o casrep/crash.casrep -e ./vuln_no_protect -c core.vuln_no_protect.*
# Batch analyze multiple cores (after extracting with coredumpctl)
for core in cores/*; do
casr-core -o casrep/$(basename $core).casrep -e ./vuln_no_protect -c $core
done
casr-cluster: Deduplicate and cluster crashes
# Cluster all reports from casrep/ directory by call stack and crash type
casr-cluster -c casrep/ clustered/
# Leave only reports with unique crash lines in each cluster
casr-cluster -c casrep/ clustered/ --unique-crashline
# Deduplicate reports (remove duplicates, keep unique)
casr-cluster -d casrep/ deduped/
# Merge new reports into existing cluster directory
#casr-cluster -m new_crashes/ clustered/
# Update existing clusters with new reports
#casr-cluster -u new_crashes/ clustered/
# Calculate clustering quality (silhouette score)
casr-cluster -e clustered/
# Compare two crash sets (find new unique crashes)
#casr-cluster --diff new_crashes/ old_crashes/ diff_output/
# Parallel processing
#casr-cluster -c casrep/ clustered/ -j 8
casr-cli: TUI for browsing crash reports
# Launch interactive tree browser (default)
casr-cli casrep/
# View mode options: tree, slider, stdout
casr-cli -v tree casrep/
casr-cli -v slider casrep/
casr-cli -v stdout casrep/
# Print only unique crash lines in statistics
casr-cli -u casrep/
# Generate SARIF report from CASR reports
casr-cli --sarif output.sarif casrep/
# SARIF with source root for proper file paths
casr-cli --sarif output.sarif --source-root /home/dev/crash_analysis_lab casrep/
# Strip path prefix from crash paths in statistics
casr-cli --strip-path /home/dev/crash_analysis_lab/ casrep/
AFL++ Fuzzing to CASR Triage
cd ~/crash_analysis_lab/
# 1. Create a simple vulnerable target (heap overflow)
cat > src/fuzz_target.c << 'EOF'
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
volatile char sink; // Prevent optimization
void process_input(char *data, size_t len) {
char buffer[64];
// Vulnerability 1: Stack buffer overflow
if (len > 0 && data[0] == 'A') {
memcpy(buffer, data, len); // No bounds check
sink = buffer[0]; // Force use
}
// Vulnerability 2: Heap overflow
if (len > 1 && data[0] == 'B') {
char *heap = malloc(32);
memcpy(heap, data, len); // Overflow if len > 32
sink = heap[0]; // Force use before free
free(heap);
}
// Vulnerability 3: Use-after-free
if (len > 1 && data[0] == 'C') {
char *ptr = malloc(16);
free(ptr);
sink = ptr[0]; // UAF read (more reliable than write)
}
// Vulnerability 4: Double free
if (len > 1 && data[0] == 'D') {
char *ptr = malloc(16);
free(ptr);
free(ptr); // Double free
}
}
int main(int argc, char **argv) {
if (argc < 2) return 1;
FILE *f = fopen(argv[1], "rb");
if (!f) return 1;
fseek(f, 0, SEEK_END);
size_t len = ftell(f);
fseek(f, 0, SEEK_SET);
char *data = malloc(len + 1);
fread(data, 1, len, f);
fclose(f);
process_input(data, len);
free(data);
return 0;
}
EOF
# 2. Build with AFL++ instrumentation and ASan
mkdir -p bin
export CC=afl-clang-fast
export AFL_USE_ASAN=1
$CC -g -O0 -fno-omit-frame-pointer src/fuzz_target.c -o bin/fuzz_target_asan
# Build without sanitizer for GDB analysis comparison
$CC -g src/fuzz_target.c -o bin/fuzz_target_plain
# 3. Create seed corpus (seeds that will trigger crashes)
mkdir -p afl_input
python3 -c "import sys; sys.stdout.buffer.write(b'A' + b'X'*100)" > afl_input/seed_stack
python3 -c "import sys; sys.stdout.buffer.write(b'B' + b'X'*50)" > afl_input/seed_heap
python3 -c "import sys; sys.stdout.buffer.write(b'CX')" > afl_input/seed_uaf
python3 -c "import sys; sys.stdout.buffer.write(b'DX')" > afl_input/seed_double
echo -n "test" > afl_input/seed_normal
# 4. Run AFL++ fuzzing (run for a few minutes to generate crashes)
# Use tmux or screen for longer sessions
timeout 300 afl-fuzz -i afl_input -o afl_output -m none -- ./bin/fuzz_target_asan @@
# Check crashes found
ls -la afl_output/default/crashes/
# 5. Triage crashes with casr-afl
casr-afl -i afl_output/default -o afl_casrep -t 10 -j 4 -f -- ./bin/fuzz_target_asan @@
# 6. View clustered results
ls afl_casrep/
# cl1/ cl2/ cl3/ ... (each cluster = unique crash type)
# 7. Generate statistics (just pass the directory)
casr-cli afl_casrep/
# 8. Optional: Add GDB analysis for non-sanitizer crashes
casr-afl -i afl_output/default -o afl_casrep_gdb -f -- ./bin/fuzz_target_plain @@
Timeouts and Hangs Are Bugs Too
Why Timeouts Matter
- Denial of Service: A single malicious input causing 100% CPU for hours
- Algorithmic Complexity: O(n²) or O(n!) behavior with crafted input
- Deadlocks: Multithreaded code stuck waiting forever
- Resource Exhaustion: Memory growth without bounds
Creating a Hang-Prone Test Program
First, let's create a program that can hang to practice these techniques:
// ~/crash_analysis_lab/src/hang_test.c
// ~/crash_analysis_lab/src/hang_test.c
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
// Simulates algorithmic complexity attack
void slow_parse(char *input, int len) {
// O(n²) behavior - gets very slow with large input
for (int i = 0; i < len; i++) {
for (int j = 0; j < len; j++) {
if (input[i] == input[j]) {
usleep(100); // Simulate work
}
}
}
}
// Multiple infinite loop patterns based on input
void process_command(char *cmd) {
if (strncmp(cmd, "LOOPA", 5) == 0) {
printf("[*] Entering loop pattern A...\n");
while(1) { } // Pattern A
}
if (strncmp(cmd, "LOOPB", 5) == 0) {
printf("[*] Entering loop pattern B...\n");
for(;;) { } // Pattern B - different stack location
}
if (strncmp(cmd, "LOOPC", 5) == 0) {
printf("[*] Entering loop pattern C...\n");
volatile int spin = 1;
while(spin) { } // Pattern C
}
if (strncmp(cmd, "LOOP", 4) == 0) {
printf("[*] Entering default loop...\n");
while(1) { } // Default pattern
}
printf("[*] Command processed: %s\n", cmd);
}
// Recursive function that can stack overflow or hang
void recursive_parse(char *data, int depth) {
if (depth > 10000) return; // Safety limit
if (data[0] == 'R') {
recursive_parse(data, depth + 1);
}
}
int main(int argc, char **argv) {
char buffer[1024];
if (argc < 2) {
printf("Usage: %s <1|2|3> [input]\n", argv[0]);
printf(" 1 <input> - Slow O(n²) parsing\n");
printf(" 2 - Infinite loop (reads from stdin)\n");
printf(" LOOPA/LOOPB/LOOPC for different patterns\n");
printf(" 3 <input> - Deep recursion\n");
return 1;
}
int test = atoi(argv[1]);
switch(test) {
case 1:
if (argc < 3) return 1;
slow_parse(argv[2], strlen(argv[2]));
break;
case 2:
if (fgets(buffer, sizeof(buffer), stdin)) {
process_command(buffer);
}
break;
case 3:
if (argc < 3) return 1;
recursive_parse(argv[2], 0);
break;
}
printf("[*] Done\n");
return 0;
}
Build the hang test program:
cd ~/crash_analysis_lab
# Build without optimizations for clear stack traces
gcc -g -O0 src/hang_test.c -o hang_test
# Build with ASAN for timeout analysis
gcc -g -O0 -fsanitize=address -fno-omit-frame-pointer src/hang_test.c -o hang_test_asan
Collecting Stack Dumps from Hangs
cd ~/crash_analysis_lab
# Create a hang input
echo "LOOP" > crashes/hang_input.txt
# Run with timeout - program hangs and gets killed
timeout --signal=SIGABRT 10s ./hang_test 2 < crashes/hang_input.txt
# Output: "[*] Entering infinite loop..." then "timeout: the monitored command dumped core"
# The GDB batch approach doesn't work well for hangs because timeout kills
# the entire GDB process. Instead, use the attach method:
# Start the hang in background:
./hang_test 2 < crashes/hang_input.txt &
HANG_PID=$!
# Wait a moment for it to enter the loop
sleep 1
# Attach GDB and get backtrace:
sudo gdb -batch -p $HANG_PID \
-ex "bt" \
-ex "info registers" \
-ex "x/5i \$pc" \
-ex "detach" 2>&1 | tee crashes/hang_analysis.txt
# Example output:
#process_command (cmd=0x7fff499d9fd0 "LOOP\n") at src/hang_test.c:36
#36 while(1) { } // Default pattern
#0 process_command (cmd=0x7fff499d9fd0 "LOOP\n") at src/hang_test.c:36
#1 0x000064fc7f622532 in main (argc=2, argv=0x7fff499da508) at src/hang_test.c:70
# ...
#=> 0x64fc7f622375 <process_command+221>: nop
# 0x64fc7f622376 <process_command+222>: jmp 0x64fc7f622375 <process_command+221>
#
# The jmp-to-itself pattern confirms an infinite loop!
# Clean up the hung process
kill $HANG_PID 2>/dev/null
CASR Classification for Hangs
CASR is designed for crash analysis, not hang detection. It requires the program to actually crash (receive a signal like SIGSEGV or SIGABRT from within the program):
cd ~/crash_analysis_lab
mkdir -p casrep
# This does NOT work - timeout kills the process externally, CASR sees "no crash"
casr-san -o casrep/hang.casrep -- timeout 10s ./hang_test_asan 2 < crashes/hang_input.txt
# Error: Program terminated (no crash)
# For hangs, use the GDB attach method instead (shown above)
# CASR is best suited for actual crashes, not timeouts
Key insight: Hangs and timeouts are different from crashes:
- Crash: Program receives a signal (SIGSEGV, SIGABRT) due to internal error
- Hang: Program runs forever, must be killed externally
- CASR: Only analyzes crashes, not externally-killed processes
For hang analysis, use the GDB attach method shown in Method 1 above.
When to use CASR: Use it for actual crashes from the Day 1-2 test binaries:
cd ~/crash_analysis_lab
# CASR works great for actual crashes
casr-san -o casrep/stack_overflow.casrep -- ./vuln_asan 1 $(python3 -c "print('A'*200)")
cat casrep/stack_overflow.casrep | jq '.CrashSeverity'
# Output: { "Type": "EXPLOITABLE", "ShortDescription": "stack-buffer-overflow", ... }
Simple Hang Bucketing
When you have many timeouts from fuzzing, bucket by stack signature:
#!/bin/bash
# ~/crash_analysis_lab/bucket_hangs.sh
cd ~/crash_analysis_lab
mkdir -p hang_buckets
for hang in crashes/hang_*.txt; do
[ -f "$hang" ] || continue
# Start the program in background
./hang_test 2 < "$hang" &
pid=$!
sleep 0.3
# Get stack, strip addresses, keep only function names and line info
sig=$(sudo gdb -batch -p $pid -ex "bt 5" 2>&1 | \
grep "^#" | \
sed 's/0x[0-9a-f]*//g' | \
sed 's/cmd=[^ ]*/cmd=/g' | \
md5sum | cut -d' ' -f1)
kill -9 $pid 2>/dev/null
wait $pid 2>/dev/null
mkdir -p hang_buckets/$sig
cp "$hang" hang_buckets/$sig/
done
echo "Unique hang patterns:"
ls -1 hang_buckets/ | wc -l
Test the bucketing script:
cd ~/crash_analysis_lab
# Create multiple hang inputs
echo "LOOPA" > crashes/hang_a1.txt
echo "LOOPA" > crashes/hang_a2.txt
echo "LOOPB" > crashes/hang_b1.txt
echo "LOOPC" > crashes/hang_c1.txt
# Run bucketing
chmod +x bucket_hangs.sh
./bucket_hangs.sh
Infinite Loop Detection Patterns
When analyzing hangs interactively, GDB helps identify the specific loop pattern. The key is distinguishing between a program waiting for input (blocked in read()) versus an actual infinite loop (spinning CPU).
Common Mistake: Blocking vs Spinning
cd ~/crash_analysis_lab
# WRONG: Running without input - program blocks waiting for stdin
gdb ./hang_test
(gdb) run 2
# Press Ctrl+C...
# You'll see it's blocked in read(), NOT in an infinite loop:
# #0 __GI___libc_read () at read.c:26
# #1 _IO_file_underflow ()
# #5 fgets ()
# #6 main () at src/hang_test.c:69 <-- Waiting for input!
# This is NOT a hang - it's waiting for you to type something
Correct Approach: Provide Input First
cd ~/crash_analysis_lab
# Method 1: Use a pipe to provide input, then attach
echo "LOOP" | ./hang_test 2 &
HANG_PID=$!
sleep 0.5 # Let it enter the loop
# Now attach and analyze
sudo gdb -batch -p $HANG_PID \
-ex "bt" \
-ex "x/5i \$pc" \
-ex "detach" 2>&1
# Expected output shows we're IN the loop, not waiting for input:
# #0 0x0000555555555375 in process_command (cmd=...) at src/hang_test.c:36
# #1 0x000055555555551e in main () at src/hang_test.c:70
#
# => 0x555555555375 <process_command+221>: nop
# 0x555555555376 <process_command+222>: jmp 0x555555555375
#
# The jmp-to-itself pattern confirms an infinite loop!
kill $HANG_PID 2>/dev/null
# Method 2: Interactive GDB with input redirection
# First ensure the input file exists (created earlier in this section):
echo "LOOP" > crashes/hang_input.txt
gdb ./hang_test
(gdb) run 2 < crashes/hang_input.txt
# Now Ctrl+C will catch it in the actual loop
^C
(gdb) bt
# #0 process_command (cmd=0x7fffffffdfd0 "LOOP\n") at src/hang_test.c:36
# #1 main () at src/hang_test.c:70
Distinguishing Hang Types:
cd ~/crash_analysis_lab
# 1. Blocked on I/O (NOT a bug - waiting for input)
gdb ./hang_test
(gdb) run 2
^C
(gdb) bt
# Shows: read() -> fgets() -> main()
# PC is in libc read(), program is WAITING not SPINNING
(gdb) info proc status
# CPU time will be near zero - not consuming CPU
# 2. True infinite loop (BUG - spinning CPU)
echo "LOOP" | ./hang_test 2 &
PID=$!; sleep 1
sudo gdb -batch -p $PID -ex "info proc status" 2>&1 | grep -E "utime|stime"
# Shows high CPU time - actively spinning
kill $PID
# 3. Mutex deadlock (multithreaded programs)
# Would show multiple threads in __lll_lock_wait
(gdb) info threads
# Thread 1: __lll_lock_wait() <- waiting for lock
# Thread 2: __lll_lock_wait() <- also waiting = DEADLOCK
Testing Different Loop Patterns:
cd ~/crash_analysis_lab
# Each LOOP variant creates a loop at a different source line
# This tests whether your deduplication correctly groups them
for pattern in LOOPA LOOPB LOOPC LOOP; do
echo "=== Testing $pattern ==="
echo "$pattern" | ./hang_test 2 &
PID=$!
sleep 0.3
# Get the crash location
sudo gdb -batch -p $PID -ex "bt 2" 2>&1 | grep "process_command"
kill $PID 2>/dev/null
wait $PID 2>/dev/null
done
# Output shows different line numbers but same function:
# process_command at src/hang_test.c:23 (LOOPA)
# process_command at src/hang_test.c:27 (LOOPB)
# process_command at src/hang_test.c:32 (LOOPC)
# process_command at src/hang_test.c:36 (LOOP)
Identifying Algorithmic Hangs vs Infinite Loops:
cd ~/crash_analysis_lab
# Algorithmic hang (O(n²) with usleep - gets very slow with large input)
time timeout 5s ./hang_test 1 $(python3 -c "print('A'*100)")
# Completes in ~1-2 seconds
# real 0m1.6s
# user 0m0.05s <- Low CPU (usleep dominates)
time timeout 30s ./hang_test 1 $(python3 -c "print('A'*500)")
# Times out! 500 chars = 25x more iterations than 100 chars (O(n²))
# Would need ~40+ seconds to complete
# True infinite loop (never completes, high CPU)
echo "LOOP" | timeout 5s ./hang_test 2
# Always killed by timeout, prints "[*] Entering default loop..."
# Key differences when debugging:
# - Algorithmic: PC changes on each Ctrl+C, low-ish CPU if I/O bound
# - Infinite loop: PC stays at same instruction (jmp to itself), 100% CPU
# - I/O blocked: PC in read()/recv(), near-zero CPU
Algorithmic Complexity Attack Detection
cd ~/crash_analysis_lab
# Test O(n²) behavior with increasing input sizes
echo "Testing algorithmic complexity..."
for size in 100 200 400 800; do
input=$(python3 -c "print('A'*$size)")
echo -n "Size $size: "
time timeout 30s ./hang_test 1 "$input" 2>/dev/null
done
# You'll see execution time grow quadratically:
# Size 100: ~1 second
# Size 200: ~6 seconds
# Size 400: ~24 seconds
# Size 800: timeout (would be >64 seconds)
CASR Severity Classes
CASR classifies crashes into three main categories with 23 specific types:
EXPLOITABLE (High Severity):
-
SegFaultOnPc: Instruction pointer controlled by attacker
"ShortDescription": "SegFaultOnPc" // PC/IP register contains attacker-controlled value -
ReturnAv: Return address overwrite
"ShortDescription": "ReturnAv" // Return address corrupted, likely stack overflow -
BranchAv: Branch target controlled
"ShortDescription": "BranchAv" // Indirect jump/call to
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 195k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 192k | Updated | Claude CodeCodex | Verified | Install |