Profiling Applications: Finding Bottlenecks in Go, Python, and Node.js

Your application is slow. Requests that should take 50ms are taking 800ms. Memory usage is creeping up every hour and you’re not sure why. You’ve already tried the obvious things — added an index here, reduced a payload there — but the problem persists.

The instinct is to scan the code and optimize what looks expensive. This almost always fails. The things that look slow often aren’t, and the actual bottleneck is frequently somewhere mundane: a string format call inside a tight loop, a missing cache, a goroutine that never exits, a JSON parse that happens on every request.

Profiling is the tool that ends the guessing. It tells you, with evidence, exactly where time and memory are actually going. This guide covers systematic profiling for Go, Python, and Node.js — the commands to run, the output to read, and how to turn a profile into a fix.

Why Profiling Beats Guessing

Donald Knuth said it: “Premature optimization is the root of all evil.” The less-cited corollary is that late optimization — done after you understand the actual problem — is essential engineering. The two rules together produce the correct workflow: don’t optimize until you measure, then optimize exactly what the measurement shows.

What Profiling Actually Reveals

A profiler is not just a speedometer. Depending on the type, it reveals:

• CPU hotspots — which functions consume the most processor time
• Memory allocation patterns — which code paths allocate, and how much
• Memory leaks — allocations that grow without bound over time
• Lock contention — threads or goroutines blocking each other on mutexes
• I/O wait — time spent blocked on disk reads, network calls, or database queries
• GC pressure — garbage collection pauses caused by excessive allocations
• Event loop stalls — in Node.js, synchronous work that starves async handlers
• Goroutine/thread leaks — concurrency primitives that pile up and never exit

Types of Profiling

Development vs Production Profiling

Profiling in development is low-risk: run a benchmark, collect a profile, iterate. The tradeoffs appear in production.

CPU profiling has real overhead — typically 5–10% for sampling profilers, more for instrumentation-based ones. Memory profiling is heavier still. The strategy for production is usually:

1. Use sampling profilers (py-spy, 0x, Pyroscope) which have sub-1% overhead
2. Profile under actual load, not idle — many bugs only appear under concurrency
3. Enable profiling endpoints but protect them behind authentication or a firewall rule
4. Collect short profiles (10–30 seconds) rather than running continuously at full detail

The Profiling Workflow

1. Reproduce the problem (benchmark, load test, or manual trigger)
         ↓
2. Collect a profile during the slow period
         ↓
3. Identify the hotspot (wide bars in flame graph, top functions by time)
         ↓
4. Read the actual code — does this hotspot make sense?
         ↓
5. Form a hypothesis and make the fix
         ↓
6. Measure again — did it actually improve?
         ↓
7. Check for regressions in other areas

Never skip step 6. Performance work that skips the after-measurement produces surprising results — often the fix didn’t help, sometimes it made things worse.

Go Profiling: pprof

Go ships with one of the best profiling ecosystems of any language. The pprof package is built into the standard library, the tooling is mature, and flame graphs are a go tool pprof -http flag away.

Enabling pprof in Your Application

The fastest way to add profiling to an HTTP service is a single blank import:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

import (
    "net/http"
    _ "net/http/pprof"  // registers /debug/pprof/ handlers automatically
)

func main() {
    // Your application setup...

    // Run the pprof HTTP server (separate port from your main server)
    go func() {
        log.Println(http.ListenAndServe("localhost:6060", nil))
    }()

    // ... rest of main
}

This exposes the following endpoints:

http://localhost:6060/debug/pprof/          — index page
http://localhost:6060/debug/pprof/profile   — 30-second CPU profile
http://localhost:6060/debug/pprof/heap      — current heap snapshot
http://localhost:6060/debug/pprof/goroutine — all goroutine stack traces
http://localhost:6060/debug/pprof/block     — goroutine blocking events
http://localhost:6060/debug/pprof/mutex     — mutex contention
http://localhost:6060/debug/pprof/trace     — execution trace

Security note: Bind to localhost:6060, not 0.0.0.0:6060. The pprof endpoint exposes internal memory contents and should never be public-facing.

Collecting Profiles with go tool pprof

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# 30-second CPU profile (application must be under load during collection)
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

# Current heap snapshot (shows live objects and their allocation sites)
go tool pprof http://localhost:6060/debug/pprof/heap

# All goroutine stacks (great for detecting goroutine leaks)
go tool pprof http://localhost:6060/debug/pprof/goroutine

# Goroutine blocking profile (mutexes, channels, syscalls)
go tool pprof http://localhost:6060/debug/pprof/block

# Mutex contention (which mutexes are contested, and by whom)
go tool pprof http://localhost:6060/debug/pprof/mutex

To enable block and mutex profiling, you need to set the sampling rate at startup:

1
2
3
4
5
6
7

import "runtime"

func main() {
    runtime.SetBlockProfileRate(1)     // profile every blocking event
    runtime.SetMutexProfileFraction(1) // profile every mutex contention
    // ...
}

Interactive pprof Commands

After running go tool pprof, you enter an interactive shell:

(pprof) top
Showing nodes accounting for 2.31s, 94.69% of 2.44s total
      flat  flat%   sum%        cum   cum%
     1.23s 50.41% 50.41%      1.23s 50.41%  encoding/json.(*encodeState).marshal
     0.54s 22.13% 72.54%      1.77s 72.54%  main.processRequests
     0.31s 12.70% 85.25%      0.31s 12.70%  runtime.mallocgc

Key columns:

• flat — time spent in this function itself (not callees)
• flat% — percentage of total profile time
• cum — cumulative time including all functions this one calls
• cum% — cumulative percentage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Top by cumulative time (includes callee costs — shows the call chain)
(pprof) top -cum

# Annotated source for a specific function
(pprof) list processRequests

# Open interactive flame graph in browser (requires graphviz)
(pprof) web

# Open web UI with full flame graph, call graph, and source view
go tool pprof -http=:8080 http://localhost:6060/debug/pprof/profile?seconds=30

Reading Flame Graphs

A flame graph shows the call stack over time. The x-axis is not time — it’s alphabetical ordering within each stack level. The width of each bar represents the proportion of samples where that function was on the stack.

What to look for:

• Wide, flat bars near the top — functions that consume a lot of time and don’t call deeper. These are your hotspots.
• Tall narrow spikes — deep call chains that are individually fast but called frequently.
• Unexpected functions — runtime.mallocgc, runtime.gcBgMarkWorker, JSON parsers in hot paths.

Writing and Profiling Benchmarks

For library code or functions you can call in isolation, Go’s benchmark framework integrates directly with pprof:

1
2
3
4
5
6
7
8

// mypackage_test.go
func BenchmarkProcessOrder(b *testing.B) {
    order := generateTestOrder()
    b.ResetTimer() // don't count setup time
    for i := 0; i < b.N; i++ {
        ProcessOrder(order)
    }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# Run benchmarks and collect CPU and memory profiles
go test -bench=BenchmarkProcessOrder -benchmem \
    -cpuprofile=cpu.prof \
    -memprofile=mem.prof \
    -count=3

# Analyze with pprof
go tool pprof cpu.prof
go tool pprof mem.prof

# Or open the web UI directly
go tool pprof -http=:8080 cpu.prof

The -benchmem flag adds allocation counts per operation to the benchmark output:

BenchmarkProcessOrder-8    142857    8432 ns/op    2048 B/op    31 allocs/op

That 31 allocs/op is often the key number. Reducing allocations in hot paths directly reduces GC pressure.

The Execution Tracer

For goroutine scheduling issues, GC pause timing, and network I/O blocking, the trace tool gives you a timeline view that pprof can’t:

1
2
3
4
5

# Collect 5 seconds of trace data
curl -o trace.out http://localhost:6060/debug/pprof/trace?seconds=5

# Open the trace viewer
go tool trace trace.out

The trace viewer shows goroutine activity, GC events, syscall blocks, and the GOMAXPROCS scheduler. Look for:

• Long GC pauses (STW — stop the world)
• Goroutines stuck in syscall (gray blocks)
• Goroutines waiting on channel or mutex (purple blocks)
• Goroutine count growing over time (leak indicator)

Common Go Performance Issues Found with pprof

fmt.Sprintf in hot paths — fmt.Sprintf uses reflection and allocates on every call. In a function called millions of times, this adds up.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// Slow — allocates on every call
key := fmt.Sprintf("user:%d:session:%s", userID, sessionID)

// Fast — no allocation
var buf strings.Builder
buf.WriteString("user:")
buf.WriteString(strconv.Itoa(userID))
buf.WriteString(":session:")
buf.WriteString(sessionID)
key := buf.String()

Repeated JSON parsing — decoding the same schema on every request:

1
2
3
4
5
6
7
8

// Bad — allocates a new struct on every request and uses reflection
func handler(w http.ResponseWriter, r *http.Request) {
    var req MyRequest
    json.NewDecoder(r.Body).Decode(&req)
    // ...
}

// Consider: easyjson, jsoniter, or sonic for schemas decoded in hot paths

sync.Mutex where sync.RWMutex belongs — if many goroutines read and few write, a read-write mutex dramatically reduces contention:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

// If reads dominate writes
var mu sync.RWMutex
var cache map[string]Item

func get(key string) Item {
    mu.RLock()         // multiple goroutines can hold RLock simultaneously
    defer mu.RUnlock()
    return cache[key]
}

func set(key string, item Item) {
    mu.Lock()
    defer mu.Unlock()
    cache[key] = item
}

Goroutine leaks — the goroutine profile is your first signal. If goroutine count grows monotonically with traffic, you have a leak. Common causes: goroutines waiting on channels that never receive, or HTTP clients without timeouts.

Small repeated allocations — use sync.Pool to reuse objects:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

var bufPool = sync.Pool{
    New: func() interface{} {
        return new(bytes.Buffer)
    },
}

func process(data []byte) string {
    buf := bufPool.Get().(*bytes.Buffer)
    buf.Reset()
    defer bufPool.Put(buf)
    // ... use buf
    return buf.String()
}

Continuous Profiling with Pyroscope

For always-on profiling in production, Grafana Pyroscope (open source) integrates directly with Go’s pprof and adds a time-series dimension — you can query “what was the CPU profile at 2:47 AM when the latency spike happened?”

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

import "github.com/grafana/pyroscope-go"

func main() {
    pyroscope.Start(pyroscope.Config{
        ApplicationName: "myapp",
        ServerAddress:   "http://pyroscope:4040",
        ProfileTypes: []pyroscope.ProfileType{
            pyroscope.ProfileCPU,
            pyroscope.ProfileAllocObjects,
            pyroscope.ProfileAllocSpace,
            pyroscope.ProfileInuseObjects,
            pyroscope.ProfileInuseSpace,
        },
    })
    // ...
}

Python Profiling

Python’s reputation as “slow” is often a distraction. CPython is not the fastest runtime, but in practice the bottleneck is almost always a specific piece of code rather than the language as a whole — an O(n²) loop, a missing cache, an ORM generating N+1 queries. Profilers tell you which one.

cProfile: The Built-in CPU Profiler

cProfile is deterministic (records every function call) and part of the standard library. The overhead is significant for very tight loops but acceptable for most applications:

1
2
3
4
5

# Profile a script from the command line, sorted by cumulative time
python -m cProfile -s cumtime myscript.py

# Save profile data to a file for later analysis
python -m cProfile -o output.prof myscript.py

To profile specific code programmatically:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

import cProfile
import pstats

profiler = cProfile.Profile()
profiler.enable()

# --- code to profile ---
result = process_large_dataset(data)
# --- end profiled section ---

profiler.disable()
profiler.dump_stats('output.prof')

Key columns in cProfile output:

Sort by tottime to find where time is actually spent; sort by cumtime to follow the call chain down.

pstats: Analyzing Saved Profiles

1
2
3
4
5
6
7

import pstats

p = pstats.Stats('output.prof')
p.sort_stats('cumulative')
p.print_stats(20)          # top 20 functions
p.print_callers('mymodule.process')  # what calls this function?
p.print_callees('mymodule.process')  # what does this function call?

snakeviz: Visual Flame Graph for cProfile

snakeviz turns a .prof file into a browser-based icicle chart — far easier to read than the text output:

1
2

pip install snakeviz
snakeviz output.prof

This opens a browser with a clickable icicle chart. Wider sections are more time-consuming. Click any bar to zoom in on that subtree.

line_profiler: Line-by-Line Profiling

cProfile tells you which function is slow. line_profiler tells you which line of that function is slow. It’s the tool you reach for once cProfile has narrowed the search:

1

pip install line_profiler

Decorate the functions you want to profile:

1
2
3
4
5
6
7
8
9

# myscript.py
@profile  # this decorator is injected by kernprof at runtime
def process_records(records):
    result = []
    for record in records:
        parsed = json.loads(record)         # line 6
        cleaned = clean(parsed)             # line 7
        result.append(cleaned)              # line 8
    return result

1

kernprof -l -v myscript.py

Output:

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
     6     10000      8432312    843.2     71.2%          parsed = json.loads(record)
     7     10000      2891044    289.1     24.4%          cleaned = clean(parsed)
     8     10000       510233     51.0      4.3%          result.append(cleaned)

In this example, 71% of the time is in json.loads. If these records share the same schema, you can pre-compile a validator or switch to orjson for a 3–10x speedup.

Practical comparison — list comprehension vs generator:

1
2
3
4
5
6
7
8
9

@profile
def sum_squares_list(n):
    # Builds entire list in memory before summing
    return sum([x * x for x in range(n)])

@profile
def sum_squares_gen(n):
    # Generator: yields one value at a time, no list allocation
    return sum(x * x for x in range(n))

For large n, line_profiler will show the list comprehension spending measurable time on memory allocation that the generator avoids entirely.

memory_profiler: Finding Memory Usage and Leaks

1

pip install memory_profiler

Line-by-line memory usage:

1
2
3
4
5
6
7
8
9

from memory_profiler import profile

@profile
def load_and_process(filepath):
    with open(filepath) as f:
        data = f.read()          # line 5
    records = data.split('\n')   # line 6
    processed = [transform(r) for r in records]  # line 7
    return processed

1

python -m memory_profiler myscript.py

Output:

Line #    Mem usage    Increment   Line Contents
================================================
     5    45.3 MiB    45.3 MiB    data = f.read()
     6    73.1 MiB    27.8 MiB    records = data.split('\n')
     7   198.4 MiB   125.3 MiB    processed = [transform(r) for r in records]

For memory-over-time profiling (catching leaks in long-running services):

1
2

mprof run python myscript.py
mprof plot               # generates a matplotlib chart of memory over time

Common Python memory issues:

• References in module-level lists — RESULTS = [] with RESULTS.append(thing) in a function that’s called repeatedly
• Circular references — Python’s GC handles these, but they create garbage collection pauses
• Unclosed file handles — use with statements; files not closed under load accumulate OS-level descriptors
• DataFrame copies — pandas operations that seem in-place (.apply(), boolean indexing) silently copy data

py-spy: Sampling Profiler for Production

py-spy is a sampling profiler that attaches to a running Python process without any code changes or restart. It uses ptrace (Linux) or equivalent, adding negligible overhead to the target process:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

pip install py-spy

# Live top-like view of function time in a running process
py-spy top --pid 1234

# Record a 30-second flame graph SVG
py-spy record -o profile.svg --pid 1234 --duration 30

# Profile from the start (for startup issues)
py-spy record -o profile.svg -- python myscript.py

The resulting SVG opens in any browser and is interactive — click to zoom, hover for details. This is the tool for production performance investigations where you can’t stop the process or add instrumentation.

Note: On Linux, profiling processes you don’t own requires sudo. In containers, you may need --pid-mode=host or SYS_PTRACE capability.

Profiling Async Code

Standard cProfile misattributes time in asyncio — it records the coroutine scheduling overhead rather than where the coroutines actually spend time.

yappi (Yet Another Python Profiler) handles async correctly:

1

pip install yappi

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

import asyncio
import yappi

async def main():
    yappi.start()
    await your_async_code()
    yappi.stop()

    stats = yappi.get_func_stats()
    stats.sort("tsub")  # sort by total time excluding callees
    stats.print_all()

asyncio.run(main())

For quick debugging of async slowness, enable asyncio’s debug mode:

1

PYTHONASYNCIODEBUG=1 python myscript.py

This logs coroutines that take more than 100ms and enables asyncio.get_event_loop().slow_callback_duration warnings.

Common Python Performance Issues

String concatenation in loops — += on strings creates a new object every iteration:

1
2
3
4
5
6
7

# O(n²) — don't do this
result = ""
for item in items:
    result += str(item) + ","

# O(n) — use join
result = ",".join(str(item) for item in items)

List membership tests — in on a list is O(n); on a set it’s O(1):

1
2
3
4
5
6

# Slow if valid_ids is large
if user_id in valid_ids_list:

# Fast
valid_ids_set = set(valid_ids_list)  # build once
if user_id in valid_ids_set:

Repeated attribute lookups in tight loops:

1
2
3
4
5
6
7
8

# Slower — resolves self.cache on every iteration
for item in items:
    self.cache[item.key] = item.value

# Faster — resolve once, use local variable
cache = self.cache
for item in items:
    cache[item.key] = item.value

N+1 queries — the most common database performance problem in ORMs:

1
2
3
4
5
6
7

# Django: N+1 — runs one query per user to get their profile
users = User.objects.all()
for user in users:
    print(user.profile.bio)  # triggers a query each time

# Fix: use select_related for foreign key traversals
users = User.objects.select_related('profile').all()

GIL contention — Python’s Global Interpreter Lock means CPU-bound threads don’t actually run in parallel. Use multiprocessing for CPU-bound work, asyncio for I/O-bound.

Node.js Profiling

Node.js’s event loop model makes performance problems manifest differently than in Go or Python. The most common Node.js performance issue is not “this function is slow” but “this function is blocking the event loop.” Understanding the distinction shapes which tools you reach for.

Built-in V8 Profiler

Node.js includes the V8 profiler. Enable it with a flag:

1
2
3
4
5

# Run app with profiling enabled — generates isolate-*.log
node --prof app.js

# Process the raw log into human-readable output
node --prof-process isolate-0xNNNNNNNN-v8.log > processed.txt

The processed output shows a breakdown by JavaScript, C++, and garbage collection time, along with a summary of the heaviest functions.

Chrome DevTools: The Most Powerful Node.js Profiler

For interactive profiling with a visual UI, Node.js’s --inspect flag connects to Chrome DevTools:

1
2
3
4
5

# Start with inspector (listens on 127.0.0.1:9229 by default)
node --inspect app.js

# Or pause at the first line (useful for startup profiling)
node --inspect-brk app.js

Then open chrome://inspect in Chrome and click “inspect” on your Node process. This gives you:

• Profiler tab — CPU profiles and allocation timelines. Click “Start” → apply load → “Stop” to get a flame chart
• Memory tab — heap snapshots. Take a snapshot, apply some load, take another, and compare to find objects that were allocated but not freed
• Performance tab — timeline of CPU, memory, and V8 GC events

For heap snapshot comparison, the workflow is:

1. Take baseline snapshot
2. Exercise the feature that might be leaking
3. Force GC (click the garbage can icon)
4. Take a second snapshot
5. Switch the dropdown to “Objects allocated between Snapshot 1 and Snapshot 2”

Any objects still alive that were allocated during step 2 are candidates for leaks.

clinic.js: The Comprehensive Node.js Profiling Toolkit

clinic.js is the most practical first-stop for Node.js profiling. Its doctor command diagnoses the type of performance problem before you go digging:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

npm install -g clinic

# Diagnose — tells you what kind of problem you have
clinic doctor -- node app.js

# Flame graph — where is CPU time going?
clinic flame -- node app.js

# Async operation visualization — what's blocking async work?
clinic bubbleprof -- node app.js

# Heap allocation profiling — where is memory coming from?
clinic heapprofiler -- node app.js

After running clinic doctor, send load to your app with wrk or hey, then press Ctrl+C. clinic generates an HTML report that says things like “This process has a high event loop delay indicating CPU bound work” or “This process has growing memory indicating a memory leak.”

Start with clinic doctor. It saves time by ruling out whole categories of problems before you look at flame graphs.

0x: Flame Graphs for Node.js

0x is a single-command flame graph tool:

1
2
3
4

npm install -g 0x

# Profile app — generates an HTML flame graph
0x app.js

Send load to the running application, then Ctrl+C. 0x generates an interactive HTML file you can open in a browser. It’s also what clinic flame uses under the hood.

Event Loop Lag Monitoring

The event loop blocking problem is specific to Node.js: if any synchronous code runs longer than ~10ms, it delays all other requests because they’re waiting for the event loop to free up.

Detecting event loop lag in a running application:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

// Simple lag monitor — measures how late setInterval fires
const INTERVAL = 500; // ms
let lastCheck = Date.now();

setInterval(() => {
    const now = Date.now();
    const lag = now - lastCheck - INTERVAL;
    if (lag > 50) {
        console.warn(`Event loop lag: ${lag}ms`);
    }
    lastCheck = now;
}, INTERVAL);

For diagnosing what is blocking the loop, clinic doctor is the fastest path. For programmatic detection, the blocked-at package can pinpoint the blocking call:

1

npm install blocked-at

1
2
3
4

const blocked = require('blocked-at');
blocked((time, stack) => {
    console.log(`Blocked for ${time}ms, operation started here:`, stack);
}, { threshold: 100 }); // report blocks longer than 100ms

Heap Dumps on Demand

For memory leaks in long-running processes, trigger a heap dump on SIGUSR2:

1
2
3
4
5
6
7
8

const heapdump = require('heapdump');

process.on('SIGUSR2', () => {
    const filename = `/tmp/heapdump-${Date.now()}.heapsnapshot`;
    heapdump.writeSnapshot(filename, (err, filename) => {
        if (!err) console.log(`Heap dump written to ${filename}`);
    });
});

1
2

# Trigger heap dump without restarting the process
kill -USR2 <pid>

Load the .heapsnapshot file in Chrome DevTools (Memory tab → Load).

Common Node.js Performance Issues

Synchronous fs operations in request handlers — readFileSync, writeFileSync, and existsSync block the entire event loop:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// Bad — blocks all other requests while reading
app.get('/config', (req, res) => {
    const config = fs.readFileSync('./config.json', 'utf8'); // blocks event loop
    res.json(JSON.parse(config));
});

// Good — async, allows other requests to proceed
app.get('/config', async (req, res) => {
    const config = await fs.promises.readFile('./config.json', 'utf8');
    res.json(JSON.parse(config));
});

JSON parsing of large objects on every request:

1
2
3
4
5

// If config.json is large and unchanging, parse it once at startup
const config = JSON.parse(fs.readFileSync('./config.json', 'utf8'));

// Don't re-parse on every request
app.get('/config', (req, res) => res.json(config));

Memory leaks from closures and global caches:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

// Bad — requestCache grows forever, no eviction
const requestCache = new Map();
app.get('/data/:id', async (req, res) => {
    if (!requestCache.has(req.params.id)) {
        requestCache.set(req.params.id, await fetchData(req.params.id));
    }
    res.json(requestCache.get(req.params.id));
});

// Good — use an LRU cache with a size limit
const LRU = require('lru-cache');
const cache = new LRU({ max: 1000, ttl: 60_000 });

Event emitter listener leaks — adding listeners without removing them:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

// Bad — adds a new listener on every request
app.get('/stream', (req, res) => {
    emitter.on('data', (chunk) => res.write(chunk)); // never removed!
});

// Good — remove listener when request ends
app.get('/stream', (req, res) => {
    const handler = (chunk) => res.write(chunk);
    emitter.on('data', handler);
    req.on('close', () => emitter.removeListener('data', handler));
});

Not streaming large responses:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

// Bad — buffers entire file in memory
app.get('/download', (req, res) => {
    const data = fs.readFileSync('./bigfile.bin'); // 500MB in memory
    res.send(data);
});

// Good — stream directly to response
app.get('/download', (req, res) => {
    fs.createReadStream('./bigfile.bin').pipe(res);
});

Profiling Web Applications Under Load

You cannot profile a problem you cannot reproduce. If an API is slow only under concurrent traffic, profiling it while idle tells you nothing.

Load Testing to Trigger the Problem

wrk — fast HTTP benchmarking:

1
2
3
4
5

# 4 threads, 100 concurrent connections, 30 seconds
wrk -t4 -c100 -d30s http://localhost:8080/api/endpoint

# With a Lua script for custom request bodies
wrk -t4 -c100 -d30s -s post.lua http://localhost:8080/api/endpoint

hey — simple, portable, shows latency percentiles:

1
2
3
4
5

# 10,000 requests, 50 concurrent
hey -n 10000 -c 50 http://localhost:8080/api

# Duration-based
hey -z 30s -c 50 http://localhost:8080/api

k6 — scriptable load testing with JavaScript:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

// loadtest.js
import http from 'k6/http';
import { check } from 'k6';

export const options = {
    vus: 50,
    duration: '30s',
    thresholds: {
        http_req_duration: ['p(95)<200'], // 95th percentile under 200ms
    },
};

export default function () {
    const res = http.get('http://localhost:8080/api/endpoint');
    check(res, { 'status 200': (r) => r.status === 200 });
}

1

k6 run loadtest.js

Continuous Profiling in Production

Grafana Pyroscope — open-source always-on profiling:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# docker-compose.yml addition
services:
  pyroscope:
    image: grafana/pyroscope:latest
    ports:
      - "4040:4040"
    volumes:
      - pyroscope-data:/data

volumes:
  pyroscope-data:

Pyroscope supports Go (native pprof integration), Python (py-spy), Java (async-profiler), and Node.js. The key advantage over ad-hoc profiling: when an incident occurs at 3 AM, the profile data is already there.

In Grafana, add Pyroscope as a data source and you can correlate a CPU spike in your metrics dashboard directly with the flame graph from that moment.

eBPF-based profiling — tools like Pixie and Polar Signals profile all languages at the kernel level without any code instrumentation. They’re particularly valuable for polyglot environments where per-language profilers are impractical.

Database Query Profiling

The application profiler often points at a database call that’s taking 400ms. The next question is which query, and the answer requires database-side tooling.

PostgreSQL — EXPLAIN ANALYZE:

1
2
3
4
5
6
7
8

-- Shows actual vs estimated row counts, actual time, index usage
EXPLAIN (ANALYZE, BUFFERS, FORMAT TEXT)
SELECT u.*, p.bio
FROM users u
JOIN profiles p ON p.user_id = u.id
WHERE u.created_at > NOW() - INTERVAL '7 days'
ORDER BY u.created_at DESC
LIMIT 100;

Look for: Seq Scan on large tables (usually needs an index), actual rows far from rows= estimate (stale statistics, run ANALYZE), and high Buffers: shared hit/read ratios.

pg_stat_statements — top slow queries over time:

1
2
3
4
5

-- Enable in postgresql.conf: shared_preload_libraries = 'pg_stat_statements'
SELECT query, calls, total_exec_time, mean_exec_time, rows
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 10;

MySQL slow query log:

1
2
3
4
5

# my.cnf
slow_query_log = 1
slow_query_log_file = /var/log/mysql/slow.log
long_query_time = 1      # log queries over 1 second
log_queries_not_using_indexes = 1

Redis — MONITOR and SLOWLOG:

1
2
3
4
5
6
7
8

# Live stream of all commands (use with care in production — high overhead)
redis-cli MONITOR

# Last 10 slow commands
redis-cli SLOWLOG GET 10

# Set threshold (microseconds) — default 10000 (10ms)
redis-cli CONFIG SET slowlog-log-slower-than 5000

Distributed Tracing for Microservices

When a request fans out across multiple services, profiling a single service won’t explain total latency. Distributed tracing instruments the full request path.

OpenTelemetry is now the standard instrumentation layer — instrument once, send to any compatible backend:

1
2

# Python example
pip install opentelemetry-sdk opentelemetry-exporter-otlp

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://jaeger:4317"))
)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

def process_order(order_id):
    with tracer.start_as_current_span("process_order") as span:
        span.set_attribute("order.id", order_id)
        # ... work happens here

Send traces to Jaeger, Tempo (Grafana stack), or Zipkin. In the trace waterfall view, the slow service stands out as the longest bar — you then profile that service to find the internal bottleneck.

The Profiling Workflow in Practice: A Worked Example

Here’s a concrete example using Python. The same workflow applies to Go and Node.js with the tools described above.

The problem: An API endpoint that aggregates user activity is returning in ~2 seconds. Business requirement is under 200ms.

Step 1: Reproduce with a load test

1

hey -n 100 -c 5 http://localhost:8000/api/user-summary

Output confirms the problem: p50=1923ms, p99=3102ms.

Step 2: Collect a profile

1
2

# Profile the running application
py-spy record -o profile.svg --pid $(pgrep -f "python manage.py") --duration 30

While py-spy runs, apply load with hey.

Step 3: Identify the hotspot

Opening profile.svg reveals that 73% of time is in a function called get_user_summary → get_recent_events → Event.objects.filter(...). The ORM call is consuming most of the request time.

Step 4: Read the code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

def get_user_summary(user_id):
    user = User.objects.get(id=user_id)
    events = Event.objects.filter(user=user).order_by('-created_at')

    summary = {}
    for event in events:                         # iterates all events
        if event.type not in summary:
            summary[event.type] = []
        summary[event.type].append(event)        # loads every event into memory

    return {
        k: len(v) for k, v in summary.items()   # then just counts them
    }

The problem is visible now: the function loads every event for the user into memory (potentially thousands of rows) and then just counts them by type. The database could do this aggregation in a single query.

Step 5: Fix

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

from django.db.models import Count

def get_user_summary(user_id):
    # Single aggregation query — database does the work
    return dict(
        Event.objects.filter(user_id=user_id)
                     .values('type')
                     .annotate(count=Count('id'))
                     .values_list('type', 'count')
    )

Step 6: Measure again

1

hey -n 100 -c 5 http://localhost:8000/api/user-summary

p50=18ms, p99=41ms. The fix worked — 100x improvement by eliminating the N×row memory load.

Step 7: Add a regression test

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# tests/test_performance.py
import time

def test_user_summary_performance(client, user_with_1000_events):
    start = time.monotonic()
    response = client.get(f'/api/user-summary')
    elapsed = time.monotonic() - start

    assert response.status_code == 200
    assert elapsed < 0.1, f"Response took {elapsed:.3f}s, expected <100ms"

Add this to CI. Future regressions in this endpoint will fail the test before they reach production.

Checklist: Profiling a Slow Application

Before you start:

• Can you reproduce the slowness? What’s the exact scenario (concurrent users, data size, specific endpoint)?
• Do you have a baseline measurement? (response time, memory usage, CPU%)
• Do you know which part of the system is slow (application, database, network, external API)?

Collecting profiles:

• Go: Is pprof enabled? Are you profiling under realistic load?
• Python: Start with py-spy (no restart needed) or cProfile + snakeviz
• Node.js: Run clinic doctor first to identify the problem type

Reading profiles:

• CPU profile: what function has the highest flat%? What’s widest in the flame graph?
• Memory profile: where is the allocation growth? Is any allocation monotonically increasing?
• Goroutine/thread profile: is the count growing? Are goroutines stuck waiting?

After fixing:

• Did you measure after the fix? (Before/after benchmark)
• Is the fix in all environments (dev/staging/prod)?
• Do you have a performance test that will catch this regression?
• Did you check whether the fix introduced new problems elsewhere?

Summary

Performance work without measurement is folklore. With the right profiler, a day of guessing becomes an hour of evidence: you see exactly which function allocates too much, which mutex is contested, which database query scans a million rows, which event loop callback blocks for 500ms.

The tools in this guide — Go’s pprof, Python’s cProfile/py-spy/line_profiler, and Node.js’s clinic.js/Chrome DevTools — are all available today, most for free, and all give you the same fundamental capability: evidence rather than intuition.

Pick the profiler for your language. Run it under real load. Read the flame graph. Find the wide bar. Read the code. Fix it. Measure again.

That’s the whole loop. Everything else is details.