How to Profile a Slow .NET Endpoint

Last spring one of our payout endpoints started taking eleven seconds at the 95th percentile. Not always. Just often enough that support tickets trickled in and nobody could reproduce it on a laptop. The first three engineers who looked at it did the same thing I've watched people do for fifteen years: they opened the controller, stared at the code, and started guessing.

Guessing is the enemy. A slow endpoint is a measurement problem before it is a code problem, and the whole game is refusing to touch a single line until you know where the time actually goes. Here is how I run that hunt, in the order I run it.

Reproduce it before you touch a profiler

You cannot profile a ghost. The very first thing I want is a request I can fire on demand that is reliably slow, because a profiler pointed at a fast request tells you nothing except that fast requests are fast. In the payout case the slowness only showed up for merchants with more than about 400 settlement records, so a synthetic account with three test transactions looked perfectly healthy. We wasted a full afternoon on that.

Get the real shape of the input. Pull an anonymized copy of a production-sized payload, find an account that actually triggers the latency, and confirm you can hit the slow path from a script or a saved HTTP request. Write down the number before you start. "The endpoint is slow" is not a baseline. "GET /payouts/summary for merchant 88213 takes 9.4s cold, 6.1s warm, measured five times" is a baseline. Everything you do after this either moves that number or it doesn't.

Read the timings you already have

Before any tooling, look at what the server already tells you. If you have structured logging with request durations, you often get the answer for free. We log a coarse breakdown on every request, and it takes about twenty minutes to add if you don't have it:

using var _ = logger.BeginScope(new Dictionary<string, object>
{
    ["merchant_id"] = merchantId
});

var sw = Stopwatch.StartNew();
var settlements = await _repo.GetSettlementsAsync(merchantId, ct);
logger.LogInformation("stage=fetch elapsed_ms={Ms} count={Count}",
    sw.ElapsedMilliseconds, settlements.Count);

sw.Restart();
var summary = _calculator.Build(settlements);
logger.LogInformation("stage=calc elapsed_ms={Ms}", sw.ElapsedMilliseconds);

That is crude and I don't care, because crude data beats elegant assumptions. In our case the log line was blunt about it: fetch was 5,800ms and calc was 40ms. The math nobody wanted to do was already pointing at the database, not the C#. If you expose Server-Timing headers you can even see this breakdown in the browser dev tools, which is a nice way to hand the evidence to a skeptical teammate without asking them to trust you.

Decide whether you are CPU-bound or waiting

This is the fork in the road, and picking the wrong branch here is how people burn a week. Roughly everything falls into two buckets:

• You are burning CPU — tight loops, serialization, LINQ that allocates a landfill of objects, regex, crypto. Here a sampling CPU profiler is your friend and it will point at a hot method.
• You are waiting — on a database, an HTTP call, a lock, a connection pool, the thread pool itself. Here a CPU profiler shows almost nothing, because the CPU is idle while your request twiddles its thumbs. You need timing and tracing, not sampling.

The cheap test: watch CPU on the box while you hammer the slow endpoint. If one core pegs at 100%, you're CPU-bound. If CPU stays low and latency stays high, you're waiting on something. Our payout endpoint sat at maybe 8% CPU while taking six seconds. That alone told me the profiler most people reach for first — a CPU sampler — was going to be almost useless. The time was being spent somewhere the CPU wasn't.

Reach for dotnet-trace and dotnet-counters

The built-in dotnet diagnostics tools are underrated and they run anywhere, including a locked-down production container where you can't attach a GUI. I install them as global tools and keep them in my back pocket. Start with dotnet-counters to watch live vitals — thread pool queue length, GC pause, lock contention, requests per second — while traffic flows:

dotnet-counters monitor --process-id 1234 \
    System.Runtime Microsoft.AspNetCore.Hosting

If the thread pool queue length is climbing and GC looks fine, you probably have thread pool starvation, which is nearly always someone calling .Result or .Wait() on an async call and blocking a pool thread. I have found that exact bug in four different codebases. It is the single most common self-inflicted latency wound in .NET, and it gets worse under load, not better, which is why it never reproduces on your laptop. For a deeper capture, dotnet-trace records events you can open in a viewer and read the actual call durations, not just samples.

Assume the database until proven innocent

In web services the answer is the database far more often than anyone wants to admit. Not because databases are slow — because we ask them foolish questions. Once dotnet-counters confirmed we were waiting on IO, I turned on EF Core query logging with parameter values in a staging environment and read what was actually being sent to Postgres.

The query log is the most honest document in your entire system. It does not care about your intentions, your mental model of the ORM, or the clever abstraction you wrote. It shows you exactly what left the process, and it is frequently horrifying.

What I found was the classic: an N+1. The summary loaded a list of settlements, then for each settlement lazily loaded its line items, one round trip apiece. For a small merchant that's five queries. For merchant 88213 it was 431 queries, each taking about thirteen milliseconds of round-trip latency, which is where our six seconds went. The fix was a single .Include() to fetch line items in one join. Latency dropped from 6.1 seconds to 340 milliseconds. No algorithm changes, no caching, no new infrastructure. Just asking the database once instead of 431 times.

Confirm at the database, not just the app

When the query itself is slow rather than repeated, get the plan. In Postgres that means EXPLAIN (ANALYZE, BUFFERS); in SQL Server it's the actual execution plan. Read it for the two things that account for most sins: a sequential scan where you expected an index seek, and a wildly wrong row estimate, which usually means stale statistics or a query the planner can't reason about.

EXPLAIN (ANALYZE, BUFFERS)
SELECT s.id, s.amount, li.description
FROM settlements s
JOIN line_items li ON li.settlement_id = s.id
WHERE s.merchant_id = 88213
  AND s.settled_at >= '2026-01-01';

I've seen a missing index on a foreign key turn a 20ms query into a 2-second table scan the moment a table crossed a few million rows. Nothing in the application code changed; the table just got big enough that the shortcut the planner was taking stopped being cheap. This is why load matters and why a query that's fine in dev quietly rots in production. Measure it where the data lives.

Profile production carefully, not just staging

Staging lies. It has less data, cleaner caches, no noisy neighbors, and none of the connection-pool exhaustion you get at 3pm on a busy Tuesday. Some latency only exists in production, so you have to be willing to look there. The trick is doing it without becoming the outage you're investigating.

My rules for touching production: use sampling profilers over instrumenting ones because the overhead is bounded and predictable; capture short windows, thirty to sixty seconds, not a running-all-day trace that fills a disk; and never attach a debugger to a live process serving customers, because a breakpoint on a request thread freezes it and you'll take the service down while trying to make it faster. Continuous, low-overhead profilers that run at a small sampling rate earn their keep once you're past a certain scale — being able to click on a spike from three hours ago and see the flame graph beats trying to catch it live.

Measure the fix or you didn't fix it

Here is the discipline that separates engineers from people who move code around until the symptom disappears: you re-measure against the exact baseline you wrote down at the start, under the same load, and you keep the two numbers next to each other. 6.1 seconds to 340 milliseconds. Write it in the pull request. If you can't state the before and after as numbers, you optimized a feeling.

And watch for the whack-a-mole. Fixing the N+1 sometimes just moves the bottleneck one layer down — now the single big join is the slow part, or now you're saturating the connection pool because each request is faster and you're doing more of them. That's fine. It means you fixed the first thing. Profiling is iterative: you kill the biggest cost, re-measure, find the new biggest cost, and you stop when the endpoint is fast enough for the people using it, not when it's theoretically perfect.

Conclusion

The uncomfortable truth is that most slow endpoints are slow for boring, findable reasons — a chatty ORM, a blocked thread, a missing index — and almost never for the exotic reason the room wants to debate. The engineers who are fast at this aren't smarter. They're just more stubborn about measuring before they touch anything, and they resist the ego pull of the clever fix. So the next time an endpoint drags, don't open the controller. Open the timing data, and let it tell you where to look. Your first guess is wrong more often than you'd like, and the profiler will say so without sparing your feelings.