The first time an auditor asked me to prove who changed a customer's credit limit on a specific Tuesday eight months earlier, I learned exactly how good our audit log was. It was not good. We had application logs that had already rotated out of retention, a database with no history table, and a Slack thread where someone said "I think it was the batch job." That afternoon cost us three engineers a full day and a very uncomfortable email to the auditor promising we would do better.
An audit log is one of those systems nobody thanks you for when it works and everybody blames you for when it doesn't. It is also, in a regulated shop, not optional. So it is worth building properly the first time, because retrofitting one onto a live payments platform is a special kind of misery I would not wish on anyone.
What an audit log actually is
Let me be precise, because the word gets abused. An audit log is an append-only record of who did what to which resource, when, and from where. It is not your application log. It is not your metrics pipeline. It is not a debugging tool you grep when a customer complains. Those things are useful, but they are lossy by design and they get thrown away on a schedule. An audit log is evidence, and evidence has different rules.
The distinction matters because the two get conflated constantly. I have seen teams point an auditor at Datadog and get politely destroyed, because Datadog was sampling, dropping high-cardinality tags, and expiring records after 30 days. An audit record for a limit change on a lending product might need to survive seven years depending on your jurisdiction. Those are not the same system and pretending they are will bite you.
If you cannot answer "who changed this, when, and what did it look like before" for any regulated resource, for the full retention period, from a source you would be comfortable handing to a regulator, you do not have an audit log. You have hope.
The write path is not the read path
The mistake that kills audit logs at scale is treating writes and reads as the same problem. They are opposites. Writes are enormous in volume, latency-sensitive, and must never fail silently. Reads are rare, tolerant of a few seconds of latency, and need rich filtering. If you try to serve both from one Postgres table with a dozen indexes, the indexes will slow your writes and the writes will bloat your reads, and you will end up unhappy on both ends.
My strong preference: capture on the write path as cheaply as possible, then move the data to a store optimized for querying. On the hot path you want to do the absolute minimum. Serialize the event, hand it to a durable queue or an outbox, and get out. Do not enrich it inline, do not resolve the actor's full name from another service, do not do anything that can throw. Enrichment and indexing happen downstream where a slow query hurts nobody.
Capture through an outbox, not a side call
Here is the trap I fell into early: writing the audit event as a separate call after the business transaction commits. Business write succeeds, audit write fails because the queue hiccuped, and now you have a change with no record. In a payments system that is a compliance gap, not a bug ticket.
The fix is the transactional outbox. You write the audit event to a table in the same database transaction as the business change. Either both commit or neither does. A separate relay process reads that table and ships events onward. The audit record shares the fate of the thing it describes, which is exactly the property you want.
-- Business change and audit event commit atomically
BEGIN;
UPDATE credit_accounts
SET credit_limit = 15000
WHERE account_id = 'acct_8842';
INSERT INTO audit_outbox (
event_id, occurred_at, actor_id, actor_type,
action, resource_type, resource_id,
before_state, after_state, request_id
) VALUES (
gen_random_uuid(), now(), 'user_331', 'operator',
'credit_limit.update', 'credit_account', 'acct_8842',
'{"credit_limit": 10000}'::jsonb,
'{"credit_limit": 15000}'::jsonb,
'req_a19f2c'
);
COMMIT;
-- A relay drains audit_outbox and forwards to the audit store.
-- If COMMIT fails, neither row exists. That is the point.What every record must carry
Getting the schema right upfront saves you from a data migration across billions of rows later, which is a sentence I have said out loud while staring at a very slow backfill. A useful audit event is not just "something changed." It is a small, complete story. Over the years I have converged on a fixed set of fields that I now treat as non-negotiable on any new service:
- Actor — the identity that caused the change, plus its type. A human operator, a service account, and an automated batch job are different and you will want to filter on that difference.
- Action — a stable, namespaced verb like credit_limit.update. Free-text descriptions are useless for querying; enums drift; namespaced strings age well.
- Resource — type and id together. resource_id alone is ambiguous once you have forty resource types.
- Before and after state — the single most requested thing in every real audit I have sat through. Reconstructing prior state from a chain of deltas is a trap; store the snapshot.
- Correlation — a request or trace id so you can tie the audit event back to the wider system when you inevitably need to.
- Timestamp in UTC — with the source clock recorded honestly. Never local time. I have watched a daylight-saving transition make an hour of events look like they happened twice.
Make it genuinely append-only
An audit log that can be edited is worth roughly nothing to an auditor, because the obvious question is "how do I know this wasn't changed after the fact?" You need to be able to answer that with something better than "trust us."
At the database level, that means the application role has INSERT and SELECT on the audit table and nothing else. No UPDATE, no DELETE, revoked explicitly. For the stronger guarantee, chain your records: each event stores a hash of its own contents plus the hash of the previous event, so any tampering breaks the chain from that point forward. It is a lightweight scheme, not a blockchain, and it does not need to be. The point is detectability, not decentralization.
One caveat that trips people up: append-only fights with data deletion laws. If a customer exercises a right to erasure, you cannot just DELETE the row without breaking your immutability story. The pattern I use is to keep the audit record's structure and hashes intact but store the personal fields encrypted, then destroy the key on erasure. The event still exists, the chain still verifies, and the personal data is genuinely unrecoverable. Talk to your legal team before you pick an approach here; this is one place where the right technical answer depends entirely on the regulation.
Storage that survives seven years of growth
Volume is where good intentions go to die. A mid-sized platform can generate tens of millions of audit events a month, and if you are keeping them for seven years, you are talking billions of rows that are almost never read. Keeping all of that in your primary transactional database is expensive and pointless.
Tier it by age. Recent events, say the last 90 days, live in a queryable store that supports the interactive lookups your operations and fraud teams do daily. Older events roll off to cheap object storage in a columnar format like Parquet, partitioned by date, queryable on demand when an auditor comes knocking. The read latency on cold data goes from milliseconds to seconds, and that is completely fine, because nobody is doing interactive analysis on events from 2021. They are answering one specific question about one specific Tuesday, and they can wait ten seconds for it.
Design the query before you need it
The read side is where you find out whether your write side was any good. Every real audit request I have handled reduces to a handful of query shapes, and if your data model does not serve them cleanly, you will be writing one-off scripts under time pressure while an auditor waits.
Three shapes cover almost everything. Everything a specific actor did in a window. Everything that happened to a specific resource across its whole life. Every occurrence of a specific action across all resources, which is how you answer "show me all credit limit increases above 20,000 last quarter." Design your partitioning and indexing around those three, not around some abstract idea of flexibility. I would rather have three fast queries than infinite slow ones. Optimize for the questions you will genuinely be asked, not the ones you can imagine.
Discipline about what not to capture
The instinct once you have a good pipeline is to log everything, and it is the wrong instinct. Audit logs are not a firehose; they are a curated record of security- and compliance-relevant events. Logging every read of every record turns your evidence store into noise and buries the events that matter under millions that do not. Worse, over-logging is how secrets end up in your audit trail. I have seen a full request body, complete with a plaintext card number, land in an audit event because someone logged the whole payload "just in case." That is now a PCI incident, and the audit log became the thing being audited.
Be deliberate. Log state changes to regulated resources, authentication and authorization decisions, permission grants, and administrative actions. Do not log routine reads unless a regulation specifically requires it. And run every audit payload through the same redaction rules you apply everywhere else, because an audit log full of sensitive data is a liability dressed up as a control.
Conclusion
The thing I would tell my younger self, staring at that empty query result and drafting an apologetic email to the auditor, is this: an audit log is a product, and its users are future-you, your fraud team, and a regulator you have not met yet. You are building for a single moment years out, when someone asks a pointed question about one event and either the answer is right there or your afternoon is ruined. Do the work and that moment becomes a boring five minutes of running a query. Boring is the goal. In this corner of the business, boring is the highest praise there is, and I have stopped apologizing for chasing it.