The best staff engineer I ever worked with left on a Friday with three weeks' notice, and by the following Wednesday I understood that he had been the sole human who knew how our settlement reconciliation job recovered from a partial failure. Not the only person who wrote it. The only person who understood it. There was a runbook. The runbook was four years old and wrong in two places that mattered.
We got through it, barely, with a lot of screen-sharing and one genuinely frightening Saturday. But the lesson stuck harder than any postmortem I have ever run. Succession planning is not an HR ritual for the executive layer. In a regulated payments business it is an operational control, and if you treat it as a soft skill you will eventually pay for it in an incident.
Bus factor is a risk register item, not a joke
Engineers say "bus factor" with a grin, as if it were dark humor rather than a number you can actually compute. I stopped grinning about it years ago. For every critical system we run, I want to know how many people could keep it alive at 3am without escalating to someone who is on a plane. If the answer is one, that is a finding, and it goes on the same risk register as an expired certificate or an unpatched dependency.
The uncomfortable truth is that your most dangerous single points of failure are usually your best people. They are fast, they are trusted, so work flows to them, and every time it does the concentration gets a little worse. I have watched a brilliant engineer become a liability purely through everyone else's dependence on him. He did nothing wrong. The org failed to spread the load, and then quietly organized itself around the fact that he would always be there.
Once you frame it as risk rather than sentiment, the conversation gets easier. Nobody argues about whether we should have a backup for our HSM keys. So why would we accept that exactly one person understands how a payout batch gets rebuilt after a mid-run crash?
Two kinds of succession, and people confuse them
There are really two problems hiding under the same phrase, and conflating them wastes everyone's time. The first is planned succession: a senior person is going to be promoted, move teams, or leave on good terms with notice. You have weeks or months. The second is unplanned succession: someone is hit by the proverbial bus, quits in anger, gets poached with two days of garden leave, or is simply out sick during the worst possible week.
Planned succession is a project management problem. You can sequence a handover, shadow real incidents, transfer relationships one introduction at a time. Unplanned succession is a resilience problem, and the only defense is work you did months earlier. You cannot cram for it. The distinction matters because most leaders only prepare for the planned kind, feel prepared, and then discover the hard way that the second kind is the one that actually hurts.
My rule of thumb: assume the unplanned version. If you are resilient to someone vanishing overnight, the graceful three-month handover becomes a luxury rather than a lifeline.
The lieutenant trap
Here is a failure mode I fell into personally. As a young manager I picked a deputy, a sharp senior engineer, and I routed everything through him. Architecture calls, on-call rotations, the tricky customer escalations, the thorny vendor conversations. He was excellent, and it felt efficient. I had built myself a second-in-command.
What I had actually built was a bus factor of two where I thought I had improved things. When he took paternity leave for four months, half the team's institutional knowledge left with him, and the other half had never been given the chance to develop because he had always been the answer. I had concentrated knowledge in the act of trying to delegate it. Real succession depth is not one strong understudy. It is a bench.
A single deputy is not redundancy. It is your key-person risk wearing a slightly different hat.
Knowledge that lives only in heads
The knowledge you most need to preserve is precisely the knowledge nobody bothers to write down, because to the person who holds it, it feels obvious. The location of the config flag that quietly disables 3-D Secure for one acquirer. The reason a particular retry is set to exactly 90 seconds and not 60. The fact that the batch job must run before the currency rates refresh or Tuesdays go strange.
Documentation helps, but I have grown skeptical of the belief that a wiki solves this. Wikis rot. Nobody reads them until the fire is already burning, and by then they discover the page was last edited in 2022. What actually transfers deep knowledge is doing the work alongside someone. A few concrete practices that have worked for my teams:
- Rotate on-call so the person carrying the pager changes every week and every rotation touches a real incident eventually, not just the same two veterans.
- Make the author of a change explain it to a reviewer who has never touched that system, so understanding is forced to cross at least one boundary.
- Run game days where you deliberately break something in a staging environment and let a non-expert drive the recovery while the expert sits on their hands and only answers questions.
- Keep runbooks executable and tested. A runbook you have not run in six months is a hypothesis, not a procedure.
The game day one is the single most useful thing on that list, and it is the one people skip because it feels like theater. It is not theater. It is the closest you get to a fire drill, and the awkward silence when the non-expert doesn't know the first command is exactly the gap you needed to find before it found you.
Growing successors on purpose
You do not get succession depth by hoping. You get it by deliberately handing people work that is slightly above their current level and then resisting the urge to take it back when they struggle. This is genuinely hard, because in the short term the expert is always faster, and every quarter has a deadline that rewards using the expert.
I try to run a rough calculation in my head for critical systems. If I let the second-most-knowledgeable person handle the next three non-trivial changes to that system, they will be slower and I will spend review time. Call it two weeks of drag over a quarter. Against that, I am buying down a key-person risk that could cost me a multi-day outage and a very unhappy conversation with the regulator. That trade is not close. I take the drag every time, and I have never once regretted it during an actual incident.
The other half is telling people plainly that you are doing this. "I want you to own reconciliation within six months, here is what that means, here is who will help you get there." People rise to explicit expectations far more reliably than to vague hope that they will absorb things by osmosis.
Documenting the parts that embarrass you
The most valuable document I have ever made a team write is not the architecture diagram. It is the honest list of what is fragile, what is held together with tape, and what you would never do to this system if you were being careful. The scars. The "do not deploy this on a Friday" knowledge.
Teams resist writing this down because it reads like an admission of failure, and there is a real temptation to keep the diagram pretty and the caveats verbal. Fight that. When a critical person leaves, the pretty diagram survives fine on its own. It is the informal folklore, the "oh, never touch that without checking with billing first," that walks out the door and takes a piece of your operational safety with it. Write the ugly parts down precisely because they embarrass you.
The CTO is not exempt
It is easy to run this exercise on your senior engineers and quietly exempt yourself. I did, for a while. Then a board member asked me, more pointedly than I liked, what would happen to the engineering org if I were unavailable for a month. I did not have a good answer, and the fact that I did not have one was itself the problem.
If you are the leader, your relationships, your context on why past decisions were made, and your read on which fires are real are all single points of failure too. I now keep a short document that I would not want anyone outside a small circle to read: the vendor relationships that hinge on a personal contact, the decisions that look wrong but were deliberate, the political landmines a successor would step on. It is morbid to write. It is also the most responsible thing I do all year.
What good actually looks like
You will not reach a state where losing anyone is painless, and you should not aim for it. Some pain is the cost of having people who care and go deep. The goal is narrower: no single departure should be able to threaten the business, and no critical system should have a bus factor of one for longer than it takes you to notice and fix it.
Concretely, in a healthy org I can name at least two people who could run each critical system, on-call rotations pull in more than the usual suspects, handovers when people leave are tense but not terrifying, and the answer to "what if this person disappeared tomorrow" is a plan rather than a held breath. That is not a finished state. It is a discipline you keep paying into, quietly, when nothing is on fire.
Conclusion
The thing I would tell my younger self is that succession planning feels like insurance you are paying for a claim that never comes, right up until the day it comes. Every hour you spend spreading knowledge is an hour you could have spent shipping, and the market never claps for the outage you prevented. Do it anyway. The best compliment your architecture can receive is that the person who built it took a real vacation, uncontactable, and the pager stayed quiet. If your team cannot survive a two-week silence from any one member, you have not built a system. You have built a hostage situation, and you are the one being held.