If you died in a car accident on your way to work tomorrow would your replacement be able to access your systems and work products? Or would they be locked out of everything and struggle for weeks or months to gain access and move forward? Business continuity planning is something most people associate with CEOs and big companies. However, as the sole software developer in my company for many years I was acutely aware of how screwed my employer would be if I died suddenly.
In this post in going to show you the steps I took to ensure my company wouldn’t be crippled if I died.
Document whatever you can in a private wiki and share it with your boss
The first rule of business continuity planning is documentation. I created a private wiki (hosted by a third party) and documented everything I deemed important. Then I shared my boss on the wiki with full admin privileges so he could share it with my replacement in the event of my death.
I put all kinds of useful information in our wiki like:
- an overview of all the pieces of our production environment and how they fit together
- SaaS products we use like BitBucket, Jira, Campaign Monitor, etc.
- web server configurations
- firewall configurations
- project specific configuration files
- router configuration settings with screenshots
- network layout including static and dynamic ranges along with switch and printer IP addresses
- DNS settings
- SSL certificate details
- how our database backups work and where the backups are stored
I had three criteria for creating this kind of documentation. It must:
- not damage our security if the wrong person sees it (that means no usernames, passwords, or private encryption keys)
- help me recreate our systems if the need ever arises
- help my replacement take over my duties if I die
When we grew from one programmer to two, the wiki also helped me orient our new programmer and delegate work to him, which was a nice side benefit.
Some tips
It is easiest to create this documentation when you are setting up a new system. That’s what I did most of the time. But sometimes I would stumble upon something I forgot to document and I found it a bit more of a pain to document after the fact.
I documented to the expected skill level of my replacement. That means I assumed my audience was a programmer with a reasonable understanding of Windows and Linux administration, web servers, networking, etc.
I freely documented text commands and made screenshots of settings where I didn’t write a script to execute all the commands for a task. My goal was to take as much guesswork out of the process as possible.
Finally, I updated the documentation whenever I used it to:
- reduce ambiguities
- record new settings
- document updates to GUIs or commands
Business continuity planning for passwords and private encryption keys
Because I was an admin on everything, I felt I needed to take extreme precautions with my passwords and private encryption keys. I was very aware that someone could literally destroy our business if this information fell into the wrong hands.
On the other hand, I also needed to be able to access this information on a regular basis, ensure I never lost it, and somehow make it available to my boss if I died suddenly.
My approach
I took a two pronged approach. First I setup separate accounts for my boss and made him an owner on as many accounts as appropriate. This is stuff like AdWords, Google Analytics, shared Google Docs, our domain name provider, DNS, etc.. In the event of my death, he could simply remove my access and add my replacement on those accounts.
But, if you are following the principle of least privilege–which you should be–then there are certain accounts you shouldn’t share with anyone such as server logins, ssh passwords, database logins, router passwords, private encryption keys, and the like. Most non-technical people aren’t in a position to protect highly sensitive credentials. And if they don’t need access to these accounts to do their job, they shouldn’t have access to them.
So, for the second prong of my approach I:
- created strong, long, random passwords (the more important account, the longer the password)
- stored them in an encrypted container with a super-strong master password that I memorized (and never wrote anywhere)
- stored the encrypted container in the cloud so I can access it from anywhere and to take advantage of the automated versioning and backups of my preferred storage provider
- created a printout of my passwords and periodically and securely shared it with my boss (see details below)
How I shared my passwords with my boss without exposing them
I printed a copy of my passwords and sealed them in an envelope, which I signed and dated on the seal. Then I made arrangements to store the envelope in the company safe. And I actually watched the safe keeper put it into the safe because I’m that kind of paranoid.
My boss knew to request the envelope in the event of my death. And the safe keeper knew to only give the envelope to me if I was alive or to my boss if I was dead.
Every couple of months–or more often if something important changed–I’d replace the envelope in the safe with an updated version. If the old envelope was missing or had been opened, I’d know my passwords were compromised. I’d also open the old envelope to ensure it contained the password file I was expecting and then I’d cross-cut shred it and the envelope (so nobody would see how I signed and dated it).
That’s it. My boss was happy because he knew he could access all our systems if I died. And I was happy because I knew that our most sensitive passwords weren’t written on a Post-It note stuck to the bottom of someone’s keyboard. Nor were they vulnerable to social engineering attacks like Kevin Mitnick was famous for perpetrating.
Just a little warning
Pay particular attention to the stuff you’ve encrypted with symmetric or public key cryptography. Your replacement isn’t going to be access that stuff with a simple password reset or a call to your vendor. So make sure those passwords (and private keys) are part of your plan and that they are always up to date.
Wrapping up
Business continuity planning is important for every key employee in your company, not just for owners and senior managers.
I believe everybody should be documenting their systems. That’s just common sense. It really wasn’t that much work and it has paid for itself many times over in the time it’s saved me over the years.
As the sole programmer in my company I had to get a little creative with my business continuity planning for passwords and encryption keys. I believe I struck a good balance between security and convenience, given my circumstances. But what’s right for you may be different. I encourage you to think about your circumstances and engage in some business continuity planning of your own so your company won’t be crippled if you die in a car accident on your way to work one day.
Agree or disagree? I’d love to hear your thoughts.
Leave a Reply