Small Business Programming

Learn how to be a wildly successful small business programmer

Are you working on the most important thing?

Is it possible that you’ve undervalued or overlooked stories in your backlog that will return thousands of dollars on every hour of your effort? Are you sure you’re working on the most important thing?

That’s exactly what happened to me. I’d been ignoring a bunch of “low priority” stories until I discovered that a similar story returned well over $15K per hour of my time.

In this post, I’m going to tell you what happened, how you can find those kind of stories in your own backlog, and how you can make sure you’re working on the most important thing.

I made a landing page that effectively prints money

About 5 years ago I got interested in keyword analysis. So, I spent a little time investigating it between work assignments. I soon discovered a really great keyword that didn’t appear anywhere on our site. So, I quickly built a landing page, made an ad group for it on AdWords, and started pushing traffic to it.

My landing page was ugly. It was just a bunch of benefits I threw together and a couple of calls to action. Because it was an experiment, I didn’t spend much time on it. Yet, despite its amateurish look, we soon learned that it converted like crazy–way beyond anything else we had in AdWords. I was proud of myself and thought about taking it further. But I had lots of other work to do so I moved on to other things.

The years passed and I continued to be proud of my landing page. It was always our best ad group–by far–and it was doing well in organic search too. But I was busy so I left it alone.

I waited 5 years to assess the performance of my landing page

An unrelated work assignment prompted me to finally assess my landing page. And I was floored when I learned that my ugly landing page has made us in excess of $15K per hour of effort (so far). And that its annual revenue is increasing every year. So, there’s no telling how much money that one landing page will make my employer over its lifetime.

landing page revenue 2013-2017

landing page revenue 2013-2017

Now, my goal is to always be working on the most important thing. And I thought I was doing pretty well at it. But given the rate of return on this one landing page, I’d have to say that I haven’t been working on the most important thing because I don’t do anything on a regular basis that brings in five figures per hour.

So, I spent some time looking at new keywords (I’d be crazy not to, right?). And after a couple of hours I found a bunch more that look like they have the potential to be in the same league as our best keyword.

I’d been ignoring a huge opportunity for years.

How to ensure you’re working on the most important thing

We knew that my landing page was good. We talked about it every time we talked about AdWords or Google Analytics. I even built another landing page on different keyword that my boss and I brainstormed one day. But we never actually sat down and calculated the value of my landing page. Other tasks always seemed to be a higher priority.

It’s difficult be to identify the most important thing because the solution space is huge and it’s difficult to compare stories. But I’ve found The Theory of Constraints to be a helpful way to quickly zero in on high value areas without looking at the whole business in excruciating detail. And last year I learned about cost of delay and CD3, which combine very nicely with the Theory of Constraints. I wrote about these concepts here, here, and here. And yet I still failed to find the most important thing for at least 5 years. What gives?

You need to close the loop

I recently read The Lean Startup and I think Eric Ries has the answer. He spends a good portion of the book talking how important it is for startups to gain something he calls “validated learning” by engaging in a “build-measure-learn” cycle. Well, I did the build. But I didn’t measure or learn for five years (author comically applies palm to forehead).

So, my solution is simple. Use the Theory of Constraints, Cost of Delay, and CD3 to prioritize your work. But also take the time apply a build-measure-learn feedback loop whenever you can.

You must convert your results to dollars

CD3 forces you to convert your stories to dollars and that’s the key to this whole thing.

We were able to turn inaction into action when we calculated an actual dollar value of my landing page. We knew it was good but I do lots of things that have good results so it didn’t really stand out. But when we attached a dollar value to how “good” my landing page is, both my boss and I were instantly convinced that we needed to make more landing pages if we can find more similarly attractive keywords.

Without a dollar value attached to your outcomes, its impossible to accurately calculate the relative “goodness” of different stories. For example, should you spend 5 hours squashing bugs or 5 hours adding a new feature to your software? See the problem? Your whole backlog is filled with things that are difficult to compare. That’s why you need to prioritize your backlog by CD3.

Update your estimates based on the results of your validated learning

Finally, I think Ries’ build-measure-learn feedback loop hints at something more subtle. The value of each story in your backlog relies on two estimates.

You must estimate the value of the story to your customers and you must estimate the cost of implementing that story (possibly adding some maintenance costs as well). You use these two estimates to determine the ROI of your stories. But you won’t know how accurate either of those estimates are until you complete the story, get it in front of your customers, and analyze your results.

So, here’s the important bit. If you quantify both the value of your stories and your actual effort in dollars, you can calculate two things:

  • how close to optimal you ordered your stories in the past
  • if you need to update your value and/or cost estimates for similar future stories

With this data in hand, you can prioritize your backlog more accurately and ensure you are–to the extent possible–always working on the most important thing.

Resist the urge to use your gut to prioritize your backlog

I know the cowboys software developers out there will want to ignore all this analysis stuff and just get down to coding the next thing. That’s a mistake that will lead you to develop needlessly expensive software in the best case, worthless software in the average case, and no finished software at all in the worst case.

The value of your stories likely follows a power law distribution, which has significant consequences.

power law distribution

power law distribution

You need a sound method of consistently identifying those few truly valuable stories at the far left side of your distribution. Without such a method you’ll waste the majority of your resources working on low priority stories. You need CD3 if you don’t want to be mediocre.

Takeaways

The more I learn about project management, the more I realize the extent to which managers are needlessly making decisions based on gut feelings and delivering poor results. In some cases, following your gut is absolutely the right thing to do. But for the vast majority of business decisions, we are just engaging in sub-optimal behavior when we don’t take the time to do the ‘measure-learn’ phases of the ‘build-measure-learn’ feedback loop.

It literally took me less than five minutes quantify the value of my ugly landing page. I don’t really have any excuse for not doing it sooner other than that my focus was divided among the other three hundred things in our backlog. But now that I’ve looked, we’ve moved landing pages way up in our backlog.

So, dear reader, are you working on the most important thing? If so, how do you know? If not, tell me why in the comments.

Business continuity planning: how screwed would your employer be if you died suddenly?

If you died in a car accident on your way to work tomorrow would your replacement be able to access your systems and work products? Or would they be locked out of everything and struggle for weeks or months to gain access and move forward? Business continuity planning is something most people associate with CEOs and big companies. However, as the sole software developer in my company for many years I was acutely aware of how screwed my employer would be if I died suddenly.

In this post in going to show you the steps I took to ensure my company wouldn’t be crippled if I died.

Document whatever you can in a private wiki and share it with your boss

The first rule of business continuity planning is documentation. I created a private wiki (hosted by a third party) and documented everything I deemed important. Then I shared my boss on the wiki with full admin privileges so he could share it with my replacement in the event of my death.

I put all kinds of useful information in our wiki like:

  • an overview of all the pieces of our production environment and how they fit together
  • SaaS products we use like BitBucket, Jira, Campaign Monitor, etc.
  • web server configurations
  • firewall configurations
  • project specific configuration files
  • router configuration settings with screenshots
  • network layout including static and dynamic ranges along with switch and printer IP addresses
  • DNS settings
  • SSL certificate details
  • how our database backups work and where the backups are stored

I had three criteria for creating this kind of documentation. It must:

  1. not damage our security if the wrong person sees it (that means no usernames, passwords, or private encryption keys)
  2. help me recreate our systems if the need ever arises
  3. help my replacement take over my duties if I die

When we grew from one programmer to two, the wiki also helped me orient our new programmer and delegate work to him, which was a nice side benefit.

Some tips

It is easiest to create this documentation when you are setting up a new system. That’s what I did most of the time. But sometimes I would stumble upon something I forgot to document and I found it a bit more of a pain to document after the fact.

I documented to the expected skill level of my replacement. That means I assumed my audience was a programmer with a reasonable understanding of Windows and Linux administration, web servers, networking, etc.

I freely documented text commands and made screenshots of settings where I didn’t write a script to execute all the commands for a task. My goal was to take as much guesswork out of the process as possible.

Finally, I updated the documentation whenever I used it to:

  • reduce ambiguities
  • record new settings
  • document updates to GUIs or commands

Business continuity planning for passwords and private encryption keys

Because I was an admin on everything, I felt I needed to take extreme precautions with my passwords and private encryption keys. I was very aware that someone could literally destroy our business if this information fell into the wrong hands.

On the other hand, I also needed to be able to access this information on a regular basis, ensure I never lost it, and somehow make it available to my boss if I died suddenly.

My approach

I took a two pronged approach. First I setup separate accounts for my boss and made him an owner on as many accounts as appropriate. This is stuff like AdWords, Google Analytics, shared Google Docs, our domain name provider, DNS, etc.. In the event of my death, he could simply remove my access and add my replacement on those accounts.

But, if you are following the principle of least privilege–which you should be–then there are certain accounts you shouldn’t share with anyone such as server logins, ssh passwords, database logins, router passwords, private encryption keys, and the like. Most non-technical people aren’t in a position to protect highly sensitive credentials. And if they don’t need access to these accounts to do their job, they shouldn’t have access to them.

So, for the second prong of my approach I:

  • created strong, long, random passwords (the more important account, the longer the password)
  • stored them in an encrypted container with a super-strong master password that I memorized (and never wrote anywhere)
  • stored the encrypted container in the cloud so I can access it from anywhere and to take advantage of the automated versioning and backups of my preferred storage provider
  • created a printout of my passwords and periodically and securely shared it with my boss (see details below)

How I shared my passwords with my boss without exposing them

I printed a copy of my passwords and sealed them in an envelope, which I signed and dated on the seal. Then I made arrangements to store the envelope in the company safe. And I actually watched the safe keeper put it into the safe because I’m that kind of paranoid.

safe

My boss knew to request the envelope in the event of my death. And the safe keeper knew to only give the envelope to me if I was alive or to my boss if I was dead.

Every couple of months–or more often if something important changed–I’d replace the envelope in the safe with an updated version. If the old envelope was missing or had been opened, I’d know my passwords were compromised. I’d also open the old envelope to ensure it contained the password file I was expecting and then I’d cross-cut shred it and the envelope (so nobody would see how I signed and dated it).

That’s it. My boss was happy because he knew he could access all our systems if I died. And I was happy because I knew that our most sensitive passwords weren’t written on a Post-It note stuck to the bottom of someone’s keyboard. Nor were they vulnerable to social engineering attacks like Kevin Mitnick was famous for perpetrating.

Just a little warning

Pay particular attention to the stuff you’ve encrypted with symmetric or public key cryptography. Your replacement isn’t going to be access that stuff with a simple password reset or a call to your vendor. So make sure those passwords (and private keys) are part of your plan and that they are always up to date.

Wrapping up

Business continuity planning is important for every key employee in your company, not just for owners and senior managers.

I believe everybody should be documenting their systems. That’s just common sense. It really wasn’t that much work and it has paid for itself many times over in the time it’s saved me over the years.

As the sole programmer in my company I had to get a little creative with my business continuity planning for passwords and encryption keys. I believe I struck a good balance between security and convenience given my circumstances. But what’s right for you may be different. I encourage you to think about your circumstances and engage in some business continuity planning of your own so your company won’t be crippled if you die in a car accident on your way to work one day.

Agree or disagree? I’d love to hear your thoughts.

Software security is hopelessly broken

As software developers, we are doing a terrible job of protecting the data we collect from our users because software security is hopelessly broken. This is a huge topic so I’ll restrict my comments to coding, encryption/hashing, web server configurations, regulation, and what we can do about the security of the software we create and maintain.

Programming needs to be significantly safer by default

We’re failing on the easy stuff. We are guilty of hard coding API passwords in our code bases and then posting it on github, insecurely storing user passwords, writing code that’s vulnerable to injection and overflow attacks, failing to properly validate data before using it, not protecting our backups, not deleting data we no longer need, etc., etc..

I bought a book on secure programming around 2002 and all the risks identified in that book are still very much with us today. We’ve barely moved the needle at all in the last 15 years!

The only way we are going to make significant progress on software security issues is to make programming safer by default. Trying harder hasn’t worked and is unlikely to work for the vast majority of projects in the future.

Sure, it would be great if every developer had security training and lived by the programmer’s oath. It would definitely help. I wrote quite a popular post about software professionalism and security if you’re interested.

The problem is that we already have a thousand things to remember every time we write a line of code and it’s naive to think that humans (with our pitiful working memory of 5 +/-2) will every remember to do everything right all the time. (Or that your boss will let you take another month to review your code for security problems before you release it.)

Secure programming in C is basically impossible

Have you ever looked at the security guidelines for properly range checking an array index in C? Not fun. Who’s going to get that 100% correct every time? If you write a significant project in C, you are going to have trouble ensuring that you never get an array index out of bounds, or an overflow, or a null pointer, etc..

You can staple copies of the MISRA C standard to your developer’s foreheads and review every commit 5 times but you’ll still miss piles of errors.

What’s safer look like?

  • computer languages that have automatic bounds checking
  • database abstraction layers that automatically escape inputs to prevent SQL injection attacks
  • templating engines that automatically escape output by default to prevent cross-site scripting
  • form builders that automatically add and check for a unique token on every submission to prevent cross-site request forgery
  • data validators that make it easy to prevent injection attacks
  • web frameworks that have well designed and tested authentication and authorization capabilities
  • tools that allow software developers to statically and dynamically examine their code and report a prioritized list of problems
  • security scanners that are easy to use

These things work because you get the security benefits for free (but only if you actually use them). Secure coding has to be automatic and free if we expect it to work.

Password hashing and encryption need to be idiot-proof

We need simple ways to get the benefits of the most-up-to-date programming practices without becoming experts. PHP developers have actually done some impressive work in this area.

Secure password hashing

For example, password hashing in PHP is now simple to use and strong by default. PHP has three functions in the core that do everything you need to securely store and verify passwords. We upgraded one of our websites in a couple of hours. So, PHP now takes care of the salting and secure hashing of our passwords. Our code will even upgrade our hashes automatically in the future when something better comes along.

Here’s the best part: people using PHP’s secure hashing functionality don’t need to understand security best practices, salting, rainbow tables, or the difference between md5 and sha-256. And that’s the way it should be.

Secure application level encryption

Application level encryption should be dirt-simple to use. Anybody should be able to call encrypt($cleartext, $key) and decrypt($cyphertext, $key) and know that it’s secure without understanding anything about how encryption works.

If you’re an expert go ahead and use the lower level functions. But most of us just need to encrypt a string and store it safely so we can decrypt it later. So just give us something safe to use and we’ll use it. Encryption isn’t quite as easy to use as password hashing in PHP but it’s getting close. Check out this implementation (scroll down for example code). I imagine simpleEncrypt() and simpleDecrypt() or something similar will eventually make it into the PHP core.

Servers need to be easier to configure and more secure by default

Have you ever tried to setup a web server and make it secure? I have, and it’s not fun on Windows or Linux. The level of knowledge you need to do this well is insane. But even if you do manage to create what you believe is a “secure” configuration, you have no guarantees that your server will remain secure tomorrow or next week.

What would be better? Imagine if Apple developed the GUI for a web server OS that was built to the security standards of the OpenBSD project. This is out of my wheelhouse so forgive me if I say something silly.

Here are some features I’d like to see in a secure web server OS:

  • it’s easy to see the configuration of the system and how it has changed over time (and who changed it)
  • the server monitors the behavior of logged-in users and reports anything suspicious (along with a recording of everything they did and saw during their session)
  • it’s easy to see if someone is attacking your system, how they are attacking it, and what the OS is doing to stop the attack from succeeding
  • the server should contact the admin if it needs help defending itself from an attack (and suggest actions the human should take)
  • the OS should only allow software it thinks is safe to be executed (I know this is very challenging in practice but I can dream)
  • configuration changes are made through wizards (or scripts) and the system won’t allow you to make silly configuration mistakes (like creating an ssh account with an easily guessed password)
  • the OS should monitor how it is used and suggest or automatically turn off unneeded functionality
  • the OS should automatically install signed updates without requiring a reboot but allow rollback if necessary (or have a configurable update policy)
  • built-in encrypted full backups with one click restores
  • the OS should be wary of new hardware and anything plugged into its USB ports
  • the file system is encrypted by default
  • the OS uses address space layout randomization by default
  • multiple servers can be managed from a single interface with ease
  • the server should fail safely (never reveal sensitive information about itself or its data)
  • the OS should be able to run a self-test and tell you all the places it can be accessed/exploited
  • the OS should learn from the successes and failures of other systems to improve its security and performance (like anti-virus software does today)
  • all firmware is cryptographically signed

I know this stuff is easier said than done but you can’t dispute the fact that there’s lots of room for improvement here. There’s also no shortage of controversy around making computing safer. In many ways freedom and flexibility are at odds with security.

New regulations are going to force us to change the way we design and construct software

I’m interested to see what is going to happen to the software world when the EU’s new data protection regulations come into effect on May 25, 2018. These regulations are specific and the penalties for not complying with them are steep but the details of how it’s going to be enforced are still unclear. I’d be surprised if 2% of the software in the wild that contains user data complies with these regulations. And making your existing software compliant is going to be expensive.

Plus, this is just the beginning of the regulation of non-safety critical software. I predict more and more regulation will be thrown at us as people get tired of data breaches and the damage caused by our crappy software. People will seek government protection.

I also wonder when insurance companies are going to routinely set premiums for businesses based on what kind of software they develop and how carefully they develop it.

It should be interesting to see how it all turns out.

Okay, software security is hopelessly broken. What happens next?

I believe we’ll get slightly better at writing secure software in the coming years. But the bad guys will continue to steal our data with ease.

We’ll use safer languages, better tools, and incrementally better software engineering practices to create software that offers our users slightly more protection (like testing, static analysis, and code reviews). Big companies like Google, Microsoft, and Facebook will do a better job of writing secure software than small companies. Apps and IOT devices will remain an absolute disaster area but almost all software will remain vulnerable because, like I’ve said before, software security is hopelessly broken.

There are just too many ways make a programming or configuration mistake, trick you into defeating your own security, or attacking your system at another level (network, router, OS, hardware, firmware, physical, etc.).

Plus there are billions of lines of code out there that will never and can never be upgraded to be secure because:

  • the software has been abandoned
  • of the expense of modifying existing software is prohibitive
  • it’s basically impossible to add effective security controls to an existing insecure system
  • we don’t have enough security experts to go around
  • there’s no money in fixing it

Conclusion

Here’s the thing: our entire modern computing apparatus held together with duct tape. There is no bedrock layer in computing that we could drop down to and say “everything below this point is safe so we’re just going to rebuild from this level.”

Nothing like that exists. We could employ security experts to redesign/rewrite everything from scratch (hardware, firmware, OS, and applications, protocols, etc.) with insane attention to detail. But we don’t yet know how to make complex software without errors, certainly not at the scale we are talking about here.

Plus, what are you going to do about the people? They’re part of the system too. Remember, the bad guys can just drug you and hit you with a wrench until you give up all your passwords.

And you also have to worry about physical security as well because someone could slip a key logger between your keyboard and your computer. Or remotely read the EMF off your keyboard (it works with wired keyboards too). Or just install a small camera in your room near your computer and take videos of your screen. Or activate your webcam and read your screen in the reflection in your glasses. Or any of a million other things.

Nope. The truth is that software security is hopelessly broken.

What can you do?

  • keep your software up to date–security updates are the best defense you have
  • comply with all applicable laws and regulations such as GDPR, HIPPA, PCI-DSS, etc.
  • educate yourself about security best practices, the tools, and the languages available to you to help you reduce the cost of writing secure software
  • use a risk mitigation strategy to protect your most sensitive data first because you can’t fix everything all at once
  • allocate time to fix lower priority security issues because they are never going to fix themselves
  • raise awareness about your security issues by talking about them with your coworkers (both programmers and non-programmers)

What do you think? Do you believe software security is hopelessly broken? I’d love to hear your thoughts.

Books I read in 2017

In this post, I’m going to share some of the books I read this year. In the nonfiction realm, some of them are very much on point for software developers. Others are just good books that let you know what’s going on in the world.

In fiction, I’m drawn to technology-driven Sci-Fi. I found some great reads this year. I also listed some books at the bottom of this post that weren’t that good. I think it’s just as important to tell you about the good books as the not so good books.

Enjoy!

My rating: 5/5

The Principles of Product Development Flow: Second Generation Lean Product Development
This is the most important book I read in 2017. In fact, it’s probably the most important book I’ve read in several years.

Here’s the deal: agile software development (and SCRUM in particular) borrowed successful ideas from manufacturing and applied them to software development and expected to see the same results. The problem is that the fundamental assumptions that make LEAN and the Toyota Production System so successful in manufacturing don’t hold up in software development.

So, team after team adopted SCRUM and then wondered why they weren’t going any faster and why they were having so many problems. So they called in the consultants and the consultants said that they weren’t doing it right. Try harder! So they tried harder but SCRUM still didn’t work as advertised.

Well this book has the answers! I’m not going to lie, this is a serious book. But, I’m telling you, Reinertsen knows what he’s talking about.

I did a couple of blog posts on chapter two if you want to have a look:

And here’s a video of Reinertsen giving an overview of the ideas in his book. Great stuff!


Dark Territory: The Secret History of Cyber War

This is a chilling history of America’s cyber war activities and a look at the current battle space.

Make no mistake, cyber is just as much a domain as air, land, sea, and space. The military needs to control it if it wants to control the battlefield.

America is in a unique position. Because it has the largest and most technologically advanced economy in the world, it has the biggest attack surface. Some people in the government want to defend and harden the economy against attacks while others in the government want to collect exploits so they can use them in offensive operations.

Spying is rampant (even among allies). Corporate IP theft is rampant. Nations are hacking nations. And nobody seems to have a coherent plan to protect themselves.

Our technology has reached the point where a strong cyber strike could cripple a nation. And unlike a missile strike or bombing campaign, it’s very difficult to determine were it came from to know against whom to retaliate.

This book is especially relevant to software developers because we are the people writing the crappy, exploitable code that makes all this possible.


The Martian
Awesome fiction for your geeky soul.

I decided to read this book after I watched the movie and I wasn’t disappointed. The movie largely follows the book but on a compressed scale.

Instead of that two hour feel-good feeling you got from watching Mark Watney struggle to “science the shit out of this” in the movie, you get to experience that feeling the whole time you’re reading the book.

Excellent!


Daemon

Amazing Sci-Fi. The obituary of a tech titan triggers an AI he created to execute his plan from beyond the grave. It’s killing people, hacking, recruiting a citizen army, and we don’t know why. It’s always one step ahead of the FBI and all the other law enforcement agencies. As they struggle to comprehend what’s happening, the AI is moving pieces into place to obtain its primary objective.

It’s interesting to think the tech that’s just around the corner and how it can be used for good or evil. The book’s got AI, AR, self-driving cars, IOT, and so much more.

Highly recommended.


Freedom (TM) (Daemon Book 2)
Freedom is the second book in the series and you won’t want to miss it. It picks up right after the first book and the action is awesome and unrelenting.

I actually enjoyed the second book more than the first.

 

 

 

 

 

My rating: 4/5

Test Driven Development: By Example

What you think of this book will be heavily influenced by your views on TDD and your level of skill level as a TDD practitioner.

This book is aimed at TDD beginners and I think it serves that target audience well. Beck resolves lots of misconceptions about TDD. And he shows you how to actually go about applying those three simple TDD rules with realistic and lengthy examples.

If you are an intermediate or advanced TDD practitioner, you will not find much new material in this book.


America’s War for the Greater Middle East: A Military History

Have you ever wondered why the American military is permanently camped in the middle east? Have you wondered which half of the crap on TV about the middle east is true? Or how does this all end?

I sure have. Bacevich lays it all out and it’s ugly and deeply concerning. If you have the slightest interest in America’s foreign policy, you should definitely check out this book.

 

 

 


Change Agent: A Novel
Set 28 years in the future, an Interpol agent is injected with a mysterious substance in a subway. He wakes up in a hospital handcuffed to the bed and everyone thinks he’s a notorious crime lord. His DNA and appearance have been altered, which is supposed to be impossible.

He spends the rest of the book trying to get his body and his life back and we get to come along for the ride.

In 2045 there’s advanced genetic engineering and medicine, autonomous military drones, self-driving cars, AR, VR, total surveillance, and massive numbers of climate refugees.

I liked the Daemon series better but this Suarez novel is well worth your time.


Spaceman: An Astronaut’s Unlikely Journey to Unlock the Secrets of the Universe
I’ve always been fascinated with NASA and the Astronaut Corp. What does it take to make it as an Astronaut? What do you have to do to get in? How do you get a mission? Mike Massimino tells all in this fascinating and entertaining story.

This man went “all in” over and over again for the slimmest chance to become an astronaut. Any rational person would have given up right at the beginning. But he persisted despite overwhelming odds, got into the program, and made it into space.

Inspirational.

 


Good to Great: Why Some Companies Make the Leap and Others Don’t
What’s the difference between a good company and a great company? This book aims to answer that question.

I really like Jim Collins’ work. This is good material for a software developer trying to get a better understanding of how successful businesses are structured and how they think.

Collins’ model of how to build a long-term successful company is very different from what you typically see in Silicon Valley and that’s a good thing.

This is an interesting read and I believe it will be well worth your time.


Great by Choice: Uncertainty, Chaos, and Luck–Why Some Thrive Despite Them All
In this book, Collins investigates the roll of luck (both good and bad) in running a successful company.

All companies have both good and bad luck but the most successful companies make plans to minimize the bad luck which will inevitably come and maximize the good luck should an opportunity present itself.

Just like in Good to Great, Collins shows that successful companies look very different than the typical Silicon Valley start-up. While these high-flying tech companies are getting all the press, the great by choice companies are quietly and methodically doing their thing and crushing the market and the competition in the process.

 


Lock In: A Novel of the Near Future
In the near future a highly contagious virus spreads across the world. Most people recover after experiencing something similar to the flu but a small percentage become trapped in their bodies unable to move a muscle (known as being “locked in”). Fast forward 25 years and the best solution anyone can come up with is to put computers in the victims’ brains and give them androids to act as their surrogates in the real world.

This brings up all kinds of interesting issues. What happens to your internet connected brain computer when the manufacturer goes out of business? Who does the security updates? How do you protect the software on really old computers from being hacked? How can you tell that the android you’re talking to is really being controlled by the person you think is supposed to be controlling it?

And then there are all kinds of issues with discrimination against persons with disabilities and the proper role of government in helping  find cures for serious illnesses. It’s an interesting read.


CyberStorm
What would happen if your communications technology and electricity all stopped working at once? No phones, no computers, no internet, no power–nothing. What if it happened in New York city right before a wicked winter storm?

The answer is that things get ugly. The civilized part of civilization goes away pretty almost immediately when the power goes out and people get cold, hungry, and desperate.

I couldn’t put this book down!

 

My rating: 3/5

The Difference: When Good Enough Isn’t Enough
The thesis of this book it that its very difficult to have a successful company if your people don’t give a crap.

The author suggests that you need to adopt a ‘caring mindset’ if you really want to make a difference.

A caring mindset is achieved through practicing the STAR principles of being: straightforward, thoughtful, accountable, and having resolve.

That’s a totally fair position but I’m not sure this is an idea that needed a whole book. Some Amazon reviewers talked about the book being one big advertisement for the author’s services and I can’t disagree with them.


Lean Six Sigma For Dummies
Lean was born in Toyota and Six Sigma came out of Motorola. This book introduces a mashup of the two methodologies.

I read this book hoping to find a way to improve the effectiveness of our software development efforts but it isn’t very applicable to software development.

If you work in an environment where other departments are doing one or both of these methodologies, then you might get a good introduction in this book. Otherwise, I’d say you are much better off reading The Principles of Product Development Flow.

 


Disrupted: My Misadventure in the Start-Up Bubble
Find out what it’s like to work for a tech company run by people who don’t have the slightest idea of how to be mature adults. Funny, cringe-worthy, and sad all at the same time.

It’s a cynical take on start-up “culture.”

 

 

 

 


Trapped Under the Sea : One Engineering Marvel, Five Men, and a Disaster Ten Miles Into the Darkness(Hardback) – 2014 Edition
This book is not directly related to software development.

I’ve been reading books on engineering to see how other disciplines deliver solutions to complex problems. This is the story of a team of divers that attempted a one-of-a-kind mission deep into a tunnel in the bedrock below the ocean floor.

The mission failed and two divers died. There were plenty of signs that the mission was unlikely to succeed but time pressure, greed, and wishful thinking on the part of many people kept the project moving forward.

There are many parallels with doomed software projects here.


Ingenious: A True Story of Invention, Automotive Daring, and the Race to Revive America
This is the story of the X-Prize hyper car competition. The author followed several teams as they prepared for the competition, competed, and a little bit of follow-up afterwards.

I’m always interested to see how people come together to make something and how I can use that information to make better software. Unfortunately, I didn’t learn much I could use in this book.

It’s an okay read, at best.

 

 


The Art of War Publisher: Shambhala
This book is a classic among the business people who see business as as a metaphor for war. My professors mentioned it now and then when I was in business school. I came across it at my library and thought I would check it out.

It’s not the first book I would recommend for a software developer looking to understand business. Hell, it’s probably not the 100th.

The book is filled with cryptic almost riddle-like wisdom about waging warfare in China several hundred years ago. Unless that’s your thing, look elsewhere.

Cost of delay: how to calculate quantitative cost of delay and CD3

In my last post, I introduced you to qualitative cost of delay and CD3. I argued that rational teams should order their backlog for maximum economic benefit and that you can use cost of delay divided by duration (CD3) to do that. Because software developers aren’t accountants, I used the easier to understand qualitative cost of delay and CD3 in my examples. But in this post I’m going to discuss at the advantages of switching to quantitative cost of delay.

I know this stuff isn’t as interesting as reading about the latest advances in AI or your favorite framework. But trust me when I say that learning to order your backlog by quantitative CD3 will help you take your effectiveness to the next level.

Why calculate quantitative cost of delay?

Qualitative cost of delay just gives us a unit-less score. It’s a crude measure of the urgency and value of a story. When you divide it by duration, you can use it to roughly order your backlog but that’s about it. Quantitative CD3 is more accurate than qualitative CD3 for ordering the stories in your backlog, which is a huge benefit. But that’s just the beginning of what you can do with quantitative cost of delay.

Make better tradeoffs using quantitative cost of delay

You can use quantitative cost of delay to make rational decisions about the hundreds of little and big tradeoff decisions your team makes every day.

For example, is it better to:

  • release a new version of your software today or spend another month fixing bugs?
  • hire another software developer or work with the people you have?
  • refactor a messy module now or leave it for another time?
  • update the slightly misleading variable name so the code is a little easier to read for the next guy or leave it?
  • keep your people busy or reduce their utilization and focus on reducing cycle time?

You can use quantitative cost of delay to help you answer these kinds of questions and more. I plan to show you how to do that in a future post.

Speak the language of business

Money is the language of business. If you want to get your ideas and changes implemented, show the decision makers in your company the economic benefits of your ideas.

Want to tackle some technical debt? Show them how much it’s costing your company not to tackle it. Want buy some software make something go faster? Show them the numbers. Want to spend some time improving your security? Show them the impact.

How to calculate quantitative cost of delay

Okay, so how do you actually calculate quantitative cost of delay? The short answer is that you calculate the marginal impact on after-tax lifecycle profits of your potential actions.

Don’t panic. I promise that it isn’t as bad as it sounds. I’m going to break this into small steps for you. First, I’m going to present you with some general advice. And then I’ll show you some examples.

Partner with an expert in your company to help you calculate the cost to delay

You will likely want a partner to help you figure this stuff out. This might be the product owner, somebody from accounting, or somebody with specialized knowledge of the economics of your projects. Business people are used to thinking in dollars and cents and you shouldn’t have too much trouble getting them to help you once you explain what you are trying to do.

Use an appropriate level of effort and rigor

You need a different level of rigor in your cost of delay calculations for developing all the software for a new fighter jet as compared to adding a newsletter sign up form to your company blog.

Use common sense and stop when you have answers that are good enough. For example, if the top three stories in your backlog have CD3s that are orders of magnitude larger than the other stories further down in your backlog, you probably don’t need to analyze those other stories now.

In most scenarios, the marginal impact on after-tax lifecycle profits of a story are going to be overwhelmingly driven by only a few factors. This means you can use simple models to get answers that are good enough.

Tip: you can use the Theory of Constraints to guide you to potentially valuable stories, which are usually related to your system constraint.

Use a spreadsheet

If you build your cost of delay models in a spreadsheet and make your assumptions explicit, you’ll find it easier to update and change your assumptions as new information becomes available (see the example below).

The importance of marginal economics

You only want to consider costs and profit that will change with your decision. Lots of people, even business people, make mistakes here. The biggest mistake people make is when they consider “sunk” costs. Sunk costs are costs that you’ve already incurred (or that you haven’t yet incurred but you can’t avoid).

Let’s look at a quick example. If you’ve worked on a feature for five weeks and you’re got two more weeks to go, those first five weeks are a sunk cost. This has an interesting effect on your CD3 over time. If you assume a fixed cost of delay of say $5,000 per week, you might start with a CD3 of $5,000/7 weeks = 714. Five weeks into the projects, you only have two weeks remaining so the math looks like this: $5,000/2 weeks = 2,500. As the duration remaining decreases, your CD3 score will approach infinity.

This makes sense when you think about it. Would you ever shelf a feature that will make $5,000/week in after tax profit that you could finish in one hour? I wouldn’t (unless another story had a higher CD3). Let’s work out the CD3. One hour out of 40 hours in a work week = 0.025. So, $5,000/.025 = 200,000. That is a very different answer than the CD3 of 714 you get when you calculate the non-marginal CD3.

Here is the same data displayed graphically (with CD3 rising as duration remaining decreases).

marginal cd3 vs non-marginal cd3

Lifecycle profit vs cashflow

Most of the time you should be interested in lifecycle profits when you are calculating quantitative cost of delay. However there are cases where cashflow is more important. Companies with extremely high growth rates or companies fighting for their survival may need to worry about cashflow more than lifecycle profits.

If you’re in this situation, the smart move is to sacrifice some long term profitability to make sure you can meet your short term financial obligations (like making payroll and paying your suppliers). You’ll likely know if you’re in this situation but if you aren’t sure any business type in your company will definitely know the answer. Adjust your cost of delay calculations to meet your company’s goals as required.

An example of calculating quantitative cost of delay and CD3

Suppose you work for an e-commerce website and you are considering two new features and you want to determine which feature has the higher priority using quantitative CD3:

  • Feature A: streamline your checkout pages to increase their conversion rate.
  • Feature B: update your tax rate calculations so you can expand into a new market with a different tax rate.

I made a spreadsheet to calculate the cost of delay and CD3. It looks like this:

spreadsheet of quantitative cost of delay for two features

With the assumptions I used, feature A has a CD3 score of 300 and feature B has a CD3 score of 7,200. So if everyone agrees with those assumptions you should work on feature B first.

But, everyone might not agree. And if that’s the case, you can easily tweak your assumptions (yellow cells) and see how they effect your outcome (purple cells). Software developers are already familiar with estimating duration. This is just a small extension of that idea.

You can take another approach too. You could ask how much the conversation rate on feature A would have to rise to make its CD3 score the same as feature B’s CD3 score? Let’s see what that looks like.

change in conversion rate required to have equal cd3 scores

I punched in a few values for the “new conversion rate” and found that it would have to be 52% to have the same CD3 score as feature B. I’ve never heard of a 52% conversion rate for a long term e-commerce site. So, I’d say that it’s a pretty safe bet that feature B is a higher priority than feature A.

Wrapping up

I know you might not be excited about spending your time calculating cost of delay and CD3. But your team will have a very difficult time being effective unless somebody’s doing it and you’ll find it also gets easier with a bit of practice. Our intuition for finding and working on the most valuable thing isn’t good enough to wing it. You’ll be shocked by what you find if you spend some time quantifying the cost of delay for your high priority backlog items.

In my next post in this series, I’ll show you how to deal with stories where their urgency changes over time. Stay tuned.

Agree, disagree, comments or questions? I’d love to hear from you.

Additional resources

You might find the following cost of delay resources helpful:
* excellent website with a great three minute explainer video
* book: “The Principles of Product Development Flow: Second Generation Lean Product Development” (Donald Reinertsen) Chapter 2
* book: “Essential Scrum: A Practical Guide to the Most Popular Agile Process” (Kenneth S. Rubin) Chapter 16
* video of a talk by Donald Reinertsen on product development

Cost of delay: prioritize your product backlog by CD3

Ordering your backlog for maximum economic benefit is a hard problem. The average team is being pulled in several directions at once. New features requests, technical debt, existing defects, non-functional requirements, security, risk mitigation, resource shortages, and more are all competing for your team’s attention. So, how does a rational team decide what to work on? In this post, I’m going to show you how to prioritize your product backlog by CD3, which is cost of delay divided by duration.

This post is part of a series on cost of delay. You can find the first post here.

What’s the problem?

Have you ever noticed that scrum and agile writers have vague descriptions of how you should order your backlog? They write stuff like “work on the highest ROI stories first” or “the product owner should prioritize the backlog by customer value.” That kind of advice sounds good when you read it but it’s not going help you answer my next question.

If you only had the following three stories in your backlog, how should they be ordered?

  1. encrypting your users table and updating the security of your password hashes to the current best practice
  2. removing an unused feature from your code base that’s adding to the complexity of some really important code
  3. automating the creation of a report for your sales manager

Which story is the highest priority? To whom? Your insurance company would probably pick story 1. You’d probably pick story 1 or 2. And your sales manager would probably pick story 3, right? So where do you go from here? See who can shout the loudest? Pull out your org chart and go with decision of the person who is closest to the top? Rank the stories by ROI (return on investment)? FIFO (First in, first out queue)? Ouija board?

You should prioritize your product backlog by CD3

The value of the stories in your backlog can span several orders of magnitude. And people have very poor intuition about the value of any given story. So, no amount of effort, skill, or talent is going to get you where you need to go if can’t identify the highest value stories and get them to the top of your backlog.

There is no consensus on the Internet about how you should prioritize your product backlog. And I actually cringed as I read some really horrible advice in articles with very good search engine rankings. I looked at a couple of books too and they weren’t much better.

It’s impossible to compare the value of radically different stories because their values are expressed in incomparable units of measure. To get around this problem, you can convert everything to money using cost of delay. And then can divide the cost of delay by duration to calculate CD3. The higher the CD3 score, the higher the story should be in your backlog.

How to prioritize your product backlog by CD3

I’m going to show you how to calculate the qualitative cost of delay and CD3 for the three stories I wrote about above. Read this excellent article on qualitative cost of delay and come back here.

Okay, you’re back. Did you read the article? If not, go read it now. What follows isn’t going to make sense to you unless you read the article.

We can’t do math on text labels so I assigned them integer values as follows:

scoring value, urgency, and duration for CD3

I calculated the following cost of delay and CD3:

CD3 calculations for three stories

Let me explain what’s happening here.

Encrypt users table and upgrade password hashes

I assigned this story a killer value (8). If we get hacked like Ashley Madison we will literally lose our business. So this is a very serious situation. However, getting hacked isn’t guaranteed. I estimated that we have a 1/20 chance of being hacked so I divided 8 by 20 and get a value of 0.4. I ranked this story’s urgency as ASAP (8) because it is very perishable; it will have no value after we’ve been hacked. I multiplied the value by the urgency to get a cost of delay of 3.2. And I estimated that it would take one week to complete the story. So CD3 = 3.2/1 = 3.2.

Automate the sales report creation

The sales manager has a member of her staff create the report by hand. It takes one hour per week. So automating this task would save about $40/week. This story isn’t very valuable or urgent. We expect the cost of creating this report to remain fairly flat for the foreseeable future. I estimate that this report will take half a week to automate. So CD3 = 1/0.5 = 2.

Remove unneeded feature

This is a technical debt story. Removing this feature will help us reduce the complexity of important code in our project. And since complexity scales non-linearly, removing this feature should help us move faster in the future. But when I read the value and urgency descriptions, I have to admit that it is meh-level (1). And that that urgency is whenever (1). It’s pretty difficult to remove code that runs through the heart of the system. So, I estimate that it will take two weeks to get it out of there. So we get a CD3 = 0.5.

Shifting the conversation

So how do you feel about these rankings? Do you want to argue that the risk of being hacked is 1/50, or 1/100? Fine. Do some planning poker with your team and come up with a more realistic value. Do you think that removing the unneeded feature duration estimate is too big? Okay, let’s talk about it. If you don’t like the 1, 3, 8 values for value and urgency, then change them to something more appropriate.

My point is that we’re shifting the conversation from who’s highest on the org chart or which story was in the backlog first to a framework where we can rationally break things down and try to figure out what’s actually going on.

The benefits of being able to justify the order of your product backlog

I can easily imagine a situation where the sales manager starts to make arguments to try to get the report automation story to the top of the backlog. “The report is hard to create manually.” Or “It’s been in the backlog for 5 months and I’m losing my patience.” Maybe she phones and emails you multiple times per week to say how much she needs this report automated.

If you have a cost of delay and CD3 calculation readily available you can defend yourself against these kinds of pressures. You show the sales manager your spreadsheet and that her story has a CD3 = 2. And she says, “You don’t get it. My staff member made two mistakes in creating that report in the last year. One cost us $30,000 and the other one cost of $10,000. I don’t want it automated to save time. I need it automated to prevent costly errors!”

Did you see that coming? A miscommunication led you to make a bad assumption. Would you care to recalculate the CD3 of that story and reorder the backlog?

Don’t worry about the accuracy of your inputs

Don’t worry about getting super accurate inputs into the framework. Reinertsen says our intuition about this stuff is so bad that any reasonable estimates will improve outcomes.

Duration

You want to know how many weeks (on the calendar) it will take to deliver each story. This is the sum of the actual time people spend working on the story and all the time the story is waiting in various queues. This is probably a departure from the way you normally measure effort but it’s important in the context of calculating CD3.

Qualitative vs. quantitative cost of delay

When you are getting started qualitative cost of delay is easier for people to understand than quantitative cost of delay. But you should consider switching to quantitative cost of delay as soon as possible.

Qualitative CD3 can create distortions in your assessment of value and urgency that lead you to make suboptimal decisions about the order of the stories in your backlog.

Qualitative CD3 is a score and is only useful for ranking the stories in your backlog. On the other hand quantitative cost of delay and CD3 are rates. And they allow you to make economically optimal decisions about the cost of queues, staffing, and many other decisions that we usually make poor guesses at while we are developing software.

Don’t make it more complicated than it needs to be

You can really go down the rabbit hole with this stuff but I don’t recommend you do that unless the value clearly outweighs the cost.

It’s totally common for the values of your inputs to change based on how you order your stories because the stories are interdependent.

For example, if you put a bunch of technical debt repayment stories at the top of your backlog, the duration of many of the other stories below them in the backlog should decrease. Conversely, if you add a big new feature to a certain part of your application, you might increase the complexity enough to make another feature in that same area much more difficult to implement.

Value and urgency can change over time as well.

Practical advice

Step 1: Ensure all the stories anywhere near the top of your backlog won’t result in worthless software or waste.

Step 2: Rank the stories with quantitative or qualitative CD3. Since the value of your stories typically span several orders of magnitude, it shouldn’t be hard for you to identify the really high value work. As long as you update your variables and recalculate your CD3 periodically (depending on how fast your variables are changing), this ranking should be good enough for the vast majority of projects.

Step 3: (rarely needed) Consider interactions between variables to fine tune the order of your backlog.

Wrapping up

You can’t be effective if you are working on unimportant things. Experience shows that our intuition about which stories are the most economically valuable is shockingly poor. But you can use CD3 to create a rational basis for the ordering of your product backlog. It helps you shift the conversation from opinions and seniority to value, urgency, and hidden assumptions.

Cost of delay isn’t just about prioritizing your backlog. I’ll expand on these ideas and show you more complicated examples in follow-up posts. Stay tuned.

Additional resources

You might find the following cost of delay resources helpful:

  • excellent website with a great three minute explainer video
  • book: “The Principles of Product Development Flow: Second Generation Lean Product Development” (Donald Reinertsen) Chapter 2
  • book: “Essential Scrum: A Practical Guide to the Most Popular Agile Process” (Kenneth S. Rubin) Chapter 16
  • video of a talk by Donald Reinertsen on product development

Cost of delay: simultaneous feature development

Simultaneous feature development is a common but often economically sub-optimal software development practice. In this post, I’m going to look at the financial implications of simultaneous feature development and show you what you can do instead to achieve better outcomes.

What is simultaneous feature development?

Simultaneous feature development is a common software development practice where a team works on more than one feature at the same time. For example, if you are doing scrum, you’ll pull a bunch of stories into your sprint. And everyone is more or less happy if you complete all your stories by the end of the sprint. Your team has no limits on work in progress. So you work on whatever’s convenient.

Teams engage in simultaneous feature development to keep everyone on the team busy all the time, which is the incorrect variable to optimize.

Why should you care about simultaneous feature development?

The problem with simultaneous feature development is that by working on multiple features at the same time, you are delaying the delivery of every feature. And if those features increase revenue or decrease costs, you are costing your company profit. So, your team might be hitting all its velocity targets, quality targets, and getting the most important work into your sprints and still be missing out on substantial profit opportunities.

Of course your team might have very good reasons to develop features in parallel. You might have a large team and it just doesn’t make sense to have 50 people working exclusively on a 200 line feature. Alternatively, you might have a deadline for delivering a whole basket of functionality and the only way you can meet that deadline is by working in parallel.

How you can improve your outcomes by using cost of delay

You can’t dogmatically apply a “no simultaneous feature development” rule. You need to calculate your cost of delay and choose the path that makes the most financial sense.

An example

Suppose you are starting a new four week sprint and you have 4 features to complete. Each feature will take one week to develop. And you lose $5,000 every week you delay the launch of each feature–this is the cost of delay.

Here’s the financial impact if you develop all your features simultaneously and deploy them at the end of the sprint.

simultaneous feature development

And here’s the financial impact if you develop your features sequentially and deploy them as soon as they are done.

sequential feature development

You can have a huge impact on the profitability of your project just by making a simple change to the way you schedule your work. In this scenario, you can decrease the cost of delay by 37.5%! ((50K – 80K)/80K = 37.5%)

Do you see how huge this is? You don’t need to work any harder, hire more people, or learn functional programming. All you have to do is schedule your features to minimize cost of delay.

Key insight: waiting has a cost attached to it. If your feature takes 50 hours of developer time to complete but it spends several months waiting in various queues before it is finally delivered to your customers, the true cost of that feature could be many multiples of the 50 hours of programmer wages. Most businesses are focused on that 50 hours of programmer time and making sure all their people are running a very high utilization rates. But they are completely blind to the potentially tens of thousands of dollars they lost because the feature took months to deliver when it could have been delivered in days.

Wrapping up

I was completely blind to this huge opportunity to increase profitability until I learned about cost of delay. We were happy if we worked on high priority tasks in our sprints and met our velocity targets. But we were definitely leaving a lot of money on the table. We would frequently allow stories to linger in multiple sprints if they happened to fail multiple code reviews. Those stories could block whole features for weeks on end. But it never occurred to us that we were bleeding money.

Calculating cost of delay and using it to schedule our work required us to make some adjustments but it is so worth the effort. I’ll expand on these ideas and show you more complicated examples in follow-up posts. Stay tuned.

Additional cost of delay resources

You might find the following cost of delay resources helpful:

  • excellent website with a great three minute explainer video
  • book: “The Principles of Product Development Flow: Second Generation Lean Product Development” (Donald Reinertsen) Chapter 2
  • book: “Essential Scrum: A Practical Guide to the Most Popular Agile Process” (Kenneth S. Rubin) Chapter 16
  • video of a talk by Donald Reinertsen on product development

A love letter to my IDE

I am not a poet nor am I prone to making public displays of affection but I can’t help myself. I love you so much.

In the beginning

I’ve always had some kind of tool to help me write code. I’m almost embarrassed to say that I started my life as a programmer with a syntax highlighting text editor and an FTP program. I’d write my code, save it, FTP it to the server, and then run it to see if it “worked”. I know that sounds crazy now but it was the turn of the century and I was young and didn’t know any better.

My first IDE

After a few years of programming like that, I switched to my first IDE. Its capabilities blew me away. I loved its automatic code formatting, intuitive code navigation, auto-completion, Subversion integration, and that it was free. But it wasn’t a perfect relationship. My IDE was slow, ugly, and hard to use at times. We had more than a few rough patches when it threw a tantrum and refused to comply with my commands. The more I grew as a programmer, the more I struggled with its limitations.

When I met you

And then I found you and I fell in love at first sight. When you imported my project from Git and were up and running within minutes, I knew I made the right decision. I dumped my other IDE immediately and never looked back. You can do everything my other IDE could do and so much more. You are worth every penny.

How you make my life easier

You understand that I, like all humans, have extremely limited cognitive capabilities. My working memory is pathetically small and my ability to think is limited and easily exhausted. Unsurprisingly, I make mistakes when I get tired. But you do whatever you can to reduce my cognitive load and that makes us excellent partners.

Your basic features are perfection

For example, if I just type the first couple of letters of something, you suggestion completions. And when I’m trying to remember what arguments to give to a method call, you helpfully show them below my cursor. You’re amazing at refactoring–the best I’ve ever seen. And when I need a hand remembering how something works, documentation, method signatures, and more are just a hotkey away.

You helped me standardize the formatting on my project and wrangle my line endings, file encoding, and convert my tabs to spaces. You take care of all the brackets and code formatting so I can focus on the logic and problem I’m trying to solve.

Your search and replace functionality is powerful, configurable, and lightning fast. I couldn’t imagine it being any better.

Inspections and static analysis

I love your code inspections. You constantly scan my code and point out errors or possible problems. It was humbling to see how many problems you found with my code when I first ran your inspections but you never gloated.

Yet, as good as your inspections are, you also support my favorite third-party static analyzers right out of the box. And I don’t mean a button I can press to run the other static analyzers in a terminal. I mean full integration so they run automatically and appear on my screen like native inspections. Brilliant.

I love that you are helping me rid my world of the ternary operator. It’s a conditional disguising itself as an assignment and I think it’s evil. I know this is my personal pet peeve and I’m grateful that you are configurable enough to find it and flag it any time it appears in my code.

inline if statement not allowed example

Git integration

You help me get along with Git, which is no small feat. When we look back at Git in thirty years, we’ll be baffled as to how such a terrible tool became the dominant version control system. Can you think of another piece of popular software that’s harder to master? Anyway, Git is the best VCS for my needs right now and you help smooth off the rough edges and hide the hideous details.

Your graphical merge conflict resolution feature is amazing. And I can’t imagine what I’d do without the plug-in that periodically fetches from origin and unobtrusively lets me know what’s changed. Thank you.

Testing support

You were built with testing in mind. I can run my unit tests at the touch of a button or hotkey. I can easily run all my tests, or just a folder, or just a file, or just one test or just one test case. You also support more complicated testing setups thanks to your support for unlimited test suite configuration files. And when I wanted to experiment with property-based testing, you were super-cool about it. That’s flexibility.

You are so easy to use

I don’t personally know the people who designed and wrote you but please pass along my compliments. They spared no expense in making you insanely as easy to use, highly configurable, and extremely powerful all at the same time. You get better with every update and I can’t wait to see everything you’ll be able to do as you continue to improve over the next five to ten years.

You’ve helped me increase the quality of the code I write and the speed with which I write it. So, even though we’ve only been together for a couple of years, I can’t imagine my life without you.

Love,

Blaine

With great power comes great responsibility

I recently got into a discussion in the comments section of someone else’s blog where I argued that many software developers are overly confident in their abilities. I further argued that this overconfidence leads to the kind of behavior that would be considered professional malpractice if a lawyer, doctor, or professional engineer did something similar in their respective field.

A fellow software developer expressed an opposing view. He made the following points:

  • only a small percentage of software can cause as much damage as medical or legal malpractice and that software is highly regulated
  • if we stop people from pursuing their interests it will stifle innovation, which he was very much against

I hear variations on this view quite often and I think it is worth exploring.

Software as a force for good

Software has enabled our modern world. We can communicate with anyone virtually anywhere in the world for free or very low cost. It puts the world’s knowledge at our fingertips. It multiplies our productivity, manages our supply chains, opens up new markets, and keeps track of our money. We use software to discover new science, improve our health, and fight disease. Plus, software reduces the cost of just about everything.

And we, the software developers, are right in the middle of it and it’s glorious!

But we do have some problems

Let me paint a picture for you:

  • the industry average is 15-50 errors per KLOC for delivered software (that’s one error every 20-67 lines of code on average!)1
  • only 29% of projects completed in 2015 were considered successful (on time, on budget, with a satisfactory result). 19% of projects outright failed and the rest were somewhere in between2
  • software failures cost hundreds of billions of dollars EACH YEAR3
  • 90% of mobile health and finance apps are vulnerable to critical security risks4
  • consumer routers. Need I say more?5

Do you see what I’m getting at here? As a profession, we’re not exactly knocking it out of the park on every at bat.

Software developed for unregulated environments can’t cause that much damage. Really?

I don’t think that’s a defensible position. We (software developers) are creating plenty of harm in unregulated industries through our mistakes and negligent practices.

While our software probably isn’t directly responsible for killing people very often, I have no doubt we are indirectly responsible for countless deaths. Plus we enable property damage, theft of personal data and intellectual property, financial ruin, lost productivity, privacy violations, voyeurism, stalking, blackmail, discrimination, loss of reputation, interference in elections, espionage, and all kinds of criminal enterprises.

I can come up with links if you don’t believe me or you can just take a look at the thousands and thousands of entries in the ACM’s Risks Digest database. Here’s just a taste from recent issues:

I purposely chose examples from unregulated industries to illustrate a point. We don’t have to build drones with guns mounted on them or faulty autopilots that fly planes into mountains to hurt people and cause serious damage.

I know we kind of expect software to be terrible but keep in mind that none of these things are supposed to happen if we are doing our jobs correctly!

Evading responsibility

I expect that someone will want to split hairs in the comments about email not being secure and it not being the programmers’ fault that someone broke into the real estate agent’s email account and impersonated him because his password was “password123456”. And that might be true if you’re looking at an individual developer. But we (software developers) know how people are using email and we know better than anyone that it’s not secure but we’re doing very little about it.

We can also consider another plausible scenario. Perhaps the real estate agent created an account for some harmless website. Perhaps the website didn’t store their user’s passwords securely. Further imagine a copy of the website’s database ended up in the wrong hands and the criminals either just read the clear text passwords straight from the database or broke the unsalted MD5 hashes and recovered the password that the real estate agent used for all of his accounts.

Here, again, we know people re-use passwords and we should know better than to store them in clear text or unsalted hashes, even if our website doesn’t contain particularly sensitive information. But this could never happen, right?

The software we create is powerful, which means it can also be dangerous. So, you need to be thinking about security and the unintended consequences of any software you produce. Security isn’t just for sensitive projects.

Stifling innovation?

The claim here is that I somehow want to stop new people and ideas from entering the field and stifle innovation. I haven’t proposed any actual action and I have no power in the industry so I’d say that my power to stifle innovation is pretty minimal.

But let’s say I did propose something for the sake of argument. Maybe you can’t work on software where you’d have access to personal information if you’ve been convicted of identity theft or computer crimes. Is that an unreasonable rule where innovation is concerned?

How about this one: if you want to work with money or personal information in a system that’s connected to the Internet, you have to pass a simple test. Passing the test would show you have a basic grasp of security principles (TLS, password hashing, SQL injection, cross site scripting, maybe that it’s a bad idea to post your private encryption keys on your blog, etc.). And let’s throw in some ethics while we’re at it. Unreasonable?

I can’t think of any reason why a person who is capable of creating innovation in any area of software development would have any trouble demonstrating his or her basic competency as a programmer. Nor do I believe a reasonable testing process would stifle innovation.

Unleashing innovation?

What if keeping certain people out of software development increases innovation? We know there are huge differences between the productivity of the best and the worst developers–5-28 times by some estimates6.

So what if we had some licensing mechanism to keep the worst of the worst from becoming professional software developers. Instead of having them bounce from job to job causing chaos wherever they go, maybe we could create our own version of the bar exam or something but set the bar really low. The mechanism and details aren’t important right now but just imagine if you never had to deal with another net negative producing programmer for the rest of your career. How much more could you accomplish?

Wrapping up

Pretty much every jurisdiction in the world requires people to demonstrate basic competency to drive a car. While cars are incredibly useful machines, they can also be incredibly dangerous. Ensuring the basic competency of all drivers is in the public interest. Likewise, I’d argue that ensuring the basic competency of computer programmers is also in the public interest for similar reasons.

Whether you agree with my view or not, licensing software developers is not going to happen any time soon. So, go build the next amazing thing! Please just keep in mind that any software worth building can also cause harm. So, I hope you’ll skim the Risks Digest and maybe think about the specific risks of your software and how you can minimize them.

With great power comes great responsibility.

What do you think? Agree or disagree? I’d love to hear your thoughts.

Additional resources

Here are some links you might enjoy if you want to dig a little deeper:


  1. Code Complete (Steve McConnell) page 610. If you have a 100,000 line application it will contain 1,500-5,000 errors! 
  2. Standish Group 2015 Chaos Report 
  3. The estimates are all over the map but they are all HUGE. See Software failures cost $1.1 trillion in 2016 and Worldwide cost of IT failure: $3 trillion 
  4. 90% of mobile health and finance apps are vulnerable to critical security risks 
  5. The quality of consumer routers is brutal in just about every way imaginable. And people can do bad things to you if they break into your router, really bad things 
  6. Facts and Fallacies of Software Engineering (Robert Glass) Fact 2. But I’d actually argue that the spread is infinite. I’ve met programmers who couldn’t solve a toy question in a screening interview. So their productivity is going to be zero (or negative), which makes the spread infinite. 

Worthless software: risks and prevention

We (software developers) write astounding amounts of worthless software. I find it hard to fathom just how much junk we are producing. As someone who spends a fair bit deal of time thinking and writing about how to be a 10x programmer and effectiveness, I believe we have lots of room for improvement here. In this post, I’m going to examine the problem of worthless software and what you can do about it.

How much worthless software are we producing?

Tons. The Standish Group 2015 Chaos Report, concluded that only 29% of projects completed in 2015 were considered successful (on time, on budget, with a satisfactory result). 19% of projects outright failed and the rest were somewhere in between. Ouch. But project failures aren’t the only kind of worthless software.

Features users don’t use or care about

None of the Chaos Report numbers take into account all the stupid, useless, rarely used, and unused features in successful projects. (I’m looking at you, Clippy.) But seriously, look through the menus in Microsoft Word or Excel. Do you even know what half of those things do? I don’t. Can you imagine being the Microsoft developer who maintains the kerning feature in Word? It’s hard to get a good handle on the number of features that fall into this category but I feel confident saying that it is well above 25%.

Features that cost too much

And then there are the successful features within a successful project that cost way too much to deliver and maintain. I don’t have a statistic for you on these costly software engineering failures but I think we can agree this is the rule more often than the exception.

Unneeded code

Plus, you have all kinds of code that doesn’t need to be in the project at all. I’m talking about dead code, automated tests for said dead code, duplicated code, over-engineered code, piles of code that can be replaced by one or two standard library calls, etc.

Nonviable business model

We must also count all the software produced by failed startups. The vast majority of that software dies with the startup that wrote it.

But it gets worse, so much worse

You don’t just pay for worthless software when you create it. You pay for it in each phase of the development process forever. So you might look at a feature that you wrote that turned out to be worthless and think it took 100 hours to implement, no big deal, right? Wrong.

Measuring the true cost of worthless software

If your team is like my team, you record your time spent directly building, testing, integrating, and releasing a feature but you don’t count the cost of:

  • the initial discussion and investigation of the feature
  • all the time you spend talking about it in meetings
  • reporting on its progress
  • reading the code when maintaining other parts of the system
  • coupling other parts of the system to the feature (while you attempt to keep your software DRY)
  • people leaving your team because your software is bloated mess and a nightmare to work on
  • hiring new programmers
  • bringing new programmers up to speed
  • upgrading your code to deal with deprecated libraries, language features, or APIs
  • keeping your code styled and formatted like the rest of your project
  • tracking, isolating, and fixing defects
  • extending the code
  • waiting for its tests to run a thousand times a day when you are working on other features
  • trying to maintain and evolve your architecture
  • increasing the attack surface of your system to malicious users
  • supporting the feature until the end of time (almost always happens) or porting it to the next version of the software (frequently happens) or eventually removing the feature (almost never happens)

Complexity

As if all this stuff isn’t bad enough, software complexity scales non-linearly. So, after you reach a certain level of complexity, every worthless feature you add to your project makes it significantly harder to work on the rest of your software for the life of the project. Let that sink in for a minute.

Opportunity cost

And the final kick in the groin is the opportunity cost of your worthless software. What could you have done instead of pursuing and maintaining this worthless software?

So, what’s the total lifetime and opportunity cost of building, supporting, and maintaining that “100 hour” feature? Who knows–nobody keeps track. It’s almost certainly more costly than anybody would be comfortable admitting.

How to keep worthless software out of your project

I hope I’ve sufficiently scared the crap out of you. Keeping worthless software out of your projects is a major engineering challenge. Software is so easy to change compared to hardware or physical things like buildings or cars, that people find it impossible to resist the temptation to change it. So, you can’t expect your stakeholders to stop making change requests because that’s not going to happen. What you need is a system for dealing with change requests and filtering out the ones that will lead to worthless software.

This is a huge leverage point. If you can consistently filter out ideas for worthless software from your project before you build them, you are so much better off than trying to figure out how to create software more efficiently. And think about how much time you devote to that.

I don’t have a silver bullet here. But I do have some ideas for how you can keep worthless software from creeping into the various stages of your development process.

Talk to your stakeholders about the total cost of ownership of worthless software

Your first step is just to talk about the problem of worthless software and the total cost of ownership with your stakeholders. Look at your project and identify the worthless software in it. How bad is it? In which development stage did you introduce most of the worthless software in your project? To what extent is your progress on high priority features slowed down by worthless software? How much effort are you expending to maintain this software?

Can you make a business case that it would be cheaper to remove some of the worthless software from your project than it would be to continue supporting and maintaining it?

Spike and stabilize

Instead of bringing every feature to 100% production quality before you release it, in this patten you produce a sort of working prototype of a feature and get it in front of your users as fast as you can. Then you gather feedback from your real users to determine whether the feature should be killed right there or receive additional investment to make it more complete and production quality. Dan North describes spike and stabilize a million times better than I can so just checkout his talk on the subject.

Always work on the highest priority tasks

You must advocate for working on only the highest priority tasks at all times. Add your change requests to your backlog but make sure the truly important stuff is always at the top. If someone pushes something potentially worthless to the top of the backlog, don’t argue against it directly. Pick the highest priority thing in the backlog and make your case for why task B is a higher priority than task A. If that’s not enough, you can “go negative” on the task you think will lead to worthless software and make a case for why it shouldn’t be at the top of your backlog. You won’t always get your way but if you are discussing things openly and honestly, you should keep the majority of the bad ideas out of your project.

Use your lack of staff to question every requirement

Almost all software projects are understaffed or behind schedule. And if you are in this situation, you can use it to your advantage to trim the “fat”.

Just because the overall task is important, doesn’t mean that every single part of it is also important. So you need to question every requirement. It’s way too easy for anyone to write a few words into a requirements document that turns into weeks or months of work or completely thwarts your architecture.

Find the “fat” and trim it wherever you can. And if you can’t trim it, at least schedule it in the later stories. With a little luck, your stakeholders will change their minds and drop or deprioritize the stories that include the “fat” before you implement them.

This strategy also gives you time to come up with good arguments for why the “fat” shouldn’t be in the project if those stories do eventually rise to the top of your backlog.

“Fat” in this case spans all types of requirements, not just stuff your users can see. Really think about if you need to add logging and email alerts for a new feature when you’re not even sure anyone will use yet. Does it really need to be that fast at the beginning? Documentation? A full suite of automated unit tests?

We’re trying to make “question every requirement” into an official step in our development process right now because we see it as such a high leverage point.

Use vertical slicing

We often split our features into multiple stories to align more closely with our optimal pull request size. We work to implement the high uncertainty and “must have” requirements in the earlier stories and work towards the “nice to have” requirements in subsequent stories. And it’s shocking how often one of two things happens. Either we completely drop the “nice to have” stories because something more important comes up and our efforts are redirected elsewhere. Or, we learn something by deploying the early stories that radically changes how we want to approach the later stories.

Either way, it helps us avoid adding worthless software to our project. And it will work for you too.

KISS and YAGNI

KISS means “keep it simple stupid” and YAGNI means “you aren’t going to need it.” You should burn these acronyms into your brain. Every line of code you add to your project is a liability. So, if you’re adding code “just in case” or because this is where the books told you should should use the observer pattern or for any other reason other than that you need it right now, I urge you to reconsider. It’s almost always pure downside.

Similarly, you should ignore most automation, infrastructure, and scaling concerns if you’re just starting out. Address them as they become inadequate, not before.

Further reading

Here are some resources you might find helpful:

Wrapping up

Because you can add worthless software to your project any stage of the development process, you need multiple strategies to combat it. It can be difficult to maintain this level of vigilance, but the cost of supporting and maintaining worthless software is so high that you can expend significant effort up front to keep that stuff out of your project and still end up ahead.

How do you keep worthless software out of your projects? I’d love to hear your thoughts?

« Older posts