YAGNI and the scourge of speculative design

Robert Anning Bell [Public domain], via Wikimedia Commons

I’ve been programming professionally for five years. One of the things that I’ve learned is YAGNI, or “You aren’t gonna need it”.

It’s taken me a long time to learn the importance of this principle. When I was a senior in college, I had a course that involved programming the artificial intelligence (AI) of a real-time strategy game. For our final project, our team’s AI would be plugged in to fight against another team’s. I got hung up on implementing a complicated binary protocol for the robots on our team to communicate efficiently and effectively, and our team ended up doing terribly. I was mortified. No other team spent much time or effort on their communication protocol, and only after getting everything else up and running.

In this essay I’ll primarily be talking about producing code that’s not necessary now, but might be in the future. I call this “speculative design” and it’s what the YAGNI philosphy prevents.

First, let’s discuss how and why this speculative design happens. Then we’ll discuss the problems with giving into the temptation.

Why does it happen

I can only speak to my own experience. The times I’ve fallen into this trap can be classified into a few categories:

• It’s fun to build new features
• It feels proactive to anticipate needs
• Bad prioritization

Building features is fun

Programming is a creative outlet. It’s incredibly satisfying to have an idea, build it in code, and then see it in use. It’s more fun than other parts of development, like testing, refactoring, fixing bugs, and cleaning up dead code. These other tasks are incredibly important, but they’re ‘grungy’ and often go unrewarded. Implementing features is not only more fun, it get you more visibility and recognition.

Proactive to anticipate needs

A second reason one might engage in speculative design is to be proactive and anticipate the needs of the customer. If our requirements say that we must support XML export, it’s likely that we’ll end up having to support JSON in the future. We might as well get a head start on that feature so when it’s asked for we can delight the customer by delivering it in less time.

Bad prioritization

This is the case with the story I started this piece with. I overestimated the importance of inter-robot communications and overengineered it to a point where it hurt every other feature.

In this case, the feature was arguably necessary and should have been worked on, but not to the extent and not in the order that I did. In this case I should have used a strategy of satisficing and implemented the bare minimum after all of the more important things were done.

Why is it problematic

I’ve described a few reasons speculative code exists. You’ve already seen one example of why it’s problematic. I’ll detail some other reasons.

More time

Let’s start simple. Time spent building out functionality that may be necessary in the future is time not spent on making things better today. As I mentioned at the start of this post, I ended up wasting hours and hours on something that ended up being completely irrelevant to the performance of teams in the competition, at the expense of things that mattered a lot more, like pathfinding.

Less focus

Since there is more being developed, it’s likely that the overall software product is less focused. Your time and attention are being divided among more modules, including the speculatively designed ones.

More code

Software complexity is often measured in lines of code; it’s not uncommon for large software projects to number in the millions. Windows XP, for instance, had about 45 million lines.

Edsger Dijkstra, one of the most influential computer scientists, has a particularly good quote about lines of code:

My point today is that, if we wish to count lines of code, we should not regard them as “lines produced” but as “lines spent”: the current conventional wisdom is so foolish as to book that count on the wrong side of the ledger.

I once equated lines of code produced to productivity, but nothing could be further from the truth. I now consider it a very good week if I decrease the lines of code in the system, by deleting chunks of code or rewriting them to be simpler and shorter.

The extra code and complexity associated with speculative coding is very expensive.

• It slows down readers of the code
• It slows down building the software (especially if it pulls in more dependencies)
• It adds additional tests that slow down the test suite
• It is likely to add more bugs (more code generally equals more bugs)

Sets unrealistic expectations

Say that you design a feature because you think that the customer is going to want. Imagine that you actually got it right – what they end up asking for is essentially identical to what you’ve implemented. You deliver it to the customer a full week before you promised it.

You might look like a hero, but this sets a very bad precedent. It sets unrealistic expectations as to how much work the feature took to implement, and might lead to the customer setting impossible deadlines for features of similar scope. If you were able to finish that feature early, they might reason, there’s no reason you shouldn’t produce the next feature just as quickly.

You’re probably a bad judge of what will be needed in the future

It’s hard enough to build software from detailed specifications and requirements. Guessing about what the specifications and requirements of a feature that isn’t needed yet is likely to end up with a product that doesn’t make anyone happy. It will likely match the designers’ mental model but not the users, since there was inadequate input from them.

It’s hard to remove features once they exist

Say that you’re designing the export feature of your software. You imagine there will be a whole lot of formats you want to support, but at the moment the only hard and fast requirement is CSV (comma separated value) format. As you’re writing the CSV export code, you see how it would be trivial to implement JSON encoding. And while you’re at it, you throw in XML. You were required to produce CSV but now you have JSON and XML support too. Great!

Well, maybe. Maybe not. A year down the line you notice that only a small percentage of your users export to XML, but the feature has led to a disproportionate number of support tickets. Now you’re in a tough place – if you kill the feature, you’ll irritate these power users. Furthermore, you will have effectively wasted all of the time in implementing the feature in the first place, and all the subsequent patches.

I have seen little-used features remain in production because they’re too much trouble to delete and alienate the few users of said feature. Which leads to…

Increased risk of dead code

Imagine that you’ve implemented a new feature but it’s not ready for prime time yet. Or maybe you used it once or twice but it’s not worth turning on for your normal service. You don’t want to kill the feature entirely, as it might have some utility down the line. (Warning bells should be going off about now) You decide to hide the feature behind a configuration flag that defaults to off. Great! The feature can easily be reenabled should you ever need it again.

There’s just one problem – it gets turned on accidentally interacts catastrophically with the rest of the system. Your software deals with financial transactions and it ends up costing your company 460 million dollars.

This sounds unlikely – except it’s true. This is essentially what happened to Knight Capital in 2012.

From the Security and Exchange Commission report of the incident:

Knight also violated the requirements of Rule 15c3-5(b) because Knight did
not have technology governance controls and supervisory procedures
sufficient to ensure the orderly deployment of new code or to prevent the
activation of code no longer intended for use in Knight’s current operations
but left on its servers that were accessing the market; and Knight did not
have controls and supervisory procedures reasonably designed to guide
employees’ responses to significant technological and compliance
incidents;

This is one of the most visible failures caused by dead or oxbow code. I am not suggesting that Knight Capital speculatively developed the feature that malfunctioned. What I am saying is that

• It’s dangerous to leave dead code around in a system
• Speculative development is likely to lead to features that are not used often and are more likely to be dead code than if they were completely spec’ed out as in normal development
• Therefore speculative development puts you at a greater risk of dead code problems

Don’t allow dead code stay in the codebase. If you should ever need it again, you should be able to retrieve it from the version control system. You almost certainly won’t.

Conclusion

As an engineer, it’s easy to fall into the trap of implementing features before they’re actually needed. You’ll look productive and proactive. In the end, it’s best to avoid this temptation, for all of the problems I’ve mentioned. These include

• the extra code takes time to write, test, debug, and code review
• it contributes to a lack of conceptual focus in the system
• if done to please a customer, it sets unrealistic expectations for the development of other features
• it imparts an extra maintenance cost for the rest of the lifetime of said feature
• it will be difficult to remove the feature if and when its lack of use becomes apparent
• it puts you at increased risk of leaving dead code in the system, code which may later be accessed with bad consequences

I love Dijkstra’s notion of ‘lines spent’. Do you want to spend your time and lines of code on a speculative feature? Just remember – you aren’t gonna need it.

“Everyone should be able to pull and analyze data”

“Data overload” Islam Elsedoudi via flickr- cc http://creativecommons.org/licenses/by-sa/2.0/

Everyone should be able to write spaghetti code, and everyone should be able to pull and analyze data. And I’m not just talking about business-folk here.

…
Look at what’s going on in the digital humanities. Now, even literature, history, and religious scholars can use data to shed new insight on old texts. How awesome is that? But you have to be able to actually analyze the data. That means being able to query and scrub; that means knowing a bit of probability and statistics. The difference between a median and mean would be a start.

So yes, it’s no longer acceptable to say, “I suck at math!” and then ignore that part of the world.

I suck at physical exercise, but that doesn’t mean it’s OK for me to melt into a chair all day. We all need to work at the important stuff in life, and understanding data has become terribly important.

• John Foreman, chief data scientist at MailChimp. Read the full interview on chartio.com.

I agree with the overall sentiment of the quote, that more people should be able to do basic data scraping and analysis. Unfortunately, I don’t see it happening anytime soon for two reasons – the tools to analyze data are complicated to non-engineers and most people do not receive training in programming (to script and pull the data in the first place) or statistics (to crunch the data and draw valid insights).

Even if everyone had the skills and tools necessary to pull and analyze the data, there would still be a need for skilled analysts / data scientists. Executives and product managers often don’t have the time to do analysis themselves; it’s not efficient for them to do so. Analysts fulfill an important role by distilling raw data into products and insights.

Of bugs and purple wires

I picked up a copy of Frederick Brooks, Jr.’s seminal work, “The Mythical Man Month: Essays on Software Engineering” a few weeks ago and finished it this past weekend.  Despite being written over three decades ago, there is a lot of great content here, and much of it still relevant.  One quote that struck me comes towards the tail end of the book, and orthogonal to the more commonly cited content about the futility of adding more developers to a late project.

In System/360 engineering models, one saw occasional strands of purple wire among the routine yellow wires.  When a bug was found, two things were done.  A quick fix was devised and installed on the system, so testing could proceed.  This change was put on in purple wire, so it stuck out like a sore thumb.  It was entered in the log.  Meanwhile, an official change document was prepared and started into the design automation mill.  Eventually this resulted in updated drawings and wire lists, and a new back panel in which the change was implemented in printed circuitry or yellow wire.  Now the physical model and the paper were together again, and the purple wire was gone.

Programming needs a purple-wire technique, and it badly needs tight control and deep respect for the paper that ultimately is the product.  The vital ingredients of such technique are the logging of all changes in a journal and the distinction, carried conspicuously in source code, between quick patches and thought-through, tested, documented fixes.

System Debugging, p. 149

I wonder, has much changed in the past thirty plus years in software engineering?  What is the equivalent of the purple wire?  The best I can think of is ensuring that no bug is fixed without a corresponding unit test being put into place to validate the change and to guard against regression.  I am very new to the industry, however, and I would love for the more experienced of my readers to share their thoughts on the matter.  How do you ensure that the quick fix that inevitably gets put in with a quick // TODO: Remove this hack comment actually gets the attention it deserves?  Is it just a matter of every coder being ultra-vigilant?  Does your company have certain standards and practices relating to this problem?

Interview with Twitter dev explaining use of Scala

So what were our criteria for choosing Scala? Well first we asked, was it fast, and fun, and good for long-running process? Does it have advanced features? Can you be productive quickly? Developers of the language itself had to be accessible to us as we’d been burned by Ruby in that respect. Ruby’s developers had been clear about focusing it on fun, even sometimes at the expense of performance. They understood our concerns about enterprise-class support and sometimes had other priorities.

We wanted to be able to talk to the guys building the language, not to steer the language, but at least to have a conversation with them.

There’s a new, good post interviewing one of the lead back-end engineers of Twitter.  I knew that they had replaced some Ruby code with Scala due to performance concerns, but there’s new information in here to me, such as all the systems it’s used in.  It also gives a great summary of Scala’s benefits as a language. He makes a good point that you can easily hire Java programmers and train them to use Scala; it’s a lot harder to teach/hire a Haskell programmer, for instance.

Well worth a read.

Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. Flat is better than nested. Sparse is better than dense. Readability counts. Special cases aren’t special enough to break the rules. Although practicality beats purity. Errors should never pass silently. Unless explicitly silenced. In the face of ambiguity, refuse the temptation to guess. There should be one— and preferably only one —obvious way to do it. Although that way may not be obvious at first unless you’re Dutch. Now is better than never. Although never is often better than *right* now. If the implementation is hard to explain, it’s a bad idea. If the implementation is easy to explain, it may be a good idea. Namespaces are one honking great idea — let’s do more of those!

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren’t special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one— and preferably only one —obvious way to do it.
Although that way may not be obvious at first unless you’re Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it’s a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea — let’s do more of those!

The Zen of Python, by Tim Peters

A little easter egg built into the python interpreter; type “import this” at the command prompt.  This is the first thing I learned from O’Reilly’s Learning Python book I bought yesterday.

To write clean code, you must first write dirty code; and then clean it.

To write clean code, you must first write dirty code; and then clean it.

Robert C. Martin

http://www.objectmentor.com/resources/articles/Clean_Code_Args.pdf

To illustrate the difference between Web applications that don’t use MVC and those that do, think about the difference between Rocky Road and Neapolitan ice cream. Both may be delicious, but if you want to make any changes to Rocky Road, think about how much trouble it would be to switch the almonds for walnuts. The almonds are too deeply embedded in the ice cream to do the switch without affecting everything else. On the other hand, because Neapolitan is cleanly separated into layers, switching one flavor for another is an easy task. Think of Neapolitan as MVC compliant, and Rocky Road as not.

To illustrate the difference between Web applications that don’t use MVC and those that do, think about the difference between Rocky Road and Neapolitan ice cream. Both may be delicious, but if you want to make any changes to Rocky Road, think about how much trouble it would be to switch the almonds for walnuts. The almonds are too deeply embedded in the ice cream to do the switch without affecting everything else. On the other hand, because Neapolitan is cleanly separated into layers, switching one flavor for
another is an easy task. Think of Neapolitan as MVC compliant, and Rocky Road as not.
Jakarta Struts For Dummies Mike Robinson, Ellen Finkelstein
As Seth Glickman puts it, “They were almost definitely high when they wrote that.”

[Java] might be successful – after all, MS DOS was – and it might be a profitable thing for all your readers to learn Java, but it has no intellectual value whatsoever… Java is clearly an example of a money oriented programming (MOP).

[Java] might be successful – after all, MS DOS was – and it might be a profitable thing for all your readers to learn Java, but it has no intellectual value whatsoever… Java is clearly an example of a money oriented programming (MOP).

http://www.stlport.org/resources/StepanovUSA.html

A C++ STL author lays the smack down on Java and Object Oriented Programming.  Also interesting for the STL work he describes.

He usually appears in public with a lost umbrella in each hand. He prefers to face the blackboard and turn his back on the class. He writes A; he says B; he means C; but it should be D.

He usually appears in public with a lost umbrella in each hand. He prefers to face the blackboard and turn his back on the class. He writes A; he says B; he means C; but it should be D.

How to Solve It: A New Aspect of Mathematical Method – G. Polya

I am hoping to buy this book soon.  Apparently Microsoft used to give copies of it to all of its junior developers.