I am a programmer who has worked a lot on hardware, and from that point of view, software testing looks horrible, since software is full of bugs. And you can explain this with, well, they push programmers to ship buggy code since you can always fix the bugs later, and so the program is always full of bugs to be fixed later — whereas with hardware, bugs are often (not always!) catastrophic, so they have no choice but to let you fix them, even if it takes some time.
From this angle, the only question is, are the decision-makers right in thinking that bugs aren’t a problem, or are they deluding themselves and losing more money due to bugs than it would cost to avoid them? (I think that sometimes they’re right, and when they’re wrong, you are unlikely to persuade them absent a visceral, visual argument like the hardware case of “a pile of chips thrown in the garbage,” or regulatory requirements.)
But we could look at it from another angle — how is software actually tested, and why? I mean, if you don’t care about bugs, you could have no tests at all, which indeed was fairly common till the 2010s. Now, on the other hand, you have unit tests roughly doubling the size of the code base on average. What happened, and what does this testing buy the programmers doing it?
Unit testing definitely doesn’t do much in terms of squashing non-trivial bugs, for two reasons:
- Many tough bugs and performance issues come from the way components interact, so you need “integration tests” running many components from different teams to discover these — and like other kinds of “not unit tests,” integration tests are very rare. In fact, people will do unthinkable things to “mock” their dependencies and avoid touching anything outside their “unit” (for example, only use abstract classes to make mocking easier, or create a “stub” of an API per test, so that you can’t change your API without not only updating the call sites, but also all these stubs, etc.)
- You need a lot of different input combinations to find tricky bugs, whereas the unit test approach is to hardcode a handful of inputs and their respective outputs. In hardware circles, it is common to generate random inputs to the “design under test,” which makes you think about the input distribution (you don’t want most inputs to be garbage quickly rejected by the code) and also how to tell if an output is correct — rather than just running the code and hardcoding its output as “correct,” a typical origin story of unit test reference outputs.
If unit tests are much less effective at finding bugs than integration tests and/or input randomizers, why are they the one testing method which became pervasive in the industry, and why in the 2010s?
My answer is that the main purpose of unit tests is keeping your code from being violently shat on by others in an environment where everything changes uncontrollably all the time. The “we can fix bugs later” problem, after all, is but a special case of the “we can always change the program” problem — a problem that hardware doesn’t have (or you could call the ease of changing code an advantage, of course, but in a way it isn’t, since hard things are easy, and easy things are hard.)
In this situation, where everyone is constantly told to change everything, and a culture evolves where you have no code ownership (“shared ownership” being the euphemism), how can you keep people from breaking your code in the most basic sense? “Breakage” is a social construct; I only “broke” something if I am made to fix it, preferably before I pushed my changes, since goodness knows that I can’t be bothered to fix anything afterwards.
Unit tests are the perfect mitigation for this insanity:
- They are very easy to write
- They’re cheap and fast to run — so you can make them gate commits without much pushback
- They tick the “quality” box at just about the cost software organizations are willing to pay to tell themselves they are doing something about quality
- Everyone can change everything — of course! — but not if tests break — also of course! There’s something too ridiculous in removing someone’s tests or brutally changing them to make them pass, without even talking to the author; so there you have it — a mechanism to keep your code from being blown to smithereens by passers-by, they now need to either keep the tests passing or talk to you
- …And most programmers will enthusiastically support these tests, seeing how easy they are to write and how much they do for them — not to find tricky bugs, but to be able to count on things still being there after you put them in, dammit
This is the perfect equilibrium which both management and programmers are happy with, and which genuinely lets you move faster — compared to moving really fast without such tests, and then backpedaling frantically, with everybody trying to undo the damage brought about by everybody else. Sure, the program is full of both weird bugs and pretty shallow ones not at all covered by unit tests — but at least you have a program made of parts mostly recognizable to whoever made them, despite the team size and the sheer amount of changes.
This I think is why tests roughly double the code size, BTW — they're a bit like writing the code twice: first, the spelling everyone can shit on — shared ownership and all — and then the tests are a second respelling that they can't just totally shit on after all, because it's actually better for them not to be able to. You basically repeat yourself another time, saying, like, I really meant it the first time, don't just rip it to shreds, please.
(When you test to find bugs as opposed to marking territory, tests could be both much longer and much shorter than the code, depending — but the amount of territorial markings tends to be proportionate to the size of the territory. “This function is still in and does something like it was supposed to, this other one is also still in…”)
Why the 2010s? That’s when DVCSes went mainstream, with their easy branching and merging of entire code bases — which, imagine what this does to your code after a while absent CI full of unit tests (and CI & DVCS got near-ubiquitous at around the same time; interestingly, the early high-profile use of DVCS was in the Linux kernel which didn’t have unit tests and only recently started to add them, and unit tests had been popular with the “agile” people since before DVCS was a thing, and then later they exploded in popularity together.)
Now, if you look at it from a hardware guy’s POV, it’s still weird. The hardware guy mostly ignores unit tests, seeing how little they do for correctness, and says, OK, why don’t you also set up a dedicated verification team which might write and run integration tests, and randomizers? You’ll find loads of bugs and for sure some of them will cost more if left alone than it costs you to have this team.
The answer to which is, dude. What do you think it’s even gonna look like? I mean, first of all, we don’t want a team filing bugs for code we already committed — we have moved on, get it? You say integration tests, randomizers — what APIs are they going to work against? We now need to care about API stability for these people when the whole point is to change everything all the time (in practice a lot becomes immutable for various reasons but this is rarely acknowledged)? It’s not just that you want us to spend money on quality — you are going to slow us down, which is obviously strictly forbidden!
And this, in fact, is the general principle of software quality: quality improvements are perceived as worthwhile if and only if they accelerate the rate at which the program is changed. Anyone trying to improve quality at the expense of the proverbial “velocity1” meets this principle in action, in an unpleasant way. And this is why the widespread testing methods in software are so different from the ones in hardware, and why software teams so rarely adopt the very effective approaches used in hardware, where it’s squashing the bugs which matters rather than the rate of change.
Thanks to Dan Luu, a programmer who has worked a lot on hardware, for reviewing a draft of this post and for prompting me to write this by repeatedly mentioning how bad software testing is, most recently in a piece showing agents copying the habits of human programmers in this area.
P.S.: randomizers recommended
If you write tricky code and want to get it right, I do heartily recommend a randomizer — not necessarily running 24/7 on many machines, but just a few tens of thousands of times or whichever number you can afford — and on deterministic seeds, so you can in fact put it in CI and keep people from breaking your code at least on this set of inputs. (If your random inputs are actually random, CI will be non-deterministic, and people can just rerun it until it passes — the problem keeping people from running TSan in CI; I wonder why nobody made a deterministic thread scheduler work with TSan builds to mitigate this.)
I randomize at “module” level — a few thousand or tens of thousands of lines of code — so the randomizers works against a stable API that is hard to change rather than ever-changing internal ones you can’t afford to invest this much work into. The randomizer can take a week or several to write, but once you do this, you’ll approximate hardware-grade reliability for the code you test. And you can run it 24/7 for a period of time once in a while so it finds a few rarely reproducing bugs, if you’re the kind of person who really wants to fix bugs proactively (which is unlikely to be incentivized.)
My point here is that serious integration testing is not something one person can do if the organization doesn’t want it, which it usually doesn’t; but one person definitely can put random testing to good use even if nobody else does this, and reap the benefits, and it makes sense for certain kinds of code. Perhaps I will elaborate on this method in an upcoming post, “DDT (Development-Driven Testing).”
P.P.S. agents and unit tests
Today's agents seem to have learned from human code bases more than from some reinforcement learning process teaching them to test well, and so they roughly double the code size with pretty silly tests. In a code base mostly edited by agents, is it better than nothing? Funnily enough, currently, I would think yes.
An agent is like a quick-witted, experienced new hire knowing nothing about your code whom you've just onboarded. If you work at a place where most changes are done by someone like that, code will quickly lose features and any semblance of structure.
Since agents are primed to treat test failures with respect, you have a guardrail against destructive changes of this sort — though not against additions of complicated unnecessary structures, themselves protected from future “refactoring” by tests. But human programmers do this sort of thing, too; here as in many other contexts, agents just get you there faster, whether “there” is your goal or your worst nightmare.