What constitutes managerial competence? As a vague starting point for an answer, we could say that competent management sets achievable objectives and then achieves them, by organizing and incentivizing the necessary work.
It turns out that even this near-tautological banality is enough to see why competent management puts many desirable things out of reach. This becomes apparent when looking at examples where incompetent management does better than most well-run places can hope for.
Efficiency
Improving efficiency tends to be against the interest of most people in an org, because it’s equivalent to shrinking your budget. Here’s what I’m told is a true story about how things work with actual budgets. A relatively inexperienced VP attends a meeting where senior management is asked to shrink their budgets due to the adverse economic climate brought about by the coronavirus pandemic. He eagerly cuts his equipment budget from $10 million to $6 million - over the loud and desperate objections of his team (whom the VP nearly accuses of lacking patriotism, loyalty to the company and commitment to the common good.)
Next year, the coronavirus mutates some more, and profits go back up. Our VP submits a $10 million equipment budget to the finance department, where they cheerfully inform him that the extra $4 million will not go well with the CEO. Why, a 66% increase over last year’s $6 million!
Wait a minute, thinks the VP, a sensation running through his whole body of rapidly gaining that invaluable experience which he so sorely lacked. I voluntarily cut 40% of my budget - a share way larger than anyone else - due to an unforeseen, extraordinary emergency. And now I’m rewarded with this cut becoming permanent?.. I see. Well, I’m always eager to learn.
This year being already lost, he quietly resubmits a $6 million budget (approved more swiftly by the CEO than any other, thanks to zero YoY growth.) Next year, he uses some real or perceived crisis to increase this budget to $20 million. And now he learned how to operate in a well-run company.
Of course you could say that this is a badly run company, and to avoid arguing what that means, let’s stick to the definition of managerial competence as the ability to set and achieve objectives. Whatever objective you are expected to achieve, a bigger budget makes it easier. And while asking for more resources gets you yelled at, the yelling is for show, and ends once the budget increase is approved (or isn’t approved; but it never really hurts to try.) But if you fail to achieve your objective, the yelling will be for realsies, go on and on, be followed by career setbacks, and continue long afterwards, quite possibly with no way to resuscitate said career.
Set objectives create a simple zero-sum game over resources - you want more resources to do what they asked, and they want you to do the same things with less. Optimization, budget cuts or relinquishing resources under any other name simply registers as losing a round in this game. It’s awfully sweet to save company resources, but expecting it to do you any good just means that you don’t understand the game.
I mean, what do you expect to happen? That we'll ask you to do less, or forgive you for doing less? No way, we asked you to do those things because they must be done. Then maybe you expect to be given more resources? Obviously ridiculous, you just had resources, there’s no sense in hoping to get them again as a reward for giving them back when you already had them?.. Maybe you’d like a stock grant for being such a good citizen? No, if we do that, everyone will inflate their budget, and then cut it to get a stock grant. Could that be what you did here?.. Like, why was the budget so large to begin with?..
But wait, seriously though, what’s the math here? What are we maximizing? Revenue minus cost? Revenue divided by cost? I mean, shrinking the cost has got to be helping with these?.. Well, sure it’s helping, but it’s not helping you, because you don’t bring any revenue by yourself, unlike cost, which you very much do incur all by yourself. The math with you is, we tell you to do something if the cost is below a threshold. If you won’t commit to doing it cheaply enough, we’ll find someone who will, and if we can’t, we won’t do the thing, or reconsider the options in some other way. But exactly what the cost below the threshold is changes nothing in any math related to you, except for a lower cost making your job harder, since you have the same objectives to achieve. The firm’s bottom line - sure, lower costs help there. But the impact on the firm’s “revenue - cost” doesn’t trickle down to your “cost < threshold,” because you have no revenue1.
Things work the same with any resource, not just actual money - it could be team size, or processor cycles and memory bytes. If you free up 200 ms of CPU cycles and 500 megs of RAM, someone else can deploy their functionality using these newly available resources, and then you won’t be able to. In fact, a mature, well-run CI system will measure everyone’s resource footprint after each commit, and will not let you exceed your budget, which was frozen at some point based on how much you were using at the time (hope it was a lot! - always spend like crazy before the baseline is established!) Is it any wonder that people learn to never optimize their code - unless they want to deploy something new themselves, and only after asking for more resources to deploy it and not getting any?
I like it when people ask “why is this code so slow? Why don't we optimize it?” And it still makes me sad when people ask instead, “how much CPU time do I have for running this code?” when it's obviously 5-10x slower than it could be, and they're asking to reserve 2-3x more CPU time than they're already wasting. But that's what happens when people have worked at well-run places and aren’t stupid.
What happens in a badly run place? In a badly run place, management is bad at setting objectives, so you have people aimlessly wandering about, lacking clear goals, and just doing stuff because they want to. They see an optimization opportunity and they gladly pursue it - it’s interesting, it’s fun, it’s good for the company, it’s what they’re here for. If a patch must be submitted to a team, that team might gladly accept it - they don’t mind shrinking their resource footprint, because nobody monitors the resource budget properly, nor presses them to meet any targets very hard - which is also why they don’t really mind spending some time on something not helping them achieve any such target. In fact, they might get interested enough to actively help whoever found the problem to fix it.
Your legs don’t fight your heart, brain and each other for the oxygen budget; every organ only uses what it needs, and is optimized for efficiency. The selfish corporation is yet to make its parts behave as selflessly as our body parts sharing our supposedly selfish genes. Yet people do have a tendency to do the right thing regardless of incentives - no doubt because they mistake their corporation for their tribe, thinking their coworkers share more of their genes than they do2. But if there’s a reliable & scalable way for vigorous, systematic management to reward the spontaneous human drive towards efficiency instead of punishing it, I am yet to see it. Certainly honest people working for the trillion-dollar heavyweight champions of the industry testify that this problem is far from solved.
It’s an exercise both fun and depressing to come up with ways to “manage for efficiency.” For example, we could reward people for performance savings, right? Great idea - now I can commit some CPU or memory hog, then you can fix it, and we’ll split the reward. Or, more realistically, first we all go on a crazy resource spending spree to meet a deadline. And then later on, we optimize away the lower-hanging fruit in the crazy-inefficient system and get a reward - with not-so-low-hanging fruit from that spending spree probably left hanging forever.
(Of course, we probably won’t tell ourselves that we’re deliberately overspending more than is actually helpful for meeting the deadline to game the system. Rather, the culture will just kinda shift in that direction. People are very good at doing fucked-up things without admitting it to themselves - which would make them sad and less energized to do the fucked-up thing they have compelling reasons to do.)
Perverse incentives always appear wherever incentives are deployed, because the very notion of an incentive is fundamentally perverse. But a competent manager is forced to use incentives, instructions, and incentives to follow instructions, because what else could he use?
Sprawl
“These teams are like bulldozers with no brakes,” mused my acquaintance, who’d managed a team in a poorly-run company and had recently become a director in a much better-run one. “You only have a steering wheel, and you need to be steering all the time, or this thing is going to dig a giant hole in the ground, or raze a building or something. If you don’t tell them exactly what to do, they’re still always going to do something, and then it’ll be too late.”
You see, he was used to people doing pretty much nothing when left unmolested. Of course, from the employer’s point of view, this habit is straightforwardly wasteful, because you’re still paying their salaries. To weed out such do-nothing people, competent management sets up a performance evaluation process, so that we always know what every person has done for us every year, and who should get outsized rewards and who should get fired.
This system leaves people very worried if they don’t have clear goals to work towards. However, even a competent organization cannot set actually useful goals for everyone at all times, just like you generally need your legs, but you don’t really have a use for them at every moment. And thus, you have people with spare bandwidth making up their own goals, so that they have something to show in the performance review.
If we now revisit the situation from the employer’s point of view, it is no longer trivially wasteful, because everyone is always busy. However, it’s likely more wasteful than before, because people are building stuff you didn’t really need, and yet you almost certainly need now, because actually productive activities are hopelessly intertwined with this stuff.
This is a big reason why successful software companies end up with mountains of code. The cycle repeats and branches out exponentially, as every team who’s built the once-needless and now-necessary thing asks for more headcount, gets it, and inevitably ends up with some of it idle some of the time. Then these new people invent more goals to pursue, persuade everyone that these fake goals are actual sub-goals of the real goals, and entangle existing systems with their new systems.
And now figuring out where the waste is will be much harder than just spotting idle people, since all the needless work was done for no other purpose than looking very important, and people are pretty good at making the right impression when they’re trying. And of course when people lie, they lie first and foremost to themselves - we’re all natural-born Method actors - so if you spot a decoy and try to cancel the work on that system, not only will the people working on it fight this with all their might, but they'll be genuinely heartbroken if you do cancel it. And by the time you’ve actually dealt with one of these weeds, if you’re a weird manager actually trying, two more will have sprouted in another part of the org.
If you’re used to such sprawl, you’d be surprised how effective sleepy HR practices are at preventing it. Suppose you always get a standard, shitty raise at the end of the year by default, unless you bargain loudly, which works rarely and only if you’ve really made an impression throughout the year. There is no defined budget for raises; every significant raise is hard to get, and you never get it proactively without bargaining, but there’s no formal system to avoid spending too much on raises except for the reluctant, reactive approach to giving them. There’s also no system for firing low performers, and it’s only very rarely that you see anyone fired - like that crazy fuck who went on and on about how your source control sucked and should be completely different, and then used a single dot character, “.”, as the commit message when he finally committed something.
A similar system is used for managing other resources: for example, every team gets to grow at some low annual rate, no department is ever cut, and it’s very hard to grow your department faster than the base rate even if you get more responsibilities.
A place like this evolves the healthy laziness that keeps animals from moving their body parts all the time, needlessly burning calories, in order for the claws, wings and tails to get a good performance review from their head at the end of the year. Sure, many people do nothing much of the time, and you need some effort to make them do something when it becomes necessary; “the hedgehog is too proud a bird to fly without a kick,” as the wise Russian proverb goes. But on the upside, nobody doing anything unless it’s really necessary means you don’t have all this unnecessary stuff.
Healthy laziness begets agility - you have way less code, less systems, less everything, and therefore way more ability to maneuver and actually change things with a small number of motivated people - and there’s always a small number of motivated people in any place, and this place might even keep them, if they learn to bargain for raises. And you also don’t need to grow as much, because you don’t need to be adding people to take care of all these sprawling systems that you quickly come to depend on.
Bugs
Bug fixes work a lot like efficiency improvements, the main difference being that competent management makes things much worse. You can’t make fixing bugs into a “goal,” same as you can’t make optimization into a goal, because people will just add more bugs up front and then fix some of them. But at least with optimization, you can have teams doing it across the organization, and it claws back some of the performance lost in the first place.
A team optimizing others’ systems cannot hunt down the tens of thousands of little performance hogs created by everyone else. But it can often find tens or hundreds of relatively small changes with a fairly big performance impact. They’re probably not “fully incentivized” for this outsized impact, because with rewards anywhere close to how much money this is worth to the business, the incentive is quite likely to become extremely perverse. But you definitely can make “everyone else’s performance” a team’s job description, combine it with your venerable performance evaluation & promotion process, and get something - often a big something.
Another kind of teams with some form of “someone else’s efficiency” in their job description is people working on compiler, language runtime or kernel optimizations, custom compute or networking accelerators, and other such. They could be inefficient in their own work for the same reasons mentioned above, but they might still be increasing others’ efficiency, because it’s legitimately an example of a goal that their competent management is good at setting and achieving.
The problem with bugs is that you can’t have people solve others’ bugs as much as you can have them improve others’ efficiency. It is generally much easier for a relative outsider to see where a system spends its resources than where its bugs are. That’s because all systems spend similar kinds of resources, but what constitutes a bug varies from system to system, and there’s almost never a machine-readable, formal, or even just a reasonably complete and written-down definition of correctness. The few exceptions are things like programming language semantics, and indeed this is where a lot of progress has been made - think sanitizers, race detectors, etc.
Another problem with bug fixes which you don’t have as much with optimizations is that it’s harder to measure the impact. With efficiency improvements you can usually give a ballpark number of how much resources it would save - perhaps a range of possibilities rather than one high-confidence number, but you’ll have something. With bugs, well, you could A/B test them to try to quantify the impact on some metric management cares about, but who does that?
With performance, you deal in resources to begin with, and you have some number speaking of resource savings by definition, or you couldn’t call it an optimization. And now there might be an argument of what multipliers to apply to this number to arrive at a cost estimation, but at least you have a starting point. With a bug fix, you have the bug and the fix, and you’re seriously going to suggest to A/B test the impact for no benefit to the employer except your ability to claim this impact is worthy of a promotion? This is a great plan especially for internal systems without A/B testing infrastructure or any preconditions for it, but it’s a great plan in general, employers love this.
(And also, most bugs you fix tend to come from your own team, and then all high impact proves is that you messed up big time when you put the bug in. You’re not supposed to have bugs in the first place, punk.)
“I have this potential employer who says they’re interested in performance and correctness,” said another acquaintance. “I told them that I can work on performance anywhere in the industry, so I can probably find an offer better in other respects elsewhere. But correctness sounds interesting. I don’t know anywhere caring about correctness!”
Well, it’s not like they don’t care, as much as they don’t have a mechanism for caring or even registering it. Correctness is not a goal in itself that management can set for the teams without perverse side-effects. Of course, you have to fix “showstopper bugs” or you haven’t achieved your goal. Any further bug-fixing takes resources from achieving your nominal goals, and is avoided - not outright, which would look bad, but through slow-walking and other acceptable forms of sabotage.
It’s true that Microsoft Teams (to take one example too many are familiar with) can get away with bugs because it’s bundled with Outlook and other stuff, and because whoever pays for it doesn’t use it that much, but rather foists it upon helpless internal users. But it’s also true that fixing those bugs would be money very well-spent for Microsoft, because it would almost certainly improve their reputation and increase sales at the margins and more than offset the cost of the work. The problem is that it’s hard for a well-run place to get people to fix non-showstopper bugs.
(One way to work on correctness, if you're into this, is to go to areas where more bugs are showstoppers, so fixing them becomes a part of the nominal goals. If you’re a hardware developer, FPGAs, where you can fix bugs very cheaply, are a worse context for this than ASICs, where you cannot, making you eager to find and fix them proactively. And hardware running lots of software, which can't be patched to work around hardware bugs, like a CPU, will face more pressure to be correct than something like a peripheral device controller, which is only touched by comparatively little code written at the company making the hardware, where it’s “easy” for software developers to add workarounds to this code3. If you’re a software developer, you could try an industry with high reliability requirements, where many more bugs are defined as showstoppers.)
Of course, having more sprawl means having more bugs (and more performance issues, and more machine resources spent on running all that sprawling code), and even defining what “correct” means becomes harder when the system is larger. The unfortunate side effects of competent management compound each other.
The problem with incompetent management
The main disadvantage of incompetent management is its definitional inability to set and achieve key goals, which can endanger the survival of the organization. Incompetent management can only thrive in situations where basic survival and even growth are somehow taken care of, and any major changes in that situation create an existential risk.
It is theoretically possible for management to respond to an external crisis by “changing gears” from a sleepy indifference to what’s going on in the organization to a vigorous push to get something huge done, as required by the new external situation. The hope is to kick the suddenly awakened, terrified hedgehog into the stratosphere, and then go back to the sleepy ways of old once it’s orbiting the Earth.
In practice, the risk is high for this attempt to fail - a place not used to the mobilized state of subordinating all efforts to top-down goals will need time to learn, where “learning” might involve firing or otherwise replacing key people (which is a big part of what “learning” means for organizations, and what people mean when they say such learning is “hard.”)
If the war effort does succeed, there’s quite likely no going back - the hedgehog will have been thoroughly transformed and militarized by the ordeal. It will be the usual mix of competent management and cargo cult management from now on.
Cargo cult management vs straightforward incompetence
Speaking of which - a most unfortunate side effect of competent management is the widespread desire to emulate its look and feel, which contaminates the wonderful natural incompetence of so many managers, robbing us of its many advantages.
Mostly incompetent management which is very bad at setting and achieving goals is perfectly capable and all too likely to cargo-cult effective management by setting up an elaborate bureaucracy for assigning work and tracking its status, thus preventing work from happening spontaneously. This has all the downsides of actually competent management without any of the benefits.
Things work much better when incompetent managers embrace their laziness and do close to nothing. This is possible if there's a culture where a manager gets to look good through means other than appearing to be on top of plans and status - for example, by presenting shiny things the team is working on (regardless of their exact impact on the bottom line or even chances to be deployed in production.)
What is to be done?
“What is to be done?” is a pamphlet by Lenin, who proposed some things to be done, and went on to do them and then some, with results most charitably described as mixed.
I don't know how it ever happened to me, but I somehow got infected with the absurd idea that there's always a good way for things to work in an organization, and furthermore, somehow this good way always makes the org more effective than the commonly observed not-so-good alternatives. I was brought up with natural immunity to the Soviet strain of this Panglossian optimism with respect to our ability to shape organizations in the all-around optimal way:
We shall wholly destroy the world of oppression
Down to the foundations, and then
We'll build a new world of our own.
But it turned out that my Soviet antibodies don’t automatically work against the Western strain:
…the most important advantage of being good is that it acts as a compass. … you have so many choices. How do you decide?
Here's the answer: Do whatever's best for your users. You can hold onto this like a rope in a hurricane, and it will save you if anything can. Follow it and it will take you through everything you need to do.4
I mean, I guess I’ve always had antibodies good enough to protect against severe illness; I never imagined that companies mostly succeeded by holding onto their goodness like a rope in a hurricane. But if you pressed me, I’d say they probably could, and they would be better off if they did.
Which, if you think about it, why on Earth would this have to be correct? Few people would say that you can always make your code faster without making it uglier, and those who say it tend to be a bit insane, in a professional sense. So why would making a company more effective always make it better instead of worse, according to, well, any definition, really?.. Just because the opposite thought is depressing? Well, the thought of faster code tending to be uglier isn’t a very happy one, either.
So, now that I have immunity strong enough to prevent infection and transmission of the effective goodness virus, I don’t think you have to find a solution to an organizational problem just because you happen to observe it5. From an individual’s POV, the environment is nearly impossible to change, hard to truly understand, and fairly easy to fit into for most values of “environment,” and companies are no different. You probably can’t make most well-run places “truly care” about efficiency or correctness, but you can make a great living optimizing stuff, and even debugging or seriously testing, if you find the right place for it.
Of course, if you put a gun to my head, I could add a few paragraphs on “combining the best of both worlds” and how it’s been known to happen in small teams over short periods of time, and so on. And, not gonna lie, I almost did put a gun to my own head to write these paragraphs - old habits die hard. But I came to my senses and deleted it. It’s more likely to make you feel sad than happy, and most of all, it’s likely to make you bored.
(Update - see a comment on this write-up describing the rise and fall of Creo, a company that they say did start out combining the best of both worlds, then regressed to the mean after acquiring another company with a lot of people and a "standard" culture, and went downhill both as a place to work and a viable business. Like I said, these "best of both worlds" stories will make you more sad than happy.)
Conclusion
Competent management sets goals to achieve. Whatever can’t be made into a goal cannot be achieved by definition. Whether this sounds trivial or absurd, it has many surprising undesirable consequences which are surprisingly hard to avoid.
A company’s board is unlikely to raise the need for less competent management in their annual meeting, and for good reasons. A prospective employee is another matter. If someone invites you to work for a company that’s run very badly, there might well be a good story there - this is far from guaranteed, but you might want to hear the details. And by “a good story”, I don’t mean “yay, here’s a place to slack off at,” but “maybe I can finally get some work done that I hardly ever get the chance to do.”
See also
It's very possible to make sure you only hire people who can answer algorithms questions in an interview. But don't expect these carefully filtered employees to then actually solve, rather than create, problems equivalent to basic phone screen questions on the job, for reasons related to the above.
Thanks to Dan Luu and Tim Pote for reviewing a draft of this post.