10x more selective
There's this common notion of "10x programmers" who are 10x more productive than the average programmer. We can't quantify
productivity so we don't know if it's true. But definitely, enough people appear unusually productive to sustain the "10x
programmer" notion.
How do they do it?
People often assume that 10x more productivity results from 10x more aptitude or 10x more knowledge. I don't think so. Now
I'm not saying aptitude and knowledge don't help. But what I've noticed over the years is that the number one factor is 10x more
selectivity. The trick is toΒ consistentlyΒ avoid shit work.
And by shit work, I don't necessarily mean "intellectually unrewarding". Rather, the definition of shit work is that its
output goes down the toilet.
I've done quite a lot of shit work myself, especially when I was inexperienced and gullible. (One of the big advantages of
experience is that one becomes less gullible that way β which more than compensates for much of the school knowledge having
faded from memory.)
Let me supply you with a textbook example of hard, stimulating, down-the-toilet-going work: my decade-old adventures with
fixed point.
You know what "fixed point arithmetic" is? I'll tell you. It's when you work with integers and pretend they're fractions, by
implicitly assuming that your integer x actually represents x/2^N for some value of N.
So to add two numbers, you just do x+y. To multiply, you need to do x*y>>N, because plain x*y would represent x*y/2^2N,
right? You also need to be careful so that this shit doesn't overflow, deal with different Ns in the same expression, etc.
Now in the early noughties, I was porting software to an in-house chip which was under development. It wasn't supposed to
have hardware floating point units β "we'll do everything in fixed point".
Here's a selection of things that I did:
- There was a half-assed C++ template class called InteliFixed<N> (there still is; I kid you not). I put a lot of effort
into making it, erm, full-assed (what's the opposite of
half-assed?) This included things like making operator+ commutative when it gets two fixed point numbers of different types
(what's the type of the result?); making sure the dreadful inline assembly implementing 64-bit intermediate multiplications
inlines well; etc. etc.
- My boss told me to keep two versions of the code β one using floating point, for the noble algorithm
developers, and one using fixed point, for us grunt workers fiddling with production code. So I manually kept the two in
sync.
- My boss also told me to think of a way to run some of the code in float, some not, to help find precision bugs. So
I wrote a heuristic C++ parser that automatically merged the two versions into one. It took some functions from
the "float" version and others from the "fixed" version, based on a header-file-like input telling it what should come from
which version.
- Of course this merged shit would not run or even compile just like that, would it? So I implemented macros where you'd pass
to functions, instead of vector<float>&, a REFERENCE(vector<float>), and a horrendous bulk of code
making this work at runtime when you actually passed a vector<InteliFixed> (which the code inside the
function then tried to treat as a vector<float>.)
- And apart from all that meta-programming, there was the programming itself of course. For example, solving 5Γ5 equation
systems to fit polynomials to noisy data points, in fixed point. I managed to get this to work using hideous normalization
tricks and assembly code using something like 96 bits of integer precision. My code even worked better than
single-precision floating point without normalization! Yay!
For months and months, I worked as hard as ever, cranking out as much complicated, working code as ever.
And here's what I should have done:
- Convince management to put the damned hardware floating point unit into the damned chip. It didn't cost that many square
millimeters of silicon β I should have insisted on finding out how many. (FPUs were added in the next chip
generation.)
- Failing that, lay my hands on the chip simulator, measure the cost of floating point emulation, and use it wherever
it was affordable. (This is what we ended up doing in many places.)
- Tell my boss that maintaining two versions in sync like he wanted isn't going to work β they're going to
diverge completely, so that no tool in hell will be able to partially merge them and run the result. (Of course this is exactly
what happened.)
Why did this end up in many months of shit work instead of doing the right thing? Because I didn't know what's what, because
I didn't think I could argue with my management, and because the work was challenging and interesting. It then promptly went
down the toilet.
The hardest part of "managing" these 10x folks β people widely known as extremely productive β is actually convincing them to
work on something. (The rest of managing them tends to be easy β they know what's what; once they decide to do something, it's
done.)
You'd expect the opposite, kind of, right? I mean if you're so productive, why do you care? You work quickly; the worst thing
that happens is, nothing comes out of it β then you'll just do the next thing quickly, right? I mean it's the slow, less
productive folks that ought to be picky β they're slower and so get less shots at new stuff to work on to begin with, right?
But that's the optical illusion at work: the more productive folks aren't that much quicker β not 10x quicker. The reason
they appear 10x quicker is that almost nothing they do is thrown away β unlike a whole lot of stuff that other people
do.
And you don't count that thrown-away stuff as productivity. You think of a person as "the guy who did X" where X was famously
useful β and forget all the Ys which weren't that useful, despite the effort and talent going into those Ys. Even if something
else was "at fault", like the manager, or the timing, or whatever.
To pick famous examples, you remember Ken Thompson for C and Unix β but not for Plan 9, not really, and not for Go, not yet β
on the contrary, Go gets your attention because it's a language by those Unix guys. You remember Linus Torvalds even though
Linux is a Unix clone and git is a BitKeeper clone β in fact because they're clones of successful products which
therefore had great chances to succeed due to good timing.
The first thing you care about is not how original something is or how hard it was to write or how good it is along any
dimension: you care about its uses.
The 10x programmer will typically fight very hard to not work on something that is likely enough to not get used.
One of these wise guys asked me the other day about checkedthreads
which I've just finished, "so is anyone using that?" with that trademark irony. I said I didn't know; there was a comment on HN
saying that maybe someone will give it a try.
I mean it's a great thing; it's going to find all of your threading bugs, basically. But it's not a drop-in replacement for
pthreads or such, you need to write the code using its interfaces β nice, simple interfaces, but not the ones you're already
using. So there's a good chance few people will bother; whereas Helgrind or the thread sanitizer, which have tons of false
negatives and false positives, at least work with the interfaces that people use today.
Why did I bother then? Because the first version took an afternoon to write (that was before I decided I want to have
parallel nested loops and stuff), and I figured I had a chance because I'd blog about it (as I do, for example, right now). If I
wrote a few posts explaining how you could actually hunt down bugs in old-school
shared-memory parallel C code even easier than with Rust/Go/Erlang, maybe people would notice.
But there's already too much chances of a flop here for most of the 10x crowd I personally know to bother trying. Even though
we use something like checkedthreads internally and it's a runaway success. In fact the ironic question came from the guy who
put a lot of work in that internal version β because internally, it was very likely to be used.
See? Not working on potential flops β that's productivity.
How to pick what to work on? There are a lot of things one can look at:
- Is there an alternative already available? How bad is it? If it's passable, then don't do it β it's hard to
improve on a good thing and even harder to convince that improvements are worth the switch.
- How "optional" is this thing? Will nothing work without it, or is it a bell/whistle type of thing that can
easily go unnoticed?
- How much work do users need to put in to get benefits? Does it work with their existing code or data? Do
they need to learn new tricks or can they keep working as usual?
- How many people must know about the thing for it to get distributed, let alone used? Will users
mostly run the code unknowingly because it gets bundled together with code already distributed to them, or do they need to
actively install something? (Getting the feature automatically and then having to learn things in order to use it is often
better than having to install something and then working as usual. Think of how many people end up using a new Excel feature vs
how many people use software running backups in the background.)
- How much code to deliver how much value? Optimizing the hell out of a small kernel doing mpeg decompression
sounds better than going over a million lines of code to get a 1.2x overall speed-up (even though the latter may be worth it by
itself; it just necessarily requires 10x the programmers, not one "10x programmer").
- Does it have teeth?If users do something wrong (or "wrong"), does it silently become useless to them (like
static code analysis when it no longer understands a program), or does it halt their progress until they fix the error (like a
bounds-checked array)?
- ...
You could easily expand this list; the basic underlying question is, what are the chances of me finishing this thing and then
it being actually used? This applies recursively to every feature, sub-feature and line of code: does it contribute to the
larger thing being used? And is there something else I could do with the time that would contribute more?
Of course it's more complicated than that; some useful things are held in higher regard than others for various reasons.
Which is where Richard Stallman enters and requires us to call Linux "GNU/Linux" because GNU provided much of the original
userspace stuff. And while I'm not going to call it "Gah-noo Lee-nux", there's sadly some merit to the argument, in the sense
that yeah, unfortunately some hard, important work is less noticed than other hard, important work.
But how fair things are is beside the point. After all, it's not like 10x the perceived productivity is very likely to give
you 10x the compensation. So there's not a whole lot of reasons to "cheat" and appear more productive than you are. The main
reason to be productive is because there's fire raging up one's arse, more than any tangible benefit.
The point I do want to make is, to get more done, you don't need to succeed more quickly (although that helps) as
much as you need to fail less often. And not all failures are due to lack of knowledge or skill; most of them
are due to quitting before something is actually usable β or due to there being few chances for it to be used in the first
place.
So I believe, having authored a lot of code that went down the toilet, that you don't get productive by working as
much as by not working β not on stuff that is likely to get thrown away.
Were you referring to yours truly in the article above? I guess you
did. I just want to debate some of the thesis above. Yes, being
selective/avoiding failures is important, but also having the courage to
dare and fail in something is necessary. And yes, I think the 10x is
certainly also an aptitude/knowledge and mind-set thing. You need to
avoid wasted effort, that is for sure. Indeed I guess the 10x man will
just write at the first try what an 'average' one will take 10 versions
to achieve if at all ...
I am not sure the 10x is also the correct metric, as you said,
productivity can not be measured. The more important contributions are
sometimes just things that would not have existed/thought of at all by
the 1x / average person.
I am also not sure your personal example above was in place, as I do
think you are in the 10x category, and apparently you did produce quite
a lot of non trivial code/work during that time too. With hindsight you
know it was wasted/doomed to fail effort, but that's because you've
become more experienced over the years, like all of us ...
Well if I accept your compliment then my example was surely in place,
because avoiding wasted effort is much of the difference between then
and now. More generally nobody can change innate aptitude (to the extent
that aptitude is innate...), but you can change your attitude.
So yeah, there's more to it and I admitted that; I focus on this one
thing because it contributes a lot and it can be changed. What
kind of things you come up with might be innate or very
"person-specific" somehow β I don't know; I know for sure that there are
people with seemingly similar aptitudes, working on roughly equally hard
stuff, except that one's stuff gets used and the other's much less.
Regarding courage to fail β sure, when it's hard and one can fail
through the "fault" of one's own; but less so when
circumstantially the thing looks like having poor chances even if you do
a perfect job. In particular, I very easily see you taking the first
type of risk; the second type, much less likely, not?
That is why the most productive programmers use template based code
generators. The programmer just has to click the boxes to specify the
features needed and then finally click the Generate Code button and the
generator creates the code for you.
All you need to do is connect the output from one code generator to
the input from another.
You ought to be kidding.
From my experience β a lot of coders end up writing 10x more code to
do the same work than the 10x coders.
Often because they don't seem to step back and understand things and
just throw more code at it until it "works"
Your notion of success/measure of selectivity will vary depending on
the context of your work and on your position within an organization. I
see the following contexts for programming work:
Β
1) employee in a shop that works on 'product'
2) employee in a shop that works on 'site/service' / 'customer
project'
3) independent consultant
4) work on open source project
working on 'products' values more such things as 'completeness' at
the expense of 'efficiency'.
Working on 'service' your work must reasonably fulfill the customers use
cases; most often at the expense of 'completeness'; As a consultant you
might try to minimize the effort that has to be put into a task, while
Β completing the customers spec.
Β
So what ever you do, you can be as selective as the realities around you
allow you to be. Most freedom
you have with your own pet projects, that is; but how to achieve some
wider impact so far has been beyond me ;-)
Β
Β
@John M: a part of it is seeing code as a good thing, so that more
must be better; let's add this and let's add that. A special case of
gladly doing needless work under the assumption that hard work is always
a good thing.
@Michael Moser: I think you always want to minimize the effort you
put into a task (including not doing it altogether because it can be
avoided). It's not different for consultants and full-time employees,
except if the employee has a manager that insists on getting needless
work done, which is perhaps more likely than someone hiring a consultant
and paying his own money insisting on getting needless work done.
An interesting, thought provoking post.
I think this has to do with real programming craftsmanship being more
like an art than a science. A seasoned programmer will know, almost
intuitively, what efforts are worth pursuing. What doesn't need to be
reinvented. What can be tolerated by "it works.." and what has to be
avoided at all costs. The best artist isn't the one who paints at the
greatest speed or with the highest accuracy β he's the one that does the
most meaningful work.
Being 10x productive, I agree, has less to do with the speed at which
a given task can be accomplished, and more with _which_ tasks get
accomplished.
This, btw, is exactly the reason why every single software project
does't finish on time. People love to focus on how _fast_ things get
done (estimates, performance) and they neglect the fact that _what_
actually gets done is the tricky part. All that stuff you didn't realize
you were going to do as part of the project, and all the stuff you ended
up doing that ultimately went to the trash bin. That's why being truly
productive often just means being more careful, considering things a
little longer, planning ahead a bit more before diving in.
I think "project" is a red flag, by itself; I can tell as someone who
underwent project management training by the Project Management
Institute, qualifying me to attempt to pass an exam and become a
certified Project Management Professional β PMP β’...
A project is basically something one-off with a rather precise ROI
estimation, and that ROI is not a whole lot of different from what you'd
get by lending money to someone else. So with "projects" if you're a bit
late then you're probably losing money.
The decent thing to work on is products, where there's no "final"
delivery but rather things are incrementally improved, and where the
thing might be a big-enough hit to more than cover very large
development expenses.
I have to agree with John. In one of the projects I'm working on, to
use a single example, I have a 68 line udf in the database that does the
equivalent of this:
SELECT * FROM [table]
Seriously. It has comments, with nice stars and well aligned fields
that nobody will ever care about. It has beautifully aligned field
names. It has... stuff.
(And no, we don't have a DBA. We programmers have full access to
everything we want in the project β we can redesign both the code and
the database at will. For some reason, this programmer thought a 68-line
UDF was just what was needed for this.)
I just remembered another example β I can't even decide if it's worse
than the previous one. I've seen a programmer write a 3-page switch to
convert a string to its Enum equivalent (I think it was a list of
country codes). I can understand not knowing about Enum.Parse, but I
can't for the life of me understand *not asking*. After you write the
same thing for the 10th time, shouldn't you figure out that there has to
be a better way to do this?
I saw a look-up table where table[i] gave you 1<
...table[i] gave you 1<<i, I meant to say but WordPress
wouldn't let me get away so easily.
I do think selectivity and negotiation have a huge impact in
productivity. Being able to spot a rabbit hole before you start. Offer
alternatives and give the customer the 80% solution that still solves
their problem. But that usually comes with good logic skills and falling
in the rabbit holes enough to know when they're coming.
Oh, don't mind me: I was "finally" studying the new C++11 features
when I suddenly felt the need to scream but I didn't want to alert other
people β then, after an epiphany(?), I found myself here...
I actually suggest to everyone to upgrade to C++11 from C++98/03; it
sure has a ton of horrible stuff in it, but a few things making it worth
the trouble β notably auto, "smart for", lambdas and initializer
lists.
Russian translation: http://habrahabr.ru/post/178553/
I'd gladly write a Russian version myself if you asked for it, at
least as a first draft :) While my Russian is admittedly not very well
developed when it comes to programming terminology/slang (I say that
programs "run", while Russians say that they "walk"/"go", etc.), I think
it could come out nicely if edited by a true Russian programmer from
Russia.
@gd Failure and "failing fast" to iterate toward success are not
quite the same thing. The later might better be called 'testing',
'iteration' or my favorite Succession. See: https://www.facebook.com/notes/facebook-engineering/software-design-glossary/10150309412413920
I've found some of the most successful engineers I've known to employ
both your suggestions and Yossi's. Part of being good and not doing shit
work β is figuring out what is shit work. This is hard, and beyond
experience, the best way to do that is paradoxically doing shit work and
failing fast. But the key here is to fail fast and and to use data to
guide decisions. Being smart about choosing to do the easy things that
you have some evidence of having the most impact and then iterating or
switching manages the risk (and cost) of failure which means you may
just end up achieving success.
You have such a colorful English that I cannot resist temptation to
translate it myself.
For the next articles, I will gladly edit your drafts in Russian trying
as much as I can to preserve your style
Hi Yossi,
Did you implement your ideas in practice? What where the results? I
long for a sequel to this article.
Well, I don't know β do you expect me to say that I'm now 10x more
productive than before or than someone else?.. Yeah, I think I got less
gullible over the years, I think it helped... But others should be the
judges...
Although I am still inexperienced, I thought that I should add my two
cents. Interning at a company at the moment, I feel that I am definitely
doing "crap work", (but difficult much like your example).
However, reading this article reminded me of a fellow programmer who
never re-invents the wheel and gets a lot done. The only problem is that
he has the creativity of a plank.
I think that solving difficult problems, and solving problems which
have yet to be uncovered (these are usually revolutionary) requires a
lot of useless work before an epiphany.
Well, it depends on context; an experienced programmer easily tempted
into doing shit work or not protesting when ordered to do it has a real
weakness, someone just starting out is a bit different.
There is in fact some tension between efficiency and creativity, but
efficiency can also be a force multiplier, to some extent, if it makes
you focus on those hard things which actually matter.
I'm a 1x, and I think it's at least in part due to the fact that my
main motivation is to have fun rather than to get shit done. It's fun to
learn a new language, or write a new tool, or read this blog, and it can
kind of be considered productive, so it gets rationalized. I'm hoping
that at some point I'll reach a level of maturity where my priorities
will change.
Well, I think at some point getting shit done becomes the fun; as in,
the first several languages arouse more curiosity than the next, etc.,
and at some point it's not as much fun to learn a new language as it is
to get this particular piece of shit done already, and in fact people
start to seek ways to learn as little new info as necessary just because
so much of the info they learned in the past turned out perfectly
useless either immediately or in the long run.
Post a comment