Entries Tagged 'wetware' ↓

Digital asses in the computing industry

Ever noticed how academic asses are analog and industrial asses are digital? It's legitimate to not know whether P equals NP, or to not know what x is if x*2=y but we don't know y, for that matter. But it isn't legitimate to not know how many cycles, megabytes or – the king of them all – man-months it will take, so numbers have to be pulled out of one's ass.

The interesting thing is that the ass adapts, that the numbers pulled out of this unconventional digital device aren't pure noise. Is it because digital asses know to synchronize? Your off-by-2-months estimation is fine as long as other estimations are off by 5. But it's not just that, there must be something else, a mystery waiting to be discovered. We need a theory of computational proctology.

Ever noticed how painful the act of anal estimation is for the untrained, um, mind, but then eventually people actually get addicted to it? Much like managers who learn that problems can be made to go away by means such as saying a firm "No", without the much harder process of understanding the problem, not to mention solving it? Anal prophecy is to the technical "expert" the same raw enjoyment that the triumph of power over knowledge is to the manager. "Your powers are nothing compared to mine!"


There once was a company called ArsDigita (I warmly recommend the founder's blog and have his Tenth Rule tattooed all over my psyche), a name I tend to misread as "ArseDigital" – a tribute to an important method of numerical analysis and estimation in the computing industry.

The Virtue of a Manager

I never managed a group larger than 5 people, luckily for the people in the group (perhaps more so for those remaining outside). Good managers are hard to find, which is the basis of my self-motivating motto: "This job could have been done worse". Such is the background for the hereby presented pearls of wisdom assortment. As to "The Virtue of a Manager" title, it's a ripoff of Paul Krugman's exquisite title "The Conscience of a Liberal". "The Private Part of a Self-Important Self-Description" is a great template.

***

A prime virtue of a manager is the ability to take pride in someone else's work.

No, seriously. We've recently deployed a debugger internally and an algorithm developer had a look at it. I knew it was good, but it's used to debug the sort of thing algo devs hate: code with an anal-retentive performance focus. So the last thing I expected was praise, but praise it the guy did.

Now, I had previously known proud moments from having done things myself, and here I had this proud moment with 90% of the work done by someone else. And I'm telling you, it was just like the real thing.

***

The defining trait of a manager is the distinctly wide gap between responsibility and understanding.

By far the funniest spot to have a gap at, hence the easiest target for a low blow: try to make jokes about a gap between one's teeth and you'll soon be exhausted, but this here is gold. This is mean-spirited though. Imagine living with a gap between your responsibility and your understanding and everybody laughing at you – how would that make you feel? Show compassion.

***

One can have the title of a manager or nominal reports for any of a number of reasons:

  • An HR system with per-title wage ceilings: can't give someone a raise without faking a title.
  • A diametrically opposed case: some forms of brain damage cause people to accept lower paychecks given more impressive titles, larger rooms, etc.
  • Someone is too senior to report to a team leader but doesn't want a team to report to him, either.

In a roomful of managers, how do you find the real ones among this variety – not "real" as opposed to incompetent or unimportant, but "real" as opposed to fake?

There are several cues, for example, only real managers can have other managers report to them. But the perfect, if-and-only-if discriminator is that real managers don't write code. (The precise rule is that they can spend up to 2% of their time on a favorite piece of code without getting disqualified.)

***

The principal function of a manager is being the responsible adult.

Some managers occasionally point this out in frustration, both mourning their technical skills which dry up during their current gig where they only get to exercise adulthood, and because being the adult means getting tired of the annoying kids. A gal who both managed and met literally hundreds of managers during her career in some consulting agency said "Now I really understand management" when she got to babysit.

This is why I have hard time believing management can be taught – you can't teach adulthood, it can only result from people growing up by themselves. I'm not sure if this feeling is fully aligned with reality, but quite some very successful managers never went to a management school (at least one of those is somewhat critical of MBAs), and some of those who went say it was worthless in terms of useful things learned.

The opposite is also true: childishness is fitting for a programmer. We were two fake code-writing managers in a meeting with one real one, and at one point the real one said: "Let's not be childish about this". The technically correct reply to her would have been "I'M NOT CHILDISH ABOUT THIS, HE IS!", but I suppressed it for tactical reasons. Some time later I told her: "You don't want us to stop being childish about this, not as long as you're interested in our output as programmers. Recall: the reason you aren't still programming is because of not being childish enough to truly enjoy this sort of game."

And in fact since she started managing 20 programmers, she's been talking about her work all the time, which she didn't when she was programming. Well, some people like to play and some prefer to babysit. (I'm not sure where this leaves the quasi-managers who write code; presumably some are the elder and most responsible kid while others are the most restless who invent games for the gang.)

***

I've recently got a driving license. One thing I learned was that someone pushing his (presumably broken) car along the road is a "driver" as far as the law is concerned. I find this counter-intuitive, probably because pushing a car is not categorized in my head as "driving experience", but, at least in Israel, that's the law.

Likewise, doing the work of three people is not what most of us associate with "managerial responsibility". However, if you're given two reports without a drive of their own to work, that's what your responsibility will be.

***

A manager will have favorite words. For example: acute (critical), priorities, agenda, rationale, integrity (shoot this manager first), responsibility (ownership), stakeholder.

Keep laughing at them. Once you become a manager, you'll have favorite words whether you want it or not – it is useless to resist the dynamics inherent to your situation. My favorite word is "dynamics". Its connotations are deep and its applicability wide – heartily recommended.

***

Managers get to do a lot of knowledge-free decision making, which necessarily drives them insane. Here's how the manager's bipolar disorder works.

During maniacal periods, the manager is the only one who can do anything around here. This frequently happens when the manager is under external pressure, and he feels that control is slipping out of his hands. He's trying to compensate for his lack of knowledge by immense concentration and willpower. (Managers always have ample emergency supplies of both.) "Concentration" translates to an ability to derive general and far-reaching conclusions from insignificant details, then "willpower" translates to aggression.

Then depression follows: "Don't bother me with details". This results partly from exhaustion quickly arrived at during the mania (especially if reports were wise enough to not argue with the manager, letting his efforts defeat their own purpose.) The manager has delivered his trademark concentration and willpower, so he no longer feels guilty on that front. However, he's overwhelmed by information and (rightly) feels that he doesn't know what's going on. He decides it is none of his business and concentrates on the Big Picture (does nothing). Usually, the cycle repeats upon a new wave of external pressure.

Awareness of the management cycle on behalf of the manager himself can help soften the cycle but not eliminate it. It is up to reports to apply counter-cycle measures by scheduling most work into depression periods when it is least disrupted. Special attention must be given to long-term projects, frequently characterized by a prolonged depressive apathy period at the beginning followed by a period of maniacal frenzy lasting until the end.

***

There's a naive brain model in the spirit of "the brain has a reptilian part, a mammal part and a human part". For example, if a student fails to answer a question in an oral exam with his human brain, the mammal brain feels bad about it and complains to the reptilian brain. The reptilian brain then cheerfully replies, "Who's causing the trouble? Oh, that little guy behind the table? Not to worry – I'll kill him". The higher brains then supposedly suppress this – "What do you think this is, reptile – Jurassic Park?", and the tension is translated into sweating.

The manager is the team's reptilian brain; he doesn't know enough to do real thinking, but he's good at "taking responsibility", bargaining, fighting, socializing, etc. A manager doesn't know how to implement the feature, except for suspecting, based on experience, that it will conflict with a couple other features and it will take a week or three for the whole thing to stabilize (with him taking the heat when things break during those weeks). Therefore, instead of technical advice (which he might be otherwise qualified to give), he'll propose something which solves the problem at his favorite social plane:

  • Prioritize the feature away, delaying the implementation until forever
  • Negotiate the feature away, by talking to whoever wants it out of it for something in return
  • Redefine the feature away, by reducing the scope to the few scenarios which absolutely can't be ignored

Do not drag management into anything you actually want solved. Presented with a question, the manager will answer it by killing the little guy behind the table, so only go to him if you really want that. And once awakened, he might take a lot of sweat to suppress. (If he's really a programmer posing as a quasi-manager, the chances for an actual solution can actually be worse: he's more likely to feel guilty about his managerial ability and use the opportunity to exercise and develop that ability, instead of using his technical ability to think about the issue.)

***

There's this quote from The Mythical Man-Month, supposedly by a pessimistic manager:

All programmers are optimists. Perhaps this modern sorcery especially attracts those who believe in happy endings and fairy god-mothers. Perhaps the hundreds of nitty frustrations drive away all but those who habitually focus on the end goal. Perhaps it is merely that computers are young, programmers are younger, and the young are always optimists. But however the selection process works, the result is indisputable: "This time it will surely run," or "I just found the last bug."

This is backwards. In reality, programmers are the more pessimistic people. Perhaps it's because experience teaches programmers that programs always have bugs while teaching managers that programs always ship. Perhaps it's because the programmer is the one with the actual knowledge, and the ignorant are always optimists. But however the selection process works, how many programmers have you seen saying "it will never work" and how many managers?

A programmer might be more optimistic locally, hoping in vain to have fixed this one piece of code where he has the illusion of complete understanding. However, it is invariably the manager who believes that everything will work out. A programmer can't really believe that because there are so many things nobody even understands that are yet to be faced.

But the manager is used to knowing little and understanding less, and thus has learned to translate uncertainty to optimism. In fact a programmer can learn it, too, in the areas which are of little interest to him. I know a programmer who doesn't care about optimization and who consequently describes others' efforts to fit a program into a given performance budget as doomed to success: "It runs at the word of command" – a programmer's expression of the managerial worldview worthy of a seasoned manager.

***

We don't know how to test for programming ability. The best tech companies spend 5 to 10 interviews to solidly confirm that the candidate knows what is taught during the first 1.5 years of an undergraduate CS curriculum. Other processes measure less accurately by asking less relevant questions; the inaccuracy is somewhat ameliorated by the lack of precision – the non-uniform quirks of interviewers and general randomness of the process eliminate biases, causing all kinds of good candidates to sneak through the gates.

It is well known that we can't find out during the interview what we inevitably find out once someone gets the job, but what are the corollaries? Here's one I've heard a lot: trust recommendations more than interviews. Here's another I haven't: let others interview and get the new hires, then steal the best.

(Objection: the first recommendation is good for the company while the other is only good for the manager following it. Well, "competition between managers over team members isn't a zero-sum game – it improves teamwork across the company", this one we weasel out of in a snap.)

***

We have a VIP club at work called Bottleneck, its principal activity being the collective purchasing and consumption of alcoholic beverages. The club operates during work hours (regular meetings held on Thursdays, emergency meetings scheduled upon arrival of packages from abroad). Our room being the headquarters, I'm naturally a member. By now the club has shifted to high-end liquors at prices causing the consumption to contract to a sip per cup of coffee, but originally it was affordable to actually drink.

I noticed that minor alcoholic intoxication has a notable impact on my programming ability. I can still lay my hands on the right variable, but by the time I do I forget what you do next with these things. There's that handy member somewhere in it, dot something, but dot what?

However, managerial ability is not affected. Things I can do just fine following a meeting of the Bottleneck club include progress monitoring, planning, risk assessment, general technical advice, and requirement negotiation. Now that I think of it, perhaps the managerial functions are affected for the better.

The nomadic programmer

There's this broad metaphor I have, with no conclusions attached – just an attempt to describe dynamics. I've recently shared it with the commie ex-VP school teacher and he liked it, so I thought it could fit well with the other borderline stuff I host at Proper Fixation under the "wetware" category.

So. One recurring theme in the history of civilization is the conflict between nomadic and settled people. Nomads think that land is for feeding cattle and you move elsewhere once there's nothing left to graze. Villagers figure that land is for growing food, so you settle on it and fertilize it and irrigate it and stuff. Initially, nomads typically dominate the landscape, periodically attacking the settled villagers and taking their crops. However, the settled people eventually accumulate enough surplus to support cities, nation states and standing armies, extending their control to more and more lands and eventually exterminating the nomadic lifestyle altogether.

The way I painted this picture, I tend to side with the hard-working settled folk, the nomads being the parasitic losers I've depicted, and I think most of us civilized humans share similar sentiments. However, in my metaphor I side with the nomadic programmer, at least to a large extent, and I do so because of the meaning my metaphor assigns to "land".

The thing I find analogous to land in programming is problems, because that's where programmers live. Programmers live on (in?) problems in the sense of dealing with broken things most of the time – once something starts working, you move on to something that doesn't. In another sense, large problems or problem areas a programmer deals with define that programmer's territory. The programmer is in immediate demand to the extent that solutions to "his" problems are in demand; problems feed programmers. Strong programmers seek, in one way or another, to expand their responsibility to encompass more problems, and to preserve their existing responsibilities. And so on.

Now if we restate the respective worldviews of nomads and settlers in the terms of this metaphor, we'll get this. Nomads think that problems exist for solving them and you move elsewhere once there's nothing left to graze. Settlers think that problems exist for growing them, so they settle on them and fertilize them and irrigate them and stuff.

And now you can see why I'm inclined to sympathize with the nomadic programmer. Two other things fueling this sympathy are issues of personality to be discussed soon, and the fate of the nomad to be discussed immediately. And while the nomad is no longer the parasite, rest assured that he's still, in the long run, the loser.

Initially – in a young and small organization – nomadic programmers tend to dominate the landscape. There are more problems than people around. The nomadic programmer travels from one urgent problem to another, grazing through them as fast as he can. Occasionally he stumbles upon a settler who has settled on a problem near the nomad's territory and grown crops of code there. Well, if the problem occupied by the settler becomes urgent, or if the crops stand in the way of solving the nomad's adjacent urgent problem, the nomad will go ahead and brutally solve the settler's problem, wiping out his crops. The politics of the invasion will be trivial – a promise to deliver by the nomad carries lots of weight at this stage and the settler will not issue a counter-promise (to deliver in his own way) because he's a peaceful code-growing villager who isn't into stress which necessarily comes with delivering quickly.

However, the time goes by and sure enough, the settled people accumulate quite some surplus. What you grow on land is surplus wheat; what you grow on problems is surplus code. Code that wouldn't naturally grow on a problem – but now that the problem was fertilized by the original settlers, they've grown enough code on it to support whole cities, a nation state, and a standing army of programmers, all making a living by fiddling with this code.

The nomad starts running out of pasture. Sure enough, there are lots of problems just like there used to be. But you can no longer solve them because (1) now it's the majority and not a minority of problems that are already owned by someone (growing them rather than solving them) and (2) in most cases invasion is no longer an option. Now that the problem is owned by a nation state, responsible for lots of code and with lots of people working on that code, the nomad's promise to deliver quickly carries very little weight compared to the danger of irritating the sovereign. While it is quite likely still true that a nomad will probably deliver more quickly than the whole nation state team, the nomad will not be able to take over the entire responsibility of the team. (It is possible that the single reason for the latter is the problems grown by the team itself and that a few nomads could in fact handle the original problem. But it is irrelevant since problems that could have been avoided are no less real than problems that couldn't.)

So if the organization, by some decision making mechanism, lets the nomad invade the territory of the settled team and solve the stupid problem, and then the offended team, by some decision making mechanism, fights back by effectively going on strike, there is nothing the nomad will be able to offer the organization at this point. Of course it doesn't have to come to this, just like political conflicts don't have to come to full-scale wars, or personal conflicts to fist fights or court hearings. It's enough for the worst case scenario to be likely to work out in favor of A rather than B to shift the balance decisively in favor of A. Even if neither A nor B nor anyone making decisions affecting A and B actually think in terms of this scenario, things tend to evolve and adapt such that decisions are made in favor of A. And in our case, the nomadic programmer is B.

Solving problems just isn't the big thing in this organization anymore, just like the quality of life experienced by the inhabitants of some territory isn't the main theme in international politics. Perhaps there are ways to improve the quality of life in Siberia, however this is not nearly as important politically as the fact that there's already a guy exclusively responsible for the quality of life in Siberia. Perhaps Socialism with Chinese Characteristics could yield improvements in the lives of Siberians that Managed Democracy could not, however, if the Chinese try to act on this assumption, there will be a nuclear war. If what remains of a nomadic tribe somewhere in the region makes a similar attempt, then it will remain no more.

The disgruntled nomadic programmer reduces his ambition to merely being left alone to wander the remaining wilderness. However, this option is no more real for him now than the option of being left alone was available to the settler in the old days. Back then, the settlers were never safe since a nomad could always bump into them in an attempt to solve a related problem, and if their stuff got in the way, he'd rewrite or delete/disable their stuff. Now it is the nomad who is never safe since the nation states keep expanding their responsibilities into neighboring problems – having enough people to have some of them free for that some of the time.

(Actually having even partially idle workers on a team leaves few satisfying alternatives to an attempt at expanding the team's responsibilities since other teams are always happy to seize an idle worker. Likewise, back in the old days the nomadic programmer had few satisfying alternatives to invading and solving others' problems since otherwise he couldn't keep his promises to deliver. It's not (just) the intentions that fuel wars, it's (also) the situation.)

The nation states seeking to expand won't fight each other since the nomad is a much easier target, not having resources (time and reports) to look over his entire territory. Once a nation state team managed to take over some of that ever-shrinking territory, the nomad will never gain it back. Increasingly, the nomad has to reach compromises with neighboring nation states whenever his work is related to their work. Then it turns out that in order to be able to work on what he wants at all, he has to do it the way a chief commander or an officer of a nation state team wants him to do it – and then that in order to work on anything at all, he has to report to such a manager.

At this point the nomadic programmer can use his reputation and seniority to get pseudo-promoted to a non-productive position. Alternatively, he can actually become a report of a nation state team manager with whom the relationship is likely already strained – and his seniority, reputation and ambitions won't make the transition into this particular position of a report any smoother. Alternatively he can quit. His failure is now complete.

(It may sound like a natural thing for a nomad to change jobs fairly frequently – part of a lifestyle rather than the failure of that lifestyle. However, nomadic programmers are those who like to travel from problem to problem – not from job to job; some like the latter but some don't. A new job at a new place means a temporary, but possibly significant loss of confidence and efficiency. An African nomad won't necessarily welcome a relocation to Alaska.)

As I've said above, I have reasons having to do with my personality to side with the nomadic programmer, especially at the stage of mounting pressure from nation state teams. The people I tend to relate to most easily seem to be those who prefer freedom to power. A talented freedom-seeker with a strong sense of responsibility will accumulate, well, responsibilities much more quickly than reports – a lot of territory to wander, and no standing army to protect it. (The problem with reports is that you take their freedom by telling them what to do and they take your freedom through your responsibility for their actions; who wants reports?) Since many freedom-lovers disdain politics, they won't respect international borders – a problem should be solved, dammit; hence they're likely to initiate invasions.

However, while this means that I personally will tend to find myself sympathizing with particular nomadic programmers, this does not mean that theirs is the right way or something. For example, it is unclear which share of programming problems out there can really be "solved" – grazed through and left alone – and which problems actually require continuous care and gardening that a true nomad is not likely to supply. Also, whether there's a "solution" you only need to "maintain" or an "infrastructure" you want to "extend", the code needs a permanent owner. I don't believe in collective code ownership any more than in collective ownership of anything else – what it usually means is that everybody collectively fights over something. Therefore I think that ownership should generally be respected, and so a compromise which is, from a technical viewpoint, quite moronic, can otherwise be a great thing – a belief outside the nomad's way.

So while I know where my sympathies lie, I don't know which camp I'm in and this is why this metaphor doesn't come with any conclusions, just the dynamics. In fact I'd rather leave it without conclusions but I wouldn't mind expanding more on the dynamics. For example, some – but not all – settled civilizations were actually started by nomads enslaving argicultural villagers and settling among them. Apparently a similar distinction can be made between nation state teams of programmers; it is then interesting whether differences in their behavior can be traced to their different origins. Perhaps a person more entertained than appalled by the sort of perspective on the adventurous lives of programmers here presented is also the kind of person more entertained than appaled by the history of mankind in general and so could help develop this line of thought based on his knowledge of history. Could be fun.

Update (2009-08-18) – Chuck Moore: "I’ve met too many people who want to make a career out of a project instead of completing it" – the nomad's view of the settlers. Nomadism is apparent in other writing by Chuck Moore – his disdain for "complexity" (which implies dependency on large teams of people you ought to manage, annoying constraints imposed by systems made by someone else and other things nomads don't like), his firm opinion that distinct projects should have distinct code bases (customizability and "reuse" imply complexity and otherwise reduce the chances to "hermetically close" and truly complete a project), etc.

Pearls of wisdom

Proper Fixation always had more unfinished drafts than posts, but recently it's getting ridiculous. I do have a couple of drafts I seriously intend to finish (usually the drafts which don't make it to posthood during the first 4 hours or so go to the eternal drafthood land.) Until I'm able to think this stuff out to the point where I can share the results of my thinking, I figured I could share the far less scarce resource of Wisdom with ya.

***

Since I've violated the Golden Rule of Helping Friends with their PC Problems and attempted to help a friend with his PC problem, expectedly wiping out his hard drive in vain, I had many opportunities to explain the Programmer Paradox: how can a programmer fail to make a computer do as he wishes? While the difficulty of debugging a program without the source proved hard to explain to laymen, I think I've found a metaphor that does a good job. A programmer is to the blue screen of death what Mikhail Kalashnikov is to a loaded AK-47: just as helpless a victim as any other mortal, except for having a profound understanding of the mechanisms of his execution.

***

I would like to get some statistics on file encryption. For example, of all the files on the planet, X% are encrypted. Of all those files, Y% will never be read by someone due to encryption. Of all those files, Z% will never be read by malicious intruders. If I could lay my hands on the value of just one of these unknowns, I'd pick Z, because at least 100-Z% of the files will never be read by their owners. I would bet on Z lying somewhere between 0 and 1.

***

One of the key traits of good code is the ease at which it can be modified. One of the key traits of bad code is the high cost of modifying it. So good code is likely to deteriorate until it's bad enough to become hard to change, and bad code is likely to stay bad. In short, code has a strong tendency to end up bad.

This can sound worthlessly pessimistic, similarly, for example, to "It is easier to break a leg than it is to cure it, therefore, most legs end up broken." However, I think it's more analogous to aging – the accumulation of changes in an organism, observably causing most animals to end up dead. Similarly, code that is used will be changed, code that is changed will degrade, and code that degrades beyond a certain point will die.

***

Health tends to be simpler than disease. For example, everybody can brush their teeth but few people can treat cavities. Similarly, it's not very hard to maintain a sane development environment, but pretty hard to deal with the tide of bugs and of long-living branches resulting from a failure to do so. However, I'm generally optimistic about the chances of such cavities to be treated, and as usual, the optimism is based on the pain they cause – a strong incentive to seek and reward treatment.

***

There's this evolution vs Intelligent Design debate. Well, I don't know about life on Earth, but I sure have hard time believing in Intelligent Design in software. Code has to repeatedly survive exposure to users upon whom its fate depends. Yes, "users" can be a set containing just the author, but only if it's honest-to-God USAGE, that is, the author has to pay a price when the program is hard to use – like not getting important things done properly. Show me a program that someone finds useful and that wasn't subject to such evolutionary pressure, but rather was Intelligently Designed as useful.

I think that my intense hatred of the word "design" has to do with its prominent place in the speech of software creationists. These people are likely to constantly complain about not having enough resources to do The Right Thing in the ugly real world. They are also likely to give you software that you hate enough to wish to kill them, and be articulate enough to convince you that the problem is at your end, and fail to notice how this latter ability quadruples your desire to slash their body into square millimeter pieces.

***

I'll conclude with an off-topic request: if you know a good text advocating a collectivistic or other kind of heterodox approach to economics, I'd be very grateful for a reference. By "advocacy", I mean a text for laymen expressing support for a certain set of policies (as opposed to merely criticizing the effects of existing policies) – like Milton Friedman's "Capitalism and Freedom", for example.

The internal free market

This is going to be a bit atypical, because I'm going to talk, like, about organizing large teams of programmers. Which I rarely do, for the simple reason that it's not my problem. I'm not a manager, I don't think I'm likely to do a particularly good job as a manager in the near future, and I don't want to be a manager. As far as I'm concerned – if your problem is organizing lots of people, you brought it upon yourself. So this "internal free market" thing, which tends to work well according to my observations, is an exception to my general rule of not making or thinking much about "organizational" observations.

So, free markets. Basically a way to create incentives (because you have to compete) at the cost of redundancy (because of duplicated efforts by many competitors). A redundancy vs dependencies issue, if you like – several competitors means less dependence on each – and since I generally think redundancy is a fair price to pay for removing dependencies, you can guess that I'm leaning towards free market fundamentalism.

At this stage I'm skipping the detours where I'm dragging exciting pseudo-related stuff into this, like the subprime mortgage crisis and the enigmatically overwhelming support for Barack Obama by top programmers and tech bloggers. I'm skipping that to get to the simple point of there being no good way for an employer to create incentives for programmers with money.

How exactly this trick fails to work, and what kind of LOCs you get when you pay by the LOC is out of my scope. If you're sincerely surprised, there's lots of material for you to browse – Econ 101 Management being as good a place to start as any. The single thing I have to say about the "financial incentive" method here is that its failure isn't at all surprising to the free market fundamentalist.

In a free market, people solve their own problems, and (pay to) use the output of other people only when it helps them solve their problems, in a completely distributed way. Setting prices for things and subsidizing them is the trademark of a centralized, controlled economy. Now, does an employer paying by the LOC or by 1/#defects or whatever create an "internal free market", or an "internal government bureaucracy"? Without looking at soft stuff, like a hypothetical offense to sacred engineering values carried by the act of creating incentives, we can see that subsidizing LOCs will create surplus LOCs, just the way it works with agricultural surplus and everywhere else.

What would we do if we wanted a real internal free market? "Free market" means that we want to have people solving their problems, and let them "pay" each other in some way to solve them, without us controlling the latter process. In our specific context:

  1. "People" means "strong programmers" (or at least decent – or else why did "we", the employer, hire them, dammit?) Folks who, at the very least, like to be productive and have their stuff used. Maybe they also "like to solve puzzles", but not all do. For example, I hate puzzles, and have a strong preference for Alexandrian solutions. But you should still hire me.
  2. For those people, "solving their problems" means delivering user-visible features. This is the basic responsibility of developers towards the organization, this is what the organization is capable of judging (it better be), and this is what links the whole operation to reality through the external market forces.

The only question is, what does it mean for a developer to "pay" another developer? Paying with money makes no sense, not with our definition of "people". People who're into those transactions tend to be self-employed. However, developers do have their own currency, karma points or whatever term you prefer to use for it (they are all irksome; economics is to life what proctology is to anatomy – it's ugly because it's true). I know two kinds:

  1. Time. You can "pay" to developers, teams and the whole organization by volunteering to do particularly unsatisfying but important/urgent work.
  2. Code. When someone uses your code, they are paying you (I repeat, it's not you who are giving something to them; do not make this error or they'll stop using your code.)

Time and code are not unlike gold and printed money, because you can't make more time but you can make more code. However, proceeding with this analogy and trying to scale it to include inflation and such will expose my economical illiteracy and general lunacy to an extent making me want to stop now.

What we'll do now is examine how "trading" time and code works in programming, and how it creates incentives to invest efforts into the most needed things similarly to the way prices do in free market economies. We'll start with "trading code" – the less intuitive but the more fundamental kind of transaction.

You have to deliver something user-visible. The user couldn't care less about the guts making up your program – how you parse things, what your communication protocols are, which optimized libraries you use for processing bottlenecks, etc. However, you do care about these things, because they are really needed for things to work. With all these things, you could reuse "infrastructure" others work on (for example, me), or you could roll your own (let's ignore "international trade" where you use a third-party library for the moment).

To you, depending on me is a risk – who knows how many bugs I have, how well my stuff maps to your needs, etc. To me, on the other hand, having you reuse my code is the best thing that can happen to me during work hours. After work, better things can happen, in particular, those having to do with spending the money earned during work hours. But at work, the best thing that I can do is be productive and have others use my code. Lots of users is the workplace fortune equivalent to being rich in the real world. Do you see who pays who here?

What can an organization do to manage these infrastructure transactions?

  1. The "economical", "capitalist" solution: leave them alone except for securing them. "Leaving them alone" means not controlling them – not mandating the development and reuse of infrastructure and not assigning workforce to it. This means that by making my modules reusable, I'm only trying to please my internal users, so I'm likely to (try to) invest most effort into what they find important and helpful for doing their ultimate job. "Securing transactions" means something similar to the way public companies are forced to expose their accounting. If something becomes reusable code, it ought to have proper documentation, versioning, etc., and the organization must make sure it does.
  2. The "political", "socialist" solution: assign the task of developing a parser, an optimized library, etc. to a person/team – subsidize the parser (the price to a user is now lower – even if the parser is all buggy, a person is officially assigned to fix the bugs, and the responsibility for failures moves to that person and not to the one who decided to reuse the code). This means that the parser will be created even if in a "free company" nobody would want to develop and maintain it, knowing that most people wouldn't take the risk of using it for the benefits it provides. Leading to surplus crops of parsers.
  3. A further improvement on 2, the "communist" solution: force everyone to use The Parser. This means there are no "economical" means to punish the author whatsoever – where "punishing" means "not paying" and "not paying" means "stop using the code". However, there's still hope: you have political means to punish the author. For example, poke fun at the goddamn nightmare infrastructure, yell at the author, yell at his manager, ask your manager to yell at his manager's manager – a whole slew of counter-infrastructure measures. Victims of infrastructural communism use them all the time.

So this is how "trading code" is (more accurately, "can be") a better way of evolving reusable "infrastructure" than centralized planning. In general, the only thing I'm discussing is the reusable stuff – that's what organizations can optimize (or pessimize, creating useless "reusable" modules and not creating the actually needed ones). Nothing can be done about things which aren't reusable by definition, belonging to a single "feature"/"project" – those will have to be written once and only once no matter what.

What's wrong with this picture? Could be many things, but one thing I'll talk about (because I have a good answer for it) is the problem of "instant gratification"/"disruptive changes"/"local optimums"/etc. There are grand things that just can't be done by small incremental changes, the by-products of work on "specific features". You really need a person/team assigned to these things. This is somewhat similar to economies of scale which can be achieved by purchasing expensive machinery. How are many small farmers/shoe makers/etc. going to raise money for that machinery without central planning, if they're all busy with their small short-term profits?

This is where entrepreneurs come into play. Entrepreneurs are people with fire up their asses. Normal people want enough money to get by, enough money to not worry about money, or enough money to not have to work for money. Entrepreneurs want more money than they can sensibly spend during the decades of their lifetime. And they want it because they desperately need that money to feed the fire raging up their asses. When they see a potential for making truckloads of money, many of them are willing to put their own savings on the line to chase that chance.

This psychological profile is a speculation of mine – my best attempt to comprehend the inexplicable behavior of making efforts and burning nerve endings to make more money than you could possibly need. However, I do have motivation which is quite similar in the context of our "economics of programming" analogy. I'm a "programming entrepreneur", or at least I have, um, the same trademark proctological fireworks. I'm thrilled by opportunities to make stuff that, like, everybody will use, everything will depend on, …and everyone will want a piece of me when it breaks – so? It's still worth it.

I can't make such stuff as a by-product of working on something reasonably user-visible. I need to be assigned to it. What are the savings that I can put on the line? Time invested into doing unsatisfying, but important work. I call my own way of making these deals "buying development with debugging". I'm usually willing to debug the weirder of the urgent problems, although it's not much fun by itself, because it translates to a lot of karma points. I can then spend those karma points by working on what I want 80% of the time, 20% being the continuous urgent debugging tax.

Again, there's more than one way for that kind of "entrepreneur" to start a programming venture:

  1. The "economical" way – spend my own time implementing my ideas. Like a "real" entrepreneur putting his savings on the line and forced to look at his company bleeding that money if it doesn't take off, I will want to stop as soon as possible when I realize that I'm failing. Those so-called "organizational karma points" you gain in the trenches have better uses than wasting on the development of worthless stuff nobody will use.
  2. The "political" way – convincing "the government" (a manager) that my idea is worth implementing, and have someone assigned to it. Now nobody wants to admit the failure early on. I'm not losing anything when someone else struggles with the implementation – "I could do it better". The person working on the thing isn't really held responsible for the failure, either – not his idea, so why not keep trying to make it work? Everybody wants to make his stuff work and be used, after all. And the manager won't want to admit the failure because of all people, he'll get most of the blame. Therefore, the worthless effort will not be stopped for a lot of time.

Free market supporters are sometimes blamed for disrespecting people and reducing human nature to primitive egoism. Well, the only thing I can say is that I sure am a Good Person (how could it be different?), I respect myself lots, I successfully "launched" more than one "programming venture" both ways – "economical" (DIY) and "political" (persuasion), and of each of these two kinds, some succeeded and some failed.

And believe you me, deep down I refuse to take responsibility for the failing "politically launched" projects even now when we talk about it. On the other hand, the "economically launched" failures are – seriously – the best thing that happened to me in my professional life. I attribute most of my occasional successes – or, more accurately, non-failures – to lessons learned from the DIY failures, which I had no choice but admit responsibility for. (Damn, that was painful. To the extent that wasn't on my job description.)

Now, I'm not an "internal free market fundamentalist", simply because I know much more about programming than I do about economics, and obnoxious/oversimplified opinions usually correlate with ignorance. However, my experience seems to show that "internal free markets" are healthy enough to sustain continuous improvements on many scales, and eventually punish both "greedy" "instant gratification" techniques of pleasing managers/customers and architectural masturbation, promoting solid work.

And if you're not a manager (I mostly care about non-managers, guys and gals like me, you know), I think this quasi-economical angle can contribute to your ability to look at some young initiative around you and say "Hm, this might work out" and conversely "Epic fail on the way, I'm not going to touch this with a laser pointer, man". So, FYI.

The cardinal programming jokes

I'm depressed. What I'll do is I'll tell you the 3 cardinal programming jokes. And if it helps cheer me up, I'll consider my job well done.

I must warn you about those jokes. Firstly, they are translated from Russian and Hebrew by yours truly, which may cause them to lose some of their charm. Secondly, I'm not sure they came with that much charm to begin with, because my taste in jokes (or otherwise) can be politely characterized as "lowbrow". In particular, all 3 jokes are based on the sewer/plumber metaphor. I didn't consciously collect them based on this criterion, it just turns out that I can't think of a better metaphor for programming.

By the way, I was recently told by a very strong programmer that of all things, he wanted to become a plumber as a kid. 'Cause it was very interesting to him, the tools, the pipes, how you make the whole thing work. And then he felt he understood enough of it, so he figured he'd become a programmer instead. And now he is, and he has enough (virtual) pipes full of (virtual) shit to keep him curious about how to make it work for the rest of his life. By which I mean to say, hey, it's not just my bad taste, it is a good metaphor, see?

So, the jokes. Lowbrow, depressing stuff. You have been warned.

Expanding your skill set

A very important thing. You should be learning stuff. Yada yada.

With many things though, people have this strange tendency to avoid knowing them, and instead ask someone else unfortunate enough to already know them. Say, Makefiles. Is it just my experience or do people worldwide pretend to be incapable of dealing with a hairy Makefile, and leave its regularly scheduled tweaking to a small set of knowledgeable victims?

Or debugging of the lowest kind, with race conditions and creative memory corruption. People like to give up on that, as long as someone else can take over. "I just don't know how to proceed". Right.

Sometimes I wish I could put this claim to a test. Check if they'd say this at gunpoint. Or, more humanely and therefore much less cheaply, propose them $1M if they do know how to proceed. I bet they'd think a bit harder. If you're working on AI, specifically on preparing it to the Turing test, don't forget to teach it this principle, or else it has no chance of passing for a human.

I find that the following describes the double-edged sword that is skill set expansion quite well:

A plumber and his apprentice pay a visit to a manhole requiring their attention. The plumber goes down the manhole, and the apprentice stays above with the toolbox. The plumber asks for wrench #3, and the apprentice puts the wrench into his hand. 2 minutes pass. "Wrench #5!" The apprentice finds the wrench and passes it to the plumber. 5 more minutes. "Wrench #6!" The plumber is given that, takes a couple more minutes and finally comes out.

The next scene should really be a small piece of pantomime, but I'll have to get by with words alone. Not unexpectedly for this type of joke, the plumber comes out with his arms covered with excrement. He slowly sweeps his right hand over his left arm, then the left hand over the right arm, shakes his hands and reaches for something to wipe them with. And to the apprentice he says:

"Watch and learn, son, or you'll be passing wrenches for the rest of your life".

Really, you should learn things. Expand your skill set. Who wants to be passing wrenches?

Layers of abstraction

Abstraction is good. Or should I say legitimate. Or should I say inevitable. I mean, you have to count on something. Something has to work, because you can't build things on top of nothing.

Except it won't work. That something you build things on top of won't work.

What's that? "Whining"? Yep, definitely. This here is whining.

Whining is good. Or should I say legitimate. Or should I say inevitable. Because if you aren't allowed to whine about frigging data channels which drop chunks of data and duplicate chunks of data because some fucking hardware subcontructor couldn't be bothered to implement arbitration for shared data access, if you aren't allowed to whine about that…

If you aren't allowed to whine about that, you should be allowed to whine about memory, which flips bits, and zeros bytes, and it does so once per hour for some weird sequence of accesses having nothing to do with the address where data actually changes. Fuck that, OK? Fuck DDR2. Fuck its controllers and the zillions of their configuration parameters.

A plumber climbs out of a manhole, this time without a preamble, and his arms are covered with – guess what? – excrement! A beautiful little girl in a beautiful white dress happens to pass by. The plumber seizes the opportunity and (another piece of pantomime) quickly, but firmly sweeps his hands over the girl's white dress.

Little girl (appalled): AAAH!!

Plumber (outraged): Oh yeah? I bet you love to take a shit though.

Yep. You love to allocate objects in memory, don't you? Megabytes of them. And then a board designer decides to wipe his filthy hands with your beautiful white huge software system. Debug that, you perverted memory-addicted individual.

Taking pride in your work

And still, I actually like my work, on a level. Why? It feels inherently cool to design stuff that becomes this bunch of tiny parts, transistors and all, switching hundreds of millions of times a second, and then to write code that manages all the flying circus.

I know people who feel the same about computer vision. People for whom it's a personal priority to work on computer vision, where they are given images and they look for stuff in them. Who wants to be doing that? Who wants to be responsible for the solution of a problem that can't even be precisely defined? Me, I wanna be doing hardware.

What do I actually do most of the time though? I eat hexadecimal. I sit near a debugger, and I keep hitting Page Up in a memory view window, to find the beginning of the array that overwrote this piece of data (I guess the element size from the repetitive patterns and such), and along comes a computer vision geek and he says, "damn it, man, you got out of the Matrix!"

Well, I dunno, I find it much easier to guess what buggy code did to my memory than to find out "why" an algorithm thinks this here is a person when in fact it's a shade of a tree. Because if you look closely at the pixels, the shade kinda looks like a person, but of course we could reject it based on its motion, but of course that would mean we'd approve these reflections over here based on their motion, but, but, but…

What my bogus example is saying is that you have lots and lots of cues but each can work both for you and against you, and now how do you weigh all that, without even a formal spec? I'd rather eat hexadecimal, thank you very much.

And we look at each other, and sincerely think that our jobs are pretty nifty, but the other guy's job is awful and how can he be doing it. And I suspect that if one looks at this from aside, one might wonder where the actual fun is, because there is actual fun in here, or so all the participants testify. And I think I know the answer.

An airplane lands, and passengers come out. One of them notices a guy underneath the airplane. As you'd guess, the guy is a plumber. The plumber touches some lock, and immediately gets covered by excrement streaming from an opening at the bottom of the plane.

The pantomime cleanup routine follows, and then comes the turn of the dialog.

Passenger (appalled): What on Earth makes you keep this job?

Plumber (proudly): Hey, I'm in the aerospace business!

The aerospace effect happens to different people with different things. With some, it's "Hey, I'm making real hardware!" With others, it's "Hey, I'm finding real objects in real images!" It's a good thing people are different, because so are the currents of excrement, and someone ought to swim in each. We can't all be passing wrenches.

Extreme Programming Explained

When I saw Kent Beck's "Extreme Programming Explained" in our office, I was shocked. I've already accepted the inevitable occasional purchasing of obscure C++ wisdom, exemplified by titles such as "Effective C++", "Exceptional C++", "Imperfect C++", "Modern C++ Design" and so on. Yikes. Oh well. As long as nobody uses the boost libraries in production code, they can entertain themselves by ordering whatever they want as far as I'm concerned.

But XP? A software development methodology? A programmer ordered a book on methodology? I have to find out who that was. Well, I found out and the shock became worse, because it was one of the cooler kids. Largely out of respect for that guy, I grabbed the book, took it home and read it. What follows is my book review. Spoiler: I say quite some positive things there.

XP and hire/fire dynamics

I didn't look for quotes on this subject, it's just that all of them caught my eye. Maybe it's because HR implications of things are inherently interesting, maybe it's because everything else in a "methodology" is life-threateningly boring, and maybe both. Anyway, the quotes are:

"Given the choice between an extremely skilled loner and a competent-but-social programmer, XP teams consistently choose the more social candidate."

Gotta love the "but". Well, what can I say about this approach? This approach would prevent the best pieces of software and hardware I've seen developed from happening. In all those cases, there was one or more "extremely skilled loners". More on that later, when we get to code ownership, open spaces, pair programming and the like.

For now, it's sufficient to say that a methodology preferring mediocre to "extremely skilled" and calling itself "Extreme Programming" is, um, interesting. That "competent-but-social" is a euphemism for "mediocre" is as clear to me as the fact that mediocrity, while related to lack of talent, is first and foremost a personal value.

"Here's a sad but repeated story: a development team begins applying XP, dramatically improves quality and productivity, but then is disbanded, its leaders fired and the rest of the team scattered. Why does this happen? … The team's improved performance shifted the constraint elsewhere in the organization. The new constraint (e.g. marketing, who can't decide what they want fast enough) doesn't like the spotlight."

I have a fairly developed imagination, especially when it comes to organizational dysfunctions. For example, I can imagine how a team delivering a good result without working overtime will be appreciated less than a team delivering a bad result through heroic efforts (this example is from the book, too; of course the unappreciated team is the one "practicing" XP).

Getting fired over productivity? Because marketing can't invent features fast enough? What??

Now, that certainly challenges one's imagination skills. But we aren't the kind of people to turn down a challenge, are we? OK, lez do it. Imagine. Imagine an organization so rotten, so disgusting, so peculiarly brain-damaged that it can fire people because they work too fast to invent new work for them. Hooray! We did it! I see it right in front of my closed eyes!

But wait, what's that? Are those unfortunate victims of corporate idiocy Extreme Programmers? The competent-but-social programmers? If they're so "social", can't they see what stinky a hole they've landed at? You don't get fired from such places; you quit. I mean, realizing that you're in a hopeless human quagmire is among the most basic social skills – even I have it.

I failed the challenge. I can't imagine this picture.

Brad Jensen, Senior VP, Sabre Airline Solutions: "If programmers won't pair or if they insist on owning code, have the courage to fire them. The rest of the team will bail you ought."

Aha. XP, the two-edged sward. Get fired for using XP or get fired for refusing to use it. Just ducky.

No, really. I don't want to be mean. I won't even pick on the VP's "courage to fire" wording. You know what – I'll even praise this wording. The employer depends on a strong programmer more than the programmer depends on them, and firing an evil, antisocial strong programmer is likely the most courageous act a manager can commit.

All I'm saying is, XP is basically this religious cult, as more quotes will show. This cult is spreading in your organization. They fire people. They easily fire exceedingly productive people, "skilled loners" that won't "pair" and insist on code ownership – nothing evil or antisocial. They fire people just for refusing to accept their most controversial "practices".

Could it be the real reason for the occasional extermination of productive XP teams, assuming Kent Beck is right and the disbanded XP teams he saw actually were productive? With all those "loners" I work with and the kind of "process" we use, I can easily imagine my reaction to a hypothetical XP epidemic at my workplace. Probably something along the lines of "Come and get me! Let's see who gets fired first."

XP "practices"

Why do I place the word "practice" in quotes? I dunno, do you know a better way to typographically mark stupidity? The word kinda reminds of "spiritual practices", the religion thing again. But primarily it reminds me that there's just too much money in this industry. I mean, I have a relative who makes and sells ceramics. Neither she nor her employees use any "best practices". There's not enough money in ceramics for "practices"; you have to work. No time to listen to people talking about work, hence no market for their talks and writings.

This term is annoying on so many levels that I could go on for ages. For example, the words "best practices" and "questionable practices" convey infinite amount of ignorant, arrogant idiocy. You could never pass an exam citing "best practices" – you are supposed to analyze the problem and prove your solution. But in the industry it's perfectly possible to get away with, and even get promoted by substituting analysis with rules of thumb, irrelevant to the case in point as they may be.

OK, enough said about the term "practices"; maybe it's just me. Let's talk about some practices.

Sit Together

XP likes open space. I don't like open space. I'm exceptionally good at ignoring external events when I concentrate, to the point where it takes me 10 seconds to understand what you're saying when you interrupt me in the middle of something. But there's a limit to this autism, and when lots of conversations I can relate to, intellectually and emotionally, happen around me, I can't work.

Many people have talked about the state of "flow" and how you don't want to interrupt someone when they're in it because they're extremely productive, and how interrupting someone all the time means they work at a fraction of their full speed. Myself, I "value" communication as this would be called in XP. I like to talk to people, and people like to talk to me. Lots of them. And much as like it, I like flow, too, and closing the door in my room is quite a Best Practice.

To be fair, with XP, you won't be able to concentrate behind a closed door, either, because of…

Pair Programming

We'll start with some quotes:

"Personal hygiene and health are important issues when pairing. Cover your mouth when you cough. Avoid strong colognes that might affect your partner."

"When programmers aren't emotionally mature enough to separate approval from arousal, working with a person of the opposite gender can bring up sexual feelings that are not in the best interest of the team."

"In Figure 6 the man has moved closer to the woman than is comfortable for her. Neither is making his or her best technical decision at this point."

"Programmers that won't pair." Sounds more like "programmers that won't couple."

"I like to program with someone new every couple of hours…"

OK, OK, this is neither emotionally mature nor in the best interest of the team. Enough with this sort of quotes.

XP measures time in "pair-hours", since all production code must be written in pairs. Since we programmers are good at math, it's easy to see that this halves your task force. Kent Beck claims that pairs are more than 2x more productive than 2 people working alone, so it's a net gain. Is that really so, even if each of the 2 people can close the door?

I worked in teams of size 2 or 3, with everybody going pretty much at full speed, and I can't imagine gaining back the 2x spent on sitting together. Maybe if you're doing something so extraordinarily boring and trivial that you can barely move without talking to someone about it.

No, really, I think XP is for writing huge piles of straightforward code. And here's my proof.

Shared Code

As we've seen, in XP, there's no code ownership unless you want to get fired. The code is owned by the team, and everybody can and is encouraged to change ("refactor") any piece of code, any time.

This concludes the triad of XP practices that I'll call "The Coupling Practices", both because of my emotionally immature amusement with the quotes above and because of the coupling between everything (it's one system owned by everyone, not separate modules with separate owners). The Coupling Practices are:

  • Open space
  • Work in pairs
  • No code ownership

Kent Beck raises just one objection to the shared code practice – people might act irresponsibly and make expedient changes. Well, paranoid attitude towards coworkers is not one of my many sins; if you trust someone to work for your company, then you trust them to act responsibly most of the time, don't you think? My objections are different.

First, the coupling. I'm a great believer in Conway's Law: Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations. I'm the great believer that I am simply because I've never seen this law fail in practice. Strong module boundaries mostly appear at social boundaries – code ownership and responsibility – or at technical boundaries (different programming languages, different processes, different machines, kernel space/user space).

If everybody owns all modules, and unless each module is a separate kernel module, they'll quite likely become one big hairy program. And I don't like coupling, at least not in this sense. Sure, XP encourages refactoring, which should take care of the problem; I just never saw anything beat Conway's Law. To me, it's like defeating the law of gravity with spiritual practices.

My second objection is that shared ownership can't work except for trivial code, and this is the proof of the "XP is for trivial stuff" claim I've made above. Let me start with an example.

I used to think that ASIC engineers, the kind that write hardware description code in Verilog or VHDL, are sociopaths. I mean, the ones I've met were very helpful and patient when it came to explaining things to my feeble programmer mind, but apparently they couldn't stand each other. If one of them even touched another's code, the other guy would go to pieces. As a programmer, I was used to the Shared Code practice (you don't need XP to have a happy tar pit of coders friendly patching each other's stuff). So these hardware types scared me off.

And then I began to understand what you were saying in Verilog. In hardware, you have a bunch of variables, and you have code that computes the next values of variables given their current values. Hardware is like a huge recursive function of its registers' state, with the reset logic being the base of the recursion. Now, it's trivial to understand how a variable is updated – it's spelled using the usual arithmetics and logic and if-then-else and stuff. But understanding how all these updates propagate, and what will happen in, say, 4 cycles, and how the pipelines work together is, I dunno, impossible.

So you see, when you have this mental model of how this monstrous state machine works, and someone makes a change to it, you tend to be upset, because it's hard to read someone else's change and update your mental model. And why are they so sure their change makes sense in the first place? They can't possibly understand everything there is to understand! You could say that it's all irrelevant since XP is about software, and this is hardware, but to me, the line is quite blurred. It's code in a programming language. You can run this code on a simulator or an FPGA without manufacturing anything, just by copying bits. It's software.

Now, obviously we have the other extreme – code so straightforward that you can read it from top to bottom and understand it completely. Say, a one-screen script shoveling through the file system, doing simple regexpy parsing and printing statistics. You know, Practical Extraction and Report. They even have a Language for it. The question is, where is most software – is it like a Verilog module or like a screenful of Perl? (BTW, my Verilog and Perl guru is the same person.)

Well, let's see. Low-level interrupt handling and scheduling code is definitely like Verilog – touch my code and I'll go to pieces. Compilers are also like Verilog, because you have complicated algorithms and you can't just read them and say "aha, I see why this is doing what it's supposed to be doing". There are lots of heuristics and lots of knowledge about interaction of passes and complicated data structures. Optimized image processing code is also like Verilog, because there are all the precision considerations and knowledge about the target optimizer and platform. Computer vision code is also like Verilog… Everything I currently deal with is like that.

I mean, code is not like descriptive prose you'd put in a local wiki. You can't read it, say "I get it!", update it, and make the author happy with the "refactoring". Code is more like poetry: change this line, and now the next line doesn't rhyme, or you've broken the rhythm, or you've put angry words into a happy poem, that sort of trouble. Which is one reason to like code ownership.

It's probably useful to have a second programmer with "read access" to important pieces of code, so that at least 2 people can help debug each piece. You can get there with code reviews, and without the Coupling Practices, which I can't imagine working except for code so straightforward that I doubt there's much of its kind.

XP and CMM

Brad Jensen, Senior VP, Sabre Airline Solutions: "The pure XP projects have very few defects… (Even) the (impure) XP projects have very competitive defect rates, one to two defects per thousand lines of code. The Bangalore SPIN, consisting of ten CMM level-five organizations, reports an average of 8 defects per thousand lines of code."

"Defects per LOC". Interesting metric. How do we improve it? First, we don't record "defects". What's a "defect"? Are 10 reported problems caused by one defect or several? Depends on the "root cause analysis", which is of course up to us coders. The second important thing to do is to write more code. That guy working on the data-driven layout rendering should be taken out and shot. We have editing macros for that. You could squeeze 50K LOC out of rendering if it was spelled as code like it should be.

Who is stupid enough to use a metric encouraging people to misattribute problem reports, increase code size, or simply quit the silly job? Why, it's the Capability Maturity Model, of course. Meet XP's competitor: the dreadful CMM.

I could have received a certificate telling that I was trained in CMM, but I couldn't take it and asked my manager to bail me out (and so he did, THANKS!!). It was stupid to the point of physical pain. The CMM instructor, making $160/hour, talked like this: "Blah-blah-blah-PRACTICE!! Blah-blah-blah-ESTABLISH-AND-MAINTAIN!! Blah-blah-blah-GOAL!!" – yelling at some random word, so you couldn't even fall asleep in the absence of any kind of rhythm your brain could learn to ignore. The sleep deprivation and the countless repetitions caused some of the garbage to be ingrained in my memory. Here's an excerpt for ya.

CMM has 5 levels. CMM level 1 is where you operate right now: it denotes the ability to ship something. CMM level 5, the one mentioned in the interview from Kent Beck's book, denotes complete paralysis. To do anything, you have to write or update so many documents, have so many meetings with "relevant stakeholders", and to perform so many pointless measurements of the defects/LOC kind, that it's much more productive to just go postal and at least remove a CMM auditor or two from the face of the Earth in the process.

Levels 2 to 4 indicate various intermediate stages where paralysis is spreading, but you can still ship. For example, level 2 is called "Managed Process". "Managed Process is distinguished by the degree to which the process is Managed". I'm not making this up. There's a book called Capability Maturity Model Integration, and this book, heavy enough to kill a human, is full of this sort of stuff. Reading it is impossible.

And this is why I think XP is a great thing. (See? I promised I'll say positive things; and I'm not done yet!) I've actually read Extreme Programming Explained. About 75% of it seemed meaningless, but I made it through. The CMMI book, on the other hand, is pretty much infeasible.

CMM has hundreds of practices. XP has a couple dozens. CMM has extremely costly certification process. XP doesn't. CMM forces you to write a zillion documents. XP forces you to write a zillion tests, documentation of features, and story cards. CMM is about lengthening the development cycle. XP is about shortening the development cycle.

CMM is inflicted on you by customers who believe the lies of the worthless bastards from the Software Engineering Institute that their "process" will make you ship quality software. XP attempts to establish itself as an alternative legitimate "process" in the realm of suspicious customers and costly vendors. XP tries to sell commonsense stuff like automated testing to the desperate programmers working on a single-customer product with scarce resources. XP also tries to sell itself to the customer, relieving the desperate vendor's programmers from the insane, intolerable overhead of level 5 CMM paralysis.

XP brings hope to dark, wet, stinky corners around the world. I sincerely think it's great. The permanent brain damage of that CMM training course makes me admire XP.

But then there are people who work for product companies and inflict XP upon themselves. It's like paying taxes the government doesn't ask you to pay. XP is an alternative to CMM. You need it if your customer or manager requires you to use a ready-made methodology, out of lack of trust. If you don't have that problem, make yourself your own "process". You are blessed with an opportunity to just work, and organize your work as you go and as you see fit. Why "practice" stuff when you can just work?

How many product companies use CMM, XP or any other ready-made process, and how many make their own process? How many successful companies borrow existing "methodologies" without being forced to do so? I don't know the numbers, but my bet is that most of them don't.

XP and religion

"EMBRACE CHANGE"

This quote is right from the book cover. "Extreme Programming Explained. EMBRACE CHANGE." Does it freak you out the way it freaks me out? Maybe it's because of the cultural gap created by my Russian origins? Nay, I know plenty of English slogans I can relate to. Say, "Trust a condom". Beats "Embrace change" hands-down. Changes come in two flavors, good and bad. Should I "embrace" both kinds?

"Even programmers can be whole people in the real world. XP is an opportunity to test yourself, to be yourself, to realize that maybe you've been fine all along and just hanging with the wrong crowd."

Is this a religious cult or what?

"The key to XP is integrity, acting in harmony with my true values… The past five years have been a journey of changing my actual values into those I wanted to hold."

"Journey". Talking about being good. Do you like hippies? I like hippies more than nazis. I like XP more than CMM. But IMO the hippie world view and general style is suboptimal.

"With XP, I work to become worthy of respect and offer respect to others. I'm content to do my best and strive always to improve. I hold values I'm proud of and act in harmony with those values."

"I have seen people applying XP bring renewed hope to their software development and their lives. You know enough to get started. I encourage you to start right now. Think about your values. Make conscious choices to live in harmony with them."

It's a religion, people.

I'm not strictly against religion. I even have vague respect towards religious people, those stating that their purpose in life is to be good and that they know how to do it. (I similarly respect the intentions and discipline of various development methodology followers.) Of course when a religious person behaves as an asshole, it's way more annoying than a secular asshole is, because the latter at least doesn't state that he's all about being good and living right. (I noticed that many people who love to talk about "proper process" commit the worst atrocities.)

Some religions or sects are awful, to the point of human sacrifices. (CMM.) Most modern mainstream religions are kinda nice though, and they promote good Values and Practices, like not killing, not stealing and helping each other. (XP promotes automated testing, short build, integration and release cycles.) Many religions have peculiar regulations when it comes to sex. (XP has The Coupling Practices.) This is when religious people start to get annoying, to the point of throwing stones at people who are improperly dressed, not to mention engaging in forbidden sexual relationships. (Refuse to "pair" and XP gets you fired.)

Interestingly enough, the majority of religious people feel much stronger about the peculiar regulations of their religion than the universally recognized atrocities, which are also forbidden by their religion. They never gather around prisons to throw stones at convicts going on vacation, they are more likely to do so near a gay parade. (The Senior VP from Kent Beck's interview didn't recommend firing over lack of tests, which all of us would recognize as a crime; he recommended firing over the violation of The Coupling Practices.)

And this is when my patience towards religious people evaporates. So there's a gay parade. Got a problem with it? Shove it! I'm not gay, but the idea of harassing people based on this sort of criteria is disgusting. (I think I can blend in very well in a team engaging in pair programming and collective ownership, but I know awesome programmers who can't work that way; just try to get them fired.)

But still, religion is not all bad. I think that there are religious communities which are less inflicted with drug abuse compared to your average secular community. (It's not unlikely that many XP teams have more tests and less bugs than "secular" teams using no "standard" process.) Few people have the courage and the patience needed to openly attack religion; in particular, holy scriptures tend to be long and obscure. (Skim through an XP or especially a CMM book and try to stay awake or find refutable claims.)

Young people who aren't cynical enough to realize that bullshit is free and hence everybody is going to bullshit you if only you let them tend to be an easy target for religions seeking expansion. (When I started working as a programmer, I was shocked how messy everything was, and cheerfully welcomed every "methodologist" about to create order out of the chaos. Today, even saying "Best Practice" near me is nor recommended.)

Summary

I treat XP as a religious lifestyle (true of any methodology). XP is not a bad religion. Its primary virtue is the threat it poses to the cannibalistic cults such as the CMM. While I don't want to be anywhere near XP churches, I think religion is an inevitable attribute of human existence, and as far as religions go, XP is not at all bad.