Extreme Programming Explained

June 30th, 2008

When I saw Kent Beck's "Extreme Programming Explained" in our office, I was shocked. I've already accepted the inevitable occasional purchasing of obscure C++ wisdom, exemplified by titles such as "Effective C++", "Exceptional C++", "Imperfect C++", "Modern C++ Design" and so on. Yikes. Oh well. As long as nobody uses the boost libraries in production code, they can entertain themselves by ordering whatever they want as far as I'm concerned.

But XP? A software development methodology? A programmer ordered a book on methodology? I have to find out who that was. Well, I found out and the shock became worse, because it was one of the cooler kids. Largely out of respect for that guy, I grabbed the book, took it home and read it. What follows is my book review. Spoiler: I say quite some positive things there.

XP and hire/fire dynamics

I didn't look for quotes on this subject, it's just that all of them caught my eye. Maybe it's because HR implications of things are inherently interesting, maybe it's because everything else in a "methodology" is life-threateningly boring, and maybe both. Anyway, the quotes are:

"Given the choice between an extremely skilled loner and a competent-but-social programmer, XP teams consistently choose the more social candidate."

Gotta love the "but". Well, what can I say about this approach? This approach would prevent the best pieces of software and hardware I've seen developed from happening. In all those cases, there was one or more "extremely skilled loners". More on that later, when we get to code ownership, open spaces, pair programming and the like.

For now, it's sufficient to say that a methodology preferring mediocre to "extremely skilled" and calling itself "Extreme Programming" is, um, interesting. That "competent-but-social" is a euphemism for "mediocre" is as clear to me as the fact that mediocrity, while related to lack of talent, is first and foremost a personal value.

"Here's a sad but repeated story: a development team begins applying XP, dramatically improves quality and productivity, but then is disbanded, its leaders fired and the rest of the team scattered. Why does this happen? ... The team's improved performance shifted the constraint elsewhere in the organization. The new constraint (e.g. marketing, who can't decide what they want fast enough) doesn't like the spotlight."

I have a fairly developed imagination, especially when it comes to organizational dysfunctions. For example, I can imagine how a team delivering a good result without working overtime will be appreciated less than a team delivering a bad result through heroic efforts (this example is from the book, too; of course the unappreciated team is the one "practicing" XP).

Getting fired over productivity? Because marketing can't invent features fast enough? What??

Now, that certainly challenges one's imagination skills. But we aren't the kind of people to turn down a challenge, are we? OK, lez do it. Imagine. Imagine an organization so rotten, so disgusting, so peculiarly brain-damaged that it can fire people because they work too fast to invent new work for them. Hooray! We did it! I see it right in front of my closed eyes!

But wait, what's that? Are those unfortunate victims of corporate idiocy Extreme Programmers? The competent-but-social programmers? If they're so "social", can't they see what stinky a hole they've landed at? You don't get fired from such places; you quit. I mean, realizing that you're in a hopeless human quagmire is among the most basic social skills – even I have it.

I failed the challenge. I can't imagine this picture.

Brad Jensen, Senior VP, Sabre Airline Solutions: "If programmers won't pair or if they insist on owning code, have the courage to fire them. The rest of the team will bail you ought."

Aha. XP, the two-edged sward. Get fired for using XP or get fired for refusing to use it. Just ducky.

No, really. I don't want to be mean. I won't even pick on the VP's "courage to fire" wording. You know what – I'll even praise this wording. The employer depends on a strong programmer more than the programmer depends on them, and firing an evil, antisocial strong programmer is likely the most courageous act a manager can commit.

All I'm saying is, XP is basically this religious cult, as more quotes will show. This cult is spreading in your organization. They fire people. They easily fire exceedingly productive people, "skilled loners" that won't "pair" and insist on code ownership – nothing evil or antisocial. They fire people just for refusing to accept their most controversial "practices".

Could it be the real reason for the occasional extermination of productive XP teams, assuming Kent Beck is right and the disbanded XP teams he saw actually were productive? With all those "loners" I work with and the kind of "process" we use, I can easily imagine my reaction to a hypothetical XP epidemic at my workplace. Probably something along the lines of "Come and get me! Let's see who gets fired first."

XP "practices"

Why do I place the word "practice" in quotes? I dunno, do you know a better way to typographically mark stupidity? The word kinda reminds of "spiritual practices", the religion thing again. But primarily it reminds me that there's just too much money in this industry. I mean, I have a relative who makes and sells ceramics. Neither she nor her employees use any "best practices". There's not enough money in ceramics for "practices"; you have to work. No time to listen to people talking about work, hence no market for their talks and writings.

This term is annoying on so many levels that I could go on for ages. For example, the words "best practices" and "questionable practices" convey infinite amount of ignorant, arrogant idiocy. You could never pass an exam citing "best practices" – you are supposed to analyze the problem and prove your solution. But in the industry it's perfectly possible to get away with, and even get promoted by substituting analysis with rules of thumb, irrelevant to the case in point as they may be.

OK, enough said about the term "practices"; maybe it's just me. Let's talk about some practices.

Sit Together

XP likes open space. I don't like open space. I'm exceptionally good at ignoring external events when I concentrate, to the point where it takes me 10 seconds to understand what you're saying when you interrupt me in the middle of something. But there's a limit to this autism, and when lots of conversations I can relate to, intellectually and emotionally, happen around me, I can't work.

Many people have talked about the state of "flow" and how you don't want to interrupt someone when they're in it because they're extremely productive, and how interrupting someone all the time means they work at a fraction of their full speed. Myself, I "value" communication as this would be called in XP. I like to talk to people, and people like to talk to me. Lots of them. And much as like it, I like flow, too, and closing the door in my room is quite a Best Practice.

To be fair, with XP, you won't be able to concentrate behind a closed door, either, because of...

Pair Programming

We'll start with some quotes:

"Personal hygiene and health are important issues when pairing. Cover your mouth when you cough. Avoid strong colognes that might affect your partner."

"When programmers aren't emotionally mature enough to separate approval from arousal, working with a person of the opposite gender can bring up sexual feelings that are not in the best interest of the team."

"In Figure 6 the man has moved closer to the woman than is comfortable for her. Neither is making his or her best technical decision at this point."

"Programmers that won't pair." Sounds more like "programmers that won't couple."

"I like to program with someone new every couple of hours..."

OK, OK, this is neither emotionally mature nor in the best interest of the team. Enough with this sort of quotes.

XP measures time in "pair-hours", since all production code must be written in pairs. Since we programmers are good at math, it's easy to see that this halves your task force. Kent Beck claims that pairs are more than 2x more productive than 2 people working alone, so it's a net gain. Is that really so, even if each of the 2 people can close the door?

I worked in teams of size 2 or 3, with everybody going pretty much at full speed, and I can't imagine gaining back the 2x spent on sitting together. Maybe if you're doing something so extraordinarily boring and trivial that you can barely move without talking to someone about it.

No, really, I think XP is for writing huge piles of straightforward code. And here's my proof.

Shared Code

As we've seen, in XP, there's no code ownership unless you want to get fired. The code is owned by the team, and everybody can and is encouraged to change ("refactor") any piece of code, any time.

This concludes the triad of XP practices that I'll call "The Coupling Practices", both because of my emotionally immature amusement with the quotes above and because of the coupling between everything (it's one system owned by everyone, not separate modules with separate owners). The Coupling Practices are:

Kent Beck raises just one objection to the shared code practice – people might act irresponsibly and make expedient changes. Well, paranoid attitude towards coworkers is not one of my many sins; if you trust someone to work for your company, then you trust them to act responsibly most of the time, don't you think? My objections are different.

First, the coupling. I'm a great believer in Conway's Law: Organizations which design systems are constrained to produce designs which are copies of the communication structures of these organizations. I'm the great believer that I am simply because I've never seen this law fail in practice. Strong module boundaries mostly appear at social boundaries – code ownership and responsibility – or at technical boundaries (different programming languages, different processes, different machines, kernel space/user space).

If everybody owns all modules, and unless each module is a separate kernel module, they'll quite likely become one big hairy program. And I don't like coupling, at least not in this sense. Sure, XP encourages refactoring, which should take care of the problem; I just never saw anything beat Conway's Law. To me, it's like defeating the law of gravity with spiritual practices.

My second objection is that shared ownership can't work except for trivial code, and this is the proof of the "XP is for trivial stuff" claim I've made above. Let me start with an example.

I used to think that ASIC engineers, the kind that write hardware description code in Verilog or VHDL, are sociopaths. I mean, the ones I've met were very helpful and patient when it came to explaining things to my feeble programmer mind, but apparently they couldn't stand each other. If one of them even touched another's code, the other guy would go to pieces. As a programmer, I was used to the Shared Code practice (you don't need XP to have a happy tar pit of coders friendly patching each other's stuff). So these hardware types scared me off.

And then I began to understand what you were saying in Verilog. In hardware, you have a bunch of variables, and you have code that computes the next values of variables given their current values. Hardware is like a huge recursive function of its registers' state, with the reset logic being the base of the recursion. Now, it's trivial to understand how a variable is updated – it's spelled using the usual arithmetics and logic and if-then-else and stuff. But understanding how all these updates propagate, and what will happen in, say, 4 cycles, and how the pipelines work together is, I dunno, impossible.

So you see, when you have this mental model of how this monstrous state machine works, and someone makes a change to it, you tend to be upset, because it's hard to read someone else's change and update your mental model. And why are they so sure their change makes sense in the first place? They can't possibly understand everything there is to understand! You could say that it's all irrelevant since XP is about software, and this is hardware, but to me, the line is quite blurred. It's code in a programming language. You can run this code on a simulator or an FPGA without manufacturing anything, just by copying bits. It's software.

Now, obviously we have the other extreme – code so straightforward that you can read it from top to bottom and understand it completely. Say, a one-screen script shoveling through the file system, doing simple regexpy parsing and printing statistics. You know, Practical Extraction and Report. They even have a Language for it. The question is, where is most software – is it like a Verilog module or like a screenful of Perl? (BTW, my Verilog and Perl guru is the same person.)

Well, let's see. Low-level interrupt handling and scheduling code is definitely like Verilog – touch my code and I'll go to pieces. Compilers are also like Verilog, because you have complicated algorithms and you can't just read them and say "aha, I see why this is doing what it's supposed to be doing". There are lots of heuristics and lots of knowledge about interaction of passes and complicated data structures. Optimized image processing code is also like Verilog, because there are all the precision considerations and knowledge about the target optimizer and platform. Computer vision code is also like Verilog... Everything I currently deal with is like that.

I mean, code is not like descriptive prose you'd put in a local wiki. You can't read it, say "I get it!", update it, and make the author happy with the "refactoring". Code is more like poetry: change this line, and now the next line doesn't rhyme, or you've broken the rhythm, or you've put angry words into a happy poem, that sort of trouble. Which is one reason to like code ownership.

It's probably useful to have a second programmer with "read access" to important pieces of code, so that at least 2 people can help debug each piece. You can get there with code reviews, and without the Coupling Practices, which I can't imagine working except for code so straightforward that I doubt there's much of its kind.

XP and CMM

Brad Jensen, Senior VP, Sabre Airline Solutions: "The pure XP projects have very few defects... (Even) the (impure) XP projects have very competitive defect rates, one to two defects per thousand lines of code. The Bangalore SPIN, consisting of ten CMM level-five organizations, reports an average of 8 defects per thousand lines of code."

"Defects per LOC". Interesting metric. How do we improve it? First, we don't record "defects". What's a "defect"? Are 10 reported problems caused by one defect or several? Depends on the "root cause analysis", which is of course up to us coders. The second important thing to do is to write more code. That guy working on the data-driven layout rendering should be taken out and shot. We have editing macros for that. You could squeeze 50K LOC out of rendering if it was spelled as code like it should be.

Who is stupid enough to use a metric encouraging people to misattribute problem reports, increase code size, or simply quit the silly job? Why, it's the Capability Maturity Model, of course. Meet XP's competitor: the dreadful CMM.

I could have received a certificate telling that I was trained in CMM, but I couldn't take it and asked my manager to bail me out (and so he did, THANKS!!). It was stupid to the point of physical pain. The CMM instructor, making $160/hour, talked like this: "Blah-blah-blah-PRACTICE!! Blah-blah-blah-ESTABLISH-AND-MAINTAIN!! Blah-blah-blah-GOAL!!" – yelling at some random word, so you couldn't even fall asleep in the absence of any kind of rhythm your brain could learn to ignore. The sleep deprivation and the countless repetitions caused some of the garbage to be ingrained in my memory. Here's an excerpt for ya.

CMM has 5 levels. CMM level 1 is where you operate right now: it denotes the ability to ship something. CMM level 5, the one mentioned in the interview from Kent Beck's book, denotes complete paralysis. To do anything, you have to write or update so many documents, have so many meetings with "relevant stakeholders", and to perform so many pointless measurements of the defects/LOC kind, that it's much more productive to just go postal and at least remove a CMM auditor or two from the face of the Earth in the process.

Levels 2 to 4 indicate various intermediate stages where paralysis is spreading, but you can still ship. For example, level 2 is called "Managed Process". "Managed Process is distinguished by the degree to which the process is Managed". I'm not making this up. There's a book called Capability Maturity Model Integration, and this book, heavy enough to kill a human, is full of this sort of stuff. Reading it is impossible.

And this is why I think XP is a great thing. (See? I promised I'll say positive things; and I'm not done yet!) I've actually read Extreme Programming Explained. About 75% of it seemed meaningless, but I made it through. The CMMI book, on the other hand, is pretty much infeasible.

CMM has hundreds of practices. XP has a couple dozens. CMM has extremely costly certification process. XP doesn't. CMM forces you to write a zillion documents. XP forces you to write a zillion tests, documentation of features, and story cards. CMM is about lengthening the development cycle. XP is about shortening the development cycle.

CMM is inflicted on you by customers who believe the lies of the worthless bastards from the Software Engineering Institute that their "process" will make you ship quality software. XP attempts to establish itself as an alternative legitimate "process" in the realm of suspicious customers and costly vendors. XP tries to sell commonsense stuff like automated testing to the desperate programmers working on a single-customer product with scarce resources. XP also tries to sell itself to the customer, relieving the desperate vendor's programmers from the insane, intolerable overhead of level 5 CMM paralysis.

XP brings hope to dark, wet, stinky corners around the world. I sincerely think it's great. The permanent brain damage of that CMM training course makes me admire XP.

But then there are people who work for product companies and inflict XP upon themselves. It's like paying taxes the government doesn't ask you to pay. XP is an alternative to CMM. You need it if your customer or manager requires you to use a ready-made methodology, out of lack of trust. If you don't have that problem, make yourself your own "process". You are blessed with an opportunity to just work, and organize your work as you go and as you see fit. Why "practice" stuff when you can just work?

How many product companies use CMM, XP or any other ready-made process, and how many make their own process? How many successful companies borrow existing "methodologies" without being forced to do so? I don't know the numbers, but my bet is that most of them don't.

XP and religion

"EMBRACE CHANGE"

This quote is right from the book cover. "Extreme Programming Explained. EMBRACE CHANGE." Does it freak you out the way it freaks me out? Maybe it's because of the cultural gap created by my Russian origins? Nay, I know plenty of English slogans I can relate to. Say, "Trust a condom". Beats "Embrace change" hands-down. Changes come in two flavors, good and bad. Should I "embrace" both kinds?

"Even programmers can be whole people in the real world. XP is an opportunity to test yourself, to be yourself, to realize that maybe you've been fine all along and just hanging with the wrong crowd."

Is this a religious cult or what?

"The key to XP is integrity, acting in harmony with my true values... The past five years have been a journey of changing my actual values into those I wanted to hold."

"Journey". Talking about being good. Do you like hippies? I like hippies more than nazis. I like XP more than CMM. But IMO the hippie world view and general style is suboptimal.

"With XP, I work to become worthy of respect and offer respect to others. I'm content to do my best and strive always to improve. I hold values I'm proud of and act in harmony with those values."

"I have seen people applying XP bring renewed hope to their software development and their lives. You know enough to get started. I encourage you to start right now. Think about your values. Make conscious choices to live in harmony with them."

It's a religion, people.

I'm not strictly against religion. I even have vague respect towards religious people, those stating that their purpose in life is to be good and that they know how to do it. (I similarly respect the intentions and discipline of various development methodology followers.) Of course when a religious person behaves as an asshole, it's way more annoying than a secular asshole is, because the latter at least doesn't state that he's all about being good and living right. (I noticed that many people who love to talk about "proper process" commit the worst atrocities.)

Some religions or sects are awful, to the point of human sacrifices. (CMM.) Most modern mainstream religions are kinda nice though, and they promote good Values and Practices, like not killing, not stealing and helping each other. (XP promotes automated testing, short build, integration and release cycles.) Many religions have peculiar regulations when it comes to sex. (XP has The Coupling Practices.) This is when religious people start to get annoying, to the point of throwing stones at people who are improperly dressed, not to mention engaging in forbidden sexual relationships. (Refuse to "pair" and XP gets you fired.)

Interestingly enough, the majority of religious people feel much stronger about the peculiar regulations of their religion than the universally recognized atrocities, which are also forbidden by their religion. They never gather around prisons to throw stones at convicts going on vacation, they are more likely to do so near a gay parade. (The Senior VP from Kent Beck's interview didn't recommend firing over lack of tests, which all of us would recognize as a crime; he recommended firing over the violation of The Coupling Practices.)

And this is when my patience towards religious people evaporates. So there's a gay parade. Got a problem with it? Shove it! I'm not gay, but the idea of harassing people based on this sort of criteria is disgusting. (I think I can blend in very well in a team engaging in pair programming and collective ownership, but I know awesome programmers who can't work that way; just try to get them fired.)

But still, religion is not all bad. I think that there are religious communities which are less inflicted with drug abuse compared to your average secular community. (It's not unlikely that many XP teams have more tests and less bugs than "secular" teams using no "standard" process.) Few people have the courage and the patience needed to openly attack religion; in particular, holy scriptures tend to be long and obscure. (Skim through an XP or especially a CMM book and try to stay awake or find refutable claims.)

Young people who aren't cynical enough to realize that bullshit is free and hence everybody is going to bullshit you if only you let them tend to be an easy target for religions seeking expansion. (When I started working as a programmer, I was shocked how messy everything was, and cheerfully welcomed every "methodologist" about to create order out of the chaos. Today, even saying "Best Practice" near me is nor recommended.)

Summary

I treat XP as a religious lifestyle (true of any methodology). XP is not a bad religion. Its primary virtue is the threat it poses to the cannibalistic cults such as the CMM. While I don't want to be anywhere near XP churches, I think religion is an inevitable attribute of human existence, and as far as religions go, XP is not at all bad.