This is going to be a bit atypical, because I'm going to talk, like, about organizing large teams of programmers. Which I rarely do, for the simple reason that it's not my problem. I'm not a manager, I don't think I'm likely to do a particularly good job as a manager in the near future, and I don't want to be a manager. As far as I'm concerned – if your problem is organizing lots of people, you brought it upon yourself. So this "internal free market" thing, which tends to work well according to my observations, is an exception to my general rule of not making or thinking much about "organizational" observations.
So, free markets. Basically a way to create incentives (because you have to compete) at the cost of redundancy (because of duplicated efforts by many competitors). A redundancy vs dependencies issue, if you like – several competitors means less dependence on each – and since I generally think redundancy is a fair price to pay for removing dependencies, you can guess that I'm leaning towards free market fundamentalism.
At this stage I'm skipping the detours where I'm dragging exciting pseudo-related stuff into this, like the subprime mortgage crisis and the enigmatically overwhelming support for Barack Obama by top programmers and tech bloggers. I'm skipping that to get to the simple point of there being no good way for an employer to create incentives for programmers with money.
How exactly this trick fails to work, and what kind of LOCs you get when you pay by the LOC is out of my scope. If you're sincerely surprised, there's lots of material for you to browse – Econ 101 Management being as good a place to start as any. The single thing I have to say about the "financial incentive" method here is that its failure isn't at all surprising to the free market fundamentalist.
In a free market, people solve their own problems, and (pay to) use the output of other people only when it helps them solve their problems, in a completely distributed way. Setting prices for things and subsidizing them is the trademark of a centralized, controlled economy. Now, does an employer paying by the LOC or by 1/#defects or whatever create an "internal free market", or an "internal government bureaucracy"? Without looking at soft stuff, like a hypothetical offense to sacred engineering values carried by the act of creating incentives, we can see that subsidizing LOCs will create surplus LOCs, just the way it works with agricultural surplus and everywhere else.
What would we do if we wanted a real internal free market? "Free market" means that we want to have people solving their problems, and let them "pay" each other in some way to solve them, without us controlling the latter process. In our specific context:
- "People" means "strong programmers" (or at least decent – or else why did "we", the employer, hire them, dammit?) Folks who, at the very least, like to be productive and have their stuff used. Maybe they also "like to solve puzzles", but not all do. For example, I hate puzzles, and have a strong preference for Alexandrian solutions. But you should still hire me.
- For those people, "solving their problems" means delivering user-visible features. This is the basic responsibility of developers towards the organization, this is what the organization is capable of judging (it better be), and this is what links the whole operation to reality through the external market forces.
The only question is, what does it mean for a developer to "pay" another developer? Paying with money makes no sense, not with our definition of "people". People who're into those transactions tend to be self-employed. However, developers do have their own currency, karma points or whatever term you prefer to use for it (they are all irksome; economics is to life what proctology is to anatomy – it's ugly because it's true). I know two kinds:
- Time. You can "pay" to developers, teams and the whole organization by volunteering to do particularly unsatisfying but important/urgent work.
- Code. When someone uses your code, they are paying you (I repeat, it's not you who are giving something to them; do not make this error or they'll stop using your code.)
Time and code are not unlike gold and printed money, because you can't make more time but you can make more code. However, proceeding with this analogy and trying to scale it to include inflation and such will expose my economical illiteracy and general lunacy to an extent making me want to stop now.
What we'll do now is examine how "trading" time and code works in programming, and how it creates incentives to invest efforts into the most needed things similarly to the way prices do in free market economies. We'll start with "trading code" – the less intuitive but the more fundamental kind of transaction.
You have to deliver something user-visible. The user couldn't care less about the guts making up your program – how you parse things, what your communication protocols are, which optimized libraries you use for processing bottlenecks, etc. However, you do care about these things, because they are really needed for things to work. With all these things, you could reuse "infrastructure" others work on (for example, me), or you could roll your own (let's ignore "international trade" where you use a third-party library for the moment).
To you, depending on me is a risk – who knows how many bugs I have, how well my stuff maps to your needs, etc. To me, on the other hand, having you reuse my code is the best thing that can happen to me during work hours. After work, better things can happen, in particular, those having to do with spending the money earned during work hours. But at work, the best thing that I can do is be productive and have others use my code. Lots of users is the workplace fortune equivalent to being rich in the real world. Do you see who pays who here?
What can an organization do to manage these infrastructure transactions?
- The "economical", "capitalist" solution: leave them alone except for securing them. "Leaving them alone" means not controlling them – not mandating the development and reuse of infrastructure and not assigning workforce to it. This means that by making my modules reusable, I'm only trying to please my internal users, so I'm likely to (try to) invest most effort into what they find important and helpful for doing their ultimate job. "Securing transactions" means something similar to the way public companies are forced to expose their accounting. If something becomes reusable code, it ought to have proper documentation, versioning, etc., and the organization must make sure it does.
- The "political", "socialist" solution: assign the task of developing a parser, an optimized library, etc. to a person/team – subsidize the parser (the price to a user is now lower – even if the parser is all buggy, a person is officially assigned to fix the bugs, and the responsibility for failures moves to that person and not to the one who decided to reuse the code). This means that the parser will be created even if in a "free company" nobody would want to develop and maintain it, knowing that most people wouldn't take the risk of using it for the benefits it provides. Leading to surplus crops of parsers.
- A further improvement on 2, the "communist" solution: force everyone to use The Parser. This means there are no "economical" means to punish the author whatsoever – where "punishing" means "not paying" and "not paying" means "stop using the code". However, there's still hope: you have political means to punish the author. For example, poke fun at the goddamn nightmare infrastructure, yell at the author, yell at his manager, ask your manager to yell at his manager's manager – a whole slew of counter-infrastructure measures. Victims of infrastructural communism use them all the time.
So this is how "trading code" is (more accurately, "can be") a better way of evolving reusable "infrastructure" than centralized planning. In general, the only thing I'm discussing is the reusable stuff – that's what organizations can optimize (or pessimize, creating useless "reusable" modules and not creating the actually needed ones). Nothing can be done about things which aren't reusable by definition, belonging to a single "feature"/"project" – those will have to be written once and only once no matter what.
What's wrong with this picture? Could be many things, but one thing I'll talk about (because I have a good answer for it) is the problem of "instant gratification"/"disruptive changes"/"local optimums"/etc. There are grand things that just can't be done by small incremental changes, the by-products of work on "specific features". You really need a person/team assigned to these things. This is somewhat similar to economies of scale which can be achieved by purchasing expensive machinery. How are many small farmers/shoe makers/etc. going to raise money for that machinery without central planning, if they're all busy with their small short-term profits?
This is where entrepreneurs come into play. Entrepreneurs are people with fire up their asses. Normal people want enough money to get by, enough money to not worry about money, or enough money to not have to work for money. Entrepreneurs want more money than they can sensibly spend during the decades of their lifetime. And they want it because they desperately need that money to feed the fire raging up their asses. When they see a potential for making truckloads of money, many of them are willing to put their own savings on the line to chase that chance.
This psychological profile is a speculation of mine – my best attempt to comprehend the inexplicable behavior of making efforts and burning nerve endings to make more money than you could possibly need. However, I do have motivation which is quite similar in the context of our "economics of programming" analogy. I'm a "programming entrepreneur", or at least I have, um, the same trademark proctological fireworks. I'm thrilled by opportunities to make stuff that, like, everybody will use, everything will depend on, ...and everyone will want a piece of me when it breaks – so? It's still worth it.
I can't make such stuff as a by-product of working on something reasonably user-visible. I need to be assigned to it. What are the savings that I can put on the line? Time invested into doing unsatisfying, but important work. I call my own way of making these deals "buying development with debugging". I'm usually willing to debug the weirder of the urgent problems, although it's not much fun by itself, because it translates to a lot of karma points. I can then spend those karma points by working on what I want 80% of the time, 20% being the continuous urgent debugging tax.
Again, there's more than one way for that kind of "entrepreneur" to start a programming venture:
- The "economical" way – spend my own time implementing my ideas. Like a "real" entrepreneur putting his savings on the line and forced to look at his company bleeding that money if it doesn't take off, I will want to stop as soon as possible when I realize that I'm failing. Those so-called "organizational karma points" you gain in the trenches have better uses than wasting on the development of worthless stuff nobody will use.
- The "political" way – convincing "the government" (a manager) that my idea is worth implementing, and have someone assigned to it. Now nobody wants to admit the failure early on. I'm not losing anything when someone else struggles with the implementation – "I could do it better". The person working on the thing isn't really held responsible for the failure, either – not his idea, so why not keep trying to make it work? Everybody wants to make his stuff work and be used, after all. And the manager won't want to admit the failure because of all people, he'll get most of the blame. Therefore, the worthless effort will not be stopped for a lot of time.
Free market supporters are sometimes blamed for disrespecting people and reducing human nature to primitive egoism. Well, the only thing I can say is that I sure am a Good Person (how could it be different?), I respect myself lots, I successfully "launched" more than one "programming venture" both ways – "economical" (DIY) and "political" (persuasion), and of each of these two kinds, some succeeded and some failed.
And believe you me, deep down I refuse to take responsibility for the failing "politically launched" projects even now when we talk about it. On the other hand, the "economically launched" failures are – seriously – the best thing that happened to me in my professional life. I attribute most of my occasional successes – or, more accurately, non-failures – to lessons learned from the DIY failures, which I had no choice but admit responsibility for. (Damn, that was painful. To the extent that wasn't on my job description.)
Now, I'm not an "internal free market fundamentalist", simply because I know much more about programming than I do about economics, and obnoxious/oversimplified opinions usually correlate with ignorance. However, my experience seems to show that "internal free markets" are healthy enough to sustain continuous improvements on many scales, and eventually punish both "greedy" "instant gratification" techniques of pleasing managers/customers and architectural masturbation, promoting solid work.
And if you're not a manager (I mostly care about non-managers, guys and gals like me, you know), I think this quasi-economical angle can contribute to your ability to look at some young initiative around you and say "Hm, this might work out" and conversely "Epic fail on the way, I'm not going to touch this with a laser pointer, man". So, FYI.