Work on unimportant problems

June 9th, 2012

"Work on important problems": ~40900 results.

"Work on unimportant problems": ~18 results.

– Google (at the time of writing), tempting the contrarian in me

It seems obvious that some problems are important to solve and some aren't, as in, curing cancer is more important than delivering social gaming. Often, people lament the abundance of tech firms working on ultimately unimportant stuff, and advise to work on important problems and not just chase the money.

I guess I agree that some problems are ultimately more important than others. But I don't think it follows that working on the important ones is better.

Working on unimportant problems can create important side-effects. A whole lot of mission-critical, world-changing and even life-saving tech is a by-product of "unimportant" things – time-wasting infotainment products, or personal pet projects started without a grand noble cause.

For instance, GPU hardware was developed to run first-person shooters with increasingly fancier graphics. Today, it powers some of the largest high-performance computing clusters where "important" science is done.

Other types of processors powering HPC clusters weren't designed for HPC, either. Hardware originally designed for scientific computing is dead – Cray is the iconic example – and replaced by cheaper and more powerful microprocessors designed to run things like office software. Office software arguably solves no important problem: as Berglas convincingly argues, office automation results not in increased productivity, but in increased complexity of rules and regulations.

All popular programming languages and operating systems, without a single exception I can think of, began either as personal projects or commercial projects not aiming to solve any problem "important" by itself. People hacked on the stuff for pleasure (C, Unix, Linux, Python, Ruby, PHP), or to conquer the world of businessy/officy/enterprisey software (Windows, VB, Java, C#, ASP). One language more specifically designed for the implementation of important software is Ada – but most important programs are written in something else.

It can even seem that not much important computer hardware or software came out of institutions directly dealing with important problems. Much of the Internet protocols is one big exception, and arguably so is HTML – but probably not JavaScript.

And, certainly, it's the "unimportant" social companies that made publishing and coordination via Internet universally accessible. Myself, I'm not very fond of either Facebook, Twitter, etc. or the kind of political activity that's coordinated through these sites nowadays, but it's "important", without doubt – another important side-effect of unimportant time-wasting projects.

One might wonder how anything of importance can possibly come out of, say, FarmVille. I really don't know – however, I couldn't guess how anything of importance could come out of DOOM, and it did.

And then there's a reason why so much of the best tech comes out of the least "important" markets. These markets are big, and they're free. Important problems tend to imply a smallish scale, or heavy regulation, or both. So you can't finance the work, and/or can't get any work done anyway.

Consider the aerospace software market – there aren't many planes, but a whole lot of regulation. Philip Greenspun, a software entrepreneur, a flight instructor and an expert witness in both software-related and aviation-related lawsuits, had this to say about the Colgan 3407 disaster:

Who crashed Colgan 3407? Actually the autopilot did. ... The airplane had all of the information necessary to prevent this crash. The airspeed was available in digital form. The power setting was available in digital form. The status of the landing gear was available in digital form. ...

How come the autopilot software on this $27 million airplane wasn’t smart enough to fly basically sensible attitudes and airspeeds? Partly because FAA certification requirements make it prohibitively expensive to develop software or electronics that go into certified aircraft. It can literally cost $1 million to make a minor change. Sometimes the government protecting us from small risks exposes us to much bigger ones.

The same is happening in the automotive market, the healthcare market, etc. There's progress, of course, just nowhere near the progress in more frivolous areas – and much of the progress in "important" areas is a byproduct of progress in frivolous areas. As in, the best system for managing patients' records may well be Google Docs that doctors access from their iPads.

By the way, the importance of an issue correlates with the stupidity of rules, not just in technology, but in most things in life. The hoops you must jump through to get an "important" product out the door are not fundamentally different from airport security checks.

The airport security theater results in little added security. Likewise, the quality theater necessarily surrounding any life-saving technology results in little added quality. However, for much the same reasons, both are unavoidable. I've been working on automotive accident prevention systems for the last decade, and as time goes by, the regularly scheduled cavity searches are only getting worse.

So if you ask me – by all means, work on unimportant problems. They're often more fun to work on, and ultimately you never know how important they really are.

1. SusanJun 9, 2012

Or...work on something that is important to you – something that you know a lot about, something you are passionate about.

2. Prasd ChakkaJun 9, 2012

What do you think about FDA?

3. Yossi KreininJun 9, 2012

@Susan: I already do, I guess – it's processors and development tools and stuff – not end-user products, these, and could be a part of many kinds of products. I'm rather impassionate about end-user products...

@Prasd Chakka: FDA I don't know much about. I guess they get some of people killed by delaying the approval of drugs and then save others by that very delay; I'm not sufficiently informed to tell which group is larger.

4. Brian CardarellaJun 9, 2012

Some of the most progress I've made as a developer has been when I work on side projects just for fun. My day job I am working in domains I am already comfortable with, when I work in new areas is where I find great challenges.

5. Senthil NayagamJun 9, 2012

Unimportant ones keeps us to set lower expectations, so stress free and higher productivity

6. Byron HathawayJun 9, 2012

Don't forget to include daydreaming in your suggestion to work on "unimportant" things. So many of our truly great and important ideas, not to mention art, comes from dreamers.

Thanks for the post...

7. Donnie BerkholzJun 9, 2012

The parallel here is scientific research, with a balance between basic (no immediate payoff) versus applied projects.

There's an important distinction between things that are expected to be unimportant and those that may provide foundational work for future efforts.

In the former case, it may be such a tiny niche that it's very unlikely to have a big impact. But in the latter case, the payoff may come 10 years down the road — but it never would've happened if the initial work never did either.

8. Aristotle PagaltzisJun 9, 2012

I cannot decide whether this is a generalisation of A Mathematician’s Apology itself or of its opposite. Both, perhaps.

9. Yossi KreininJun 9, 2012

An Angry Bird's Apology, perhaps.

I liked, in that Mathematician's Apology, how chess was to math what pop music was to music. (Indeed, my own appreciation of math doesn't extend that very far beyond chess, just like my appreciation of music doesn't extend very significantly beyond pop music.)

10. JohnJun 9, 2012

Feynman is one of the great ambassadors for this approach. He worked on hard problems but also let his curiosity lead him into seemingly frivolous areas.

11. DmitryJun 10, 2012

Можно я по-русски? Ответ на английском меня не смутит, если что.

1. Не меньшее количество вещей, изменивших мир, появилось в результате работы именно над важными задачами. Lisp родился в попытках создать AI, TeX родился в попытках опубликовать хорошую книгу, а Норберт Винер в своей "Кибернетике" много пишет о плодотворном сотрудничестве с мексиканским врачом по фамилии Розенблют, на тему импульсов сердечной мышцы и прочей медицины. Ну, про интернет вы сами написали.

2. Если человек влюблён в то, что он делает, то его труд не пропадёт для человечества, даже если это фейсбук или твиттер. Но в 95% случаев случается другое: профессионалы уныло пашут за бабло над какими-нибудь Аллодами (игра такая), к которым без бабла они бы и близко не подошли. Эффект косвенных ноу-хау они могут использовать как плохое оправдание — а на самом деле они лишь получат своё бабло, а ноу-хау будут сделаны остальными 5%. Каждому нравится думать, что он именно в тех 5% :)
(http://en.wikiquote.org/wiki/House_(TV_series)/Season_4#Living_the_Dream_.284.14.29)

12. Yossi KreininJun 10, 2012

@Dmitry: I believe AI is a swindle, illustrating a part of the reason why the importance of "important" stuff is debatable... People can't be trusted with "importance evaluation", neither morally nor intellectually. This is partly why I don't quite share the contempt for "money chasing" that I sense in your comment.

As to "poor excuses": working on life-saving automotive apps, yada yada, I really don't need any, however, I wouldn't ask for any if I worked on games or whatever. I personally don't think technical people have anything like a responsibility to advance the state of mankind through technology – mostly because I don't think it's technology that, by itself, advances the state of mankind. (A simple thought experiment taking it to the limit: is mankind better off stripped of all technology and malice, or equipped by the best of technology and most cunning malice? This shows you what our biggest problems really are.)

13. DmitryJun 10, 2012

AI, может, и swindle, но в то же время это и мечта! И господин МакКарти, мир его праху, никого не хотел обмануть, он же во всё это искренне верил типа. Человечество хоть и осталось без AI — зато с Лиспом.

Насчёт мысленного эксперимента: похожий провёл Станислав Лем. Какой-то учёный (забыл в какой книге) в одной пробирке разводил адово злобных, эгоистичных и коварных микросуществ, а в соседней — добрых и альтруистичных. И оказалось, что технологически и вообще по любым объективным признакам популяции развиваются одинаково :)

14. Yossi KreininJun 10, 2012

Dreams or swindles – they have to be financed by someone :)

As to populations – our bodies are themselves populations of microorganisms, and these microorganisms kind of do play nicely together (when they don't, something is happening along the lines of cancer).

15. DmitryJun 10, 2012

Признаю свою (или Лема) неправоту. С другой стороны, тезис о том, что добро/зло важнее технологии, кажется натянутым, потому люди по-любому меняются очень неспешно. За последние сто лет люди остались такими же, а технология рванула по экспоненте. Попытки же рывком изменить людскую природу заканчивались плачевно.

Раньше, чем люди додумаются быть добрее, родится технология, которая просто их такими сделает, хехе. Слава эпохе добрых биороботов.

16. Yossi KreininJun 11, 2012

I wasn't proposing to change human nature – just saying that it's not technological limits that are the problem.

17. SchlingelJun 11, 2012

I wanted to add a comment but Susan nailed it. So I just have to second her thought.

18. Brian BalkeJun 11, 2012

I'm afraid that I find your excerpt from Greenspun's blog to be somewhat misleading. The comments on the post indicate that there is better software available, and that there was a fair amount of human involvement in the development of the disaster.

19. Yossi KreininJun 11, 2012

Well, there are many comments on that post and they are not all in agreement.

I can tell you this with certainty: for better or worse, the pace of change in automotive software, specifically, is much slower than in consumer software. The "calm, rational" way of looking at it is, when errors are costly, you get conservative and try less things and when you try less things, you find less things that turn out really good.

20. Alexis YL ChanJun 11, 2012

I agree with Susan's comment. I think we need to separate the nature of the problem and our approach to it.

Imho, "unimportant" problems lead to "important side effects" because of the relentless drive by the people working on the problems – this relentless drive can be born out of passion, or out of prescience.

What is needed in addition to that drive is a metric driving the improvement of your solution and your ability to make consistent progress towards that metric. Note that this metric may not be considered "important" by conventional standards, and may not even be individually defined. In fact, it may be borne organically out of collective, iterative, assessment.

21. Alexis YL ChanJun 11, 2012

Oh btw great article!

22. Simon HEfferJun 12, 2012

Of course, one of the most popular computer languages was written specifically to write business oriented programs and there are billions of lines of code using it today (in banks, ATMs, insurance companies) despite the fact it's 50 years old – COBOL.

23. Micky LatowickiJun 23, 2012

There's some merit to your overall argument. I'd like to warn, though, against the salience fallacy. The story of GPUs is interesting, but let's not over-generalize from it. And the fact that some important developments resulted from unimportant work does not establish that the likelihood of important results is as high in unimportant efforts as it is in important efforts.

Fleming stumbled upon penicillin during his work on staphylococci, which was serious work. The Wright brothers weren't striving to develop better entertainment – they wanted a flying machine, and they worked until they had one. The question we should ask is: in what ratio of "serious" projects are the results important? is this ratio higher than in "important" work? I think it is.

As you allude to, lots of unimportant work is better financed than important work, or is more rewarding, so it gets done. Since so much unimportant work gets done, I suspect that the conclusion we should come to is the opposite: that it is relatively rarely, in fact, for important technology to result from the frivolous work.

Well, neither of us have any numbers to back up our positions. So we can throw anecdotes at one another. You'd probably win in that particular game, because you know so much. Good for you!

Last, let's remember that the category of "unimportant" work includes lots of things that do not lead to any discovery at all, not even by accident. If someone makes a career choice between advertising and pharmaceutical research, there's not even a one-in-billion chance that he'd stumble onto something really important in the more-frivolous line of work.

24. Yossi KreininJun 23, 2012

You could look at a different ratio – instead of counting projects, count results. If one-two projects out of a mullion lead to enormously influential results, then it's more significant than 99 out of 100 leading to results that are important, but on a limited scale. Not that I know how I'd go about getting this alternative kind of measurements.

The thing is, my real point isn't that I could win the game that you invented (I doubt it), but that, just as you pointed out, it's hard to impossible to truly convincingly back up any claim about importance – in other words, people aren't very good at judging the significance and place of anything in this world.

For instance, I wouldn't want to do advertising because it irks me – a reasonable approach in my opinion; but I don't think of advertising as worthless because it doesn't lead to discovery. Of course it does lead to discovery. Google leads to discovery because researchers get information through Google. Google is financed almost exclusively through advertising.

So all I'm saying is, it makes no sense to go into advertising "because it leads to important discoveries – look, it finances Google and most of the Internet, etc." – and it doesn't make sense to not go into advertising "because it leads to no discovery". It makes sense to do what you're good at (as long as it's not extortion or something of the sort, I guess, though these people will also end up doing what they're good at...) and, beyond trying not to be too much of an asshole in concrete everyday situations, I don't think we have much to do to make sure our actions are ultimately leading to the betterment of mankind.

25. Yossi KreininJun 23, 2012

Also a question: do you do algorithms in ME to save people, or because you like math? :) And, would someone with a more burning desire to save people but not as much into math be better at it than you or worse?

26. Micky LatowickiJun 23, 2012

I'm not sure I like math all that much. I like the sort of challenges that I find at ME, and when I went job hunting five years ago, the kind of impact that the company has on mankind was definitely a consideration. All other things being equal, I would have definitely preferred a job in safety engineering to a job in gaming. I try to balance these considerations.

Regarding your second question – I'm not sure, but being goal-driven rather than process-driven is usually considered to be a good thing, so I suppose that someone with a burning desire to save lives would indeed do my job better than I do it. Alas, I am not that person.

By the way, I love your "Human?" field. Brilliant.

27. Yossi KreininJun 23, 2012

Actually, the spam filtering through obscurity approach was brought to my attention by Aristotle Pagaltzis (at a time when commenting here required subscription).

28. Jasper PaulsenMar 4, 2014

Actually, C and Unix were developed to solve an important problem. C was developed to build Unix. Unix was developed to be the operating system for AT&T's phone switches.

The automated phone switches freed the world from needing people (like M*A*S*H's Radar O'Reilly) to connect every phone call or data transmission.

29. Yossi KreininMar 4, 2014

I recall that Ken Thompson said in his Coders at Work interview that at the time he wrote Unix, OS development was "banned" in AT&T and he actually expected to be fired for his efforts.

30. Big MacSep 18, 2014

The argument has been made a few places that porn has driven some internet innovations:
http://www.pcworld.com/article/155745/porn_on_the_web.html
http://finweek.com/2013/12/04/opinion-how-the-porn-industry-has-driven-internet-innovation/
http://www.businessinsider.com/how-porn-drives-innovation-in-tech-2013-7



Post a comment