Suppose you have a sparse RAM API, something along the lines of:
- add_range(base, size)
- write_ram(base, bytes)
- read_ram(base, size)
People use this API for things like running a simulated CPU:
- define the accessible memory with add_range()
- pass the initial state to the simulator with write_ram()
- run the simulation, get the final state with read_ram()
Suppose this API becomes a runaway success, with a whopping 10 programmers using it (very little irony here, >95% of the APIs in this world are used exclusively by their designer). Then chances are that 9 of the 10 programmers are API users, and 1 of them is an API wrapper. Here's what they do.
API users
The first thing the first API user does is call you. "How do I use this sparse thing of yours?" You point him to the short tutorial with the sample code. He says "Uhmm. Errm…", which is userish for "Come on, I know you know that I'm lazy, and you know I know that docs lie. Come over here and type the code for me." And you insist that it's actually properly documented, but you will still come over, just because it's him, and you personally copy the sample code into a source file of his:
add_range(0x100000, 6) # input range add_range(0x200000, 6) # output range write_ram(0x100000, "abcdef") # run a program converting the input to uppercase print read_ram(0x200000, 6) # should print "ABCDEF"
It runs. You use the opportunity to point out how your documentation is better than what he's perhaps used to assume (though you totally understand his frustration with the state of documentation in this department, this company and this planet). Anyway, if he has any sort of problem or inconvenience with this thing, he can call you any time.
The next 8 API users copy your sample code themselves, some of them without you being aware that they use or even need this API. Congratulations! Your high personal quality standards and your user-centric approach have won you a near-monopoly position in the rapidly expanding local sparse RAM API market.
Then some time later you stumble upon the following code:
add_range(0x100000,256) add_range(0x200000,1024) add_range(0x300000,1024) ... add_range(0xb00000,128) ... add_range(0x2c00000,1024) ...
Waitaminnit.
You knew the API was a bit too low-level for the quite common case where you need to allocate a whole lot of objects, doesn't matter where. In that case, something like base=allocate_range(size) would be better than add_range(base,size) – that way users don't have to invent addresses they don't care about. But it wasn't immediately obvious how this should work (Nth call to allocate_range() appends a range to the last allocated address, but where should the first call to allocate_range() put things? What about mixing add_range() and allocate_range()? etc.)
So you figured you'd have add_range(), and then whoever needed to allocate lots of objects, doesn't matter where, could just write a 5-line allocate_range() function good enough for him, though not good enough for a public API.
But none of them did. Why? Isn't it trivial to write such a function? Isn't it ugly to hard-code arbitrary addresses? Doesn't it feel silly to invent arbitrary addresses? Isn't it actually hard to invent constant addresses when you put variable-sized data there, having to think about possible overlaps between ranges? Perhaps they don't understand what a sparse RAM is? Very unlikely, that, considering their education and experience.
Somehow, something makes it very easy for them to copy sample code, but very hard to stray from that sample code in any syntactically substantial way. To them, it isn't a sparse RAM you add ranges to. Rather, they think of it as a bunch of add_range() calls with hexadecimal parameters.
And add_range() with hex params they promptly will, just as it's done in the sample. And they'll complain about how this API is a bit awkward, with all these hex values and what-not.
API wrappers
If there's someone who can see right through syntax deep into semantics, it's the tenth user of your API, or more accurately, its first wrapper. The wrapper never actually uses an API directly in his "application code" as implied by the abbreviation, standing for "Application Programming Interface". Rather, he wraps it with another (massive) layer of code, and has his application code use that layer.
The wrapper first comes to talk to you, either being forced to use your API because everybody else already does, or because he doesn't like to touch something as low-level as "RAM" so if there's already some API above it he prefers to go through that.
In your conversation, or more accurately, his monologue, he points out some admittedly interesting, though hardly pressing issues:
- It's important to be able to trick a program using the sparse RAM API into allocating its data in specific address ranges, so that the resulting memory map is usable on certain hardware configurations and not just in simulations.
- In particular, it is important to be able to extract the memory map from the section headers of executables in the ELF and COFF format.
- Since add_range() calls are costly, and memory map formats such as the S-Record effectively specify a lot of small, adjacent ranges, there is a need for a layer joining many such ranges.
- An extensible API for the parsers of the various memory map formats is needed.
- …
When you manage to terminate the monologuish conversation, he walks off to implement his sparse RAM API on top of yours. He calls it SParser (layer lovers, having to invent many names, frequently deteriorate into amateur copywriters).
When he's done (which is never; let's say "when he has something out there"), nobody uses SParser but him, though he markets it heavily. Users won't rely on the author who cares about The Right Thing but not about their problems. Other wrappers never use his extra layers because they write their own extra layers.
However, even with one person using it, SParser is your biggest headache in the sparse RAM department.
For example, your original implementation used a list of ranges you (slowly) scanned through to find the range containing a given address. Now you want to replace this with a page table, so that, given an address, you simply index into a page array with its high bits and either find a page with the data or report a bad address error.
But this precludes "shadowing", where you have overlapping segments, one hiding the other's data. You thought of that as a bug in the user code your original implementation didn't detect. The wrapper thought it was a feature, and SParser uses it all over to have data used at some point and then "hidden" later in the program.
So you can't deploy your new implementation, speeding up the code of innocent users, without breaking the code of this wrapper.
What to do
Add an allocate_range() API ASAP, update the tutorial, walk over to your users to help replace their hex constants with allocate_range() calls. Deploy the implementation with the page table, and send the complaining wrapper to complain upwards along the chain of command.
Why
Your users will switch to allocate_range() and be happy, more so when they get a speed-up from the switch to page tables. The wrapper, constituting the unhappy 10% of the stakeholders, will have no choice but fix his code.
Ivan drank half a bottle of vodka and woke up with a headache. Boris drank a full bottle of vodka and woke up with a headache. Why drink less?
Users are many, they follow a predictable path (copy sample code) and are easily satisfied (just make it convenient for them to follow that path). Wrappers are few, they never fail to surprise (you wouldn't guess what and especially why their layers do), and always fail to be satisfied (they never use APIs and always wrap them). Why worry about the few?
The only reason this point is worth discussing at all is that users offend programmers while wrappers sweet-talk them, thus obscuring the obvious. It is natural to feel outrage when you give someone an add_range() function and a silly sample with hex in it, and not only do they mindlessly multiply hex numbers in their code, but they blame you for the inconvenience of "your API with all the hex in it". It is equally natural to be flattered when someone spends time to discuss your work with you, at a level of true understanding ("sparse RAM") rather than superficial syntactic pattern matching ("add_range(hex)").
He who sees through this optical illusion will focus on the satisfaction of the happy many who couldn't care less, securing the option to ignore the miserable few who think too much.
118 comments ↓
So, so true. I've had this scenario play out pretty much word for word. More than once!
Peer Reviews help
a) the API developer to find others need
b) the API users in the way that they know that there is a API at all
c) the smart API users in the way that they know understand the API
d) the dump API users in the way that they get a free example
e) the wrapper if he get encouraged to refactor/improve the original code rather than writing a wrapper
This is at least my experiance. I'm still a freshman and I learn usually a lot from code reviews (my code & foreign code)
It is always sad to see how many implementations exist for exactly the same issue – and the most attractive solution (which may not be the best) is copied over and over by others.
@The Fiddler: did this actually work in a similar way with OpenTK?
@anon: smart/dumb is not the distinction I made (I'm not sure you intended it to be a way to spell the distinction I made; it's the "free example" part that makes me think this is what you meant). Anyway, I don't think user/wrapper and dumb/smart correlate, and if there is a correlation, I'd guess it to be weak and I wouldn't guess which way they correlate. Even "naive"/"sophisticated" is the wrong distinction; I don't know a way to reduce it to cognitive traits, I only know the behavioral distinction – "go with the flow of the sample code"/"go against the grain wrapping things".
That aside, you sound like an optimist (spreading knowledge through reviews and other educational means, sad how Tower-of-Babelish and otherwise suboptimal things get); well, you've arrived at a very pessimistic site.
Still, it's better to go with the optimism in this case, since code reviews do help in more other ways. And if they also happen to pre-empt some wrapping and/or mindless copying attempts, well, all the better for you! Not to mention that pessimism can create its own causes by the way of the broken window effect. So while it's true that sometimes you end up in a shithole with no way out, in general it pays to be as optimistic as possible.
@mathrick: code reviews are a tricky subject so I won't argue about that at the moment – except that the wrapper type is well-versed in the software bullshit jargon and has stamina for endless arguing, hence a code review or any other social decision-making mechanism is unlikely to help, though a shotgun could.
As to the broken window effect – I agree, hence the tutorial for the API in the hypothetical example, but there's a difference: here it is "you" the API author who's keeping the windows non-broken, so being "pessimistic" here (not doing it because hey, the world sucks anyway) amounts to being lazy; however, the way "they" use APIs isn't something you control, so being "optimistic" here is, well, also being lazy (count on others to be thoughtful where it's easy to predict they won't and where you and I quite likely wouldn't be and where simple measures on your side could ameliorate the consequences of someone else's predictable and hardly avoidable behavior).
Oh dear, I recognised myself in your wrapper description. But some apis are just so tedious! Take stl's file handling.
Please, take it!
C++ doesn't have much file handling in its standard library, does it? There are the more elaborate boost APIs but I don't touch boost with a laser beam so I wouldn't know much about that; the standard interfaces include fstream, which is, well, usable, and stuff like ostream_iterator, which no wrapping can save.
"My name is k5.user, and I have written many API wrappers and forced others to use my wrapper."
Confession, good for the soul.
well, yes, I'm (still) an optimist.
A collegue mentioned today, that he have never found a way to describe the intetion from his framework. We both fully agreed, that minimal code examples cannot show that. So I thought I forward the question to you:
Do you think that people will use the API in a proper way when they understand the intention of the framework? (we do)
How to describe this intention?
I also should mention that my collegue had some terrible examples how users misused and abused his frameworks ending up in changes in within the framwork so that someone elses wrapper works.
So he is on the pessimistic side, too
@anon: I think intentions can be conveyed either through sample code or, more subtly, through a tendency to shape the user's code in certain ways, but hardly through words – generally, the consumer can almost never reach a level of thoughtfulness about the product anywhere near that of the producer, so it's very hard to correct bad user behavior through education (unless people are willing to invest lots in education – say, when the "consumer" is a musician learning to play a violin, and even then you must make your violin compatible with existing education rather than hoping to reshape the education to suit your newer violin).
That said, I'm somehow very curious about the details – what it is that you're doing and what it is that people don't get about it, so if you want to discuss this either here or at Yossi.Kreinin@gmail.com, you're most welcome.
@k5.user: may God forgive you.
"Do you think that people will use the API in a proper way when they understand the intention of the framework?"
Maybe, but consider: They'll see possibilities in it that you never dreamed of.
Following on the above example of the violin, it helps to consider how scandalous "The Four Seasons" was in Vivaldi's day. One critic mused, "If rapid, high tones are a sin, Signore Vivaldi will have much to account for."
@gus3: it usually isn't a Vivaldi who is the problem.
It seems like the lowest-energy solution would be for the API wrapping guy to send you a patch that adds allocate_range(). Then he would have to care about your users' problems in order to get you to accept the patch, you'd have more people working on your users' problems, and he'd have more people using his code. Wins all around, right?
Yes, but that's not what API wrappers do. API wrappers don't extend, they wrap, hiding the original API behind their own API operating with different concepts. People who are so cooperative and thoughtful that they gracefully extend are a distinct group that I didn't mention because of its apparently tiny size.
Almost nobody is successful at extending gracefully on their first attempt, especially in a company with code ownership; instead they blunder around violating all your style rules and encapsulation, while still adding new features that people actually need. Getting somebody up to speed enough that they can give you a patch that you'll want to accept without rewriting it can take a certain amount of time.
I'm a step behind you – I believe one will usually have hard time even finding someone with enough interest in the problem plus the right psychological profile.
I think your "wrapper" type is an instance of the general immature programmer type. These like *adding* complexity, instead of removing it, and then enjoy themselves being able to construct such universal abstractions. A young and capable mind contributes to this obsession. I was one myself. Still, this could be a problem even for experienced programmers, they have to keep themselves in check constantly too :)
Thanks for the C++ FQA, BTW :)
Well, there exist wrappers who are mature in pretty much every sense of the word, so whether you and I like what they've matured into or not, they surely ain't gonna get any maturer. As to love of complexity as the psychological root cause – can be part of it, though there can be other angles as well, for instance, repressed fear of the application domain (wrap it to hide the horror from the eyes), an unbendable belief in The Right (wrap it to right its wrongs), etc. As far as I'm concerned as an API maintainer, wrappers are a single cluster; to what extent they are psychologically homogeneous is an interesting question.
And, you're welcome :)
I consider something really special in this site.
I simply must tell you that you have an excellent and unique site that I really enjoyed reading.
You got yourself a new follower.
bing took me here. Cheers!
very Great post, i actually love this web site, carry on it
very interesting post, i actually enjoyed this web site, carry on it
Good, this is what I was looking for in google
I like this website its a master peace ! Glad I found this on google .
I love reading through and I believe this website got some genuinely utilitarian stuff on it! .
Great article to see, glad that duckduck brought me here, Keep Up cool Work
google got me here. Cheers!
I must say, as a lot as I enjoyed reading what you had to say, I couldnt help but lose interest after a while.
This i like. Cheers!
Respect to website author , some wonderful entropy.
Great, this is what I was looking for in yahoo
Great, bing took me stright here. thanks btw for info. Cheers!
Cheers, great stuff, Me enjoying.
Hey, yahoo lead me here, keep up nice work.
Enjoyed reading through this, very good stuff, thankyou .
I’m impressed, I have to admit. Genuinely rarely should i encounter a weblog that’s both educative and entertaining, and let me tell you, you may have hit the nail about the head. Your idea is outstanding; the problem is an element that insufficient persons are speaking intelligently about. I am delighted we came across this during my look for something with this.
Thanks for this web. I definitely agree with what you are saying.
I consider something really special in this site.
I like this website its a master peace ! Glad I found this on google .
I consider something really special in this site.
I like this website its a master peace ! Glad I found this on google .
I really enjoy examining on this blog , it has got fine goodies .
Some truly interesting content on this web site , appreciate it for contribution.
I simply must tell you that you have an excellent and unique post that I must say enjoyed reading.
This i like. Cheers!
Very interesting points you have remarked, appreciate it for putting up.
I was looking at some of your articles on this site and I believe this internet site is really instructive! Keep on posting .
Can I just say what a comfort to uncover somebody
that actually understands what they are talking about on the web.
You actually realize how to bring an issue to light and make it important.
A lot more people really need to look at this and understand this
side of your story. I can't believe you are not
more popular because you definitely have the gift.
I am not rattling great with English but I get hold this really easygoing to read .
Very interesting points you have remarked, appreciate it for putting up.
Thanks for finally writing about > API users & API wrappers < Liked it!
I must say, as a lot as I enjoyed reading what you had to say, I couldnt help but lose interest after a while.
Me like, will read more. Cheers!
I have read so many articles or reviews on the topic of the blogger
lovers except this post is genuinely a fastidious post, keep it up.
I really got into this website. I found it to be interesting and loaded with unique points of interest.
Hello there! This post could not be written much better!
Reading through this article reminds me of my previous roommate!
He constantly kept preaching about this. I am going to send this information to
him. Fairly certain he's going to have a very good read.
I appreciate you for sharing!
Howdy! I know this is kinda off topic but
I'd figured I'd ask. Would you be interested in trading links or maybe guest writing a blog article or vice-versa?
My website addresses a lot of the same topics as yours and I think
we could greatly benefit from each other. If you're interested
feel free to shoot me an e-mail. I look forward to hearing from you!
Awesome blog by the way!
What's up, its pleasant post about media print, we all understand media is a great source of data.
Post writing is also a fun, if you know after that you can write if not it
is complex to write.
Hey There. I found your blog using msn. This is an extremely well written article.
I'll be sure to bookmark it and come back to read more of your useful info.
Thanks for the post. I will certainly comeback.
If some one desires to be updated with newest technologies after that he must be pay a visit
this website and be up to date every day.
What a material of un-ambiguity and preserveness of valuable experience about unexpected feelings.
I read this post completely on the topic of the
comparison of newest and preceding technologies,
it's awesome article.
Thanks for the good writeup. It if truth be told was once
a entertainment account it. Glance complicated to more brought agreeable from you!
By the way, how can we communicate?
Hello There. I discovered your weblog the usage of msn. This is a really smartly written article.
I will make sure to bookmark it and return to learn more of your useful info.
Thank you for the post. I will certainly return.
Good post. I will be facing some of these issues as well..
It is in reality a great and helpful piece of information. I'm happy that you shared
this helpful information with us. Please stay us up to date like this.
Thank you for sharing.
stays on topic and states valid points. Thank you.
Some truly wonderful posts on this web site , appreciate it for contribution.
Enjoyed reading through this, very good stuff, thankyou .
I kinda got into this web. I found it to be interesting and loaded with unique points of view.
Some truly good posts on this web site , appreciate it for contribution.
Found this on bing and I’m happy I did. Well written article.
Appreciate it for this howling post, I am glad I observed this internet site on yahoo.
I dugg some of you post as I thought they were very beneficial invaluable
I like this site because so much useful stuff on here : D.
This does interest me
I consider something really special in this site.
Cheers, i really think i will be back to your site
I dugg some of you post as I thought they were very beneficial invaluable
Hi, Neat post. There is an issue together with your website in web explorer, would
test this? IE nonetheless is the marketplace chief
and a good section of other people will miss your magnificent writing because
of this problem.
I like this site, because so much useful stuff on here : D.
Me enjoying, will read more. Thanks!
I like this site because so much useful stuff on here : D.
Just wanna input on few general things, The website layout is perfect, the articles is very superb : D.
I consider something really special in this site.
I am not rattling great with English but I get hold this really easygoing to read .
Good Morning, glad that i saw on this in bing. Thanks!
This is nice!
Parasite backlink SEO works well :)
Found this on yahoo and I’m happy I did. Well written web.
Great, google took me stright here. thanks btw for post. Cheers!
Some truly cool goodies on this web site , appreciate it for contribution.
Thanks for this article. I definitely agree with what you are saying.
I consider something really special in this site.
Hi, glad that i saw on this in bing. Thanks!
Just wanna input on few general things, The website layout is perfect, the articles is very superb : D.
very cool post, i actually like this web site, carry on it
Does your site have a contact page? I'm having problems locating it but, I'd like to send you an email.
I've got some ideas for your blog you might be interested in hearing.
Either way, great website and I look forward to seeing it improve over time.
Cheers, great stuff, Me enjoying.
great advice you give
Hello! This is my first visit to your blog! We are
a group of volunteers and starting a new project in a community in the same niche.
Your blog provided us beneficial information to work on. You have done a wonderful
job!
I know this if off topic but I'm looking into starting
my own weblog and was curious what all is required to get set up?
I'm assuming having a blog like yours would cost a pretty penny?
I'm not very web savvy so I'm not 100% sure. Any suggestions or advice would be greatly appreciated.
Many thanks
Skyking, this note is your next bit of information. Feel free to contact the agency at your convenience. No further information until next transmission. This is broadcast #4533. Do not delete.
Hi there, after reading this remarkable article i am as well glad to share my
familiarity here with mates.
I conceive you have mentioned some very interesting details , appreciate it for the post.
I could not resist commenting. Well written!
I conceive this web site holds some real superb information for everyone : D.
Write more, thats all I have to say. Literally, it
seems as though you relied on the video to make your point.
You obviously know what youre talking about, why throw away your intelligence on just posting videos
to your blog when you could be giving us something informative to
read?
I absolutely love your site.. Excellent colors
& theme. Did you make this web site yourself? Please reply back as I'm
trying to create my own personal website and would
like to find out where you got this from or what the
theme is named. Many thanks! natalielise pof
You got yourself a new rader.
Very interesting points you have remarked, appreciate it for putting up.
Hmm is anyone else encountering problems with the images on this blog loading?
I'm trying to determine if its a problem on my end
or if it's the blog. Any feed-back would be greatly appreciated.
Thanks for this article. I definitely agree with what you are saying.
Hello, i read your blog from time to time and i own a similar
one and i was just wondering if you get a lot of spam comments?
If so how do you reduce it, any plugin or anything you
can advise? I get so much lately it's driving me insane so any help is very much appreciated.
natalielise plenty of fish