Low-level is easy

My previous entry has the amazingly interesting title "Blogging is hard". Gee, what a wuss, says the occasional passer-by. Gotta fix my image, fast. Think, think, think! OK, here's what I'm going to tell you: low-level programming is easy. Compared to higher-level programming, that is. I'm serious.

For starters, here's a story about an amazing developer, who shall rename nameless for the moment. See, I know this amazing developer. Works for Google now. Has about 5 years of client-side web programming under his belt. And the word "nevermore" tattooed all over his body (in the metaphorical sense; I think; I haven't really checked). One time, I decided that I have to understand this nevermore business. "Amazing Developer", I said, "why have you quit the exciting world of web front-ends?" "I don't like it", says the Amazing Developer. "But, but! What about The Second Dot Com Bubble? VC funds will beg you to take their $1M to develop the next Arsebook or what-not. Don't you wanna be rich?" "I really don't like web front-ends very much", he kept insisting. Isn't that strange? How do you explain it? I just kept asking.

Now that I think of it, he probably was a little bit irritated at this point. "Look, pal", he said (in the metaphorical sense; he didn't actually say it like that; he's very polite). "I have a license to drive a 5-ton truck. But I don't want a career in truck driving. Hope this clarifies things". He also said something about your average truck driver being more something than your typical web developer, but I don't remember the exact insult.

Now, I've been working with bare metal hardware for the last 4 years. No OS, no drivers, nothing. In harsh environments in terms of hardware, by comparison. For example, in multi-core desktops, when you modify a cached variable, the cache of the other processor sees the update. In our chips? Forget it. Automatic hardware-level maintenance of memory coherency is pretty fresh news in these markets. And it sucks when you change a variable and then the other processor decides to write back its outdated cache line, overwriting your update. It totally sucks.

A related data point: I'm a whiner, by nature. Whine whine whine. You've probably found this blog through the C++ FQA, so you already know all about my whining. And it's not like I haven't been burnt by low-level bugs. Oh, I had that. Right before deadlines. Weekends of fun. So how come I don't run away from this stuff screaming and shouting? Heck, I don't mind dealing with bare metal machines for the rest of my career. Well, trying out other stuff instead can be somewhat more interesting, but bare metal beats truck driving, I can tell you that. To be fair, I can't really be sure about that last point – I can't drive. At all. Maybe that explains everything?

I'll tell you how I really explain all of this. No, not right in this next sentence; there's a truckload of reasons (about 5 tons), so it might take some paragraphs. Fasten your seatbelts.

What does "high level" basically mean? The higher your level, the more levels you have below you. This isn't supposed to matter in the slightest: at your level, you are given a set of ways to manipulate the existing lower-level environment, and you build your stuff on top of that. Who cares about the number of levels below? The important thing is, how easily can I build my new stuff? If I mess with volatile pointers and hardware registers and overwrite half the universe upon the slightest error, it sucks. If I can pop up a window using a single function, that's the way I like it. Right? Well, it is roughly so, but there are problems.

Problem 1: the stuff below you is huge at higher levels. In my humble opinion, HTML, CSS, JavaScript, XML and DOM are a lot of things. Lots of spec pages. A CPU is somewhat smaller. You have the assembly commands needed to run C (add/multiply, load/store, branch). You have the assembly commands needed to run some sort of OS (move to/from system co-processor register; I dunno, tens of flavors per mid-range RISC core these days?). And you have the interrupt handling rules (put the reset handler here, put the data abort handler here, the address which caused the exception can be obtained thusly). That's all.

I keep what feels like most of ARM946E-S in my brain; stuff that's still outside of my brain probably never got in my way. It's not a particularly impressive result; for example, the fellow sitting next to me can beat me ("physically and intellectually", as the quote by Muhammad Ali goes; in his case, the latter was a serious exaggeration – he was found too dumb for the US army, but I digress). Anyway, this guy next to me has a MIPS 34Kf under his skull, and he looks like he's having fun. That's somewhat more complicated than ARM9. And I worked on quite some MFC-based GUIs (ewww) back in the Rich Client days; at no point I felt like keeping most of MFC or COM or Win32 in my head. I doubt it would fit.

Problem 2: the stuff below you is broken. I've seen hardware bugs; 1 per 100 low-level software bugs and per 10000 high-level software bugs. I think. I feel. I didn't count. But you probably trust me on this one, anyway. How many problems did you have with hardware compared to OS compared to end-user apps? According to most evidence I got, JavaScript does whatever the hell it wants at each browser. Hardware is not like that. CPUs from the same breed will run the user-level instructions identically or get off the market. Memory-mapped devices following a hardware protocol for talking to the bus will actually follow it, damn it, or get off the market.

Low-level things are likely to work correctly since there's tremendous pressure for them to do so. Because otherwise, all the higher-level stuff will collapse, and everybody will go "AAAAAAAAAA!!" Higher-level things carry less weight. OK, so five web apps are broken by this browser update (or an update to a system library used by the browser or any other part of the pyramid). If your web app broke, your best bet is to fix it, not to wait until the problem is resolved at the level below. The higher your level, the loner you become. Not only do you depend on more stuff that can break, there are less people who care in each particular case.

Problem 3: you can rarely fix the broken things below your level. Frequently you don't have the source. Oh, the browser is open source? Happy, happy, joy, joy! You can actually dive into the huge code base (working below your normal level of abstraction where you at least know the vocabulary), fix the bug and… And hope everyone upgrades soon enough. Is this always a smart bet? You can have open source hardware, you know. Hardware is written in symbolic languages, with structured flow and everything. The only trouble is, people at home can't recompile their CPUs. Life cycle. It's all about life cycle. Your higher-level thingie wants to be out there in the wild now, and all those other bits and pieces live according to their own schedules. You end up working around the bug at your end. Sometimes preventing the lower-level component author from fixing the bug, since that would break yours and everybody else's workaround. Whoopsie.

How complicated is your workaround going to be? The complexity raises together with your level of abstraction, too. That's because higher-level components process more complicated inputs. Good workarounds are essentially ways to avoid inputs which break the buggy implementation. Bad workarounds are ways to feed inputs which shouldn't result in the right behavior, but do lead to it with the buggy implementation. Good workarounds are better than bad workarounds because bad workarounds break when the implementation is fixed. But either way, you have to constrain or transform the inputs. Primitive sets of inputs are easier to constrain or transform than complicated sets of inputs. Therefore, low-level bugs are easier to work around. QED.

Low-level: "Don't write that register twice in a row; issue a read between the writes". *Grump* stupid hardware. OK, done. Next.

High-level: "Don't do something, um, shit, I don't know what exactly, well, something to interactive OLE rectangle trackers; they will repaint funny". I once worked on an app for editing forms, much like the Visual Studio 6 resource editor. In my final version, the RectTracker would repaint funny, exactly the way it would in Visual Studio 6 in similar cases. I think I understood the exact circumstances back then, but haven't figured out a reasonable way to avoid them. Apparently the people working at that abstraction level at Microsoft couldn't figure it out, either. What's that? Microsoft software is always crap? You're a moron who thinks everything is absolutely trivial to get right because you've never done anything worthwhile in your entire life. Next.

Problem 4: at higher levels, you can't even understand what's going on. With bare metal machines, you just stop the processor, physically (nothing runs), and then read every bit of memory you want. All the data is managed by a single program, so you can display every variable and see the source code of every function. The ultimate example of the fun that is higher-level debugging is a big, slow, hairy shell script. "rm: No match." Who the hell said that, and how am I supposed to find out? It could be ten sub-shells below. Does it even matter? So what if I couldn't remove some files? Wait, but why were they missing – someone thought they should be there? Probably based on the assumption that a program should have generated them previously, so that program is broken. Which program? AAARGH!!

OK, so shell scripts aren't the best example of high-level languages. Or maybe you think they are; have fun. I don't care. I had my share of language wars. This isn't about languages. I want to move on to the next example. No shell scripts. You have JavaScript (language 1) running inside HTML/CSS (languages 2 & 3) under Firefox (written in language 4) under Windows (no source code), talking to a server written in PHP (language 5, one good one) under Linux (yes source code, but no way to do symbolic debugging of the kernel nonetheless). I think it somewhat complicates the debugging process; surely no single debugger will ever be able to make sense of that.

Problem 5: as you climb higher, the amount of options grows exponentially. A tree has one root, a few thick branches above it, and thousands of leaves at the top. Bend the root and the tree falls. But leaves, those can grow in whichever direction they like.

Linkers are so low-level that they're practically portable, and they're all alike. What can you come up with when you swim that low? Your output is a memory map. A bunch of segments. Base, size, bits, base, size, bits. Kinda limits your creativity. GUI toolkits? The next one is of course easier to master than the first one, but they are really different. What format do you use to specify layouts, which part is data-driven and which is spelled as code? How do you handle the case where the GUI is running on a remote machine? Which UI components are built-in? Do you have a table control with cell joining and stuff or just a list control? Does your edit box check spelling? How? I want to use my own dictionary! Which parts of the behavior of existing controls can be overridden and how? Do you use native widgets on each host, surprising users who switch platforms, or roll your own widgets, surprising the users who don't?

HTML and Qt are both essentially UI platforms. Counts as "different enough" for me. Inevitably, both suck in different ways which you find out after choosing the wrong one (well, it may be obvious with those two from the very beginning; Qt and gtk are probably a better example). Porting across them? Ha.

The fundamental issue is, for many lower-level problems there's The Right Answer (IEEE floating point). Occasionally The Wrong Answer gains some market share and you have to live with that (big endian; lost some market share recently). With higher-level things, it's virtually impossible to define which answer is right. This interacts badly with the ease of hacking up your own incompatible higher-level nightmare. Which brings us to…

Problem 6: everybody thinks high-level is easy, on the grounds that it's visibly faster. You sure can develop more high-level functionality in a given time slot compared to the lower-level kind. So what? You can drive faster than you can walk. But driving isn't easier; everybody can walk, but to drive, you need a license. Perhaps that was the thing mentioned by the Awesome (Ex-Web) Developer: at least truck drivers have licenses. But I'm not sure that's what he said. I'll tell you what I do know for sure: every second WordPress theme I tried was broken out of the box, in one of three ways: (1) PHP error, (2) SQL error and (3) a link pointing to a missing page. WordPress themes are written in CSS and PHP. Every moron can pick up CSS and PHP; apparently, every moron did pick them up. Couldn't they keep the secret at least from some of them? Whaaaam! The speedy 5-ton truck goes right into the tree. Pretty high-level leaves fall off, covering the driver's bleeding corpse under the tender rays of sunset. And don't get me started about the WordPress entry editing window.

Now, while every moron tries his luck with higher-level coding, it's not like everyone doing high-level coding is… you get the idea. The other claim is not true. In fact, this entry is all about how the other claim isn't true. There are lots of brilliant people working on high-level stuff. The problem is, they are not alone. The higher your abstraction level, the lower the quality of the average code snippet you bump into. Because it's easy to hack up by the copy-and-paste method, it sorta works, and if it doesn't, it seems to do, on average, and if it broke your stuff, it quite likely your problem, remember?

Problem 7: it's not just the developers who think it's oh-so-easy. Each and every end user thinks he knows exactly what features you need. Each and every manager thinks so, too. Sometimes they disagree, and no, the manager doesn't always think that "the customer is always right". But that's another matter. The point here is that when you do something "easy", too many people will tell you how it sucks, and you have to just live with that (of course having 100 million users can comfort you, but that is still another matter, and there are times when you can't count on that).

I maintain low-level code, and people are sort of happy with it. Sometimes I think it has sucky bits, which get in your way. In these cases, I actually have to convince people that these things should be changed, because everybody is afraid to break something at that level. Hell, even bug fixes are treated like something nice you've done, as if you weren't supposed to fix your goddamn bugs. Low-level is intimidating. BITS! REGISTERS! HEXADECIMAL! HELP!!

Some people abuse their craft and actively intimidate others. I'm not saying you should do that; in fact, this entry is all about how you shouldn't do that. The people who do it are bastards. I've known such a developer; I call him The Bastard. I might publish the adventures of The Bastard some day, but I need to carefully consider this. I'm pretty sure that the Awesome Developer won't mind if someone recognizes him in a publicly available page, but I'm not so sure about The Bastard for some reason or other.

What I'm saying is, maintaining high-level code is extremely hard. Making a change is easy; making it right without breaking anything isn't. You can drive into a tree in no time. High-level code has a zillion of requirements, and as time goes by, the chance that many of them are implicit and undocumented and nobody even remembers them grows. People don't get it. It's a big social problem. As a low-level programmer, you have to convince people not to be afraid when you give them something. As a high-level programmer, you have to convince them that you can't just give them more and more and MORE. Guess which is easier. It's like paying, which is always less hassle than getting paid. Even if you deal with a Large, Respectful Organization. Swallowing is easier than spitting, even for respectful organizations. Oops, there's an unintended connotation in there. Fuck that. I'm not editing this out. I want to be through with this. Let's push forward.

The most hilarious myth is that "software is easy to fix"; of course it refers to application software, not "system" software. Ever got an e-mail with a ">From" at the beginning of a line? I keep getting those once in a while. Originally, the line said "From" and then it got quoted by sendmail or a descendant. The bug has been around for decades. The original hardware running sendmail is dead. And that hardware had no bugs. The current hardware running sendmail has no bugs, either. Those bugs were fixed somewhere during the testing phase. Application software is never tested like hardware. I know, because I've spent about 9 months, the better part of 2007, writing hardware tests. Almost no features; testing, exclusively. And I was just one of the few people doing testing. You see, you can't fix a hardware bug; it will cost you $1M, at least. The result is that you test the hardware model before manufacturing, and you do fix the bug. But with software, you can always change it later, so you don't do testing. In hardware terms, the testing most good software undergoes would be called "no testing". And then there's already a large installed base, plus quick-and-dirty mailbox-parsing scripts people wrote, and all those mailboxes lying around, and no way to make any sense of them without maintaining bugward compatibility (the term belongs to a colleague of mine, who – guess what – maintains a high-level code base). So you never fix the bug. And most higher-level code is portable; its bugs can live forever.

And the deadlines. The amount of versions of software you can release. 1.0, 1.1, 1.1.7, 1.1.7.3… The higher your abstraction level, the more changes they want, the more intermediate versions and branches you'll wind up with. And then you have to support all of them. Maybe they have to be able to read each other's data files. Maybe they need to load each other's modules. And they are high-level. Lots of stuff below each of them, lots of functionality in them. Lots of modules and data files. Lots of developers, some of whom mindlessly added features which grew important and must be supported. Damn…

I bet that you're convinced that "lower-level" correlates with "easier" by now. Unless you got tired and moved elsewhere, in which case I'm not even talking to you. QED.

Stay tuned for The Adventures of The Bastard.

138 comments ↓

#1 c0re on 02.16.08 at 11:09 pm

I agree with most of what you said here but I don't see how you (By you I mean low level devs) are immune from the High Level – Low Level Chain of responsibility.

For e.g. If you're a device driver writer for a GPU and you can still mess up some pipeline code and have other people dependent on your buggy code which you have to forever maintain because the next big game pimps your GPU and the big bosses couldn't care less.

Admittedly this level is still higher than machine level, but I assume low level > machine level.

Re: C++ FQA – I agree 99.9%

I'm constantly amazed at how C++ "expert" devs are just scholars of trivia instead of actually being able to write up decent code. Or do they have a new title for that now? System Architect?

bah!

#2 Yossi Kreinin on 02.17.08 at 10:27 am

Well, yeah, you can't get out of the food chain, just choose your position in it… If you're doing low-level, and your stuff got popular, every move of yours can break things, so you have to watch your step. If you're doing high-level, then until your stuff gets *really* popular, you have to work around all those lower-level bugs/quirks; then, when you become awfully important, the low-level crowd will actually bend their development towards your needs.

Note that the low-level people have trouble /after/ the big success and the high-level people have it /before/ that. If your thing is so popular that loads of stuff depends on it and you're afraid to break something, you're already in a very good position.

#3 I can’t believe I’m praising Tcl on 04.21.08 at 1:54 pm

[...] On second thought, I don't know if I'd really recommend it. Remember how I told low-level programming was easy? It is, fundamentally, but there's this other angle from which it's quite a filthy [...]

#4 DoctorNick on 04.23.08 at 5:25 am

I wrote a blog post about this last November entitled "I'm afraid of low level programming." http://www.litanyagainstfear.com/blog/2007/11/27/im-afraid-of-low-level-programming/

I've done a bit of reflecting on what I wrote and what you wrote, and I'm starting to think that newcomers to software engineering can't handle the low level well. For example, to use a calculator to do simple math usually requires a basic knowledge of how those operators work. Even for trignometry it helps to know and understand the sin, cos, and tan functions, but there's no way you'd have to dive into deep calculus topics like Taylor series and the like, which calculators use in order to compute those functions.

I personally have had a bit of exposure to C and assembly through my classes, and it scares the crap out of me. To know that your program will fail due to the slightest buffer overrun or because you misplaced one bit is just frightening, and to newcomers downright frustrating.

Low-level may be easy to you since you've been in the business (i'm assuming) for quite a few years, but for apprentices like me, I'll stay high level for now. If you have any suggestions as to what I could do to understand more low-level stuff I'd appreciate it.

#5 Yossi Kreinin on 04.23.08 at 11:43 am

There are two kinds of low-level: the user space kind and the kernel kind.

For the user space kind, the most cost effective way to become comfortable with it I can think of is to write a C compiler (for a subset, not the whole language). If it's relevant for you, you can usually get academic credit for it by taking a compiler construction course (check the syllabus; it could be called "compiler construction" and not have a bunch of assignments about a C compiler in it). I'd go for the whole toolchain – a lexer, a parser, an assembly code generator, an assembler and a linker, but using an existing assembler is probably cool, too.

The kernel kind of low-level you don't care about unless you're in the kernel or on bare metal. There's a single insight there – most hardware is memory-mapped. That is, you can tell it what to do by reading/writing C pointers. From there, it's a bunch of specific rules for each piece of hardware, and a sprinkle of crap for interrupt handling and kernel mode CPU instructions. I wouldn't worry about it anyway since this knowledge doesn't give you much value if you program on top of a PC or a mobile OS. The user space stuff is what can really help (to optimize and to debug and to hack on code bases written in C++ when they should have really been written in Java or C#).

As to fear – the whole thing isn't anywhere near, say, advanced math in terms of complexity. It has scary failure modes (buffer overflows and stuff), but you don't need that much brain power to handle it. Which makes it a cheap superpower (lots of people are afraid of it). So bringing yourself to the point where you can skim through compiler-generated assembly and follow it, is a fairly cost-effective investment. And if you write your own toy compiler, you'll most likely get there.

As to years of experience: their value is mostly in building/destroying character. But you don't learn nearly as much as you do in school, so it's not a big deal.

#6 AvailableName on 04.25.08 at 10:55 am

This is a great post and is reinforcing my desire to learn assembly.

Reading this led me to a strange thought – We have to hope that there isn't a "bug" in reality (lower than the hardware), because we can't get the source for that… yet.

#7 Yossi Kreinin on 04.25.08 at 11:36 am

To me, assembly is something which is good to be able to read (so you can see what a compiler does), and to generate (so you can write a compiler; although the right thing these days is usually to generate C, not assembly, so it's mostly about understanding what an existing compiler does).

Writing assembly is something you do for "system" code (boot loaders, kernels, that kind of thing). For optimization, C with intrinsics should do the trick. Assembly is gnarly to write.

#8 jemptymethod on 06.03.08 at 5:22 pm

The wonderful thing about high-level is, it's still never high enough. Consider the following, the crux of what quite possibly is the world's smallest MVC engine for web apps:

static function sendResponse
(IBareBonesController $controller) {
$controller->setMto(
$controller->applyInputToModel());
$controller->mto->applyModelToView();
}

The rest of the framework totals fewer than 60 lines, and implements everything except "applyInputToModel()". The jury is still out as to whether web developers will love me or hate me for introducing "yet another level".

#9 Yossi Kreinin on 06.04.08 at 1:41 pm

<60LOC? How does it applyModelToView?

#10 Eli on 10.11.08 at 1:52 am

This rings a bell, thanks for bringing this topic up.

As someone who codes both high-level and low-level, I agree about the relative difficulty of high-level programming.

For me there's another aspect. At low level, I can understand "the whole picture", which is important for becoming proficient. As you say, you can hold the entire ARM CPU in your head. Which can't be done for a jQuery / JS / XML & DOM & JSON / AJAX / PHP / CSS & HTML stack. The mere fact that you have something in your head causes:
1) You to enjoy the work more, because there are no dark dusty corners you're afraid to look into
2) Be more productive, because you spend less time worrying about stuff you don't know and reading tutorials on another cool JS library / framework. Nothing new in the ARM architecture. So just do the work.

#11 Yossi Kreinin on 10.11.08 at 2:53 am

I think there are people who actually do hold a large portion of the web stack in their brains, which is quite a feat. I'd say that it takes more space in the brain compared to lower-level kind of crud though, and I'd guess it's still harder to work with even if you know it well, because you have to "consult your knowledge" more often ("what if this?.. what if that?..") – be a "lawyer" more of the time and thus a "hacker" less of the time.

#12 Insulted... on 10.21.08 at 4:12 pm

Any idiot can bend steel, but it takes a genius to come up with the eifel tower.

#13 Insulted... on 10.21.08 at 4:26 pm

What I meant to say is that high-level engineers don't care how the plumbing gets into it, but structures like the Guggenheim or the Westminster Palace are not thought up by plumbers or carpenters. The people who glue and hammer are important, but it is rare when genius is found there.

I admit, I am frustrated by the people who only know the high-level stuff. But, I know both, and I would prefer to design on a grander scale rather than be pigeon-holed into CPU instruction work.

#14 Yossi Kreinin on 10.23.08 at 12:44 pm

I wonder what could make you "Insulted" in what I wrote; I basically said low-level was easier to deal with, which doesn't seem to contradict your claim that low-level is somehow a lesser domain. Of course it doesn't support your claim, either; regardless of what I wrote up there, I do think you're wrong, and in particular, the "right" analogy would be comparing the "macro-architecture" of the tower to the "micro-architecture" of the joints making it up or, still lower, the engineering behind the process of melting the steel, or something. It's basically a design-to-design comparison, not a design-to-"labor" one. So while it's perfectly legitimate to prefer a "larger scale", it's basically a matter of taste and not a matter of objective complexity measurement.

#15 TNO on 10.25.08 at 2:54 am

To Insulted: the main problem of the software design/architecture is that you don't have to be a genius to convince you boss to build a Guggenheim. With all the sad consequences arising therefrom…

Or, in other words, in the software world any idiot can build Eiffel tower, but it takes a genius to bend a silicon.

#16 E on 06.29.09 at 11:47 am

There's a low level error on this whole page.

#17 B on 10.23.09 at 6:03 pm

i've been programming in java and c# for a while now, but i want to get a little closer to the hardware. i read a simple internet tutorial to get the basic syntax for c++, and i understand the basics of pointers and memory allocation. however, what is the process by which one goes about to locate a manipulate specific hardware implementations?

#18 Yossi Kreinin on 10.24.09 at 2:34 am

@B: I'm not quite sure what you mean, but you could say that "low-level" breaks down into (1) raw pointers and memory management, (2) programming in assembly/intrinsics for speed, (3) OS-level assembly, memory-mapped hardware and interrupt handling. If you're on the desktop, (1) and (2) happen at user-level programs for optimization and (3) happens at kernels and device drivers, so this is where you go if you specifically want to get to lower levels (although I'd usually be dragged into a new programming environment because of having to do new sort of work and not vice versa).

#19 noah on 06.06.10 at 7:14 pm

I agree with this guy, I like low level stuff, I want to be able to write in assembly, but first I feel like I should learn all about how to write gui libraries in c++, but no one seems to want to help, and the libraries are impossible to read

#20 Elena on 08.07.10 at 2:17 am

Now I'm curious about how hardware tests are performed. Could software be tested in a similar way, too? Please send answers to egarrulo at gmail.com too, since this blog doesn't provide an option to enable notifications about updates.

#21 Yossi Kreinin on 08.07.10 at 3:05 am

First of all, there are tools for hardware verification – formal verification by static code analysis, and automatic random test generation attempting to cover as many execution paths as possible. I don't have experience with these, so despite having reasonable success in hardware testing, I'm in one potentially important way not qualified to discuss it (actually I'm in a frame of mind where I prefer to actively avoid such tools, but I don't have experience to back it up). The way I've seen hardware tested is you implement a software emulator, usually implemented quite differently than the hardware model and more simply, without the parallelism inherent to hardware and with little optimization; then you generate random tests, as well as tests for some extreme cases, run on both and compare the results (usually raw register or memory snapshots at the end of the run), the tricky part being to identify the extreme cases and to randomize from an interesting distribution. This is different from software testing in that you have 2 full-blown implementations, and you run plenty of tests on plenty of machines for plenty of days; more on those differences below.

There are two ways in which hardware is easier to test than software, IMO fundamental enough to preclude the testing of software along the lines of hardware testing.

Firstly, hardware tends to be designed to minimize dependence on accumulated state. For instance, the execution of a CPU instruction will depend, in a complicated pipelined CPU, on what instructions executed previously, but only on a relatively small window; a Markov chain, if you like. It is not the only way to design hardware and I've seen real hardware designs that were very hard to test because of executing long processes with data-dependent decision making all in hardware, and in fact those tended to be manufactured with bugs. It is also not impossible to design software such that it depends on a limited amount of state; class invariants are a simple example of being able to free oneself of thinking through many different cases of what sequences of events are possible and would the program work correctly in all these cases. So basically it's a question of "designing for testability" (though this term is reserved for other sorts of testing in the HW design jargon, namely, testing the production rather than the design itself); arguably, the problems solved in hardware tend to be better suited for testing because of there being less context affecting the behavior of a component. A related question is where the complexity is that bugs come from. In hardware, the complexity comes from implementation – vast parallelism and tricky speed or space optimizations, whereas the spec is usually relatively simple, so that a reference implementation and a set of tests giving good spec coverage are also relatively easy to produce. In software, the complexity quite often comes from definition, where it would be silly to write two implementations and if one did, it would be hard to tell which one behaved correctly, because there are actual holes in the spec. And in fact software that is not unlike hardware in the sense of being easy to specify but hard to implement – say, a classifier taking a region of interest in an image and saying if it's a snapshot of a vehicle or not – can be and frequently is tested similarly to hardware: a gold model (manually marked database in the case of machine learning) and tests performed for hours or days.

Secondly, hardware is easier to test because it is expensive not to test it. That is, people who specify hardware are more careful not to overload the spec with features with unclear interactions, and people who control its schedule would never think of optimizing away the testing. That's because of the palpable high price of a hardware error compared to a very uncertain price of software errors (in reality you can work around many hardware bugs but hardware bugs sound like something extremely scary). This social aspect of the problem seems to me not less critical, if not more critical, than the technical aspect above.

In short, I think software is destined to be orders of magnitude more buggy than hardware.

P.S. Is there actually a way to configure WP so that it notifies you when someone replies to your comment (as opposed to anything on the site)? WP doesn't seem to even know who talks to whom, so I figured it couldn't properly notify, either.

#22 bob l'eponge on 09.07.10 at 12:22 am

If i understand correctly, lower level is easyer because :
- you have a clear idea of what you are trying to achieve.
- you depend less, on the clear ideas of others
- you can actually implement your clear idea pretty fast into some high level languages ..
- and then, you sometimes have a clearer idea of what could be actually going on (whenever your stuff is running).

I grew up with the illusion that Assembly langage was low level. At first i knew it was not, then i just forgot about it. Because in most case you "can" learn the assembly langage, and even understand it (although it sometimes becomes difficult like with modern Intel cpus, where i have no idea of what's going on any more). And 20 years later, i suddenly realize (again) that i never did any low level thinking (i don't even knwo what a Verilog statement is).

I have seen some high level assembly langages, and i really wonder what C has to offer in terms of speed of developpement that is better than those. C is nothing more to me than a macroassembler (and a lot of people do look at the disassembled version of their C software).

What "high level" really lack, is some standard way to do things. It really should be all about standardazing. As i see it everything (high level) should be written out of a standard assembly langage. So you would have the tool to do fast whatever simple thing you need to get done. But would also have means to "understand" what you are doing. Where understanding would be "to know what standard assembly instructions your program maps to".

But now that i reallize that assembly is such a high level beast, i wonder if one could not work up something better than that as the basic low level representation any high level langage can map their result to. What's the use in cutting off the understanding of high level programmers from what it physically means ?

#23 Yossi Kreinin on 09.11.10 at 7:51 am

The trouble is that there's an infinite amount of way to build a physical (virtual) machine in the first place. There's a Tower of Babel problem with languages that can't be solved, namely, that there's always an infinite amount of ways to spell things and no way to choose.

#24 Michael S on 06.20.11 at 7:06 am

I couldn't agree with this post any more. I personally develop with C++ (consistently for about 4 years now) but just don't see how I could have understood C++ without writing my toy compiler. Things such as how come there exists a this pointer? I just could never thought of a thorough explanation without attempting to implement a compiler. I just don't think a developer can truly call themselves a developer (especially for C/C++) without knowing how a compiler would work. Another example is function pointers. Something like const char *(*f(int index))(const double *const val, struct tree *); Just how can someone understand the meaning of (to an intuitive level) all this without trying to implement a compiler and thus be able to parse a declaration into an abstract syntax tree? May be its just me but I don't see how its possible.

#25 blOOp on 08.12.11 at 10:52 pm

Oh sigh, how I've longed to have a complete understanding of hardware/bios interrupts, for kernel space programming. But there is no real, 'go do this br0, it's l3g1t I swear!!!!!!!!11111', like we higher level programmers have been led to learn by.

For instance; I can barely learn on my own without a video tutorial holding my hand the whole way. Personally I find my dependency terribly disgusting, but it's just the world now day's… ANYWHO, let's stop the banter and get to the questions:

1: The main thing that I don't get, is when getting the BIO's intertupt for let's say, keyboard input, you would write something like:

mov ebx, int8030

(granted that's the first thing that popped into my head, I know it's wrong)

I don't understand _what_ the int8030 is. I can understand that you're moving the int8030 instruction into the 32 bit base register, but is that it?

2: On my browsing of several Operating System Development forums (OSDev.org mainly (Yes, I do want to be an OSDev)), they seem to throw hissy fit's whenever one of their members uses said BIOS intrupt. Is there a reason to use something else? Or are they just PM's-ing?

3: I'm trying to limit my time with ASM as much as possible, as it tends to bend my mind in ways that I'm sure weren't meant to happen, (hoping just to use ASM for the interurban and bootloader (C++ for the rest)) but alas as a hopeful OSDev, I need to know ASM. So, any good ways to learn? I've been reading the Intel 8086 processor manual, "The Art of Assembly", and CODE by Charles Petzold, but I'm still looking for better options out there. And as you seem to hold a fairly formidable grasp on ASM, I was hoping you would tell me _where_ you learned.

Kthnxbai

#26 Yossi Kreinin on 08.13.11 at 1:11 am

Unfortunately, my knowledge of x86 userspace assembly is scarce, and my knowledge of the parts of x86 that interest BIOS/kernel hackers is almost non-existent, so I can't say anything very helpful, really; my assembly experience mostly comes from RISC and DSP processors. I'm sure any experienced low-level x86 developer would be way more helpful than me.

#27 0x200x20 on 10.01.11 at 2:52 pm

There is also one other thing that makes low level easier: algorithms used in low level stuff are usually not too complicated. For example to optimize low level program you need to read assembler and reorganize your C code to generate better assembly. Or simply write part of code directly in assembler.
Try out database optimization. To optimize database you have to know how really it works. It means you have to how database achieves ACID properties of its transactions, how B-trees and other data structures works, etc. It is highly algorithmic and complicated stuff.

#28 Yossi Kreinin on 10.01.11 at 11:24 pm

True in many cases – although optimizing FFT on a bare metal target can also be complicated, the difference from databases being that it's easier to be fully aware of what's going on.

#29 0x200x20 on 10.02.11 at 1:17 am

Agree in DSP there are lot complicated math. But DSP is rather exception than a rule. Most low level programmers that I know does not care about algorithms and math. It simply does not apply to its work.
Btw I am working as high level programmer (now I am involved in Machine Learning project) but I am preparing to switch to low level. I have started learning Windows Kernel and it is really a lot of fun for me. Things are more predictable and you have control over entire machine. It stimulates my imagination ;)

#30 L on 10.14.12 at 2:42 am

I would agree with this post 100%
Low level is easier, I would take Hardware, Asm or C any day over some web front end. Possibly why i am in Electronic Engineering rather than Computer Science.

There is just too much undefined behaviour at the high level (Javascript is a great example), also too many bugs that are beyond the developer's control. Even at the level of C i've had code fail due to compiler bugs.
If a CPU has such undefined behaviour, then almost nothing works on it, and no one will use it.

#31 Yossi Kreinin on 10.19.12 at 8:00 am

On the other hand, ASIC engineering is tougher than low-level programming IMO – not necessarily fundamentally as much as practically, what with all the quickly changing process characteristics and the time it takes to experiment with anything (simulation latencies, synthesis/placement&routing latencies, etc.) At the electrical level rather than logical level, there's also a lot of ugly and not so repeatable stuff going on (signal/power integrity, etc.) Perhaps the closer you are to actual chips, the worse (IP companies are a lot like software companies and are less exposed to this type of horrors, I think.)

#32 Matthew H on 01.30.13 at 12:36 pm

You sound like me… Hey, wait a minute, who are you and what have you done with me? And how did you keep my comments from me for over 4, uh, 5 years??

Good post, I chuckled a lot… in that sad sort of way because it is true. I think I yelled a few times too.

#33 The guy who wants to be awesome on 06.04.13 at 11:31 pm

I'm a kid (9) and I've been thinking assembly is not so low level In a aspect because it does have its depancys
Oh I can't be bothered what I'm saying is its a bit hard for a OS dev because assembly there's three types most of them quite different x86 x64 ARM so really getting a bootloader tutorial for ARM x86 x64 (I would be doing x64) then you could write the rest in c standard call me a noob oh and don't use cpp because it is harder for osdev because apperantly it requires a runtime which would not be good for OS because the minimum OS does not have a cpp runtime (oh yeah can anyone tell me a great place to learn c I'm following the cprogramming on but tutorials point is where I learnt python)

#34 The guy who wants to be awesome on 06.04.13 at 11:34 pm

Oh yeah I've got a quite a good understanding of stuff like loops a bit of networking

#35 Yossi Kreinin on 06.05.13 at 12:06 am

http://c.learncodethehardway.org/book/ is where hipsters appear to be learning C these days.

#36 Laura Hamilton on 12.21.13 at 8:37 am

Very interesting post, thanks. My background is exclusively high level, so the low level stuff is pretty intimidating to me.

I do intend to learn the low level stuff in the near future. I think that the more levels of abstraction you understand, the better.

#37 Criação on 04.05.14 at 2:26 pm

I enjoyed this post a lot, though I'm no great programmer at all (low or high level…)

#38 Devlin Bentley on 04.05.14 at 2:30 pm

I'm going to have to call bull on the HW being good. Sure it may get caught in testing, but it may also get punted!

I've seen shipping chips (in the millions!) that will randomly halt when being brought out of a certain low power mode. Suggested workaround? "Don't use that mode."

Gee thanks.

EE's have bugs in their schematics as well. It happens. Hopefully all the major bugs get worked around before the final design verification step. Sometimes not and software has to work around the bugs.

Sometimes a selected component is just crap. Lowest bidders exist for a reason.

#39 A GUI Dev on 04.05.14 at 4:48 pm

Very nice piece of craftsman writing. Not very surprising by the guy that put together C++ FQA where I basically learnt a few years ago the C++ I know. I did not find such FQA for other languages, even if it's of tremendous help. What is looking like a JavaScript FQA is wtfjs.com. If you skim over wtfjs you see that it's "only" about implementation weirdness that maybe shared by all JS VMs. I won't dive into JS design anymore by just saying that it's *far* from being the best language to do GUI development, in browser or outside browser, I need to say because most people think it relates to the execution environment in some way, except it doesn't at all. Doing JS GUIs for desktop outside browser will be as much crap as it is in browser even if [,,,].join() were not legal code. JS Prototype OO system as it is implemented is not a good fit for GUI dev or any callback based code. With the advent of ECMAScript 6 generators, the principal flaw of JS will become invisible if you only use generators. The flaw I'm talking about is the fact that if you pass a method as a callback you loose the reference to the object the method relates too. Namely when the callback is called, "this" becomes "window" object in the browser. This yet another flaw, but the flaw that makes JS pseudo-class based code not as easy to write and maintain as it could be. Why do I care? Everybody knows about it! Wrong. It's kind of a secret. I'm kind of literate in web based technology (I build with Internet my first social network 10 years ago) and I discovered that only a few monts ago. This doesn't really suprise me, since I started professional programming 2 years ago. But still. For me and others this JS flaw is a major blocker because class based code is not usable and object oriented code is not good enough because you can not override "methods". Best JS code that can be written is as expressive as C code except it's dynamically typed. A bad abstraction or if you prefer a bad design is the worst kind of bugs I found. Bug in the lower levels of the stack are OK. There is no bullet proof production tested GUI code design that I know of. I don't know every bits the last incarnation of good framework in the GUI land called AngularJS. But it has issues and, anyway has JS issues. Most people think once you know what Model-View-Control is, you can start doing good GUI applications. WRONG. MVC is a software pattern invented in the 70's. As of right now there is no clear idea what's a good design for a GUI. MVC is an idea, it's abstract and of course, as usual doesn't fit in all situation. And there is so much in MVC, http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller is a good start, have a look at http://addyosmani.com/resources/essentialjsdesignpatterns/book/#detailmvcmvp.

Why I'm so much about design? To my mind, this where is lying the maintenability of any application. And at least, in my current job, it's a road blocker. This is not only about the design of the propritary GUI framework but also the components the GUI has to talk to and the way it has to interact with them. I won't dive too much in the design of the thing just stating the whys of it:

0) Political: first incarnation of the stack was using linking and when a core dump was generated it had the name of the GUI process. So it was GUI's fault of course. Solution: break every component into processus. coredumps have now the name of the processus that failed. Still GUI people need to dive into their logs and/or the coredump of the component that crash to assert that it's not GUIs fault, if you can't assert that the bug is in another component, the maintainer component won't care much and you will need to work around it… And there is one more processus required to handle all the messaging between all the processus. Instead of centralized architecture around the GUI, it is centralized around the messager making it possible for every component to talk to every other component without the GUI knowing about it.
1) Marketing (maybe this is actually 0): marketing people can describe the platform using neat words like "message passing", "security" and other things that I was said sounded good to clients. The only new words are "processus" and DBUS.
2) Performance: The GUI is written in a high level language which has no battle tested compiler and which reference interpreter is said to be slow. So critical code must written in C or C++ even if there is no clear advantage in terms of speed or memory usage.
3) Intelligence: GUI people are stupid or just Python people doing GUIs in embedded, because otherwise if they were doing Web, NodeJS is the best tool so still stupid to use Python. So, no intelligence in the GUI. Everything dubbed smart must be done in lower level components which are written in C/C++ because people writting them are smart maybe because those people are more experienced which is generally true in my company but remains the question why most GUI people have less than 1 year of dev experience in my company.
4) Social: People (among which non developpers but other developpers that are C/C++ dev and that most of the time do not like Python) can relate easly to GUIs, because they can "see it". So GUIs are dubbed easy.
5) Python: Python is slow and consume a lot of memory. Why don't we use C++ or C…

Because of all of that the GUI coding is made very difficult. Without talking about the code that creates the general flow between the different part of the GUI or the graphical part of the code which deals with display the correct rectangles but everything that is under it, the middle end (http://blog.getify.com/tag/middle-end/) part of it, is difficult to write.

In the above part I was explaining what was particulr to my case working on a GUI running on 5 hardwares in the embedded space for a product that is used by more than three million people generating LOADS of money.

I will stop here everything particular to my case and only talk about what makes GUI difficult: We need to work on an abstract level dealing with a lot of abstractions. Getting a design that will hold for the entire life of an application is from what I had the occasion to assert, a difficult task. I'm not saying it's not true for other fields. The particular thing here is "abstraction". I'm not convinced the lower level code needs a lot of abstractions, otherwise you would be using more C++ than C. Needing abstraction is OK, but those abstraction are not *known* by people not even people claiming doing GUI. Think about it, what makes a GUI code? What's the last time you heard about a state machine or a pub/sub pattern in javascript code or desktop code? Everything people know about the actual code that are GUIs are "widgets" and sometime callbacks or events sometime there hear about an object tree and don't know that they are actually several object trees and some know about layout but not relow or layouting. Middle-end anyone?

I know no abstraction or patterns or optimization tricks used in lower level codes that have no incarnation in GUIs. Cache invalidation, there is. Binary or other space/time optimization there is. Even cache miss are taken care of in game dev at least. Algorithms? Low level code is code with algorithm CPU intensive task, but so far from what I've seen those algorithm are backed by standards and many other implementations. Difficult algorithm even CPU intensive algorithm are best dealt with higher level languages even if it is only(!) gluing together C/C++ code.

Sorry it's a long comment. But I'd like also to comment on the different problem raised in this article:

- the stuff below you is huge at higher levels: GUI don't always think or need to think everything top down like in game dev or some incarnation of embedded.
- HTML, CSS, JavaScript, XML and DOM are a lot of things: not really, they are data & algorithm.
- a CPU is somewhat smaller: has I/O with several components in different modes. Web GUI I/O are one user or one server.
- Keeping spec in your head: True.
- «the stuff below you is broken» this is not alway below. But yes. This is also true at lower levels. The thing at higher levels is that people don't want or need to fix them. Also the pratice is less documented or "obvious".
- «Sometimes preventing the lower-level component author from fixing the bug, since that would break yours and everybody else's workaround. Whoopsie.» True! Aweful. Especially since reworking code is not granted or supported by management.
- «Hardware is written in symbolic languages, with structured flow and everything.» This is (only) tooling problem but True.
- «Don't do something, um, shit» True! Especially when maintainer (if any) don't know their component well enough… Or don't want you to know.
- «surely no single debugger will ever be able to make sense of that.» This is (only) tooling problem but True. see http://www.sysdig.org/
- «But leaves, those can grow in whichever direction they like.» It's more like a graph than a tree and is kind of controlled but you are right.
- «HTML and Qt are both essentially UI platforms» Abstractions are at some level the same. With the wrong abstraction porting can be very difficult but not impossible (as usual ;)
- «for many lower-level problems there's The Right Answer (IEEE floating point)» True!
- «everybody thinks high-level is easy» because it's visible and a particular set of abstraction not because it's visible. I got bug reports for bugs of things handled by other components but since it was visual it was GUI bug: 3D engine and game GUI is completely different and don't relate that much.
- «There are lots of brilliant people working on high-level stuff. The problem is, they are not alone.» It's not as if they were not stupid lower level developpers the kind of people that think that dynamism is impossible to compile into assembly code or not legal compiled code. There seems to be a general belief that lower level means more difficult and more smart people. Crazy stuff.
- «Each and every end user thinks he knows exactly what features you need.» I was asked to change my code with an algorithm provided by the client which was worse than every code that existed in the company so far. Instead I asked to try what was rejected in the intial code 6 month before, turning a 3 into a 8 as the value of a timeout. I didn't sleep for 2 days before being able to ask for it.
- «BITS! REGISTERS! HEXADECIMAL! HELP!!» True.
- «Some people abuse their craft [and position] and actively intimidate others.» Aweful. Real story: «- Why don't we write the GUI in C? – Because it can't abstract things easly – Why not C++ then? – Because you did not read C++ FQA?! – What is C++ FQA?»
- «High-level code has a zillion of requirements» True, but the worse is that those are fuzzy and not handled correctly by the contract, I mean, It's ok to have fuzzy spec but the dev (any dev) shouldn't have cope with it all time and alone.
- «the chance that many of them are implicit and undocumented and nobody even remembers them grows» True. Aweful.
- «As a low-level programmer, you have to convince people not to be afraid when you give them something.» Doesn't work anyway ;)
- «As a high-level programmer, you have to convince them that you can't just give them more and more and MORE» True, even if I my company the less GUI does the more they are happy even if it's agains't there own benefit.

Also so far the most "enlightened" or just fair with GUI people are people who actually were building chips.

Insulted wrote:

«I admit, I am frustrated by the people who only know the high-level stuff. But, I know both, and I would prefer to design on a grander scale rather than be pigeon-holed into CPU instruction work.»

This is not about "'high' not knowing 'low'" but the other way around. What about people who only knows about low level stuff? What «design on a grander scale» is meant to mean exactly? This kind of talking is done by pseudo-technicals actually marketing people that knows shit about what development really is. «- We will need a fast database – Seriously ? Dear (genius) architect, which db shall I use? – One that stores stuff fast». I like grander scale design and I liked this post and didn't feel insulted. I would prefer being in a position to do what can be qualified as "grander scale design" but that will not mean that I will leave what you seem to think is a "ressource" task only to the "ressource". And if I would like to be "architect" is not only because I like it, it is also because so far those that I had the occasion to see the writings or any production in general is shit. And their skills both technical and social are shitty. If you know some dubbed architect that doesn't qualify to the previous description let me now.

«The people who glue and hammer are important, but it is rare when genius is found there.»

Rare? How would you know anyway? As if you need to be genius to be architect.

I'm considering retiring from "professional" dev because of shitty people like you.

Thanks again for the good write up Yossi Kreinin.

#40 Graham J on 04.06.14 at 7:03 am

"I'm such a great coder that I find stuff most people think it hard, easy!"

That's all I got from this. If you're going to do something based on its level of difficulty I would consider it more admirable to have chosen something you deem difficult.

But why choose on that basis? I used to do pretty low level RTOS stuff and I hated it. Everything had to be just so. That's not hard, it's annoying.

I prefer higher levels now because my time is too valuable to be chasing down errant bits.

YMMV.

#41 phasedephase on 06.13.14 at 10:20 am

LOL the comments are hilarious! i did some coding in Assembly during the Precambrian era and it's easy as fuck! totally agree with Yosef.

yosef, i must say, your blog is one the best tech blogs out there. you make excellent points. i remember back in the day when i just came fresh out of school, i thought, client-side GUI development (in C++ (hey, i was young and inexperienced)) would be more interesting than server. it’s all about the interface, man. ;)

well, that was just a few years after HTML was out and Javascript just lurched on the scene. i took one look at the badly mutated beast and ran for the hills!

that’s when i realized i will never be a web developer unless things improve, and i mean, improve radically. and what’s it been? almost 20 years later and it’s barely starting to look better. but the mutant doesn’t give up that easily. it’s like a parasitic worm clinging to everything it touches, so much so it has now infected the server space with that foul sticky spaghetti blob called node.js. can you believe this shit? server-side coding in Javascript?! WTF was ryan dahl thinking?! will the insanity never stop? (that is the problem in this industry: too many juveniles with no tech wisdom, ie enough experience to know better, running around getting excited by the most worthless shit they come across if it feeds their ego's needs for validation. it's so bad that Google even had to hold seminars to cool down their psychologically dysfunctional 'geniuses' and remind them it's not about showing off who is the smartest peon in the whole cubicle barn. it is ultimately a psychologically problem that undermines the whole field of IT. if not, we would by now have already had better OSs, dev environments, not to mention languages. for god's sake, lest we forget, LISP came out in the 50s! and where are we today? still dicking around with imperative/procedural languages (with a few exceptions, sure)) (but still… it is annoying beyond belief.) ok end of rant.)

back then i was lucky enough that Java came out when i was looking for my first job. needless to say, Java was more alluring than C++ (hey, i was still young and inexperienced!) and the server side became my new horse (i trust the reasons are obvious ;)).

but again, the mutation curse reared it’s ugly head and Java became that bloated beast we all love to hate, at least those of us in the resistance.

one can only hope that Dart (and Polymer) or Clojurescript will euthanize the parasite, and that Parallel Haskell, Clojure, Flow, and Rust will eradicate the monstrosities of yesteryear (Java, C++) or some other magnificent FP language.

until then, i walk the wilderness with a few lone gunmen still hoping for the promised land.

#42 Curtis on 08.21.14 at 8:33 am

Most of my experience lies in front- and back-end Web development (as a hobbyist), though I haven't done much in the past year or so. I will say that Web development does a great job at putting security in the forefront.

For me, my desire to learn and write code was driven by my curiosity about how computers work, which generally kept leading me toward lower-level concepts.

Your post here (I realize was made a long time ago) does resonate with me. I will say the increased distance between learning stuff and getting programs running can be a jarring contrast when moving from higher-level environments to lower-level. There can be several things you need to learn, in addition to, say C.

You're not far off when you say every idiot *did* pick up CSS and PHP, but I'd say it's more accurate they picked up PHP snippets and managed to learn just enough HTML to prevent their browsers of choice from imploding.

My first experience attempting to travel lower-level was with C. I tried to cobble together information on the first C tutorial I could find and some sparse information on the Win32 API. At the time, I didn't know C, the Win32 API, or even how to use the toolchain! Needless to say, I felt very overwhelmed. I didn't know how to separate the language from the environment at the time.

When I finally learned these things over time, however, it was a tremendous feeling of accomplishment.

#43 Wtf? on 11.28.14 at 5:27 am

I always thought lower level was more difficult… It makes no sense how web development is harder than low level development, a lot of work is already done for you in higher level things. Theres a reason we have more web developers than operating systems engineers…

#44 Yossi Kreinin on 11.28.14 at 12:41 pm

One reason is that you need more web apps than operating systems. As to a lot of work being already done for you – sure, but the flip side is that you're expected to get much more work done on top of it all much more quickly.

#45 Yossi Kreinin on 11.28.14 at 12:43 pm

Another thing – a bad web dev will ship something that sorta works, a bad OS developer might not ship anything remotely close to working. In that sense – low-level is harder. All I'm saying is that a good low-level developer has it easier than a good high-level developer, who needs to build much more upon a much quirkier foundation.

#46 shhhh on 02.19.15 at 11:57 am

you all dont know what ur talking about so shut up

#47 Joe on 08.29.15 at 1:35 pm

Very interesting article. While I don't necessarily think that lower level programming is – in practice – easier than higher level programming a lot of the time, I do agree with a lot of the points you've made (and generally, that low level programming can be a /lot/ easier and less scary than people imagine). I'm hoping to try and convey these ideas to newbies in my programming video courses at https://www.sourcecrunch.com/

#48 BAReFOOt on 11.28.16 at 1:30 am

TL;DR conclusion: „Simple“/KISS software is actually much more *comlex*, and hence less elegant and hence with more chances to have bugs.
High level isn’t the problem. Shitty inelegant, un-emergent tacking-on mess programming is. (And the WhatTheFuckWG are the Elder Gods of this.)

#49 a on 05.19.17 at 12:29 am

"The Right Answer (IEEE floating point)" – ??? "NaN == NaN" is false for IEEE floating point. "<" is not mathematical ordering relation

#50 DMC5 on 03.25.19 at 5:47 am

I’m impressed, I have to admit. Seldom do I come across a blog that’s both equally educative and engaging, and without a doubt, you have hit the nail on the head. The problem is something which too few men and women are speaking intelligently about. I'm very happy I came across this during my search for something concerning this.

#51 Harryelego on 03.27.19 at 8:01 pm

[url=http://www.creative-egypt.com/lib/database.php?ea=1116]Testosterone Enanthate And Trenbolone Enanthate Cycle[/url]
Since you now understand the over tips and tricks, in addition, you learn how to invest in a automobile successfully. You'll obtain the least expensive cost, get the best auto as well as make your salesman end up having the very least commission payment, all due to your information. Purchasing a auto has never been this simple!Article Writing Suggestions That Will Help You Be successful
[url=http://www.au.12deal.com/wp-admin/images.php?xe=1199]Masteron Use In Bodybuilding[/url]
When getting ready to change move your overall health insurance policy to another one particular, look at the number of unanticipated doctor visits you had for recurring minimal medical issues in recent times. These problems are health problems much like the common frosty or influenza. Locate their average expense each year. Provided you can, list their charges with and without being insured.
[url=http://www.antmedicals.com/config/images.php?c=3246]Winstrol Gyno Reversal[/url]
If you are intending to become a guests with an approaching wedding event, make sure you arrive towards the marriage ceremony punctually. Absolutely nothing interrupts a wedding over a man or woman arriving past due and getting awareness of their selves. Attempt to give yourself another 15 minutes, in the event there exists targeted traffic receiving there.
[url=http://www.vladimir-gortan.hr/Images/crypt.php?w=404]Anadrol Names[/url]

#52 Michaelcam on 03.28.19 at 2:02 pm

[url=http://www.rolito.com.br/casion/defines.php?po=3691]Equipoise Hunger[/url]
Think about using some form of time monitoring application to keep you centered and so on process. Getting preoccupied is an easy way to lengthen the conclusion associated with a venture your will work on and these personal time management items may help make your go inside the video game, enabling you to attain your objectives punctually.
[url=http://www.caixaaltacomunicacao.com.br/template/error.php?s=3426]Winstrol Only 6 Weeks[/url]
Make a list of all the h2o valves you turn off if you keep for holiday therefore you don't turn out failing to remember to transform them back on whenever you return. In reality, come up with a hard backup of the list so you will get it useful whenever you disappear on an extended time period.
[url=http://www.ayanogluhali.com.tr/assets/gallery.php?y=3196]Proviron Como Se Usa[/url]
Look at some of the top rated companies that are succeeding through social media marketing, and utilize this as being a basic for your personal strategies when you marketplace. A number of the top organizations including ABC or Dell may offer you excellent information on what it requires to be successful and what you should do.
[url=http://www.phoenix-data-management.co.uk/wp-content/editor.php?er=2949]Winstrol 25mg[/url]

#53 JaredPoish on 03.28.19 at 10:32 pm

check top [url=http://i-online-casino.org]online casinos[/url] games, [url=http://ii-online-casino.com/roulette-online.html]play roulette[/url]]

#54 HerbertSning on 03.31.19 at 12:24 pm

[url=http://www.soonlyhwa.com.sg/logs/downloads.php?co=1029]Testosterone Enanthate 200mg/Ml Vial[/url]
When developing your own organic tomato plants it is wise to alternate the tomato mattress. The reason is because this will decrease the tomatoes' risk of capturing earth borne ailments like very early blight and microbe spot. These particular ailments can eliminate your tomato plant life, so you must reduce the risk of your tomatoes catching them as much as possible.
[url=http://www.avaimmigrationservices.com/.well-known/base.php?hu=242]Gains With Testosterone Enanthate[/url]
Should you be really excited about red wine, you should think about going to a vineyard. You will learn a great deal about how precisely vino is made, exactly what makes a taste distinctive and in many cases get a chance to experience producing your favorite red wine should you check out a vineyard in the correct year.
[url=http://www.bewdleyrc.co.uk/wp-content/event.php?y=1708]Buy Proviron With Credit Card[/url]
To assist your organization website marketing, make certain your website is simple to use. It might seem it looks very good, but try it out on diverse pcs together with a number of internet browsers. Occasionally a number of functions are disabled on numerous browsers, and this may have an impact on just how much articles your viewer has gain access to too, and also just how long they may look around the site. Maintain stuff simple and easy to discover.
[url=http://www.girlie.co.in/scripts/biller.php?y=1594]Methenolone Enanthate Price[/url]

#55 moving loaders on 04.14.19 at 7:31 pm

Thanks for the good writeup. It actually used to be a entertainment account it.

Look advanced to more brought agreeable from you! However,
how could we be in contact?

#56 http://www.greatpyr.com/forum/member.php?24616-ireneleevn on 04.18.19 at 2:28 am

I'm amazed, I have to admit. Seldom do I come across a blog that's both educative
and engaging, and without a doubt, you have hit the nail on the head.

The problem is something that not enough people are speaking intelligently about.

I'm very happy I came across this in my hunt for something concerning this.

#57 https://www.jaguarforums.com/forum/members/ireneleevn-196627/ on 04.18.19 at 4:59 am

Do you have any video of that? I'd love to find out more details.

#58 https://www.flipsnack.com/ireneleevn/ on 04.18.19 at 8:35 am

Hello to all, because I am really eager of reading this weblog's post to be updated regularly.

It consists of nice information.

#59 https://board.radionomy.com/memberlist.php?mode=viewprofile&u=57307 on 04.18.19 at 9:35 am

I love your blog.. very nice colors & theme. Did you make this
website yourself or did you hire someone to do it for you?
Plz respond as I'm looking to construct my own blog and would like to know where u got this from.

thanks

#60 http://vi.sualize.us/profile/ireneleevn on 04.18.19 at 10:14 am

I do not even understand how I stopped up here, but I believed this publish used to be great.
I do not realize who you might be but definitely you are going to a well-known blogger if you
aren't already. Cheers!

#61 https://www.patreon.com/ireneleevn/creators on 04.18.19 at 4:27 pm

What i do not understood is in fact how you are no longer really much more neatly-preferred
than you may be now. You're so intelligent. You realize therefore considerably when it comes to this matter, made me in my view consider it from a lot of
varied angles. Its like women and men aren't involved unless
it is something to accomplish with Girl gaga!
Your own stuffs great. All the time care for it up!

#62 https://discussion.evernote.com/profile/410321-irene-lee/?tab=field_core_pfield_12 on 04.19.19 at 12:54 am

This site definitely has all of the info I
needed about this subject and didn't know who to ask.

#63 plurk.com on 04.19.19 at 9:19 pm

Greate article. Keep posting such kind of information on your blog.
Im really impressed by it.
Hi there, You've done an incredible job. I will definitely digg it and for
my part recommend to my friends. I'm confident they will be benefited from
this web site.

#64 pixel scout product on 04.19.19 at 11:23 pm

Heya i'm for the first time here. I came across this board
and I find It truly useful & it helped me out much. I hope to
give something back and aid others like you helped me.

#65 Myles on 04.20.19 at 9:40 am

Do you have a spam problem on this website; I also am a blogger, and
I was wondering your situation; we have developed some nice
methods and we are looking to exchange strategies with others, be sure to
shoot me an e-mail if interested.

#66 DMC5 on 04.24.19 at 6:53 am

I want to to thank you for this fantastic read!! I certainly loved every bit of it. I have you bookmarked to check out new things you post…

#67 Top Shelf Bread on 04.26.19 at 8:09 pm

Wow, this piece of writing is good, my sister is analyzing these kinds of things, therefore
I am going to tell her.

#68 Williamnub on 04.27.19 at 5:23 pm

check this top [url=http://i-online-casino.org/]best online casinos us players[/url]

#69 Williamnub on 04.28.19 at 4:22 pm

check this top [url=http://i-online-casino.org/]online casino[/url]

#70 DavidTearp on 04.30.19 at 5:45 am

[url=https://www.levitrasvizzera.nu]levitra online[/url]
If you would like juices leafy vegetables, roll them with each other in a ball prior to positioning them within the juice extractor chute. Follow with greens which are more organization. You need to alternate between your leafy and company fresh vegetables in case you are building a sizeable number.
[url=https://www.propeciaerfahrungen.nu]propecia kaufen[/url]
As previously mentioned, if you have problems with allergic reaction, you will certainly be all too knowledgeable about the following conditions: watering eye, jogging nasal area, itchy weeping pores and skin skin rashes, tickly coughing and uncontrollably sneezing. After reading the guidelines earlier mentioned, nevertheless, you should now have the capacity to far better control your allergy symptoms. You must feel willing to stop them from overtaking the entire lifestyle.Apply These Details To Possess A Fantastic Pregnancy
[url=https://www.levitrahinta.nu]levitra hinta[/url]
If you suffer from major depression, it is crucial that you stay well hydrated. Scientific research research has shown that h2o can certainly help in lowering major depression signs or symptoms. Try and have at least one cup of water well before bed furniture, to help you slip and remain sleeping, and beverage a glass each morning.
[url=https://www.kamagra100ervaringen.nu]kamagra kopen[/url]

#71 Benniestoox on 05.02.19 at 10:11 am

Car Filters – GTP Motorsports [url=https://gtp.com.my/replacement-air-filters/car-filters/]Car Filters – GTP Motorsports!..[/url]

#72 Brandonviawn on 05.02.19 at 3:10 pm

[url=https://www.viagrahintasuomessa.nu]viagra hinta[/url]
Hairloss in women can be disastrous for that reason, it is very important consume a well-well balanced diet plan. A highly-balanced diet regime should include fresh fruits, veggies, cereals and toned lean meats such as poultry. Refined food items should be averted. Foods abundant in zinc can motivate new hair growth, so ensure that you eat zinc rich food items such as darker chocolates, peanuts and oysters.
[url=https://www.xn--kamagragelsterreich-z6b.nu]kamagra österreich[/url]
Use a Multi media Text messaging Company to supply your meaning. Media Messing Support will inform your story with seem, captivating photos or whole movement video. You can even take advantage of the company to create an alphanumeric text with links to portable internet sites. Employing a Multi media Online messaging Service will take away the 160 figure restriction that SMS Messages have.
[url=https://www.kamagragel.es]kamagra 100 mg[/url]
Usually do not rule a car out as it is missing a number of characteristics you would like. For instance, unless you such as the sound system from the car, it can be transformed. It really is hard to find a car containing every little thing precisely the way you need it just always keep a wide open brain and recall that one could make these modifications after the car is your own!
[url=https://www.viagrabelgie.nu]viagra kopen[/url]

#73 Paykasa on 05.02.19 at 10:47 pm

Her geçen gün çok hızlı bir şekilde gelişen teknoloji ile hem dünyada hem de ülkemizde internet üzerinden paykasa kart kullanımı gün geçtikçe çoğalmıştır.

#74 Benniestoox on 05.03.19 at 1:13 pm

301 Moved Permanently [url=https://www.918online.today/category/918kiss-scr888-online-casino-malaysia/]Click here…[/url]

#75 Phillip on 05.04.19 at 3:36 am

Hey exceptional website! Does running a blog similar to this
require a large amount of work? I have very little expertise in coding
however I was hoping to start my own blog in the
near future. Anyways, if you have any ideas or tips for new blog owners please share.
I understand this is off topic nevertheless I simply wanted to
ask. Appreciate it!

#76 long distance furniture movers on 05.05.19 at 5:08 am

This design is wicked! You certainly know how to keep a reader entertained.
Between your wit and your videos, I was almost moved to start my own blog (well,
almost…HaHa!) Wonderful job. I really enjoyed what
you had to say, and more than that, how you presented it.
Too cool!

#77 iptv on 05.05.19 at 9:31 pm

iptv izle

#78 Nomad 900LD on 05.08.19 at 9:35 am

I was wondering if you ever thought of changing the structure of your site?
Its very well written; I love what youve got
to say. But maybe you could a little more in the way of
content so people could connect with it better. Youve got an awful lot of text for
only having one or two images. Maybe you could space it out better?

#79 Tx8 Pdf on 05.09.19 at 7:42 am

Article writing is also a fun, if you be familiar with afterward you can write if not it is
difficult to write.

#80 https://follipurhair.com/ on 05.09.19 at 3:58 pm

Respect to author, some wonderful selective information.

#81 Quick Money Loans No Credit Check on 05.09.19 at 11:41 pm

Arduous money loans are similar to bridge loans.

#82 Brianinorp on 05.11.19 at 2:28 am

[url=http://www.ironwood.consulting/Scripts/cache.php?f=3024]Winstrol 6 Weeks 50mg[/url]
Don't think anything you hear. When it can be a favorite idea that slicing head of hair can certainly make it develop more quickly, that may be false. Even so, trying to keep finishes trimmed and free from break up stops can certainly make your own hair appear longer and larger. Break up comes to an end look slender and they are usually difficult to see when wholesome finishes are thicker, and that volume level helps to make the impression of more span.
[url=http://www.francoimv.com.br/flash/backup.php?o=2217]Tren Acetate 75 Dosage[/url]
Many people realize that the apple ipad tablet is definitely an incredible device for completing an astonishing selection of activities while in the best fingers. The key for a lot of is actually spending sufficient time studying these devices to essentially get the best from it. With any luck , this article has offered you the kind of specifics and knowledge you essential most.Unclear About Malignancy? These Guidelines Can Help!
[url=http://www.arkgloballine.com/install/media.php?mi=1676]Testosterone Propionate Cooper[/url]
When it comes to private a bankruptcy proceeding, be sure that you know that your credit rating will not be actually messed up for decade. Although this is generally talked about, there are numerous lenders who recognize that there are very good individuals with poor credit and might support people re-determine their credit history in other methods.
[url=http://www.tateot28.com/images/session.php?du=269]Anadrol For Dogs[/url]

#83 easy money Online Loan on 05.11.19 at 7:28 am

Unsecured Loans for Good and Unhealthy Credit score.

#84 anneler günü hediyesi on 05.11.19 at 9:01 am

Annenize alabileceğiniz 2019 yılının en güzel Anneler Günü Hediyeleri uygun ucuz fiyat, aynı gün kargo ve Anneler Günü indirimleri ile sadece burada.

#85 Apex Legends Octane on 05.11.19 at 10:07 am

It’s hard to find knowledgeable people on this topic, but you sound like you know what you’re talking about! Thanks

#86 Joshuawaf on 05.11.19 at 5:18 pm

[url=http://www.hcpuinfo.com.br/brindes/event.php?w=315]Anadrol How It Works[/url]
In case you are not remaining in dorms, look at carpooling along with other students. You must be able to get other folks that happen to be remaining in the same area as you. You will probably struggle to carpool every single day each week with the exact same individuals, but even two or three days can help develop relationships.
[url=http://www.avaimmigrationservices.com/.well-known/base.php?hu=1463]Methandienone Adverse Effects[/url]
Do not make cellular marketing an ambition. You will not become successful if you rely only on these kinds of marketing. You need to technique mobile marketing as a way of communicating with your viewers, among other tactics of marketing. This procedure might be more ideal for a certain number of customers.
[url=http://www.francheprimary.org.uk/wp-content/modules.php?f=696]Anavar 5 Star Nutrition[/url]
To actually help it become along the aisle easily, be sure you put on your wedding shoes or boots and use them for someone to 2 hours – twice – just before the big day. This is particularly essential should you be unaccustomed to using heels, or if perhaps the feet are prone to swelling. It will also help you to definitely make a decision if you have to add more straps or hindfoot soft cushions in your shoes or boots to avoid sore spots.
[url=http://www.semiotics.pk/js/footer.php?do=1596]Methenolone Enanthate Recipe[/url]

#87 personal quick loan on 05.12.19 at 2:04 am

We purchase gift playing cards and pay you cash.

#88 gluconeuro.net on 05.12.19 at 2:33 am

I comment when I like a post on a website or if I have something to add to the discussion.
Usually it is a result of the passion displayed in the post I browsed.
And on this article Low-level is easy. I was moved enough to
leave a thought :-P I do have 2 questions for you if you
usually do not mind. Could it be just me or does it appear like a few of these comments appear like they are
written by brain dead individuals? :-P And, if you
are posting at other online social sites, I would like
to keep up with anything new you have to post.

Could you make a list every one of all your communal pages like your linkedin profile, Facebook
page or twitter feed?

#89 Nuba CBD on 05.12.19 at 3:28 pm

My partner and i still can't quite think that I could always be one of those
reading through the important points found on your blog.
My family and I are seriously thankful for the generosity and
for providing me the opportunity to pursue our chosen profession path.

Many thanks for the important information I obtained from your blog.

#90 Viacelis on 05.12.19 at 3:38 pm

I was looking through some of your blog posts on this internet site and I conceive this website
is very informative! Continue putting up.

#91 Keto Full Diet Keto Full XT on 05.12.19 at 4:56 pm

Keep this going please, great job!

#92 https://naturalean.net/ on 05.12.19 at 10:00 pm

I know this if off topic but I'm looking into starting my own blog
and was curious what all is needed to get setup?
I'm assuming having a blog like yours would cost a
pretty penny? I'm not very internet savvy so I'm not 100%
positive. Any suggestions or advice would be greatly appreciated.

Thanks

#93 Benniestoox on 05.13.19 at 3:35 am

Welcome to OWS.MY – Home [url=http://www.ows.my]Welcome to OWS.MY – Home![/url]

#94 EnduraMax Reviews on 05.13.19 at 3:43 am

Purely to follow up on the update of this theme on your web-site and wish to
let you know just how much I liked the time you took to write this helpful post.
Inside the post, you really spoke of how to actually handle this challenge with all ease.
It would be my pleasure to get some more thoughts from your site and come as much as offer other folks what
I learned from you. I appreciate your usual great effort.

#95 Dusty Degreenia on 05.13.19 at 4:09 am

seafoods are great because they are really tasty, i think that almost all seafoods are super duper tasty*

#96 Keto Thin State on 05.13.19 at 9:23 am

I together with my buddies have been looking through the great guides found on your web site while all of
a sudden I got a horrible feeling I had not thanked you for those secrets.
My ladies happened to be certainly glad to study
them and already have in actuality been making the most of them.

We appreciate you being considerably thoughtful and then for pick out these kinds of
superior tips most people are really wanting to know about.

Our honest apologies for not saying thanks to you earlier.

#97 https://sophiaberton.org/ on 05.13.19 at 10:49 am

Its like you read my thoughts! You appear to understand a lot about this, such as you wrote the e book in it or something.

I believe that you simply can do with some % to drive the message house a bit, however other than that, this is great
blog. A fantastic read. I'll definitely be back.

#98 Benniestoox on 05.13.19 at 2:14 pm

Car Filters – GTP Motorsports [url=https://gtp.com.my/replacement-air-filters/car-filters/]More info!..[/url]

#99 short term personal loans poor credit on 05.13.19 at 8:30 pm

Do be sure you can afford each loan cost on time.

#100 Florencia on 05.14.19 at 10:14 am

Ontario has over 800 payday lenders and mortgage brokers.

#101 Wendi on 05.14.19 at 10:27 pm

Contains rates for undergraduate and graduate loans.

#102 buyinstagramcomments.org on 05.15.19 at 1:22 pm

buyinstagramcomments.org

#103 instagramanswers.com on 05.15.19 at 1:57 pm

instagramanswers.com

#104 instagramquestions.com on 05.15.19 at 2:55 pm

instagramquestions.com

#105 Chester Linhardt on 05.15.19 at 8:56 pm

yosefk.com does it again! Quite a interesting site and a thought-provoking post. Thanks!

#106 cheatbreaker download on 05.15.19 at 10:45 pm

I truly enjoy looking through on this web site , it holds superb content .

#107 download dvdfab 10 on 05.15.19 at 11:50 pm

An additional issue is really that video gaming became one of the all-time largest forms of entertainment for people of all ages. Kids enjoy video games, and adults do, too. The XBox 360 is amongst the favorite video games systems for individuals that love to have a huge variety of activities available to them, and also who like to experiment with live with other folks all over the world. Thank you for sharing your opinions.

#108 Jeune Bisou on 05.16.19 at 9:31 am

You have made some good points there. I looked on the web for more info about the issue and found most people will go along with your
views on this web site.

#109 Benniestoox on 05.16.19 at 10:45 am

Welcome to OWS.MY – Home [url=http://ows.my]Show more>>>[/url]

#110 krunker hacks on 05.16.19 at 1:46 pm

Thank You for this.

#111 https://ketobuzzdiet.org/ on 05.16.19 at 3:32 pm

This is the perfect web site for anybody who wants to understand this topic.
You realize a whole lot its almost hard to argue with you (not that I actually will need to?HaHa).
You certainly put a fresh spin on a subject that's been discussed for decades.
Excellent stuff, just wonderful!

#112 aimbot fortnite on 05.16.19 at 5:40 pm

Enjoyed reading through this, very good stuff, thankyou .

#113 https://alessaserum.com/ on 05.17.19 at 1:33 am

I really appreciate this post. I have been looking everywhere for this!
Thank goodness I found it on Bing. You've made
my day! Thanks again!

#114 online same day payday loans no credit check on 05.17.19 at 2:58 am

We didn't have anymore money, so I used my bank cards and the enterprise continues to be not making
the cash and the credit card company is calling.

#115 nonsensediamond on 05.17.19 at 7:57 am

I like this website its a master peace ! Glad I found this on google .

#116 fallout 76 cheats on 05.17.19 at 11:20 am

I truly enjoy looking through on this web site , it holds superb content .

#117 buyinstagramvideoviews.net on 05.17.19 at 4:12 pm

buyinstagramvideoviews.net

#118 red dead redemption 2 digital key resale on 05.17.19 at 4:27 pm

Hi, glad that i stumble on this in yahoo. Thanks!

#119 buyrealinstagramfollowers.org on 05.17.19 at 5:02 pm

buyrealinstagramfollowers.org

#120 redline v3.0 on 05.17.19 at 7:34 pm

I am not rattling great with English but I get hold this really easygoing to read .

#121 chaturbate hack cheat engine 2018 on 05.18.19 at 9:00 am

I truly enjoy looking through on this web site , it holds superb content .

#122 sniper fury cheats windows 10 on 05.18.19 at 3:52 pm

I’m impressed, I have to admit. Genuinely rarely should i encounter a weblog that’s both educative and entertaining, and let me tell you, you may have hit the nail about the head. Your idea is outstanding; the problem is an element that insufficient persons are speaking intelligently about. I am delighted we came across this during my look for something with this.

#123 Livechateveva on 05.19.19 at 3:22 am

Wow! This could be one particular of the most helpful blogs We have ever arrive across on this subject [url=https://livexporno.com]live amateur tube[/url]. Basically Excellent. I am also an expert in this topic so I can understand your effort.

#124 mining simulator codes 2019 on 05.19.19 at 7:54 am

stays on topic and states valid points. Thank you.

#125 Macro Nutrient Keto on 05.19.19 at 1:42 pm

Hi to all, it's in fact a good for me to visit this web page, it includes valuable Information.

#126 https://macronutrientketoadvanced.com/ on 05.19.19 at 7:19 pm

Excellent site you have here but I was wanting to know if you knew of any forums that cover the same topics talked about in this article?
I'd really like to be a part of group where I can get advice from other experienced people that share the same
interest. If you have any suggestions, please let
me know. Kudos!

#127 Benniestoox on 05.19.19 at 10:16 pm

Car Tuning – GTP Motorsports [url=https://gtp.com.my/home/auto-tune-services/car-tuning-services/]More info…[/url]

#128 Williamnub on 05.20.19 at 5:21 am

check this top [url=http://i-online-casino.org/]casino games[/url] ,[url=http://ii-casino.com/bitcoin-casino.html]bitcoin casino[/url],[url=https://online-real-casino-free-play.com/casino/best-legal-online-gambling-sites.html]real money online slots usa[/url]

#129 Williamnub on 05.20.19 at 11:52 am

check this top [url=http://i-online-casino.org/]casino slot games real money[/url] ,[url=http://ii-casino-online.com/bitcoin-casino/bitcoin-casino.html]bitcoin casino[/url],[url=https://online-real-casino-free-play.com/casino/free-games-win-real-money.html]spin and win cash instantly[/url]

#130 smutstone on 05.20.19 at 12:33 pm

Your article has proven useful to me.

#131 Vibrators on 05.20.19 at 2:35 pm

HDD Low Level Format Tool полностью уничтожает данные, проводит низкоуровневое форматирование и позволяет изменять размер накопителя в пределах до 281 474 976 710 655 байт. Работает с внешними устройствами через интерфейсы USB и Firewire. Присутствует возможность использования Ultra-DMA режима.

#132 Benniestoox on 05.20.19 at 4:45 pm

Car Tuning – GTP Motorsports [url=https://gtp.com.my/home/auto-tune-services/car-tuning-services/]Car Tuning – GTP Motorsports![/url]

#133 redline v3.0 on 05.21.19 at 8:05 am

I consider something really special in this site.

#134 viagra_cheap on 05.21.19 at 1:09 pm

Hello!

#135 free fire hack version unlimited diamond on 05.21.19 at 5:26 pm

I have interest in this, danke.

#136 8 ball pool facebook on 05.22.19 at 1:17 am

A further issue is that video games are normally serious in nature with the principal focus on mastering rather than entertainment. Although, it comes with an entertainment factor to keep your young ones engaged, each game is often designed to work towards a specific set of skills or program, such as mathmatical or technology. Thanks for your publication.

#137 longUrl on 05.22.19 at 4:20 am

I believe everything composed made a lot of sense. But, what about this?

suppose you wrote a catchier post title? I am not suggesting your information is not good, however suppose you added a
title that grabbed people's attention? I mean Low-level is easy is a little plain. You could peek
at Yahoo's home page and note how they create news titles to get
viewers to click. You might try adding a video or a related picture
or two to get people interested about what you've got to
say. In my opinion, it could make your posts a little
livelier.

#138 Noticeably Skin Anti Aging Cream on 05.22.19 at 9:49 am

Truly no matter if someone doesn't understand afterward its up to other users that they will assist, so here it happens.

Leave a Comment