Hardware macroarchitecture vs mircoarchitecture

The comp-arch.net wiki defines "computer architecture" as the union of two things:

  • Macroarchitecture - the "visible" parts, the contract between hardware and software. For example, branch instructions.
  • Microarchitecture - the "invisible" parts, the implementation strategy which affects performance but not semantics. For example, branch prediction.

I think this distinction is very interesting for three reasons:

  1. It pops up everywhere. That is, many hardware problems can be addressed at the macro level or the micro level, explicitly or implicitly.
  2. The choice of macro vs micro is rarely trivial – for most problems, there are real-world examples of both kinds of solutions.
  3. The choice has common consequences across problems. The benefits and drawbacks of macro and micro are frequently similar.

I'll use examples from diverse types of hardware – CPUs, DSPs, GPUs, FPGAs, CAPPs, and even DRAM controllers. We'll discuss some example problems and how they can be solved at the macro or micro level. I'll leave the discussion of the resulting trade-offs to separate write-ups. Here, we'll go through examples to see how practical macro and micro solutions to different problems look like.

Our examples are:

  • Data parallelism: SIMD vs SIMT
  • Multiple issue: VLIW vs superscalar
  • Running ahead: exposed latencies vs OOO
  • Local storage: bare RAM vs cache
  • Streaming access: DMA vs hardware prefetchers
  • Data processing: logic synthesis vs instruction sets
  • Local communication: routing vs register addresses
  • Avoiding starvation: pressure signals vs request aging

Data parallelism: SIMD vs SIMT

Suppose you want to have a data-parallel machine: software issues one instruction that processes multiple data items.

The common macro approach is wide registers and SIMD opcodes. To use the feature, software must explicitly break up its data into 16-byte chunks, and use special opcodes like "add_16_bytes" to process the chunks.

One mirco approach is what NVIDIA marketing calls SIMT. The instruction set remains scalar. However, hw runs multiple scalar threads at once, with simultaneously running threads all executing the same instruction. That way, 16 pairs of values are added in a single cycle using scalar instructions.

(If you're interested in SIMT, a detailed comparison with SIMD as well as SMT – the more general simultaneous multithreading model – is here.)

Multiple issue: VLIW vs superscalar/OOO

Suppose you want to have a multiple issue machine. You want to simultaneously issue multiple instructions from a single thread.

The macro approach is VLIW, which stands for "very long instruction word". The idea is, those multiple instructions you issue become "one (very long) instruction", because software explicitly asks to run them together: "ADD R0, R1, R2 and MUL R3, R0, R5". Note that ADD and MUL "see" the same value of R0: MUL gets R0's value before it's modified by ADD.

VLIW also lets software choose to say, "ADD R0, R1, R2; afterwards, MUL R3, R0, R5" – that's two separate instructions yielding vanilla serial execution. This is not only slower (2 cycles instead of 1), but has a different meaning. This way, MUL does see ADD's change to R0. Either way, you get what you explicitly asked for.

(If you're interested in VLIW, an explanation of how programs map to this rather strangely-looking architecture is here.)

The micro approach, called superscalar execution, is having the hardware analyze the instructions and run them in parallel – when that doesn't change the hw/sw contract (the serial semantics). For example, ADD R0, R1, R2 can run in parallel with MUL R3, R1, R2 – but not with MUL R3, R0, R5 where MUL's input, R0, depends on ADD. Software remains unaware of instruction-level parallelism – or at least it can remain unaware and still run correctly.

Running ahead: exposed latencies vs OOO

We've just discussed issuing multiple instructions simultaneously. A related topic is issuing instructions before a previous instruction completes. Here, the macro approach is to, well, simply go ahead and issue instructions. It's the duty of software to make sure those instructions don't depend on results that are not yet available.

For example, a LOAD instruction can have a 4-cycle latency. Then if you load to R0 from R1 and at the next cycle, add R0 and R2, you will have used the old value of R0. If you want the new value, you must explicitly wait for 4 cycles, hopefully issuing some other useful instructions in the meanwhile.

The micro approach to handling latency is called OOO (out-of-order execution). Suppose you load to R0, then add R0 and R2, and then multiply R3 and R4. An OOO processor will notice that the addition's input is not yet available, proceed to the multiplication because its inputs are ready, and execute the addition once R0 is loaded (in our example, after 4 cycles). The hw/sw contract is unaffected by the fact that hardware issues instructions before a previous instruction completes.

Local storage: local memories vs caches

Suppose you want to have some RAM local to your processor, so that much of the memory operations work with this fast RAM and not the external RAM, which is increasingly becoming a bottleneck.

The macro approach is, you just add local RAM. There can be special load/store opcodes to access this RAM, or a special address range mapped to it. Either way, when software wants to use local RAM, it must explicitly ask for it – as in, char* p = (char*)0×54000, which says, "I'll use a pointer pointing to this magic address, 0×54000, which is the base address of my local RAM".

This is done on many embedded DSPs and even CPUs – for example, ARM calls this "tightly-coupled memory" and MIPS calls this "scratch pad memory".

The micro approach is caches. Software doesn't ask to use a cache – it loads from an external address as if the cache didn't exist. It's up to hardware to:

  • Check if the data is already in the cache
  • If it isn't, load it to the cache
  • Decide which cached data will be overwritten with the new data
  • If the overwritten cached data was modified, write it back to external memory before "forgetting" it

The hardware changes greatly, the hw/sw contract does not.

Data streaming: DMA vs hardware prefetchers

Suppose you want to support efficient "streaming transfers". DRAM is actually a fairly poor random access memory – there's a big latency you pay per address. However, it has excellent throughput if you load a large contiguous chunk of data. To utilize this, a processor must issue loads without waiting for results of previous loads. Load, wait, load, wait… is slow; load, load, load, load… is fast.

The macro approach is, sw tells hw that it wants to load an array. For example, a DMA – direct memory access – engine can have control registers telling it the base address and the size of an array to load. Software explicitly programs these registers and says, "load".

DMA starts loading and eventually says, "done" – for example, by setting a bit. In the meanwhile, sw does some unrelated stuff until it needs the loaded data. At this point, sw waits until the "done" bit is set, and then uses the data.

The micro approach is, software simply loads the array "as usual". Naturally, it loads from the base address, p, then from p+1, then p+2, p+3, etc. At some point, a hardware prefetcher quietly inspecting all the loads realizes that a contiguous array is being loaded. It then speculatively fetches ahead – loads large chunks beyond p+3 (hopefully not too large – we don't want to load too much unneeded data past the end of our array).

When software is about to ask for, say, p+7, its request is suddenly satisfied very quickly because the data is already in the cache. This keeps working nicely with p+8 and so on.

Data processing: logic synthesis vs instruction sets

Let's get back to basics. Suppose we want to add a bunch of numbers. How does software tell hardware to add numbers?

The micro approach is so much more common that it's the only one that springs to mind. Why, of course hardware has an ADD command, and it's implemented in hardware by some sort of circuit. There are degrees here (should there be a DIV opcode or should sw do division?) But the upshot is, there are opcodes.

However, there are architectures where software explicitly constructs data processing operations out of bit-level boolean primitives. This is famously done on FPGAs and is called "logic synthesis" – effectively software gets to build its own circuits. (This programming model is so uncommon that it isn't even called "software", but it certainly is.)

Less famously, this is also what effectively happens on associative memory processors (CAPPs/APAs) – addition is implemented as a series of bit-level masked compare & write operations. (The CAPP way results in awfully long latencies, which you're supposed to make up with throughput by processing thousands of elements in parallel. If you're interested in CAPPs, an overview is available here.)

Of course, you can simulate multiplication using bitwise operations on conventional opcode-based processors. But that would leave much of the hardware unused. On FPGAs and CAPPs, on the other hand, "building things out of bits" is how you're supposed to utilize hardware resources. You get a big heap of exposed computational primitives, and you map your design to them.

Local communication: routing vs register addresses

Another problem as basic as data processing operations is local communication: how does an operation pass its results to the next? We multiply and then add – how does the addition get the output of multiplication?

Again, the micro approach is by far the better known one. The idea is, you have registers, which are numbered somehow. We ask MUL to output to the register R5 (encoded as, say, 5). Then we ask ADD to use R5 as an input.

This actually doesn't sound as "micro" – what's implicit about it? We asked for R5 very explicitly. However, there are two sorts of "implicit" things going on here:

  • The numbers don't necessarily refer to physical registers – they don't on machines with register renaming.
  • More fundamentally, even when numbers do refer to physical registers, the routing is implicit.

How does the output of MUL travel to R5 and then to the input port of the adder? There are wires connecting these things, and multiplexers selecting between the various options. On most machines, there are also data forwarding mechanisms sending the output of MUL directly to the adder, in parallel to writing it into R5, so that ADD doesn't have to wait until R5 is written and then read back. But even on machines with explicit forwarding (and there aren't many), software doesn't see the wires and muxes – these are opaque hardware resources.

The macro approach to routing is what FPGAs do. The various computational units are connected to nearby configurable switches. By properly configuring those switches, you can send the output of one unit to another using a path going through several switches.

Of course this uses up the wires connecting the switches, and longer paths result in larger latencies. So it's not easy to efficiently connect all the units that you want using the switches and the wires that you have. In FPGAs, mapping operations to computational units and connecting between them is called "placement and routing". The "place & route" tools can run for a couple of hours given a large design.

This example as well as the previous illustrate micro vs macro at the extreme – a hardware resource that looks "all-important" in one architecture is invisible in another to the point where we forget it exists. The point is that they're equally important on both – the only question is who manages the resource, hardware or software.

Avoiding starvation: pressure signals vs request aging

One thing DRAM controllers do is accept requests from several different processors, put them in a queue, and reorder them. Reordering helps to better utilize DRAM, which, as previously mentioned, isn't that good at random access and prefers streaming access to consequent locations.

So if two processors, A and B, run in parallel, and each accesses a different region, it's frequently better to group requests together – A, A, A, A, B, B, B, B – then to process them in the order in which they arrive – say, A, B, A, A, B, A, B, B.

In fact, as long as A keeps issuing requests, it's frequently better to keep processing them until they're over, and keep B waiting. Better, that is, for throughput, as well as for A's latency – but worse for B's latency. If we don't know when to stop, serving A and starving B could make the system unusable.

When to stop? One macro solution is, the DRAM controller has incoming pressure signals, and both A and B can complain when starved by raising the pressure. Actually, this is "macro" only as far as the DRAM controller is concerned – it gives outside components explicit control over its behavior. The extent of software control over the generation of the pressure signal depends on the processors A and B.

One micro solution is to use request aging. Older requests are automatically considered more urgent. This method is implemented in many DRAM controllers – for instance, Denali's. The macro approach is implemented in the Arteris DRAM scheduler.

The micro approach is safer – the controller itself is careful to prevent starvation, whereas in the macro option, a non-cooperative processor can starve others. It also uses a simpler bus protocol, making compatibility easier for processors. However, it results in a lesser throughput – for instance, if B is a peripheral device with a large FIFO for incoming data, and can afford to wait for long periods of time until the FIFO overflows.

Whatever the benefits and drawbacks – and here, we aren't going to discuss benefits and drawbacks in any depth – this last example is supposed to illustrate that macro vs micro is relevant outside of "core"/"processor" design but extends to "non-computational" hardware as well.

Blurred boundaries

Micro vs macro is more of a continuum than a strictly binary distinction. That is, we can't always label a hardware feature as "visible" or "invisible" to programmers – rather, we can talk about the extent of its visibility.

There are basically two cases of "boundary blurring":

  • Hardware features "quite visible" even though they don't affect program semantics. These are "technically micro" but "macro in spirit".
  • Hardware features "quite invisible" even though they do affect program semantics. These are "technically macro" but "micro in spirit".

Let's briefly look at examples of both kinds of "blurring".

Technically micro but macro in spirit

A good example is memory banking. The point of banking is increasing the number of addresses that can be accessed per cycle. A single 32K memory bank lets you access a single address per cycle. 2 16K banks let you access 2 address, 4 8K banks let you access 4 addresses, and so on.

So basically "more is better". What limits the number of banks is the overhead you pay per bank, the overhead of logic figuring out the bank an address belongs to, and the fact that there's no point in accessing more data than you can process.

Now if we look at banking as implemented in NVIDIA GPU memory, TI DSP caches and superscalar CPU caches, then at first glance, they're all "micro solutions". These machines seem to mostly differ in their mapping of address to bank – for instance, NVIDIA GPUs switch banks every 4 bytes, while TI DSPs switch banks every few kilobytes.

But on all these machines, software can remain unaware of banking and run correctly. If two addresses are accessed at the same cycle that map to the same bank, then the access will take two cycles instead of one – but no fault is reported and results aren't affected. Semantically, banking is invisible.

However, I'd call GPUs' and DSPs' banking "macroish", and superscalar CPUs' banking "microish". Why?

GPUs and DSPs "advertise" banking, and commit to a consistent address mapping scheme and consistent performance implications across different devices. Vendors encourage you to know about banking so that you allocate data in ways minimizing contentions.

CPUs don't advertise banking very much, and different CPUs running the same instruction set have different banking schemes which result in different performance. Moreover, those CPU variants differ in their ability to access multiple addresses in parallel in the first place: a low-end CPU might access at most one address but a high-end CPU might access two.

GPUs and DSPs, on the other hand, have explicit multiple load-store units (a macro feature). So software knows when it attempts to accesses many addresses in parallel – one reason to "advertise" which addresses can actually be accessed in parallel.

This shows why hardware features that don't affect program semantics aren't "completely invisible to programmers" – rather, there are "degrees of visibility". A feature only affecting performance is "quite visible" if vendors and users consider it an important part of the hw/sw contract.

Technically macro but micro in spirit

SIMD and VLIW are both visible in assembly programs/binary instruction streams. However, SIMD is "much more macro in spirit" than VLIW. That's because for many programmers, the hw/sw contract isn't the semantics of assembly, which they never touch, but the semantics of their source language.

At the source code level, the effect of SIMD tends to be very visible. Automatic vectorization rarely works, so you end up using intrinsic functions and short vector data types. The effect of VLIW on source code can be close to zero. Compilers are great at automatic scheduling, and better than humans, so there's no reason to litter the code with any sort of annotations to help them. Hence, SIMD is "more macro" – more visible.

Moreover, there's "macroish VLIW" and "microish VLIW" – just like there's "macroish banking" and "microish banking" – and, again, the difference isn't in the hardware feature itself, but in the way it's treated by vendors and users.

An extreme example of "microish VLIW" is Transmeta – the native binary instruction encoding was VLIW, but the only software that was supposed to be aware of that were the vendor-supplied binary translators from x86 or other bytecode formats. VLIW was visible at the hardware level but still completely hidden from programmers by software tools.

An opposite, "macro-centric" example is TI's C6000 family. There's not one, but two "human-writable assembly languages". There's parallel assembly, where you get to manually schedule instructions. There's also linear assembly, which schedules instructions for you, but you still get to explicitly say which execution unit each instruction will use (well, almost; let's ignore the A-side/B-side issues here.)

Why provide such a "linear assembly" language? Josh Fisher, the inventor of VLIW, didn't approve of the concept in his book "Embedded Computing: a VLIW Approach".

That's because originally, one of the supposed benefits of VLIW was precisely being "micro in spirit" – the ability to hide VLIW behind an optimizing compiler meant that you could speed up existing code just by recompiling it. Not as easy as simply running old binaries on a stronger new out-of-order processor, but easy enough in many cases – and much easier to support at the hardware end.

Linear assembly basically throws these benefits out the window. You spell things in terms of C6000's execution units and opcodes, so the code can't be cross-platform. Worse, TI can't decide to add or remove execution units or registers from some of the C6000 variants and let the compiler reschedule instructions to fit the new variant. Linear assembly refers to units and registers explicitly enough to not support this flexibility – for instance, there's no silent spill code generation. Remove some of the resources, and much of the code will stop compiling.

Then why is linear assembly shipped by TI, and often recommended as the best source language for optimized code? The reason is that the code is more "readable" – if one of the things the reader is after is performance implications. The same silent spill code generation that makes C more portable makes it "less readable", performance-wise – you never can tell whether your data fits into registers or not, similarly it's hard to know how many operations of every execution unit are used.

The beauty of linear assembly is that it hides the stuff humans particularly hate to do and compilers excel at – such as register allocation and instruction scheduling – but it doesn't hide things making it easy to estimate performance – such as instruction selection and the distinction between stack and register variables. (In my opinion, the only problem with linear assembly is that it still hides a bit too much – and that people can choose to not use it. They often do – and preserve stunning unawareness of how the C6000 works for years and years.)

Personally, I believe that, contrary to original expectations, VLIW works better in "macro-centric" platforms than "micro-centric" – a viewpoint consistent with the relative success of Transmeta's chips and VLIW DSPs. Whether this view is right or wrong, the point here is that hardware features "visible" to software can be more or less visible to programmers – depending on how visible the software stack in its entirety makes them.

Implications

We've seen that "macro vs micro" is a trade-off appearing in a great many contexts in hardware design, and that typically, both types of solutions can be found in practical architectures – so it's not clear which is "better".

If there's no clear winner, what are the benefits and drawbacks of these two options? I believe that these benefits and drawbacks are similar across the many contexts where the trade-off occurs. Some of the implications were briefly mentioned in the discussion on VLIW's "extent of visibility" – roughly:

  • Micro is more compatible
  • Macro is more efficient
  • Macro is easier to implement

There are other common implications – for example, macro is harder to context-switch (I like this one because, while it's not very surprising once you think about it, it doesn't immediately spring to mind).

I plan to discuss the implications in detail sometime soon. I intend to focus, not as much on how things could be in theory, but on how they actually tend to come out and why.

128 comments ↓

#1 GD on 05.11.12 at 12:16 pm

Well, this time I am the first to comment. Nice writeup, though I was quite familiar with the source material :)
One implication which you mentioned but not explicitly in the general case, is that for a macro approach it is easier to analyze performance of code. However that indeed breaks down in the FPGA routing vs register file example where you put FPGA as macro while register file as micro. I am not so sure calling current FPGA place and route tools "macro" is correct in this context. I guess it is more technically macro, but micro in spirit ….

#2 Jaen on 05.12.12 at 5:51 am

Awesome article! Glad that you're writing up the drafts.

#3 Yura on 05.12.12 at 5:37 pm

A very interesting view angle.
One minor gripe I have with your presentation is terminology. "Macro" and "micro" have little relation to the nature of the distinction you talk about — that is, visibility of low-level considerations to the programmer at the higher level. When reading, I had trouble remembering which is which. I'd personally prefer terms like "transparent" and "opaque" or something like that.

#4 Adrian on 05.12.12 at 6:28 pm

Fabulous post! Thanks!

#5 Yossi Kreinin on 05.12.12 at 9:57 pm

Glad some people liked it – it came out long and (therefore) somewhat unfinished/"to be continued", so I had my doubts about it…

@Yura: I borrowed the terms from comp-arch.net; microarchitecture/uarch is a standard term meaning more or less what I used it here for, and "macro" sounds complementary to it… If it feels unintuitive to you, I guess there isn't much I can do to convince you of the opposite :) – I do like these terms myself though because it feels intuitive to me. For instance, "macro-heavy" architectures tend to feel "bulkier", made of bigger parts – as in, large SIMD registers, VLIW instructions glue together into "very large words", etc. I used "micro" and "macro" in my own conversations before finding them at comp-arch.net, and I was pleased to find them there because it hinted that they are in fact intuitive if not very standard – which, however, turns out debatable given your remark…

#6 Nick Zivkovic on 05.17.12 at 9:00 am

Firstly: excellent article! I, quite literally, didn't know that such a distinction existed, for hardware. There is a very similar distinction in software such as kernels, systems-level libraries, or daemons/services. For example, the slab allocators (used in most kernels) are "macro", in that they require that the user know the size of an object when allocating _and_ freeing. Or the cache in which an object resides, if it has a cache. Whereas any malloc implementation is "micro", in that it has the same interface it ever did, but the internals change, over the generations, often incorporating ideas from slab allocators, buddy-systems, etc.

Secondly: Not to be pedantic, but in the following sentence:

"The micro approach is safer – the controller itself is careful to prevent starvation, whereas in the __micro__ option, a non-cooperative processor can starve others."

Shouldn't the second occurrence of "micro" be "macro"?

i.e.:

"The micro approach is safer – the controller itself is careful to prevent starvation, whereas in the __macro__ option, a non-cooperative processor can starve others."

#7 Yossi Kreinin on 05.17.12 at 11:18 pm

Thanks – fixed the typo

#8 Jussi Santti on 06.05.12 at 8:23 am

On similar grounds as Yura I would like to reminder about one more choice terms: "abstract" instead of macro and "concrete" or "hidden" instead of micro. Yes I know they do not fit HW tradition, but make a connection to SW and math. The engineers better know what "abstract" means.

The programmers better know it too. That is why software term API is re-spelled as "abstract program interface" by some SW practitioners and theoreticians.

#9 sed on 06.07.12 at 12:08 pm

s/mirco/micro/, twice…

#10 paladins aimbot on 05.15.19 at 3:32 pm

Just wanna input on few general things, The website layout is perfect, the articles is very superb : D.

#11 fortnite aimbot download on 05.16.19 at 12:34 pm

Just wanna input on few general things, The website layout is perfect, the articles is very superb : D.

#12 aimbot fortnite on 05.16.19 at 4:29 pm

I like this website its a master peace ! Glad I found this on google .

#13 nonsense diamond key on 05.17.19 at 6:42 am

I have interest in this, thanks.

#14 Mavis Soper on 05.17.19 at 8:58 am

yosefk.com does it again! Very thoughtful site and a well-written post. Thanks!

#15 fallout 76 cheats on 05.17.19 at 10:09 am

I really got into this article. I found it to be interesting and loaded with unique points of interest.

#16 red dead redemption 2 digital key resale on 05.17.19 at 3:20 pm

I truly enjoy looking through on this web site , it holds superb content .

#17 redline v3.0 on 05.17.19 at 6:23 pm

I dugg some of you post as I thought they were very beneficial invaluable

#18 badoo superpowers free on 05.18.19 at 7:48 am

Thank You for this.

#19 forza horizon 4 license key on 05.18.19 at 2:41 pm

I simply must tell you that you have an excellent and unique site that I must say enjoyed reading.

#20 mining simulator codes 2019 on 05.19.19 at 6:40 am

google took me here. Cheers!

#21 smutstone on 05.20.19 at 11:20 am

Enjoyed reading through this, very good stuff, thankyou .

#22 redline v3.0 on 05.21.19 at 6:48 am

google took me here. Thanks!

#23 free fire hack version unlimited diamond on 05.21.19 at 4:03 pm

Great, yahoo took me stright here. thanks btw for info. Cheers!

#24 nonsense diamond on 05.22.19 at 5:53 pm

I like this website its a master peace ! Glad I found this on google .

#25 krunker aimbot on 05.23.19 at 6:11 am

I am glad to be one of the visitors on this great website (:, appreciate it for posting .

#26 bitcoin adder v.1.3.00 free download on 05.23.19 at 9:49 am

I simply must tell you that you have an excellent and unique web that I kinda enjoyed reading.

#27 vn hax on 05.23.19 at 6:34 pm

Respect to website author , some wonderful entropy.

#28 eternity.cc v9 on 05.24.19 at 7:22 am

I love reading through and I believe this website got some genuinely utilitarian stuff on it! .

#29 cheats for hempire game on 05.26.19 at 6:15 am

Very interesting points you have remarked, appreciate it for putting up.

#30 iobit uninstaller 7.5 key on 05.26.19 at 9:01 am

Great writing to see, glad that duckduck brought me here, Keep Up good job

#31 smart defrag 6.2 serial key on 05.26.19 at 3:21 pm

Your website has proven useful to me.

#32 resetter epson l1110 on 05.26.19 at 5:56 pm

Some truly interesting content on this web site , appreciate it for contribution.

#33 sims 4 seasons free code on 05.27.19 at 7:13 am

Some truly wonderful posts on this web site , appreciate it for contribution.

#34 rust hacks on 05.27.19 at 7:47 pm

I simply must tell you that you have an excellent and unique web that I really enjoyed reading.

#35 gamefly free trial on 05.28.19 at 7:16 am

Hello there! I know this is kinda off topic nevertheless I'd figured I'd ask.
Would you be interested in trading links or maybe
guest writing a blog article or vice-versa? My site discusses a lot of the same
subjects as yours and I think we could greatly benefit from each
other. If you happen to be interested feel free to shoot me an e-mail.

I look forward to hearing from you! Great blog
by the way!

#36 strucid hacks on 05.28.19 at 10:05 am

Great, google took me stright here. thanks btw for this. Cheers!

#37 expressvpn key on 05.28.19 at 7:08 pm

Enjoyed examining this, very good stuff, thanks .

#38 ispoofer pokemon go license key on 05.29.19 at 8:20 am

Intresting, will come back here later too.

#39 aimbot free download fortnite on 05.29.19 at 12:19 pm

I am glad to be one of the visitors on this great website (:, appreciate it for posting .

#40 redline v3.0 on 05.29.19 at 4:46 pm

Hey, glad that i found on this in yahoo. Thanks!

#41 gamefly free trial on 05.30.19 at 5:53 am

Having read this I thought it was rather informative.
I appreciate you finding the time and energy to put this
information together. I once again find myself spending a significant amount of time both reading and commenting.
But so what, it was still worth it!

#42 vn hax on 05.30.19 at 5:58 am

I am not rattling great with English but I get hold this really easygoing to read .

#43 Solange Djuric on 05.30.19 at 5:03 pm

Thanks for the great site you've set up at yosefk.com. Your enthusiastic take on the subject is definitely contagious. Thanks again!

#44 how to get help in windows 10 on 05.31.19 at 9:31 am

Awesome! Its actually remarkable article, I have got much clear idea on the topic of from this
article.

#45 xbox one mods free download on 05.31.19 at 12:32 pm

Great, bing took me stright here. thanks btw for post. Cheers!

#46 fortnite aimbot download on 05.31.19 at 3:17 pm

I am not rattling great with English but I get hold this really easygoing to read .

#47 gamefly free trial on 05.31.19 at 4:41 pm

I'm not positive the place you are getting your
info, but great topic. I needs to spend some time finding out more or working out
more. Thanks for fantastic information I used to be looking for this information for my mission.

#48 mpl pro on 06.01.19 at 6:17 pm

Enjoyed examining this, very good stuff, thanks .

#49 gamefly free trial on 06.02.19 at 1:02 am

Very nice post. I just stumbled upon your blog and
wanted to say that I've truly enjoyed browsing your blog posts.
In any case I will be subscribing to your rss feed and I hope you write again soon!

#50 hacks counter blox script on 06.02.19 at 6:22 am

Respect to website author , some wonderful entropy.

#51 gamefly free trial on 06.02.19 at 5:39 pm

What's up, for all time i used to check blog posts here early
in the break of day, as i enjoy to gain knowledge
of more and more.

#52 gamefly free trial on 06.02.19 at 8:55 pm

Wow, marvelous blog layout! How lengthy have you been running a blog
for? you made blogging look easy. The entire glance
of your site is fantastic, as well as the content!

#53 protosmasher download on 06.03.19 at 10:12 am

very Great post, i actually like this web site, carry on it

#54 gamefly free trial on 06.04.19 at 2:57 am

Hola! I've been reading your site for some time now and finally got the bravery to go ahead and
give you a shout out from Huffman Texas! Just wanted to say keep up the fantastic job!

#55 gamefly free trial on 06.04.19 at 3:06 pm

What a stuff of un-ambiguity and preserveness of valuable experience about unexpected emotions.

#56 gamefly free trial on 06.05.19 at 6:37 am

Asking questions are in fact pleasant thing if you are
not understanding anything fully, except this article gives nice understanding yet.

#57 gamefly free trial on 06.06.19 at 8:04 pm

I like the valuable info you provide in your articles.
I'll bookmark your weblog and check again here frequently.
I am quite certain I'll learn lots of new stuff right here!
Good luck for the next!

#58 Edwin Midgett on 06.07.19 at 6:45 am

Excellent read. I just now sent this on 6/6/2019 to a fellow student who has been involved in a little research of his own on this subject. To say thanks, he just bought me dinner! So, I should probably say: Cheers for the meal!

#59 gamefly free trial on 06.08.19 at 4:51 am

I am sure this post has touched all the internet viewers, its really really fastidious post on building
up new web site.

#60 best ps4 games 2019 on 06.08.19 at 9:20 am

Magnificent web site. Lots of helpful info here.

I am sending it to several buddies ans also sharing
in delicious. And certainly, thank you for your sweat!

#61 http://tinyurl.com on 06.09.19 at 11:23 am

Hello there! Do you know if they make any plugins to assist with SEO?
I'm trying to get my blog to rank for some targeted keywords but I'm not seeing very good
success. If you know of any please share. Cheers!

#62 http://tinyurl.com/y25qeqxw on 06.10.19 at 12:47 am

I'm gone to tell my little brother, that he should also go to see this weblog
on regular basis to obtain updated from most recent reports.

#63 gamefly free trial 2019 coupon on 06.10.19 at 2:25 pm

If you want to increase your know-how just keep visiting
this web site and be updated with the most up-to-date information posted here.

#64 http://tinyurl.com/yxnwx792 on 06.12.19 at 12:46 pm

Hey there this is somewhat of off topic but I was wanting
to know if blogs use WYSIWYG editors or if you have to manually code
with HTML. I'm starting a blog soon but have no coding knowledge so I wanted to get guidance from someone with experience.
Any help would be enormously appreciated!

#65 ps4 best games ever made 2019 on 06.12.19 at 6:46 pm

I like what you guys are up too. Such clever work and exposure!
Keep up the terrific works guys I've added you guys to my own blogroll.

#66 quest bars cheap on 06.14.19 at 12:58 pm

I always emailed this web site post page to all my associates, for the reason that if like to read it
after that my links will too.

#67 quest bars cheap on 06.15.19 at 2:53 am

Today, I went to the beachfront with my kids. I found a sea shell and
gave it to my 4 year old daughter and said "You can hear the ocean if you put this to your ear." She put the shell to her ear
and screamed. There was a hermit crab inside and it pinched her ear.
She never wants to go back! LoL I know this is totally off topic but I had to tell someone!

#68 roblox script executor on 06.16.19 at 11:39 pm

Me like, will read more. Cheers!

#69 proxo key on 06.19.19 at 9:32 am

I was looking at some of your articles on this site and I believe this internet site is really instructive! Keep on posting .

#70 vn hax on 06.20.19 at 6:26 pm

Hi, here from yahoo, me enjoyng this, will come back again.

#71 nonsense diamond on 06.21.19 at 7:39 am

I simply must tell you that you have an excellent and unique site that I must say enjoyed reading.

#72 plenty of fish dating site on 06.21.19 at 10:06 pm

Hello! I've been following your weblog for a long time now
and finally got the bravery to go ahead and give you a shout out from New
Caney Tx! Just wanted to mention keep up the excellent work!

#73 star valor cheats on 06.23.19 at 5:06 pm

I conceive this web site holds some real superb information for everyone : D.

#74 gmod hacks on 06.24.19 at 3:11 pm

Ha, here from bing, this is what i was looking for.

#75 Explain Like I’m Five on 06.25.19 at 5:01 am

I like this website its a master peace ! Glad I found this on google .

#76 qureka pro apk on 06.25.19 at 7:51 pm

This is interesting!

#77 krunker aimbot on 06.26.19 at 6:30 am

You got yourself a new follower.

#78 ispoofer key on 06.27.19 at 5:57 am

Thanks for this article. I definitely agree with what you are saying.

#79 synapse x serial key on 06.27.19 at 8:45 pm

stays on topic and states valid points. Thank you.

#80 strucid aimbot on 06.28.19 at 7:19 am

Awesome, this is what I was searching for in yahoo

#81 advanced systemcare 11.5 on 06.28.19 at 1:25 pm

Ni hao, here from google, me enjoyng this, will come back again.

#82 zee 5 hack on 06.29.19 at 8:29 am

Thank You for this.

#83 cryptotab script hack free download on 06.29.19 at 2:50 pm

Enjoyed examining this, very good stuff, thanks .

#84 roblox generator no verification on 07.01.19 at 9:03 am

I have interest in this, xexe.

#85 http://tinyurl.com/y3gqxgyr on 07.01.19 at 7:45 pm

That is very attention-grabbing, You're an overly professional blogger.
I have joined your rss feed and look ahead to in the hunt for more of your magnificent
post. Additionally, I've shared your website in my social networks

#86 cheat fortnite download no virus on 07.01.19 at 7:48 pm

stays on topic and states valid points. Thank you.

#87 escape from tarkov cheats and hacks on 07.02.19 at 7:42 am

Awesome, this is what I was searching for in yahoo

#88 skin swapper on 07.02.19 at 1:13 pm

stays on topic and states valid points. Thank you.

#89 vn hax on 07.03.19 at 7:21 am

I dugg some of you post as I thought they were very beneficial invaluable

#90 cyberhackid on 07.03.19 at 7:17 pm

I conceive this web site holds some real superb information for everyone : D.

#91 prison life hacks on 07.04.19 at 7:18 am

Great read to see, glad that yandex took me here, Keep Up great Work

#92 why you should care about seo on 07.04.19 at 3:19 pm

Parasite backlink SEO works well :)

#93 subbot on 07.04.19 at 7:04 pm

Intresting, will come back here again.

#94 dego hack on 07.05.19 at 7:18 am

I have interest in this, xexe.

#95 tom clancy's the division hacks on 07.05.19 at 7:33 pm

I conceive this web site holds some real superb information for everyone : D.

#96 synapse x free on 07.06.19 at 6:52 am

Thank You for this.

#97 gx tool apk pubg uc download on 07.06.19 at 11:05 am

Hello, google lead me here, keep up good work.

#98 rekordbox torrent on 07.06.19 at 10:15 pm

I must say, as a lot as I enjoyed reading what you had to say, I couldnt help but lose interest after a while.

#99 black ops 4 license key free on 07.07.19 at 8:33 am

I was looking at some of your articles on this site and I believe this internet site is really instructive! Keep on posting .

#100 spyhunter 5.4.2.101 key on 07.08.19 at 8:37 am

I love reading through and I believe this website got some genuinely utilitarian stuff on it! .

#101 quest bars cheap 2019 coupon on 07.09.19 at 10:03 am

First off I would like to say excellent blog! I had a quick question that I'd like
to ask if you don't mind. I was curious to find
out how you center yourself and clear your mind before writing.

I've had a difficult time clearing my mind in getting my ideas out.
I truly do enjoy writing but it just seems like the first 10
to 15 minutes are usually lost simply just trying to figure out how to begin. Any suggestions or tips?
Thanks!

#102 roblox fps unlocker download on 07.09.19 at 10:13 am

This does interest me

#103 angie_koks on 07.14.19 at 2:08 am

Thank you for the great read!

#104 sex chat cams on 07.15.19 at 2:23 am

some great ideas this gave me!

#105 legal porno on 07.16.19 at 12:00 am

great advice you give

#106 legalporn on 07.16.19 at 12:40 am

great advice you give

#107 how to get help in windows 10 on 07.16.19 at 3:15 pm

Hey there would you mind letting me know which webhost you're working with?
I've loaded your blog in 3 different browsers and I must
say this blog loads a lot quicker then most.
Can you recommend a good internet hosting provider at a honest price?

Thanks, I appreciate it!

#108 Jere Heagle on 07.17.19 at 2:52 pm

Mr.s Fister, this message is your next piece of info. Feel free to message the agency at your earliest convenience. No further information until next transmission. This is broadcast #6685. Do not delete.

#109 how to get help in windows 10 on 07.18.19 at 2:13 am

This is very interesting, You are a very skilled blogger.
I have joined your rss feed and look forward to seeking more of your magnificent post.
Also, I've shared your site in my social
networks!

#110 plenty of fish dating site on 07.18.19 at 3:29 pm

Fantastic beat ! I wish to apprentice even as you amend your web site, how
could i subscribe for a weblog site? The account aided me a acceptable deal.
I have been a little bit familiar of this your broadcast provided
vibrant transparent concept

#111 ines_lenvin on 07.19.19 at 1:47 am

thx you so much for posting this!

#112 sandra_luberc on 07.19.19 at 2:16 am

you are so great

#113 buydrugonline on 07.19.19 at 2:52 am

This blog is amazing! Thank you.

#114 plenty of fish dating site on 07.19.19 at 4:01 pm

Excellent blog post. I certainly appreciate this website.
Stick with it!

#115 how to get help in windows 10 on 07.20.19 at 5:40 pm

naturally like your web site but you need to take a look at the spelling on quite a few of your posts.
Many of them are rife with spelling problems and
I find it very troublesome to tell the reality nevertheless I will
definitely come back again.

#116 how to get help in windows 10 on 07.21.19 at 9:40 am

If some one wants to be updated with newest technologies then he must
be go to see this web site and be up to date daily.

#117 prodigy hacked on 07.21.19 at 2:55 pm

I have interest in this, cheers.

#118 natalielise on 07.22.19 at 6:02 pm

Excellent blog here! Also your website loads up very fast!
What host are you using? Can I get your affiliate link to your
host? I wish my website loaded up as quickly as yours lol pof natalielise

#119 acidswapper on 07.23.19 at 12:56 pm

Enjoyed examining this, very good stuff, thanks .

#120 date cougarr on 07.23.19 at 10:42 pm

I am 43 years old and a mother this helped me!

#121 xate cougar on 07.23.19 at 11:00 pm

I am 43 years old and a mother this helped me!

#122 date coujgar on 07.23.19 at 11:56 pm

I am 43 years old and a mother this helped me!

#123 plenty of fish dating site on 07.24.19 at 1:23 am

Since the admin of this site is working, no uncertainty very shortly it will be famous,
due to its quality contents.

#124 pphud download on 07.24.19 at 1:18 pm

Morning, i really think i will be back to your website

#125 ezfrags on 07.25.19 at 3:08 pm

Great read to Read, glad that bing led me here, Keep Up good Work

#126 plenty of fish dating site on 07.25.19 at 10:43 pm

I was curious if you ever thought of changing the structure of your
website? Its very well written; I love what youve got to say.
But maybe you could a little more in the way of content so people
could connect with it better. Youve got an awful lot
of text for only having one or two images. Maybe you could
space it out better?

#127 Digital Marketing on 07.26.19 at 4:11 am

After looking over a few of the blog articles on your site, I really like your technique of blogging. I saved it to my bookmark webpage list and will be checking back soon. Please check out my website as well and let me know what you think.

#128 ezfrags on 07.26.19 at 4:15 pm

Enjoyed reading through this, very good stuff, thankyou .

Leave a Comment