An unusual hardware architecture: APA (Associative Processing Array)

We're living in the golden age of hardware design – it probably won't get any cheaper to make a new chip. Yes, there's the downside that it's harder to be wildly incompatible with everything than in the old days – there's plenty of standard interfaces to support. On the other hand, it's a benefit, not just a drawback – make a chip with a CPU that runs C, an Ethernet controller and a DRAM interface, and it's now usable with plenty of software and hardware developed by others.

And then for the wilder, "incompatible" innovations – if you really want to invent interfaces and not just implementations – there's the "accelerator" realm: an MPEG decoder, a DSP, a GPU. A decade ago, they said that "standard" microprocessors killed all high-performance architectures. Today, a system that kept the "fastest computer in the world" title for almost a year is based on GPUs – and "non-standard"/"incompatible" accelerators are everywhere (not to mention SIMD instruction set extensions right inside the otherwise "compatible"/"commodity" CPUs).

Still, while it indeed became way cheaper to make your own chips since you don't have to erect your own fab to do that, "cheap" still means costs somewhere in the millions – not quite what the word "cheap" brings to mind. And mistakes still can't be corrected, not really. Which results, among other things, in an understandable risk aversion with respect to design.

A language geek, who's naturally curious about programming languages that don't look mainstream (C with classes), is going to meet many such languages, and plenty of implementations to play with. A hardware geek, who's equally curious about designs that don't look mainstream (RISC/VLIW with bells and whistles), is going to find few such designs and very few implementations. Risk aversion is innovation aversion.

APA thus combines two traits that are rare – it gets at least 9 out of 10 in the "non-mainstream" category, and implementations were actually manufactured and shipped, specifically, in early smartphones by NeoMagic (BDTI and EETimes articles are some of the architecture overviews). The phones apparently weren't very successful, but it seems a poor measure of the architecture's merit in this case, just because there are so many ways to fail regardless of your accelerator quality. So the platform's demise gives us not evidence but a lack of evidence (the platform wasn't widely targeted by 3rd party developers whose opinions could be interesting).

Overview

APA stands for Associative Processing Array, a kind of content-addressable memory. Just memory. How do you compute using just memory? APA (Associative Processing Array)The operations are done right there, near the cells where your bits are kept. No need to read your data word by word into a processor, than write it back – rather, your operations run in parallel, inside the memory, on every data word! SIMD operations on steroids.

What operations? Bitwise operations – masked compare and masked write:

  • Compare: tag[i] = (word[i]&mask)==(compare_data&mask)
  • Write: if(tag[i]) word[i]=(word[i]&mask) | (write_data&~mask)

…where a tag bit is kept for every data word (row), and mask & compare/write data are broadcasted to all rows. You can also move words – either to the adjacent rows or to rows at distance 8 (or, say, 16, depending on the hardware configuration – but not at an arbitrary distance). And you can move the data in or out, sequentially.

Where's the code – who decides what to write, compare and move in what order? One option is to use a low-end CPU running "normal" code to control the contents of the compare/write data & mask registers, to issue APA operations and to interact with the host system (the "real" CPU and external memory). This little processor isn't "accelerating" computations by itself, just controls the APA accelerator hardware.

That's it. No add, no multiply, no nothing. How's that for non-mainstream?

So, how do you actually do anything with this SIMD-on-steroids machine – say, add two vectors of numbers? Well, you need to have both vectors in the array – each element pair occupying some of the bits of a row (for instance, the NeoMagic 512-row APA had 160 bits per row, so for 16 bit vectors, you'd use 32 bits out of 160). Then you perform the addition bit by bit, using masked compare and write operations.

That's right, addition done entirely in software. Latency: 48 cycles for 16 bits, throughput: >10 16b additions/cycle for NeoMagic's 512-row APA. More latency for 32 bits, less for 8 bits, still less for 3 bits. Which means optimization opportunities you're not used to having in software (you gain no speed-up using only 3 bits of a 32b CPU register for addition). On the other hand, it means large penalties for high precision (floating point takes thousands of cycles).

Tools and libraries would be supplied with an APA implementation, but the exciting thought for a programmer is to be able to bypass them where needed and get down to bit-level manipulation! (The exciting thought for a decision maker would be to always use the tools, never pay someone for the bit fiddling; well, different people get their excitement from different things.)

Pros & cons

In a way, it's an exceedingly generic and elegant architecture, and a very impressive one. One "gets it" like one doesn't get any other. Just try to describe any other sort of programmable machine to a level of detail sufficient for accurate predictions about performance.

Surprisingly – or, perhaps, unsurprisingly given just how unusual this machine is – I simultaneously also don't get it like any other. I'm not used to thinking of computations that way, both in terms of precision and in terms of data access. I have no idea what share of my 8-bit numbers only have just 5 or 2 bits and it can matter a lot here.

Likewise, even for algorithms that don't require random access – and APA is simply not good for random access – I have little idea of just how non-random my access is. That is, similarly to the precision case where constants I'm not used to care about matter, there's a difference between accessing a row at distance 1 and a row at distance 5, and I don't know how frequent different distances between adjacent things are in a code base I care about.

So while in the abstract, a shorter than ever architecture description is sufficient here for accurate performance predictions, I lack intuition and experience that would make such predictions easy.

A great thing about the APA design is its "hardware friendliness". The typical processor design involves a truckload of convoluted circuits that are likeable for their function, but unimaginable to the human mind as physical objects – so cumbersome, infinitely configurable, finicky tools are used to translate a functional description of said circuits to actual electric circuits occupying actual space.

The typical hardware design is not hardware friendly (sounds strange – but the average piece of software isn't very considerate towards its target processor and software stack, either). The APA, on the other hand, has a natural spatial mapping with its 2 dimensions (row bit width and the number of rows) and regular connections between just the adjacent or relatively close rows.

Again, surprisingly – or unsurprisingly given how unusual the APA is – this is also a drawback, in the sense that the standard hardware design tools will do a very poor job implementing APA. For that matter, standard tools will likely do an even worse job implementing plain RAM – which also has a natural spatial mapping and regular structure.

RAM is analog design, not digital: rather than describing it on a functional level, someone implements it as a physical circuit description. Basically, RAM is a library for digital tools, like flip-flops or simple gates – a library that can't be implemented using the tool itself.

I really appreciate the von Neumann architecture, but it's scary to realize just how strong the von Neumann grip really is. RAM is basically the only interface to a large array of bits that is relatively easily accessible to a hardware designer. (Not that it's very easily accessible, mind you – there is no portable interface for RAM, believe it or not, but no matter how you build your chips, you'll be able to get some sort of RAM).

What happens if you want your own bag of bits instead of RAM? You can do what they call full custom design – make your own physical circuit description. This isn't "portable", where "porting" means changing the manufacturing process – either to move from 65nm to 40nm, or from one manufacturer to another. It's also a longer and more complicated process. But it's certainly doable, especially if you outsource it to the inventors, as you'd have to do anyway because the thing is patented.

The hostility to hardware design tools (despite the friendliness to the basic constraints of hardware) and patent protection make APA more of a possible off-the-shelf solution than inspiring source of ideas, and so does the design simplicity. Unlike, say, VLIW, which is a design style with basic ideas in the public domain and countless possible variations and extensions (starting with the "let's add this one spiffy instruction" variety), APA is more or less complete. There are important constants to tweak – row width and the distance to easily accessible neighbors – but while it could be a hard choice to make, it's not very creative. (I do believe it could be a source of ideas if only through expanding one's horizons, which is why I write about it.)

If we attempt to discuss the efficiency of APA qualitatively, in terms of hardware resource utilization rather than throughput per mm^2 or per mW for a given app, three things come to mind:

  • Benefit - a perennial bottleneck of accessing memory through a bus very narrow compared to the amount of bits stored is eliminated. This "ought to be good" for algorithms where access is "far from random" since these gain nothing from conventional RAM's flexibility but pay the full price of its low throughput.
  • Drawback - the cycle is "wasted": values are read from flip-flops, undergo very few transformations and are written back. This "ought to be bad" because it won't translate to high frequency (you can't read then write a flip-flop very fast – and doing this with them all at once dissipates power, so in fact low frequencies enabled by parallelism, not high frequencies enabled by circuit simplicity are APA's potential advantage in the frequency department). A competing design running at a similar frequency, however, will do much more per cycle (like actual addition) because circuits implementing combinatorial logic are fast. The question thus becomes how big those circuits are compared to flip-flops: if enough APA rows can be packed instead, the throughput will still be competitive despite the abysmal latency – provided that you simultaneously process sufficiently large amounts of data.
  • Benefit - ability to use DRAM cells. This "ought to be good" since DRAM cells are much smaller than normal flip-flops (which is achieved through having them leak their charge so they have to be periodically refreshed). AFAIK, nobody uses them for computation directly because they don't fit in traditional hardware models – neither as registers (that's plain dumb) nor as local RAM (they have poor latency and imply a cumbersome controller to access as a RAM). An APA, on the other hand, could potentially run well on DRAM cells if the required simple circuitry were implemented near the cells. One problem with this is that custom chips may be easy to make these days, but not custom DRAM chips. In particular, Andy Glew, formerly from Intel then AMD, said at some place that it's "hard to influence DRAM makers", and if it's hard for Intel or AMD, well, it's probably hard in general.

Overall, I have a lot of reservations here – not just because of the way originality by itself seems to complicate matters here (I'd hate to admit it as the only reason…), but because it's a step in the opposite direction to what successful SIMD systems are doing. That is, you get high throughput given unusually low precision and unusually restricted data access patterns. A DSP from the late 90s would attempt to process numbers with more bits and would let you fetch them from more places. A GPU from the 2000s still more so – floating point numbers, parallel random access with transparent contention handling (at least in CUDA GPUs). It seems that the direction is removing restrictions on the set of things you run fast, not very high throughput for a very restricted set.

On the other hand, there exist algorithms with low precision and high locality. And it's exciting to see an architecture which is not RAM-based, naturally represented in space, etc. – all those things which get people excited in hardware discussions – but practical enough for a real world delivery, and with some things connecting it to more usual programming models (for instance, SIMD commands broadcasted from a CPU instead of clever local rules you have no idea how to come up with as in cellular automata). So it's definitely very interesting.

Update: as a better informed commenter pointed out, APA is also known as CAPP – Content Addressable Parallel Processor, and implementations date as early as 1972 (which makes you wonder what could legitimately remain outside the public domain by now). This seems an evidence of having failed the test of time – or having some really deep trouble with commercialization. I'd be curious to hear a software developer's experience with this – BDTI's article from 2003 talked about the development experience in hypothetical terms and entirely in the future tense, whereas some past evidence has to be available.

134 comments ↓

#1 Bryce on 08.22.11 at 5:15 pm

Pretty interesting stuff, as usual. You could use this to replace 80% of SIMD instructions with a much smaller resistor count. Hopefully this will happen some day though it doesn't really benefit intel much to make standards that skimp on resistor count.

#2 Kragen Javier Sitaker on 08.22.11 at 8:38 pm

"it probably won’t get any cheaper to make a new chip" — but it was cheaper when MOSIS was founded, wasn't it? Around 1980? You can still get chips fabbed through MOSIS and similar services, and it only costs a few thousand dollars. Designing the chip may cost more than a few thousand dollars, but it's still within reach of the occasional middle-class US hobbyist: tens of thousands, maybe, but not millions.

At least in theory. Chuck Moore is the only one I know of who ever succeeded.

#3 Yossi Kreinin on 08.22.11 at 10:46 pm

@Bryce: I think the market is open enough to make any particular large company unable to prevent any sort of competition; the market may have sufficient inertia (it's inherently cheaper to keep doing what you're doing than switching to something else) to prevent some sorts of innovation without any sort of "sabotage" by anyone. BTW Intel buys loads of IPs and it would most certainly buy the APA patents if it figured it was worth it.

As to "resistor count" ("per performance") – I don't know, really, as it's very much a matter of constants – many of them – depending on what you do and how you do it; could reduce or increase that count as far as I can tell without (much) additional info and experiments.

@Kragen: I think you'll almost certainly reach millions if you want a marketable product, things you'll have to spend heavily on including IPs such as DRAM, Ethernet, etc. controllers (and, very likely, CPU and on-chip interconnect); EDA tools for optimizing implementation; and verification – both tools and man hours. Also, if you count time spent – a million dollars is what, ten man years? Not that much time.

Chuck Moore's chips don't appear very marketable so currently don't quite disprove this point; or rather, they aren't a huge hit – but we (or at least I) don't know that any generation was particularly cheap to make overall, either.

#4 Øystein Gran Larsen on 08.22.11 at 10:54 pm

Want to read more? My favorite, although old, is Caxton C Foster's "Content Addressable Parallel Processors" published by Van Nostrand Reinhold in 1976.

#5 Yossi Kreinin on 08.22.11 at 11:12 pm

@Øystein Gran Larsen: thanks!

#6 maht on 08.23.11 at 1:01 am

I also found "Content Addressable Parallel Processor" by Lambert M. Surhone

but Foster's was only $10 so I bought that :)

#7 Z.T. on 08.23.11 at 7:26 am

"there are so many ways to fail irregardless of your accelerator quality" – s/irregardless/regardless/

A real world example of an algorithm well suited to implementation on such devices and how much better (throughput and latency per mm and per watt) it would run on such a device compared to an Intel chip or an ARM chip manufactured using the same process technology is what's really lacking for me to appreciate this idea. All I can think of is how weird it would be to program.

#8 Yossi Kreinin on 08.23.11 at 10:29 pm

@Z.T.: well, image convolution with a small fixed-sized filter is a good example; >10 additions/cycle is also not bad at all for an accelerator from 2003 (ARM had no Neon back then). As to how much better things would be in terms of performance per cost metric – well, it's the numbers everyone wants to lay their hands on when comparing architectures and nobody can because you get drowned in details – different algorithm implementations will have different precision characteristics on different machines, will be optimized to a different extent due to programmer performance differences, and the chips won't be comparable because of being manufactured using a different process and under different constraints. This means you have to gather statistics carefully for your case and then rely on gut feeling and general attraction/aversion towards an architecture – one reason the best performing machines aren't necessarily the most popular; anyway, I figured spitting out poorly supported numbers would be worse than nothing, so I didn't.

#9 nick black on 08.23.11 at 10:40 pm

what an absolutely outstanding writeup! thank you, yossi!

#10 Yossi Kreinin on 08.23.11 at 10:53 pm

@nick: thanks! :)

#11 gus3 on 08.26.11 at 1:46 pm

I heard about ten years ago, of a memory-mapped PCI card that did near-instantaneous encryption. (It was probably DES, but it might have been RSA.) A program wrote parameters (key, algo, and enc/dec operation) to the card, and then the fun began. As soon as a block of input data was finished transferring to the card, the block of output data was ready to be read back by the program. Being PCI, read/write via DMA was also supported, so however much data the bus could transfer, was how fast the card could operate.

I don't remember many specific details about the card, except that the developer supplied drivers only for Windows NT, with a Linux driver in development at the time.

Could this card have been using an architecture similar to APA/CAPP? I mean, would APA/CAPP lend itself to a task like this?

#12 Yossi Kreinin on 08.27.11 at 5:28 am

@gus3: whether you could encrypt efficiently with an APA I don't know since I'm not familiar with the different encryption algorithms enough to envision their different data flows; however, if you saw low latency, it rules out APA as well as any other high-throughput, high-latency design. Generally, anything that can be done to a stream of data without keeping much state – roughly O(1) rather than O(N) state, assuming N is large or unbound – is very naturally handled by non-programmable hardware with a deep pipeline, where you have a piece of hardware dedicated to each processing phase and inputs flow into the first box and a few cycles later, outputs flow out of the the last box. The APA, on other hand, wants to have a large chunk of the stream put into it, then processes the entire chunk and then spits it out. (Of course the "large" chunk could be quite small in absolute terms – it's just not the typical way people parallelize things in hardware when they make fixed-function stream-processing boxes, it's more usual to see stage 1 run in parallel with stage 2, 3, … N than to see stage 1 run in parallel on inputs 1..K). So this is how it worked if I had to bet.

#13 Zee on 09.01.11 at 4:16 am

Hi.

You've promised to make new section in your FQA when С++0x is going to be accepted.
Hope you didn't change your mind =)

#14 Yossi Kreinin on 09.01.11 at 8:43 am

I do intend to write about it, however, my writing style has since changed, so it's going to be a bit of a different experience.

#15 Yossi Kreinin on 10.19.11 at 12:03 am

@Vladimir: Sure; your friend can reach me at Yossi.Kreinin@gmail.com :) – the same address listed everywhere on the site, and it is quite functional – or you can give me an e-mail address I could contact first instead. Comment deleted…

#16 Vladimir on 10.19.11 at 2:09 pm

Hi Yossi, I was indeed being dyslexic — repeatedly I misspelled your email address!

Anyhow, I sent you and Ed an e-mail to introduce you to each other. Hopefully you got it.

#17 Ray McConnell on 04.16.12 at 3:32 am

Been there, done that. Not recommended.

The ratio of memory storage vs computational hardware is critical. Even though one can simpliy increase the parallelism, in the end, for any given algorithm there has to be a minimal amount of storage, the working set, for the algorithm to work. So, increasing parallism will increase this working set as fast as you increase the computational resources. If the balance between memory and compute swings toward more memory, then bit serial hardware isn't efficient. Your hadware will be mostly memory, poorly utilised. Its just a fact of life that few algorithms will gain from this bit squeeze, because the replicated footprint of the memory working set will just remove any gains you may achieve.

Minimising the footprint of the memory by utilising DRAM is fraught with silicon process issues. There are few processes that can do this (IBM for instance) and they are expensive. In the end its a route that doesn't provide enough benefit over and above what you could achieve by conventional means.

FPGAs have evolved to generalise logic function at the bit level. (what really is the difference in a single bit ALU and an FPGA CLB?) However if you look at the trends, they are nowhere near as area efficient as fixed function or wider bit operations. Indeed, newer FPGAs have embedded DSP units sprinkled in a sea of configurable hardware.

#18 Yossi Kreinin on 04.16.12 at 10:17 pm

Well, I certainly don't recommend this, either :) And yeah, there's memory vs compute density problem [paralleling your comment at the GPU article :) ]

Regarding DRAM: it's an interesting point in a geeky discussion, which doesn't make it a good thing to bet for a real business (I know a real case when this bet didn't work out; BTW – "been there, done that" – you mean literally, or with something similar but not actually the same?)

Regarding FPGAs – and largely agreeing with the fact that DSP slices in modern FPGAs are a telling data point with respect to the efficiency of configurable logic – I think one big difference is how routing works; in FPGAs, bits will undergo quite a bit of processing before reaching a flip-flop, so it seems a more computationally dense design. (Of course there are also bits needed to configure which processing it should be, taking some of the density away, but I'd still think that overall, in an FPGA without fixed logic, the ratio of memory to compute resources is smaller than in CAPPs.)

#19 Ray McConnell on 04.24.12 at 8:45 am

Hired a small team with a retired DRAM designer, and build a custom DRAM that coupled directly to a small but highly replicated processor. It wasn't so much the DRAM design but the pain needed in additional process steps to support the DRAM cell. So many unexpected side effects on other blocks, (including SRAMS) made the effort way too hard. Best to avoid mixed designs.

However, IBM have worked a great side effect free process with DRAM macros. Extra process steps are still necessay. They even use a modified cell for decoupling. And bizarrely DRAMS are also more rad hard than SRAMS by many orders of magnitude, but thats another story.

Take your point re FPGA. Hard to really measure without a detail spreadsheet. As you acked, the point is they have evolved to hard macros to do number crunching and this is telling.

#20 red piter on 05.15.19 at 8:53 pm

Appreciate it for this howling post, I am glad I observed this internet site on yahoo.

#21 vn hax on 05.16.19 at 1:28 pm

Good Morning, glad that i found on this in bing. Thanks!

#22 fortnite aimbot download on 05.16.19 at 5:22 pm

I have interest in this, cheers.

#23 nonsensediamond on 05.17.19 at 7:38 am

Great read to see, glad that google led me here, Keep Up cool job

#24 Sal Graceffo on 05.17.19 at 10:21 am

Kudos for the noteworthy blog you've created at yosefk.com. Your enthusiasm is certainly inspiring. Thanks again!

#25 fallout 76 hacks on 05.17.19 at 11:02 am

Some truly great stuff on this web site , appreciate it for contribution.

#26 red dead redemption 2 digital key resale on 05.17.19 at 4:11 pm

I consider something really special in this site.

#27 redline v3.0 on 05.17.19 at 7:16 pm

Great stuff to check out, glad that bing led me here, Keep Up awsome job

#28 chaturbate hack cheat engine 2018 on 05.18.19 at 8:42 am

Yeah bookmaking this wasn’t a risky decision outstanding post! .

#29 forza horizon 4 license key on 05.18.19 at 3:33 pm

Good Morning, happy that i found on this in yahoo. Thanks!

#30 mining simulator codes 2019 on 05.19.19 at 7:36 am

I dugg some of you post as I thought they were very beneficial invaluable

#31 smutstone on 05.20.19 at 12:14 pm

Thank You for this.

#32 redline v3.0 on 05.21.19 at 7:46 am

I like this article, because so much useful stuff on here : D.

#33 free fire hack version unlimited diamond on 05.21.19 at 5:05 pm

Enjoyed reading through this, very good stuff, thankyou .

#34 nonsense diamond on 05.22.19 at 6:55 pm

I’m impressed, I have to admit. Genuinely rarely should i encounter a weblog that’s both educative and entertaining, and let me tell you, you may have hit the nail about the head. Your idea is outstanding; the problem is an element that insufficient persons are speaking intelligently about. I am delighted we came across this during my look for something with this.

#35 krunker hacks on 05.23.19 at 7:14 am

I like this site because so much useful stuff on here : D.

#36 bitcoin adder v.1.3.00 free download on 05.23.19 at 10:54 am

I have interest in this, cheers.

#37 vn hax on 05.23.19 at 7:37 pm

I conceive this web site holds some real superb information for everyone : D.

#38 eternity.cc v9 on 05.24.19 at 8:26 am

I was looking at some of your articles on this site and I believe this internet site is really instructive! Keep on posting .

#39 ispoofer pogo activate seriale on 05.24.19 at 6:56 pm

I conceive you have mentioned some very interesting details , appreciate it for the post.

#40 cheats for hempire game on 05.26.19 at 6:58 am

I like, will read more. Cheers!

#41 iobit uninstaller 7.5 key on 05.26.19 at 9:43 am

Your post has proven useful to me.

#42 smart defrag 6.2 serial key on 05.26.19 at 4:12 pm

I conceive you have mentioned some very interesting details , appreciate it for the post.

#43 resetter epson l1110 on 05.26.19 at 6:59 pm

Great read to Read, glad that bing brought me here, Keep Up nice Work

#44 sims 4 seasons free code on 05.27.19 at 8:16 am

I love reading through and I believe this website got some genuinely utilitarian stuff on it! .

#45 rust hacks on 05.27.19 at 8:43 pm

I conceive you have mentioned some very interesting details , appreciate it for the post.

#46 strucid hacks on 05.28.19 at 11:00 am

I must say, as a lot as I enjoyed reading what you had to say, I couldnt help but lose interest after a while.

#47 expressvpn key on 05.28.19 at 8:02 pm

I love reading through and I believe this website got some genuinely utilitarian stuff on it! .

#48 gamefly free trial on 05.28.19 at 11:17 pm

Hi there! This post could not be written any
better! Reading through this post reminds me of my old room mate!
He always kept chatting about this. I will forward this post
to him. Pretty sure he will have a good read.
Thanks for sharing!

#49 how to get help in windows 10 on 05.29.19 at 2:27 am

Appreciate the recommendation. Let me try it out.

#50 ispoofer key on 05.29.19 at 9:21 am

I have interest in this, thanks.

#51 aimbot free download fortnite on 05.29.19 at 1:21 pm

yahoo brought me here. Cheers!

#52 redline v3.0 on 05.29.19 at 5:46 pm

Ha, here from yahoo, this is what i was looking for.

#53 vn hax on 05.30.19 at 7:03 am

This does interest me

#54 xbox one mods free download on 05.31.19 at 1:35 pm

Enjoyed reading through this, very good stuff, thankyou .

#55 fortnite aimbot download on 05.31.19 at 4:18 pm

Intresting, will come back here again.

#56 gamefly free trial on 06.01.19 at 2:06 am

Highly energetic blog, I enjoyed that a lot. Will there be
a part 2?

#57 gamefly free trial on 06.01.19 at 8:50 am

I got this website from my friend who informed me concerning this web page and at
the moment this time I am visiting this web site
and reading very informative articles or reviews at this place.

#58 mpl pro on 06.01.19 at 6:59 pm

Deference to op , some superb selective information .

#59 gamefly free trial on 06.01.19 at 11:35 pm

Have you ever considered publishing an ebook or
guest authoring on other blogs? I have a blog centered on the same subjects you discuss and
would love to have you share some stories/information. I know my viewers would appreciate your work.
If you're even remotely interested, feel free to send me
an email.

#60 hacks counter blox script on 06.02.19 at 7:10 am

I like this article, useful stuff on here : D.

#61 gamefly free trial on 06.02.19 at 10:01 pm

Very rapidly this site will be famous amid all blog visitors, due to it's nice articles
or reviews

#62 gamefly free trial on 06.03.19 at 3:32 am

Why users still use to read news papers when in this technological globe all is accessible on web?

#63 how to crack fortnite accounts on 06.03.19 at 10:59 am

I have interest in this, danke.

#64 gamefly free trial on 06.04.19 at 4:10 am

Simply want to say your article is as astonishing.
The clearness in your post is simply cool and that i could assume you're knowledgeable on this subject.
Fine along with your permission allow me to grasp your feed to stay updated with drawing close post.
Thank you a million and please continue the enjoyable work.

#65 gamefly free trial on 06.05.19 at 10:03 am

We're a gaggle of volunteers and opening a brand new scheme in our community.
Your website provided us with helpful information to
work on. You have performed a formidable process and our entire neighborhood will likely be thankful to you.

#66 gamefly free trial on 06.06.19 at 4:35 am

This blog was… how do I say it? Relevant!! Finally I have found
something that helped me. Thanks!

#67 gamefly free trial on 06.06.19 at 7:38 am

Howdy, i read your blog from time to time and i own a similar
one and i was just wondering if you get a lot of spam responses?

If so how do you reduce it, any plugin or anything you can suggest?
I get so much lately it's driving me crazy so any support is very much appreciated.

#68 Rigoberto Kumpf on 06.07.19 at 11:06 pm

In my estimation, yosefk.com does a excellent job of covering issues of this sort! While often intentionally controversial, the information is more often than not well researched and stimulating.

#69 Harold Levy on 06.09.19 at 9:26 pm

6/9/2019 I'm pleased by the manner in which yosefk.com deals with this type of subject! Generally on point, often controversial, always well-researched and also stimulating.

#70 gamefly free trial 2019 coupon on 06.10.19 at 1:38 pm

I'm really loving the theme/design of your blog.
Do you ever run into any web browser compatibility problems?
A handful of my blog readers have complained about
my blog not working correctly in Explorer but looks great in Chrome.
Do you have any advice to help fix this issue?

#71 playstation 4 best games ever made 2019 on 06.12.19 at 8:19 am

I visit every day some web sites and blogs to read articles, but this weblog presents quality based writing.

#72 ps4 best games ever made 2019 on 06.12.19 at 9:44 am

Howdy, i read your blog from time to time and i own a similar one and i was just wondering if you get a lot of spam comments?
If so how do you protect against it, any plugin or anything you can recommend?
I get so much lately it's driving me crazy so any
support is very much appreciated.

#73 Calvin Rynn on 06.14.19 at 5:48 am

yosefk.com is a well-written piece. I just sent this on 6/13/2019 to a coworker who's been doing some work of their own on the topic. To say thank you, he just bought me dinner! So, I should probably say: Cheers for the meal!

#74 quest bars cheap on 06.14.19 at 11:47 am

If you would like to increase your familiarity just
keep visiting this web page and be updated with the latest news update posted here.

#75 quest bars cheap on 06.14.19 at 2:52 pm

I'm extremely inspired along with your writing abilities as well
as with the format on your weblog. Is that this a paid theme or did you
modify it your self? Either way keep up the nice quality writing, it is uncommon to see a nice blog like this one today..

#76 gx tool download on 06.17.19 at 1:52 pm

stays on topic and states valid points. Thank you.

#77 tinyurl.com on 06.18.19 at 3:30 am

Hey I know this is off topic but I was wondering if
you knew of any widgets I could add to my blog that
automatically tweet my newest twitter updates.

I've been looking for a plug-in like this for quite some time and
was hoping maybe you would have some experience with something like this.
Please let me know if you run into anything. I truly enjoy reading your blog and I look forward to your new updates.

#78 proxo key on 06.19.19 at 2:29 pm

Some truly great goodies on this web site , appreciate it for contribution.

#79 vn hax pubg on 06.20.19 at 11:01 pm

Good Day, happy that i saw on this in bing. Thanks!

#80 nonsense diamond on 06.21.19 at 12:06 pm

Good Morning, happy that i found on this in bing. Thanks!

#81 badoo superpowers free on 06.23.19 at 9:23 pm

I conceive you have mentioned some very interesting details , appreciate it for the post.

#82 gx tool pro on 06.24.19 at 7:17 pm

Great, bing took me stright here. thanks btw for info. Cheers!

#83 geometry dash 2.11 download on 06.26.19 at 12:02 am

Ni hao, i really think i will be back to your site

#84 krunker aimbot on 06.26.19 at 10:27 am

I love reading through and I believe this website got some genuinely utilitarian stuff on it! .

#85 ispoofer activation key on 06.27.19 at 9:27 am

Appreciate it for this howling post, I am glad I observed this internet site on yahoo.

#86 synapse x serial key on 06.28.19 at 12:32 am

I am glad to be one of the visitors on this great website (:, appreciate it for posting .

#87 strucid hacks on 06.28.19 at 11:28 am

very Great post, i actually love this web site, carry on it

#88 serial advanced systemcare 11.5 on 06.28.19 at 4:01 pm

I am not rattling great with English but I get hold this really easygoing to read .

#89 how to get help in windows 10 on 06.29.19 at 2:46 am

As the admin of this website is working, no question very soon it will be renowned, due to its feature contents.

#90 cryptotab hack script free download 2019 on 06.29.19 at 10:18 am

I really got into this site. I found it to be interesting and loaded with unique points of interest.

#91 cryptotab script hack free on 06.29.19 at 4:40 pm

This i like. Cheers!

#92 codes for mining simulator 2019 on 07.01.19 at 10:37 pm

I like this site because so much useful stuff on here : D.

#93 escape from tarkov cheats and hacks on 07.02.19 at 10:41 am

Enjoyed examining this, very good stuff, thanks .

#94 nonsense diamond on 07.02.19 at 3:41 pm

Intresting, will come back here more often.

#95 Ebony Janszen on 07.03.19 at 2:57 am

This really answered my problem, thank you!

#96 vn hax on 07.03.19 at 10:02 am

I kinda got into this website. I found it to be interesting and loaded with unique points of view.

#97 cyberhackid on 07.03.19 at 9:58 pm

Intresting, will come back here later too.

#98 xforce keygen on 07.04.19 at 9:59 am

I enjoying, will read more. Thanks!

#99 seo principles 2018 on 07.04.19 at 3:03 pm

Parasite backlink SEO works well :)

#100 phantom forces aimbot on 07.04.19 at 9:52 pm

Deference to op , some superb selective information .

#101 erdas foundation 2015 on 07.05.19 at 10:31 pm

Enjoyed examining this, very good stuff, thanks .

#102 synapse x roblox on 07.06.19 at 8:40 am

Great article to see, glad that duckduck led me here, Keep Up awsome job

#103 gx tool pubg uc hack apk download on 07.06.19 at 12:46 pm

Deference to op , some superb selective information .

#104 rekordbox torrent on 07.07.19 at 4:17 am

Great, this is what I was searching for in bing

#105 call of duty black ops 4 licence key on 07.07.19 at 12:33 pm

I truly enjoy looking through on this web site , it holds superb content .

#106 spyhunter 5.4.2.101 portable on 07.08.19 at 1:21 pm

Hello, glad that i found on this in yahoo. Thanks!

#107 quest bars cheap 2019 coupon on 07.09.19 at 5:24 am

My partner and I stumbled over here coming from a different web address
and thought I might check things out. I like what I see so i am just following you.

Look forward to looking over your web page for a
second time.

#108 roblox fps unlocker download on 07.09.19 at 3:18 pm

I really enjoy examining on this page , it has got cool goodies .

#109 quest bars cheap on 07.11.19 at 1:50 am

I all the time emailed this weblog post page to all my associates,
for the reason that if like to read it after that my friends will too.

#110 crazy_catty on 07.14.19 at 2:03 am

Thank you for the great read!

#111 live sex chat on 07.15.19 at 2:15 am

some great ideas this gave me!

#112 legal porno on 07.16.19 at 12:12 am

great advice you give

#113 legalporn on 07.16.19 at 12:25 am

great advice you give

#114 how to get help in windows 10 on 07.17.19 at 4:17 am

Having read this I believed it was rather informative.

I appreciate you taking the time and energy to put this article together.
I once again find myself personally spending a lot of time both reading and leaving comments.
But so what, it was still worth it!

#115 Granville Vink on 07.17.19 at 6:58 pm

Alex9, this message is your next bit of information. Please transceive the agency at your convenience. No further information until next transmission. This is broadcast #5109. Do not delete.

#116 how to get help in windows 10 on 07.18.19 at 2:31 am

Does your site have a contact page? I'm having trouble locating it but, I'd like to shoot
you an email. I've got some recommendations for your blog you might be interested in hearing.
Either way, great blog and I look forward to seeing it develop
over time.

#117 sonya on 07.19.19 at 1:57 am

you are a great writer!

#118 ashley_adams on 07.19.19 at 2:06 am

thx you so much for posting this!

#119 buydrugsonline on 07.19.19 at 2:57 am

This blog is amazing! Thank you.

#120 plenty of fish dating site on 07.19.19 at 10:29 am

Wow, this post is nice, my sister is analyzing these kinds of
things, therefore I am going to let know her.

#121 how to get help in windows 10 on 07.20.19 at 6:49 am

Very great post. I just stumbled upon your blog and wished to say that
I've truly loved surfing around your blog posts. After all I'll be
subscribing to your rss feed and I am hoping
you write once more very soon!

#122 prodigy hacks on 07.21.19 at 6:43 pm

Great article to check out, glad that Yahoo led me here, Keep Up nice Work

#123 how to get help in windows 10 on 07.22.19 at 10:20 am

In fact no matter if someone doesn't be aware of after that its up to other users that they will help, so
here it occurs.

#124 natalielise on 07.22.19 at 8:27 pm

My brother recommended I might like this website.
He used to be totally right. This submit truly made my day.
You cann't imagine simply how a lot time I had spent for this
info! Thanks! pof natalielise

#125 plenty of fish dating site on 07.23.19 at 9:35 am

I do not even know how I stopped up here, but I believed this publish was good.
I do not recognize who you are but certainly you are going to a famous blogger for those who are
not already. Cheers!

#126 evogame.net/wifi on 07.23.19 at 5:51 pm

I was looking at some of your articles on this site and I believe this internet site is really instructive! Keep on posting .

#127 date coufar on 07.23.19 at 10:54 pm

I am 43 years old and a mother this helped me!

#128 date ocugar on 07.23.19 at 11:31 pm

I am 43 years old and a mother this helped me!

#129 darte cougar on 07.23.19 at 11:48 pm

I am 43 years old and a mother this helped me!

#130 dtae cougar on 07.24.19 at 12:00 am

I am 43 years old and a mother this helped me!

#131 forza horizon 4 keygen on 07.24.19 at 6:09 pm

Just wanna input on few general things, The website layout is perfect, the articles is very superb : D.

#132 plenty of fish dating site on 07.25.19 at 1:26 pm

Hi there this is kind of of off topic but I was wondering if blogs use WYSIWYG editors or if you have to manually code with HTML.
I'm starting a blog soon but have no coding experience so I wanted to get advice from someone with
experience. Any help would be greatly appreciated!

#133 skisploit on 07.25.19 at 8:48 pm

You got yourself a new rader.

#134 skisploit on 07.26.19 at 10:01 pm

Cheers, great stuff, I like.