Making data races manifest themselves

Or, synchronization, valgrind and poset dimension.

This story is perhaps more technical than I ever allowed myself to get online. If I may say so – it may be worth following; I can say that it really changed my perspective on what data races are.

I'll need to explain a bit about the context:

  • how we structure our parallel apps for multi-core targets
  • how we catch synchronization bugs using a custom valgrind tool
  • which of the bugs were hard to catch that way

Then I can explain how we now catch them all (at least those having any chance to happen on our test inputs; you'll see what I mean).

I'll also discuss valgrind's standard race detection tool, helgrind. I'll try to show that the interface to parallelism it works against – pthreads – poses a much harder problem, and how races can go undetected because of that.

I hope this can be interesting outside of my specific work context because "catching all the races" seems to be a claim rarely made. Also we got to stumble upon a well-known math problem during mundane pointer-chasing – a cliche of sorts, but amusing when it actually happens.

So, here goes.

Our parallelism frameworkThe framework - goals & dependencies

…known in its current incarnation as GSF – "Goal/Graph Scheduling Framework". It's fairly simple – 4 paragraphs and 2 pseudo-code listings should be a reasonable introduction.

The idea is, your app is a graph of goals. The nodes, goals, are things to do – typically function pointers (could be function objects or closures). The edges are dependencies – A depends on B, so A must be executed after B (denoted with B A in the diagrams). The main loop does this:

goal = first_goal # a dummy everything depends on
while goal != last_goal: # a dummy, depends on everything
  update_dependent_goals_status(goal) # find READY goals
  for chosen in sched.choose_goals(): # scheduler
    chosen.resource.enqueue(chosen) # outgoing message Qs
  goal = dequeue_finished_goal() # incoming message Q

This runs all the goals in a graph, and can itself be invoked in a loop – in our case, we run all the goals for every grabbed video frame.

Whatever the scheduling policy, a correct execution order will be chosen. Only READY goals can run, and a goal only becomes READY when everything it depends on becomes DONE. This is implemented in update_dependent_goals_status:

goal.status = DONE
for dep in goal.dependent_goals:
  dep.num_predecessors -= 1
  if dep.num_predecessors == 0:
    dep.status = READY

The scheduling policy is implemented partly in sched.choose_goals – the scheduler, and partly in resource.enqueue – the queues. The scheduler decides who is enqueued first; some queues may reorder waiting goals by priority, or even suspend the running goal(s) when a more important one arrives. While the scheduling policy can be made complicated almost without bound, that's all there is to the framework itself.

To summarize in standard terms, it's a kind of a message passing framework, without explicit threads and without locking, but with (massive) data sharing. "Goal scheduled"/"goal finished" notifications are the only messages, and only used by the framework – the goals themselves always communicate through shared data. Dependencies are the only synchronization primitive. OS locks are never used, except in the guts of various memory allocators.


Users, or "application programmers", usually hate this framework with a passion, because code is hard to read and modify.

To break your app into the sort of graph described above, you need to specify the goals and the dependencies. Currently we do both in ugly ways. Goals are specified as global functions, with context passed through global variables: void goal_A() { g_ctxt… }. The hairy dependency graph is constructed by a Python script running at build time, with ifs, loops, and function calls – if do_A: make_goal("A",dep="B C"). This adds ugliness to C++ code, and then makes you follow ugly Python code to figure out the execution order of the ugly C++ code.

Some of it was even uglier in the past. Some of it could become less ugly in the future. At least one serious limitation is irremediable – the inability to create goals at run time. For example, you can't say "create a goal per processed data item" – rather, you need to explicitly, statically create a goal per CPU and distribute the data between the goals. You can't say "if X happens, create that goal" – you need to always have that goal, and have it do nothing unless X happens. People don't like to look at their flow that way.


Efficiency and correctness, basically. Efficiency is tangential to our story, so I won't discuss it (and frankly, we aren't sure how much efficiency we gain, compared to other approaches). As to correctness – the hateful thing indeed "scales", which is to say, it doesn't break down. We've managed to ship working code for several years, given:

  • 8 to 10 heterogeneous target cores
  • Hundreds of goals
  • Dozens of applications
  • Dozens of developers

…the worst thing being "dozens of developers" – most unaware of parallelism, as I think they should be. Not that it proves we couldn't have done it differently – I can only give my theory as to why it would be harder.

The biggest problem with parallel imperative programs is synchronizing access to shared data. Tasks/threads can, and are encouraged to communicate through messages, but eliminating sharing is hard:

  • People frequently don't even realize they're sharing data

  • If tools make implicit use of shared data impossible, things can get painful (think of explicitly passing around a particularly "popular" data item, or copying large/intertwined objects)

Sharing data is natural and efficient in imperative programs, and outlawing it is hard. And locks are problematic since

  • You can have deadlocks, and
  • You can forget to lock.

Now, if you schedule a statically known flow graph rather than an unconstrained threaded app, it turns out that you solve both of these problems:

  • Deadlocks are trivially prevented - at build time, check that the dependency graph has no cycles

  • Undeclared dependencies – sharing data without proper synchronization – can be found using program instrumentation

Our subject is this second bit – using program instrumentation to find race conditions (which makes everything up to this point an unusually long introduction).

We use a custom valgrind plug-in ("tool" as they call it) for program instrumentation. It works somewhat similarly to valgrind's helgrind tool, though helgrind's implementation is much more complicated.

However, helgrind has false negatives – it will consistently miss some of the data races, as I believe will any tool lacking the knowledge of the overall app structure. A simple example of a data race unreported by helgrind appears after the discussion on race condition detection, when the problem should become more clear.

Race conditions in a static graph

In an imperative program, goals communicate through memory. A memory access can thus be thought of as a proof of dependence. If goal A accesses memory at address X, and goal B was the last to modify X, then A depends on B. If there is no path from A to B in the dependency graph, the program is buggy. A could run before B or after it, so the result of accessing X is undefined.Write then read: the simplest dependence proof

Suppose we could intercept all load/store instructions in the program. Then we could maintain, for every byte, its current "owner" – a goal ID. A store to a location would update its owner to the ID of the currently running goal. Loads and stores could then check whether the currently running goal depends on the owner of the accessed location – all it takes is accessing a 2D array, the path matrix. Upon error, print the call stack, and the names of the 2 goals.

This sort of thing is surprisingly easy to implement on top of valgrind, almost as easy as implementing an interface with abstract onLoad and onStore methods. I thought of posting sample code but it looks like the example tool shipped with valgrind, "lackey", comes with a load/store interception example these days.

As to the "shadow memory" you need for keeping owner IDs – you can do that with a page table, along the lines of a virtual memory implementation in an OS. The high bits of a pointer select a page and the low bits are the offset within the page, with pages allocated upon first access. Standard valgrind tools do it this way, too.

Our valgrind tool is called shmemcheck for "shared memory checker". A lame pun on memcheck, valgrind's most famous tool, the one reporting uninitialized memory access. "Memcheck-shmemcheck". Probably the last time I used this sort of name – funny for the first few times, embarrassing for the next few thousands of times.

Despite the embarrassment, when we got an implementation good enough to be systematically used, it was really great – data races were decimated. Valgrind is awesome, unbelievably so. It managed to unimaginably expand what can be done to a natively compiled program, decades after the tools for building native programs were cast in stone.

The trouble with this version of, cough, shmemcheck, is that it doesn't really work. That is, sometimes you have false negatives, so you still get to dive into core dumps. Why?

What about the opposite case?

Read then write: another shadow cell?If A loads from X that was last modified by B, A depends on B, and we detect it alright. What if A writes to X that was previously read by B? This also proves that A should depend on B. Otherwise B will sometimes run after A, reading its update to X, and sometimes before A, reading the previous value. In order to detect this dependency, we have to remember that X was read by B until A stores to X.

…What if, in addition to B, X was read by C, D, and E, all before A's update?Reads then write: too many shadow cells

Every location always has one "owner" but can have many "users". We can afford keeping an ID – 2 bytes in our case – for every byte. Keeping – and checking – a list of users that could grow to tens or even hundreds of IDs per location sounds impractical.

We used all sorts of application-specific arguments to convince ourselves that this problem is confined to a few particular scenarios. We had a few hacks removing some of the false negatives, at the cost of adding some false positives, and lived with that.

Choosing the order

We got used to think that the question was, how do you know who reads X before A? But then it dawned upon us that the right question was: why do they read X before A?!read then write: why not reverse the order?

And really – if B, C and D run before A because it is stated in the dependency graph that A depends on B, C and D – then there's no bug to worry about. But if there's no dependency declared between, say, A and B, then A could run before B just as well – and we'd detect the bug. So we only missed the race condition because of a randomly chosen order. And when we do find races, we don't know if B depends on A or vice versa – only that we were lucky to have A run first.

What if we choose different random orders and run the same app many times? Then in some run, A will run before B and we'll find the bug – with just one shadow cell keeping the owner.

…Or will it? We have in our dependency graph many "A, B" pairs of independent goals. If we pick random orders, how likely are these pairs to "flip" to "B, A" in some orders but not others?

The first conjecture was – very likely. Just randomize schedules by having sched.choose_goals pick a random goal among the READY ones. "There seems to be no bias in the selection; A and B are independent so the chance for A to come before B is 1/2; therefore, with N orders, the chance for A to always follow B is 1/2^N – small enough."

Interestingly, it still sounds reasonable to me, though I found a counter-example, just out of fear that it's too good to be true (what, years of core dumps are now behind me?) The counter example is, suppose you have two long processes, P and Q, each made up of 100 goals, P1…P100 and Q1…Q100. P and Q are declared independent – Pi never depends on Qj or vice versa. However, Pi+1 depends on Pi, similarly for Q, so P and Q goals can run in just one order.

Just two schedules would "flip" all independent pairs: (1) Run P, then Q and (2) Run Q, then P. However, sched.choose_goals has a snowflake's chance in hell to pick these schedules. The set of READY goals will contain, at all times, 2 goals: one from P and one from Q. sched.choose_goals must either choose the one from P 100 times in a row, or the one from Q 100 times in a row. Since that's a 1 in a 2^100 event, P1 will always run before Q100. If P1 loads from X and Q100 stores to X, we'll never find the undeclared dependency.

One could say that things that aren't likely to flip in such random trials are unlikely to flip in reality and it'd be 100% wrong. In reality, execution order depends on real run times and those create a very biased, and an unpredictably biased sampling of the distribution of schedules.

Now, with N goals, N*2 orders would clearly be enough to flip all pairs – for every goal, create an order where it preceeds everything it doesn't depend on, and another order where it follows everything it doesn't depend on. But our N is above 1000, so N*2, though not entirely impractical, is a bit too much.

Poset dimension

Our pessimism resulting from the collapse of the first conjecture lead to an outburst of optimism yielding a second conjecture – just 2 orders are enough to flip all pairs. Specifically, the orders we thought of were a DFS postorder and its "reversal", where you reverse the order in which you traverse the children of nodes.

Works on my P & Q example indeed, but there's still a counter-example, just a bit more elaborate. We kept looking at it until it turned out that we shouldn't. The dependency graph makes the goals a partially ordered set or poset. Finding the minimal number of schedules needed to flip all independent pairs would find the order dimension of that poset. Which is a known problem, and, like all the good things in life, is NP-hard.

Curious as this discovery was, I was crushed by this NP-hardship. However, there still could be a good practical heuristic yielding a reasonably small number of orders.

We tried a straightforward one – along the lines of the N*2 upper bound, "flip them one by one", but greedier. Simply try to flip as many pairs as you can in the same schedule:

  1. Start with a random schedule, and make a list of all the pairs that can be flipped.
  2. Starting with the real dependencies, add fake dependencies forcing pairs from the list to flip, until you can't add any more without creating a cycle.
  3. Create a schedule given those extended dependencies.
  4. Remove the pairs you flipped from the list. If it isn't empty, go to step 2.

This tended to produce only about 10 schedules for more than 1000 goals, and there was much rejoicing.

Massive testing

Instrumentation slows things down plenty, so you can't run many tests. This is a pity because instrumentation only catches data races that actually occurred – it's actual loads by A from location X owned by B that prove that A depends on B. But if you can't run the program on a lot of data, you may not reach the flow path in A that uses X.

However, if you can, with just 10 schedules, flip all "A, B" pairs, you don't need instrumentation in the first place. Just run the app with different orders on the same data, see if you get the same results. If you don't, there's a race condition – then run under instrumentation to have it pin-pointed to you. The order is "purposefully diversified", but deterministic, so problems will reproduce in the second, instrumented run. Thus massive runs can be uninstrumented, and therefore, more massive.

So not only does this cheap poset dimension upper bound business solve the false negatives problem with instrumentation – it also greatly increases coverage. Of course there can still be a race that never occurred in any test runs. But now the quality of testing is defined just by the quality of the test data, never by chance. With races, whether you detect them or not is thought of as something inherently having to do with chance. It turns out that it doesn't have to.

Helgrind's problem

Helgrind – or any tool lacking knowledge about the app structure – can not meaningfully reorder things this way, in order to "make races manifest themselves". A race can be detected if A that stores to X is moved to happen before B that loads from X, but in order to do that, you have to know where A and B are.

Helgrind's only markers breaking the app into meaningful parts are synchronization calls, like mutex locks/unlocks. But it doesn't know where they'll happen in advance. So it can't stir the execution so that "this lock happens as early as possible" – there's no "this lock" until a lock actually happens. By then it's too late to make it happen before anything else that already happened. (In theory, the app structure could be inferred dynamically from the flow, I just don't see how I'd go about it.)

Therefore, helgrind seems to have the same problem our first versions of shmemcheck had. You can keep the single owner of a location but not its many readers; it appears that helgrind keeps one reader – the last. Helgrind's "owner/reader" is a lock ID rather than a goal ID, but I think the principle is very similar.  I haven't studied helgrind's interals so I only speculate about how it works, but I easily managed to create an example where it gives a false negative:

  1. In thread A, read X without locking, as many times as you want (…or don't want – in a real program it would be a bug, not an achievement…)
  2. After that, read X once with proper locking.
  3. In thread B, modify X, again with proper locking.

If, during testing, thread B happens to modify X after all of thread A's reads from X, the bug in A – unsynchronized access to X – will go unnoticed. It won't go unnoticed if thread A never locks properly – since helgrind will remember and report the last of its unsynchronized reads:

#include <pthread.h>
#include <stdio.h>

pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER;
int shared=1;
int unused=0;
typedef struct { int read_unlocked, read_locked; } sopt;

void* child_fn(void* arg) {
  sopt* opt = (sopt*)arg;
  if(opt->read_unlocked) {
    unused += shared;
  if(opt->read_locked) {
    unused += shared;
  return 0;

void testhelgrind(int read_unlocked, int read_locked,
                   const char* expected) {
  fprintf(stderr,"expected behavior: %sn", expected);
  sopt opt = { read_unlocked, read_locked };
  pthread_t child;
  pthread_create(&child, 0, child_fn, &opt);
  //lock & modify shared data
  shared = 2;
  pthread_join(child, 0);

int main(void) {
 testhelgrind(0,1,"no errors reported [true negative]");
 testhelgrind(1,0,"data race reported [true positive]");
 testhelgrind(1,1,"no errors reported [false negative]");

Which, under valgrind –tool=helgrind, prints:

expected behavior: no errors reported [true negative]
expected behavior: data race reported [true positive]

==15041== Possible data race during write of size 4 at 0x804a024 by thread #1
==15041==    at 0x8048613: test_helgrind (helgrind_example.c:33)
==15041==    by 0x8048686: main (helgrind_example.c:41)
==15041==  This conflicts with a previous read of size 4 by thread #3
==15041==    at 0x804856E: child_fn (helgrind_example.c:14)

expected behavior: no errors reported [false negative]

…or at least it works that way on my system, where the child thread runs first.

So if helgrind does report a data race – be sure to look careful. If you have several buggy reads, and only fix the last one and run again, helgrind will very likely not report the previous one. Your new synchronization code around the last read now "masks" the previous read.

The happy ending

We still have false negatives in our data race detection, but these are just easy-to-fix low-level loose ends – or so I think/hope. Things like tracking memory writes by devices other than CPUs, or handling custom memory allocators. I'm looking forward to have these false negatives eliminated, very sincerely. Few things are uglier than debugging data races.

I never thought this through enough to tell whether this purposeful reordering business can be utilized by a framework more restrictive than "use raw threads & locks", but less restrictive than "your app is a static graph". If it can be (is already?) utilized elsewhere, I'd be very curious to know.

Special thanks to Yonatan Bilu for his help with the math.


#1 Antiguru on 12.06.10 at 10:31 pm

Thanks for posting this. Very interesting read.

#2 Vlad Patryshev on 12.07.10 at 5:42 pm

Reads if not like a novel, like a good story. Thanks!!!

#3 Yossi Kreinin on 12.10.10 at 6:27 am

Thanks to those who liked it for letting me know, this being an atypically hard read among my probably less interesting but probably more entertaining writing.

#4 Dan on 12.14.10 at 7:34 am

I'll have to come back and read this again some other time when half my brain cells aren't still sleeping. But I do have to say that this illustrates perfectly the scale of the morass that is known as "parallelism". I've not seen any other programming can unleash as many worms as when opening the can of parallelism. It seems to me the tools we use today still are simply not up to the task of properly grappling with the problem. Often, when faced with a parallelism bug, I feel like a caveman armed with nothing but a stone hammer trying to troubleshoot the Infinite Improbability Drive from an alien space ship. Sometimes this makes diving up to your elbows in excrement, as an alternative, look quite attractive!

#5 Yossi Kreinin on 12.14.10 at 8:06 am

@Dan: Well, on the other hand, it illustrates (or at least attempts to) that it doesn't have to be that bad, even in an imperative program. At the moment I have the impression that our chances of seeing a parallelism bug in a production environment are close to zero.

#6 Frank on 01.07.11 at 7:35 pm

I found this article very interesting to read!
One question kind of suggests itself, though:
Couldn't this be solved more efficiently (and with 100% reliability) with static code analysis rather than runtime analysis?

#7 Yossi Kreinin on 01.08.11 at 7:39 am

@Frank: this question did suggest itself to us; there are two problems – static flow analysis has its limitations unless the language is constrained somehow (the halting problem and such), and then parsing C++ is a PITA so you can't easily run an experiment to see how many problems static analysis would run into on your actual code. Therefore ideas along that direction never materialized; the run time instrumentation option is a couple of orders of magnitude easier to follow.

#8 Frank on 01.08.11 at 2:58 pm

Oh, I see. Both points.
I wonder if it was possible to replace C++ with something better.
I mean, what do we need in an implementation language for a distributed system under soft realtime constraints?
Enum Wishlist:
1. Efficinecy: High speed, small memory footprint.
2. Power: Easy things should be easy. Hard things, too (we dream!).
3. Enum Meta:
3.1 Efficient reasoning about the semantic properties of the runtime code.
3.2 Efficient manipulation of the semantic properties of the runtime code.
3.3 Have 3.1 and 3.2 with a semantic that captures temporal effects.
Okay, I got a bit carried away there in the end. But one must have a goal in order to know ones direction… =)
I'm curious if I've missed any essentials…

#9 Frank on 01.08.11 at 3:40 pm

Oh, and I've missed the identifiers in enum meta:
3.1 Introspection
3.2 Intromanipulation (yeah, I made that one up ;-)
3.3 I dont't know! Alternating-time temporal logic?
Okay, with 3.3 I'm really well an far out of my depth now and just dropping random names.

#10 Andy Wingo on 06.28.11 at 2:03 am

Nice article. The partial ordering permutations trick was a particularly nice touch.

#11 Yossi Kreinin on 06.28.11 at 4:11 am


#12 jonathan on 11.23.11 at 7:10 am

I just found your website about a week ago (via, I think reddit/programming) and I really enjoy your writing – subject matter and style. Coming from a math background I particularly liked this post. Posting this message 'cause I just wanted to encourage you to continue.

#13 Yossi Kreinin on 11.23.11 at 7:53 am

Thanks! Not coming from a math background, I spent a particularly large amount of time writing this post…

#14 resharper crack on 05.15.19 at 2:14 pm

Enjoyed reading through this, very good stuff, thankyou .

#15 krunker hacks on 05.16.19 at 12:20 pm

yahoo brought me here. Thanks!

#16 Faustino Maccarter on 05.16.19 at 2:57 pm

5/16/2019 I'm gratified with the way that handles this kind of subject matter! Usually on point, often controversial, consistently well-researched and more often than not quite thought-provoking.

#17 fortnite aimbot download on 05.16.19 at 4:14 pm

Found this on yahoo and I’m happy I did. Well written article.

#18 nonsensediamond on 05.17.19 at 6:27 am

Morning, i really think i will be back to your page

#19 fallout 76 hacks on 05.17.19 at 9:55 am

Great, this is what I was searching for in google

#20 red dead redemption 2 digital key resale on 05.17.19 at 3:05 pm

I like this site because so much useful stuff on here : D.

#21 redline v3.0 on 05.17.19 at 6:08 pm

Hi, google lead me here, keep up good work.

#22 badoo superpowers free on 05.18.19 at 7:33 am

Just wanna input on few general things, The website layout is perfect, the articles is very superb : D.

#23 forza horizon 4 license key on 05.18.19 at 2:27 pm

Good Day, glad that i saw on this in yahoo. Thanks!

#24 mining simulator codes 2019 on 05.19.19 at 6:25 am

Respect to website author , some wonderful entropy.

#25 smutstone on 05.20.19 at 11:05 am

I really got into this site. I found it to be interesting and loaded with unique points of interest.

#26 redline v3.0 on 05.21.19 at 6:33 am

I was looking at some of your articles on this site and I believe this internet site is really instructive! Keep on posting .

#27 free fire hack version unlimited diamond on 05.21.19 at 3:45 pm

Great article to check out, glad that yandex brought me here, Keep Up nice job

#28 nonsense diamond on 05.22.19 at 5:36 pm

Some truly cool content on this web site , appreciate it for contribution.

#29 krunker aimbot on 05.23.19 at 5:54 am

Cheers, great stuff, Me enjoying.

#30 bitcoin adder v.1.3.00 free download on 05.23.19 at 9:32 am

I really got into this post. I found it to be interesting and loaded with unique points of interest.

#31 vn hax on 05.23.19 at 6:16 pm

I really got into this article. I found it to be interesting and loaded with unique points of interest.

#32 v9 on 05.24.19 at 7:04 am

I consider something really special in this site.

#33 ispoofer pogo activate seriale on 05.24.19 at 5:25 pm

Cheers, great stuff, Me enjoying.

#34 cheats for hempire game on 05.26.19 at 6:02 am

Your post has proven useful to me.

#35 iobit uninstaller 7.5 key on 05.26.19 at 8:50 am

Found this on bing and I’m happy I did. Well written post.

#36 smart defrag 6.2 serial key on 05.26.19 at 3:09 pm

Intresting, will come back here more often.

#37 resetter epson l1110 on 05.26.19 at 5:39 pm

Appreciate it for this howling post, I am glad I observed this internet site on yahoo.

#38 sims 4 seasons code free on 05.27.19 at 6:56 am

I’m impressed, I have to admit. Genuinely rarely should i encounter a weblog that’s both educative and entertaining, and let me tell you, you may have hit the nail about the head. Your idea is outstanding; the problem is an element that insufficient persons are speaking intelligently about. I am delighted we came across this during my look for something with this.

#39 Delores Hovanesian on 05.27.19 at 6:24 pm

I'm pleased with the way that deals with this type of subject matter! Generally on point, sometimes controversial, always well-researched and thought-provoking.

#40 rust hacks on 05.27.19 at 7:32 pm

Yeah bookmaking this wasn’t a risky decision outstanding post! .

#41 how to get help in windows 10 on 05.28.19 at 7:06 am

Hi there! Do you know if they make any plugins to assist with Search Engine Optimization? I'm trying to get my blog to rank for some targeted keywords but I'm not seeing very good results.
If you know of any please share. Cheers!

#42 strucid hacks on 05.28.19 at 9:49 am

Just wanna input on few general things, The website layout is perfect, the articles is very superb : D.

#43 expressvpn key on 05.28.19 at 6:53 pm

Intresting, will come back here more often.

#44 gamefly free trial on 05.28.19 at 8:31 pm

Hello there! Would you mind if I share your blog with my twitter group?
There's a lot of people that I think would really
appreciate your content. Please let me know.

#45 ispoofer license key on 05.29.19 at 8:03 am

Enjoyed reading through this, very good stuff, thankyou .

#46 aimbot free download fortnite on 05.29.19 at 12:02 pm

Hi, glad that i saw on this in bing. Thanks!

#47 redline v3.0 on 05.29.19 at 4:30 pm

I conceive you have mentioned some very interesting details , appreciate it for the post.

#48 vn hax on 05.30.19 at 5:41 am

Ha, here from yahoo, this is what i was browsing for.

#49 xbox one mods free download on 05.31.19 at 12:14 pm

I must say, as a lot as I enjoyed reading what you had to say, I couldnt help but lose interest after a while.

#50 fortnite aimbot download on 05.31.19 at 3:00 pm

I truly enjoy looking through on this web site , it holds superb content .

#51 mpl pro on 06.01.19 at 6:04 pm

I must say, as a lot as I enjoyed reading what you had to say, I couldnt help but lose interest after a while.

#52 gamefly free trial on 06.02.19 at 2:01 am

Hi, Neat post. There's a problem along with your site in internet
explorer, would test this? IE still is the market leader and a large component to other folks will pass over your
magnificent writing due to this problem.

#53 hacks counter blox script on 06.02.19 at 6:09 am

Great article to check out, glad that yandex took me here, Keep Up good Work

#54 chaturbate tokens hack generator 2018 pc on 06.03.19 at 9:58 am

Good, this is what I was looking for in google

#55 gamefly free trial on 06.03.19 at 12:43 pm

Pretty! This was a really wonderful article. Thanks for supplying this information.

#56 gamefly free trial on 06.04.19 at 8:20 pm

Stunning quest there. What happened after? Thanks!

#57 gamefly free trial on 06.06.19 at 6:00 pm

Yesterday, while I was at work, my cousin stole my iphone and
tested to see if it can survive a twenty five
foot drop, just so she can be a youtube sensation. My iPad is now broken and she
has 83 views. I know this is entirely off topic
but I had to share it with someone!

#58 on 06.07.19 at 4:54 pm

You can definitely see your enthusiasm within the work you write.
The arena hopes for even more passionate writers such as you who are not
afraid to say how they believe. Always follow your heart.

#59 gamefly free trial on 06.07.19 at 5:37 pm

Hello just wanted to give you a quick heads up. The text in your
post seem to be running off the screen in Internet explorer.

I'm not sure if this is a format issue or something to do with
web browser compatibility but I figured I'd post to let you know.
The layout look great though! Hope you get the issue fixed soon. Kudos

#60 free ps4 games on 06.08.19 at 2:36 pm

What a information of un-ambiguity and preserveness of valuable experience on the topic of unexpected emotions.

#61 quest bars on 06.16.19 at 4:28 pm

Thankfulness to my father who told me regarding this weblog,
this webpage is actually remarkable.

#62 krunker aimbot on 06.16.19 at 8:20 pm

I consider something really special in this site.

#63 on 06.17.19 at 5:02 pm

Excellent post! We are linking to this great article on our
website. Keep up the good writing.

#64 proxo key on 06.19.19 at 8:22 am

Respect to website author , some wonderful entropy.

#65 vn hax download on 06.20.19 at 5:27 pm

I am not rattling great with English but I get hold this really easygoing to read .

#66 nonsense diamond 1.9 on 06.21.19 at 6:38 am

Just wanna input on few general things, The website layout is perfect, the articles is very superb : D.

#67 plenty of fish dating site on 06.21.19 at 10:14 pm

Wonderful blog! I found it while browsing on Yahoo News.

Do you have any tips on how to get listed in Yahoo News?

I've been trying for a while but I never seem to get there!
Many thanks

#68 star valor cheats on 06.23.19 at 4:10 pm

I’m impressed, I have to admit. Genuinely rarely should i encounter a weblog that’s both educative and entertaining, and let me tell you, you may have hit the nail about the head. Your idea is outstanding; the problem is an element that insufficient persons are speaking intelligently about. I am delighted we came across this during my look for something with this.

#69 gx tool apk download on 06.24.19 at 2:15 pm

Ha, here from yahoo, this is what i was browsing for.

#70 how do we KNOW on 06.25.19 at 4:01 am

You got yourself a new rader.

#71 qureka pro apk on 06.25.19 at 6:56 pm

This does interest me

#72 viacutan olie on 06.26.19 at 3:11 am

A round of applause for your article post.Really thank you! Cool.

#73 krunker aimbot on 06.26.19 at 5:35 am

Intresting, will come back here again.

#74 ispoofer activation key on 06.27.19 at 5:07 am

Awesome, this is what I was searching for in bing

#75 synapse x cracked on 06.27.19 at 7:52 pm

I must say, as a lot as I enjoyed reading what you had to say, I couldnt help but lose interest after a while.

#76 strucid aimbot script on 06.28.19 at 6:24 am

Enjoyed examining this, very good stuff, thanks .

#77 advanced systemcare 11.5 key on 06.28.19 at 12:47 pm

Just wanna input on few general things, The website layout is perfect, the articles is very superb : D.

#78 cryptotab hack script free download 2019 on 06.29.19 at 8:04 am

I like this website its a master peace ! Glad I found this on google .

#79 cryptotab script hack free download on 06.29.19 at 2:24 pm

I have interest in this, cheers.

#80 mm2 hack download on 07.01.19 at 8:23 am

Just wanna input on few general things, The website layout is perfect, the articles is very superb : D.

#81 on 07.01.19 at 3:48 pm

Helpful information. Lucky me I found your website by accident,
and I'm stunned why this twist of fate didn't happened earlier!

I bookmarked it.

#82 fortnite cheats on 07.01.19 at 7:11 pm

Good, this is what I was looking for in google

#83 hacking apex legends pc on 07.02.19 at 7:00 am

Enjoyed reading through this, very good stuff, thankyou .

#84 skin swapper on 07.02.19 at 12:35 pm

This i like. Cheers!

#85 vn hax on 07.03.19 at 6:42 am

Great, google took me stright here. thanks btw for post. Cheers!

#86 cyberhackid on 07.03.19 at 6:39 pm

Good Morning, yahoo lead me here, keep up great work.

#87 prison life hacks on 07.04.19 at 6:37 am

Yeah bookmaking this wasn’t a risky decision outstanding post! .

#88 seo tutorial w3school on 07.04.19 at 3:04 pm

Parasite backlink SEO works well :)

#89 phantom forces hack on 07.04.19 at 6:24 pm

Intresting, will come back here again.

#90 dego pubg hack on 07.05.19 at 6:37 am

Deference to op , some superb selective information .

#91 tom clancy's the division hacks on 07.05.19 at 6:50 pm

Very interesting points you have remarked, appreciate it for putting up.

#92 synapse x roblox on 07.06.19 at 6:25 am

I am not rattling great with English but I get hold this really easygoing to read .

#93 Erasmo Hooton on 07.06.19 at 10:18 am

The next step of the mystery is to understand the order of the pyramid. This is your third secret clue. 517232125

#94 gx tool apk pubg uc hack download on 07.06.19 at 10:39 am

I am not rattling great with English but I get hold this really easygoing to read .

#95 rekordbox torrent download on 07.06.19 at 8:45 pm

Respect to website author , some wonderful entropy.

#96 call of duty black ops 4 activation key on 07.07.19 at 7:36 am

very cool post, i actually like this web site, carry on it

#97 spyhunter serial on 07.08.19 at 7:39 am

Respect to website author , some wonderful entropy.

#98 fps unlocker download on 07.09.19 at 9:14 am

I consider something really special in this site.

#99 quest bars cheap on 07.11.19 at 8:10 am

Hello there! Quick question that's entirely off topic.
Do you know how to make your site mobile friendly? My site
looks weird when browsing from my iphone 4. I'm trying to find a theme or plugin that might be able to
correct this problem. If you have any suggestions, please share.
Appreciate it!

#100 legal porno on 07.15.19 at 11:54 pm

great advice you give

#101 how to get help in windows 10 on 07.17.19 at 8:45 am

Appreciate this post. Will try it out.

#102 Bertram Erven on 07.17.19 at 10:21 pm

Alex9, this message is your next bit of info. Do transceive the agency at your earliest convenience. No further information until next transmission. This is broadcast #3718. Do not delete.

#103 how to get help in windows 10 on 07.18.19 at 7:07 am

Hurrah! After all I got a website from where I can actually get
useful data regarding my study and knowledge.

#104 plenty of fish dating site on 07.18.19 at 5:22 pm

Hey there, I think your website might be having browser compatibility issues.

When I look at your blog in Safari, it looks fine but when opening in Internet Explorer, it has some overlapping.

I just wanted to give you a quick heads up! Other then that, very good blog!

#105 ellie on 07.19.19 at 1:43 am

you are a great writer!

#106 Buy Drugs Online on 07.19.19 at 2:49 am

This blog is amazing! Thank you.

#107 plenty of fish dating site on 07.20.19 at 3:09 am

Good post. I learn something new and challenging on sites I stumbleupon every day.
It will always be interesting to read articles from other authors and practice a little something from
other web sites.

#108 how to get help in windows 10 on 07.20.19 at 6:14 pm

Awesome post.

#109 prodigy game files on 07.21.19 at 2:10 pm

I like this site because so much useful stuff on here : D.

#110 natalielise on 07.23.19 at 6:18 am

This excellent website truly has all the info I needed concerning this subject and didn't
know who to ask. natalielise plenty of fish

#111 acidswapper on 07.23.19 at 11:44 am

This does interest me

#112 plenty of fish dating site on 07.23.19 at 6:02 pm

whoah this blog is wonderful i love studying your articles.
Keep up the great work! You realize, a lot of persons are
looking around for this information, you could aid them greatly.

#113 date coiugar on 07.23.19 at 10:39 pm

I am 43 years old and a mother this helped me!

#114 fortnite skin changer by darkshoz on 07.24.19 at 12:11 pm

Me enjoying, will read more. Thanks!

#115 skisploit on 07.25.19 at 1:45 pm

I simply must tell you that you have an excellent and unique web that I really enjoyed reading.

#116 on 07.26.19 at 8:20 am

Hello there! This is my first visit to your blog! We are a collection of volunteers and starting a new project in a community in the same niche.
Your blog provided us useful information to work on. You
have done a extraordinary job! natalielise pof

#117 skisploit on 07.26.19 at 2:48 pm

Thank You for this.