"Multiprocessing", "multi-threading", "parallelism", "concurrency" etc. etc. can give you two kinds of benefits:
- Doing many things at once – 1000 multiplications every cycle.
- Waiting for many things at once – wait for 1000 HTTP requests just issued.
Some systems help with one of these but not the other, so you want to know which one – and if it's the one you need.
For instance, CPython has the infamous GIL – global interpreter lock. To what extent does the GIL render CPython "useless on multiple cores"?
- Indeed you can hardly do many things at once – not in a single pure Python process. One thread doing something takes the GIL and the other thread waits.
- You can however wait for many things at once just fine – for example, using the multiprocessing module (pool.map), or you could spawn your own thread pool to do the same. Many Python threads can concurrently issue system calls that wait for data – reading from TCP sockets, etc. Then instead of 1000 request-wait, request-wait steps, you issue 1000 requests and wait for them all simultaneously. Could be close to a 1000x speed-up for long waits (with a 1000-thread worker pool; more on that below). Works like a charm.
So GIL is not a problem for "simultaneous waiting" for I/O. Is GIL a problem for simultaneous processing? If you ask me – no, because:
- If you want performance, it's kinda funny to use pure Python and then mourn the fact that you can't run, on 8 cores, Python code that's 30-50x slower than C to begin with.
- On the other hand, if you use C bindings, then the C code could use multiple threads actually running on multiple cores just fine; numpy does it if properly configured, for instance. Numpy also uses SIMD/vector instructions (SSE etc.) – another kind of "doing many things at once" that pure Python can't do regardless of the GIL.
So IMO Python doesn't have as bad a story in this department as it's reputed to have – and if it does look bad to you, you probably can't tolerate Python's slowness doing one thing at a time in the first place.
So Python – or C, for that matter – is OK for simultaneous waiting, but is it great? Probably not as great as Go or Erlang – which let you wait in parallel for millions of things. How do they do it? Cheap context management.
Context management is a big challenge of waiting for many things at once. If you wait for a million things, you need a million sets of variables keeping track of what exactly you're waiting for (has the header arrived? then I'm waiting for the query. has it arrived? then I ask the database and wait for it etc. etc.)
If those variables are thread-local variables in a million threads, then you run into one of the problems with C – and hence OS-supported threads designed to run C. The problem is that C has no idea how much stack it's gonna need (because of the halting problem, so you can't blame C); and C has no mechanism to detect that it ran out of stack space at runtime and allocate some more (because that's how its ABIs have evolved; in theory C could do this, but it doesn't.)
So the best thing a Unixy OS could do is, give C one page for the stack (say 4K), and make say the next 1-2M of the virtual address space unaccessible (with 64b pointers, address space is cheap). When C page-faults upon stack overflow, give it more physical memory – say another 4K. This method means at least 4K of allocated physical memory per thread, or 4G for a million threads – rather wasteful. (I think in practice it's usually way worse.) All regardless of us often needing a fraction of that memory for the actual state.
And that's before we got to the cost of context switching – which can be made smaller if we use setjmp/longjmp-based coroutines or something similar, but that wouldn't help much with stack space. C's lax approach to stack management – which is the way it is to shave a few cycles off the function call cost – can thus make C terribly inefficient in terms of memory footprint (speed vs space is generally a common trade-off – it's just a bad one in the specific use case of "massive waiting" in C).
So Go/Erlang don't rely on the C-ish OS threads but roll their own – based on their stack management, which doesn't require a contiguous block of addresses. And AFAIK you really can't get readable and efficient "massive waiting" code in any other way – your alternatives, apart from the readable but inefficient threads, are:
- Manual state machine management – yuck
- Layered state machines as in Twisted – better, but you still have callbacks looking at state variables
- Continuation passing as in Node.js – perhaps nicer still, but still far from the smoothness of threads/processes/coroutines
The old Node.js slides say that "green threads/coroutines can improve the situation dramatically, but there is still machinery involved". I'm not sure how that machinery – the machinery in Go or Erlang – is any worse than the machinery involved in continuation passing and event loops (unless the argument is about compatibility more than efficiency – in which case machinery seems to me a surprising choice of words.)
Millions of cheap threads or whatever you call them are exciting if you wait for many events. Are they exciting if you do many things at once? No; C threads are just fine – and C is faster to begin with. You likely don't want to use threads directly – it's ugly – but you can multiplex tasks onto threads easily enough.
A "task" doesn't need to have its own context – it's just a function bound to its arguments. When a worker thread is out of work, it grabs the task out of a queue and runs it to completion. Because the machine works - rather than waits - you don't have the problems with stack management created by waiting. You only wait when there's no more work, but never in the middle of things.
So a thread pool running millions of tasks doesn't need a million threads. It can be a thread per core, maybe more if you have some waiting – say, if you wait for stuff offloaded to a GPU/DSP.
I really don't understand how Joe Armstrong could say Erlang is faster than C on multiple cores, or things to that effect, with examples involving image processing – instead of event handling which is where Erlang can be said to be more efficient.
Finally, a hardware-level example – which kind of hardware is good at simultaneous working, and which is good at simultaneous waiting?
If your goal is parallelizing work, eventually you'll deteriorate to SIMD. SIMD is great because there's just one "manager" – instruction sequencer – for many "workers" – ALUs. CPUs, DSPs and GPUs all have SIMD. NVIDIA calls its ALUs "cores" and 16-32 ALUs running the same instruction "threads", but that's just shameless marketing. A "thread" implies, to everyone but a marketeer, independent control flow, while GPU "threads" march in lockstep.
In practice, SIMD is hard despite thousands of man-years having been invested into better languages and libraries – because telling a bunch of dumb soldiers marching in lockstep what to do is just harder than running a bunch of self-motivating threads each doing its own thing.
(Harder in one way, easier in another: marching in lockstep precludes races – non-deterministic, once-in-a-blue-moon, scary races. But races of the kind arising between worker threads can be almost completely remedied with tools. Managing the dumb ALUs can not be made easier with tools and libraries to the same extent – not even close. Where I work, roughly there's an entire team responsible for SIMD programming, while threading is mostly automatic and bugs are weeded out by automated testing.)
If, however, you expect to be waiting much of the time – for memory or for high-latency floating point operations, for instance – then hoards of hardware threads lacking their own ALUs, as in barrel threading or hyper-threading, can be a great idea, while SIMD might do nothing for you. Similarly, a bunch of weaker cores can be better than a smaller number of stronger cores. The point being, what you really need here is a cheap way to keep context and switch between contexts, while actually doing a lot at once is unlikely to be possible in the first place.
Conclusions
- Doing little or nothing while waiting for many things is both surprisingly useful and surprisingly hard (which took me way too long to internalize both in my hardware-related and software/server-related work). It motivates things looking rather strange, such as "green threads" and hardware threads without their own ALUs.
- Actually doing many things in parallel – to me the more "obviously useful" thing – is difficult in an entirely different way. It tends to drag in ugly languages, intrinsics, libraries etc. about as much as having to do one single thing quickly. The "parallelism" part is actually the simplest (few threads so easy context management; races either non-existent [SIMD] or very easy to weed out [worker pool running tasks])
- People doing servers (which wait a lot) and people doing number-crunching (work) think very differently about these things. Transplanting experience/advice from one area to the other can lead to nonsensical conclusions.
See also
Parallelism and concurrency need different tools – expands on the reasons for races being easy to find in computational code – but impossible to even uniformly define for most event handling code.
122 comments ↓
One interesting solution to this in the parallel computing space is the object based decomposition and Structured Dagger (SDAG) control-flow mechanism implemented in the Charm++ parallel programming environment. Each concurrent execution context is a user-defined object driven by asynchronous calls to particular 'entry methods'. Thus, the state for processing a message is given by both the statically known members of the recipient object and the contents of the message itself – no stack problem. When those messages need to be processed in a particular arrangement, SDAG generates code inside each object to store them and consume them as intended.
Thus, when an object's control flow needs to 'block' waiting for some remote message to arrive, the generated code sets up its own continuation in its internal structures, and then simply returns from the executing method. The runtime scheduler driving each worker thread then continues by picking up some other message, and delivering it to whatever object it's directed to. Hence, 'context switching' purely by function call/return.
http://ppl.cs.illinois.edu/research/dagger
http://ppl.cs.illinois.edu/papers/13-36 (section 3.2)
http://charm.cs.illinois.edu/manuals/html/charm++/5.html
Interesting stuff. Could you link to a page with simple example code doing something concrete? (5.html above has code doing something rather abstract – Input a, Input b etc.; I think it'd be easiest to get a gut feeling of how this kind of approach stacks up compared to others by looking at example code doing something concrete – sort something, filter something, send/receive something etc., something an application programmer could be doing.)
I'm not going to get into the whole threads / continuation passing / etc. (I see threads as CPS with the continuation address stored on the stack, FWIW – the fact that the stack is a big array causes most of the problems) – I just wanted to get a bit "well, actually" on Python GIL.
You argue that the GIL isn't important because if you want performance you shouldn't be using Python. But IMO that's not the sequence of events. You start out using something like Python because it's productive, there's heaps of libraries that are available and easily installed for it, and you can glue them together into a solution that works well. Over time it gets more featured, bigger, and slower. Where things like the GIL hit you is that you can't use multithreading to get much of that performance back.
You can't simply write it in C to begin with, because that's a major engineering effort – half the libraries you're using don't use C, some of them are pure Python, some of them use C++, etc. You can't even extract out the inner loop, because there is no hot loop.
You say "if you want performance" – but that's a false question. Everybody wants performance. Performance is just another requirement to be prioritized, and if something isn't performed in a finite time, it might as well not be performed at all. But as you do more and more work – all software grows over time – the time required to do this work tends to increase, but the time allotted tends not to increase quite as liberally. Eventually you hit the deadline, and performance moves up the priority list.
tl/dr: performance always matters, even if you're starting out 50x slower – you're getting the 50x back somewhere else.
Well, the thing is, if you use C bindings on an 8-core machine you'll get your 8x times the 30-50x spent on using Python. Given that Python does not have particularly good tools for debugging multi-threaded code – C does better there – it makes sense to first implement your bottleneck in C and then maybe do multi-threading even in a world without the GIL:
* Moving things to C gives a larger performance boost than multithreading
* It is less buggy than multithreading
* If you want to parallelize the code then doing it in C is easier to get right so you'd want to move to C anyway
That said, I realize that temptation exists to just somehow multithread whatever it is you've coded up in Python under time pressure, so I see your point…
TempleOS has master/slave multicore support, not SMP. For real-time graphics, you want to break a problem like a flight simulator into 8 pieces and have each core do one strip of land. You do not want the OS scheduler to move tasks between cores. 7 cores with an extra job doubled-up is not the same as 8 cores!! Putting two tasks on one core makes refresh time 30fps instead of 60fps!
Here is a flight simulator:
https://www.youtube.com/watch?v=FxEq6IM43sA&feature=player_embedded
The code for the flight simulator is here:
http://www.templeos.org/Wb/Demo/Games/EagleDive.html
TempleOS:
http://www.templeos.org
TempleOS does master/slave multicore, not SMP. For my flight simulator, I put one task on each of the 8 cores. I do NOT want the scheduler to change it. If the scheduler put 2 tasks on one core instead of one task on each of 8 cores, that would cause the refresh to glitch and wait a whole nother cycle!!
Here's the SDAG code for parallel LU factorization without pivoting (for demonstration only, since it would be fine with real numbers, but can be numerically junk in floating point):
http://charm.cs.uiuc.edu/gerrit/gitweb?p=charmlu.git;a=blob;f=lu.ci;h=6d9f5aa0b5dce28982d14fd73bdf86cb0db2c053;hb=31fb59692d2d5ee337cb0ad3e115cd8cf3b9fc16
Here's the correct numerical code that implements 'partial' pivoting:
http://charm.cs.uiuc.edu/gerrit/gitweb?p=charmlu.git;a=blob;f=lu.ci;h=9a1d0650191320efce5172dd31210b760c740a18;hb=HEAD
Excellent untangling of these two problem domains. I'll just avoid using the words "concurrency" and "parallelism" altogether. There are just too many competing definitions and if you look in the dictionary – "at the same time" is mentioned in both. The potential for misinterpretation is far too great.
As is accepted for "function", "routine", "procedure" and "subroutine". If you try and convey different ideas using words that are so strongly related, or are so broadly used to mean the same thing – you're going to have a bad time.
So I'm adopting your "waiting for many things at once" vs "doing many things at once".
PS nodejs has a GIL though it's less famous perhaps because it doesn't have a name.
I thought Node.js was actually completely single-threaded because JS is; I guess you could call that the ultimate GIL…
Well i think it also depends on how you are allocating resources when you are creating a specific thread, if it has a low footprint then you could achieve big improvements.
As reported in a node.js module, kernel threads where way faster than simple concurrency.
"If you want performance, it's kinda funny to use pure Python and then mourn the fact that you can't run, on 8 cores, Python code that's 30-50x slower than C to begin with."
Not at all. Python programmers don't measure themselves against the arbitrary yardstick of "C", just like C programmers don't measure themselves against assembly language or machine language. Faster is always better, even if it's not as fast as some other environment.
When the JVM upgraded from green threads to native threads, it became possible to write "working simultaneously" programs in pure Java (though they still wouldn't be as fast as C, at least at the time). Programmers now had the *choice* of using C FFI or pure Java for their threading needs — and I know of zero who use C FFI today, given this choice. Whatever the technical merits may be, it seems that the population in general would rather use their existing language, than escape to C, even if it's slower.
When I write a program for other people, and most of my users tell me that they really wish it worked a different way, my response is not to dig in my heels and insist that this way is technically superior. I change my program to work the way they think it should.
"Moving things to C gives a larger performance boost than multithreading"
Yes or no. This depends entirely on the extent to which the problem is parallelizable.
"It is less buggy than multithreading. If you want to parallelize the code then doing it in C is easier to get right so you'd want to move to C anyway"
Am i reading this correctly? Are you saying C coding is the *less* buggy alternative? I'm a longtime C programmer and I strongly disagree.
Threading in an HLL isn't terribly hard — worst case, you add too many locks, and it doesn't run as efficiently as it could.
Doing *anything* in C is an exercise in getting every detail perfect, something which even seasoned professionals have trouble with — worst case, your process has a security hole the size of Nebraska, and half the internet needs an immediate software update.
Well, Python has a "special relationship" with C, as acknowledged but its creator, and it assumes you use C much more than Java does – Java very much prefers you to stay "pure" IMO. I think that for a Java programmer it is more sensible to only know Java and not C than it is for a Python programmer to only know Python and not C; for one thing, Python is way way slower than Java.
As to multi-threaded code vs serial C code – I do prefer the latter, and if I have to do multiple threads, the last thing I would reach to would be explicit locks, and have enough experience with both serial C code and multithreading to be entitled to my own opinion, though of course others are entitled to theirs.
If Python lost the GIL, letting one gun for an 8x less latency with threads (and 1x higher throughput, or less… using multiple cores means you'll need more cores to transfer the speedup benefits to the user, eventually) with threads instead of a 30-50x less latency and more throughput with C, I wouldn't object. What I would like to see then though, is good frameworks atop threads (not unlike pool.map) and good race detectors (currently I don't know about these for Python). And even then I'd hardly recommend it for the hard cases. For easy cases with trivial parallelization and not that much code and just one developer touching that code – perhaps.
Response to #4 above:
Hi Yossi,
I agree fundamentally with your position here, but if the context is a programmer or team that isn't expert in C, then Barry's point carries more weight. There is a bit of a price for introducing C into a pure Python program, because even if all the interfaces are good, you have the worry of debugging / configured ng that code.
(I'm speaking in the abstract; I'm much more conversant with C than Python)
wrt. "Layered state machines as in Twisted – better, but you still have callbacks looking at state variables"
You might want to read
https://glyph.twistedmatrix.com/2014/02/unyielding.html
which goes over possible concurrency mechanisms available for Python and Twisted (including implicit coroutines you seem to like) and points out why you actually don't want implicit context switches in your code.
I don't know if it's just me or if perhaps everyone else experiencing
issues with your website. It appears like some of the text within your
posts are running off the screen. Can someone else please comment and let me know
if this is happening to them too? This might be a problem with my
web browser because I've had this happen before.
Cheers
I don't even know how I ended up right here, however I thought
this submit was once good. I do not know who you might be but
certainly you're going to a well-known blogger in the event you aren't already.
Cheers!
Can I simply just say what a relief to find an individual who truly understands what they are discussing on the internet.
You certainly understand how to bring a problem to light and make it important.
A lot more people should look at this and understand this side
of your story. I can't believe you're not more
popular because you certainly possess the gift.
Howdy just wanted to give you a quick heads up and let you know a few of the
images aren't loading properly. I'm not sure why but I think its
a linking issue. I've tried it in two different browsers and both show
the same results.
I think this is among the most significant information for me.
And i'm glad reading your article. But should remark on some
general things, The website style is great, the articles is really nice :
D. Good job, cheers
These are really impressive ideas in regarding blogging.
You have touched some fastidious things here.
Any way keep up wrinting.
I don't know if it's just me or if everyone else encountering
problems with your blog. It appears as though some of the written text on your content are running off the screen. Can somebody else please comment and let me know if this is happening to them too?
This may be a issue with my web browser because I've had this happen previously.
Thank you
I got this web page from my pal who informed me about this site and at the
moment this time I am visiting this web page and reading very informative content here.
If some one desires expert view regarding blogging then i propose him/her to
go to see this website, Keep up the fastidious job.
I am regular reader, how are you everybody?
This article posted at this website is actually good.
In my estimation, yosefk.com does a great job of handling issues of this sort! While frequently deliberately controversial, the material posted is more often than not well-written and thought-provoking.
Cheers, great stuff, I like.
This information is invaluable. When can I
find out more?
I was suggested this web site by my cousin. I'm not sure whether this post
is written by him as nobody else know such detailed about my difficulty.
You're wonderful! Thanks!
This article will assist the internet viewers for creating new blog or even a blog from start to
end.
Wow, incredible blog structure! How lengthy have you
been running a blog for? you made running a blog glance easy.
The entire glance of your website is excellent, as neatly as the content material!
I simply could not leave your site before suggesting that I
really loved the usual information an individual provide to your visitors?
Is going to be again often to inspect new posts
6/7/2019 @ 6:16:13 PM In my opinion, yosefk.com does a excellent job of covering issues of this sort. Even if often intentionally polemic, the material posted is in the main well researched and challenging.
With havin so much content and articles do you ever
run into any issues of plagorism or copyright infringement?
My blog has a lot of completely unique content I've either written myself or outsourced
but it appears a lot of it is popping it up all over the web without my agreement.
Do you know any methods to help reduce content from being
stolen? I'd genuinely appreciate it.
I have read so many articles concerning the blogger lovers
however this post is really a nice article, keep it up.
Because the admin of this web page is working, no doubt very
shortly it will be famous, due to its feature contents.
Hi, i think that i saw you visited my web site so i came to “return the favor”.I am attempting to find things to improve my
website!I suppose its ok to use some of your ideas!!
This blog was… how do you say it? Relevant!!
Finally I have found something that helped me. Many thanks!
I like this article, because so much useful stuff on here : D.
My brother suggested I would possibly like this website.
He was once totally right. This put up truly made
my day. You can not consider just how so much time I had spent for this information! Thank you!
I must thank you for the efforts you have put in writing this website.
I'm hoping to check out the same high-grade content from you later on as well.
In truth, your creative writing abilities has encouraged me to get my very own blog now ;)
I consider something really special in this site.
Cheers, great stuff, Me like.
I'm more than happy to discover this site. I wanted to thank you for your time for this particularly
fantastic read!! I definitely enjoyed every part of
it and I have you book marked to look at new information in your site.
I'm really impressed with your writing skills as
well as with the layout on your blog. Is this a paid theme or did you modify it yourself?
Either way keep up the excellent quality writing,
it's rare to see a great blog like this one these days.
Great, google took me stright here. thanks btw for post. Cheers!
Everything is very open with a very clear clarification of the issues.
It was really informative. Your website is very useful.
Many thanks for sharing!
whoah this blog is magnificent i like studying your posts.
Stay up the great work! You understand, many persons are
looking around for this info, you can aid them greatly.
I am glad to be one of the visitors on this great website (:, appreciate it for posting .
I was looking at some of your articles on this site and I believe this internet site is really instructive! Keep on posting .
Piece of writing writing is also a excitement, if you be acquainted with after that you can write or else it is complicated to write.
Great, this is what I was browsing for in bing
Just wanna input on few general things, The website layout is perfect, the articles is very superb : D.
Hi there everyone, it's my first pay a quick visit at this web site, and paragraph is actually fruitful
designed for me, keep up posting these types of articles or reviews.
You actually make it appear really easy along with your presentation however I find this topic to be really something which
I feel I might by no means understand. It seems too complicated and very extensive for me.
I'm looking forward in your next put up, I'll attempt to get the hold of it!
This is interesting!
If you are going for best contents like me, only pay a quick visit this web site every day for the reason that it
offers quality contents, thanks
Respect to website author , some wonderful entropy.
I really got into this site. I found it to be interesting and loaded with unique points of interest.
This is my first time go to see at here and i am actually pleassant to read all at single place.
I constantly emailed this web site post page to all my associates, as if like to read it next my links will too.
My family members every time say that I am killing my time here at web,
but I know I am getting knowledge all the time by reading thes good
articles or reviews.
Hello, yeah this post is really good and I have learned lot of things from it on the
topic of blogging. thanks.
I conceive this web site holds some real superb information for everyone : D.
Respect to website author , some wonderful entropy.
Deference to op , some superb selective information .
all the time i used to read smaller content that also
clear their motive, and that is also happening with this piece
of writing which I am reading at this time.
Deference to op , some superb selective information .
Parasite backlink SEO works well :)
Thanks for this post. I definitely agree with what you are saying.
yahoo took me here. Thanks!
Amazing! This blog looks just like my old one!
It's on a completely different subject but it has pretty much the same layout and design. Superb choice of colors!
This helps. Thanks!
First off I would like to say terrific blog! I had a quick question in which I'd like to ask if you do not mind.
I was curious to find out how you center yourself and clear your head prior to
writing. I've had trouble clearing my thoughts in getting
my thoughts out there. I do enjoy writing however it
just seems like the first 10 to 15 minutes are wasted simply
just trying to figure out how to begin. Any
recommendations or tips? Thank you!
Interesting blog! Is your theme custom made
or did you download it from somewhere? A design like
yours with a few simple adjustements would really make my blog shine.
Please let me know where you got your theme. Thanks
Whoa! This blog looks just like my old one! It's on a completely
different topic but it has pretty much the same layout and design.
Great choice of colors!
What's up, yeah this piece of writing is in fact pleasant
and I have learned lot of things from it on the topic of blogging.
thanks.
You got yourself a new rader.
Intresting, will come back here more often.
Your style is really unique compared to other people I have read stuff from.
Many thanks for posting when you have the opportunity, Guess I will just
book mark this web site.
Hi! Do you know if they make any plugins to assist with SEO?
I'm trying to get my blog to rank for some targeted keywords but I'm not seeing very good success.
If you know of any please share. Thank you!
When some one searches for his essential thing, thus
he/she wishes to be available that in detail, so
that thing is maintained over here.
Great writing to see, glad that Yahoo led me here, Keep Up nice job
Wow, marvelous weblog layout! How long have you ever been running a blog for?
you made blogging look easy. The entire glance of your
site is wonderful, let alone the content!
thx for sharing this!
I am 45 and a mother and I need to read this thank you.
Thank you for the great read!
some great ideas this gave me!
great advice you give
I do not know if it's just me or if perhaps everyone else experiencing problems with your
website. It seems like some of the text on your content are running off the
screen. Can somebody else please comment and let me know if this is
happening to them too? This could be a problem with my browser because I've had this happen before.
Appreciate it
Great weblog here! Additionally your web site loads up fast!
What host are you the usage of? Can I get your associate hyperlink in your host?
I desire my web site loaded up as quickly as yours lol
I read this paragraph completely concerning the resemblance of most
up-to-date and earlier technologies, it's awesome article.
Hi! I just wanted to ask if you ever have any trouble with hackers?
My last blog (wordpress) was hacked and I ended up losing
months of hard work due to no backup. Do you have any solutions to prevent hackers?
Very energetic blog, I liked that a lot. Will there be a part
2?
Pretty! This has been an extremely wonderful post.
Thank you for supplying this info.
I'm not sure exactly why but this site is loading extremely slow for me.
Is anyone else having this issue or is it a problem on my end?
I'll check back later and see if the problem
still exists.
great post looking forward to more
This blog is amazing! Thank you.
Alex9, this note is your next bit of information. Immediately message the agency at your earliest convenience. No further information until next transmission. This is broadcast #4649. Do not delete.
Pretty nice post. I just stumbled upon your blog
and wanted to say that I've really enjoyed surfing around your
blog posts. After all I will be subscribing to your feed and I hope you write again soon!
What's up friends, how is everything, and what you wish
for to say about this post, in my view its really remarkable for me.
Very energetic post, I enjoyed that a lot. Will there
be a part 2?
Please let me know if you're looking for a article author for
your site. You have some really great articles and I think I would be a good asset.
If you ever want to take some of the load off, I'd love to
write some content for your blog in exchange for a
link back to mine. Please send me an email if interested.
Regards!
very nice post, i actually enjoyed this web site, carry on it
Very nice post. I just stumbled upon your blog and wished to say that I have really enjoyed surfing around your blog posts.
In any case I'll be subscribing to your feed and I hope you write again very soon!
natalielise pof
It's awesome designed for me to have a site,
which is valuable in favor of my know-how.
thanks admin
What's up, after reading this amazing piece of writing i am also cheerful to share my experience here with friends.
I consider something really special in this site.
I am 43 years old and a mother this helped me!
I am 43 years old and a mother this helped me!
I am 43 years old and a mother this helped me!
I am 43 years old and a mother this helped me!
It is not my first time to visit this web site,
i am visiting this site dailly and take fastidious facts from here everyday.
Hey I know this is off topic but I was wondering if you knew of
any widgets I could add to my blog that automatically tweet my
newest twitter updates. I've been looking for a plug-in like this for
quite some time and was hoping maybe you would have some experience with something like this.
Please let me know if you run into anything. I truly enjoy reading
your blog and I look forward to your new updates.
Hey, I think your site might be having browser compatibility
issues. When I look at your blog site in Ie, it looks
fine but when opening in Internet Explorer, it has some overlapping.
I just wanted to give you a quick heads up! Other then that, wonderful blog!
Just wanna input on few general things, The website layout is perfect, the articles is very superb : D.
I'm not sure where you are getting your information, but great topic.
I needs to spend some time learning more or understanding more.
Thanks for excellent information I was looking for this information for my mission.
Cheers, great stuff, Me like.
Write more, thats all I have to say. Literally, it seems as though you relied on the video to make your point.
You obviously know what youre talking about, why throw away
your intelligence on just posting videos to your blog when you could be giving us something informative to read?
natalielise pof
Your site has proven useful to me.
Thank you, I've just been searching for information approximately this subject
for ages and yours is the greatest I've came upon so far.
However, what concerning the bottom line? Are you certain concerning the source?
I really like your blog.. very nice colors & theme.
Did you design this website yourself or did you hire
someone to do it for you? Plz respond as I'm looking to construct my
own blog and would like to know where u got this from. thank you