Suppose you run on the x86 and you don't like its default FPU settings. For example, you want your programs to dump core when they divide by zero or compute a NaN, having noticed that on average, these events aren't artifacts of clever numerical algorithm design, but rather indications that somebody has been using uninitialized memory. It's not necessarily a good idea for production code, but for debugging, you can tweak the x86 FPU thusly:
//this is a Linux header using GNU inline asm
#include <fpu_control.h>
void fpu_setup() {
unsigned short cw;
_FPU_GETCW(cw);
cw &= ~_FPU_MASK_ZM;//Divide by zero
cw &= ~_FPU_MASK_IM;//Invalid operation
_FPU_SETCW(cw);
}
So you call this function somewhere during your program's initialization sequence, and sure enough, computations producing NaN after the call to fpu_setup result in core dumps. Then one day someone computes a NaN before the call to fpu_setup, and you get a core dump the first time you try to use the FPU after that point. Because that's how x86 maintains its "illegal operation" flags and that's how it uses them to signal exceptions.
The call stack you got is pretty worthless as you're after the context that computed the NaN, not the context that got the exception because it happened to be the first one to use the FPU after the call to fpu_setup. So you move the call to fpu_setup to the beginning of main(), but help it does not. That's because the offending computation happens before main, somewhere in the global object construction sequence. The order of execution of the global object constructors is undefined by the C++ standard. So if you kindly excuse my phrasing – where should we shove the call to fpu_setup?
If you have enough confidence in your understanding of the things going on (as opposed to entering hair-pulling mode), what you start looking for is the REAL entry point. C++ is free to suck and execute parts of your program in "undefined" (random) order, but a computer still executes instructions in a defined order, and whatever that order is, some instructions ought to come first. Since main() isn't the real entry point in the sense that stuff happens before main, there ought to be another function which does come first.
One thing that could work is to add a global object to each C++ translation unit, and have its constructor call fpu_setup(); one of those calls ought to come before the offending global constructor – assuming that global objects defined in the same translation unit will be constructed one after another (AFAIK in practice they will, although in theory the implementation could, for example, order the constructor calls by the object name, so they wouldn't). However, this can get gnarly for systems with non-trivial build process and/or decomposition into shared libraries. Another problem is that compilers will "optimize away" (throw away together with the side effects, actually) calls to constructors of global objects which aren't "used" (mentioned by name). You can work around that by generating code "using" all the dummy objects from all the translation units and calling that "using" code from, say, main. Good luck with that.
The way I find much easier is to not try to solve this "portably" by working against the semantics prescribed by the C++ standard, but instead rely on the actual implementation, which usually has a defined entry point, and a bunch of functions known to be called by the entry point before main. For example, the GNU libc uses a function called __libc_start_main, which is eventually called by the code at _start (the "true" entry point containing the first executed instruction, AFAIK; I suck at GNU/Linux and only know what was enough to get by until now.) In general, running `objdump -T <program> | grep start` (which looks for symbols from shared libraries – "nm <program>" will miss those) is likely to turn up some interesting function. In these situations, some people prefer to find out from the documentation, others prefer to crawl under a table and die of depression; the grepping individuals of my sort are somewhere in between.
Now, instead of building (correctly configure-ing and make-ing) our own version of libc with __libc_start_main calling the dreaded fpu_setup, we can use $LD_PRELOAD – an env var telling the loader to load our library first. If we trick the loader into loading a shared library containing the symbol __libc_start_main, it will override libc's function with the same name. (I'm not very good at dynamic loading, but the sad fact is that it's totally broken, under both Windows and Unix, in the simple sense that where a static linker would give you a function redefinition error, the dynamic loader will pick a random function of the two sharing a name, or it will call one of them from some contexts and the other one from other contexts, etc. But if you ever played with dynamic loading, you already know that, so enough with that.)
Here's a __libc_start_main function calling fpu_setup and then the actual libc's __libc_start_main:
#include <dlfcn.h>
typedef int (*fcn)(int *(main) (int, char * *, char * *), int argc, char * * ubp_av, void (*init) (void), void (*fini) (void), void (*rtld_fini) (void), void (* stack_end));
int __libc_start_main(int *(main) (int, char * *, char * *), int argc, char * * ubp_av, void (*init) (void), void (*fini) (void), void (*rtld_fini) (void), void (* stack_end))
{
fpu_setup();
void* handle = dlopen("/lib/libc.so.6", RTLD_LAZY | RTLD_GLOBAL);
fcn start = (fcn)dlsym(handle, "__libc_start_main");
(*start)(main, argc, ubp_av, init, fini, rtld_fini, stack_end);
}
Pretty, isn't it? Most of the characters are spent on spelling the arguments of this monstrosity – not really interesting since we simply propagate whatever args turned up by grepping/googling for "__libc_start_main" to the "real" libc's __libc_start_main. dlopen and dlsym give us access to that real __libc_start_main, and /lib/libc.so.6 is where my Linux box keeps its libc (I found out using `ldd <program> | grep libc`).
If you save this to a fplib.c file, you can use it thusly:
gcc -o fplib.so -shared fplib.c
env LD_PRELOAD=./fplib.so <program>
And now your program should finally dump core at the point in the global construction sequence where NaN is computed.
This approach has the nice side-effect of enabling you to "instrument" unsuspecting programs without recompiling them s.t. they run with a reconfigured FPU (to have them crash if they compute NaNs, unless of course they explicitly configure the FPU themselves instead of relying on what they get from the system.) But there are niftier applications of dynamic preloading, such as valgrind on Linux and .NET on Windows (BTW, I don't know how to trick Windows into preloading, just that you can.) What I wanted to illustrate wasn't how great preloading is, but the extent to which C++, the language forcing you to sink that low just to execute something at the beginning of your program, SUCKS.
Barf.
Corrections - thanks to the respective commenters for these:
1. Section 3.6.2/1 of the ISO C++ standard states, that “dynamically initialized [objects] shall be initialized in the order in which their definition appears in the translation unit”. So at least you have that out of your way if you want to deal with the problem at the source code level.
2. Instead of hard-coding the path to libc.so, you can pass RTLD_NEXT to dlsym.
142 comments ↓
Good stuff. I'd be interested to know how to do this on Windows; I'm going to research that at some point, and maybe I'll post about it on my blog if and when I succeed.
I'm currently doing battle with a static library that uses global objects with non-trivial constructors that allocate memory, which doesn't interact too nicely with our memory manager. We're currently using lazy initialization of the memory manager, but shutting it down and checking for leaks on exit is problematic – not even using atexit() works, since sometimes some global objects manage to register their destructors to run after the memory manager shuts down and *BOOM*.
The man himself wrote a whole section about the problem of global object initialization order in "The Design and Evolution of C++" but doesn't really offer a solution.
I realize you have to continue serving your function as a high level C++ critic but your post really doesn't have much to do with C++, though, does it? What you want is better control over the linker/loader which is really an OS thing. I'm not even sure if you can cover all cases because some other process might have already loaded some of the dynamic libraries.
You might want to look into the GNU linker's –wrap function. It looks like it does exactly what you want, though I don't know that it will work in a low level function like you are looking for. If it *does* work then you at least can launch without the LD_PRELOAD env var stuff.
"info ld invocation options" to bring up the info page then search for "wrap".
Dynamic loading is not broken. It has well defined semantics. Read up on dlopen() and dlsym() in the Single Unix Specification v2 or v3. For example, there is no need to dlopen() libc explicitly in your code, you should just use dlsym() with RTLD_NEXT as handle.
@queisser: "I realize you have to continue serving your function as a high level C++ critic but your post really doesn’t have much to do with C++, though, does it? What you want is better control over the linker/loader which is really an OS thing."
No, C++'s ability to execute static constructors before main() really does suck, and IMO really does exacerbate the problem of finding the "real entry point" of a non-trivial program. It's true that even C programs execute a lot of code before main() — code that initializes the heap, sets up stdin and stdout, and whatnot — but that code is generally written by trusted sources(TM) and can basically be ignored when debugging. C++ allows Joe Random Programmer to insert code before main(), which is far, far worse.
Now, I'd argue that the answer is Don't Do That. If yosefk had had the foresight and (probably more critically) authority to enforce a project-wide coding standard that forbids the definition of any static object with a constructor or destructor, then he wouldn't have had to hack around the problem this way. (Hindsight is 20/20, yeah.)
Nice article. One thing to consider trying is rather than hardcoding the path to libc, you can use the pseudo-handle RTLD_NEXT in your call to dlsym().
.NET executables have a traditional entrypoint that jumps to mscoree.dll!_CorExeMain which kicks off managed code execution, so I don't think this counts as magic of the same order as LD_PRELOAD.
Regarding global constructors + custom memory manager: yeah, been there, two. Since it was on an embedded target, I ended up adding the memory manager initialization as another hack to the already hacked libc startup code.
Regarding RTLD_NEXT – thanks for the tip. Regarding "dynamic loading not being broken" – "defined behavior" isn't the opposite of "broken behavior". When I say "broken", I mean (1) that it's not "The Right Thing" (and there would be some hubris here if we weren't discussing something as trivial as detecting redefinition, where The Right Thing is damn easy to define), and (2) the fact that actual compilers out there generating shared objects produce output compatible with the spec of shared objects doesn't make that output compatible with the spec of their source language (try throwing a C++ exception from one .so file and catch it in a caller function located in another .so file and you'll get the idea.)
Regarding this not being a C++ issue – as mentioned above, it is. Do you have a problem doing something "at the [real] beginning" of your C or Lisp or Python program?
Regarding my presumed duty to criticize C++ – um, how do I put this. I get to use the fucking shit a lot. When I no longer do, people will have to get C++ hate elsewhere.
FYI
> assuming that global objects defined in the same
> translation unit will be constructed one after another
> (AFAIK in practice they will, although in theory the
> implementation could, for example, order the
> constructor calls by the object name, so they
> wouldn’t)
Section 3.6.2/1 of the ISO C++ standard states, that "dynamically initialized [objects] shall be initialized in the order in which their definition appears in the translation unit"
Thanks!
I should update the article to quote this.
I don't know about Linux, but dyld on Mac OS X lets you delcare a function with the "constructor" attribute, i.e.
void do_something(void) __attribute__((constructor))
It'll get called by the dynamic linker before entering main(). But I'm not sure how it gets called in relation to C++'s static constructors. They get called after your program's image has been loaded, since you can initialize globals in said constructors.
Would it have been possible for you to do the FPU tweaks and then re-exec(2)? If the FPU control word is set on a per-address space basis, that should do the trick.
@Damien: interesting, I didn't think about either. I'd guess __attribute__((constructor)) simply adds the address of the function to the .init section, so it gets called at some undefined point during the pre-main initialization sequence. Regarding exec(2) – I don't know whether the FPU mask is supposed to survive that, but it's a pretty violent measure – for example, if someone prints before main(), the text will be printed twice, etc. That is, it's basically OK to do this only if you assume that the program's init sequence is "tame" enough – in which case you wouldn't have the trouble of taming it in the first place, or something.
Why not add a floating point operation to the end of fpu_setup, like x=1.0+2.0; (with appropriate un-optimization settings) Then at least if you have a pending exception you will get your core dump when fpu_setup is called, not at some unknown later time when another function tries to do f.p.?
You'd stll have to debug your constructors without benefit of core dumps on NaNs, but seems like fair trade for not having to trick the loader into doing something it doesn't want to do, and how many constructors need to do f.p., anyway?
@Matt: It's a good idea to amend fpu_setup with an fp operation to save the head scratching when the program fails upon its first attempt to use fp elsewhere. However, I vigorously reject the claim that debugging global constructors is reasonably easy without core dumps at the point of failure :) Seriously, a 1/5M-1M LOC program uses what, 500-2000 translation units? Each can instantiate globals, which can have constructors calling constructors ad nauseam, and this shit can depend on getenv or files or the command line (accessible via stuff like the /proc/ file system.) How am I going to shovel through all that, and where do I even start?
Of course I don't recommend to use the LD_PRELOAD shite in production environments, only for debugging.
Instead of doing it dynamically, you can do it statically. Compile your fpu_setup in a separate .o adding
asm(".section .initncall fpu_setup");
to it, then make sure you pass this as the first thing to the linker. Nothing is guaranteed by the language, of course, but the C run-time it's built on is pretty reliable.
Dynamic linking semantic of ELF is horrible indeed, Windows is much better though. It does not silently pick up the first definition — each dll is in it's own namespace and you explicitly specify from which dll you want your symbol.
And btw, inserting static constructor into each .cpp file (that includes some header file) is the classic trick used to initialize iostream library "before anything else", as you're allowed to use it from any static constructor.
This relies on the standard initialization order in one translation unit and absense of "optimizations". C++ compilers are not allowed to optimize away any constructors, static or not, as hell knows what it can do inside. The only thing it can do is "inline and dissolve".
I've had the pleasure to witness the iostream initialization trick under the unfortunate circumstances of running on butt-slow targets (RTL simulators.) Fun.
However, I distinctly remember gcc "optimizing" away global constructors – which meant I had to give up on "automatic registration" (where you have a global object that adds itself to some map before main to register a library with a framework – you know the drill.) That only worked if the global was defined in a .cpp whose .o was passed directly to the linker; archiving the .o into a .a caused it to be optimized away. (Now that I think of it, perhaps "touching" the global in the library code would help.) This seemed completely broken exactly because static constructors can have side effects, which in this case they actually did, so it seemed like a broken build, but I double-checked and found no way around this.
If it works in .o, but not in .a it obviously has nothing to do with gcc optimizing it. gcc has created the code, you can see it in .o, so it's done it's job. It does not know or care what you're doing with .o.
This is a standard static linker behaviour. Were you putting static constructor in it's own .o inside .a? This won't work, because linker won't pick .o from .a, unless something it already picked needs some symbol, defined in that .o. This has nothing to do with what constructor does or whether it's "used" at a language level (e.g. you can use it from the same .o, it won't help).
Nothing was ever updated for C++ in this scheme, and in fact it's not even clear how it should work, because .a has semantics of a bunch of independent .o files that can be picked at will, not an all-or-nothing module. As it's typical for C++, noone cares, you're supposed to figure this yourself.
iostream trick works because it only needs to initialize itself if something uses it. As it inserted itself into all .os if any of them get's picked the code will run. If not, this means noone's going to use iostream in this program. If you have a header file that's included into every .cpp (that uses floating point operations) in your program, and you're willing to recompile the whole thing, you could add fpu_setup in static constructor in that header. Same thing.
On the original subject. In presence of dynamic libraries _start is no longer the entry point. Now it's buried inside dynamic linker. You can exploit this with preloading much simpler. When ld.so loads dynamic library it calls library's _init function. You can either try using this directly or just write a static constructor. That is, static constructor in preloaded library runs before anything in your program.
And while at it, valgrind does not use preloading (it wouldn't be able to do half of what it's doing), it uses dynamic binary compilation.
By "gcc optimized it away", I didn't mean "gcc the compiler, as opposed to the assembler and the linker, did it in an optimization pass as opposed to the linkage phase", I meant "the GNU C++ implementation" – the whole toolchain which is supposed to implement C++, which in this particular case it doesn't.
valgrind uses preloading in order to begin its dynamic binary compilation.
Is it possible to write a stub program that calls fpu_setup() and then just exec(2)s the main program? That seems easier, and since it's the same PID to the OS it seems like the FPU settings should survive.
Perhaps the FPU CSR would survive exec() – didn't check it; actually the asm(”.section .initncall fpu_setup”) trick mentioned in a comment above is probably the easiest way to do it as that way, you simply have the FPU configured before application code is called as things should be without having to tweak the way someone runs the code.
The POSIX exec() spec sez:
"The state of the floating-point environment in the new process image or in the initial thread of the new process image shall be set to the default."
http://www.opengroup.org/onlinepubs/000095399/functions/exec.html
Come up with non-buggy code before complaining about C++ – your function should've reset the fpu state before setting the flags.
Actually what you should have done is… wait for it… not do floating point calculations in global constructors. I don't know why this trips up so many people. Global initialization order is undefined like you said, so why are you doing anything in global constructors int he first place? Make them global pointers and new them at the beginning of main, problem solved.
@Zach: it wasn't my code that ran before main.
This is not a problem with C++, this is a problem of BAD code. Code heavily depending on global objects is bad – period.
Even worse if there is no documentation which global objects is constructed – If there is, simply add breakpoints in those constructors, and you will find your bug.
And if you really need global objects, dont use any global objects with constructors – rather call an init-function at the beginning of main.
If you dont have control over main, have only 1 global object with constructor, which constructs the others in defined order AND document it.
I dont see why one would complain that bad code produces bad problems… Just dont use such code, write good code instead / fix it.
You'd be surprised how much effort was expended by the C++ community to make the order of initialization and destruction sequences deterministic and "correct" to some approximation (in the lack of an explicitly defined order as well as an inclination to think that such order should exist at all). So it's not "just" a bad code problem, it's a language-specific cultural problem interacting with a loose language spec.
On the other hand, if you like C++ and the one of its subcultures you're dealing with the most, then I'm sincerely happy for you.
I don't get why you need to wrap around __libc_start_main(), would it be just easier to invoke
fp_setup() in the pre-loaded shared lib?
E.g.
fplib.c:
int foo()
{
fpu_setup();
return 0;
}
int bar = foo();
(will need to build with g++ instead of gcc to allow non-const initializer element for initializing bar.)
It could work, I guess; I don't know how loading work enough to tell if bar=foo() is the first thing that will happen but perhaps it will be.
There is an elegant solution to this, I think I read it in Alexandrescu's Modern C++ Design.
You need a .h file that's included from all the .cpp files (with VisualC++ you typically have this anyway, because of using precompiled headers).
Make a class, which increments a static variable in its contructor. When this if first incremented, do the initialization you need. Make a static global variable of this class at the beginning of the header file. (This will result in each translation unit having a copy, but your initialization code will only run once and as the first thing, because within translation units the order of initialization is the order of declaration.)
Cool stuff at your page there.
I know this method; I wouldn't call it "elegant", but to each his own. One "objective" issue with it though is, it doesn't work if you use a library with global constructors and you have its header files and object code but no source code. Whereas in C (or Python or a whole lot of other languages) you simply have your first executing statement, the second etc. etc. even if you can't recompile all the code that you're using from source.
Ha, here from yahoo, this is what i was searching for.
This is good. Cheers!
Cheers, here from yahoo, me enjoyng this, will come back again.
Love the site– extremely user-friendly and lots to consider!
Thanks for this website. I definitely agree with what you are saying.
I am not rattling great with English but I get hold this really easygoing to read .
I am not rattling great with English but I get hold this really easygoing to read .
Appreciate it for this howling post, I am glad I observed this internet site on yahoo.
Intresting, will come back here later too.
I love reading through and I believe this website got some genuinely utilitarian stuff on it! .
Yeah bookmaking this wasn’t a risky decision outstanding post! .
Respect to website author , some wonderful entropy.
Your web has proven useful to me.
I truly enjoy looking through on this web site , it holds superb content .
I must say got into this web. I found it to be interesting and loaded with unique points of view.
Respect to website author , some wonderful entropy.
You got yourself a new rader.
stays on topic and states valid points. Thank you.
Just wanna input on few general things, The website layout is perfect, the articles is very superb : D.
I must say got into this website. I found it to be interesting and loaded with unique points of view.
stays on topic and states valid points. Thank you.
very cool post, i actually enjoyed this web site, carry on it
I really enjoy examining on this blog , it has got interesting article .
yahoo got me here. Cheers!
Great, this is what I was searching for in bing
Ha, here from google, this is what i was searching for.
I really got into this web. I found it to be interesting and loaded with unique points of interest.
Thanks for sharing such a fastidious idea, paragraph is fastidious, thats why i have read
it entirely
I conceive this web site holds some real superb information for everyone : D.
Right now it sounds like BlogEngine is the preferred blogging platform out there right now.
(from what I've read) Is that what you are using on your blog?
Intresting, will come back here later too.
This is awesome!
Deference to op , some superb selective information .
When I initially commented I clicked the "Notify me when new comments are added" checkbox and now each time
a comment is added I get four emails with the same comment.
Is there any way you can remove people from that service?
Bless you!
very Great post, i actually like this web site, carry on it
Because the admin of this website is working, no doubt very soon it
will be well-known, due to its feature contents.
I'm really loving the theme/design of your site. Do you
ever run into any web browser compatibility problems?
A number of my blog visitors have complained about
my site not operating correctly in Explorer but looks great in Chrome.
Do you have any recommendations to help fix this problem?
Thanks for this web. I definitely agree with what you are saying.
Appreciate it for this howling post, I am glad I observed this internet site on yahoo.
Incredible points. Outstanding arguments. Keep up the good effort.
Hello, i think that i noticed you visited my site thus i came to go back the favor?.I am trying to
find issues to improve my web site!I suppose
its adequate to use a few of your ideas!!
You got yourself a new follower.
Hi there friends, its great post about tutoringand
fully explained, keep it up all the time.
I have interest in this, cheers.
I have been exploring for a little bit for any high-quality articles
or weblog posts on this sort of house . Exploring in Yahoo I
finally stumbled upon this web site. Reading this information So i am happy to exhibit that I've an incredibly just right uncanny feeling I found out exactly what I needed.
I so much without a doubt will make certain to don?t put out
of your mind this web site and provides it a glance on a continuing basis.
Yeah bookmaking this wasn’t a risky decision outstanding post! .
Thanks for sharing your thoughts. I really appreciate your efforts and I am waiting for
your further post thank you once again.
These are genuinely great ideas in concerning blogging.
You have touched some good points here. Any
way keep up wrinting.
Hey very interesting blog!
Having read this I thought it was rather enlightening. I appreciate you finding the time and effort to put this information together.
I once again find myself personally spending a lot of
time both reading and posting comments. But so what, it was still worth it!
fantastic points altogether, you simply gained a new reader.
What might you recommend about your put up that you simply made
some days ago? Any certain?
I'm gratified by the manner in which yosefk.com deals with this sort of subject matter! Usually to the point, sometimes contentious, always well-researched and stimulating.
We are a bunch of volunteers and opening a new scheme in our
community. Your website provided us with helpful information to work on. You have
done an impressive process and our whole group might be thankful to you.
Hey there! I could have sworn I've been to this website before
but after browsing through some of the post I realized it's
new to me. Anyhow, I'm definitely happy I found it and
I'll be book-marking and checking back frequently!
Hi there to every body, it's my first pay a visit of this website;
this webpage includes amazing and in fact good data in favor of readers.
Greate post. Keep writing such kind of info on your page.
Im really impressed by your site.
Hello there, You have performed a great job. I'll certainly digg it and personally recommend
to my friends. I'm sure they'll be benefited from this website.
You could certainly see your expertise in the article you write.
The arena hopes for more passionate writers such as you who
aren't afraid to say how they believe. At all times go after your heart.
We're a group of volunteers and opening a new scheme in our community.
Your web site offered us with valuable information to work on. You have performed an impressive process and our entire neighborhood can be grateful to you.
Found this on google and I’m happy I did. Well written web.
Very interesting points you have remarked, appreciate it for putting up.
I’m impressed, I have to admit. Genuinely rarely should i encounter a weblog that’s both educative and entertaining, and let me tell you, you may have hit the nail about the head. Your idea is outstanding; the problem is an element that insufficient persons are speaking intelligently about. I am delighted we came across this during my look for something with this.
This does interest me
very interesting post, i actually love this web site, carry on it
I conceive you have mentioned some very interesting details , appreciate it for the post.
Great, google took me stright here. thanks btw for info. Cheers!
I was looking at some of your articles on this site and I believe this internet site is really instructive! Keep on posting .
This helps. Thanks!
Cheers, great stuff, I enjoying.
I enjoying, will read more. Cheers!
Thank You for this.
Ni hao, i really think i will be back to your page
What's Happening i am new to this, I stumbled upon this
I have found It absolutely useful and it has helped me out loads.
I'm hoping to contribute & help other users like its aided me.
Good job.
I like this website its a master peace ! Glad I found this on google .
I simply must tell you that you have an excellent and unique post that I kinda enjoyed reading.
I love reading through and I believe this website got some genuinely utilitarian stuff on it! .
Hi there, just became aware of your blog through
Google, and found that it's really informative. I am
going to watch out for brussels. I will appreciate
if you continue this in future. Lots of people will be benefited from your writing.
Cheers!
I was looking at some of your articles on this site and I believe this internet site is really instructive! Keep on posting .
Very interesting points you have remarked, appreciate it for putting up.
Cheers, here from bing, i enjoyng this, will come back soon.
I am not rattling great with English but I get hold this really easygoing to read .
I must say, as a lot as I enjoyed reading what you had to say, I couldnt help but lose interest after a while.
Enjoyed reading through this, very good stuff, thankyou .
Intresting, will come back here once in a while.
Parasite backlink SEO works well :)
I simply must tell you that you have an excellent and unique post that I really enjoyed reading.
Respect to website author , some wonderful entropy.
I was looking at some of your articles on this site and I believe this internet site is really instructive! Keep on posting .
Cheers, here from bing, i enjoyng this, will come back soon.
Cheers, great stuff, Me like.
I like this website its a master peace ! Glad I found this on google .
I’m impressed, I have to admit. Genuinely rarely should i encounter a weblog that’s both educative and entertaining, and let me tell you, you may have hit the nail about the head. Your idea is outstanding; the problem is an element that insufficient persons are speaking intelligently about. I am delighted we came across this during my look for something with this.
Found this on google and I’m happy I did. Well written website.
Because the admin of this web page is working, no question very soon it will be renowned, due to its feature contents.
Deference to op , some superb selective information .
Sweet blog! I found it while searching on Yahoo News. Do you have any suggestions on how to get listed in Yahoo News?
I've been trying for a while but I never seem to
get there! Many thanks
What's up everyone, it's my first go to see at this
website, and article is actually fruitful designed for me, keep up posting such content.
Skyking, this clue is your next bit of data. Please message the agency at your convenience. No further information until next transmission. This is broadcast #4311. Do not delete.
649557 95872Some truly amazing content material on this internet internet site , appreciate it for contribution. 605918
Unquestionably believe that which you stated. Your favorite reason appeared
to be on the net the simplest thing to be aware of.
I say to you, I definitely get annoyed while people think about worries that they just don't know about.
You managed to hit the nail upon the top as well as defined out the whole
thing without having side effect , people could take a signal.
Will probably be back to get more. Thanks
Wow! This blog looks just like my old one! It's on a totally different topic but it has pretty
much the same page layout and design. Excellent choice of colors!
Hi there! I realize this is sort of off-topic however I had to
ask. Does operating a well-established blog such as yours require a large
amount of work? I am brand new to operating a blog however I do write in my journal
everyday. I'd like to start a blog so I will be able to share
my own experience and thoughts online. Please let
me know if you have any ideas or tips for brand new aspiring bloggers.
Appreciate it!
Very quickly this site will be famous among all blogging users, due to
it's pleasant posts
I was looking at some of your articles on this site and I believe this internet site is really instructive! Keep on posting .
I really enjoy examining on this website , it has got good article .
You got yourself a new follower.
I truly enjoy looking through on this web site , it holds superb content .
You actually make it appear really easy together with your presentation but
I in finding this topic to be actually one thing which I believe
I would by no means understand. It seems too complex and extremely
huge for me. I am looking ahead to your next publish, I'll try to get
the dangle of it! natalielise plenty of fish
Hi would you mind letting me know which web host you're utilizing?
I've loaded your blog in 3 different web browsers and I must say this blog loads a lot quicker then most.
Can you suggest a good hosting provider at a fair price?
Cheers, I appreciate it! plenty of fish natalielise
It is really a great and useful piece of information. I am glad that you shared this useful information with us. Please keep us informed like this. Thank you for sharing.
Intresting, will come back here again.