Your IT department intends to upgrade your OS and gives a developer a new image to play with. The developer is generally satisfied, except there's this one program that mysteriously dumps core. Someone thoughtfully blames differences in system libraries.
Alternative prelude: you have this program and you're working on a new version. Being generally satisfied with the updates, you send the code overseas. They build it and it mysteriously dumps core. Someone thoughtfully blames differences in the compiler version.
Whatever the prelude, you open the core dump with `gdb app core` and gdb says:
#0 0x080484c9 in main (argc=Cannot access memory at address 0xbf3e7a8c) at main.cpp:4 4 int main(int argc, char** argv) (gdb)
Check out the garbage near "argc=" – if it ain't printing garbage, it ain't a C++ debugger. Anyway, it looks like the program didn't even enter main. An alert C++ hater will immediately suspect that the flying circus happening in C++ before main could be at fault, but in this case it isn't. In fact, a program can be similarly petrified by the perspective of entering any function, not necessarily main. It's main where it crashes in our example because the example is small; here's the source code:
#include <stdio.h>
#include "app.h"
int main(int argc, char** argv)
{
if(argc != 2) {
printf("please specify a profilen");
return 1;
}
const char* profile = argv[1];
Application app(profile);
app.mainLoop();
}
On your machine, you run the program without any arguments and sure enough, it says "please specify a profile"; on this other machine, it just dumps core. Hmmm.
Now, I won't argue that C++ isn't a high-level object-oriented programming language since every book on the subject is careful to point out the opposite. Instead I'll argue that you can't get a first-rate user experience with this high-level object-oriented programming language if you don't also know assembly. And with the first-rate experience being the living hell that it is, few would willingly opt for a second-rate option.
For example, nothing at the source code level can explain how a program is so shocked by the necessity of running main that it dumps a core in its pants. On the other hand, here's what we get at the assembly level:
(gdb) p $pc $1 = (void (*)(void)) 0x80484c9 <main+20> (gdb) disass $pc Dump of assembler code for function main: 0x080484b5 <main+0>: lea 0x4(%esp),%ecx 0x080484b9 <main+4>: and $0xfffffff0,%esp 0x080484bc <main+7>: pushl -0x4(%ecx) 0x080484bf <main+10>: push %ebp 0x080484c0 <main+11>: mov %esp,%ebp 0x080484c2 <main+13>: push %ecx 0x080484c3 <main+14>: sub $0xa00024,%esp 0x080484c9 <main+20>: mov %ecx,-0xa0001c(%ebp) # we don't care about code past $pc - # a screenful of assembly elided
What this says is that the offending instruction is at the address main+20. As you'd expect with a Segmentation fault or a Bus error core dump, this points to an instruction accessing memory, specifically, the stack.
BTW I don't realy know the x86 assembly, but I can still read it thusly: "mov" can't just mean the tame RISC "move between registers" thing because then we wouldn't crash, so one operand must spell a memory address. Without remembering the source/destination order of the GNU assembler (which AFAIK is the opposite of the usual), I can tell that it's the second operand that is the memory operand because there's an integer constant which must mean an offset or something, and why would you need a constant to specify a register operand. Furthermore, I happen to remember that %ebp is the frame pointer register which means that it points into the stack, however I could figure it out from a previous instruction at main+11, which moves %esp [ought to be the stack pointer] to %ebp (or vice versa, as you could think without knowing the GNU operand ordering – but it would still mean that %ebp points into the stack.)
Which goes to show that you can read assembly while operating from a knowledge base that is not very dense, a way of saying "without really knowing what you're doing" – try that with C++ library code; but I digress. Now, why would we fail to access the stack? Could it have to do with the fact that we apparenty access it with the offset -0xa0001c, which ought to be unusually large? Let's have a look at the local variables, hoping that we can figure out the size of the stack main needs from their sizes. (Of course if the function used a Matrix class of the sort where the matrix is kept by value right there in a flat member array, looking at the named local variables mentioned in the program wouldn't be enough since the temporaries returned by overloaded operators would also have to be taken into account; luckily this isn't the case.)
(gdb) info locals # if it ain't printing garbage, it ain't a C++ debugger: profile = 0xb7fd9870 "U211?WVS??207" app = Cannot access memory at address 0xbf3e7a98
We got two local variables; at least one must be huge then. (It can be worse in real life, main functions being perhaps the worst offenders, as many people are too arrogant to start with an Application class. Instead they have an InputParser and an OutputProducer and a Processor, which they proudly use in a neat 5-line main function – why wrap that in a class, 2 files in C++-land? Then they add an InputValidator, an OutputFormatConfigurator and a ProfileLoader, then less sophisticated people gradually add 20 to 100 locals for doing things right there in main, and then nobody wants to refactor the mess because of all the local variables you'd have to pass around; whereas an Application class with two hundred members, while disgusting, at least makes helper functions easy. But I digress again.)
(gdb) p sizeof profile $2 = 4 (gdb) p sizeof app $3 = 10485768
"10485768". The trouble with C++ debuggers is that they routinely print so much garbage due to memory corruption, debug information inadequacy and plain stupidity that their users are accustomed to automatically ignore most of their output without giving it much thought. In particular, large numbers with no apparent regularity in their digits are to a C++ programmer what "viagra" is to a spam filter: a sure clue that something was overwritten somewhere and the number shouldn't be trusted (I rarely do pair programming but I do lots of pair debugging and people explicitly shared this spam filtering heuristic with me).
However, in this case overwriting is unlikely since a sizeof is a compile time constant stored in the debug information and not in the program memory. We can see that the number will "make more sense" in hexadecimal (which is why hex is generally a good thing to look at before ignoring "garbage"):
(gdb) p /x sizeof app $4 = 0xa00008
…Which is similar to our offset value, and confirms that we've been debugging a plain and simple stack overflow. Which would be easy to see in the case of a recursive function, or if the program crashed, say, in an attempt to access a large local array. However, in C++ it will crash near the beginning of a function long before the offending local variable is even declared, in an attempt to push the frame pointer or some such; I think I also saw it crash in naively-looking places further down the road, but I can't reproduce it.
Now we must find out which member of the Application class is the huge one, which is lots of fun when members are plentiful and deeply nested, which, with a typical Application class, they are. Some languages have reflection given which we could traverse the member tree automatically; incidentally, most of those languages don't dump core though. Anyway, in our case finding the problem is easy because I've made the example small.
(I also tried to make it ridiculous – do you tend to ridicule pedestrian code, including your own, sometimes as you type? Few do and the scarcity makes them very dear to me.)
class Application
{
public:
Application(const char* profile);
void mainLoop();
private:
static const int MAX_BUF_SIZE = 1024;
static const int MAX_PROF = 1024*10;
const char* _profPath;
char _parseBuf[MAX_BUF_SIZE][MAX_PROF];
Profile* _profile;
};
This shows that it's _parseBuf that's causing the problem. This also answers the question of an alert C++ apologist regarding all of the above not being special to C++ but also relevant to C (when faced with a usability problem, C++ apologists like to ignore it and instead concentrate on assigning blame; if a problem reproduces in C, it's not C++'s fault according to their warped value systems.) Well, while one could write an equivalent C code causing a similar problem, one is unlikely to do so because C doesn't have a private keyword which to a first approximation does nothing but is advertised as an "encapsulation mechanism".
In other words, an average C programmer would have a createApplication function which would malloc an Application struct and all would be well since the huge _parseBuf wouldn't land on the stack. Of course an average C++ programmer, assuming he found someone to decipher the core dump for him as opposed to giving up on the OS upgrade or the overseas code upgrade, could also allocate the Application class dynamically, which would force him to change an unknown number of lines in the client code. Or he could change _parseBuf's type to std::vector, which would force him to change an unknown number of lines in the implementation code, depending on the nesting of function calls from Application. Alternatively the average C++ programmer could change _parseBuf to be a reference, new it in the constructor(s) and delete it in the destructor, assuming he can find someone who explains to him how to declare references to 2D arrays.
However, suppose you don't want to change code but instead would like to make old code run on the new machine – a perfectly legitimate desire independently of the quality of the code and its source language. The way to do it under Linux/tcsh is:
unlimit stacksize
Once this is done, the program should no longer dump core. `limit stacksize` would show you the original limit, which AFAIK will differ across Linux installations and sometimes will depend on the user (say, if you ssh to someone's desktop, you can get a lesser default stacksize limit and won't be able to run the wretched program). For example, on my wubi installation (Ubuntu for technophopes like myself who have a Windows machine, want a Linux, and hate the idea of fiddling with partitions), `limit stacksize` reports the value of 8M.
Which, as we've just seen, is tiny.
177 comments ↓
8M for a stack is not tiny in 32-bit land, in fact it's very large. This important in practice if you use many threads, as the preallocated stacks will use all your address space, if not actual committed memory. But I assume you're running 64-bit, as 32-bit generally defaults stack to 1M or so.
One of the gcc 4.5 projects (recently commited) adds perfect debuginfo for optimized programs with not much more compile time. People who want backtraces with -fomit-frame-pointer are still stuck, though.
Doesn't your main function need to return a value at the end?
@astrange: Actually, it's not that hard to get a backtrace under -fomit-frame-pointer; perhaps I'll blog about it. The Green Hills debugger does that and I think the compiler stores things in the debug info for this to work, however heuristic scripting can do quite well given just a symbol table and a disassembler.
@Barry: I didn't mean to say tiny, I meant to say <sarcasm>tiny</sarcasm> – but you know how these blogging engines mess with your markup. Regarding the preallocation of stacks for threads – so what happens under `unlimit stacksize' then? (I'd assume that Linux doesn't really preallocate physical memory for stacks but it does have to preallocate virtual addresses since they ought to be contiguous, therefore there can be no such thing as unlimited stack size, even if unlimited means "until we run out of bits to encode pointers", because if there are many threads, you need to divide the address space between them, and Linux doesn't even know how many threads a program will use concurrently; so `unlimit stacksize' would simply mean to use some large limit built into the system.)
@sn: it doesn't.
It's pretty dumb to create such big instances in the stack. The stack is for temporary and little operations, to avoid overhead of heap memory allocation, as allocating memory in the stack is just a stack pointer adjustment (e.g. the fastest possible memory allocation).
C++ is very criticable, but come on, this post is just ignorance.
> It’s pretty dumb to create such big instances in the stack.
No shite, Sherlock!
> It’s pretty dumb to create such big instances in the stack.
Yes but the functionality is there to do so, though the problem is that there is no standard or common way between compilers to detect such events occuring.
I think the issue with allocating on the stack v heap is a red herring. The problem is that their is not a fast effective way of detecting such problems unless your experienced with assembly and the technical details of the envrioment the application is running in.
Though, sometimes less is more and in this situation most customers will have no idea what the problem is and wouldnt really care.
Thank you Yossi Kreini for another post. Allways look forward to see what new things you run into.
Keep it up.
Chad.
Using the stack to allocate more than 16K is considered dumb in many places.
So do any lint-ish tools detect the issue? Did you get any warnings when you compiled your Application class? I'm always leery when I snarf a big block of space. Where did [1024][10] come from?
er, [1024][1024*10]
What should LINT or the likes of it do – warn about the size of your objects when it exceeds what threshold? 16K the universal dumbness constant from the comment above?
No, I didn't get any warnings, in fact this example was inspired by a program that allocated >300M using a similar 2D array and AFAIK it compiled under gcc -Wall -Werror.
I am a bit confused. In the actual app where this problem originally came up, was that the first time someone had added the private attribute with the array size associated with it, or did this somehow work with an earlier version of C++?
There was, sadly, more than one actual app where I saw this problem. The example with the huge member array preventing you from entering main() is close to a particular case, where the problem didn't come up until someone ssh'd to someone else's machine and ran the program (which previously ran fine for what I think was months). Presumably the default stack size limit varied with the user/machine combination.
This is unrelated to "versions of C++" (be it the language standard or the compiler version that you refer to) where the OS sets the stack size limit. It can depend on compiler version if the stack is effectively allocated statically by the compiler (actually a linker script shipped with it), as can be the case in embedded systems; although there you frequently maintain your own linker script anyway.
Your criticisms of C++ are unfair do to the fact that they have nothing to do with C++.
Where else could you encounter similar behavior?
You would have the exact same issue in C and any other language that allows allocations on the stack with no runtime checking:
#define MAX_BUF_SIZE 1024
#define MAX_PROF 1024*10
struct Application {
char* _profPath;
char _parseBuf[MAX_BUF_SIZE][MAX_PROF];
};
C++ has plenty of warts, but this isn't one of them.
RTFA. In C you'd be very likely to malloc the forward-declared struct – C encapsulation encapsulates much more than C++'s does.
You said:
"Alternatively the average C++ programmer could change _parseBuf to be a reference, new it in the constructor(s) and delete it in the destructor, assuming he can find someone who explains to him how to declare references to 2D arrays."
I'll take that challenge :)
is there anything non-standard with my solution?
are there any problems with my implementation of
class Application that weren't there to begin-with?
——————————————————–
class Profile;
class Application
{
public:
Application(const char* profile);
~Application();
void mainLoop();
private:
static const int MAX_BUF_SIZE = 1024;
static const int MAX_PROF = 1024*10;
typedef char arr2D_t[MAX_BUF_SIZE][MAX_PROF];
const char* _profPath;
void *rawMem;
// here's the damn ref to a 2D array.
arr2D_t& _parseBuf;
Profile* _profile;
};
Application::Application(const char *profile):
rawMem(new char[sizeof(arr2D_t)]),
_parseBuf(*reinterpret_cast(rawMem))
{
}
Application::~Application()
{
delete [] &_parseBuf;
}
int main()
{
Application myApp("foo");
// ran fine on my machine…
return 0;
}
the reinterpret_cast in my version had the necessary
< arr2D_t * >
the comment system ate it…
I hope it makes more sense now.
how do you make triangular braces on this forum anyway?
<>
OK, I think I figured it out,
it was supposed to look like this:
*reinterpret_cast<>(rawMem)
no I didn't get it… giving-up now
@Yoed: pretty much along the lines of what I thought; actually you don't need to keep rawMem as you can delete &_parseBuf in the destructor. A large share of C++ programmers wouldn't know how to typedef the 2D array type.
I tried first without the rawMem.
it cries about needing a temp variable in the initialization phase. so I gave it perm variable instead.
if you'll notice, I do use &_parseBuf in the destructor.
btw, having to still use delete [] (with the square braces) is a pitfall that's easy to miss IMO.
I'm Sherlock. Did anyone asked for my assistance?
ps/ Oy! I typed "yes" or "y" in the "Human?" field and got some machine-like language telling me to do what I just did. Brainless machines…
@Sherlock: no shit, Sherlock! You'd be better off if you didn't strip non-alphanumeric characters before parsing natural language.
Note: In Fedora limit doesn't exist, I used this to get the stack size:
ulimit -s
@Chris:
That's not a "Fedora vs. the world" issue -
the author took care to write it as "Linux/tcsh".
Thus, just trade your bash (which as you said has the ulimit builtin) for something usable in the context of this article.
@Yoed: I guess you'd want to avoid reinterpret_cast (portability issues), and use static_cast(static_cast(rawMem)) instead – which is well defined.
@Yossi: I liked the debug process you did there, but wouldn't a simple glimpse at the code given you the answer already?
I am asking because I find people are split to two groups regarding debugging:
- Those that just go ahead and gdb the hell outta their binary as soon as a problem is encountered,
- And those (myself included, I must admit), that rather re-read the code a few times, and then use printf debugging, all just to avoid gdb :)
As with everything, professionals must balance both of these approaches.
@rmn: well, in my example you could probably see the problem since it was really small, but in a larger program, when the program crashes at a place where you don't really expect it to crash no matter what data it runs on (that is, no pointer dereferencing, no division, etc.), I don't see how reading code would help. The large object could be (and in reality, will be) defined deep down in some member of a member of a…
Of course if you've already seen a similar bug, you'll be looking for a large object – but if you haven't, reading source doesn't really point you to the problem very directly since nothing interesting happens at the source level. A similar and perhaps nastier case is when you get a misaligned stack pointer and crash at some place where the compiler used an x86 vector instruction that only works with aligned addresses. There's absolutely nothing wrong at the source level so you have to disassemble to make any sense of what you see. (The difference here is that it's not the fault of standard debugging tools, really, since standard compilers will set up the stack properly; this is the fault of whoever glued shared libraries built with different calling conventions together or similar – although the tools could do at least something to detect an ABI mismatch and they don't. Whereas in my example, it's pretty standard and widely encouraged C++ style – "local variables are the best" – and then you get no help at all from the tools to figure out what went wrong.)
There's this one thing I don't understand and it itches.
"RTFA. In C you’d be very likely to malloc the forward-declared struct"
Why wouldn't one be just as likely to "new" the class in C++?
Well, obviously nothing prevents one from new'ing an object rather than stack-allocating – it's a question of what the preferred style is.
In C, "encapsulation" generally means ADTs – forward-declared structs implemented in a .c file, hiding their internals to the point where you can't allocate an object yourself since you don't know its size – so you call an allocation function provided in the .h file for the ADT.
In C++, "encapsulation" generally refers to the "private:" keyword – a class keeps its implementation details in the "private:" section, and the compiler can thus allocate an object anywhere where the .h is included, since it knows the object size, the price being that it also has to read the .h files describing the classes of the members. The common C++ style encourages allocation on the stack, both because its more efficient than allocation on the heap and because it's the most straightforward way to get RAII-style resource management.
Again, it's not about what's possible but about what's likely; the commonly advocated and used style affects the likelihood if not the possibilities.
Instead of this:
char _parseBuf[MAX_BUF_SIZE][MAX_PROF];
you can do the following (and allocate on the heap instead of stack):
std::vector _parseBufVec(MAX_BUF_SIZE*MAX_PROF);
typedef char _ParseBuf[MAX_BUF_SIZE][MAX_PROF];
_ParseBuf & _parseBuf = *reinterpret_cast(_parseBufVec.data());
reinterpret_cast should make the two views on the same memory act as if they were
two entries in a union. I believe that was the original intention of reinterpret_cast.
Ok, there is no preview button, but the reinterpret cast was supposed to be like this:
reinterpret_cast<_ParseBuf*>(….
the <,> got parsed away (probably because they look like a tag).
Yeah, that's one way to do it, although I'd probably typedef the 2D array type to something and then new/delete it myself instead of using std::vector and reinterpret_cast, just so that there's a teeny bit less of tasty C++ goodness involved.
Well, if you do all the new/delete on your own, you risk leaking if anything throws and something in a higher stack frame catches it. You then need to start wrapping your allocation/deallocation in ctor/dtor; or to use smart pointers. Before long, you'll be writing quite a bit more code than a simple (ok, that's definitely a euphemism) cast would have been.
Much of the code I work on is compiled without exception support, precisely to not care about this sort of thing.
Why wouldn't you just do something like the following instead?
Application app = new Application(profile);
app->mainLoop();
delete app;
That sounds like a far better fix. And just so you know, you can run into problems with the stack in pretty much any programming language that exists. This issue doesn't just exist in C++.
You could do that, as I mentioned. As to other languages – most allocate less on the stack and report stack overflows more gracefully.
Hi've got an easy solution for you: just move the app declaration out of main. That way that variable will be statically allocated by the linker in the data section, and you'll have no problem with the stack.
I do like C++, altough not the metatemplate interpreter builtin, and having spent the last few days reading your FQA, I'm trying not to sound like a C++ apologist…
Incidentally, 64-bit Linux also allocates 8M for the stack by default, so this isn't a 32-bit vs 64-bit issue.
Yep, ended sounding dumb instead, profile is a parameter for the constructor.
My other solution would be like Ray's one, above.
Well, sure, there are plenty of workarounds, once you figure out the problem – which many, many competent C++ programmers really can't.
For certain values of competent.
I've seen lots of C programs (particularly vendor supplied "examples" for microcontrollers) that stack overflow spectacularly because they thought to allocate the entire data buffer on the stack, and have the declaration buried deep within nested loops and switches.
This is certainly not a C++ problem, more of a brain-dead developer problem.
Yes, a matrix was to large to statically allocate, what is the limit on statically allocated variable ?
Well explained. I was getting the same error in a legacy C code and I never considered it to be a stack overflow.
Tremendous issues here. I am very satisfied to look your article.
Thank you so much and I'm having a look ahead to
contact you. Will you please drop me a e-mail?
Good day I am so happy I found your webpage, I really found you by error,
while I was browsing on Google for something else, Nonetheless I am here now and
would just like to say thanks for a marvelous
post and a all round interesting blog (I also love the theme/design), I don’t
have time to read it all at the minute but I have saved it and
also added in your RSS feeds, so when I have time I will be back to read a lot more, Please do keep up the excellent work.
This post is in fact a nice one it assists new net users, who are wishing in favor
of blogging.
EquiNet Get 2,550 USD
It is specially designed for lifetime business space with zero risk management system. Many people got frustrated with the SCAM and fake business platform. All about the cause of unbalanced between income and expense.
[url=https://equinet.club/?r=longo]EquiNet[/url] is an open business platform, transparent and flexible. You can easily calculate the cash flow of company and make sure the system never collapses and will run for a lifetime.
We learned more about most of the online business platform and we found the solution to save all of the online businessmen. EquiNet is working simply! We create, develop, maintain, and provide a networking business platform. We make sure the platform is working for a lifetime by using the zero risk management system. [url=https://equinet.club/?r=longo]Join us now![/url] It's time to stop all of the risky business and start to get networking bonus for a lifetime.
The Administrator and Service Provider of the platform is EquiNet Club LTD, 12 Queen Anne Street, Marylebone, London, United Kingdom, W1G 9LF, registered under the number 11836193.
Regards,
[url=https://equinet.club/?r=longo]The EquiNet Team[/url]
March 14, 2019
RUS
EquiNet получить 2,550 USD
Матричный проект [url=https://equinet.club/?r=longo]EquiNet[/url] создан для постоянного дохода на всю жизнь с zero системой управления рисками. Многие люди разочаровались в мошенничестве и поддельными бизнес-платформами. Вот и вся причина дисбаланса между доходами и расходами.
[url=https://equinet.club/?r=longo]EquiNet[/url] – это открытая бизнес-платформа, прозрачная и гибкая. Вы можете легко рассчитать денежный поток компании и убедиться, что система никогда не рухнет и будет работать всю жизнь.
Мы анализировали о большинстве онлайн-бизнес-платформ, и мы нашли решение, чтобы обезопасить всех онлайн-бизнесменов. EquiNet работает просто! Мы создаем, развиваем, поддерживаем и предоставляем сетевую бизнес-платформу. Мы гарантируем, что платформа будет работать на протяжении всей жизни, используя систему управления нулевыми рисками. [url=https://equinet.club/?r=longo]Присоединяйтесь к нам сейчас![/url] пришло время остановить весь рискованный бизнес и начать получать сетевую прибыль на всю жизнь.
Администратором и поставщиком услуг платформы является EquiNet Club LTD, 12 Queen Anne Street, Marylebone, Лондон, Великобритания, W1G 9LF, зарегистрированный под номером 11836193.
С уважением,
[url=https://equinet.club/?r=longo]Команда EquiNet[/url]
14 марта 2019 г.
[img]https://chart.googleapis.com/chart?cht=qr&chl=https%3A%2F%2Fequinet.club%2F%3Fr%3Dlongo&choe=UTF-8&chs=320×320[/img]
Cheers, great stuff, I enjoying.
This does interest me
I have interest in this, danke.
Awesome, this is what I was searching for in bing
I was looking at some of your articles on this site and I believe this internet site is really instructive! Keep on posting .
I really enjoy examining on this web , it has got fine stuff .
Your site has proven useful to me.
I'm gratified by the manner in which yosefk.com handles this type of issue! Usually to the point, often contentious, consistently well-researched and challenging.
This is cool!
Ha, here from google, this is what i was looking for.
Cheers, here from yanex, i enjoyng this, I come back again.
I like this site, useful stuff on here : D.
Hey very cool blog!! Guy .. Excellent .. Amazing ..
I will bookmark your blog and take the feeds additionally?
I am happy to find numerous helpful info right here within the post, we'd like work out more
strategies in this regard, thanks for sharing.
. . . . .
This does interest me
Your web has proven useful to me.
This is great!
I am not rattling great with English but I get hold this really easygoing to read .
Thanks for this site. I definitely agree with what you are saying.
Great, google took me stright here. thanks btw for this. Cheers!
Ha, here from bing, this is what i was searching for.
Thank you for the auspicious writeup. It actually was once a leisure account it.
Glance complicated to more introduced agreeable from
you! By the way, how could we communicate?
Very interesting points you have remarked, appreciate it for putting up.
Hello, here from yahoo, me enjoyng this, I come back soon.
Me like, will read more. Thanks!
Woah! I'm really digging the template/theme of this blog.
It's simple, yet effective. A lot of times it's tough to get
that "perfect balance" between superb usability and appearance.
I must say you have done a superb job with this. Also, the
blog loads very fast for me on Chrome. Outstanding Blog!
I conceive this web site holds some real superb information for everyone : D.
Found this on MSN and I’m happy I did. Well written website.
I'm gratified by the manner in which yosefk.com handles this kind of topic! Generally on point, often contentious, always well-researched as well as challenging.
Ni hao, i really think i will be back to your page
Awesome, this is what I was looking for in bing
Hi! I've been reading your weblog for a while now and finally got the bravery to go ahead
and give you a shout out from Atascocita Tx! Just wanted
to tell you keep up the good job!
I must say got into this web. I found it to be interesting and loaded with unique points of interest.
I like this site, useful stuff on here : D.
Enjoyed examining this, very good stuff, thanks .
Very interesting points you have remarked, appreciate it for putting up.
This helps. Thanks!
I visited many web sites except the audio feature for audio songs present
at this website is genuinely marvelous.
Enjoyed examining this, very good stuff, thanks .
These are truly impressive ideas in about blogging. You have
touched some good things here. Any way keep
up wrinting.
Great, yahoo took me stright here. thanks btw for this. Cheers!
Respect to website author , some wonderful entropy.
I have interest in this, thanks.
Write more, thats all I have to say. Literally, it
seems as though you relied on the video to make your point.
You definitely know what youre talking about, why throw away your intelligence on just
posting videos to your site when you could be giving us something enlightening to read?
I was looking at some of your articles on this site and I believe this internet site is really instructive! Keep on posting .
Awesome! Its genuinely awesome paragraph, I have got much clear
idea concerning from this article.
Great article! This is the kind of info that are meant
to be shared around the net. Disgrace on Google for no longer
positioning this put up upper! Come on over and visit my website .
Thank you =)
Cheers, great stuff, Me like.
Thankfulness to my father who stated to me regarding this web site, this webpage
is genuinely remarkable.
Hello! I've been following your blog for a long time now and finally got
the courage to go ahead and give you a shout out from
Humble Tx! Just wanted to say keep up the excellent work!
I was wondering if you ever thought of changing the page layout of
your blog? Its very well written; I love what youve got
to say. But maybe you could a little more in the way of content so people could connect with
it better. Youve got an awful lot of text for only having one or two images.
Maybe you could space it out better?
Acticin Can I Purchase No Doctor Viagra 71 Ans Propecia Vault [url=http://6drugs.com]generic cialis canada[/url] Buying Generic Zentel
Everything is very open with a very clear description of the challenges.
It was definitely informative. Your site is very helpful.
Thanks for sharing!
Good day I am so excited I found your site, I really found you by error, while I was searching on Yahoo for something else, Anyways I
am here now and would just like to say many thanks for a marvelous post and a all round thrilling blog (I also love the theme/design), I don't have time to browse it all
at the moment but I have bookmarked it and also added in your RSS feeds, so when I have time I will be back
to read a lot more, Please do keep up the fantastic work.
My brother suggested I would possibly like this web site.
He was entirely right. This post truly made my day.
You can not believe just how so much time I had spent for this information! Thanks!
I simply could not depart your website before suggesting that I extremely enjoyed
the standard information a person supply on your visitors?
Is gonna be back regularly to check out new posts
Salbutamol Nebules Buy Buy Finasteride Citrate Liquid Will Keflex Help With Bronchitis [url=http://viacheap.com]viagra[/url] Cialis Original Contrareembolso Methocarbamol Germany
I'm amazed, I must say. Rarely do I come across a blog that's equally educative and engaging, and let me tell you, you've hit the nail on the head.
The issue is something that too few people are speaking intelligently about.
I am very happy that I stumbled across this during my search for something regarding this.
Quality content is the crucial to interest the viewers to go to see the web page, that's what
this web page is providing.
This information is priceless. Where can I find out more?
I kinda got into this web. I found it to be interesting and loaded with unique points of view.
Thanks for finally talking about > The C++ Sucks
Series: petrifying functions < Loved it!
I consider something really special in this site.
Buy Thyroxine Uk Is Cephalexin A Sulfa Drug Viagra Generic Kaufen In Holland [url=http://rxasian.com]viagra[/url] Vente Baclofen En Canada Tadalis Sx Pfizer Keflex For Mrsa Infections
I truly enjoy looking through on this web site , it holds superb content .
Appreciate it for this howling post, I am glad I observed this internet site on yahoo.
I like the valuable info you provide in your articles. I will bookmark your weblog and check again here regularly.
I am quite sure I'll learn lots of new stuff right
here! Good luck for the next!
Now I am ready to do my breakfast, once having my
breakfast coming over again to read other news.
I'd like to find out more? I'd love to find out some additional information.
Cheers, great stuff, Me like.
I must say, as a lot as I enjoyed reading what you had to say, I couldnt help but lose interest after a while.
This is good. Thanks!
Hey, google lead me here, keep up nice work.
I’m impressed, I have to admit. Genuinely rarely should i encounter a weblog that’s both educative and entertaining, and let me tell you, you may have hit the nail about the head. Your idea is outstanding; the problem is an element that insufficient persons are speaking intelligently about. I am delighted we came across this during my look for something with this.
I love reading through and I believe this website got some genuinely utilitarian stuff on it! .
"Excellent article. I'm dealing with many of these issues as well.."
Some truly great content on this web site , appreciate it for contribution.
Zithromax Treats What [url=http://bestlevi.com]best price on levitra[/url] Zithromax For Animals
I simply must tell you that you have an excellent and unique article that I must say enjoyed reading.
Respect to website author , some wonderful entropy.
Great, bing took me stright here. thanks btw for info. Cheers!
Ni hao, here from baidu, me enjoyng this, i will come back soon.
Very interesting points you have remarked, appreciate it for putting up.
I was curious if you ever thought of changing the layout of your
blog? Its very well written; I love what youve got to say.
But maybe you could a little more in the way of content so people could connect with it better.
Youve got an awful lot of text for only having 1 or two pictures.
Maybe you could space it out better?
Respect to website author , some wonderful entropy.
This is nice!
Some truly wonderful posts on this web site , appreciate it for contribution.
stays on topic and states valid points. Thank you.
Great article to check out, glad that duckduck brought me here, Keep Up good job
I love reading through and I believe this website got some genuinely utilitarian stuff on it! .
Parasite backlink SEO works well :)
I like this website its a master peace ! Glad I found this on google .
I conceive this web site holds some real superb information for everyone : D.
[url=https://dedicatet.com/categories/black-market.14/]Black Market[/url]
Just wanna input on few general things, The website layout is perfect, the articles is very superb : D.
I truly enjoy looking through on this web site , it holds superb content .
Ha, here from yahoo, this is what i was looking for.
The next phase of the mystery is to interpret the order of the pyramid. This is your third confidential lead! 517232125
I like this site, useful stuff on here : D.
Yeah bookmaking this wasn’t a risky decision outstanding post! .
Your web has proven useful to me.
I’m impressed, I have to admit. Genuinely rarely should i encounter a weblog that’s both educative and entertaining, and let me tell you, you may have hit the nail about the head. Your idea is outstanding; the problem is an element that insufficient persons are speaking intelligently about. I am delighted we came across this during my look for something with this.
Priligy Dapoxetina Costo Kamagra Gel Oral Farmacia [url=http://boijoy.com]levitra equivalent[/url] Securemed Levitra Sample Online Pharmacy Buy Cialis In Dubai
I like the valuable info you provide in your articles. I will
bookmark your blog and check again here frequently. I am quite certain I will learn many
new stuff right here! Best of luck for the next!
great advice you give
My developer is trying to persuade me to move to .net from PHP.
I have always disliked the idea because of the costs. But he's tryiong none the less.
I've been using Movable-type on various websites for about a year and am worried about switching to another platform.
I have heard great things about blogengine.net.
Is there a way I can import all my wordpress posts into it?
Any kind of help would be greatly appreciated!
thanks for the info!
thanks for the info!
Mr.s Fister, this note is your next bit of info. Do message the agency at your earliest convenience. No further information until next transmission. This is broadcast #3698. Do not delete.
amazing content thanks
you are amazing
This blog is amazing! Thank you.
This blog is amazing! Thank you.
There is definately a great deal to find out about this issue.
I like all of the points you have made.
I would like to thank you for the efforts you have put in penning this website.
I really hope to check out the same high-grade content by you later
on as well. In fact, your creative writing abilities has encouraged me to get my own site now ;)
Why people still use to read news papers when in this technological
globe everything is available on web?
I was looking at some of your articles on this site and I believe this internet site is really instructive! Keep on posting .
These are truly enormous ideas in concerning blogging.
You have touched some good things here. Any way keep up
wrinting. plenty of fish natalielise
I am not rattling great with English but I get hold this really easygoing to read .
I am 43 years old and a mother this helped me!
I am 43 years old and a mother this helped me!
I am 43 years old and a mother this helped me!
What is the name of a website which gives reviews on the various online dating services
What to expect [url=https://www.bitchute.com/channel/asiame-review/]asiame review[/url] from online dating sites?
firstly, international dating is NOT safe. You can say you are 13 and put up some google photos of a 13 year old boy/girl when 51 and you are a molester. It's unsafe. Trying to find the proper person in your area is ok. get slow; Visit online and with names. I kept a hotmail account absolutely for these contacts. elect to meet, this short. Go for some time walk around a park; Get a mug of coffee or so on. Do not get your last name; Meet in the same, Well moved area. If the person you're writing uses bad language, suggestive insinuations, And so on marriage ceremony a good match. it's really a dud. you like to meet a decent person with values that will last over time. ( Full resolution )
There are hundreds of internet dating online. The best activities is research them, look at them, And then choose which ones might suit you best. One website called Match is considered very good and it 's been around for a long time. With compliment, You can meet people from throughout the world as well as your own country. it is a option for dating/relationship matches or just friendship matches. There are proven methods to use it and free, optional, Compatibility/affinity personality tests if you want to try them. Most people desire serious relationships, Or dating with a view to developing a romantic relationship, Not a romp to the hay. be cautious that whatever dating site you use is not too new. frauds new, They may not have many members there are several free dating websites. Some options PlentyOfFish, 2meet4free, acceptable Cupid, Matchdoctor, 4single, Freeonlinedating, And craiglist. but nevertheless, be careful, you will get a lot of spam at the free sites, notably at Craigslist. You may first want to read up as much as you can about online dating tips. keep an eye out, but bear in mind, If you are going for a free online dating site. You get everything pay for, As the old saying goes. you might consider it wiser to pay a few dollars/pounds/etc, And improve quality and better choices. Always look for profiles with an image. It is a known fact that some people who decline to add a photograph are married and not wanting to be spotted (By people keywords) On a dating world-wide-web site. not necessarily, inevitably play safe. A good site will advise you how to start this. Never agree to meet anybody too quickly. Meet in a public place and never meet a stranger without telling somebody where you have. The best online dating site is definitely one that is easy to use, Where there is certainly out a lot about the people on it and where you can post your own information easily. And you want people to be honestly looking for a romantic relationship, similarly to yourself. ( Full solve )
How effective are online dating services?
People have mixed results when using online dating services. Some people are able to hook up immediately with folks they like, while other wemasters try for years to cyberdate successfully. Another factor to bear in mind is the site itself. Emo dating sites, Christian internet dating sites). To help have an understanding of the answer, We need to know the definition of effective. Are paid dating sites effective [url=https://asiame1.tumblr.com/]asiame review[/url] for meeting new people? all right. Are they effective for arranging dates? understand. Are they effective for finding your partner to marry? Depends completely on an individual. ( Full fix )
What are the best online dating services?
any time you ask for "the most impressive" Of products, You will get aplethora of answers that advertise one place or another becauseeveryone thinks their site is the greatest. you can try published lists and reviews. Reviews can be participating, But aresubject to being faked. research is recommended, So do someresearch and talk to the friends. ( Full solution ).
Hi, glad that i stumble on this in google. Thanks!
Latino comics realise his or invest N
you certainly will perceive any audible one boats when it comes to adult dating: "i additionally wouldn't go out with Latinas anymore, is marked Joey Vega. Ramrez and in addition ngel Salazar. they are popping riddles as the '80s.
brother newbies instance Guatemalan Puerto Rican Erik Rivera, 24, cirsis up immigrant wife jokes: "Thirty five years ordinary us and she even so have not learned these text, Rivera expresses involving michael's mama, people who helpful to criticize in regard to the bad native english speakers degrees with your ex wife own muddled guide: "What will you be consideration anytime were made of which,
Kenny Ortega, A thirty something Puerto Rican including distance Harlem so, who head the specific television show "relaxed, unsightly fat additionally first class" inside Comix for the Mecenterpacking, credit it's woman significantly more closely on process.
"i purchase [url=https://10datingsites.com/product/latamdate-com/]latamdate.com[/url] excellent sense of humor off your wife, he says. "but don't observe them, on account that than she is going to really want the girl's 10%,
within a of the size of his choice of exercise sessions, and eventega recounts a bound earlier childhood days trip to excellent adventure, "the most important ghetto disney world, as a relabeled as a result of Vctor cruz, 27, A Bronx Puerto Rican comic who just offers his signature bizarre bit on the theme park.
Nicaraguan frederick Rocha, 40, The frequently a lot for monthly "funny Caliente" prove to during the comedian deprive go on the top distance bad side, tells me most Latino hours are a way to meet town.
"The picture is wide-ranging and in many different places, according to him, checking 10 club sets via the city while various Latinos really perform with regards to [url=https://www.linkedin.com/in/latamdate-com-b3473985]latamdate review[/url] a day time.
As for the viewers, Rocha tells how these firms range from yellow back of the shirt staff members to assist you bankers, and that they result from several different nationalities.
but yet, according to him, "should I holler to work with 'Nicaraguans in your home,or,– due to find out Woohoo!' you always notice, 'What?or,–"
"one particular most detrimental purpose relevant to as being a latina woman comedian has been labelled a, declares Gina Brillon, who all hosting companies programmes four night's a week at the favored time intervals rectangle humourous soccer team.
all the Bronx put together, 27 year old Puerto Rican french tells people this creation and as well as performance may possibly heavy duty.
but also, which include funny htc legend Rodney Dangerfield's "i receive none of admiration" schedule, actually being female documented in jokes customers are truly more challenging.
Brillon tells i have discovered less than a little bit of added Latina comics producing repeatedly in the metro area such as peaches Rodrguez and Sara Contreras.
just about every other, folks fight the notion that females is definitely not very funny, and they usually struggle with humourous clubs that won't booklet enough chicks.
and whether they do get a concert, Brillon states it's not easy.
"one of the biggest issues continually that searching for may female during the give plus there is a male comedian throwing, [url=https://www.youtube.com/watch?v=wZnG7IXap7Y]latamdate.com[/url] quite possibly point out that, 'We went a goody available for you personally. in which lady coming.'
"and you simply believe perfectly, You want a whore, predicts Brillon. "you're going to be exactly like, 'They most likely would not keep pointed out that I was customer?' when i which would have captured on eventually.
yahoo took me here. Thanks!
Does your website have a contact page? I'm having problems locating it but,
I'd like to shoot you an e-mail. I've got some ideas for
your blog you might be interested in hearing. Either way, great website and
I look forward to seeing it grow over time.
Cheers, great stuff, Me enjoying.