The C++ Sucks Series: petrifying functions

Your IT department intends to upgrade your OS and gives a developer a new image to play with. The developer is generally satisfied, except there's this one program that mysteriously dumps core. Someone thoughtfully blames differences in system libraries.

Alternative prelude: you have this program and you're working on a new version. Being generally satisfied with the updates, you send the code overseas. They build it and it mysteriously dumps core. Someone thoughtfully blames differences in the compiler version.

Whatever the prelude, you open the core dump with `gdb app core` and gdb says:

#0  0x080484c9 in main (argc=Cannot access memory at address 0xbf3e7a8c) at main.cpp:4
4    int main(int argc, char** argv)
(gdb)

Check out the garbage near "argc=" – if it ain't printing garbage, it ain't a C++ debugger. Anyway, it looks like the program didn't even enter main. An alert C++ hater will immediately suspect that the flying circus happening in C++ before main could be at fault, but in this case it isn't. In fact, a program can be similarly petrified by the perspective of entering any function, not necessarily main. It's main where it crashes in our example because the example is small; here's the source code:

#include <stdio.h>
#include "app.h"

int main(int argc, char** argv)
{
  if(argc != 2) {
    printf("please specify a profilen");
    return 1;
  }
  const char* profile = argv[1];
  Application app(profile);
  app.mainLoop();
}

On your machine, you run the program without any arguments and sure enough, it says "please specify a profile"; on this other machine, it just dumps core. Hmmm.

Now, I won't argue that C++ isn't a high-level object-oriented programming language since every book on the subject is careful to point out the opposite. Instead I'll argue that you can't get a first-rate user experience with this high-level object-oriented programming language if you don't also know assembly. And with the first-rate experience being the living hell that it is, few would willingly opt for a second-rate option.

For example, nothing at the source code level can explain how a program is so shocked by the necessity of running main that it dumps a core in its pants. On the other hand, here's what we get at the assembly level:

(gdb) p $pc
$1 = (void (*)(void)) 0x80484c9 <main+20>
(gdb) disass $pc
Dump of assembler code for function main:
0x080484b5 <main+0>:    lea    0x4(%esp),%ecx
0x080484b9 <main+4>:    and    $0xfffffff0,%esp
0x080484bc <main+7>:    pushl  -0x4(%ecx)
0x080484bf <main+10>:    push   %ebp
0x080484c0 <main+11>:    mov    %esp,%ebp
0x080484c2 <main+13>:    push   %ecx
0x080484c3 <main+14>:    sub    $0xa00024,%esp
0x080484c9 <main+20>:    mov    %ecx,-0xa0001c(%ebp)
# we don't care about code past $pc -
# a screenful of assembly elided

What this says is that the offending instruction is at the address main+20. As you'd expect with a Segmentation fault or a Bus error core dump, this points to an instruction accessing memory, specifically, the stack.

BTW I don't realy know the x86 assembly, but I can still read it thusly: "mov" can't just mean the tame RISC "move between registers" thing because then we wouldn't crash, so one operand must spell a memory address. Without remembering the source/destination order of the GNU assembler (which AFAIK is the opposite of the usual), I can tell that it's the second operand that is the memory operand because there's an integer constant which must mean an offset or something, and why would you need a constant to specify a register operand. Furthermore, I happen to remember that %ebp is the frame pointer register which means that it points into the stack, however I could figure it out from a previous instruction at main+11, which moves %esp [ought to be the stack pointer] to %ebp (or vice versa, as you could think without knowing the GNU operand ordering – but it would still mean that %ebp points into the stack.)

Which goes to show that you can read assembly while operating from a knowledge base that is not very dense, a way of saying "without really knowing what you're doing" – try that with C++ library code; but I digress. Now, why would we fail to access the stack? Could it have to do with the fact that we apparenty access it with the offset -0xa0001c, which ought to be unusually large? Let's have a look at the local variables, hoping that we can figure out the size of the stack main needs from their sizes. (Of course if the function used a Matrix class of the sort where the matrix is kept by value right there in a flat member array, looking at the named local variables mentioned in the program wouldn't be enough since the temporaries returned by overloaded operators would also have to be taken into account; luckily this isn't the case.)

(gdb) info locals
# if it ain't printing garbage, it ain't a C++ debugger:
profile = 0xb7fd9870 "U211?WVS??207"
app = Cannot access memory at address 0xbf3e7a98

We got two local variables; at least one must be huge then. (It can be worse in real life, main functions being perhaps the worst offenders, as many people are too arrogant to start with an Application class. Instead they have an InputParser and an OutputProducer and a Processor, which they proudly use in a neat 5-line main function – why wrap that in a class, 2 files in C++-land? Then they add an InputValidator, an OutputFormatConfigurator and a ProfileLoader, then less sophisticated people gradually add 20 to 100 locals for doing things right there in main, and then nobody wants to refactor the mess because of all the local variables you'd have to pass around; whereas an Application class with two hundred members, while disgusting, at least makes helper functions easy. But I digress again.)

(gdb) p sizeof profile
$2 = 4
(gdb) p sizeof app
$3 = 10485768

"10485768". The trouble with C++ debuggers is that they routinely print so much garbage due to memory corruption, debug information inadequacy and plain stupidity that their users are accustomed to automatically ignore most of their output without giving it much thought. In particular, large numbers with no apparent regularity in their digits are to a C++ programmer what "viagra" is to a spam filter: a sure clue that something was overwritten somewhere and the number shouldn't be trusted (I rarely do pair programming but I do lots of pair debugging and people explicitly shared this spam filtering heuristic with me).

However, in this case overwriting is unlikely since a sizeof is a compile time constant stored in the debug information and not in the program memory. We can see that the number will "make more sense" in hexadecimal (which is why hex is generally a good thing to look at before ignoring "garbage"):

(gdb) p /x sizeof app
$4 = 0xa00008

…Which is similar to our offset value, and confirms that we've been debugging a plain and simple stack overflow. Which would be easy to see in the case of a recursive function, or if the program crashed, say, in an attempt to access a large local array. However, in C++ it will crash near the beginning of a function long before the offending local variable is even declared, in an attempt to push the frame pointer or some such; I think I also saw it crash in naively-looking places further down the road, but I can't reproduce it.

Now we must find out which member of the Application class is the huge one, which is lots of fun when members are plentiful and deeply nested, which, with a typical Application class, they are. Some languages have reflection given which we could traverse the member tree automatically; incidentally, most of those languages don't dump core though. Anyway, in our case finding the problem is easy because I've made the example small.

(I also tried to make it ridiculous – do you tend to ridicule pedestrian code, including your own, sometimes as you type? Few do and the scarcity makes them very dear to me.)

class Application
{
 public:
  Application(const char* profile);
  void mainLoop();
 private:
  static const int MAX_BUF_SIZE = 1024;
  static const int MAX_PROF = 1024*10;
  const char* _profPath;
  char _parseBuf[MAX_BUF_SIZE][MAX_PROF];
  Profile* _profile;
};

This shows that it's _parseBuf that's causing the problem. This also answers the question of an alert C++ apologist regarding all of the above not being special to C++ but also relevant to C (when faced with a usability problem, C++ apologists like to ignore it and instead concentrate on assigning blame; if a problem reproduces in C, it's not C++'s fault according to their warped value systems.) Well, while one could write an equivalent C code causing a similar problem, one is unlikely to do so because C doesn't have a private keyword which to a first approximation does nothing but is advertised as an "encapsulation mechanism".

In other words, an average C programmer would have a createApplication function which would malloc an Application struct and all would be well since the huge _parseBuf wouldn't land on the stack. Of course an average C++ programmer, assuming he found someone to decipher the core dump for him as opposed to giving up on the OS upgrade or the overseas code upgrade, could also allocate the Application class dynamically, which would force him to change an unknown number of lines in the client code. Or he could change _parseBuf's type to std::vector, which would force him to change an unknown number of lines in the implementation code, depending on the nesting of function calls from Application. Alternatively the average C++ programmer could change _parseBuf to be a reference, new it in the constructor(s) and delete it in the destructor, assuming he can find someone who explains to him how to declare references to 2D arrays.

However, suppose you don't want to change code but instead would like to make old code run on the new machine – a perfectly legitimate desire independently of the quality of the code and its source language. The way to do it under Linux/tcsh is:

unlimit stacksize

Once this is done, the program should no longer dump core. `limit stacksize` would show you the original limit, which AFAIK will differ across Linux installations and sometimes will depend on the user (say, if you ssh to someone's desktop, you can get a lesser default stacksize limit and won't be able to run the wretched program). For example, on my wubi installation (Ubuntu for technophopes like myself who have a Windows machine, want a Linux, and hate the idea of fiddling with partitions), `limit stacksize` reports the value of 8M.

Which, as we've just seen, is tiny.

123 comments ↓

#1 Barry Kelly on 09.04.09 at 1:34 pm

8M for a stack is not tiny in 32-bit land, in fact it's very large. This important in practice if you use many threads, as the preallocated stacks will use all your address space, if not actual committed memory. But I assume you're running 64-bit, as 32-bit generally defaults stack to 1M or so.

#2 astrange on 09.04.09 at 3:31 pm

One of the gcc 4.5 projects (recently commited) adds perfect debuginfo for optimized programs with not much more compile time. People who want backtraces with -fomit-frame-pointer are still stuck, though.

#3 sn on 09.04.09 at 10:59 pm

Doesn't your main function need to return a value at the end?

#4 Yossi Kreinin on 09.05.09 at 1:01 am

@astrange: Actually, it's not that hard to get a backtrace under -fomit-frame-pointer; perhaps I'll blog about it. The Green Hills debugger does that and I think the compiler stores things in the debug info for this to work, however heuristic scripting can do quite well given just a symbol table and a disassembler.

@Barry: I didn't mean to say tiny, I meant to say <sarcasm>tiny</sarcasm> – but you know how these blogging engines mess with your markup. Regarding the preallocation of stacks for threads – so what happens under `unlimit stacksize' then? (I'd assume that Linux doesn't really preallocate physical memory for stacks but it does have to preallocate virtual addresses since they ought to be contiguous, therefore there can be no such thing as unlimited stack size, even if unlimited means "until we run out of bits to encode pointers", because if there are many threads, you need to divide the address space between them, and Linux doesn't even know how many threads a program will use concurrently; so `unlimit stacksize' would simply mean to use some large limit built into the system.)

@sn: it doesn't.

#5 Antoñico on 09.05.09 at 1:49 am

It's pretty dumb to create such big instances in the stack. The stack is for temporary and little operations, to avoid overhead of heap memory allocation, as allocating memory in the stack is just a stack pointer adjustment (e.g. the fastest possible memory allocation).

C++ is very criticable, but come on, this post is just ignorance.

#6 Yossi Kreinin on 09.05.09 at 3:04 am

> It’s pretty dumb to create such big instances in the stack.

No shite, Sherlock!

#7 Entity on 09.05.09 at 3:42 am

> It’s pretty dumb to create such big instances in the stack.

Yes but the functionality is there to do so, though the problem is that there is no standard or common way between compilers to detect such events occuring.

I think the issue with allocating on the stack v heap is a red herring. The problem is that their is not a fast effective way of detecting such problems unless your experienced with assembly and the technical details of the envrioment the application is running in.

Though, sometimes less is more and in this situation most customers will have no idea what the problem is and wouldnt really care.

Thank you Yossi Kreini for another post. Allways look forward to see what new things you run into.

Keep it up.
Chad.

#8 SB on 09.05.09 at 8:16 am

Using the stack to allocate more than 16K is considered dumb in many places.

#9 Doug the newby C++ dude on 09.05.09 at 8:35 am

So do any lint-ish tools detect the issue? Did you get any warnings when you compiled your Application class? I'm always leery when I snarf a big block of space. Where did [1024][10] come from?

#10 Doug the newby C++ dude on 09.05.09 at 8:37 am

er, [1024][1024*10]

#11 Yossi Kreinin on 09.05.09 at 10:09 am

What should LINT or the likes of it do – warn about the size of your objects when it exceeds what threshold? 16K the universal dumbness constant from the comment above?

No, I didn't get any warnings, in fact this example was inspired by a program that allocated >300M using a similar 2D array and AFAIK it compiled under gcc -Wall -Werror.

#12 Vladimir Levin on 09.09.09 at 8:31 pm

I am a bit confused. In the actual app where this problem originally came up, was that the first time someone had added the private attribute with the array size associated with it, or did this somehow work with an earlier version of C++?

#13 Yossi Kreinin on 09.10.09 at 11:53 am

There was, sadly, more than one actual app where I saw this problem. The example with the huge member array preventing you from entering main() is close to a particular case, where the problem didn't come up until someone ssh'd to someone else's machine and ran the program (which previously ran fine for what I think was months). Presumably the default stack size limit varied with the user/machine combination.

This is unrelated to "versions of C++" (be it the language standard or the compiler version that you refer to) where the OS sets the stack size limit. It can depend on compiler version if the stack is effectively allocated statically by the compiler (actually a linker script shipped with it), as can be the case in embedded systems; although there you frequently maintain your own linker script anyway.

#14 Frank on 09.12.09 at 9:00 pm

Your criticisms of C++ are unfair do to the fact that they have nothing to do with C++.

#15 Yossi Kreinin on 09.12.09 at 10:41 pm

Where else could you encounter similar behavior?

#16 Sean on 09.30.09 at 10:35 am

You would have the exact same issue in C and any other language that allows allocations on the stack with no runtime checking:

#define MAX_BUF_SIZE 1024
#define MAX_PROF 1024*10
struct Application {
char* _profPath;
char _parseBuf[MAX_BUF_SIZE][MAX_PROF];
};

C++ has plenty of warts, but this isn't one of them.

#17 Yossi Kreinin on 09.30.09 at 11:41 am

RTFA. In C you'd be very likely to malloc the forward-declared struct – C encapsulation encapsulates much more than C++'s does.

#18 Yoed Stavi on 10.14.09 at 1:42 pm

You said:

"Alternatively the average C++ programmer could change _parseBuf to be a reference, new it in the constructor(s) and delete it in the destructor, assuming he can find someone who explains to him how to declare references to 2D arrays."

I'll take that challenge :)
is there anything non-standard with my solution?
are there any problems with my implementation of
class Application that weren't there to begin-with?

——————————————————–
class Profile;

class Application
{
public:
Application(const char* profile);
~Application();
void mainLoop();
private:
static const int MAX_BUF_SIZE = 1024;
static const int MAX_PROF = 1024*10;

typedef char arr2D_t[MAX_BUF_SIZE][MAX_PROF];

const char* _profPath;

void *rawMem;

// here's the damn ref to a 2D array.
arr2D_t& _parseBuf;
Profile* _profile;
};

Application::Application(const char *profile):
rawMem(new char[sizeof(arr2D_t)]),
_parseBuf(*reinterpret_cast(rawMem))
{
}

Application::~Application()
{
delete [] &_parseBuf;
}

int main()
{
Application myApp("foo");
// ran fine on my machine…
return 0;
}

#19 Yoed Stavi on 10.14.09 at 1:48 pm

the reinterpret_cast in my version had the necessary
&lt arr2D_t * &gt

the comment system ate it…

I hope it makes more sense now.

#20 Yoed Stavi on 10.14.09 at 1:51 pm

how do you make triangular braces on this forum anyway?

<>

#21 Yoed Stavi on 10.14.09 at 1:54 pm

OK, I think I figured it out,
it was supposed to look like this:

*reinterpret_cast<>(rawMem)

#22 Yoed Stavi on 10.14.09 at 1:55 pm

no I didn't get it… giving-up now

#23 Yossi Kreinin on 10.14.09 at 11:31 pm

@Yoed: pretty much along the lines of what I thought; actually you don't need to keep rawMem as you can delete &_parseBuf in the destructor. A large share of C++ programmers wouldn't know how to typedef the 2D array type.

#24 Yoed Stavi on 10.15.09 at 2:24 am

I tried first without the rawMem.
it cries about needing a temp variable in the initialization phase. so I gave it perm variable instead.

if you'll notice, I do use &_parseBuf in the destructor.

btw, having to still use delete [] (with the square braces) is a pitfall that's easy to miss IMO.

#25 S.H. on 10.27.09 at 9:36 am

I'm Sherlock. Did anyone asked for my assistance?

ps/ Oy! I typed "yes" or "y" in the "Human?" field and got some machine-like language telling me to do what I just did. Brainless machines…

#26 Yossi Kreinin on 10.27.09 at 10:56 pm

@Sherlock: no shit, Sherlock! You'd be better off if you didn't strip non-alphanumeric characters before parsing natural language.

#27 Chris on 06.30.10 at 1:57 pm

Note: In Fedora limit doesn't exist, I used this to get the stack size:
ulimit -s

#28 AndiM on 10.09.10 at 4:30 am

@Chris:
That's not a "Fedora vs. the world" issue -
the author took care to write it as "Linux/tcsh".
Thus, just trade your bash (which as you said has the ulimit builtin) for something usable in the context of this article.

#29 rmn on 05.06.11 at 1:52 am

@Yoed: I guess you'd want to avoid reinterpret_cast (portability issues), and use static_cast(static_cast(rawMem)) instead – which is well defined.

@Yossi: I liked the debug process you did there, but wouldn't a simple glimpse at the code given you the answer already?
I am asking because I find people are split to two groups regarding debugging:
- Those that just go ahead and gdb the hell outta their binary as soon as a problem is encountered,
- And those (myself included, I must admit), that rather re-read the code a few times, and then use printf debugging, all just to avoid gdb :)

As with everything, professionals must balance both of these approaches.

#30 Yossi Kreinin on 05.06.11 at 2:43 am

@rmn: well, in my example you could probably see the problem since it was really small, but in a larger program, when the program crashes at a place where you don't really expect it to crash no matter what data it runs on (that is, no pointer dereferencing, no division, etc.), I don't see how reading code would help. The large object could be (and in reality, will be) defined deep down in some member of a member of a…

Of course if you've already seen a similar bug, you'll be looking for a large object – but if you haven't, reading source doesn't really point you to the problem very directly since nothing interesting happens at the source level. A similar and perhaps nastier case is when you get a misaligned stack pointer and crash at some place where the compiler used an x86 vector instruction that only works with aligned addresses. There's absolutely nothing wrong at the source level so you have to disassemble to make any sense of what you see. (The difference here is that it's not the fault of standard debugging tools, really, since standard compilers will set up the stack properly; this is the fault of whoever glued shared libraries built with different calling conventions together or similar – although the tools could do at least something to detect an ABI mismatch and they don't. Whereas in my example, it's pretty standard and widely encouraged C++ style – "local variables are the best" – and then you get no help at all from the tools to figure out what went wrong.)

#31 Human on 08.19.11 at 6:09 am

There's this one thing I don't understand and it itches.

"RTFA. In C you’d be very likely to malloc the forward-declared struct"
Why wouldn't one be just as likely to "new" the class in C++?

#32 Yossi Kreinin on 08.19.11 at 7:31 am

Well, obviously nothing prevents one from new'ing an object rather than stack-allocating – it's a question of what the preferred style is.

In C, "encapsulation" generally means ADTs – forward-declared structs implemented in a .c file, hiding their internals to the point where you can't allocate an object yourself since you don't know its size – so you call an allocation function provided in the .h file for the ADT.

In C++, "encapsulation" generally refers to the "private:" keyword – a class keeps its implementation details in the "private:" section, and the compiler can thus allocate an object anywhere where the .h is included, since it knows the object size, the price being that it also has to read the .h files describing the classes of the members. The common C++ style encourages allocation on the stack, both because its more efficient than allocation on the heap and because it's the most straightforward way to get RAII-style resource management.

Again, it's not about what's possible but about what's likely; the commonly advocated and used style affects the likelihood if not the possibilities.

#33 Dmitry Rubanovich on 03.03.12 at 5:17 pm

Instead of this:
char _parseBuf[MAX_BUF_SIZE][MAX_PROF];

you can do the following (and allocate on the heap instead of stack):

std::vector _parseBufVec(MAX_BUF_SIZE*MAX_PROF);
typedef char _ParseBuf[MAX_BUF_SIZE][MAX_PROF];
_ParseBuf & _parseBuf = *reinterpret_cast(_parseBufVec.data());

reinterpret_cast should make the two views on the same memory act as if they were
two entries in a union. I believe that was the original intention of reinterpret_cast.

#34 Dmitry Rubanovich on 03.03.12 at 5:20 pm

Ok, there is no preview button, but the reinterpret cast was supposed to be like this:
reinterpret_cast<_ParseBuf*>(….
the <,> got parsed away (probably because they look like a tag).

#35 Yossi Kreinin on 03.03.12 at 10:57 pm

Yeah, that's one way to do it, although I'd probably typedef the 2D array type to something and then new/delete it myself instead of using std::vector and reinterpret_cast, just so that there's a teeny bit less of tasty C++ goodness involved.

#36 Dmitry Rubanovich on 04.04.12 at 8:48 am

Well, if you do all the new/delete on your own, you risk leaking if anything throws and something in a higher stack frame catches it. You then need to start wrapping your allocation/deallocation in ctor/dtor; or to use smart pointers. Before long, you'll be writing quite a bit more code than a simple (ok, that's definitely a euphemism) cast would have been.

#37 Yossi Kreinin on 04.04.12 at 8:58 am

Much of the code I work on is compiled without exception support, precisely to not care about this sort of thing.

#38 Ray Allen Gebhardt on 05.12.12 at 11:18 am

Why wouldn't you just do something like the following instead?

Application app = new Application(profile);
app->mainLoop();
delete app;

That sounds like a far better fix. And just so you know, you can run into problems with the stack in pretty much any programming language that exists. This issue doesn't just exist in C++.

#39 Yossi Kreinin on 05.12.12 at 9:48 pm

You could do that, as I mentioned. As to other languages – most allocate less on the stack and report stack overflows more gracefully.

#40 José Pedro on 05.25.12 at 11:56 am

Hi've got an easy solution for you: just move the app declaration out of main. That way that variable will be statically allocated by the linker in the data section, and you'll have no problem with the stack.

I do like C++, altough not the metatemplate interpreter builtin, and having spent the last few days reading your FQA, I'm trying not to sound like a C++ apologist…

Incidentally, 64-bit Linux also allocates 8M for the stack by default, so this isn't a 32-bit vs 64-bit issue.

#41 José Pedro on 05.25.12 at 12:00 pm

Yep, ended sounding dumb instead, profile is a parameter for the constructor.

My other solution would be like Ray's one, above.

#42 Yossi Kreinin on 05.26.12 at 4:25 am

Well, sure, there are plenty of workarounds, once you figure out the problem – which many, many competent C++ programmers really can't.

#43 Steve on 06.17.12 at 12:44 pm

For certain values of competent.

#44 Tony Kao on 08.01.12 at 3:55 pm

I've seen lots of C programs (particularly vendor supplied "examples" for microcontrollers) that stack overflow spectacularly because they thought to allocate the entire data buffer on the stack, and have the declaration buried deep within nested loops and switches.

This is certainly not a C++ problem, more of a brain-dead developer problem.

#45 Kronos on 07.18.13 at 2:42 pm

Yes, a matrix was to large to statically allocate, what is the limit on statically allocated variable ?

#46 Bacc on 01.22.14 at 7:00 pm

Well explained. I was getting the same error in a legacy C code and I never considered it to be a stack overflow.

#47 http://www.soccerjerseysclubs.com/au-da-voi-dong-doi-pato-thao-than-khoi-trung-quoc.html on 03.29.19 at 3:42 am

Tremendous issues here. I am very satisfied to look your article.

Thank you so much and I'm having a look ahead to
contact you. Will you please drop me a e-mail?

#48 https://forums.mozfr.org/memberlist.php?mode=viewprofile&u=84303 on 03.29.19 at 10:07 pm

Good day I am so happy I found your webpage, I really found you by error,
while I was browsing on Google for something else, Nonetheless I am here now and
would just like to say thanks for a marvelous
post and a all round interesting blog (I also love the theme/design), I don’t
have time to read it all at the minute but I have saved it and
also added in your RSS feeds, so when I have time I will be back to read a lot more, Please do keep up the excellent work.

#49 https://trello.com/nhacaimibet on 04.01.19 at 2:55 am

This post is in fact a nice one it assists new net users, who are wishing in favor
of blogging.

#50 EstelleEmell on 04.24.19 at 10:42 am

EquiNet Get 2,550 USD
It is specially designed for lifetime business space with zero risk management system. Many people got frustrated with the SCAM and fake business platform. All about the cause of unbalanced between income and expense.

[url=https://equinet.club/?r=longo]EquiNet[/url] is an open business platform, transparent and flexible. You can easily calculate the cash flow of company and make sure the system never collapses and will run for a lifetime.

We learned more about most of the online business platform and we found the solution to save all of the online businessmen. EquiNet is working simply! We create, develop, maintain, and provide a networking business platform. We make sure the platform is working for a lifetime by using the zero risk management system. [url=https://equinet.club/?r=longo]Join us now![/url] It's time to stop all of the risky business and start to get networking bonus for a lifetime.

The Administrator and Service Provider of the platform is EquiNet Club LTD, 12 Queen Anne Street, Marylebone, London, United Kingdom, W1G 9LF, registered under the number 11836193.

Regards,
[url=https://equinet.club/?r=longo]The EquiNet Team[/url]
March 14, 2019

RUS

EquiNet получить 2,550 USD
Матричный проект [url=https://equinet.club/?r=longo]EquiNet[/url] создан для постоянного дохода на всю жизнь с zero системой управления рисками. Многие люди разочаровались в мошенничестве и поддельными бизнес-платформами. Вот и вся причина дисбаланса между доходами и расходами.
[url=https://equinet.club/?r=longo]EquiNet[/url] – это открытая бизнес-платформа, прозрачная и гибкая. Вы можете легко рассчитать денежный поток компании и убедиться, что система никогда не рухнет и будет работать всю жизнь.
Мы анализировали о большинстве онлайн-бизнес-платформ, и мы нашли решение, чтобы обезопасить всех онлайн-бизнесменов. EquiNet работает просто! Мы создаем, развиваем, поддерживаем и предоставляем сетевую бизнес-платформу. Мы гарантируем, что платформа будет работать на протяжении всей жизни, используя систему управления нулевыми рисками. [url=https://equinet.club/?r=longo]Присоединяйтесь к нам сейчас![/url] пришло время остановить весь рискованный бизнес и начать получать сетевую прибыль на всю жизнь.

Администратором и поставщиком услуг платформы является EquiNet Club LTD, 12 Queen Anne Street, Marylebone, Лондон, Великобритания, W1G 9LF, зарегистрированный под номером 11836193.

С уважением,
[url=https://equinet.club/?r=longo]Команда EquiNet[/url]
14 марта 2019 г.

[img]https://chart.googleapis.com/chart?cht=qr&chl=https%3A%2F%2Fequinet.club%2F%3Fr%3Dlongo&choe=UTF-8&chs=320×320[/img]

#51 battlefield 1 esp on 05.15.19 at 12:54 pm

Cheers, great stuff, I enjoying.

#52 fortnite aimbot download on 05.16.19 at 12:06 pm

This does interest me

#53 aimbot fortnite on 05.16.19 at 3:59 pm

I have interest in this, danke.

#54 nonsensediamond on 05.17.19 at 6:12 am

Awesome, this is what I was searching for in bing

#55 fallout 76 cheats on 05.17.19 at 9:41 am

I was looking at some of your articles on this site and I believe this internet site is really instructive! Keep on posting .

#56 red dead redemption 2 digital key resale on 05.17.19 at 2:52 pm

I really enjoy examining on this web , it has got fine stuff .

#57 redline v3.0 on 05.17.19 at 5:54 pm

Your site has proven useful to me.

#58 Megan Guy on 05.17.19 at 7:56 pm

I'm gratified by the manner in which yosefk.com handles this type of issue! Usually to the point, often contentious, consistently well-researched and challenging.

#59 chaturbate hack cheat engine 2018 on 05.18.19 at 7:19 am

This is cool!

#60 sniper fury cheats windows 10 on 05.18.19 at 2:13 pm

Ha, here from google, this is what i was looking for.

#61 mining simulator codes 2019 on 05.19.19 at 6:11 am

Cheers, here from yanex, i enjoyng this, I come back again.

#62 smutstone on 05.20.19 at 10:50 am

I like this site, useful stuff on here : D.

#63 c++ evening courses london on 05.20.19 at 11:30 am

Hey very cool blog!! Guy .. Excellent .. Amazing ..
I will bookmark your blog and take the feeds additionally?
I am happy to find numerous helpful info right here within the post, we'd like work out more
strategies in this regard, thanks for sharing.
. . . . .

#64 redline v3.0 on 05.21.19 at 6:17 am

This does interest me

#65 free fire hack version unlimited diamond on 05.21.19 at 3:28 pm

Your web has proven useful to me.

#66 nonsense diamond on 05.22.19 at 5:19 pm

This is great!

#67 krunker aimbot on 05.23.19 at 5:37 am

I am not rattling great with English but I get hold this really easygoing to read .

#68 bitcoin adder v.1.3.00 free download on 05.23.19 at 9:15 am

Thanks for this site. I definitely agree with what you are saying.

#69 vn hax on 05.23.19 at 6:00 pm

Great, google took me stright here. thanks btw for this. Cheers!

#70 eternity.cc v9 on 05.24.19 at 6:47 am

Ha, here from bing, this is what i was searching for.

#71 how to get help in windows 10 on 05.24.19 at 7:39 am

Thank you for the auspicious writeup. It actually was once a leisure account it.
Glance complicated to more introduced agreeable from
you! By the way, how could we communicate?

#72 ispoofer pogo activate seriale on 05.24.19 at 5:08 pm

Very interesting points you have remarked, appreciate it for putting up.

#73 cheats for hempire game on 05.26.19 at 5:50 am

Hello, here from yahoo, me enjoyng this, I come back soon.

#74 iobit uninstaller 7.5 key on 05.26.19 at 8:38 am

Me like, will read more. Thanks!

#75 gamefly free trial on 05.26.19 at 11:30 am

Woah! I'm really digging the template/theme of this blog.

It's simple, yet effective. A lot of times it's tough to get
that "perfect balance" between superb usability and appearance.

I must say you have done a superb job with this. Also, the
blog loads very fast for me on Chrome. Outstanding Blog!

#76 smart defrag 6.2 serial key on 05.26.19 at 2:58 pm

I conceive this web site holds some real superb information for everyone : D.

#77 resetter epson l1110 on 05.26.19 at 5:22 pm

Found this on MSN and I’m happy I did. Well written website.

#78 Isiah Sochor on 05.27.19 at 5:44 am

I'm gratified by the manner in which yosefk.com handles this kind of topic! Generally on point, often contentious, always well-researched as well as challenging.

#79 sims 4 seasons code free on 05.27.19 at 6:39 am

Ni hao, i really think i will be back to your page

#80 rust hacks on 05.27.19 at 7:17 pm

Awesome, this is what I was looking for in bing

#81 gamefly free trial on 05.28.19 at 9:00 am

Hi! I've been reading your weblog for a while now and finally got the bravery to go ahead
and give you a shout out from Atascocita Tx! Just wanted
to tell you keep up the good job!

#82 strucid hacks on 05.28.19 at 9:34 am

I must say got into this web. I found it to be interesting and loaded with unique points of interest.

#83 expressvpn key on 05.28.19 at 6:39 pm

I like this site, useful stuff on here : D.

#84 ispoofer pokemon go license key on 05.29.19 at 7:46 am

Enjoyed examining this, very good stuff, thanks .

#85 aimbot free download fortnite on 05.29.19 at 11:46 am

Very interesting points you have remarked, appreciate it for putting up.

#86 redline v3.0 on 05.29.19 at 4:14 pm

This helps. Thanks!

#87 gamefly free trial on 05.29.19 at 8:08 pm

I visited many web sites except the audio feature for audio songs present
at this website is genuinely marvelous.

#88 vn hax on 05.30.19 at 5:24 am

Enjoyed examining this, very good stuff, thanks .

#89 how to get help in windows 10 on 05.30.19 at 3:12 pm

These are truly impressive ideas in about blogging. You have
touched some good things here. Any way keep
up wrinting.

#90 xbox one mods free download on 05.31.19 at 11:57 am

Great, yahoo took me stright here. thanks btw for this. Cheers!

#91 fortnite aimbot download on 05.31.19 at 2:43 pm

Respect to website author , some wonderful entropy.

#92 mpl pro on 06.01.19 at 5:52 pm

I have interest in this, thanks.

#93 gamefly free trial on 06.01.19 at 11:47 pm

Write more, thats all I have to say. Literally, it
seems as though you relied on the video to make your point.

You definitely know what youre talking about, why throw away your intelligence on just
posting videos to your site when you could be giving us something enlightening to read?

#94 hacks counter blox script on 06.02.19 at 5:57 am

I was looking at some of your articles on this site and I believe this internet site is really instructive! Keep on posting .

#95 gamefly free trial on 06.02.19 at 7:22 pm

Awesome! Its genuinely awesome paragraph, I have got much clear
idea concerning from this article.

#96 gamefly free trial on 06.02.19 at 10:55 pm

Great article! This is the kind of info that are meant
to be shared around the net. Disgrace on Google for no longer
positioning this put up upper! Come on over and visit my website .
Thank you =)

#97 vn hax pubg on 06.03.19 at 9:46 am

Cheers, great stuff, Me like.

#98 gamefly free trial on 06.03.19 at 10:21 am

Thankfulness to my father who stated to me regarding this web site, this webpage
is genuinely remarkable.

#99 gamefly free trial on 06.04.19 at 5:20 am

Hello! I've been following your blog for a long time now and finally got
the courage to go ahead and give you a shout out from
Humble Tx! Just wanted to say keep up the excellent work!

#100 gamefly free trial on 06.04.19 at 3:47 pm

I was wondering if you ever thought of changing the page layout of
your blog? Its very well written; I love what youve got
to say. But maybe you could a little more in the way of content so people could connect with
it better. Youve got an awful lot of text for only having one or two images.
Maybe you could space it out better?

#101 Franmuh on 06.05.19 at 6:36 am

Acticin Can I Purchase No Doctor Viagra 71 Ans Propecia Vault [url=http://6drugs.com]generic cialis canada[/url] Buying Generic Zentel

#102 gamefly free trial on 06.05.19 at 5:03 pm

Everything is very open with a very clear description of the challenges.
It was definitely informative. Your site is very helpful.
Thanks for sharing!

#103 gamefly free trial on 06.07.19 at 11:34 am

Good day I am so excited I found your site, I really found you by error, while I was searching on Yahoo for something else, Anyways I
am here now and would just like to say many thanks for a marvelous post and a all round thrilling blog (I also love the theme/design), I don't have time to browse it all
at the moment but I have bookmarked it and also added in your RSS feeds, so when I have time I will be back
to read a lot more, Please do keep up the fantastic work.

#104 https://trello.com/thuockichducnam on 06.08.19 at 8:21 pm

My brother suggested I would possibly like this web site.

He was entirely right. This post truly made my day.

You can not believe just how so much time I had spent for this information! Thanks!

#105 gamefly free trial 2019 coupon on 06.10.19 at 8:37 pm

I simply could not depart your website before suggesting that I extremely enjoyed
the standard information a person supply on your visitors?
Is gonna be back regularly to check out new posts

#106 Franmuh on 06.12.19 at 6:57 am

Salbutamol Nebules Buy Buy Finasteride Citrate Liquid Will Keflex Help With Bronchitis [url=http://viacheap.com]viagra[/url] Cialis Original Contrareembolso Methocarbamol Germany

#107 playstation 4 best games ever made 2019 on 06.12.19 at 11:24 pm

I'm amazed, I must say. Rarely do I come across a blog that's equally educative and engaging, and let me tell you, you've hit the nail on the head.
The issue is something that too few people are speaking intelligently about.
I am very happy that I stumbled across this during my search for something regarding this.

#108 quest bars cheap on 06.14.19 at 4:55 pm

Quality content is the crucial to interest the viewers to go to see the web page, that's what
this web page is providing.

#109 quest bars cheap on 06.15.19 at 6:17 am

This information is priceless. Where can I find out more?

#110 fortnite aimbot free on 06.16.19 at 4:53 pm

I kinda got into this web. I found it to be interesting and loaded with unique points of view.

#111 tinyurl.com on 06.17.19 at 2:19 pm

Thanks for finally talking about > The C++ Sucks
Series: petrifying functions < Loved it!

#112 proxo key on 06.19.19 at 7:21 am

I consider something really special in this site.

#113 Franmuh on 06.20.19 at 9:39 am

Buy Thyroxine Uk Is Cephalexin A Sulfa Drug Viagra Generic Kaufen In Holland [url=http://rxasian.com]viagra[/url] Vente Baclofen En Canada Tadalis Sx Pfizer Keflex For Mrsa Infections

#114 vn hax on 06.20.19 at 4:26 pm

I truly enjoy looking through on this web site , it holds superb content .

#115 nonsense diamond 1.9 on 06.21.19 at 5:36 am

Appreciate it for this howling post, I am glad I observed this internet site on yahoo.

#116 plenty of fish dating site on 06.21.19 at 9:36 pm

I like the valuable info you provide in your articles. I will bookmark your weblog and check again here regularly.
I am quite sure I'll learn lots of new stuff right
here! Good luck for the next!

#117 plenty of fish dating site on 06.22.19 at 8:45 am

Now I am ready to do my breakfast, once having my
breakfast coming over again to read other news.

#118 quest bars cheap on 06.23.19 at 9:07 am

I'd like to find out more? I'd love to find out some additional information.

#119 badoo superpowers free on 06.23.19 at 3:12 pm

Cheers, great stuff, Me like.

#120 gx tool pro on 06.24.19 at 1:20 pm

I must say, as a lot as I enjoyed reading what you had to say, I couldnt help but lose interest after a while.

#121 free online Q & A on 06.25.19 at 3:00 am

This is good. Thanks!

#122 fortnite mods on 06.25.19 at 6:00 pm

Hey, google lead me here, keep up nice work.

#123 skisploit on 06.26.19 at 4:39 am

I’m impressed, I have to admit. Genuinely rarely should i encounter a weblog that’s both educative and entertaining, and let me tell you, you may have hit the nail about the head. Your idea is outstanding; the problem is an element that insufficient persons are speaking intelligently about. I am delighted we came across this during my look for something with this.

Leave a Comment