Everything I know about getting the current call stack of C or C++ programs, including ones compiled with -fomit-frame-pointer or an equivalent, with or without a debugger. Hardly entertaining except for those specifically wishing to lay their hands on call stacks.
We'll start with trivia people sharing my unhealthy interest in call stacks probably know. There are two contexts where you might want to get a call stack:
- Look at the call stack in a debugger to see what the program is doing.
- Get a representation of the call stack inside the program itself. For example, a memory profiler might want to attach the call stack identifying the context where the allocation is made to each memory block to see who allocates the most.
Sometimes (2) can be implemented using (1) by running a debugger programmatically and asking it for the current call stack, and sometimes it can't (too much communication overhead, or no debugger available – for example, a program stripped of debugging information and running on a standalone embedded board).
The straightforward way of getting the current call stack, used both by debuggers and by programs curious about their own stacks, relies on the frame pointer. The idea is that a machine register is reserved for keeping a pointer into the stack, called the frame pointer, and every function is compiled to do the following in its prologue:
- Push the return address to the stack
- Push the frame pointer to the stack
- Save the address of the resulting two-pointer structure to the frame pointer register
This creates a linked list on the stack, with every node keeping a return address – this list is the call stack (and this is why debuggers show the points of return from function calls and not the points of call, a bit annoyingly for function calls spanning many source code lines – in fact we get the return stack and not the call stack). Here's how you get this list from within a program:
struct stack_frame {
struct stack_frame* next;
void* ret;
};
int get_call_stack(void** retaddrs, int max_size) {
/* x86/gcc-specific: this tells gcc that the fp
variable should be an alias to the %ebp register
which keeps the frame pointer */
register struct stack_frame* fp asm("ebp");
/* the rest just walks through the linked list */
struct stack_frame* frame = fp;
int i = 0;
while(frame) {
if(i < max_size) {
retaddrs[i++] = frame->ret;
}
frame = frame->next;
}
return i;
}
The code for getting the list head pointer depends on the platform, the list structure itself is common to many machines and compilers. The return addresses may be converted to function names and source line numbers with addr2line -f or similar. When the program can't access its own debug info during its execution as in the case of embedded devices, the translation from addresses to names will be a separate offline step.
The whole frame pointer business is fairly widely documented, with a bunch of source code available centered around getting the call stack that way. I think the GNU backtrace function and the Windows StackWalk64 function, which these days are probably a better alternative than code snippets like the one above, also use this linked list when available.
Now, what happens if the compiler is told to avoid generating the code maintaining the frame pointer (-fomit-frame-pointer for gcc, /Oy for MSVC), or not told to override its default behavior of not generating such code (-ga for Green Hills C++)?
Admittedly it doesn't appear to be such a good idea to make debugging that much harder in order to save a few instructions. However, there are reasons to do so. For example, one common consequence of Software Design is lots of functions doing little or nothing except for delegating their work to another function. Without frame pointer maintenance, such a call is just a jump instruction – your callee function will return directly to the address saved by your caller. With frame pointers, you need a whole prologue and epilogue here. Anyway, we won't discuss the benefits of frame pointer omission since measuring the overhead for your particular code will be more reliable than such a discussion anyway.
Compiling without frame pointers hits code trying to obtain its own context harder than it hits debuggers, because debuggers don't really need frame pointers and only (sometimes) rely on them for simplicity of implementation. Given a return address, a debugger can tell:
- Which function it belongs to (unlike the program itself, a debugger is necessarily supposed to have access to the symbol table)
- Where that function keeps the return address (the compiler knows that, so it can tell the debugger)
- The amount by which the function decrements the stack pointer, assuming the stack grows downwards – again something the compiler knows. Now that the debugger knows the previous return address and the previous stack pointer, it can go back to step 1.
So a debugger can do just fine without frame pointers as long as the compiler gives it enough information about the layout of the stack. I've been debugging without frame pointers for a long time with the Green Hills MULTI debugger which uses a proprietary debug info format. More recently the DWARF format, gcc and gdb seem to have caught up and now programs compiled with -fasynchronous-unwind-tables -fomit-frame-pointer are debuggable with gdb. The information generated by -fasynchronous-unwind-tables seems to go to a separate ELF section called .eh_frame_hdr.
Not only will gdb use .eh_frame_hdr, but the GNU backtrace function appears to be using it as well (it doesn't work under -fomit-frame-pointer but apparently does work when you add -fasynchronous-unwind-tables – although the docs explicitly say: "frame pointer elimination will stop backtrace from interpreting the stack contents correctly"). Nor is this section stripped from the program – it's not implemented as a "normal" debug information section but as an allocated data section, so it's always available to a program (in particular, to the backtrace function).
So under gcc, all call stack problems seem to be solved – unless you trust the docs (!?), or unless some code isn't compiled with the right flags because of not being up to date or someone being too greedy to allocate space for a debug info section. Outside gcc, or more precisely DWARF, I don't think a stripped program can access such debug info.
Is there a way to get a call stack without a frame pointer, without a debugger and without debug info?
For years I was sure that the answer was "no", hence some things will only work under a separate build mode – just like the release build but with frame pointers. Then one time the Green Hills debugger failed to list the call stack for some reason as it sometimes does, but this time we really wanted to decipher it. And we figured that we can in fact do the same thing the debugger does, except we'll understand from the assembly code what the debugger usually understood from debug information.
Specifically, to understand where the return address is kept and by what amount the stack pointer is decremented, you need to find the instructions doing (or undoing) these things in the prologue (or the epilogue) of the function. This worked, but due to either inertia or stupidity it took me months to realize that you can write code doing this. Anyway, here's how it works on a 32b MIPS processor under the Green Hills compiler. The prologue code of a function will contain instructions like these:
main+0: 27bdffe8 addiu sp, sp, -0x18 main+4: afbf0014 sw r31, 0x14(sp)
The add immediate instruction decrements the stack pointer, and the store word instruction saves the return address from the register where it's saved by the caller to some place on the stack. The high 16 bits of these instructions don't depend on the function, encoding the "addui sp, sp" and the "sw, r31 …(sp)" parts. The low 16 bits encode a signed offset. So we can obtain the call stack from our code disassembling it thusly:
/* get previous stack pointer and return address
given the current ones */
int get_prev_sp_ra(void** prev_sp, void** prev_ra,
void* sp, void* ra) {
unsigned* wra = (unsigned*)ra;
int spofft;
/* scan towards the beginning of the function -
addui sp,sp,spofft should be the first command */
while((*wra >> 16) != 0x27bd) {
/* test for "scanned too much" elided */
wra--;
}
spofft = ((int)*wra << 16) >> 16; /* sign-extend */
*prev_sp = (char*)sp - spofft;
/* now scan forward for sw r31,raofft(sp) */
while(wra < (unsigned*)ra) {
if((*wra >> 16) == 0xafbf) {
int raofft = ((int)*wra << 16) >> 16; /* sign */
*prev_ra = *(void**)((char*)sp + raofft);
return 1;
}
wra++;
}
return 0; /* failed to find where ra is saved */
}
The call stack will then be produced by the following loop:
int get_call_stack_no_fp(void** retaddrs, int max_size) {
void* sp = get_sp(); /* stack pointer register */
void* ra = get_ra(); /* return address register */
/* adjust sp by the offset by which this function
has just decremented it */
int* funcbase = (int*)(int)&get_call_stack_no_fp;
/* funcbase points to an addiu sp,sp,spofft command */
int spofft = (*funcbase << 16) >> 16; /* 16 LSBs */
int i=0;
sp = (char*)sp-spofft;
do {
if(i < max_size) {
retaddrs[i++] = ra;
}
}
while(get_prev_sp_ra(&sp,&ra,sp,ra));
return i; /* stack size */
}
get_sp and get_ra access registers so they must be assembly macros, which in the case of Green Hills can be spelled like this:
asm void* get_ra() {
move $v0, $ra
}
asm void* get_sp() {
move $v0, $sp
}
Under MIPS32 and Green Hills, this code seems to be giving decent call stacks except for the inevitable omission of function calls done without saving the return address; the most common case of those – simple delegating functions – was already mentioned above. If f calls g which does (almost) nothing except for calling h, and g doesn't bother to save the return address to the stack, having h return directly to f, then the call stack will contain f and h but not g. Not much of a problem and sometimes even an advantage as far as I'm concerned, since g is rarely interesting. Also, you can get this problem irregardless of the lack of a frame pointer – for example, gcc -O3 on x86 will not maintain the call stack accurately even without -fomit-frame-pointer, generating the following ridiculous code:
pushl %ebp movl %esp, %ebp popl %ebp ; so what was the point of setting up and ; immediately destroying a stack frame? jmp print_trace
Now, although code like get_prev_sp_ra looking for 0x27bd admittedly is a heuristic relying on undocumented platform-specific behavior, it looks like a passable way of getting a call stack on a RISC machine like MIPS, ARM or PowerPC. What about the x86 though? Effectively we have our code partially disassembling itself here, which is not nearly as easy with the x86 (in particular, I don't think there's a way to scan backwards because of the variable encoding length; although we could look at epilogues instead of prologues just as well).
Instead of dragging in a disassembler, we can use an external program, such as, well, a debugger. This obviously defeats the purpose of being able to get the call stack without a debugger. But this purpose isn't very interesting on the x86 in the first place because there you're rarely stuck in situations where a program can't run a debugger.
The only point of the disassembling business on the x86 thus remains to deal with programs compiled without a frame pointer and without debug information making it possible to get the call stack nonetheless. I don't know if anybody has such a problem these days, now that gcc has -fasynchronous-unwind-tables – perhaps someone uses compilers which can't do this or binaries compiled without this, and perhaps the problem is extinct on the x86. For what it's worth, here's a script getting the call stack from a core file without relying on gdb's bt command but relying on its disassemble command. Usage: python bt <program> <core>. No warranty, …or FITNESS FOR A PARTICULAR PURPOSE.
And this is all I know about getting the call stack in C or C++, something users of other languages can do, in the (unlikely) absence of a library function doing just that, simply by throwing an exception, immediately catching it and using its getStackTrace method or some such.
154 comments ↓
Just a note that x86_64 is more interesting than x86 as the ABI allows them to omit the frame pointer from code and requires a special DWARF section (perhaps the same one you mention.) to allow the debugger to unwind code.
That means your first loop that walks ebp (actually rbp) won't be reliable on x86_64 even if -fomit-frame-pointer is not passed to the compiler.
Well, just use Google Breakpad for this stuff.
The last place I worked, we were using gcc/gdb on MIPS and wrote functions to do this, too. They looked very similar to what you've posted here, but were a little more complicated because we found that some functions made more than one adjustment to the stack frame.
Anyway, we had to create our own mini coredumps for this embedded system because the OS (Nucleus) didn't do it for us. These stack back-tracing functions were the only tool we had for generating a stack trace for the core dumps. It worked surprisingly well. I don't recall ever getting a core dump that didn't include a valid stack trace (save the rare ones that had severe stack corruption… those were usually bad days for me).
@Dan: mini core dumps as in subsets of the program & chip state as opposed to complete snapshots? Painful stuff, I hate to decide what goes in and what stays out – it's a good thing to insist that an embedded system is always hooked to something capable of getting a full core dump during development time.
The thing I hate the most about stack corruption is that if the return address is smashed and you jump to nowhere, you don't know where you jumped from – at least MIPS is careful to first set $pc to $ra, then produce an exception at that address upon jr $ra into the chasm.
Thanks, Yossi! I haven't laughed out loud at a segment of assembly code in a while.
It would be convenient for debugging (in the stack corruption case) if ret exchanged $pc and $ra instead of just copying one to the other. It's not like you're going to return to that same address again, are you?
I don't think x86, which has a ret instruction, has a return address register – ret pops an address from the stack symmetrically to call which pushes one onto the stack, isn't it so?
On MIPS and AFAIK most RISC machines, there is no special ret instruction – there's jr (jump register) or similar; it's the same instruction used for virtual function calls so I guess clobbering the register in the general case would be wrong as it would suck through a loop calling a virtual function, for example, and special-casing $ra under the assumption that it's really used for a return address isn't The RISC Way.
Which of course does not prevent them from stuffing $pc where the last executed jr instruction resided to some kind of register – I'd happily settle for another addition to the zillion user-inaccessible coprocessor registers already available. The reason they don't do it IMO is that Unix/C, and consequently RISC which is the Unix/C back-end of choice, couldn't care less about debugging – they're proud when they correctly record the point where an exception occurred so you can at least meaningfully restart from there.
You're right about x86, and your MIPS comment is also correct for SPARC, ARM, and HP-PA, but not PowerPC, which has a special "link register" LR with magic instructions to access it, blech. But since it's generally used for either function return or virtual function calls, and I doubt the ABI makes it "callee"-saved in either of those cases, changing it from a mov to a swap wouldn't pose much risc to efficiency. It would clearly break programs, though.
(Actually the only case I can think of where it would matter would be when you're implementing direct-threaded Forth…)
Another extra register would be a good idea, but of course to work under an OS, the OS would have to know to save the value on every context-switch, just in case you might need to debug the program. If we can have that, can we also have a ring buffer of the last 64 jumps the program executed? Because that would be really handy too…
@Yossi: Mini coredumps are sometimes the only way to get them from customer equipment, when there is limited flash space (not enough to save a real core file). It also happens that on these systems, the only reliable way to get data out of the box in the face of a core dump was through a 38400kbps serial port. It would have taken up to 31 hours to dump core on one of these, if I did my math right (memory maxed at 512MB, all one monolithic image, single address space, yaddah yaddah). We got *really* creative at coming up with ways to include the most amount of useful information in the smallest amount of data.
@Dan: sure, been there. Just saying that sometimes beating up a PM can bring in more bandwidth and end up being the cheaper solution. Sometimes.
Hi, I really like your article. I'm using it to get some stack dumps in the development phase of our product. I have found that sometimes the return address register in the MIPS 24K is not 32bit aligned. And this causes your example code to throw a CPU exception when dereferencing a non aligned pointer. Just a heads up.
thanks for the effort to document this stuff. Its extremely useful.
Seriously? I'd think the 24K couldn't return (jr $ra) to a misaligned address. Is it a real return address or something located where the code expects a return address that actually isn't?
Seriously looks like that anyway. The stack dump example starts to break down when compiler optimization level is turned up AND the code is compiled with the MIPS16 option.
I'm getting return addresses like the following: 0x9e032d1f
And these optimized functions don't do a
addiu sp,sp,spofft
So now i'm trying to figure out how to get the call stack for functions being jumped to where the calling function uses the jalx instruction.
I'm probably in over my head but this is for extra credit in my down time anyway.
MIPS16 I don't know anything about, although a 0xf sounds too misaligned even for that; anyway, unfortunately it's not surprising that my code above can't handle your case…
yes, i agree 0xF sounds like a very strange offset??? but taking a look at the .map file, all the base address of the functions compiled under MIPS16 are on byte aligned addresses.
Nice article!
I was hoping to find a method to receive the frame pointer in a function when omit-frame-pointer is enabled. I've found out, that GCC provides a builtin function to receive the frame pointer:
void * __builtin_frame_address (unsigned int level);
Unfortunately, usage of that function seems to turn of omit-frame-pointer optimization for the given function.
I don't even know, since when this builtin function exists .. so, it might not be a very portable method.
I'd guess that if the callers' code is compiled with frame pointer omission, then you can't get the callstack even if you (the callee) do set up the frame pointer and have an intrinsic to get that address.
This is pure gold! I am working on MIPS as well and I had often found myself wondering about this same topic. Thank you so much.
Thanks, and I hope it actually works for you…
Thanks, it does help!
But in my MIPS compiler this code will retrieve wrong spofft in the line below.
/* funcbase points to an addiu sp,sp,spofft command */
int spofft = (*funcbase <> 16; /* 16 LSBs */
It's because the first few asm lines deal with $gp first like below:
00400b04 :
400b04: 3c1c0fc0 lui gp,0xfc0
400b08: 279c754c addiu gp,gp,30028
400b0c: 0399e021 addu gp,gp,t9
400b10: 27bdffe0 addiu sp,sp,-32
so I modified your code a little to work around it:
wra = (unsigned*)funcbase;
/* scan from the beginning of the function -
addui sp,sp,spofft may not be the first command */
while((*wra >> 16) != 0x27bd) {
/* test for "scanned too much" elided */
wra++;
}
spofft = ((int)*wra <> 16; /* sign-extend */
Just to share this.
Interesting; does this $gp fiddling happen at every function? Something's off; $gp is usually initialized to a constant (specifically the middle of the sda, "small data area") and then left alone.
Yes, it's in every function. I don't why this happened. I'm using uclibc, and the compile flags is "mipsel-linux-gcc -g -O -mtune=4kc -mips32 "
Right now it sounds like Expression Engine is the preferred blogging platform out there right now.
(from what I've read) Is that what you're using
on your blog?
Keep up the good work! Thanks.
In fact when someone doesn't know after that its up to other people that
they will assist, so here it takes place.
Yes! Finally someone writes about Blonde Milf.
These are actually impressive ideas in on the topic of blogging.
You have touched some good factors here. Any way keep up wrinting.
Hello friends, how is all, and what you want to say regarding this paragraph, in my view its genuinely
amazing in favor of me.
Good, this is what I was searching for in bing
Congrats for the inspiring site you've created at yosefk.com. Your enthusiastic take on the subject is definitely inspiring. Thanks again!
I have learn some good stuff here. Certainly price bookmarking for revisiting.
I surprise how much effort you place to create this sort of magnificent informative site.
My partner and I absolutely love your blog and find nearly all of your post's to be just
what I'm looking for. Would you offer guest writers to write content for you?
I wouldn't mind creating a post or elaborating on some of the subjects
you write regarding here. Again, awesome weblog!
Hmm it looks like your blog ate my first comment (it was super long) so I guess
I'll just sum it up what I submitted and say, I'm thoroughly enjoying your blog.
I as well am an aspiring blog writer but I'm still new to the whole thing.
Do you have any points for beginner blog writers? I'd certainly appreciate
it.
If some one wants to be updated with most recent
technologies then he must be go to see this web page and be up to date all the time.
Heya great blog! Does running a blog similar to this require a great deal
of work? I've very little expertise in computer programming however I was hoping to start
my own blog soon. Anyway, if you have any
ideas or techniques for new blog owners please share. I know this is off subject but I just needed to ask.
Kudos!
Undeniably imagine that which you said. Your favorite reason seemed to be on the web
the easiest factor to consider of. I say to you, I definitely get annoyed whilst folks consider issues that they plainly don't
understand about. You controlled to hit the nail upon the highest and also defined out the
entire thing with no need side effect , other people can take
a signal. Will likely be again to get more. Thank you
Great web site. A lot of useful info here.
I am sending it to some pals ans additionally sharing in delicious.
And obviously, thank you on your sweat!
Great post.
Great website. Lots of helpful information here.
I'm sending it to some buddies ans also sharing in delicious.
And certainly, thank you in your effort!
I blog often and I truly appreciate your information. This article
has really peaked my interest. I am going to take a note of your
site and keep checking for new details about once a week.
I opted in for your RSS feed as well.
I'm gone to convey my little brother, that he should also go to see this website on regular basis to get
updated from hottest news.
Hello! This post could not be written any better! Reading through
this post reminds me of my previous room mate! He always
kept talking about this. I will forward this page to him.
Fairly certain he will have a good read. Many thanks
for sharing!
I loved as much as you'll receive carried out right here.
The sketch is tasteful, your authored subject matter stylish.
nonetheless, you command get bought an edginess over that
you wish be delivering the following. unwell unquestionably come
further formerly again since exactly the same nearly very often inside case you shield this hike.
I'm not sure why but this web site is loading incredibly slow for me.
Is anyone else having this problem or is it a issue
on my end? I'll check back later on and see if the problem still
exists.
Can I just say what a reduction to seek out somebody who actually is aware of what theyre speaking about on the internet. You definitely know the best way to bring a difficulty to mild and make it important. More folks must read this and understand this facet of the story. I cant consider youre no more in style because you positively have the gift.
Hello! I know this is sort of off-topic but I needed
to ask. Does managing a well-established website like yours take a massive amount work?
I'm completely new to writing a blog however I do
write in my journal every day. I'd like to start a blog so I can share my personal experience and thoughts online.
Please let me know if you have any ideas or tips for
brand new aspiring bloggers. Thankyou!
Keep up the good work! Thanks.
Nice post. I study one thing more challenging on completely different blogs everyday. It is going to at all times be stimulating to learn content material from other writers and follow a little something from their store. I抎 desire to make use of some with the content material on my weblog whether you don抰 mind. Natually I抣l offer you a link on your web blog. Thanks for sharing.
There is noticeably a bundle to know about this. I assume you made certain good factors in features also.
I抦 impressed, I need to say. Really rarely do I encounter a blog that抯 both educative and entertaining, and let me let you know, you've got hit the nail on the head. Your idea is outstanding; the difficulty is one thing that not sufficient individuals are speaking intelligently about. I am very completely satisfied that I stumbled throughout this in my seek for something relating to this.
6/4/2019 yosefk.com does it yet again! Quite a interesting site and a well-written post. Nice work!
I completely agree with your take on this subject and look forward to new posts and comments here at yosefk.com. Thanks!
We're a group of volunteers and starting a new scheme in our community.
Your website provided us with helpful information to work
on. You have done an impressive process and our whole group can be grateful to you.
Hands down, Apple's app store wins by a mile. It's a huge selection of all sorts of apps vs a rather sad selection of a handful for Zune. Microsoft has plans, especially in the realm of games, but I'm not sure I'd want to bet on the future if this aspect is important to you. The iPod is a much better choice in that case.
Hands down, Apple's app store wins by a mile. It's a huge selection of all sorts of apps vs a rather sad selection of a handful for Zune. Microsoft has plans, especially in the realm of games, but I'm not sure I'd want to bet on the future if this aspect is important to you. The iPod is a much better choice in that case.
Zune and iPod: Most people compare the Zune to the Touch, but after seeing how slim and surprisingly small and light it is, I consider it to be a rather unique hybrid that combines qualities of both the Touch and the Nano. It's very colorful and lovely OLED screen is slightly smaller than the touch screen, but the player itself feels quite a bit smaller and lighter. It weighs about 2/3 as much, and is noticeably smaller in width and height, while being just a hair thicker.
Between me and my husband we've owned more MP3 players over the years than I can count, including Sansas, iRivers, iPods (classic & touch), the Ibiza Rhapsody, etc. But, the last few years I've settled down to one line of players. Why? Because I was happy to discover how well-designed and fun to use the underappreciated (and widely mocked) Zunes are.
The Zune concentrates on being a Portable Media Player. Not a web browser. Not a game machine. Maybe in the future it'll do even better in those areas, but for now it's a fantastic way to organize and listen to your music and videos, and is without peer in that regard. The iPod's strengths are its web browsing and apps. If those sound more compelling, perhaps it is your best choice.
Between me and my husband we've owned more MP3 players over the years than I can count, including Sansas, iRivers, iPods (classic & touch), the Ibiza Rhapsody, etc. But, the last few years I've settled down to one line of players. Why? Because I was happy to discover how well-designed and fun to use the underappreciated (and widely mocked) Zunes are.
Apple now has Rhapsody as an app, which is a great start, but it is currently hampered by the inability to store locally on your iPod, and has a dismal 64kbps bit rate. If this changes, then it will somewhat negate this advantage for the Zune, but the 10 songs per month will still be a big plus in Zune Pass' favor.
6/10/2019 @ 7:02:57 PM Like the site– very user-friendly and a lot of stuff to think about!
I have read some excellent stuff here. Definitely worth
bookmarking for revisiting. I wonder how much attempt you set to make this kind of fantastic informative web site.
Hey there! This post could not be written any better! Reading this post reminds me of my good old room mate!
He always kept chatting about this. I will forward this post to him.
Pretty sure he will have a good read. Thank you for sharing!
Spot on with this write-up, I absolutely feel this amazing site needs a great deal more
attention. I'll probably be back again to
read more, thanks for the info!
I'm pleased by the manner in which yosefk.com handles this sort of subject. Generally to the point, sometimes polemic, always well-researched and more often than not quite stimulating.
Really when someone doesn't know afterward its up to other visitors that they
will assist, so here it happens.
I pay a quick visit every day a few blogs and
sites to read posts, except this website gives quality based content.
How long does a copyright last on newspaper articles?. . If a service copies newspapers articles and then posts it in a database on the Internet, is there also a copyright on the Internet content?.
Currently it appears like Movable Type is the preferred blogging platform out there right now.
(from what I've read) Is that what you are using on your blog?
This site was… how do I say it? Relevant!! Finally I have found something
that helped me. Kudos!
This does interest me
I'm really enjoying the design and layout of your site.
It's a very easy on the eyes which makes it much more pleasant
for me to come here and visit more often. Did you hire out a designer to create your theme?
Superb work!
How long does a copyright last on newspaper articles?. . If a service copies newspapers articles and then posts it in a database on the Internet, is there also a copyright on the Internet content?.
I consider something really special in this site.
Ha, here from google, this is what i was searching for.
I like this article, some useful stuff on here : D.
Informative article, totally what I wanted
to find.
When someone writes an piece of writing he/she maintains the
image of a user in his/her brain that how a user can know it.
So that's why this paragraph is perfect. Thanks!
each time i used to read smaller articles which also clear
their motive, and that is also happening with this paragraph which I am reading here.
Great stuff to check out, glad that duckduck took me here, Keep Up good job
Yeah bookmaking this wasn’t a risky decision outstanding post! .
This does interest me
How long does a copyright last on newspaper articles?. . If a service copies newspapers articles and then posts it in a database on the Internet, is there also a copyright on the Internet content?.
Thanks for this web. I definitely agree with what you are saying.
Wonderful article! This is the type of info that are supposed to be shared across the internet.
Disgrace on Google for now not positioning this post upper!
Come on over and consult with my website . Thanks =)
This is a topic that is close to my heart… Cheers! Where are your contact details though?
Have you ever considered about adding a little bit more than just your articles? I mean, what you say is valuable and all. But just imagine if you added some great pictures or videos to give your posts more, “popâ€! Your content is excellent but with images and video clips, this website could undeniably be one of the most beneficial in its field.Fantastic blog!
Parasite backlink SEO works well :)
Heya! I’m at work browsing your blog from my new apple iphone! Just wanted to say I love reading through your blog and look forward to all your posts! Keep up the fantastic work!
I am glad to be one of the visitors on this great website (:, appreciate it for posting .
Good Morning, google lead me here, keep up good work.
Hi, after reading this awesome piece of writing i am too happy to share my familiarity here with
colleagues.
I truly enjoy looking through on this web site , it holds superb content .
Hey I know this is off topic but I was wondering if you knew of any widgets I could add
to my blog that automatically tweet my newest twitter
updates. I've been looking for a plug-in like this for quite
some time and was hoping maybe you would have some experience with something like this.
Please let me know if you run into anything. I truly enjoy reading
your blog and I look forward to your new updates.
Thank you for the great read!
some great ideas this gave me!
Look at MyFiveStarMusic. net My spouse and i get excellent songs as well as i am usually changing. It has the farely fresh along with I am looking to get it started. Please check it out. Thank you I love the concept of these kind of, but Personally, i such as experiencing on the net stereo. It just takes at a distance the desire to try to find songs. Merely fixed it, in addition to leave it in the background.
[url=https://blog.glstock.com/?p=11035]coque iphone[/url]
thank you web site admin
I have to thank you for the efforts you have put in writing
this site. I am hoping to check out the same high-grade blog posts by you later on as well.
In truth, your creative writing abilities has
encouraged me to get my own site now ;)
great advice you give
great advice you give
Great beat ! I wish to apprentice while you amend your website, how could i
subscribe for a blog web site? The account helped me a acceptable deal.
I had been tiny bit acquainted of this your broadcast provided bright clear concept
It's difficult to find experienced people in this particular topic, however, you seem like you know what you're talking about!
Thanks
Hey there would you mind sharing which blog platform you're working with?
I'm going to start my own blog in the near future but I'm having a difficult time making a decision between BlogEngine/Wordpress/B2evolution and Drupal.
The reason I ask is because your layout seems different then most blogs and I'm looking for something unique.
P.S Apologies for getting off-topic but I had to ask!
What's Happening i am new to this, I stumbled upon this I have discovered It positively useful and it has helped me out loads.
I am hoping to give a contribution & help other customers
like its aided me. Great job.
Mr.s Fister, this message is your next piece of info. Feel free to transceive the agency at your convenience. No further information until next transmission. This is broadcast #5629. Do not delete.
I am usually to running a blog and i actually recognize your content. The article has really peaks my interest. I am going to bookmark your site and keep checking for new information.
Heya i'm for the primary time here. I found this board and I in finding
It truly useful & it helped me out much. I'm hoping to provide one thing again and aid
others like you helped me.
Hello my family member! I wish to say that this post is amazing, nice written and include approximately all vital infos.
I'd like to peer more posts like this .
Hi there i am kavin, its my first time to commenting anyplace, when i read this piece of writing i thought i could also make comment due
to this sensible paragraph.
amazing content thanks
you are so great
This blog is amazing! Thank you.
Excellent site you have here.. It's hard to find high-quality writing like
yours these days. I honestly appreciate individuals like you!
Take care!!
Skyking, Skyking, this drop is your next bit of data. Do message the agency at your earliest convenience. No further information until next transmission. This is broadcast #6685. Do not delete.
Thanks for finally talking about > Getting the call stack without a frame pointer < Liked it!
Write more, thats all I have to say. Literally, it seems as though you relied on the video to make your point.
You obviously know what youre talking about, why
waste your intelligence on just posting videos to your weblog when you could be giving us something
informative to read?
Excellent goods from you, man. I've understand your stuff previous to and you
are just too wonderful. I really like what you have
acquired here, certainly like what you are saying and the way in which you say it.
You make it enjoyable and you still care for to keep it sensible.
I can't wait to read far more from you. This is actually a wonderful website.
Some truly fine goodies on this web site , appreciate it for contribution.
Howdy just wanted to give you a quick heads up. The words in your article seem to be running off the screen in Opera.
I'm not sure if this is a formatting issue or something to do with browser compatibility but
I figured I'd post to let you know. The style and design look great though!
Hope you get the issue fixed soon. Thanks
You really make it seem really easy together with your presentation however I to find this topic to
be really one thing which I believe I'd never understand.
It sort of feels too complex and extremely vast for
me. I am looking ahead in your next post, I'll try to get the grasp of it!
Pretty! This was a really wonderful article. Thank you for
supplying these details.
This site was… how do you say it? Relevant!!
Finally I have found something which helped me.
Cheers! natalielise plenty of fish
Appreciate it for this howling post, I am glad I observed this internet site on yahoo.
Hello superb website! Does running a blog similar to this require a
massive amount work? I have virtually no expertise in programming but I
had been hoping to start my own blog soon. Anyway,
should you have any ideas or techniques for new blog owners please share.
I understand this is off subject however I just wanted to ask.
Thank you!
I am 43 years old and a mother this helped me!
I am 43 years old and a mother this helped me!
I am 43 years old and a mother this helped me!
I am 43 years old and a mother this helped me!
I am 43 years old and a mother this helped me!
I am 43 years old and a mother this helped me!
Hi there, i read your blog occasionally and i own a similar one and i was just wondering if you get a lot of
spam responses? If so how do you protect against it, any plugin or anything you can suggest?
I get so much lately it's driving me mad so any support is very much appreciated.
Hi there! Do you use Twitter? I'd like to follow you if that would be
okay. I'm definitely enjoying your blog and look forward
to new posts.
Respect to website author , some wonderful entropy.
You made some decent factors there. I appeared on the internet for the issue and found most people will associate with with your website.
Hi there! This article could not be written much better! Looking
at this article reminds me of my previous roommate!
He always kept talking about this. I'll forward this article to him.
Pretty sure he'll have a great read. Thanks for sharing!
What i do not understood is in fact how you are not actually much more well-favored than you might be
now. You are very intelligent. You know thus considerably relating to this subject, made me individually believe it from
so many numerous angles. Its like men and women don't seem to be interested unless it's something to accomplish with Lady gaga!
Your own stuffs nice. Always deal with it up!
I think the admin of this web site is truly working hard in favor of his website,
since here every data is quality based information.
It's very simple to find out any matter on net as compared to books,
as I found this piece of writing at this site.
I'm not sure why but this site is loading extremely slow for me.
Is anyone else having this problem or is it a issue on my end?
I'll check back later and see if the problem still exists.
I like this site, useful stuff on here : D.
Greate pieces. Keep writing such kind of info on your page.
Im really impressed by your blog.
Hi there, You have performed an incredible job.
I will definitely digg it and in my view suggest to my friends.
I'm confident they will be benefited from this website.
Hi to every one, the contents existing at this website are truly amazing for people experience, well,
keep up the nice work fellows.
Greate pieces. Keep posting such kind of info on your site.
Im really impressed by your site.
Hello there, You have performed a fantastic job. I will definitely digg it and personally suggest to my friends.
I'm confident they'll be benefited from this website.
pof natalielise
I was curious if you ever thought of changing the page layout of your site?
Its very well written; I love what youve got to say. But maybe you could a
little more in the way of content so people could connect with it
better. Youve got an awful lot of text for only having
one or two images. Maybe you could space it out better?
I was recommended this website by means of my cousin. I am
no longer sure whether this publish is written via him as no
one else know such targeted about my difficulty.
You're wonderful! Thanks!
I like this website its a master peace ! Glad I found this on google .
Hello! Someone in my Facebook group shared this site with us so I came to check it out.
I'm definitely enjoying the information. I'm bookmarking and will be tweeting this to my followers!
Wonderful blog and fantastic design and style.
Good day! Do you use Twitter? I'd like to follow you if that would be okay.
I'm definitely enjoying your blog and look forward
to new updates.
Hi there, I enjoy reading all of your article. I wanted to write a little comment to support you.
Turkishgram.com, hesaplarını büyütmek ve ileri taşımak isteyen bütün Instagram kullanıcılarına en uygun fiyat garantisiyle Instagram takipçi al hizmetini sunmaktadır.
Much thanks for your letter which arrived a couple of days back. It was flawless to get notification from you.
As you probably are aware, my piano player is Yiruma and in last show I could hardly imagine how he showed up in my eyes and gave me a major embrace after his exhibition.
Just when I heard his tune from the stage I feel anythings around me appear to vanish and I can fly with numerous stars on the sky to overlook every one of my feelings of trepidation which I endured.
How about we return to my live show. It is so cool and as of recently I can't overlook my inclination about it. Do you regularly to go live shows? What sort of music do you like tuning in to? Who is your preferred artist?