Getting the call stack without a frame pointer

Everything I know about getting the current call stack of C or C++ programs, including ones compiled with -fomit-frame-pointer or an equivalent, with or without a debugger. Hardly entertaining except for those specifically wishing to lay their hands on call stacks.

We'll start with trivia people sharing my unhealthy interest in call stacks probably know. There are two contexts where you might want to get a call stack:

  1. Look at the call stack in a debugger to see what the program is doing.
  2. Get a representation of the call stack inside the program itself. For example, a memory profiler might want to attach the call stack identifying the context where the allocation is made to each memory block to see who allocates the most.

Sometimes (2) can be implemented using (1) by running a debugger programmatically and asking it for the current call stack, and sometimes it can't (too much communication overhead, or no debugger available – for example, a program stripped of debugging information and running on a standalone embedded board).

The straightforward way of getting the current call stack, used both by debuggers and by programs curious about their own stacks, relies on the frame pointer. The idea is that a machine register is reserved for keeping a pointer into the stack, called the frame pointer, and every function is compiled to do the following in its prologue:

  • Push the return address to the stack
  • Push the frame pointer to the stack
  • Save the address of the resulting two-pointer structure to the frame pointer register

This creates a linked list on the stack, with every node keeping a return address – this list is the call stack (and this is why debuggers show the points of return from function calls and not the points of call, a bit annoyingly for function calls spanning many source code lines – in fact we get the return stack and not the call stack). Here's how you get this list from within a program:

struct stack_frame {
  struct stack_frame* next;
  void* ret;
};
int get_call_stack(void** retaddrs, int max_size) {
  /* x86/gcc-specific: this tells gcc that the fp
     variable should be an alias to the %ebp register
     which keeps the frame pointer */
  register struct stack_frame* fp asm("ebp");
  /* the rest just walks through the linked list */
  struct stack_frame* frame = fp;
  int i = 0;
  while(frame) {
    if(i < max_size) {
      retaddrs[i++] = frame->ret;
    }
    frame = frame->next;
  }
  return i;
}

The code for getting the list head pointer depends on the platform, the list structure itself is common to many machines and compilers. The return addresses may be converted to function names and source line numbers with addr2line -f or similar. When the program can't access its own debug info during its execution as in the case of embedded devices, the translation from addresses to names will be a separate offline step.

The whole frame pointer business is fairly widely documented, with a bunch of source code available centered around getting the call stack that way. I think the GNU backtrace function and the Windows StackWalk64 function, which these days are probably a better alternative than code snippets like the one above, also use this linked list when available.

Now, what happens if the compiler is told to avoid generating the code maintaining the frame pointer (-fomit-frame-pointer for gcc, /Oy for MSVC), or not told to override its default behavior of not generating such code (-ga for Green Hills C++)?

Admittedly it doesn't appear to be such a good idea to make debugging that much harder in order to save a few instructions. However, there are reasons to do so. For example, one common consequence of Software Design is lots of functions doing little or nothing except for delegating their work to another function. Without frame pointer maintenance, such a call is just a jump instruction – your callee function will return directly to the address saved by your caller. With frame pointers, you need a whole prologue and epilogue here. Anyway, we won't discuss the benefits of frame pointer omission since measuring the overhead for your particular code will be more reliable than such a discussion anyway.

Compiling without frame pointers hits code trying to obtain its own context harder than it hits debuggers, because debuggers don't really need frame pointers and only (sometimes) rely on them for simplicity of implementation. Given a return address, a debugger can tell:

  1. Which function it belongs to (unlike the program itself, a debugger is necessarily supposed to have access to the symbol table)
  2. Where that function keeps the return address (the compiler knows that, so it can tell the debugger)
  3. The amount by which the function decrements the stack pointer, assuming the stack grows downwards – again something the compiler knows. Now that the debugger knows the previous return address and the previous stack pointer, it can go back to step 1.

So a debugger can do just fine without frame pointers as long as the compiler gives it enough information about the layout of the stack. I've been debugging without frame pointers for a long time with the Green Hills MULTI debugger which uses a proprietary debug info format. More recently the DWARF format, gcc and gdb seem to have caught up and now programs compiled with -fasynchronous-unwind-tables -fomit-frame-pointer are debuggable with gdb. The information generated by -fasynchronous-unwind-tables seems to go to a separate ELF section called .eh_frame_hdr.

Not only will gdb use .eh_frame_hdr, but the GNU backtrace function appears to be using it as well (it doesn't work under -fomit-frame-pointer but apparently does work when you add -fasynchronous-unwind-tables – although the docs explicitly say: "frame pointer elimination will stop backtrace from interpreting the stack contents correctly"). Nor is this section stripped from the program – it's not implemented as a "normal" debug information section but as an allocated data section, so it's always available to a program (in particular, to the backtrace function).

So under gcc, all call stack problems seem to be solved – unless you trust the docs (!?), or unless some code isn't compiled with the right flags because of not being up to date or someone being too greedy to allocate space for a debug info section. Outside gcc, or more precisely DWARF, I don't think a stripped program can access such debug info.

Is there a way to get a call stack without a frame pointer, without a debugger and without debug info?

For years I was sure that the answer was "no", hence some things will only work under a separate build mode – just like the release build but with frame pointers. Then one time the Green Hills debugger failed to list the call stack for some reason as it sometimes does, but this time we really wanted to decipher it. And we figured that we can in fact do the same thing the debugger does, except we'll understand from the assembly code what the debugger usually understood from debug information.

Specifically, to understand where the return address is kept and by what amount the stack pointer is decremented, you need to find the instructions doing (or undoing) these things in the prologue (or the epilogue) of the function. This worked, but due to either inertia or stupidity it took me months to realize that you can write code doing this. Anyway, here's how it works on a 32b MIPS processor under the Green Hills compiler. The prologue code of a function will contain instructions like these:

main+0: 27bdffe8 addiu sp, sp, -0x18
main+4: afbf0014 sw    r31, 0x14(sp)

The add immediate instruction decrements the stack pointer, and the store word instruction saves the return address from the register where it's saved by the caller to some place on the stack. The high 16 bits of these instructions don't depend on the function, encoding the "addui sp, sp" and the "sw, r31 …(sp)" parts. The low 16 bits encode a signed offset. So we can obtain the call stack from our code disassembling it thusly:

/* get previous stack pointer and return address
   given the current ones */
int get_prev_sp_ra(void** prev_sp, void** prev_ra,
                   void* sp, void* ra) {
  unsigned* wra = (unsigned*)ra;
  int spofft;
  /* scan towards the beginning of the function -
     addui sp,sp,spofft should be the first command */
  while((*wra >> 16) != 0x27bd) {
    /* test for "scanned too much" elided */
    wra--;
  }
  spofft = ((int)*wra << 16) >> 16; /* sign-extend */
  *prev_sp = (char*)sp - spofft;
  /* now scan forward for sw r31,raofft(sp) */
  while(wra < (unsigned*)ra) {
    if((*wra >> 16) == 0xafbf) {
      int raofft = ((int)*wra << 16) >> 16; /* sign */
      *prev_ra = *(void**)((char*)sp + raofft);
      return 1;
    }
    wra++;
  }
  return 0; /* failed to find where ra is saved */
}

The call stack will then be produced by the following loop:

int get_call_stack_no_fp(void** retaddrs, int max_size) {
  void* sp = get_sp(); /* stack pointer register */
  void* ra = get_ra(); /* return address register */
  /* adjust sp by the offset by which this function
     has just decremented it */
  int* funcbase = (int*)(int)&get_call_stack_no_fp;
  /* funcbase points to an addiu sp,sp,spofft command */
  int spofft = (*funcbase << 16) >> 16; /* 16 LSBs */
  int i=0;
  sp = (char*)sp-spofft;
  do {
    if(i < max_size) {
      retaddrs[i++] = ra;
    }
  }
  while(get_prev_sp_ra(&sp,&ra,sp,ra));
  return i; /* stack size */
}

get_sp and get_ra access registers so they must be assembly macros, which in the case of Green Hills can be spelled like this:

asm void* get_ra() {
  move $v0, $ra
}
asm void* get_sp() {
  move $v0, $sp
}

Under MIPS32 and Green Hills, this code seems to be giving decent call stacks except for the inevitable omission of function calls done without saving the return address; the most common case of those – simple delegating functions – was already mentioned above. If f calls g which does (almost) nothing except for calling h, and g doesn't bother to save the return address to the stack, having h return directly to f, then the call stack will contain f and h but not g. Not much of a problem and sometimes even an advantage as far as I'm concerned, since g is rarely interesting. Also, you can get this problem irregardless of the lack of a frame pointer – for example, gcc -O3 on x86 will not maintain the call stack accurately even without -fomit-frame-pointer, generating the following ridiculous code:

pushl   %ebp
movl    %esp, %ebp
popl    %ebp
; so what was the point of setting up and
; immediately destroying a stack frame?
jmp     print_trace

Now, although code like get_prev_sp_ra looking for 0x27bd admittedly is a heuristic relying on undocumented platform-specific behavior, it looks like a passable way of getting a call stack on a RISC machine like MIPS, ARM or PowerPC. What about the x86 though? Effectively we have our code partially disassembling itself here, which is not nearly as easy with the x86 (in particular, I don't think there's a way to scan backwards because of the variable encoding length; although we could look at epilogues instead of prologues just as well).

Instead of dragging in a disassembler, we can use an external program, such as, well, a debugger. This obviously defeats the purpose of being able to get the call stack without a debugger. But this purpose isn't very interesting on the x86 in the first place because there you're rarely stuck in situations where a program can't run a debugger.

The only point of the disassembling business on the x86 thus remains to deal with programs compiled without a frame pointer and without debug information making it possible to get the call stack nonetheless. I don't know if anybody has such a problem these days, now that gcc has -fasynchronous-unwind-tables – perhaps someone uses compilers which can't do this or binaries compiled without this, and perhaps the problem is extinct on the x86. For what it's worth, here's a script getting the call stack from a core file without relying on gdb's bt command but relying on its disassemble command. Usage: python bt <program> <core>. No warranty, …or FITNESS FOR A PARTICULAR PURPOSE.

And this is all I know about getting the call stack in C or C++, something users of other languages can do, in the (unlikely) absence of a library function doing just that, simply by throwing an exception, immediately catching it and using its getStackTrace method or some such.

81 comments ↓

#1 Wayne on 10.12.09 at 1:05 pm

Just a note that x86_64 is more interesting than x86 as the ABI allows them to omit the frame pointer from code and requires a special DWARF section (perhaps the same one you mention.) to allow the debugger to unwind code.

That means your first loop that walks ebp (actually rbp) won't be reliable on x86_64 even if -fomit-frame-pointer is not passed to the compiler.

#2 Alex on 10.20.09 at 9:23 am

Well, just use Google Breakpad for this stuff.

#3 Dan on 10.22.09 at 7:22 pm

The last place I worked, we were using gcc/gdb on MIPS and wrote functions to do this, too. They looked very similar to what you've posted here, but were a little more complicated because we found that some functions made more than one adjustment to the stack frame.

Anyway, we had to create our own mini coredumps for this embedded system because the OS (Nucleus) didn't do it for us. These stack back-tracing functions were the only tool we had for generating a stack trace for the core dumps. It worked surprisingly well. I don't recall ever getting a core dump that didn't include a valid stack trace (save the rare ones that had severe stack corruption… those were usually bad days for me).

#4 Yossi Kreinin on 10.23.09 at 4:08 am

@Dan: mini core dumps as in subsets of the program & chip state as opposed to complete snapshots? Painful stuff, I hate to decide what goes in and what stays out – it's a good thing to insist that an embedded system is always hooked to something capable of getting a full core dump during development time.

The thing I hate the most about stack corruption is that if the return address is smashed and you jump to nowhere, you don't know where you jumped from – at least MIPS is careful to first set $pc to $ra, then produce an exception at that address upon jr $ra into the chasm.

#5 Kragen Javier Sitaker on 11.08.09 at 10:00 am

Thanks, Yossi! I haven't laughed out loud at a segment of assembly code in a while.

It would be convenient for debugging (in the stack corruption case) if ret exchanged $pc and $ra instead of just copying one to the other. It's not like you're going to return to that same address again, are you?

#6 Yossi Kreinin on 11.08.09 at 11:02 am

I don't think x86, which has a ret instruction, has a return address register – ret pops an address from the stack symmetrically to call which pushes one onto the stack, isn't it so?

On MIPS and AFAIK most RISC machines, there is no special ret instruction – there's jr (jump register) or similar; it's the same instruction used for virtual function calls so I guess clobbering the register in the general case would be wrong as it would suck through a loop calling a virtual function, for example, and special-casing $ra under the assumption that it's really used for a return address isn't The RISC Way.

Which of course does not prevent them from stuffing $pc where the last executed jr instruction resided to some kind of register – I'd happily settle for another addition to the zillion user-inaccessible coprocessor registers already available. The reason they don't do it IMO is that Unix/C, and consequently RISC which is the Unix/C back-end of choice, couldn't care less about debugging – they're proud when they correctly record the point where an exception occurred so you can at least meaningfully restart from there.

#7 Kragen Javier Sitaker on 11.11.09 at 3:10 am

You're right about x86, and your MIPS comment is also correct for SPARC, ARM, and HP-PA, but not PowerPC, which has a special "link register" LR with magic instructions to access it, blech. But since it's generally used for either function return or virtual function calls, and I doubt the ABI makes it "callee"-saved in either of those cases, changing it from a mov to a swap wouldn't pose much risc to efficiency. It would clearly break programs, though.

(Actually the only case I can think of where it would matter would be when you're implementing direct-threaded Forth…)

Another extra register would be a good idea, but of course to work under an OS, the OS would have to know to save the value on every context-switch, just in case you might need to debug the program. If we can have that, can we also have a ring buffer of the last 64 jumps the program executed? Because that would be really handy too…

#8 Dan on 11.23.09 at 12:09 pm

@Yossi: Mini coredumps are sometimes the only way to get them from customer equipment, when there is limited flash space (not enough to save a real core file). It also happens that on these systems, the only reliable way to get data out of the box in the face of a core dump was through a 38400kbps serial port. It would have taken up to 31 hours to dump core on one of these, if I did my math right (memory maxed at 512MB, all one monolithic image, single address space, yaddah yaddah). We got *really* creative at coming up with ways to include the most amount of useful information in the smallest amount of data.

#9 Yossi Kreinin on 11.23.09 at 11:02 pm

@Dan: sure, been there. Just saying that sometimes beating up a PM can bring in more bandwidth and end up being the cheaper solution. Sometimes.

#10 GruntledEmbeddedDesigner on 02.27.12 at 6:06 pm

Hi, I really like your article. I'm using it to get some stack dumps in the development phase of our product. I have found that sometimes the return address register in the MIPS 24K is not 32bit aligned. And this causes your example code to throw a CPU exception when dereferencing a non aligned pointer. Just a heads up.
thanks for the effort to document this stuff. Its extremely useful.

#11 Yossi Kreinin on 02.27.12 at 10:27 pm

Seriously? I'd think the 24K couldn't return (jr $ra) to a misaligned address. Is it a real return address or something located where the code expects a return address that actually isn't?

#12 GruntledEmbeddedDesigner on 02.28.12 at 5:45 pm

Seriously looks like that anyway. The stack dump example starts to break down when compiler optimization level is turned up AND the code is compiled with the MIPS16 option.

I'm getting return addresses like the following: 0x9e032d1f

And these optimized functions don't do a
addiu sp,sp,spofft

So now i'm trying to figure out how to get the call stack for functions being jumped to where the calling function uses the jalx instruction.

I'm probably in over my head but this is for extra credit in my down time anyway.

#13 Yossi Kreinin on 02.28.12 at 11:56 pm

MIPS16 I don't know anything about, although a 0xf sounds too misaligned even for that; anyway, unfortunately it's not surprising that my code above can't handle your case…

#14 GruntledEmbeddedDesigner on 02.29.12 at 10:45 am

yes, i agree 0xF sounds like a very strange offset??? but taking a look at the .map file, all the base address of the functions compiled under MIPS16 are on byte aligned addresses.

#15 Protozorq on 06.27.12 at 7:13 am

Nice article!

I was hoping to find a method to receive the frame pointer in a function when omit-frame-pointer is enabled. I've found out, that GCC provides a builtin function to receive the frame pointer:

void * __builtin_frame_address (unsigned int level);

Unfortunately, usage of that function seems to turn of omit-frame-pointer optimization for the given function.

I don't even know, since when this builtin function exists .. so, it might not be a very portable method.

#16 Yossi Kreinin on 06.27.12 at 7:24 am

I'd guess that if the callers' code is compiled with frame pointer omission, then you can't get the callstack even if you (the callee) do set up the frame pointer and have an intrinsic to get that address.

#17 Aaron on 09.26.13 at 6:50 am

This is pure gold! I am working on MIPS as well and I had often found myself wondering about this same topic. Thank you so much.

#18 Yossi Kreinin on 09.26.13 at 7:35 am

Thanks, and I hope it actually works for you…

#19 Mars on 10.17.13 at 4:52 pm

Thanks, it does help!
But in my MIPS compiler this code will retrieve wrong spofft in the line below.
/* funcbase points to an addiu sp,sp,spofft command */
int spofft = (*funcbase <> 16; /* 16 LSBs */

It's because the first few asm lines deal with $gp first like below:
00400b04 :
400b04: 3c1c0fc0 lui gp,0xfc0
400b08: 279c754c addiu gp,gp,30028
400b0c: 0399e021 addu gp,gp,t9
400b10: 27bdffe0 addiu sp,sp,-32

so I modified your code a little to work around it:
wra = (unsigned*)funcbase;
/* scan from the beginning of the function -
addui sp,sp,spofft may not be the first command */
while((*wra >> 16) != 0x27bd) {
/* test for "scanned too much" elided */
wra++;
}
spofft = ((int)*wra <> 16; /* sign-extend */

Just to share this.

#20 Yossi Kreinin on 10.19.13 at 2:44 am

Interesting; does this $gp fiddling happen at every function? Something's off; $gp is usually initialized to a constant (specifically the middle of the sda, "small data area") and then left alone.

#21 Mars on 10.21.13 at 3:48 pm

Yes, it's in every function. I don't why this happened. I'm using uclibc, and the compile flags is "mipsel-linux-gcc -g -O -mtune=4kc -mips32 "

#22 Cat Bloog DaBea on 04.01.19 at 8:32 pm

Right now it sounds like Expression Engine is the preferred blogging platform out there right now.

(from what I've read) Is that what you're using
on your blog?

#23 skin care on 04.20.19 at 5:35 pm

Keep up the good work! Thanks.

#24 accounting freelancers on 05.03.19 at 3:45 am

In fact when someone doesn't know after that its up to other people that
they will assist, so here it takes place.

#25 Blonde Milf on 05.09.19 at 3:09 am

Yes! Finally someone writes about Blonde Milf.

#26 gel nails on 05.10.19 at 1:57 pm

These are actually impressive ideas in on the topic of blogging.
You have touched some good factors here. Any way keep up wrinting.

#27 Big Boobs on 05.11.19 at 4:34 pm

Hello friends, how is all, and what you want to say regarding this paragraph, in my view its genuinely
amazing in favor of me.

#28 newz aimbot on 05.15.19 at 5:07 pm

Good, this is what I was searching for in bing

#29 Trish Dornbusch on 05.17.19 at 11:20 am

Congrats for the inspiring site you've created at yosefk.com. Your enthusiastic take on the subject is definitely inspiring. Thanks again!

#30 Fuck Girls Online on 05.18.19 at 4:24 am

I have learn some good stuff here. Certainly price bookmarking for revisiting.
I surprise how much effort you place to create this sort of magnificent informative site.

#31 how to get help in windows 10 on 05.20.19 at 3:43 pm

My partner and I absolutely love your blog and find nearly all of your post's to be just
what I'm looking for. Would you offer guest writers to write content for you?
I wouldn't mind creating a post or elaborating on some of the subjects
you write regarding here. Again, awesome weblog!

#32 Nude Females on 05.23.19 at 5:25 am

Hmm it looks like your blog ate my first comment (it was super long) so I guess
I'll just sum it up what I submitted and say, I'm thoroughly enjoying your blog.
I as well am an aspiring blog writer but I'm still new to the whole thing.
Do you have any points for beginner blog writers? I'd certainly appreciate
it.

#33 gamefly free trial on 05.23.19 at 8:56 am

If some one wants to be updated with most recent
technologies then he must be go to see this web page and be up to date all the time.

#34 website on 05.23.19 at 3:23 pm

Heya great blog! Does running a blog similar to this require a great deal
of work? I've very little expertise in computer programming however I was hoping to start
my own blog soon. Anyway, if you have any
ideas or techniques for new blog owners please share. I know this is off subject but I just needed to ask.
Kudos!

#35 gamefly free trial on 05.24.19 at 7:12 am

Undeniably imagine that which you said. Your favorite reason seemed to be on the web
the easiest factor to consider of. I say to you, I definitely get annoyed whilst folks consider issues that they plainly don't
understand about. You controlled to hit the nail upon the highest and also defined out the
entire thing with no need side effect , other people can take
a signal. Will likely be again to get more. Thank you

#36 how to get help in windows 10 on 05.24.19 at 9:42 pm

Great web site. A lot of useful info here.
I am sending it to some pals ans additionally sharing in delicious.

And obviously, thank you on your sweat!

#37 Sexy Ass Cams on 05.26.19 at 7:02 pm

Great post.

#38 how to get help in windows 10 on 05.27.19 at 8:58 pm

Great website. Lots of helpful information here.

I'm sending it to some buddies ans also sharing in delicious.
And certainly, thank you in your effort!

#39 gamefly free trial on 05.27.19 at 9:02 pm

I blog often and I truly appreciate your information. This article
has really peaked my interest. I am going to take a note of your
site and keep checking for new details about once a week.
I opted in for your RSS feed as well.

#40 how to get help in windows 10 on 05.27.19 at 9:05 pm

I'm gone to convey my little brother, that he should also go to see this website on regular basis to get
updated from hottest news.

#41 gamefly free trial on 05.27.19 at 9:09 pm

Hello! This post could not be written any better! Reading through
this post reminds me of my previous room mate! He always
kept talking about this. I will forward this page to him.
Fairly certain he will have a good read. Many thanks
for sharing!

#42 how to get help in windows 10 on 05.27.19 at 9:16 pm

I loved as much as you'll receive carried out right here.
The sketch is tasteful, your authored subject matter stylish.
nonetheless, you command get bought an edginess over that
you wish be delivering the following. unwell unquestionably come
further formerly again since exactly the same nearly very often inside case you shield this hike.

#43 gamefly free trial on 05.27.19 at 9:18 pm

I'm not sure why but this web site is loading incredibly slow for me.
Is anyone else having this problem or is it a issue
on my end? I'll check back later on and see if the problem still
exists.

#44 jordan 6 on 05.30.19 at 1:33 am

Can I just say what a reduction to seek out somebody who actually is aware of what theyre speaking about on the internet. You definitely know the best way to bring a difficulty to mild and make it important. More folks must read this and understand this facet of the story. I cant consider youre no more in style because you positively have the gift.

#45 instagram takipi satn al on 05.30.19 at 12:27 pm

Hello! I know this is sort of off-topic but I needed
to ask. Does managing a well-established website like yours take a massive amount work?
I'm completely new to writing a blog however I do
write in my journal every day. I'd like to start a blog so I can share my personal experience and thoughts online.
Please let me know if you have any ideas or tips for
brand new aspiring bloggers. Thankyou!

#46 website marketing on 05.30.19 at 11:35 pm

Keep up the good work! Thanks.

#47 air max 270 on 05.31.19 at 5:59 am

Nice post. I study one thing more challenging on completely different blogs everyday. It is going to at all times be stimulating to learn content material from other writers and follow a little something from their store. I抎 desire to make use of some with the content material on my weblog whether you don抰 mind. Natually I抣l offer you a link on your web blog. Thanks for sharing.

#48 jordan 6 on 06.01.19 at 11:19 am

There is noticeably a bundle to know about this. I assume you made certain good factors in features also.

#49 air max on 06.02.19 at 7:56 pm

I抦 impressed, I need to say. Really rarely do I encounter a blog that抯 both educative and entertaining, and let me let you know, you've got hit the nail on the head. Your idea is outstanding; the difficulty is one thing that not sufficient individuals are speaking intelligently about. I am very completely satisfied that I stumbled throughout this in my seek for something relating to this.

#50 Samual Kasik on 06.04.19 at 8:05 pm

6/4/2019 yosefk.com does it yet again! Quite a interesting site and a well-written post. Nice work!

#51 Ouida Martinas on 06.08.19 at 12:25 am

I completely agree with your take on this subject and look forward to new posts and comments here at yosefk.com. Thanks!

#52 playstation 4 games 2019 on 06.08.19 at 2:53 pm

We're a group of volunteers and starting a new scheme in our community.

Your website provided us with helpful information to work
on. You have done an impressive process and our whole group can be grateful to you.

#53 son dakika yalova haberleri on 06.08.19 at 7:04 pm

Hands down, Apple's app store wins by a mile. It's a huge selection of all sorts of apps vs a rather sad selection of a handful for Zune. Microsoft has plans, especially in the realm of games, but I'm not sure I'd want to bet on the future if this aspect is important to you. The iPod is a much better choice in that case.

#54 yalova son dakika haberleri on 06.09.19 at 4:49 am

Hands down, Apple's app store wins by a mile. It's a huge selection of all sorts of apps vs a rather sad selection of a handful for Zune. Microsoft has plans, especially in the realm of games, but I'm not sure I'd want to bet on the future if this aspect is important to you. The iPod is a much better choice in that case.

#55 son dakika eskişehir haberleri on 06.10.19 at 10:00 am

Zune and iPod: Most people compare the Zune to the Touch, but after seeing how slim and surprisingly small and light it is, I consider it to be a rather unique hybrid that combines qualities of both the Touch and the Nano. It's very colorful and lovely OLED screen is slightly smaller than the touch screen, but the player itself feels quite a bit smaller and lighter. It weighs about 2/3 as much, and is noticeably smaller in width and height, while being just a hair thicker.

#56 son dakika elazığ haberleri on 06.10.19 at 11:07 am

Between me and my husband we've owned more MP3 players over the years than I can count, including Sansas, iRivers, iPods (classic & touch), the Ibiza Rhapsody, etc. But, the last few years I've settled down to one line of players. Why? Because I was happy to discover how well-designed and fun to use the underappreciated (and widely mocked) Zunes are.

#57 edirne son dakika haberleri on 06.10.19 at 11:08 am

The Zune concentrates on being a Portable Media Player. Not a web browser. Not a game machine. Maybe in the future it'll do even better in those areas, but for now it's a fantastic way to organize and listen to your music and videos, and is without peer in that regard. The iPod's strengths are its web browsing and apps. If those sound more compelling, perhaps it is your best choice.

#58 son dakika düzce haberleri on 06.10.19 at 11:54 am

Between me and my husband we've owned more MP3 players over the years than I can count, including Sansas, iRivers, iPods (classic & touch), the Ibiza Rhapsody, etc. But, the last few years I've settled down to one line of players. Why? Because I was happy to discover how well-designed and fun to use the underappreciated (and widely mocked) Zunes are.

#59 eski kadınların zayıflama sırları on 06.10.19 at 11:56 am

Apple now has Rhapsody as an app, which is a great start, but it is currently hampered by the inability to store locally on your iPod, and has a dismal 64kbps bit rate. If this changes, then it will somewhat negate this advantage for the Zune, but the 10 songs per month will still be a big plus in Zune Pass' favor.

#60 Jody Vonhagen on 06.11.19 at 2:02 am

6/10/2019 @ 7:02:57 PM Like the site– very user-friendly and a lot of stuff to think about!

#61 playstation 4 best games ever made 2019 on 06.12.19 at 6:01 am

I have read some excellent stuff here. Definitely worth
bookmarking for revisiting. I wonder how much attempt you set to make this kind of fantastic informative web site.

#62 ps4 best games ever made 2019 on 06.12.19 at 9:51 am

Hey there! This post could not be written any better! Reading this post reminds me of my good old room mate!
He always kept chatting about this. I will forward this post to him.
Pretty sure he will have a good read. Thank you for sharing!

#63 playstation 4 best games ever made 2019 on 06.12.19 at 10:07 pm

Spot on with this write-up, I absolutely feel this amazing site needs a great deal more
attention. I'll probably be back again to
read more, thanks for the info!

#64 Meghann Hurless on 06.13.19 at 9:12 pm

I'm pleased by the manner in which yosefk.com handles this sort of subject. Generally to the point, sometimes polemic, always well-researched and more often than not quite stimulating.

#65 quest bars cheap on 06.14.19 at 2:16 pm

Really when someone doesn't know afterward its up to other visitors that they
will assist, so here it happens.

#66 quest bars cheap on 06.14.19 at 4:09 pm

I pay a quick visit every day a few blogs and
sites to read posts, except this website gives quality based content.

#67 esfords on 06.15.19 at 2:58 am

How long does a copyright last on newspaper articles?. . If a service copies newspapers articles and then posts it in a database on the Internet, is there also a copyright on the Internet content?.

#68 quest bars cheap on 06.15.19 at 6:05 am

Currently it appears like Movable Type is the preferred blogging platform out there right now.

(from what I've read) Is that what you are using on your blog?

#69 quest bars on 06.16.19 at 7:56 pm

This site was… how do I say it? Relevant!! Finally I have found something
that helped me. Kudos!

#70 aimbot download fortnite on 06.17.19 at 3:46 am

This does interest me

#71 tinyurl.com on 06.17.19 at 1:17 pm

I'm really enjoying the design and layout of your site.
It's a very easy on the eyes which makes it much more pleasant
for me to come here and visit more often. Did you hire out a designer to create your theme?
Superb work!

#72 sexy on 06.18.19 at 10:45 pm

How long does a copyright last on newspaper articles?. . If a service copies newspapers articles and then posts it in a database on the Internet, is there also a copyright on the Internet content?.

#73 proxo key on 06.19.19 at 10:53 am

I consider something really special in this site.

#74 vn hax download on 06.20.19 at 7:37 pm

Ha, here from google, this is what i was searching for.

#75 nonsense diamond key generator on 06.21.19 at 8:47 am

I like this article, some useful stuff on here : D.

#76 plenty of fish dating site on 06.22.19 at 8:25 am

Informative article, totally what I wanted
to find.

#77 quest bars cheap on 06.23.19 at 7:58 am

When someone writes an piece of writing he/she maintains the
image of a user in his/her brain that how a user can know it.
So that's why this paragraph is perfect. Thanks!

#78 quest bars cheap on 06.23.19 at 11:15 am

each time i used to read smaller articles which also clear
their motive, and that is also happening with this paragraph which I am reading here.

#79 star valor cheats on 06.23.19 at 6:12 pm

Great stuff to check out, glad that duckduck took me here, Keep Up good job

#80 Explain Like I’m Five on 06.25.19 at 6:10 am

Yeah bookmaking this wasn’t a risky decision outstanding post! .

#81 geometry dash 2.11 download pc on 06.25.19 at 8:55 pm

This does interest me

Leave a Comment