Re: stackguard-like embedded protection

[Crispin Cowan <crispin () WIREX COM>]

antirez wrote:

> On Wed, Sep 13, 2000 at 08:20:41PM +0900, Hiroaki Etoh wrote:
> > I tested GCC aix, xlC aix, and Microsoft Visual C++. They don't place the
> > 'volatile' variable next to the RET.  Could someone test the following program
> > in the other compilers. I am very interested in such information.
>
> Can you describe how GCC aix behaves?
> Maybe we can distinguish classes of compilers.

I'm not sure about AIX/GCC, but IBM's native XLC compiler has a VERY strange stack
ordering behavior:

   * stacks grow down, as usual
   * but variables are allocated backwards, i.e. the first declared auto variable
     in a function gets the lowest address in that stack frame

Ok, that's confusing.  Here's an illustration:

foo() {
    int    a;
    int    b;
    int    c;

    bar();
}

bar() {
    int    d;
    int    e;
    int    f;

    ...
}

Run this code, and you get a stack that looks like this:

   * high memory
   * foo()'s activation record
   * c
   * b
   * a
   * bar()'s activation record
   * f
   * e
   * d

I discovered this the hard way seven years ago when trying to create a portable
checkpoint and rollback facility for my dissertation (
http://www.cse.ogi.edu/~crispin/hope.html ).  I was using some nice code by DJ
Bernstien that does precisely that.  Bernstein's code detects whether stacks grow
up or down by allocating two auto variables and then comparing their addresses.
XLC fooled Bernstein by doing this convoluted grow-down-but-invert-each-function
stack ordering.

I no longer have access to AIX machines, so I can't check current status.

Crispin

--
Crispin Cowan, Ph.D.
Chief Research Scientist, WireX Communications, Inc. http://wirex.com
Free Hardened Linux Distribution:                    http://immunix.org
                Olympics:  The Corruption Games