In case you haven't been around Linux a long time, back in the dark ages Linux used something other than ELF for the format of its executables. The old format was called a.out (or COFF as I recall). If you run the file command on a modern executable, it will say something about ELF. If you run file on an old a.out executable it will say something about xMAGIC (ZMAGIC or QMAGIC are the ones I remember). It had been some number of years since I'd seen an a.out executable, but recently I had to figure out how to run some on a Red Hat 7.1 system.
The first error message I ran into was "cannot execute binary file".
Turns out that Red Hat compiles kernel support for a.out executables as a
module. Run modprobe binfmt_aout and you should get a
different error message. You can make this permanent one of two ways.
You can either add the modprobe command to /etc/rc.d/rc.local or you can
add the following line to /etc/modules.conf:
alias binfmt-0064 binfmt_aout
This is based on a recommendation in the Kerneld mini-HOWTO. They actually list three different lines for various types of a.out binaries, but I found that on RedHat 7.1 none of them worked and the messages that showed up in syslog led me to the 0064.
The next error message refers to the failure to find the runtime linker at /lib/ld.so. That is the path to the old a.out linker. The last time Red Hat seems to have shipped with an RPM containing it is version 6.2. Grab the RPM from ftp://ftp.redhat.com/pub/redhat/linux/6.2/en/os/i386/RedHat/RPMS/ld.so-1.9.5-13.i386.rpm and install it.
The last error message I ran into was the lack of libc.so.4. You'll need a.out versions of a number of libraries. An RPM containing the common ones can be found in Red Hat 5.2. Grab the RPM from ftp://ftp.redhat.com/pub/redhat/linux/5.2/en/os/i386/RedHat/RPMS/aout-libs-1.4-9.i386.rpm and install it.
At this point your a.out executables should run. Good luck.