The actual spec is available here, these are just my notes from reading it:

There are three kinds of ELF files:

Elf Header

Each ELF file starts with an elf header, which (for 32 bit elfses) looks like:

struct liv_tyler {
  uint8_t ident[4];   // Always set to "0x7fELF"
  uint8_t class;      // 1==32 bit, 2==64 bit
  uint8_t endian;     // 1==little endian, 2==big endian
  uint8_t version1;   // Set to 1
  uint8_t pad[9];     // Unused (set to 0)
  uint16_t type;      // 1==relocatable.o, 2==executable, 3==sharedobject.so
  uint16_t machine;   // 2==sparc,3==x86,4==m68k,10==mips
  uint32_t version2;  // Set to 1
  uint32_t entry;     // Process entry point (virtual address)
  uint32_t phoff;     // Program Header table (file offset, usually ehsize or 0)
  uint32_t shoff;     // Section header table (file offset)
  uint32_t flags;     // "processor-specific flags", whatever that is.
  uint16_t ehsize;    // Set to 52 (elf header's size in bytes.  Why?)
  uint16_t phentsize; // sizeof(program header table entry) == 32
  uint16_t phnum;     // Count of entries in program header table (0 for *.o).
  uint16_t shentsize; // sizeof(section header table entry) == 40
  uint16_t shnum;     // Count of entries in section header table.
  uint16_t shstrndx;  // Index in section header table of .shstrtab
} eheader;

The "endian" byte indicates the endianness of the rest of the data. The default entry value for x86 is 0x8048000.

The above refers to three other data structures:

Program Header Table

Executables and shared libraries have a program header table, *.o files do not (which is why you can't run 'em). When an ELF file has a program header table, it usually starts right after the ELF header, I.E. eheader.phoff == 52. For *.o files both phoff and phnum are zero.

The program header table is an array of program header structs, each of which describes a chunk of the file relevant to actually executing a program with this file. The ELF spec says it must come before any other loadable segment in the file, although it doesn't say why.

Each of the program header structs looks like:

struct orlando_bloom {
  uint32_t type;    // LOAD=1, DYNAMIC=2, INTERP=3, NOTE=4, PHDR=6
  uint32_t offset;  // Starting location of data in file.
  uint32_t vaddr;   // virtual address to map the segment into memory at
  uint32_t paddr;   // physical address (unused in Linux?)
  uint32_t filesz;  // Number of bytes to load from file.
  uint32_t memsz;   // Number of bytes to allocate in memory.
  uint32_t flags;   // Or together: execute=1, write=2, read=4
  uint32_t align;   // loader aligns vaddr to this, must be power of 2.
} pheader;

If memsz > filesz then the memory at the end is zeroed. (If memsz < filesz your ELF file is broken.) The flags say what permissions to mmap it with.