Chapter 4. Determining Program Behavior

Table of Contents

Tracing System Calls
Tracing Library Calls

There are a couple of tools that allow us to look into program behavior at a more closer level. Lets look at some of these:

Tracing System Calls

This section is really only relevant for to our efforts under UNIX, as Windows system calls change regularly from version to version, and have unpredictable entry points.

strace/truss(Solaris)

These programs trace system calls a program makes as it makes them.

Useful options:

  1. -f (follow fork)

  2. -ffo filename (output trace to filename.pid for forking)

  3. -i (Print instruction pointer for each system call)

Tracing Library Calls

Now we're starting to get to the more interesting stuff. Tracing library calls is a very powerful method of system analysis. It can give us a *lot* of information about our target.

ltrace

This utility is extremely useful. It traces ALL library calls made by a program.

Useful options:

  1. -S (display syscalls too)

  2. -f (follow fork)

  3. -o filename (output trace to filename)

  4. -C (demangle C++ function call names)

  5. -n 2 (indent each nested call 2 spaces)

  6. -i (prints instruction pointer of caller)

  7. -p pid (attaches to specified pid)

API Monitor

API Monitor is incredible. It will let you watch .dll calls in real time, filter on type of dll call, view