David's Blog

There are many good options for tools that provide insights into your running programs for Linux including newer tools like writing eBPF filters against the kernel and older ones like strace, perf, and `gdb`. Today I want to talk a little bit about a feature of many Unix kernels that is often used by other tools but can also be useful on its own for resolving little questions or problems you might have. That feature is ProcFS.

Command line and environment variables

I wanted to write about a couple of use cases that others may find useful in certain cases. The first is the ability to dump the command line and environment of a given process. There are certainly other ways of obtaining the command line of a process but it isn’t always easy to inspect the environment variables present at runtime. This can be especially useful when trying to debug or verify some behavior when the program doesn’t otherwise expose its configuration at runtime (e.g. via logging).

Let’s pretend that we have a process that we want to inspect. Perhaps it’s a Python process that is doing some web serving. Let’s use ProcFS to determine the command line arguments and the environment variables set within this process:

$ pgrep python
2961
$ cat /proc/2961/cmdline
python3.6-mhttp.server3000
$ xxd /proc/2961/cmdline
00000000: 7079 7468 6f6e 332e 3600 2d6d 0068 7474  python3.6.-m.htt
00000010: 702e 7365 7276 6572 0033 3030 3000       p.server.3000.
$ cat /proc/2961/environ
MYENV=ohhaiSECRET=blah
$ xxd /proc/2961/environ
00000000: 4d59 454e 563d 6f68 6861 6900 5345 4352  MYENV=ohhai.SECR
00000010: 4554 3d62 6c61 6800                      ET=blah.

First, we determine the PID for our process to inspect. From there we dump the contents of /proc/<pid>/cmdline and /proc/<pid>/environ (I do so both in ascii and hex so that you can see the exact format). Note that each command line argument and environment variable is separated by a null byte. Here you can see that we have a web server listening on port 3000 using Python’s built in http.server tool. We’re also able to see that there are two environment variables that in the server process’ environment and what their names values are.

I’ve personally found this environment variable dumping ability quite useful for debugging a program at runtime. Maybe you have access to the source and are observing some behavior in logs which is not trivial to reproduce and are trying to correlate that behavior back to the source. If the program derives behavior from the environment, sometimes being able to dump the exact state of the environment can be helpful in your debugging.

File descriptors

Another possibility is to inspect the state of a given file descriptor within a process. If you know a given file descriptor ID, you can use /proc/<pid>/fdinfo/<fd> to dump the state of the file descriptor (including the current position of a file, the mode used to open the descriptor, socket or event state, and more).

Kernel Stack

One last handy thing that comes up occasionally in debugging a hung process (especially one in uninterruptible sleep, the dreaded D state) that you can’t get relevant logs for or hooked up to a debugger is to get the stack trace of the kernel syscall that the process is waiting on, if any. Especially for processes in uninterruptible sleep, where the hang is usually I/O related, this can be helpful for getting at least some small indication of what type of thing is blocking the progress from making progress. Here’s an example from a web server that is running normally:

root@server:~# cat /proc/2411/stack
[<0>] ep_poll+0x29c/0x3a0
[<0>] SyS_epoll_wait+0xc6/0xe0
[<0>] do_syscall_64+0x73/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[<0>] 0xffffffffffffffff

We can see that the process is in epoll wait, which should not be surprising for a web server waiting on socket events. Examples of things that could have been shown would be things like reading/writing to a socket, a file on disk, or some remote filesystem.

More resources

It would be well worth your time to read some sections of man 5 proc for more details of various Procfs features. You’re almost certainly guaranteed to learn something and it might surprise you to learn about all the various things that are able to be introspected via a file-like interface. Everything is a file, indeed.