A Blog

Scientists who write simulation software seem to always take the quick and dirty approach rather than hiring a Computer Scientist and some coders. I recently ran into this when compiling a wave propagation program on Purdue’s SiCortex. The dependencies were a mile long and there were cross-dependencies in the list. Most of these packages were smallish and compiled without any trouble at all. Others had absolute paths encoded in header files and incomplete or very old documentation. One library was actually just a library of other libraries and was just not wanting to be sprung into life.

After corresponding with the author a bit, I got the propagation code to start compiling without giving me too much grief. At about this time, warnings began to fly. It’s almost as if someone took a copy of this program from C and ported it to C++ without much thought about what was going on. But, most aren’t that scary so we can overlook them for now. However, the compile kept crashing. Sometimes without much warning or with spectacular errors from the compiler itself (“this isn’t right, send my output to my authors for assistance….”)

After speaking a little bit with the Pathscale guys, they pointed out that the compiler was using /tmp to keep its state during the compiling of large files and that that filesystem was becoming full. Doh! Who knew it would take more than 2GB of storage for /tmp to keep the statement of a single compile job?

A node within a SiCortex more or less “netboots” and mounts the root file system read only from a network block device. Anything that is writable goes into a tmpfs, this included /tmp. Every node has only ~8gb, so one can certainly understand why the writable file systems are kept a little small.

In the end, I set TMPDIR to point to an NFS mounted file system. The compile took ages to complete, but it did complete somewhere successfully.