Log files useful for troubleshooting

The files listed here are all text files that may be useful when troubleshooting problems:

• Written by the PBS server
  • /tmp/pbs/working/server_priv/accounting Each PBS job is logged here when it is queued, when it starts and when it exits.
  • /tmp/pbs/working/server_logs/yyyymmdd The server logs every status request it receives, job submissions, interactions with the scheduler, interactions with MOMs, and so on. It also logs problems communicating with other service nodes.

• Written by a PBS MOM
  • /tmp/pbs/working/mom_logs/yyyymmdd The MOM logs every PBS job that it starts, every job that it kills for being over it's walltime limit, and every normal job termination.
  • .e and .o files When a PBS job completes, the MOM copies the script's stdout and stderr stream to files in the submission directory. While the job is still running, these files may be found in /tmp/pbs/working/spool.
• Written by the PBS scheduler
  • /tmp/pbs/working/sched_logs/yyyymmdd The scheduler lists the jobs it is scheduling to run. We enhanced the scheduler also to list jobs it chooses not to run and the reason it does not run them. The scheduler also lists service nodes it has trouble communicating with.
• Written by yod
  • /tmp/userlog A single line is written to this file by yod when a load terminates with an error or when an application completes. It lists the nodes the application ran on, it's start and end time, it's Cplant job ID and PBS job ID, and any abnormal termination of the application processes.
  • /var/log/cplant Warnings and exceptions that occur during application load are logged by yod to this file. You must have syslogd running on the service node to enable logging.