mon_sacct
mon_sacct is a custom script to show you very detailed information about a job on M3. It is a convenient wrapper around Slurm's sacct command.
Usage
If you call mon_sacct JOB_ID with a job ID, it will you show very detailed info about a past job. Note the example output below is heavily abridged, run the command for yourself to see all of the details!
[lexg@m3-login3 ~]$ mon_sacct 41762254
Loading python/3.7.3-system
Loading requirement: gcc/8.1.0
/apps/slurm-23.11.9/bin/sacct -S 2024-12-04 -p --format="User,JobID,JobName,State,Submit,Start,End,Elapsed,WorkDir%35,TimeLimit,ReqMem,Account,Partition,Priority,MaxVMSize,AveVMSize,AveRSS,MaxRSS,MaxRSSTask,MaxRSSNode,QOS,NodeList,CPUTime,UserCPU,SystemCPU,TotalCPU,minCPU,ReqTRES,Reservation,NTasks,ExitCode,Nodelist,SubmitLine,Reason,Comment,Constraints,Comment,SystemComment,AdminComment" -j 41762254
+--------------+-------------------------------+
| Field | Value |
+--------------+-------------------------------+
| User | lexg |
| JobID | 41762254 |
| JobName | Very important job |
| State | COMPLETED |
...
+--------------+-------------------------------+
+--------------+---------------------+
| Field | Value |
+--------------+---------------------+
| User | |
| JobID | 41762254.batch |
| JobName | batch |
| State | COMPLETED |
...
+--------------+---------------------+
+--------------+---------------------+
| Field | Value |
+--------------+---------------------+
| User | |
| JobID | 41762254.extern |
| JobName | extern |
| State | COMPLETED |
...
+--------------+---------------------+
+--------------+---------------------+
| Field | Value |
+--------------+---------------------+
| User | |
| JobID | 41762254.0 |
| JobName | hostname |
| State | COMPLETED |
...
+--------------+---------------------+
If you call mon_sacct without any arguments, it will show you details about recent jobs, similar to show_job.
[lexg@m3-login3 ~]$ mon_sacct
Loading python/3.7.3-system
Loading requirement: gcc/8.1.0
/apps/slurm-23.11.9/bin/sacct -S 2024-11-18 -p --format="JobID,Start,JobName,Partition,Account,AllocCPUS,State,ExitCode,Nodelist"
+-----------------+---------------------+--------------------------+-----------+---------+-----------+--------------------+----------+---------------+
| JobID | Start | JobName | Partition | Account | AllocCPUS | State | ExitCode | NodeList |
+-----------------+---------------------+--------------------------+-----------+---------+-----------+--------------------+----------+---------------+
| ... lots of rows here, note the output is sorted from oldest to newest ... |
| 41762254 | 2024-12-17T15:51:05 | Very important job | comp | nq46 | 2 | COMPLETED | 0:0 | m3v[115-116] |
| 41762254.batch | 2024-12-17T15:51:05 | batch | | nq46 | 1 | COMPLETED | 0:0 | m3v115 |
| 41762254.extern | 2024-12-17T15:51:05 | extern | | nq46 | 2 | COMPLETED | 0:0 | m3v[115-116] |
| 41762254.0 | 2024-12-17T15:51:05 | hostname | | nq46 | 2 | COMPLETED | 0:0 | m3v[115-116] |
| 41804848 | 2024-12-18T10:56:11 | bash | comp | nq46 | 1 | COMPLETED | 0:0 | m3j009 |
| 41804848.extern | 2024-12-18T10:56:11 | extern | | nq46 | 1 | COMPLETED | 0:0 | m3j009 |
| 41804848.0 | 2024-12-18T10:56:12 | bash | | nq46 | 1 | COMPLETED | 0:0 | m3j009 |
+-----------------+---------------------+--------------------------+-----------+---------+-----------+--------------------+----------+---------------+
Output
When mon_sacct JOB_ID is called with a job ID, it will show a table for the job as well as a table for each job step. When called without a job ID, it will show a single table where each row is a separate job or job step.
In both cases, all of the field names you see are exactly as described in the sacct documentation.