slurm site_factor dampen

Implementation of a site_factor plugin that dampens the age priority boost for heavy users. The plugin applies a negative correction to the site_factor that grows with queue wait-time, but only for users classified as heavy. This directly counteracts the built-in AGE factor for those users, while leaving light users untouched.

Motivation

In Slurm, heavy HPC users accumulate massive AGE factors because their low fairshare scores force their jobs to sit languishing in the queue for extended periods, allowing the maximum amount of "queue age" to accrue. Light users, conversely, maintain pristine fairshare scores, meaning their jobs are scheduled almost immediately and rarely spend enough time in the queue to build up a significant AGE factor.

The AGE factor is intended to act as a "starvation prevention" mechanism so heavy users aren't locked out of the cluster forever. However, particular Slurm configuration values can create scenarios where heavy users have an unfair advantage over light or new users.

Conceptual design

Slurm’s total priority is :

priority = site_factor + (PriorityWeightAge × age_factor) + (PriorityWeightFairshare × fairshare_factor) + ...

The site_factor is an additive term that is completely under your control via the Slurm plugin API.

Strategy:

1. During the plugin's periodic update, we compute how long a pending job has been waiting.
2. If the job’s owner is listed in a config file as a heavy user, subtract priority points proportional to age × dampening_factor.

Result:

HPC cluster heavy users still accrue age priority from Slurm’s built-in calculation, but the site_factor term partially cancels it out. Light users get the full age benefit.

How to build

Download the repository and modify the provided Makefile to use the installed version of SLURM in your HPC cluster.

If you prefer you can also pass the location of SLURM_SRC as a key-pair value to the make command:

# If you unpacked Slurm source to /usr/src/slurm-24.05:
make SLURM_SRC=/usr/src/slurm-24.05
sudo make install DESTDIR=   # or copy manually to your Slurm plugin dir

The .so must land in Slurm’s plugin directory, typically /usr/lib64/slurm/ or /usr/lib/x86_64-linux-gnu/slurm/.

How to use

slurm.conf

Add of modify these lines to the slurm.conf file:

PriorityType=priority/multifactor
PrioritySiteFactorPlugin=site_factor/dampen_age
# Optional: if you want the plugin to read a non-default config path
# PrioritySiteFactorParameters=/etc/slurm/site_factor_dampen.conf

Do NOT set PriorityWeightAge=0 — you want the built-in age to keep working for everyone else. The plugin will only counteract it for the users you list.

Dampening config ( /etc/slurm/site_factor_dampen.conf )

This file is used to declare all the heavy HPC cluster users. The number indicates the amount of priority points subtracted per minute of qeue time.

# user    dampening_factor
# factor = priority points to subtract per minute of age
robin    3.0
batman   1.0
# any other user not listed here, gets full age benefit 

How to pick the factor

Use the sprio output to read the AGE of your heavy users after some wait. Use that to calculate how much built-in AGE is added per minute to find the dampening factor. For example a dampening factor of 3.0 means:

Wait time	Built-in AGE	Correction (site_factor)	Net age effect
1 h	~60	−180	−120 (penalized)
16 h	~960	−2880	−1920 (heavily penalized)

You can tune this empirically: start with 1.0 and increase until the desired behavior appears in sprio.

Deployment steps summary

1. Copy the plugin to the Slurm plugin directory.
2. Create the config file /etc/slurm/site_factor_dampen.conf.
3. Run slurmctld -t (or restart slurmctld on your test instance).
4. Verify it loaded in slurmctld.log:

site_factor/dampen_age: user=robin factor=3.00

5. Watch priorities with sprio -p GPU -l. You should expect to see the SITE column gets negative for heavy users as their jobs age.

Usage tips

• Reload without restart: editing the config file and running scontrol reconfig triggers site_factor_p_reconfig(), so you can tune user lists on the fly.
• Keep the config short: only list the top 2–3 heavy users. Everyone else gets the default full age benefit.
• Monitor with sprio -w: check that the SITE weight is correct.
• Capping: the plugin hard-caps the correction at 100000 points (about 28 h at 3.0/min) to avoid total starvation of the penalized users.

Other alternatives

There are two other options that achieve a similar effect:

1. Lower PriorityWeightAge — reduces age influence for everyone.
2. Shorten PriorityMaxAge — age maxes out faster, so heavy users can’t keep accumulating advantage indefinitely.

However, the plugin approach is more effective because of it's finer granularity (per-user control).