Calibrating solprop to your machine#

The native backend’s CPU execution settings are properties of the machine: how each operation threads (serially, on a few threads or on the whole team), how wide the team is and which cores it takes, how idle threads wait between parallel regions, and how much effort FFTW spends choosing its plans. A setting that runs one machine’s cases 13 % faster on average can run another’s 1.4 to 1.7 times slower. A calibration measures them on your machine and keeps the result, so that every later run uses it. It never changes a result beyond rounding: it chooses among exact algorithms and thread counts, as the built-in defaults do.

solprop calibrate                # standard: about two minutes
solprop calibrate --quick        # about 30 s
solprop calibrate --thorough     # about seven minutes
solprop calibrate --show         # the profile this job would use, and its evidence

Run solprop calibrate once after installing solprop, on the machine (or the kind of node) that runs your simulations. If you skip it, the first native run calibrates quickly by itself, with a notice (below).

What it measures#

The calibration runs in child processes, each pinned to the CPUs of your job, and times short windows of five problems that solprop ships: a GNLSE with Raman response, a four-mode capillary with plasma, a mode-averaged capillary, a radial filament and a one-dimensional free-space beam. The standard and thorough levels also time four of them at production size (16384 samples, four modes on 5120, 384 radial nodes, 4096 columns), because a setting that speeds up small problems can slow large ones down. It never runs your own simulation; for that, see refining one simulation.

Waiting and the team. Idle threads either sleep at once or spin briefly before the next region; spinning saves the 16 to 72 µs of a wake-up, sleeping frees the cores. The team may also be narrower than the job: the fastest kind of core on a chip with several, one socket of two (standard and thorough only, where the production-sized problems can show what a narrower team costs). Each candidate is a child process with those settings.
Operation grain. Each kind of operation (stage passes, responses, products, Raman histories, copies, transforms) threads above a size threshold. A batch of transforms threads by its shape: about one thread per row, more only where the rows are long, so the calibration also halves and doubles the samples per thread. The calibration moves each threshold past the work sizes the problems actually run, and tries other medium widths and compact teams for one-column problems.
FFT planning (standard and thorough). Every transform the problems create is planned alone at FFTW’s measure effort within a time limit; each group of plans (serial rows, threaded rows, transverse planes) keeps the effort that runs fastest inside the step, against plans made as a run without a profile makes them. A more exhaustive plan is not always a faster one, and a plan chosen alone not always the best in the step. Only a winning effort’s wisdom is kept, so later runs build those plans at once instead of measuring again; without one, runs plan as they would without a profile. The calibration’s problems are not yours: the plans of your own transforms are measured by the first run that needs them and kept (below).

A candidate replaces the built-in value only if it wins in every round of interleaved timings by more than the noise measured by timing the built-in setting against itself, and is slower on no problem by more than that noise plus 5 %; the winners are then validated together on fresh rounds, under the same bar. Ties and anything uncertain keep the built-in value. On a busy machine (other work on the job’s cores or their hyperthreads, time stolen by a hypervisor, or timings noisier than 10 %) the calibration says so and publishes nothing.

level	time	waiting and team	grain	FFT planning	rounds
`--quick`	about 30 s	the wait only	medium width and compact team	plan policy	3
standard	about 2 min	yes, two narrower teams	thresholds of the main operations	`measure`, 0.25 s a plan	5
`--thorough`	about 7 min	yes	thresholds and full sizes of every operation	`measure`, 20 s a plan	7

The standard and thorough levels also calibrate the smaller allocations they can pin children to inside the job, such as one socket, one NUMA node or one L3 cache group, while their time allows, so that one calibration of a whole node covers most smaller jobs on it.

Where the results live#

One directory per user, the platform’s cache directory, which SOLPROP_CACHE_DIR moves:

platform	directory
Linux	`$XDG_CACHE_HOME/solprop`, or `~/.cache/solprop`
macOS	`~/Library/Caches/solprop`
Windows	`%LOCALAPPDATA%\solprop\Cache`

A profile is kept per machine and build: the processor and its cache and memory layout, the operating system, the native build and the calibration’s own version. A new release, a rebuilt extension or another machine finds no profile and runs on the built-in table, so nothing stale is ever applied. Within a profile there is one row per allocation: the cores a job holds, of which kinds, and how many caches, NUMA nodes and sockets they span. Eight cores sharing a cache behave differently from eight across two sockets.

A job takes its own allocation’s row. Without one, it takes the workspace settings of the row with the same layout and the nearest smaller core count, and otherwise the built-in table.

The same directory keeps the FFT plans runs measure, in wisdom.json beside the machine and build’s profiles. The first run of a transform shape measures its plan in place, inside its own workspace, and stores it; every later run of that shape, in any process, builds the plan from it at once. A plan the profile holds keeps precedence: runs never read or overwrite one of the profile’s winners from this file. The file is kept with or without a profile, and goes with the profile when you clear it (Performance tuning says more).

The first run#

With no profile for the machine, the build and the job’s allocation, the first native run prints a notice like

CalibrationNotice: solprop is calibrating the native backend to this machine (quick,
about 30 s on 18 cores; once per machine, allocation and release). The profile is kept
in /home/me/.cache/solprop. Run `solprop calibrate` for a fuller calibration, or set
SOLPROP_CALIBRATION=use to never measure, or =off to ignore profiles.

and calibrates at the quick level before it starts. Importing solprop or the native backend never calibrates. A script whose native backend has not loaded yet takes the whole new profile; a notebook that has already run a native simulation applies the new operation grain at once and the new waiting and team from its next session.

SOLPROP_CALIBRATION chooses:

value	effect
`auto` (default)	use a profile; calibrate quickly on first use when there is none
`use`	use a profile; never measure implicitly
`off`	the built-in table and no stored FFT plans, whatever the store holds (a file `SOLPROP_FFTW_WISDOM` names is still kept); for bisecting and for holding settings fixed while code changes

The notice is a Python warning of its own category, shown once; silence it with the standard warnings filters (warnings.filterwarnings("ignore", category=solprop.exceptions.CalibrationNotice)) or with SOLPROP_CALIBRATION=use.

Concurrent first runs on one machine, such as the tasks of an array job, wait for the one calibrating, then use its profile. A calibration that found the machine too busy publishes nothing, and the next first run tries again after a day.

Scans and clusters#

A scan calibrates its workers’ allocation at the standard level before it starts them, in a child pinned like one worker, unless a standard or thorough row already matches; a scan usually runs for a long time, often after waiting in a queue, so a minute up front is cheap. Workers never calibrate. Remote scans calibrate on their own hosts. A scan whose workers run one thread each never calibrates: one CPU has no team, waiting or grain to choose.

On a cluster, calibrate once per node type and share the profile:

# on one node of each type, inside a job holding the whole node:
solprop calibrate --thorough
solprop calibrate --export /shared/solprop-profiles/node-type-a.json
# every user then points at the shared directory:
export SOLPROP_CALIBRATION_PATH=/shared/solprop-profiles

SOLPROP_CALIBRATION_PATH lists read-only directories searched after your own store, separated like PATH; a file exported for one node type applies only on that type. solprop calibrate --import FILE puts an exported profile into your own store.

Reading `--show`#

allocation:   18 cores
profile:      standard, generation g20261108T152341Z-1272949-c0d036, 2026-11-08T15:23:41Z
              measured on this allocation
process:      threads 18, wait spin
workspace:    medium_threads 8 (built-in 4)
fitted under: threads 18, wait spin
decided:      the built-in rules kept: no candidate won every round and its validation;
              grain: admitted: medium 8; FFT planning: the plan policy's efforts
noise:        0.02 (A/A)
candidate:    passive/18 1.272 (lost or tied in a round: worst ratio 1.280 against 0.980)

process is what the job’s native runtime starts with, workspace each operation setting beside the built-in value it replaces, and fitted under the process settings those were validated under: a job that sets its own wait or thread count keeps the built-in table for them. Each candidate’s score is its median time relative to the built-in setting. Every run records the settings it actually applied in its metadata (calibration in the backend’s settings), so a result says what produced it.

solprop calibrate --clear forgets this machine and build’s profiles, their refinements and the FFT plans runs stored, and the plans kept beside a file SOLPROP_FFTW_WISDOM names.

Refining one simulation#

For a costly campaign, a second stage fits the settings to one simulation:

report = simulation.calibrate(budget=60.0)
print(report.describe())

or solprop calibrate --recipe run.toml. It builds fresh native workspaces of the simulation in the calling process and times trial steps of its own operators at its start and a third and two thirds along it, under candidate operation settings; the simulation itself is not changed. Speed alone never admits a candidate: its steps must agree with the built-in’s to within ten times the run’s relative tolerance. The result is kept under the simulation’s execution descriptor (its grid, modes, nodes, chunk, responses, integrator and the process’s settings), and every later run or scan point with the same descriptor applies it, for as long as the machine profile it refined stays the same.

Overrides#

Explicit settings keep their precedence over any profile: SOLPROP_THREADS, SOLPROP_NATIVE_THREADS and the pool variables, OMP_WAIT_POLICY, GOMP_SPINCOUNT and KMP_BLOCKTIME, SOLPROP_FFTW_PLAN and SOLPROP_NATIVE_THREADED_FFT_MIN_SAMPLES (Performance tuning). SOLPROP_FFTW_WISDOM only moves the plans runs store; the profile’s own plans still apply. A setting measured under conditions an override changes is not used with it: operation thresholds fitted under spinning threads give way to the built-in table when you make the threads sleep.

The design and its measurements are in SEP 46.