[matthiaskoenig]

Why

The experiments of a run are independent, but ExperimentRunner.run_experiments ran them one after the other. Most of the time of a run goes into drawing figures, which used a single CPU.

What

• run_experiments(n_workers=None), also on the module function run_experiments, runs the experiments in a pool started with parallel.start_pool.

  • By default it uses one worker per EXPERIMENTS_PER_WORKER (4) experiments, at most one per CPU.
  • A run of a few experiments, and a run inside a worker, stays serial.
  • n_workers=1 is serial; a number gives a pool of that many workers, at most one per experiment.

• What a worker gets and does. A worker (_run_job) receives a pickled _Job:

  • the class of the experiment, the settings the runner created it with, the paths, the JIT level and the simulator;
  • the arguments of the run, the matplotlib parameters the run changed (_changed_rc_params, e.g. the rc_context of pkpdsim) and the levels of the loggers.

  The worker then:

  • creates the experiment again, keeps the loaded models in parallel.worker_cache for its next experiment, and runs the experiment with a simulator of one worker;
  • writes the outputs and sends back the fields of the ExperimentResult and its log records (_Outcome).

• What the parent does.

  • It keeps its own initialized experiments, which the report is created from, and logs the records of the workers in the order of the experiments.
  • It submits the most expensive experiments first (_cost: the points of the scans and FIGURE_COST per figure).
  • The forkserver preloads the modules of the experiments.
  • An experiment whose worker died (BrokenProcessPool) runs in the parent again.

• Serial runs. The run stays serial, with a log message giving the reason (_serial_reason, _scan_pool), when:

  • the figures are shown, or the results are kept;
  • the runner has a unit registry of its own;
  • an experiment cannot be pickled or imported by a worker: a class defined in a function, at runtime, or in the main script. A worker imports the script without the block of its guard.
  • by default only: the simulator runs its scans in a pool (Simulator(n_workers>1), or None with a scan of POOL_THRESHOLD points), so the scans keep that pool. An explicit n_workers pools the experiments anyway.

• Logging. While an experiment runs, the worker's root logger has a single handler, _Collect. A handler the worker set up itself, e.g. one installed by a script that configures logging when it is imported, would print every message a second time and out of order. A record that cannot be pickled is sent as plain text.

• The parent marks the results of a pooled experiment as released, as in a serial run.

Measurements

run_simulation pkdb_models.models.hctz --action simulate --experiments all (28 experiments, python 3.15, 20 CPUs):

workers	wall time	memory of all processes
1	20.8 s	0.5 GB
4	12.9 s	2.3 GB
8	11.9 s	3.6 GB
20	12.8 s	7.6 GB

The default for 28 experiments is 7 workers. The 740 SVG, TSV, JSON and HTML files of the run with 8 workers are the same as those of the serial run.

Trade-off

An experiment runs its scans serially in its worker. The most expensive experiment of the HCTZ model, a parameter scan, takes 5.6 s in a worker and is most of the remaining wall time. A run of a single large scan is faster with n_workers=1, which leaves the scan to the pool of the simulator. A script which runs a pool needs the guard if __name__ == "__main__":, as for a pooled scan.

Tests

• New file tests/experiment/test_runner_parallel.py, ten tests:
  • a pool gives the results of a serial run, with the parameters of matplotlib of the run and the messages of the workers in the order of the experiments;
  • an experiment whose worker dies runs in the parent;
  • a run of few experiments is serial by default and one of more is pooled;
  • classes which cannot be pickled run serially;
  • n_workers=0 raises;
  • a simulator with a pool keeps the experiments serial by default;
  • an experiment of the main script runs serially, without breaking the pool;
  • the results of a pooled experiment are released;
  • a worker logs only through the run;
  • a message that cannot be pickled is sent as text.

The review also found that the duplicated logs, the broken pool for a class behind the guard, and the lost scan pool under pkpdsim --workers N could happen; all three are fixed in the second commit.

• pytest --skip-slow tests/experiment tests/report tests/examples tests/plot tests/test_parallel.py: 483 passed.