0.5.4

Features

• Support Python 3.13

Bug fixes

• print-output fails gracefully if an experiment has not started yet.
• src layouts are no longer converted to flat layout but instead added to the PYTHONPATH.
• fixed fail-trace printing
• seml queue shows the executed script for non-seml jobs.
• start --debug fails gracefully if a collection is empty.

0.5.3 - Bugfix

Fixes

• Fixed multiple experiments not running concurrently with experiments_per_job>1.
• Fixed a killed experiment detection when one job of the batch was still running.

0.5.2 - Performance, Typing, Venv

Breaking Changes

• Dropped support for Python 3.8 and 3.9

Features

• seml now works with Python environments other than conda (e.g., venv, virtualenv or uv) by copying all environment variables at staging time and reusing these at execution time. Environment source files are ignored when stashing source code.
• seml now provides basic typing.

Performance

• Nearly all seml commands got a lot faster.
• Source files are uploaded via multithreading.
• Source file deletion is now done in a single query rather than one by one.
• seml status, seml reset, seml delete now all rely on bulk writes.
• seml add's duplicate detection with hashes got faster via bulk reads. Other parts of add have been optimized with regex checks.

Bug fixes

• seml will now report the right number deleted src files.
• seml will now report the right number of changed configuration in reload-sources.
• Calling seml cancel on a empty collection no longer kills all pending jobs.
• seml will no longer to attempt to copy whole virtual environments if they are within the project root dir.

0.5.1 - Bugfixes

Features

• Added -head and -tail argument to print-output command

Bugfix

• Removed extra job that sneaked into every new slurm start.

0.5.0 - Slurm config arrays, multi-node, CLI improvements

This release includes breaking changes to how jobs and experiments are associated. Further, a lot of convenience commands have been added.

New Experiment - Slurm Job Association

Previously, each experiment was directly associated with a specific job (by default, 1:1 or 1:n if experiments_per_job: n). This was possible as each seml config may only contain a single slurm config. Now, the slurm block accepts an array of slurm configs. This change implies that at submission time, the experiment is unaware of which Slurm job may execute it. Each slurm job then greedily pulls the experiments. This allows the following configuration to better utilize larger GPUs:

slurm:
  - experiments_per_job: 1
    sbatch_options:
      gres: gpu:1
      partition: gpu_gtx1080
  - experiments_per_job: 8
    sbatch_options:
      gres: gpu:1
      partition: gpu_a100

This will now spawn two job arrays, one on the gtx1080 partition and one on the gpu_a100 partition, which both greedily pull jobs to be executed.

Breaking Changes

The document layout for an experiment in the MongoDB has been altered! Old collections will be automatically migrated to the new format but there is no going back! Make sure to call seml --migration-backup <col> if you want seml to first create a backup.

• The document layout for the MongoDB has been altered. Automatic migration is added, though we urge users to be careful!
• When running with multiple experiments per job, each job now gets its own log file.
• This is likely the last release supporting Python 3.8 and Python 3.9

Features

• seml now supports multi-process jobs (and multi-node jobs) (#135)
• seml now supports multiple slurm configurations (#137)
• The collection cache for autocompletion is automatically refreshed when calling add, delete or drop.
• We now support src-layouts by converting them to a flat-layout at runtime.
• Added seml <col> restore-sources <path> to restore source files form the MongoDB.
• Added seml <col> hold and seml <col> release to hold and release slurm jobs.
• Added seml <col> print-experiment to retrieve experiment documents from the database.
• Added seml queue to print the collection of slurm jobs (only works for jobs submitted with this version or newer)
• For debugging, seml now supports clickable vscode links (#133)
• Experiment names may now contain / to create log directories.
• Cancel experiments by default when deleting them.
• Seml also suggests commands that do not need a collection like drop, queue or list in autocompletion.
• The CLI has been organized into groups.
• Autocompletion got faster.

Development

• Updated CI to use pre-commit and uv instead of pip.
• The repo has been restructured.

Fixes

• SSH connections are now handled in a separate process that tracks its health.
• Fixed multi-user SSH port forwarding.
• When encountering non-unicode symbols in reading the file output, we replace them with the Unicode replacement character.

0.4.6 - bugfix

• Fixed a bug that seml wouldn't output anything to the console if SSH forward were used.

0.4.5 - Support newer versions of typer

Bug fixes

• This version supports newer versions of typer.

0.4.4 - SSH Port Forwarding

Features

• Added support for port forwarding to access MongoDB Server #134 .
  • To use this feature install seml via pip install seml[ssh_forward].
  • Configure your MongoDB and ssh forward via seml configure --ssh_forward
• Add clickable VSCode Debugger links #133
  • You need the Debug Launcher extension for this feature.
• Add option stash_all_py_files to the seml configuration block to stash all python files within the current directory (instead of only stashing the imported ones).
• Added status widget to show which command is currently running.
• Allow the import of seml.experiment.Experiment directly from seml.

Fixes

• Fixed minor issues in source code discovery.

0.4.3 - Templates

Features

• seml now provides template(s), if you want to initialize a new project simply run seml project init <project_name> and a new folder project_name will be created with sensible defaults. List available templates via seml project list-templates.
  • Templates are managed in https://github.com/TUM-DAML/seml-templates
  • Custom repositories and versioning is supported via git
• Add a setting to set the default terminal width for experiments SETTINGS.EXPERIMENT.TERMINAL_WIDTH.
• Parallel processing for seml.get_results

Bug fixes

• Fixed node detection.
• Several fixes related to print-output.

0.4.2 - seml.experiment.Experiment

Breaking changes

• The use of sacred.Experiment has been deprecated. Please use seml.experiment.Experiment going forward. See https://github.com/TUM-DAML/seml/blob/master/examples/example_experiment.py
  • seml.experiment.Experiment already includes the common MongoDB, logging and experiment statistics collection setup
  • By default, stdout is no longer stored in the MongoDB, reenable this in your settings.py or for a specific experiment with seml.experiment.Experiment(capture_output=True).
• Some settings have been reorganized, see https://github.com/TUM-DAML/seml/blob/master/src/seml/settings.py

Features

• Added seml <col> print-output: print experiment outputs directly via CLI. This command first attempts to gather the output via the slurm log file. If that fails it will search for the correct log within the MongoDB.
• Added seml --version

TLDR:

old

import sacred
import seml

ex = sacred.Experiment()
seml.setup_logger(ex)

@ex.post_run_hook
def collect_stats(_run):
    seml.collect_exp_stats(_run)

@ex.config
def config():
    overwrite = None
    db_collection = None
    if db_collection is not None:
        ex.observers.append(
            seml.create_mongodb_observer(db_collection, overwrite=overwrite)
        )

new

from seml.experiment import Experiment

ex = Experiment()