API¶

Module: bootstrap¶

scine_puffin.bootstrap.bootstrap(config: Configuration)[source]¶

Sets up all required and also all additionally requested programs/packages for the use with Puffin. Generates a puffin.sh to be sourced before running the actual puffin.

Parameters:

configscine_puffin.config.Configuration: The current configuration of the Puffin.

Module: config¶

class scine_puffin.config.Configuration[source]¶

The Puffin configuration. All values are defaulted. The main sections of the configuration are:

daemon: The settings pertaining the execution of Puffin and its daemon process.
database: All information about the database the Puffin will be working on.
resources: The information about the hardware the Puffin is running on and is allowed to use for calculations and the execution of jobs.
programs: The settings for all the programs and packages Puffin relies on when executing jobs. Each program/packages has its own entry with the possibility of program specific settings. See the documentation for each individual program (found at scine_puffin.programs) for more details about the individual settings.

Note that the config is sensitive to environment variables when it is initialized/loaded. Each setting in the config can be set via a corresponding environment variable. The settings are given as PUFFIN_<key1>_<key2>=<value> where the keys are the chain of uppercase keys to the final setting. As an example: PUFFIN_DAEMON_MODE=debug would equal config['daemon']['mode'] = 'debug'.

In detail, the options in the configuration are:

daemon

modestr: The mode to run the Puffin in, options are release and debug. The release mode will fork the main process and run in a daemonized mode while the debug mode will run in the current shell, reporting any output and errors to stdout and stderr.
job_dirstr: The path to the directory containing the currently running job.
software_dirstr: The path to the directory containing the software bootstrapped from sources. The Puffin will generate and fill this directory upon bootstrapping.
error_dirstr: If existent, the Puffin instance will archive all failed jobs into this directory.
archive_dirstr: If existent, the Puffin instance will archive all correctly completed jobs into this directory.
uuidstr: A unique name for the Puffin instance. This can be set by the user, if not, a unique ID will automatically be generated.
pidstr: The path to the file identifying the PID of the Puffin instance. Automatically populated on start-up if left empty.
pid_dirstr: The path to a folder holding the file identifying the PID of the Puffin instance.
logstr: The path to the logfile of the Puffin instance.
stopstr: The path to a file that if existent will prompt the Puffin instance to stop taking new jobs and shut down instead. The instance will finish any running job though.
remove_stop_filebool: Upon finding a stop file the daemon will stop, if this option is set to True the found file will be deleted allowing instant restarts. In cases where multiple puffins depend on the same stop file it may be required to keep the stop file, setting this option to False
cycle_time_in_sfloat: The time in between scans of the database for new jobs that can be run.
timeout_in_hfloat: The number of hours the Puffin instance should stay alive. Once this limit is reached, the Puffin is shut down and its running job will be killed and re-flagged as new.
idle_timeout_in_hfloat: The number of hours the Puffin instance should stay alive. After receiving the last job, once the limit is reached, the Puffin is shut down. Any accepted job will reset the timer. A negative value disables this feature and make the Puffin run until the timeout_in_h is reached independent of it being idle the entire time.
touch_time_in_sfloat: The time in seconds in between the attempts of the puffin to touch a calculation it is running in the database. In practice each Puffin will search for jobs in the database that are set as running but are not touched and reset them, as they indicate that the executing puffin has crashed. See job_reset_time_in_s for more information.
job_reset_time_in_sfloat: The time in seconds that may have passed since the last touch on pending jobs before they are considered dead and are reset to be worked by another puffin. Note: The time in this setting has to be larger than the touch_time_in_s of all Puffins working on the same database to work!
repeated_failure_stopint: The number of consecutive failed jobs that are allowed before the Puffin stops in order to avoid failing all jobs in a DB due to e.g. hardware issues. Failed jobs will be reset to new and rerun by other Puffins. Should always be greater than 1.
max_number_of_jobsint: The maximum number of jobs a single Puffin will carry out (complete or failed), before gracefully exiting. Any negative number or zero disables this setting; by default it is disabled.
enforce_memory_limitbool: If the given memory limit should be enforced (i.e., a job is killed as soon as it reaches it) or not. The puffin still continues to work on other calculations either way.

database

ipstr: The IP at which the database server to connect with is found.
portint: The port at which the database server to connect with is found.
namestr: The name of the database on the database server to connect with. Multiple databases (with multiple names) can be given as comma seperated list: name_one,name_two,name_three. The databases will be used in descending order of priority. Meaning: at any given time all jobs of the first named database will have to be completed before any job of the second one will be considered by the Puffin instance.

resources

coresint: The number of threads the executed jobs are allowed to use. Note that only jobs that are below this value in their requirements will be accepted by the Puffin instance.
memoryfloat: The total amount of memory the Puffin and its jobs are allowed to use. Given in GB. Note that only jobs that are below this value in their requirements will be accepted by the Puffin instance.
diskfloat: The total amount of disk space the Puffin and its jobs are allowed to use. Given in GB. Note that only jobs that are below this value in their requirements will be accepted by the Puffin instance.
ssh_keysList[str]: Any SSH keys needed by the Puffin in order to connect to the database or to bootstrap programs.

programs

The specific details for each program are given in their respective documentation. However, common options are:

availablebool: The switch whether the program shall be available to Puffin. Any programs set to be unavailable will not be bootstrapped.
sourcestr: The link to the source location of the given program, usually a https link to a git repository
rootstr: The folder at which the program is already installed at. This will request a non source based bootstrapping of the program.
versionstr: The version of the program to use. Can also be a git tag or commit SHA.

The default version of a configuration file can be generated using python3 -m puffin configure (if no environment variables are set).

daemon() → dict[source]¶

Grants direct access to the daemon part of the configuration.

Returns:

settingsdict: A sub-dict of the total configuration.

database() → dict[source]¶

Grants direct access to the database part of the configuration.

Returns:

settingsdict: A sub-dict of the total configuration.

dump(path: str) → None[source]¶

Dumps the current configuration into a .yaml file.

Parameters:

pathstr: The file to dump the configuration into.

load(path: str | None = None) → None[source]¶

Loads the configuration. The configuration is initialized using the default values, then all settings given in the file (if there is one) are applied. Finally, all settings given as environment variables are applied.

Each setting in the config can be set via a corresponding environment variable. The settings are given as PUFFIN_<key1>_<key2>=<value> where the keys are the chain of uppercase keys to the final setting. As an example: PUFFIN_DAEMON_MODE=debug would equal config['daemon']['mode'] = 'debug'.

The exact load order is (with the latter one overriding the former):

defaults
file path
environment variables

Parameters:

pathstr: The file to read the configuration from. Default: None

programs() → dict[source]¶

Grants direct access to the programs part of the configuration.

Returns:

settingsdict: A sub-dict of the total configuration.

resources() → dict[source]¶

Grants direct access to the resources part of the configuration.

Returns:

settingsdict: A sub-dict of the total configuration.

scine_puffin.config.dict_generator(indict: Dict[str, Any], pre: List[str] | None = None)[source]¶

A small helper function/generator recursively generating all chains of keys for a given dictionary.

Parameters:

indictdict: The dictionary to traverse.
predict: The parent dictionary (used for recursion).

Yields:

key_chainList[str]: A list of keys from top level to bottom level for each end in the tree of possible value fields in the given dictionary.

Module: daemon¶

scine_puffin.daemon.check_environment(config: Configuration)[source]¶

Checks the runtime environment for problematic configurations that may interfere with job executions down the line.

Parameters:

configscine_puffin.config.Configuration: The current configuration of the Puffin.

scine_puffin.daemon.shutdown(_signum, _frame)[source]¶

A small helper function triggering the stop of the process.

Parameters:

_signumint: Dummy variable to match the signal dependent function signature.
_frame: Dummy variable to match the signal dependent function signature.

scine_puffin.daemon.start_daemon(config: Configuration, detach: bool = True)[source]¶

Starts the Puffin, using the given configuration.

Parameters:

configscine_puffin.config.Configuration: The current configuration of the Puffin.
detachbool: If true, forks the daemon process and detaches it.

scine_puffin.daemon.stop_daemon(config: Configuration)[source]¶

Stops the Puffin gracefully, allowing th current job to finish, then shutting down.

Parameters:

configscine_puffin.config.Configuration: The current configuration of the Puffin.

Module: jobloop¶

scine_puffin.jobloop.check_setup(config: Configuration) → Dict[str, str][source]¶

Checks if all the programs are correctly installed or reachable.

Parameters:

configscine_puffin.config.Configuration: The current configuration of the Puffin.

scine_puffin.jobloop.kill_daemon(config: Configuration) → None[source]¶

Kills the Puffin instantaneously without any possibility of a graceful exit.

Parameters:

configscine_puffin.config.Configuration: The current configuration of the Puffin.

scine_puffin.jobloop.loop(config: Configuration, available_jobs: dict) → None[source]¶

The outer loop function. This function controls the forked actual loop function, which is implemented in _loop_impl(). The loop has an added timeout and also a 15 min ping is added showing that the runner is still alive.

Parameters:

configscine_puffin.config.Configuration: The current configuration of the Puffin.
available_jobsdict: The dictionary of available jobs, given the current config and runtime environment.

scine_puffin.jobloop.slow_connect(manager, config: Configuration) → None[source]¶

Connects the given Manager to the database referenced in the Configuration. This version of connecting tries 30 times to connect to the database. Each attempt is followed by a wait time of 1.0 + random([0.0, 1.0]) seconds in order to stagger connection attempts of multiple Puffin instances.

Parameters:

managerscine_database.Manager: The database manager/connection.
configscine_puffin.config.Configuration: The current configuration of the Puffin.

Module: programs.program¶

class scine_puffin.programs.program.Program(settings: dict)[source]¶

A common interface for all programs and their setups

Parameters:

settingsdict: The settings for the particular program. This dictionary should be the given program’s block in the Configuration.

available_models() → List[str][source]¶

A small function returning the single point models available now with the given program loaded/installed.

Returns:

modelsList[str]: A list of names of models that are available if the program is available.

check_install()[source]¶: A small function checking if the program was installed/located correctly and does provide the expected features.

static initialize()[source]¶: Executed at Puffin start, run once for each available program

install(repo_dir: str, install_dir: str, ncores: int)[source]¶

Installs or loads the given program. After the install, the check_install function should run through with out exceptions. The choice of installation/compilation or loading of the program is based on the settings given in the constructor.

Parameters:

repo_dirstr: The folder for all repositories, if a clone or download is required for the installation, this folder will be used.
install_dirstr: If the program is actually installed and not just loaded, this folder will be used as target directory for the install process.
ncoresint: The number of cores/threads to be used when compiling/installing the program.

pip_module_source_install(repo_dir: str, install_dir: str)[source]¶

static pip_package_install(package: str, install_dir: str)[source]¶

scine_module_install(repo_dir: str, install_dir: str, ncores: int, add_lib: bool = False, add_bin: bool = False)[source]¶

setup_environment(config: Configuration, env_paths: dict, env_vars: dict)[source]¶

Appends the program specific environment variables to the given dictionaries.

Parameters:

configscine_puffin.config.Configuration: The current global configuration.
env_pathsdict: A dictionary for all the environment paths, such as PATH and LD_LIBRARY_PATH. The added settings will be appended to the existing paths, using export PATH=$PATH:....
env_varsdict: A dictionary for all fixed environment variables. All settings will replace existing variables such as export OMP_NUM_THREADS=1