Settings¶

All settings in Puffin are controlled via the Configuration. The puffin.yml file represents the content of this class. All settings are explained defined in the documentation of the Configuration, hence it is shown below.

class scine_puffin.config.Configuration[source]

The Puffin configuration. All values are defaulted. The main sections of the configuration are:

daemon: The settings peratining the execution of Puffin and its daemon process.
database: All information about the database the Puffin will be working on.
resources: The information about the hardware the Puffin is running on and is allowed to use for calculations and the execution of jobs.
programs: The settings for all the programs and packages Puffin relies on when executing jobs. Each program/packages has its own entry with the possibility of program specific settings. See the documentation for each individual program (found at scine_puffin.programs) for more details about the individual settings.

Note that the config is sensitive to environment variables when it is initialized/loaded. Each setting in the config can be set via a corresponding environment variable. The settings are given as PUFFIN_<key1>_<key2>=<value> where the keys are the chain of uppercase keys to the final setting. As an example: PUFFIN_DAEMON_MODE=debug would equal config['daemon']['mode'] = 'debug'.

In detail, the options in the configuration are:

daemon

mode :: str: The mode to run the Puffin in, options are release and debug. The release mode will fork the main process and run in a daemonized mode while the debug mode will run in the current shell, reporting any output and errors to stdout and stderr.
job_dir :: str: The path to the directory containing the currently running job.
software_dir :: str: The path to the directory containing the software bootstrapped from sources. The Puffin will generate and fill this directory upon bootstrapping.
error_dir :: str: If existent, the Puffin instance will archive all failed jobs into this directory.
archive_dir :: str: If existent, the Puffin instance will archive all correctly completed jobs into this directory.
uuid :: str: A unique name for the Puffin instance. This can be set by the user, if not, a unique ID will automatically be generated.
pid :: str: The path to the file identifying the PID of the Puffin instance. Automatically populated on start-up if left empty.
pid_dir :: str: The path to a folder holding the file identifying the PID of the Puffin instance.
log :: str: The path to the logfile of the Puffin instance.
stop :: str: The path to a file that if existent will prompt the Puffin instance to stop taking new jobs and shut down instead. The instance will finish any running job though.
remove_stop_file :: bool: Upon finding a stop file the daemon will stop, if this option is set to True the found file will be deleted allowing instant restarts. In cases where multiple puffins depend on the same stop file it may be required to keep the stop file, setting this option to False
cycle_time_in_s :: float: The time in between scans of the database for new jobs that can be run.
timeout_in_h :: float: The number of hours the Puffin instance should stay alive. Once this limit is reached, the Puffin is shut down and its running job will be killed and re-flagged as new.
idle_timeout_in_h :: float: The number of hours the Puffin instance should stay alive. After receiving the last job, once the limit is reached, the Puffin is shut down. Any accepted job will reset the timer. A negative value disables this feature and make the Puffin run until the timeout_in_h is reached independent of it being idle the entire time.
touch_time_in_s :: float: The time in seconds in between the attempts of the puffin to touch a calculation it is running in the database. In practice each Puffin will search for jobs in the database that are set as running but are not touched and reset them, as they indicate that the executing puffin has crashed. See job_reset_time_in_s for more information.
job_reset_time_in_s :: float: The time in seconds that may have passed since the last touch on pending jobs before they are considered dead and are reset to be worked by another puffin. Note: The time in this setting has to be larger than the touch_time_in_s of all Puffins working on the same database to work!
repeated_failure_stop :: int: The number of consecutive failed jobs that are allowed before the Puffin stops in order to avoid failing all jobs in a DB due to e.g. hardware issues. Failed jobs will be reset to new and rerun by other Puffins. Should always be greater than 1.
max_number_of_jobs :: int: The maximum number of jobs a single Puffin will carry out (complete or failed), before gracefully exiting. Any negative number or zero disables this setting; by default it is disabled.
enforce_memory_limit :: bool: If the given memory limit should be enforced (i.e., a job is killed as soon as it reaches it) or not. The puffin still continues to work on other calculations either way.

database

ip :: str: The IP at which the database server to connect with is found.
port :: int: The port at which the database server to connect with is found.
name :: str: The name of the database on the database server to connect with. Multiple databases (with multiple names) can be given as comma seperated list: name_one,name_two,name_three. The databases will be used in descending order of priority. Meaning: at any given time all jobs of the first named database will have to be completed before any job of the second one will be considered by the Puffin instance.

resources

cores :: int: The number of threads the executed jobs are allowed to use. Note that only jobs that are below this value in their requirements will be accepted by the Puffin instance.
memory :: float: The total amount of memory the Puffin and its jobs are allowed to use. Given in GB. Note that only jobs that are below this value in their requirements will be accepted by the Puffin instance.
disk :: float: The total amount of disk space the Puffin and its jobs are allowed to use. Given in GB. Note that only jobs that are below this value in their requirements will be accepted by the Puffin instance.
ssh_keys :: List[str]: Any SSH keys needed by the Puffin in order to connect to the database or to bootstrap programs.

programs

The specific details for each program are given in their respective documentation. However, common options are:

available :: bool: The switch whether the program shall be available to Puffin. Any programs set to be unavailable will not be bootstrapped.
source :: str: The link to the source location of the given program, usually a https link to a git repository
root :: str: The folder at which the program is already installed at. This will request a non source based bootstrapping of the program.
version :: str: The version of the program to use. Can also be a git tag or commit SHA.

The default version of a configuration file can be generated using python3 -m puffin configure (if no environment variables are set).