API documentation

remote_run.run module

Set of functions to allow execution of python functions on a remote host.

class remote_run.run.Executor(project: Project = None, scheduling_engine: SchedulingEngine = None, machine: Machine = LocalMachine(environment=(), working_directory=PosixPath('/tmp'), python_executable='python3'), record_session: bool = True, name_keyword: str = None)

Bases: Executor

Class to define an executor.

machine: Machine: Machine to run the function on.

name_keyword: str = None

Function keyword holding the submit name.

If set to None, the function argument will be treated as any normal argument.

project: Project: Representation of the local project.

record_session: bool = True: Record the current session and use it in the function execution.

scheduling_engine: SchedulingEngine = None: Scheduling engine to submit the function to.

submit(function: Callable, *args, **kwargs)

Run function on machine.

All arguments need to be pickable by dill. Arguments of type Path will be uploaded to the machine. If you need the path to refer to files on the machine, wrap it in MachinePath. This will prevent it from being uploaded.

Parameters:

function – Function to run on machine.
*args – Positional arguments to pass to function.
**kwargs – Keyword arguments to pass to pass to function.

Returns:

If no scheduling engine is specified it will return the return argument of the function, otherwise a job object will be returned.

Return type:

Any

remote_run.run.filter_paths(function_args, function_kwargs): Filter paths for upload.

remote_run.run.map_paths(local_paths: list[Path], destination: Path, root: Path = None)

Map local paths to destination.

Paths relative to root will be mapped relative to destination. Paths outside root will be mapped to destination.

remote_run.run.patch_paths(destination: Path, root: Path, function_args, function_kwargs): Patch path of function args and kwargs to point to destination.

remote_run.run.remote(function): Decorate a function to be run remotely.

remote_run.run.run_remote_python(job: Job): Run the Python job on the machine.

remote_run.run.setup_remote_python(function: Callable, *args, job: Job, environment: tuple[Environment] = (), with_session: bool = True, **kwargs) → str: Setup a remote running python script.

remote_run.environment module

Environment creation utilities.

class remote_run.environment.Environment

Bases: ABC

Abstract class to create environments from a script.

class remote_run.environment.FileEnvironment(source: Path, target: Path, bytes_source: bool = False)

Bases: Environment

Environment to create a file at target with content from source.

bytes_source: bool = False

property script: str: Script for file environment.

source: Path

target: Path

class remote_run.environment.GuixEnvironment(manifest: str, channels: str = '(list\n (channel\n (name \'guix)\n (url "https://git.savannah.gnu.org/git/guix.git")\n (branch "master")\n (introduction\n (make-channel-introduction\n "9edb3f66fd807b096b48283debdcddccfea34bad"\n (openpgp-fingerprint\n "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA"))))\n (channel\n (name \'python-remote-run)\n (url "https://gitlab.ost.ch/scl/remote-run.git")\n (branch "main")))\n')

Bases: Environment

Prepare guix environment.

channels: str = '(list\n (channel\n (name \'guix)\n (url "https://git.savannah.gnu.org/git/guix.git")\n (branch "master")\n (introduction\n (make-channel-introduction\n "9edb3f66fd807b096b48283debdcddccfea34bad"\n (openpgp-fingerprint\n "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA"))))\n (channel\n (name \'python-remote-run)\n (url "https://gitlab.ost.ch/scl/remote-run.git")\n (branch "main")))\n'

manifest: str

property script: str: Script for guix environment.

class remote_run.environment.ModuleEnvironment(modules: list[str])

Bases: Environment

Prepare environment with environment modules.

modules: list[str]

property script: str: Script for module environment.

class remote_run.environment.PythonEnvironment(code: str, python_executable: str = 'python3')

Bases: Environment

Prepare environment to execute Python code.

code: str

python_executable: str = 'python3'

property script: str: Script for Python environment.

class remote_run.environment.VariableEnvironment(variables: dict[str, str])

Bases: Environment

Environment to set variables.

property script: str: Script for file environment.

variables: dict[str, str]

class remote_run.environment.VenvEnvironment(packages: list[str | Path], directory: Path = PosixPath('.venv'), python_executable: str = 'python3')

Bases: Environment

Prepare environment with Python packages.

directory: Path = PosixPath('.venv')

packages: list[str | Path]

python_executable: str = 'python3'

property script: str: Script for venv environment.

remote_run.environment.assemble_environment_script(environments: tuple[Environment], project_root: Path, shebang: str = '#!/usr/bin/env bash', stdout_file: Path = PosixPath('stdout.txt'), stderr_file: Path = PosixPath('stderr.txt'), status_file: Path = PosixPath('status.txt')) → str: Assemble environment objects into a script.

remote_run.environment.heredoc_script(command: str, here_document: str, delimiter: str = 'EOF', with_quoting: bool = False, append_to: Path = None, output_to: Path = None) → str: Create a heredoc script.

remote_run.machine module

Module to define machines to execute code on.

class remote_run.machine.Identifier(project: Path, files: list[Path], function: Callable, *function_args, **function_kwargs)

Bases: object

Class identifying a project and a function.

hash: str

name: str

class remote_run.machine.LocalMachine(environment: tuple[Environment] = (), working_directory: Path = PosixPath('/tmp'), python_executable: str = 'python3')

Bases: Machine

Class representing the local machine.

Function will be executed in the project directory.

can_execute(identifier: Identifier) → bool: Check if execution can proceed.

download_file(file: Path, destination: Path) → Path: Download a remote absolute file to local file at destination.

exists(path: Path) → bool: Check if path exists.

is_symlink(path: Path): Return True if the path points to a symbolic link.

read_bytes(file: Path) → bytes: Read bytes from a file on the machine.

read_python_object(file: Path): Read a python object from file serialized with dill.

read_text(file: Path) → str: Read text from a file on the machine.

realpath(path: Path) → Path: Query realpath of path.

run(command: list[str]): Run a command on the machine.

symlink(link: Path, target: Path): Create a symlink for link pointing to target.

unlink(path: Path): Remove this file or symbolic link.

upload_path(path: Path, destination: Path): Upload absolute file to destination on machine.

upload_paths(paths: dict[Path, Path]): Upload absolute paths to absolute paths on machine.

write_text(file: Path, text: str): Write text to a file on the machine.

class remote_run.machine.Machine(environment: tuple[Environment] = (), working_directory: Path = PosixPath('/tmp'), python_executable: str = 'python3')

Bases: ABC

Abstract class represeting a machine.

can_execute(identifier: Identifier) → bool: Check if execution can proceed.

download_file(file: Path, destination: Path) → Path: Download an absolute file to destination.

environment: tuple[Environment] = (): List of environments to set up before running on the machine.

exists(path: Path) → bool: Check if path exists.

is_symlink(path: Path): Return True if the path points to a symbolic link.

python_executable: str = 'python3'

read_bytes(file: Path) → bytes: Read bytes from a file on the machine.

read_python_object(file: Path): Read a python object from file serialized with dill.

read_text(file: Path) → str: Read text from a file on the machine.

realpath(path: Path) → Path: Query realpath of path.

run(command: str): Run a command on the machine.

symlink(link: Path, target: Path): Create a symlink for link pointing to target.

unlink(path: Path): Remove this file or symbolic link.

upload_path(path: Path, destination: Path): Upload absolute path to destination on machine.

upload_paths(paths: dict[Path, Path]): Upload absolute paths to absolute paths on machine.

working_directory: Path = PosixPath('/tmp')

write_text(file: Path, text: str): Write text to a file on the machine.

class remote_run.machine.SshMachine(host: str, environment: Environment = (), working_directory: Path = PosixPath('/tmp'), local_working_directory: Path = PosixPath('/tmp'), python_executable: str = 'python3')

Bases: Machine

Class representing a machine accessible over ssh.

This requires the setup of a key value pair for passwordless login.

can_execute(identifier: Identifier) → bool: Check if execution can proceed.

download_file(file: Path, destination: Path) → Path: Download a remote absolute file to local file at destination.

environment: Environment = (): List of environments to set up before running on the machine.

exists(path: Path) → bool: Check if path exists on host.

host: str

is_symlink(path: Path): Return True if the path points to a symbolic link.

local_working_directory: Path = PosixPath('/tmp')

python_executable: str = 'python3'

python_version()

Query python version on remote host.

Returns a list with major, minor and patch level.

read_bytes(file: Path) → bytes: Read text from a file on the machine.

read_python_object(file: Path): Read a python object from file serialized with dill.

read_text(file: Path) → str: Read text from a file on the machine.

realpath(path: Path) → Path: Query realpath of path on host.

run(command: list[str]): Run a command on the machine.

symlink(link: Path, target: Path): Create a symlink for link pointing to target.

unlink(path: Path): Remove this file or symbolic link.

upload_path(path: Path, destination: Path): Upload absolute file to destination on machine.

upload_paths(paths: dict[Path, Path]): Upload absolute paths to absolute paths on machine.

working_directory: Path = PosixPath('/tmp')

write_text(file: Path, text: str): Write text to a file on the machine.

remote_run.project module

Module to define projects.

class remote_run.project.GitProject(location: Path = None, files: list[Path] = None, name: str = None)

Bases: Project

Class representing a git project.

class remote_run.project.Project(location: Path, files: list[Path], name: str)

Bases: object

Class representing a generic project.

files: list[Path]: A list of absolute files belonging to the project.

location: Path: Absolute path of project location.

name: str: Project name.

class remote_run.project.PyProject(location: Path = None, files: list[Path] = None, name: str = None)

Bases: Project

Class representing a pyproject.toml project.

remote_run.project.find_project_root() → Path

Find the absolute path to a projects root folder.

It first checks if the environment variable PROJECT_ROOT is set and if not, searches for a pyproject.toml file and/or the git root folder. If none is set, it returns the current working directory.

remote_run.project.validate_project(project: Project)

Project validation function.

Example

>>> validate_project(Project(location=Path("~"), files=(Path("a_file"), )))

remote_run.scheduling_engine module

Module to define scheduling engines.

class remote_run.scheduling_engine.SchedulingEngine(output_file: Path = None, is_async: bool = True)

Bases: ABC

Abstract class representing a scheduling engine.

is_async: bool = True

output_file: Path = None

class remote_run.scheduling_engine.SlurmSchedulingEngine(output_file: Path = None, is_async: bool = True, **kwargs: Mapping[str, Any])

Bases: SchedulingEngine

Class representing the slurm scheduling engine.

All keyword arguments will be passed directly to slurm.

cancel(machine: Machine, job_id: int): Cancel job with job_id in scheduling_engine.

check_command(job_id): Command to run if job has finished.

schedule(command: str): Schedule the slurm scheduling engine.

script(command: str, result_file: Path) → str: Create a slurm script.

remote_run.job module

Module implementing a ob class as a future.

class remote_run.job.Job(name: str, location: Path, machine: Machine, scheduling_engine: SchedulingEngine, id: int = None, is_scheduler_job: bool = False, _run_file_name: str = 'run.sh', _status_file_name: str = 'status.txt', _result_file_name: str = 'result.pkl', _exception_file_name: str = 'exception.pkl', _session_file_name: str = 'session.pkl', _stdout_file_name: str = 'stdout.txt', _stderr_file_name: str = 'stderr.txt', _id_file_name: str = 'id.txt', _traceback_file_name: str = 'traceback.txt', _scheduler_output_file_name: str = 'scheduler.txt', _scheduler_prefix: str = 'scheduler')

Bases: Future

An object representing a future result from an execution.

cancel()

Cancel the execution of the operation.

Returns True if the future was cancelled, False otherwise. A future cannot be cancelled if it has already completed.

cancelled() → bool: Check if execution was cancelled.

done(): Return True if the future was cancelled or finished executing.

exception(timeout=None)

Return the exception raised by the call that the job represents.

Parameters:

timeout – The number of seconds to wait for the exception if the job isn’t done. If None, then there is no limit on the wait time.

Returns:

The exception raised by the call that the job represents or None if the call completed without raising.

Raises:

CancelledError – If the job was cancelled.
TimeoutError – If the job didn’t finish executing before the given timeout.

property exception_file: File where job exception is written.

exists()

Check if job already exists.

If the job exists, retrieve the job ID and set it.

get_result(destination: Path = None): Read the result of the job and download it when of type Path.

id: int = None

property id_file: File where job id is written.

is_scheduler_job: bool = False

location: Path

machine: Machine

name: str

raise_exception(): Raise the exception thrown by the job.

read_exception() → Exception: Read the exception of the job.

read_id() → int: Read the ID of the job if it has any.

read_result() → Any: Read the result of the job.

read_scheduler_output() → int: Read the scheduler output of the job if it has any.

read_status() → Status: Read the status of the job.

read_traceback() → str: Read the traceback of the job.

result(timeout=None) → Any

Return the result of the call that the job represents.

Parameters:

timeout – The number of seconds to wait for the result if the job isn’t done. If None, then there is no limit on the wait time.

Returns:

The result of the call that the job represents.

Raises:

CancelledError – If the job was cancelled.
TimeoutError – If the job didn’t finish executing before the given timeout.
Exception – If the call raised then that exception will be raised.

property result_file: File where job result is written.

property run_file: File where job is run from.

running() → bool: Check if execution is still running.

property scheduler_output_file: File where scheduler output is written.

scheduling_engine: SchedulingEngine

property session_file: File where job session is written.

property status_file: File where job status is written.

property stderr_file: File where job stderr is written.

property stdout_file: File where job stdout is written.

property traceback_file: File where job exception traceback is written.

exception remote_run.job.RemoteError

Bases: RuntimeError

Error when remote execution failed.

class remote_run.job.Status(*values)

Bases: StrEnum

Status of the execution.

CANCELLED = 'cancelled'

FAILED = 'failed'

FINISHED = 'finished'

PENDING = 'pending'

RUNNING = 'running'

remote_run.utils module

Utility functions.

class remote_run.utils.MachinePath(*args, **kwargs)

Bases: PosixPath

Wrapper for a path which resides on the machine.

remote_run.utils.build_python_wheel(pyproject_toml_file: Path, output_directory: Path = None, python_executable: str = 'python3') → Path: Build a wheel from a pyproject.toml file and return path to wheel.

remote_run.utils.executable_script(path: Path, content: str): Create a script file and make it executable.

remote_run.utils.file_bytes_repr(file_path: Path) → bytes: Return the bytes representation of a file.

remote_run.utils.hash_list_of_files(files: list[Path]): Create a hash for a list of files.

remote_run.utils.hash_python_object(obj, hash): Hash of a Python object.

remote_run.utils.import_module_from_file(file: Path): Import module from file.

remote_run.utils.is_machine_path(path: Path | MachinePath): Check if path is of type MachinePath.

remote_run.utils.list_all_files_not_git_ignored(project_path: Path) → list[Path]: List all file in the project which are not ignored by git.

remote_run.utils.log_subprocess_run(logger: Logger, out: CompletedProcess): Log stdout of a subprocess.

remote_run.utils.only_files(local_paths: list[Path]) → list[Path]: Make a list of files from paths.

remote_run.utils.patch_path(path: Path, root: Path, destination: Path = PosixPath('.')) → Path

Patch path to point to destination.

If the path is below the destination path it will be patched relative to the destination. If it is outside, only the direct path will be patched.

remote_run.utils.query_git_project_path() → Path: Query the project path of a git project.

remote_run.utils.query_pyproject_path() → Path: Query the project path of a pyproject.toml project.

remote_run.utils.rsync_files(files, location, target): Copy absolute files path from location to target with rsync.

remote_run.utils.rsync_path(file: Path, destination: Path, host: str = None): Copy absolute file to destination with rsync.

remote_run.utils.rsync_paths(paths: list[Path], destination: Path): Copy absolute paths to destination with rsync.

remote_run.utils.setup_pyproject(): Setup a pyproject and compile the corresponding wheel.