API documentation

remote_run.run module

Set of functions to allow execution of python functions on a remote host.

class remote_run.run.Executor(project: Project = <factory>, environment: list[Environment] = None, scheduling_engine: SchedulingEngine = None, machine: Machine = <factory>)

Bases: Executor

Class to define an executor.

environment: list[Environment] = None: List of environments to set up before running the project.

machine: Machine: Machine to run the function on.

project: Project: Representation of the local project.

scheduling_engine: SchedulingEngine = None: Scheduling engine to submit the function to.

submit(function: Callable, *args, **kwargs)

Run function on machine.

All arguments need to be pickable by dill.

Parameters:

function – Function to run on machine.
*args – Positional arguments to pass to function.
**kwargs – Keyword arguments to pass to pass to function.

Returns:

If no scheduling engine is specified it will return the return argument of the function, otherwise a job object will be returned.

Return type:

Any

remote_run.run.remote(function): Decorate a function to be run remotely.

remote_run.run.run_remote_python(job: Job): Run the Python job on the machine.

remote_run.run.setup_remote_python(function: Callable, *args, job: Job, environment: list[Environment] = None, python_executable: str = 'python3', **kwargs) → str: Setup a remote running python script.

remote_run.environment module

Environment creation utilities.

class remote_run.environment.Environment

Bases: ABC

Abstract class to create environments from a script.

script: str

class remote_run.environment.FileEnvironment(source: Path, target: Path, bytes_source: bool = False)

Bases: Environment

Environment to create a file at target with content from source.

bytes_source: bool = False

property script: str: Script for file environment.

source: Path

target: Path

class remote_run.environment.GuixEnvironment(manifest: str, channels: str = '(list\n (channel\n (name \'guix)\n (url "https://git.savannah.gnu.org/git/guix.git")\n (branch "master")\n (introduction\n (make-channel-introduction\n "9edb3f66fd807b096b48283debdcddccfea34bad"\n (openpgp-fingerprint\n "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA"))))\n (channel\n (name \'python-remote-run)\n (url "https://gitlab.ost.ch/scl/remote-run.git")\n (branch "main")))\n')

Bases: Environment

Prepare guix environment.

channels: str = '(list\n (channel\n (name \'guix)\n (url "https://git.savannah.gnu.org/git/guix.git")\n (branch "master")\n (introduction\n (make-channel-introduction\n "9edb3f66fd807b096b48283debdcddccfea34bad"\n (openpgp-fingerprint\n "BBB0 2DDF 2CEA F6A8 0D1D E643 A2A0 6DF2 A33A 54FA"))))\n (channel\n (name \'python-remote-run)\n (url "https://gitlab.ost.ch/scl/remote-run.git")\n (branch "main")))\n'

manifest: str

property script: str: Script for guix environment.

class remote_run.environment.ModuleEnvironment(modules: list[str])

Bases: Environment

Prepare environment with environment modules.

modules: list[str]

property script: str: Script for module environment.

class remote_run.environment.PythonEnvironment(code: str, python_executable: str = 'python3')

Bases: Environment

Prepare environment to execute Python code.

code: str

python_executable: str = 'python3'

property script: str: Script for Python environment.

class remote_run.environment.VenvEnvironment(packages: list[str | Path], directory: Path = PosixPath('.venv'), python_executable: str = 'python3')

Bases: Environment

Prepare environment with Python packages.

directory: Path = PosixPath('.venv')

packages: list[str | Path]

python_executable: str = 'python3'

property script: str: Script for venv environment.

remote_run.environment.assemble_environment_script(environments: list[Environment], shebang: str = '#!/usr/bin/env bash') → str: Assemble environment objects into a script.

remote_run.environment.heredoc_script(command: str, here_document: str, delimiter: str = 'EOF', with_quoting: bool = False, append_to: Path = None, output_to: Path = None) → str: Create a heredoc script.

remote_run.machine module

Module to define machines to execute code on.

class remote_run.machine.Identifier(project: Path, files: list[Path], func: Callable)

Bases: object

Class identifying a project and a function.

hash: str

name: str

class remote_run.machine.LocalMachine(working_directory: Path = PosixPath('/tmp'), python_executable: str = 'python3')

Bases: Machine

Class representing the local machine.

Function will be executed in the project directory.

can_execute(identifier: Identifier) → bool: Check if execution can proceed.

download_file(file: Path, destination: Path) → Path: Download a remote absolute file to local file at destination.

exists(path: Path) → bool: Check if path exists.

read_bytes(file: Path) → bytes: Read bytes from a file on the machine.

read_python_object(file: Path): Read a python object from file serialized with dill.

read_text(file: Path) → str: Read text from a file on the machine.

realpath(path: Path) → Path: Query realpath of path.

run(command: list[str]): Run a command on the machine.

upload_file(file: Path, destination: Path): Upload absolute file to destination on machine.

upload_paths(paths: list[Path], destination: Path): Upload absolute paths to destination on machine.

class remote_run.machine.Machine(working_directory: Path = PosixPath('/tmp'), python_executable: str = 'python3')

Bases: ABC

Abstract class represeting a machine.

can_execute(identifier: Identifier) → bool: Check if execution can proceed.

download_file(file: Path, destination: Path) → Path: Download an absolute file to destination.

exists(path: Path) → bool: Check if path exists.

python_executable: str = 'python3'

read_bytes(file: Path) → bytes: Read bytes from a file on the machine.

read_python_object(file: Path): Read a python object from file serialized with dill.

read_text(file: Path) → str: Read text from a file on the machine.

realpath(path: Path) → Path: Query realpath of path.

run(command: str): Run a command on the machine.

upload_file(file: Path, destination: Path): Upload absolute file to destination on machine.

upload_paths(paths: list[Path], destination: Path): Upload absolute paths to destination on machine.

working_directory: Path = PosixPath('/tmp')

class remote_run.machine.SshMachine(host: str, working_directory: Path = PosixPath('/tmp'), local_working_directory: Path = PosixPath('/tmp'), python_executable: str = 'python3')

Bases: Machine

Class representing a machine accessible over ssh.

This requires the setup of a key value pair for passwordless login.

can_execute(identifier: Identifier) → bool: Check if execution can proceed.

download_file(file: Path, destination: Path) → Path: Download a remote absolute file to local file at destination.

exists(path: Path) → bool: Check if path exists on host.

host: str

local_working_directory: Path = PosixPath('/tmp')

python_executable: str = 'python3'

python_version()

Query python version on remote host.

Returns a list with major, minor and patch level.

read_bytes(file: Path) → bytes: Read text from a file on the machine.

read_python_object(file: Path): Read a python object from file serialized with dill.

read_text(file: Path) → str: Read text from a file on the machine.

realpath(path: Path) → Path: Query realpath of path on host.

run(command: list[str]): Run a command on the machine.

upload_file(file: Path, destination: Path): Upload absolute file to destination on machine.

upload_paths(paths: list[Path], destination: Path): Upload absolute paths to destination on machine.

working_directory: Path = PosixPath('/tmp')

remote_run.project module

Module to define projects.

class remote_run.project.GitProject

Bases: Project

Class representing a git project.

class remote_run.project.Project(location: Path, files: list[Path])

Bases: object

Class representing a generic project.

files: list[Path]: A list of absolute files belonging to the project.

location: Path: Absolute path of project location.

class remote_run.project.PyProject(location: Path = None, files: list[Path] = None)

Bases: Project

Class representing a pyproject.toml project.

wheel_file: Path = None

remote_run.project.validate_project(project: Project)

Project validation function.

Example

>>> validate_project(Project(location=Path("~"), files=(Path("a_file"), )))

remote_run.scheduling_engine module

Module to define scheduling engines.

class remote_run.scheduling_engine.SchedulingEngine

Bases: ABC

Abstract class representing a scheduling engine.

is_async: bool = True

class remote_run.scheduling_engine.SlurmSchedulingEngine(is_async: bool = True, **kwargs: Mapping[str, Any])

Bases: SchedulingEngine

Class representing the slurm scheduling engine.

All keyword arguments will be passed directly to slurm.

cancel(machine: Machine, job_id: int): Cancel job with job_id in scheduling_engine.

check_command(job_id): Command to run if job has finished.

schedule(command: str): Schedule the slurm scheduling engine.

script(command: str, result_file: Path) → str: Create a slurm script.

remote_run.job module

Module implementing a ob class as a future.

class remote_run.job.Job(location: Path, machine: Machine, scheduling_engine: SchedulingEngine, id: int = None, is_scheduler_job: bool = False, _run_file_name: str = 'run.sh', _status_file_name: str = 'status.txt', _result_file_name: str = 'result.pkl', _exception_file_name: str = 'exception.pkl', _session_file_name: str = 'session.pkl', _stdout_file_name: str = 'stdout.txt', _stderr_file_name: str = 'stderr.txt', _id_file_name: str = 'id.txt', _scheduler_prefix: str = 'scheduler')

Bases: Future

An object representing a future result from an execution.

cancel()

Cancel the execution of the operation.

Returns True if the future was cancelled, False otherwise. A future cannot be cancelled if it has already completed.

cancelled() → bool: Check if execution was cancelled.

done(): Return True if the future was cancelled or finished executing.

exception(timeout=None)

Return the exception raised by the call that the job represents.

Parameters:

timeout – The number of seconds to wait for the exception if the job isn’t done. If None, then there is no limit on the wait time.

Returns:

The exception raised by the call that the job represents or None if the call completed without raising.

Raises:

CancelledError – If the job was cancelled.
TimeoutError – If the job didn’t finish executing before the given timeout.

property exception_file: File where job exception is written.

exists()

Check if job already exists.

If the job exists, retrieve the job ID and set it.

get_result(destination: Path = None): Read the result of the job and download it when of type Path.

id: int = None

property id_file: File where job id is written.

is_scheduler_job: bool = False

location: Path

machine: Machine

read_exception() → Exception: Read the exception of the job.

read_id() → int: Read the ID of the job if it has any.

read_result() → Any: Read the result of the job.

read_status() → Status: Read the status of the job.

result(timeout=None) → Any

Return the result of the call that the job represents.

Parameters:

timeout – The number of seconds to wait for the result if the job isn’t done. If None, then there is no limit on the wait time.

Returns:

The result of the call that the job represents.

Raises:

CancelledError – If the job was cancelled.
TimeoutError – If the job didn’t finish executing before the given timeout.
Exception – If the call raised then that exception will be raised.

property result_file: File where job result is written.

property run_file: File where job is run from.

running() → bool: Check if execution is still running.

scheduling_engine: SchedulingEngine

property session_file: File where job session is written.

property status_file: File where job status is written.

property stderr_file: File where job stderr is written.

property stdout_file: File where job stdout is written.

class remote_run.job.Status(*values)

Bases: StrEnum

Status of the execution.

CANCELLED = 'cancelled'

FAILED = 'failed'

FINISHED = 'finished'

PENDING = 'pending'

RUNNING = 'running'

remote_run.utils module

Utility functions.

class remote_run.utils.Tee(*files)

Bases: object

Class to redirect stdout and stderr.

flush()

write(data)

remote_run.utils.build_python_wheel(pyproject_toml_file: Path, output_directory: Path = None, python_executable: str = 'python3') → Path: Build a wheel from a pyproject.toml file and return path to wheel.

remote_run.utils.executable_script(path: Path, content: str): Create a script file and make it executable.

remote_run.utils.file_bytes_repr(file_path: Path) → bytes: Return the bytes representation of a file.

remote_run.utils.hash_list_of_files(files: List[Path]): Create a hash for a list of files.

remote_run.utils.import_module_from_file(file: Path): Import module from file.

remote_run.utils.list_all_files_not_git_ignored(project_path: Path) → List[Path]: List all file in the project which are not ignored by git.

remote_run.utils.log_subprocess_run(logger: Logger, out: CompletedProcess): Log stdout of a subprocess.

remote_run.utils.only_files(local_paths: list[Path]) → list[Path]: Make a list of files from paths.

remote_run.utils.patch_path(path: Path, destination: Path = PosixPath('.')) → Path: Patch path to point to destination.

remote_run.utils.patch_paths(destination: Path, function_args, function_kwargs): Patch path of function args and kwargs to point to destination.

remote_run.utils.python_script_from_function(function: Callable, function_source_file: Path, function_args, function_kwargs, function_result_file: Path, status_file: Path, exception_file: Path) → str: Create a python script from a function.

remote_run.utils.query_git_project_path() → Path: Query the project path of a git project.

remote_run.utils.query_pyproject_path() → Path: Query the project path of a pyproject.toml project.

remote_run.utils.redirect_output(stdout_file: Path, stderr_file: Path): Helper function to redirect stdout and stderr to file and console.

remote_run.utils.redirect_stdout_stderr_to_file_and_console(stdout_file: Path, stderr_file: Path): Redirect stdout and stderr to file and console.

remote_run.utils.rsync_file(file: Path, destination: Path): Copy absolute file to destination with rsync.

remote_run.utils.rsync_files(files, location, target): Copy absolute files path from location to target with rsync.

remote_run.utils.rsync_paths(paths: list[Path], destination: Path): Copy absolute paths to destination with rsync.

remote_run.utils.setup_pyproject(): Setup a pyproject and compile the corresponding wheel.