Note
Go to the end to download the full example code.
Use a scheduling engine
from pathlib import Path
from remote_run import (
Executor,
SlurmSchedulingEngine,
GuixEnvironment,
remote,
SshMachine,
)
Define an executor with a scheduling engine. For SLURM you can pass slurm parameters directly here.
executor = Executor(
machine=SshMachine(
host="shpc0003.ost.ch",
working_directory=Path("/cluster/raid/home/reza.housseini"),
),
environment=[
GuixEnvironment(
channels=Path("channels.scm").read_text(),
manifest=Path("manifest.scm").read_text(),
)
],
scheduling_engine=SlurmSchedulingEngine(
job_name="test_remote_run",
mail_type="ALL",
mail_user="reza.housseini@ost.ch",
),
)
decorate your functions you want to run in an executor
@remote
def add(a, b):
return a + b
this call will run on the remote machine specified in the executor but due to the asynchronous nature of scheduling engines this will not return the result, instead you get a future to retrieve the result later.
job = add(1, 2)
now we wait for the remote execution to finish before retrieving the result
result = job.result(timeout=20)
assert result == 3