pysqa.queueadapter.QueueAdapter

class pysqa.queueadapter.QueueAdapter(directory: str | None = None, queue_type: str | None = None, execute_command: ~typing.Callable = <function execute_command>)

Bases: QueueAdapterAbstractClass

The goal of the QueueAdapter class is to make submitting to a queue system as easy as starting another sub process locally.

Parameters:

directory (str) – directory containing the queue.yaml files as well as corresponding jinja2 templates for the individual queues.

.. attribute:: config

QueueAdapter configuration read from the queue.yaml file.

.. attribute:: queue_list

List of available queues

.. attribute:: queue_view

Pandas DataFrame representation of the available queues, read from queue.yaml.

.. attribute:: queues

Queues available for auto completion QueueAdapter().queues.<queue name> returns the queue name.

__init__(directory: str | None = None, queue_type: str | None = None, execute_command: ~typing.Callable = <function execute_command>)

Initialize the QueueAdapter.

Parameters:

• directory (str) – Directory containing the queue.yaml files and corresponding templates.

• execute_command (Callable) – Function to execute commands.

Methods

__init__([directory, queue_type, ...])	Initialize the QueueAdapter.
check_queue_parameters(queue[, cores, ...])	Check the parameters of a queue.
convert_path_to_remote(path)	Convert a local path to a remote path.
delete_job(process_id)	Delete a job.
enable_reservation(process_id)	Enable reservation for a process.
get_job_from_remote(working_directory)	Get the results of the calculation - this is necessary when the calculation was executed on a remote host.
get_queue_status([user])	Get the status of the queue.
get_status_of_job(process_id)	Get the status of a job.
get_status_of_jobs(process_id_lst)	Get the status of multiple jobs.
get_status_of_my_jobs()	Get the status of the user's jobs.
list_clusters()	List available computing clusters for remote submission
submit_job([queue, job_name, ...])	Submits command to the given queue.
switch_cluster(cluster_name)	Switch to a different computing cluster
transfer_file_to_remote(file[, ...])	Transfer file from remote host to local host.

Attributes

config	Get the QueueAdapter configuration.
queue_list	Get the list of available queues.
queue_view	Get the Pandas DataFrame representation of the available queues.
queues	Get the list of available queues.
remote_flag	Get the value of remote_flag property.
ssh_delete_file_on_remote	Get the value of ssh_delete_file_on_remote property.

check_queue_parameters(queue: str, cores: int = 1, run_time_max: int | None = None, memory_max: int | None = None, active_queue: dict | None = None) → tuple[float | int | None, float | int | None, float | int | str | None]

Check the parameters of a queue.

Parameters:

• queue (str/None) – The queue name.

• cores (int) – The number of cores.

• run_time_max (int/None) – The maximum runtime.

• memory_max (int/None) – The maximum memory.

• active_queue (dict/None) – The active queue.

Returns:

A list containing the checked parameters [cores, run_time_max, memory_max].

Return type:

List

property config: dict | None

Get the QueueAdapter configuration.

Returns:

The QueueAdapter configuration.

Return type:

dict

convert_path_to_remote(path: str) → str

Convert a local path to a remote path.

Parameters:

path (str) – The local path.

Returns:

The remote path.

Return type:

str

delete_job(process_id: int) → str

Delete a job.

Parameters:

process_id (int) – The process id.

Returns:

The result of deleting the job.

Return type:

str

enable_reservation(process_id: int) → str

Enable reservation for a process.

Parameters:

process_id (int) – The process id.

Returns:

The result of enabling reservation.

Return type:

str

get_job_from_remote(working_directory: str)

Get the results of the calculation - this is necessary when the calculation was executed on a remote host.

Parameters:

working_directory (str) – The working directory.

get_queue_status(user: str | None = None) → DataFrame

Get the status of the queue.

Parameters:

user (str/None) – The user.

Returns:

The status of the queue.

Return type:

pandas.DataFrame

get_status_of_job(process_id: int) → str

Get the status of a job.

Parameters:

process_id – The process id.

Returns:

The status of the job. Possible values are [‘running’, ‘pending’, ‘error’].

Return type:

str

get_status_of_jobs(process_id_lst: list[int]) → list[str]

Get the status of multiple jobs.

Parameters:

process_id_lst – The list of process ids.

Returns:

The status of the jobs. Possible values are [‘running’, ‘pending’, ‘error’, …].

Return type:

List[str]

get_status_of_my_jobs() → DataFrame

Get the status of the user’s jobs.

Returns:

The status of the user’s jobs.

Return type:

pandas.DataFrame

list_clusters() → list[str]

List available computing clusters for remote submission

Returns:

List of computing clusters

property queue_list: list[str] | None

Get the list of available queues.

Returns:

The list of available queues.

Return type:

List[str]

property queue_view: DataFrame | None

Get the Pandas DataFrame representation of the available queues.

Returns:

The Pandas DataFrame representation of the available queues.

Return type:

pandas.DataFrame

property queues: Queues | None

Get the list of available queues.

Returns:

The list of available queues.

Return type:

List[str]

property remote_flag: bool

Get the value of remote_flag property.

Returns:

The value of remote_flag property.

Return type:

bool

property ssh_delete_file_on_remote: bool

Get the value of ssh_delete_file_on_remote property.

Returns:

The value of ssh_delete_file_on_remote property.

Return type:

bool

submit_job(queue: str | None = None, job_name: str | None = None, working_directory: str | None = None, cores: int | None = None, memory_max: int | str | None = None, run_time_max: int | None = None, dependency_list: list[int] | None = None, command: str | None = None, submission_template: str | Template | None = None, **kwargs) → int

Submits command to the given queue.

Parameters:

• queue (str/None) – Name of the queue to submit to, must be one of the names configured for this adapter (optional)

• job_name (str/None) – Name of the job for the underlying queuing system (optional)

• working_directory (str/None) – Directory to run the job in (optional)

• cores (int/None) – Number of hardware threads requested (optional)

• memory_max (int/None) – Amount of memory requested per node in GB (optional)

• run_time_max (int/None) – Maximum runtime in seconds (optional)

• dependency_list(list[str]/None – Job ids of jobs to be completed before starting (optional)

• command (str/None) – shell command to run in the job

• **kwargs – allows writing additional parameters to the job submission script if they are available in the corresponding template.

Returns:

Job id received from the queuing system for the job which was submitted

Return type:

int

switch_cluster(cluster_name: str)

Switch to a different computing cluster

Parameters:

cluster_name (str) – name of the computing cluster

transfer_file_to_remote(file: str, transfer_back: bool = False, delete_file_on_remote: bool = False)

Transfer file from remote host to local host.

Parameters:

• file (str) – The file to transfer.

• transfer_back (bool) – Whether to transfer the file back.

• delete_file_on_remote (bool) – Whether to delete the file on the remote host.