Ctrl+K

pysqa.queueadapter.QueueAdapter

pysqa.queueadapter.QueueAdapter#

class pysqa.queueadapter.QueueAdapter(directory: str = '~/.queues', execute_command: callable = <function execute_command>)[source]#

Bases: object

The goal of the QueueAdapter class is to make submitting to a queue system as easy as starting another sub process locally.

Parameters:

directory (str) – directory containing the queue.yaml files as well as corresponding jinja2 templates for the individual queues.

.. attribute:: config

QueueAdapter configuration read from the queue.yaml file.

.. attribute:: queue_list

List of available queues

.. attribute:: queue_view

Pandas DataFrame representation of the available queues, read from queue.yaml.

.. attribute:: queues

Queues available for auto completion QueueAdapter().queues.<queue name> returns the queue name.

__init__(directory: str = '~/.queues', execute_command: callable = <function execute_command>)[source]#

Methods

__init__([directory, execute_command])

check_queue_parameters(queue[, cores, ...])

param queue:

convert_path_to_remote(path)

param path:

delete_job(process_id)

param process_id:

enable_reservation(process_id)

param process_id:

get_job_from_remote(working_directory)

Get the results of the calculation - this is necessary when the calculation was executed on a remote host.

get_queue_status([user])

param user:

get_status_of_job(process_id)

param process_id:

get_status_of_jobs(process_id_lst)

param process_id_lst:

get_status_of_my_jobs()

rtype:

pandas.DataFrame

list_clusters()

List available computing clusters for remote submission

submit_job([queue, job_name, ...])

Submits command to the given queue.

switch_cluster(cluster_name)

Switch to a different computing cluster

transfer_file_to_remote(file[, ...])

Transfer file from remote host to local host

Attributes

config

Returns: dict:

queue_list

Returns: list:

queue_view

Returns: pandas.DataFrame:

queues

remote_flag

Returns: bool:

ssh_delete_file_on_remote

check_queue_parameters(queue: str, cores: int = 1, run_time_max: int | None = None, memory_max: int | None = None, active_queue: dict | None = None)[source]#
Parameters:

  • queue (str/None)

  • cores (int)

  • run_time_max (int/None)

  • memory_max (int/None)

  • active_queue (dict)

Returns:

[cores, run_time_max, memory_max]

Return type:

list

property config#

Returns: dict:

convert_path_to_remote(path: str) → str[source]#
Parameters:

path (str)

Return type:

str

delete_job(process_id: int) → str[source]#
Parameters:

process_id (int)

Return type:

str

enable_reservation(process_id: int) → str[source]#
Parameters:

process_id (int)

Return type:

str

get_job_from_remote(working_directory: str)[source]#

Get the results of the calculation - this is necessary when the calculation was executed on a remote host.

Parameters:

working_directory (str)

get_queue_status(user: str | None = None) → DataFrame[source]#
Parameters:

user (str)

Return type:

pandas.DataFrame

get_status_of_job(process_id: int) → str[source]#
Parameters:

process_id

Returns:

[‘running’, ‘pending’, ‘error’]

Return type:

str

get_status_of_jobs(process_id_lst: list[int]) → list[str][source]#
Parameters:

process_id_lst

Returns:

[‘running’, ‘pending’, ‘error’, …]

Return type:

list

get_status_of_my_jobs() → DataFrame[source]#
Return type:

pandas.DataFrame

list_clusters() → list[str][source]#

List available computing clusters for remote submission

Returns:

List of computing clusters

Return type:

list

property queue_list: list[str]#

Returns: list:

property queue_view: DataFrame#

Returns: pandas.DataFrame:

property remote_flag: bool#

Returns: bool:

submit_job(queue: str | None = None, job_name: str | None = None, working_directory: str | None = None, cores: int | None = None, memory_max: int | None = None, run_time_max: int | None = None, dependency_list: list[str] | None = None, command: str | None = None, **kwargs) → int[source]#

Submits command to the given queue.

Parameters:

  • queue (str/None) – Name of the queue to submit to, must be one of the names configured for this adapter (optional)

  • job_name (str/None) – Name of the job for the underlying queuing system (optional)

  • working_directory (str/None) – Directory to run the job in (optional)

  • cores (int/None) – Number of hardware threads requested (optional)

  • memory_max (int/None) – Amount of memory requested per node in GB (optional)

  • run_time_max (int/None) – Maximum runtime in seconds (optional)

  • dependency_list(list[str]/None – Job ids of jobs to be completed before starting (optional)

  • command (str/None) – shell command to run in the job

  • **kwargs – allows writing additional parameters to the job submission script if they are available in the corresponding template.

Returns:

Job id received from the queuing system for the job which was submitted

Return type:

int

switch_cluster(cluster_name: str)[source]#

Switch to a different computing cluster

Parameters:

cluster_name (str) – name of the computing cluster

transfer_file_to_remote(file: str, transfer_back: bool = False, delete_file_on_remote: bool = False)[source]#

Transfer file from remote host to local host

Parameters:

  • file (str)

  • transfer_back (bool)

  • delete_file_on_remote (bool)

Contents