Globus Compute Engine

class globus_compute_endpoint.engines.GlobusComputeEngine(*args, label: str | None = None, max_retries_on_system_failure: int = 0, executor: HighThroughputExecutor | None = None, container_type: Literal['docker', 'singularity', 'apptainer', 'podman', 'custom', None] | None = None, container_uri: str | None = None, container_cmd_options: str | None = None, encrypted: bool = True, strategy: str | None = None, job_status_kwargs: JobStatusPollerKwargs | None = None, run_in_sandbox: bool = False, allowed_serializers: Iterable[type[SerializationStrategy] | str | AllowlistWildcard] | None = None, **kwargs)

GlobusComputeEngine is a wrapper over Parsl’s HighThroughputExecutor

GlobusComputeEngine is a shim over Parsl’s HighThroughputExecutor, almost all of arguments are passed along, unfettered. Consequently, please reference Parsl’s HighThroughputExecutor documentation for a complete list of arguments; we list below only the arguments specific to the GlobusComputeEngine.

Parameters:

  • label (str | None) – Label used to name engine log directories and batch jobs default: None (in which case, “GlobusComputeEngine” will be set)

  • max_retries_on_system_failure (int) – Set the number of retries for functions that fail due to system failures such as node failure/loss. Since functions can fail after partial runs, consider additional cleanup logic before enabling this functionality default: 0

  • strategy (str | None) – Specify which scaling strategy to use; this is eventually given to Parsl’s Strategy. [Deprecated; use job_status_kwargs]

  • job_status_kwargs (dict | None) – Keyword arguments to be passed through to Parsl’s JobStatusPoller class that drives strategy to do auto-scaling.

  • encrypted (bool) – Flag to enable/disable encryption (CurveZMQ). Default is True.

  • run_in_sandbox (bool) – Functions will run in a sandbox directory under the working_dir if this option is enabled. Default: False

  • allowed_serializers (DeserializerAllowlist | None) – If specified, any submissions that were not serialized using the strategies given in this list will cause an error to be raised. Passed to the engine’s ComputeSerializer as allowed_deserializers.

assert_ha_compliant()

Signal to endpoint initialization code whether or not this engine, in its current configuration, complies with Globus’s High Assurance policies. This must be evaluated on a case-by-case basis. Some example criteria:

  • All manager-worker communication happens within one host machine

  • Network traffic is encrypted

containerized_launch_cmd() → str

Recompose executor’s launch_cmd to launch with containers

Return type:

str launch_cmd

get_connected_managers() → List[Dict[str, Any]]
Return type:

List of dicts containing info for all connected managers

get_connected_managers_packages() → Dict[str, Dict[str, str]]
Returns:

  • Dict mapping each connected manager ID to a dict of installed packages

  • and versions

get_total_managers(managers: List[Dict[str, Any]]) → int
Parameters:

managers (list[dict[str, Any]]) – List of dicts containing info for all connected managers

Return type:

Total number of connected managers

get_total_active_managers(managers: List[Dict[str, Any]]) → int
Parameters:

managers (list[dict[str, Any]]) – List of dicts containing info for all connected managers

Return type:

Number of managers that are accepting new tasks

get_outstanding_breakdown(managers: List[Dict[str, Any]] | None = None) → List[Tuple[str, int, bool]]
Parameters:

managers (list[dict[str, Any]] | None) – List of dicts containing info for all connected managers

Return type:

List of tuples of the form (component, # of tasks on component, active?)

get_total_tasks_outstanding() → dict
Returns:

Dict of type {str_task_type

Return type:

count_tasks}

get_total_tasks_pending(managers: List[Dict[str, Any]]) → int
Parameters:

managers (list[dict[str, Any]]) – List of dicts containing info for all connected managers

Return type:

Total number of tasks that are queued in the executor’s interchange

get_total_live_workers(managers: List[Dict[str, Any]] | None = None) → int
Parameters:

managers (list[dict[str, Any]]) – List of dicts containing info for all connected managers

Return type:

Total number of workers that are actively running tasks

get_total_idle_workers(managers: List[Dict[str, Any]]) → int
Parameters:

managers (list[dict[str, Any]]) – List of dicts containing info for all connected managers

Return type:

Total number of workers that are not actively running tasks

property scaling_enabled: bool

Indicates whether scaling is possible

get_status_report() → EPStatusReport
Return type:

Object containing info on the current status of the endpoint