Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,14 @@

* Apps expose `NetworkVolume` and `GlobalVolume` resources with explicit path-keyed mounts, lazy create-if-missing provisioning, and execution-bound filesystem access. Global volumes use GraphQL for lookup and creation.

### Bug Fixes

* Task calls support indefinite execution with bounded interruption cleanup. Failed deletion retains the pod ID for recovery.
* Network volume creation intersects catalog storage support with every sharing resource's hardware stock and datacenter constraints. Unsupported explicit pins fail before creation; existing volumes retain their authoritative IDs and datacenters.
* Global volume attachments require GPU compute for tasks and endpoints.
* Deployment artifacts apply project ignore rules and mandatory secret-file exclusions while preserving vendored dependency assets.
* Synchronous calls wait across polling timeouts on Python 3.10 and propagate operation timeouts.

## [1.12.0](https://github.com/runpod/runpod-python/compare/v1.11.0...v1.12.0) (2026-08-10)


Expand Down
24 changes: 24 additions & 0 deletions docs/cli/references/projects.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,27 @@ You may need to update the default configuration within `runpod.toml` to match y
## Ignore Files and Folders

Create a `.runpodignore` file in the root of your project to ignore files and folders from being uploaded to the Runpod platform, the same file will also be used to ignore files that should not trigger an API server reload.

### Flash deployment artifacts

`rp flash deploy` and `rp flash deploy --build-only` apply Git-style ignore
patterns in this order, with later rules taking precedence:

1. Default exclusions for local environments, caches, tests, and build archives.
2. Project `.gitignore` files, with deeper files overriding parents in their subtree.
3. The project-root `.runpodignore`.

Ancestor and global Git ignores are not read. Negation (`!`) can re-include
ordinary files, but excluded parent directories must also be re-included.

Negation cannot include `.git`, `.runpod`, `.flash`, the root `env/` or
`runpod_manifest.json`, or credential-like source paths such as `.env` variants,
PEM/key files, private SSH keys, cloud credential directories, and credentials,
secrets, or service-account files. These are filename safeguards, not secret
scanning; review the build-only artifact and supply credentials through worker
environment variables or a secret store.

The manifest and vendored `env/` are added separately. Source ignore rules do
not strip dependency CA bundles. Symlinks are omitted from source and
dependencies; the output artifact and dependency build directory are never
copied back into source.
40 changes: 37 additions & 3 deletions examples/apps/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,40 @@ workers pick up the new code automatically. the exhaustive
feature-by-feature suite lives in [`tests/e2e/examples`](../../tests/e2e/examples).

`await task.spawn.aio(...)` returns a task job that owns its pod.
`await job.wait(timeout=...)` terminates that pod when waiting finishes, including
timeout, task failure, or cancellation. use `await job.cancel()` to abandon a
spawned task explicitly. a failed termination retains `job.pod_id` for cleanup.
`await job.wait(timeout=...)` attempts to terminate that pod when waiting finishes,
including timeout, task failure, or cancellation. use `await job.cancel()` to
abandon a spawned task explicitly. leaving the client without waiting or cancelling
does not cancel intentional detached work.

cancellation waits for bounded cleanup and retries transient deletion failures.
failed deletion retains `job.pod_id`; check the console after a cleanup warning.
active tasks can run indefinitely, including detached work after client exit.
`remote()` and `job.wait()` impose no execution timeout by default. pod deletion
requires a working control-plane API and pod-scoped credentials.

## Storage

`NetworkVolume(name_or_id, size=50, datacenter=None, create=True)` references
datacenter-local storage. New volumes use one catalog capability snapshot per
provisioning run to select a datacenter with network storage support and hardware
stock for every resource sharing the volume, respecting their datacenter pins.
An unsupported explicit volume pin fails before creation instead of relocating;
catalog lookup failures also stop creation. Existing volumes retain their IDs and
datacenters regardless of new-volume eligibility. Their consumers must still be
schedulable in that datacenter. `GlobalVolume(name_or_id, create=True)`
references global storage without a datacenter constraint. Both inherit from
the abstract `Volume` base, resolve by name or ID, and create missing storage
when provisioning remote compute. Set `create=False` to require existing storage.
Global-volume lookup and creation use GraphQL; network volumes use REST.

Declare attachments with `mounts={"/path": volume}`. Tasks support one network
and one global volume at distinct, non-overlapping paths. Queue and API resources
support one volume at `/runpod-volume`. Global volumes require GPU compute for
both tasks and endpoints.

Inside worker code, `volume.path` returns the configured mount path. The runtime
binds declared references and resolved IDs before importing user code. Access
raises if the volume is unmounted or has multiple bindings. Evaluate `.path`
inside remote functions, not during local module discovery. Calling `.local()`
does not mount remote storage on the client machine.

1 change: 1 addition & 0 deletions requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ colorama >= 0.4.6, < 0.4.7
cryptography >= 50.0.1
fastapi[all] >= 0.141.1
paramiko >= 5.0.0
pathspec >= 0.12.1
prettytable >= 3.18.0
psutil >= 7.2.2
py-cpuinfo >= 9.0.0
Expand Down
27 changes: 26 additions & 1 deletion runpod/apps/api.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
absent from rest. management verbs for the wider sdk stay in runpod.api.ctl_commands.
"""

from typing import Any, Dict, List, Optional
from typing import Any, Dict, List, Optional, Set

import aiohttp

Expand Down Expand Up @@ -387,6 +387,31 @@ async def cpu_stock_status(
)
return _stock_in_datacenter(data, data_center_id)

async def network_volume_datacenters(self) -> Set[str]:
"""datacenters supporting any network volume tier chosen by the backend."""
data = await run_rest_request_async(
"GET", "/v2/catalog/datacenters", api_key=self._api_key
)
datacenters = data.get("dataCenters") if isinstance(data, dict) else None
if not isinstance(datacenters, list):
raise QueryError("datacenter catalog is missing dataCenters")
supported = set()
for dc in datacenters:
if (
not isinstance(dc, dict)
or not isinstance(dc.get("id"), str)
or not dc["id"]
or not isinstance(dc.get("networkVolumeTypes"), list)
or any(
not isinstance(tier, str) or not tier
for tier in dc["networkVolumeTypes"]
)
):
raise QueryError("datacenter catalog has invalid networkVolumeTypes")
if dc["networkVolumeTypes"]:
supported.add(dc["id"])
return supported

async def list_global_volumes(self) -> List[Dict[str, Any]]:
data = await self._execute(app_queries.QUERY_GLOBAL_VOLUMES, retry=True)
return data["myself"]["globalStoreBuckets"]
Expand Down
42 changes: 39 additions & 3 deletions runpod/apps/context.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,18 @@
"""execution context detection and the sync/async bridge."""

import asyncio
import concurrent.futures
import logging
import os
import threading
import time
from enum import Enum
from typing import Any, Coroutine

log = logging.getLogger(__name__)
CLEANUP_TIMEOUT = 30.0
BRIDGE_CLEANUP_TIMEOUT = CLEANUP_TIMEOUT + 5.0


class Context(Enum):
"""where the current process is running."""
Expand Down Expand Up @@ -68,16 +75,45 @@ def _ensure_loop(cls) -> asyncio.AbstractEventLoop:
@classmethod
def run(cls, coro: Coroutine[Any, Any, Any]) -> Any:
loop = cls._ensure_loop()
future = asyncio.run_coroutine_threadsafe(coro, loop)
completed = threading.Event()
interrupted = threading.Event()
running = []

async def run():
running.append(asyncio.current_task())
try:
if interrupted.is_set():
coro.close()
raise asyncio.CancelledError()
return await coro
finally:
completed.set()

def cancel():
interrupted.set()
if running:
running[0].cancel()

future = asyncio.run_coroutine_threadsafe(run(), loop)
try:
while True:
try:
return future.result(timeout=0.2)
except TimeoutError:
except concurrent.futures.TimeoutError:
if future.done():
raise
except BaseException:
future.cancel()
loop.call_soon_threadsafe(cancel)
deadline = time.monotonic() + BRIDGE_CLEANUP_TIMEOUT
while not completed.is_set():
try:
remaining = deadline - time.monotonic()
if remaining <= 0:
log.warning("interrupted operation cleanup is still pending")
break
completed.wait(min(remaining, 0.2))
except BaseException:
continue
raise


Expand Down
6 changes: 3 additions & 3 deletions runpod/apps/datacenter.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""datacenter selection for app resources.

only datacenters with storage support and S3 API support are listed"""
network volume creation support is discovered from the datacenter catalog."""

from enum import Enum
from typing import List
Expand Down Expand Up @@ -41,8 +41,8 @@ def all(cls) -> List["DataCenter"]:
return list(cls)


# datacenters with high cpu serverless stock, restricted to the
# storage+S3 set above. cpu5c/cpu5g are only stocked in EU-RO-1.
# datacenters with high cpu serverless stock. storage support is checked
# separately; cpu5c/cpu5g are only stocked in EU-RO-1.
CPU3_DATACENTERS: List[DataCenter] = [
DataCenter.EU_CZ_1,
DataCenter.EU_RO_1,
Expand Down
Loading
Loading