Capture and replay a native CUDA schedule
Download Python script · Download script with helper and data files (.zip)
Capture and replay a native CUDA schedule
Section titled “Capture and replay a native CUDA schedule”Can recording an execution schedule be kept distinct from advancing the aerosol? This example follows capture, physical replay, and invalidation as separate operations. You will learn why changing structural dependencies requires a new capture, and why a saved graph is not a scientific checkpoint. Read the resident schedule for the underlying inputs.
Record an owned resident schedule, replay it twice, and verify
invalidation and renewal without serializing an opaque CUDA graph.
Run with --cuda to reproduce the saved figures. CUDA is required
for publication; unavailable hardware never becomes a successful GPU run.
Python · section 1
"""Demonstrate qualified native-CUDA resident graph capture.
Lazy qualification runs before CPU-state construction, resident imports, ordevice allocation. This fixed-identity example explicitly captures once,replays twice, invalidates, retires, and renews a graph. CPU and Warp-CPU arenot fallback or emulation paths. It provides no automatic recapture, migration,resize/compaction, hidden transfer or synchronization, retry/rollback,checkpointed or serialized opaque handles, or performance claims."""
from __future__ import annotations
import importlibimport osfrom dataclasses import dataclassfrom types import SimpleNamespacefrom typing import TYPE_CHECKING, Any
import numpy as np
if TYPE_CHECKING: from particula.gas import EnvironmentData, GasData from particula.particles import ParticleData
_FORCE_NO_NATIVE_CAPTURE_ENV = "PARTICULA_EXAMPLE_FORCE_NO_NATIVE_CAPTURE"_UNAVAILABLE_OUTPUT = ( "Canonical path: docs/examples/gpu/resident/gpu_resident_graph_capture.py", "Native CUDA graph capture is unavailable or disabled.", "No CPU or Warp-CPU fallback ran; no fixture, upload, or capture ran.",)
@dataclassclass ExampleRun: """Retain bounded, address-free observations from the walkthrough.
Attributes: output: Deterministic status lines produced by the example. session: Resident session created by the enabled execution path. registry: Resource registry bound to ``session``. guard: Step guard bound to ``session`` and ``registry``. binding: Graph-capture binding used for the execution. captured: Retired graph-capture record from the first capture. renewed_capture: Fresh graph-capture record from renewal. gas_snapshot: Synchronized gas diagnostic observation, if enabled. saturation_snapshot: Synchronized saturation observation, if enabled. replay_count: Number of replays performed before retirement. invalidated: Whether structural drift invalidated the first capture. retired: Whether the first capture was explicitly retired. renewed: Whether a fresh lifecycle was created after retirement. synchronized: Whether the enabled path synchronized before host reads. """
output: list[str] session: Any | None = None registry: Any | None = None guard: Any | None = None binding: Any | None = None captured: Any | None = None renewed_capture: Any | None = None gas_snapshot: np.ndarray | None = None saturation_snapshot: np.ndarray | None = None replay_count: int = 0 invalidated: bool = False retired: bool = False renewed: bool = False synchronized: bool = False
def _disabled_output() -> list[str]: """Return deterministic output for intentionally unavailable execution.
Returns: The fixed status lines used when native CUDA capture is unavailable. """ return list(_UNAVAILABLE_OUTPUT)
def _qualified_native_cuda() -> str | None: """Find the first native CUDA device without setup work.
Capability qualification is deliberately delegated to the canonical graph capture resolver after the opaque device declaration is constructed. Only a missing Warp module or no native CUDA device is treated as unavailable. Other import, loader, or enumeration failures propagate.
Returns: The selected opaque native device name, or ``None`` when the explicit unavailable conditions are met.
Raises: Exception: Propagates unexpected Warp import, device enumeration, or capture-API probe failures. """ if os.getenv(_FORCE_NO_NATIVE_CAPTURE_ENV) == "1": return None try: warp = importlib.import_module("warp") except ModuleNotFoundError as error: if error.name == "warp": return None raise native_devices = warp.get_devices() native = next( ( str(device) for device in native_devices if bool(getattr(device, "is_cuda", False)) ), None, ) if native is None: return None return native
def _load_enabled_runtime() -> SimpleNamespace: """Load concrete resident and graph-capture seams after preflight.
Returns: A namespace containing the lazily loaded runtime modules.
Raises: ImportError: If an enabled-path concrete module cannot be imported. """ names = ( "warp", "particula.execution", "particula.execution.graph_capture", "particula.execution.resident_scheduler", ) loaded = { name.rsplit(".", 1)[-1]: importlib.import_module(name) for name in names } # The maintained resident example owns the identical bounded composition. loaded["resident_example"] = importlib.import_module( "docs.examples.gpu.resident.gpu_resident_multi_timestep" ) return SimpleNamespace(**loaded)
def _load_capability_runtime() -> SimpleNamespace: """Load only modules required for canonical capability qualification.""" names = ( "warp", "particula.execution", "particula.execution.graph_capture", ) loaded = { name.rsplit(".", 1)[-1]: importlib.import_module(name) for name in names } return SimpleNamespace(**loaded)
class _WarpNativeCaptureAdapter: """Expose only the selected Warp native capture vocabulary.
Attributes: _warp: Lazily imported Warp module. _native: Opaque native device selected during preflight. _graph_capture: Concrete graph-capture module providing callables. """
def __init__(self, warp: Any, device: Any, graph_capture: Any) -> None: """Bind Warp and graph-capture objects for one native device.
Args: warp: Lazily imported Warp module. device: Exact CUDA device selected during preflight. graph_capture: Concrete graph-capture module. """ self._warp = warp self._device = device self._graph_capture = graph_capture
def runtime_available(self) -> bool: """Return the completed preflight result without another probe.
Returns: ``True`` because construction occurs only after successful preflight. """ return True
def device_available(self, device: Any) -> bool: """Accept only the preflight-selected native CUDA device.
Args: device: Candidate graph-capture device.
Returns: Whether ``device`` has the exact preflight-selected native name. """ return device is self._device
def capture_api_available(self, device: Any) -> bool: """Accept APIs only for the selected native CUDA device.
Args: device: Candidate graph-capture device.
Returns: Whether ``device`` has the exact preflight-selected native name. """ return device is self._device
def capture_callables(self, device: Any) -> Any: """Return direct Warp capture callables and opaque-handle cleanup.
Args: device: Device whose native name is passed to Warp capture.
Returns: Concrete graph-capture callables for the preflight-selected device. """
def begin() -> None: """Begin capture on the selected opaque native device.""" self._warp.capture_begin( device=device.native, force_module_load=True, )
graphs: dict[object, Any] = {}
def end() -> object: """Own the Warp graph behind an opaque lifecycle token.""" graph = self._warp.capture_end() if graph is None: raise ValueError("Warp capture did not return a graph.") handle = object() graphs[handle] = graph return handle
def launch(handle: object) -> None: """Launch the graph owned by this adapter's live token.""" self._warp.capture_launch(graphs[handle])
def release(handle: object) -> None: """Drop ownership so Warp's graph finalizer frees native storage.
Warp 1.11 graphs use Python lifetime cleanup, not ``destroy()``. Keeping only the opaque token in lifecycle provenance avoids retaining the graph after explicit release. """ if handle not in graphs: raise ValueError("native capture handle is not owned or live.") del graphs[handle]
def abort() -> object: """End incomplete capture and return its cleanup handle.""" return end()
return self._graph_capture.GraphCaptureNativeCallables( begin, end, lambda: None, launch, release, abort, )
def _resolve_exact_capability( graph_capture: Any, device: Any, adapter: Any) -> Any: """Resolve and validate capability for the exact selected device.""" capability = graph_capture.resolve_graph_capture_capability(device, adapter) if capability.device is not device: raise RuntimeError( "graph-capture capability resolved for a different device." ) if ( capability.availability is not graph_capture.GraphCaptureAvailability.AVAILABLE ): raise RuntimeError("selected CUDA device is not capture-capable.") return capability
def _build_cpu_state() -> tuple[ParticleData, GasData, EnvironmentData]: """Build the fixed float64 resident state for the bounded walkthrough.
Returns: Particle, gas, and environment containers for the resident setup. """ return _load_enabled_runtime().resident_example._build_cpu_state()
def _compose_request( runtime: SimpleNamespace, session: Any, registry: Any, guard: Any, gas: GasData, environment: EnvironmentData, initial_total_mass: np.ndarray,) -> tuple[Any, Any, Any]: """Compose the canonical twelve-node request and publish its resources.
Args: runtime: Lazily loaded runtime modules. session: Active resident session. registry: Resource registry bound to ``session``. guard: Closed step guard bound to ``session`` and ``registry``. gas: CPU gas container used to construct the request. environment: CPU environment container used to construct the request. initial_total_mass: CPU-derived particle-plus-gas inventory.
Returns: The resident request and its gas and saturation diagnostic outputs. """ resident_runtime = runtime.resident_example._load_enabled_runtime() communication = SimpleNamespace(**vars(resident_runtime.communication))
def graph_capture_map( _form: Any, transport_mode: Any, edge_capacity: int, source_indices: Any, destination_indices: Any, species_indices: Any, edge_rates: Any, ) -> Any: """Construct the required closed one-dimensional map before publication. """ return resident_runtime.communication.CommunicationMap( resident_runtime.communication.CommunicationMapForm.ONE_DIMENSIONAL, transport_mode, edge_capacity, source_indices, destination_indices, species_indices, edge_rates, )
communication.CommunicationMap = graph_capture_map capture_runtime_values = vars(resident_runtime).copy() capture_runtime_values["communication"] = communication capture_runtime = SimpleNamespace(**capture_runtime_values) request, gas_output, saturation_output = runtime.resident_example._request( capture_runtime, session, registry, guard, 1.0, gas, environment, initial_total_mass, ) return request, gas_output, saturation_output
def _close_enabled_binding( graph_capture: Any | None, binding: Any | None, session: Any | None, registry: Any | None, guard: Any | None,) -> None: """Close capture and session independently, retaining teardown failures.""" failures: list[BaseException] = [] if graph_capture is not None and binding is not None: try: graph_capture.close_resident_graph_capture(binding) except BaseException as error: failures.append(error) if session is not None and registry is not None and guard is not None: try: session.close(registry, guard) except BaseException as error: failures.append(error) if failures: if len(failures) > 1: raise failures[0] from failures[1] raise failures[0]
def run_example() -> ExampleRun: # noqa: C901 """Capture and replay a qualified native CUDA graph without fallback.
Unavailable native capture returns deterministic status lines without fixture construction, resident setup, upload, or capture. The enabled path has one host-read boundary: explicit synchronization after two replays. It then deliberately invalidates, retires, and renews the fixed binding.
Returns: Bounded status and synchronized observations from the walkthrough.
Raises: Exception: Propagates unexpected qualification, enabled-path setup, capture, replay, renewal, or teardown failures after cleanup. """ native = _qualified_native_cuda() if native is None: return ExampleRun(output=_disabled_output()) capability_runtime = _load_capability_runtime() execution = capability_runtime.execution graph_capture = capability_runtime.graph_capture device = execution.Device(execution.Backend.WARP, native) capability_adapter = _WarpNativeCaptureAdapter( capability_runtime.warp, device, graph_capture ) capability = _resolve_exact_capability( graph_capture, device, capability_adapter ) runtime = _load_enabled_runtime() graph_capture = runtime.graph_capture device = execution.Device(execution.Backend.WARP, native) particles, gas, environment = _build_cpu_state() initial_total_mass = ( np.sum(particles.masses * particles.concentration[:, :, None], axis=1) + gas.concentration * particles.volume[:, None] ) session = None binding = None registry = None guard = None try: resident_runtime = runtime.resident_example._load_enabled_runtime() session = resident_runtime.gpu_session.setup_resident_session( particles, gas, environment, device ) registry = resident_runtime.gpu_resources.GPUResourceRegistry(session) guard = resident_runtime.gpu_session.ResidentStepGuard( session, registry ) request, gas_output, saturation_output = _compose_request( runtime, session, registry, guard, gas, environment, initial_total_mass, ) signature = graph_capture.create_resident_graph_capture_signature( request ) adapter = _WarpNativeCaptureAdapter(runtime.warp, device, graph_capture) lifecycle = graph_capture.create_graph_capture_lifecycle( capability, signature, ) binding = graph_capture.ResidentGraphCaptureBinding( request, session, registry, guard, lifecycle ) graph_capture._attach_resident_graph_capture_binding(request, binding) session.initialize_streams(registry, guard) prepared = runtime.resident_scheduler.prepare_resident_simulation( request, 1.0 ) capture_set = registry.validate_capture_resource_set( request.capture_resource_requirements ) qualification = graph_capture.qualify_prepared_resident_graph_capture( binding, prepared, capture_set, adapter ) captured = graph_capture.capture_prepared_resident_graph(qualification) for _ in range(2): graph_capture.replay_captured_resident_graph(captured, 1.0) runtime.warp.synchronize_device(session.particles.masses.device) gas_snapshot = gas_output.numpy().copy() saturation_snapshot = saturation_output.numpy().copy() original_update = request.environment_update object.__setattr__(request, "environment_update", object()) try: try: graph_capture.replay_captured_resident_graph(captured, 1.0) except ValueError: invalidated = ( binding.lifecycle.state is graph_capture.GraphCaptureLifecycleState.INVALIDATED ) else: raise RuntimeError("structural signature drift was accepted.") finally: object.__setattr__(request, "environment_update", original_update) if not invalidated: raise RuntimeError("structural signature drift did not invalidate.") graph_capture.retire_resident_graph_capture(binding) renewed_lifecycle = graph_capture.renew_resident_graph_capture( binding, graph_capture.create_resident_graph_capture_signature(request), ) if renewed_lifecycle is lifecycle: raise RuntimeError("renewal reused the retired lifecycle.") renewed_prepared = ( runtime.resident_scheduler.prepare_resident_simulation(request, 1.0) ) renewed_set = registry.validate_capture_resource_set( request.capture_resource_requirements ) renewed_qualification = ( graph_capture.qualify_prepared_resident_graph_capture( binding, renewed_prepared, renewed_set, adapter ) ) renewed_capture = graph_capture.capture_prepared_resident_graph( renewed_qualification ) result = ExampleRun( output=[ "Qualified native CUDA capture completed without fallback.", "Captured once, replayed twice, then explicitly retired and " "renewed.", "Opaque handles are not serialized; no performance claim is " "made.", ], session=session, registry=registry, guard=guard, binding=binding, captured=captured, renewed_capture=renewed_capture, gas_snapshot=gas_snapshot, saturation_snapshot=saturation_snapshot, replay_count=2, invalidated=True, retired=True, renewed=True, synchronized=True, ) except BaseException as primary_error: try: _close_enabled_binding( graph_capture, binding, session, registry, guard ) except BaseException as cleanup_error: raise primary_error from cleanup_error raise else: _close_enabled_binding(graph_capture, binding, session, registry, guard) return resultObserve actual physics after graph replay
Section titled “Observe actual physics after graph replay”Recording captures kernel launches; the two replays advance the physical state. Each box retains its own gas and thermodynamic response. Structural drift invalidates the old graph, and renewal creates a fresh binding. This small lifecycle demonstration makes no throughput claim; the prepared scaling studies measure warm replay and end-to-end costs separately.
Python · section 2
def run_publication(): """Require native CUDA capture and visualize the replayed box states.""" import matplotlib.pyplot as plt import warp as wp
if _qualified_native_cuda() is None: raise RuntimeError("This publication run requires native CUDA capture") result = run_example() if result.gas_snapshot is None or result.saturation_snapshot is None: raise RuntimeError("CUDA capture produced no diagnostic snapshots") boxes = np.arange(result.gas_snapshot.shape[0]) + 1 fig, axes = plt.subplots(1, 2, figsize=(9, 3.5), constrained_layout=True) axes[0].bar(boxes, result.gas_snapshot[:, 0], color="#287a90") axes[0].set( xlabel="Box", ylabel="Water gas after replay (kg m$^{-3}$)", xticks=boxes, ) axes[1].bar(boxes, result.saturation_snapshot[:, 0], color="#be6c39") axes[1].axhline(1, color="0.5", linestyle="--", label="Saturation") axes[1].set( xlabel="Box", ylabel="Saturation ratio after replay", xticks=boxes ) axes[1].legend() fig.suptitle("Two native graph replays · " + wp.get_device("cuda:0").name) print("\n".join(result.output)) plt.show() return result
def main() -> None: """Run the example and print its deterministic status lines.
The unavailable path prints the fixed capability status. The enabled path prints the bounded native-CUDA capture walkthrough status. """ for line in run_example().output: print(line)
if __name__ == "__main__": import sys
if "--cuda" in sys.argv[1:]: run_publication() else: main()Explicit cuda invocation
result = run_publication()Warp 1.11.1 initialized:
CUDA Toolkit 12.9, Driver 13.4
Devices:
"cpu" : "CPU"
"cuda:0" : "NVIDIA GeForce RTX 5060" (8 GiB, sm_120, mempool enabled)
Kernel cache:
~/.cache/warp/1.11.1
Module particula.execution.communication 2909959 load on device 'cuda:0' took 3.81 ms (cached)
Module particula.execution.rng c355345 load on device 'cuda:0' took 1.70 ms (cached)
Module particula.execution.state_updates 9e694ef load on device 'cuda:0' took 112.06 ms (cached)
Module particula.gpu.kernels.condensation aaea4cd load on device 'cuda:0' took 23.57 ms (cached)
Module particula.gpu.kernels.environment e11375e load on device 'cuda:0' took 1.92 ms (cached)
Module particula.gpu.kernels._condensation_native 34375de load on device 'cuda:0' took 118.43 ms (cached)
Module particula.gpu.kernels.coagulation 9ee9d75 load on device 'cuda:0' took 136.06 ms (cached)
Module particula.gpu.kernels._coagulation_native f93a244 load on device 'cuda:0' took 2.76 ms (cached)
Module particula.gpu.kernels.dilution 0472484 load on device 'cuda:0' took 2.39 ms (cached)
Module particula.gpu.kernels._population_validation 65eaa49 load on device 'cuda:0' took 1.71 ms (cached)
Module particula.gpu.kernels.wall_loss 6dc09db load on device 'cuda:0' took 4.72 ms (cached)
Module particula.gpu.kernels._wall_loss_native 314c0ac load on device 'cuda:0' took 1.59 ms (cached)
Module warp.utils f35afd3 load on device 'cuda:0' took 1.78 ms (cached)
Module warp.math edde159 load on device 'cuda:0' took 1.23 ms (cached)
Module warp.marching_cubes fb6e712 load on device 'cuda:0' took 1.82 ms (cached)
Module particula.execution.diagnostics edde159 load on device 'cuda:0' took 5.55 ms (cached)
Module particula.gpu.kernels.communication ac9b2a2 load on device 'cuda:0' took 3.13 ms (cached)
Module particula.gpu.properties.gas_properties edde159 load on device 'cuda:0' took 1.23 ms (cached)
Module particula.gpu.properties.particle_properties edde159 load on device 'cuda:0' took 0.98 ms (cached)
Module particula.gpu.dynamics.coagulation_funcs edde159 load on device 'cuda:0' took 1.12 ms (cached)
Module particula.gpu.dynamics.condensation_funcs edde159 load on device 'cuda:0' took 0.86 ms (cached)
Module particula.gpu.kernels._condensation_parallel edde159 load on device 'cuda:0' took 0.93 ms (cached)
Module particula.gpu.kernels.thermodynamics edde159 load on device 'cuda:0' took 0.71 ms (cached)
Module particula.gpu.kernels.exhaustion 616f81d load on device 'cuda:0' took 2624.61 ms (compiled)
Module particula.gpu.kernels.slot_management 0375e2c load on device 'cuda:0' took 3.88 ms (cached)
Module particula.gpu.kernels.nucleation 1124f09 load on device 'cuda:0' took 13.66 ms (cached)
Module particula.execution.gpu_resources 30f296c load on device 'cuda:0' took 6.98 ms (cached)
Module particula.execution.thermodynamic_updates edde159 load on device 'cuda:0' took 15.82 ms (cached)
Module particula.gpu.dynamics.wall_loss_funcs edde159 load on device 'cuda:0' took 7.64 ms (cached)
Module particula.gpu.warp_types edde159 load on device 'cuda:0' took 8.70 ms (cached)
Module particula.gpu.dynamics.ion_pair_data edde159 load on device 'cuda:0' took 2.58 ms (cached)
Module particula.gpu.kernels._brownian_parallel 30c33a2 load on device 'cuda:0' took 5.20 ms (cached)
Module particula.gpu.kernels.thermodynamics 10b2126 load on device 'cuda:0' took 1.69 ms (cached)
Module particula.execution.thermodynamic_updates 610ef79 load on device 'cuda:0' took 5.77 ms (cached)
Module particula.gpu.kernels.condensation ff745d0 load on device 'cuda:0' took 50.76 ms (cached)
Module particula.gpu.kernels.condensation 8cf5639 load on device 'cuda:0' took 100.95 ms (cached)
Module particula.gpu.kernels.condensation 7771c82 load on device 'cuda:0' took 1277.41 ms (compiled)
Module particula.gpu.kernels._condensation_native 9623455 load on device 'cuda:0' took 137.88 ms (cached)
Module particula.gpu.kernels.condensation 300a970 load on device 'cuda:0' took 1455.88 ms (compiled)
Module particula.gpu.kernels._condensation_native 76403ad load on device 'cuda:0' took 224.54 ms (cached)
Module particula.gpu.kernels.condensation 0e0fe7b load on device 'cuda:0' took 1473.49 ms (compiled)
Module particula.gpu.kernels._condensation_native c295a4d load on device 'cuda:0' took 250.19 ms (cached)
Module particula.gpu.kernels._condensation_native a80ebff load on device 'cuda:0' took 447.71 ms (cached)
Module particula.gpu.kernels.condensation 5141db8 load on device 'cuda:0' took 1585.92 ms (compiled)
Module particula.gpu.kernels.condensation 99a257a load on device 'cuda:0' took 1682.47 ms (compiled)
Module particula.gpu.kernels.condensation 9320f05 load on device 'cuda:0' took 1738.93 ms (compiled)
Module particula.gpu.kernels._condensation_native 731ad93 load on device 'cuda:0' took 535.49 ms (cached)
Module particula.gpu.kernels._brownian_parallel 6745116 load on device 'cuda:0' took 6.04 ms (cached)
Module particula.gpu.kernels.coagulation 542143a load on device 'cuda:0' took 257.38 ms (cached)
Module particula.gpu.kernels.coagulation 8fc901f load on device 'cuda:0' took 2.13 ms (cached)
Module particula.execution.diagnostics ade82cd load on device 'cuda:0' took 30.33 ms (cached)
Module particula.execution.diagnostics 903c22d load on device 'cuda:0' took 59.15 ms (cached)
Module particula.execution.diagnostics ce5a797 load on device 'cuda:0' took 82.51 ms (cached)
Module particula.execution.diagnostics 1ea62de load on device 'cuda:0' took 101.66 ms (cached)
Module particula.execution.diagnostics 037f64e load on device 'cuda:0' took 130.25 ms (cached)
Module particula.execution.diagnostics f921a30 load on device 'cuda:0' took 152.99 ms (cached)
Module particula.gpu.kernels.ion_condensation 79e70b9 load on device 'cuda:0' took 2.47 ms (cached)
Qualified native CUDA capture completed without fallback.
Captured once, replayed twice, then explicitly retired and renewed.
Opaque handles are not serialized; no performance claim is made.

Know what the replay result represents
Section titled “Know what the replay result represents”The two one-second replays reuse the three-box fixture and its prescribed gas updates from the resident schedule. Its fixed 800 Pa vapor-pressure reference and large gas densities are illustrative inputs. The gas and saturation bars show state after those replays; they do not measure speedup or establish convergence. Saturation one marks the configured flat-surface reference, not a universal threshold for particle equilibrium.
The deliberate structural change is rejected, then the old capture is retired and renewed. Renewal does not add a third replay to these plotted observations. For a long simulation, compare captured and ordinary final states and random continuation before benchmarking replay. Recreate captures after a checkpoint restart rather than treating an opaque graph handle as portable scientific state. See prepared lifecycle checks and measured scaling for those two comparisons.
Explore this topic: Resident execution