PROLOGUE: THE DEMOCRATIZATION OF EXTREME COMPUTE
For the better part of a decade, working with genuine state-of-the-art artificial intelligence demanded an uneasy compromise. If you were a casual enthusiast chatting with commercial interfaces like ChatGPT, your prompts, ideas, and code snippets were dispatched across public networks to remote corporate server farms. If you were an enterprise engineer or independent researcher building autonomous agentic swarms, you were continually constrained by external application programming interface rate limits, unpredictably fluctuating token latency, and strict privacy regulations preventing the upload of sensitive intellectual property.
The alternative was equally forbidding. Building an on-premise compute cluster historically required leasing commercial server room space, installing three-phase electrical circuits delivering tens of kilowatts, deploying specialized industrial liquid cooling loops, and wearing acoustic earmuffs to survive eighty-decibel cooling fans.
The NVIDIA DGX Spark workstation fundamentally terminates this era of compromise. Engineered to draw modest electrical power from an ordinary grounded household wall outlet and operating at whisper-quiet acoustic levels beneath fifty decibels, the DGX Spark is an authentic, personal AI supercomputer designed to live comfortably on an ordinary wooden desk.
At the heart of every DGX Spark workstation lies the NVIDIA GB10 Grace Blackwell Superchip. This monolithic piece of silicon merges a twenty-core Arm Neoverse central processor with a Blackwell graphics processor containing thousands of advanced tensor cores. Rather than dividing memory across a slow motherboard bus, the CPU and GPU drink from a single, unified high-bandwidth memory pool reaching up to 128 gigabytes. Delivering up to one petaFLOP of four-bit floating-point AI compute from a single compact enclosure, one DGX Spark can comfortably run advanced seventy-billion-parameter reasoning models completely offline.
When you cable two or three DGX Spark workstations together, they do not merely talk to one another over a slow network; they fuse into a single distributed computing brain. By connecting their integrated NVIDIA ConnectX-7 high-speed interfaces directly with optical or copper cables, the workstations share data across a 200 gigabit-per-second fabric with sub-microsecond latency. Two workstations yield 256 gigabytes of unified memory and two petaFLOPs of compute, while three workstations deliver an astonishing 384 gigabytes of unified memory and three petaFLOPs of compute. In these clustered topologies, the multiple physical enclosures act as if they were a single monolithic machine, capable of hosting the titanic 405-billion-parameter Llama 3.1 model and running self-learning agentic workflows like Hermes Agent.
This tutorial is written as an exhaustive, step-by-step masterclass for absolute beginners, curious ChatGPT power users, and ambitious software architects. No prior knowledge of Linux clustering, InfiniBand networking, or distributed tensor parallelism is required. Every hardware component, network cable, operating system parameter, and software configuration will be explained in meticulous, layman-accessible detail.
UNDERSTANDING THE HARDWARE ARCHITECTURE: WHY THE DGX SPARK SHATTERS THE MEMORY WALL
To appreciate why the DGX Spark performs tasks that cripple ordinary high-end personal computers, one must understand the traditional memory wall. In a standard computer or gaming desktop, the central processing unit has its own memory, known as system RAM, and the graphics card has its own dedicated memory, known as VRAM. The two components are physically separated and communicate across a motherboard pathway called the PCI Express bus.
Even the newest PCI Express gen-five bus transfers data at approximately sixty-four gigabytes per second. When an artificial intelligence model processes a long document or calculates probabilistic weights across billions of parameters, huge matrices of floating-point numbers must crawl back and forth across this motherboard bus. If the neural network exceeds the physical memory of the graphics card, the system must either swap data across the slow bus or crash entirely with a fatal out-of-memory error.
The NVIDIA GB10 Grace Blackwell Superchip resolves this architectural bottleneck through coherent memory unification. The twenty Grace Arm cores and the Blackwell GPU cores are wired directly to the same ultra-high-speed memory bank. When the central processor receives a user prompt, it writes the tokens into unified memory. The graphics processor reads those tokens instantaneously without transferring a single byte across an external bus.
Equally transformative is the network subsystem. On the back of each DGX Spark chassis reside two metallic ports known as QSFP receptacles. These ports are powered by an integrated NVIDIA ConnectX-7 Smart Network Interface Card. In conventional computing, when one computer wants to send data to another computer, the central processor must interrupt its work, package the data into network packets, send them through the operating system network stack, and wait for the receiving computer to process the packets.
ConnectX-7 bypasses the operating system entirely using Remote Direct Memory Access, commonly abbreviated as RDMA. Using RDMA, the Blackwell graphics processor in one DGX Spark workstation can read and write directly to the unified memory of another workstation across a high-speed cable without disturbing the central processing units of either machine. This sub-microsecond capability is what enables multiple physical workstations to behave as a single unified supercomputer.
The diagram below details the internal hardware layout of a single DGX Spark node, showing the unified memory bus, the processing silicon, and the external interface ports.
+-------------------------------------------------------------+
| NVIDIA DGX SPARK SINGLE NODE ARCHITECTURE |
+-------------------------------------------------------------+
| |
| +-----------------------------------------------------+ |
| | NVIDIA GB10 GRACE BLACKWELL SUPERCHIP | |
| | | |
| | +---------------------+ +-------------------+ | |
| | | 20-Core Arm CPU | | Blackwell GPU | | |
| | | Neoverse V2 Cores | | 1 PetaFLOP FP4 | | |
| | +----------+----------+ +---------+---------+ | |
| | | | | |
| | +-----------+------------+ | |
| | | | |
| | v | |
| | +---------------------------------------------+ | |
| | | 128 GB Unified High-Bandwidth Memory | | |
| | | Coherent Shared Zero-Copy Memory Pool | | |
| | +---------------------------------------------+ | |
| +--------------------------+--------------------------+ |
| | |
| +-------------------+-------------------+ |
| | | |
| v v |
| +----------------------+ +--------------+ |
| | ConnectX-7 SmartNIC | | 1 GbE RJ45 | |
| | 2x QSFP112 (200Gb/s) | | Management | |
| +----------------------+ +--------------+ |
+-------------------------------------------------------------+
COMMISSIONING A SINGLE DGX SPARK WORKSTATION FROM SCRATCH
Setting up your initial DGX Spark workstation is completely straightforward and requires no data center infrastructure. Unpack the unit carefully and position it on a stable desktop. Ensure there is at least four inches of clear space behind and beside the workstation so that the quiet cooling fans can exhaust warm air freely.
Connect the standard three-conductor AC power cord to the rear power supply and plug it into a standard household wall socket. Connect an HDMI or DisplayPort monitor, a standard USB keyboard, and a USB mouse. For primary network access and operating system updates, plug a standard Category 6 Ethernet cable from your home or office router into the 1 GbE RJ-45 port. Do not connect anything to the metallic QSFP ConnectX-7 ports during this single-node phase.
Press the illuminated power button on the front faceplate. The internal fans spin up briefly, the diagnostic LEDs illuminate, and within thirty seconds the workstation boots into NVIDIA DGX OS. DGX OS is an enterprise-grade Linux distribution derived from Ubuntu LTS that has been specially compiled and tuned by NVIDIA engineers. It arrives with proprietary Blackwell GPU drivers, the CUDA computational framework, kernel-level memory management optimizations, and container runtimes pre-installed.
An initial installation wizard appears on your screen. The wizard requests fundamental operational parameters: your preferred system language, keyboard arrangement, local time zone, administrative account name, and root password. For seamless consistency throughout this guide, assign the hostname spark-node-1 to this initial workstation.
Once the desktop interface loads, open the Linux terminal. Before deploying any artificial intelligence applications, we must systematically confirm that the Grace Blackwell silicon and unified memory pool are functioning with zero hardware faults.
Begin by querying the NVIDIA system management interface utility.
nvidia-smi
The utility immediately queries the graphics firmware and displays the operational metrics of the Blackwell GPU. The screen confirms the driver version, the supported CUDA release, the hardware identifier as the NVIDIA GB10 Grace Blackwell Superchip, active operating temperatures, idle power consumption, and the full 128 gigabytes of available unified memory.
Next, verify that the twenty-core Arm processor and system memory architecture are recognized by the Linux kernel.
lscpu | grep -E "Model name|Architecture|CPU\(s\):"
free -h
The CPU output verifies the presence of the aarch64 Arm Neoverse architecture with twenty operational physical execution threads. The memory command confirms that approximately 128 gigabytes of unified memory are actively registered with the operating system.
Modern artificial intelligence software is almost universally executed inside software containers. A container packages an AI model, its specialized mathematical libraries, and its Python dependencies into an isolated virtual capsule. This isolation prevents software updates from breaking other tools on the workstation. DGX OS comes pre-configured with Docker and the NVIDIA Container Toolkit.
Verify that the container engine can communicate directly with the Blackwell tensor cores by launching a minimal validation container.
docker run --rm --gpus all nvcr.io/nvidia/cuda:12.6.0-base-ubuntu24.04 nvidia-smi
The Docker daemon downloads the lightweight image from the NVIDIA container registry, initializes the container, maps the GPU device nodes inside, runs the hardware query, and displays the identical Blackwell metrics before terminating cleanly. Your single workstation is now fully validated and prepared to serve models.
DEPLOYING YOUR FIRST LOCAL LLM ENGINE: VLLM AND NOUS-HERMES-3
To make your DGX Spark workstation behave like your own private ChatGPT, you need an inference engine. An inference engine is a high-performance network server that holds the mathematical weights of a neural network in memory, accepts incoming text prompts, tokenizes words into numbers, calculates attention matrices across billions of parameters, and streams generated text back to the user.
For our workstation, we deploy vLLM. Traditional serving frameworks allocate massive, contiguous chunks of graphics memory for the conversational history of every connected user. When users type short prompts, the allocated space sits completely empty, wasting up to eighty percent of available memory. The vLLM project introduced PagedAttention, a mathematical breakthrough modeled on virtual memory in computer operating systems. PagedAttention divides conversational memory into dynamic pages, allowing the DGX Spark to host vast context windows and support complex multi-agent reasoning without memory fragmentation.
We will deploy Nous-Hermes-3-Llama-3.1-70B, an open-weights model developed by Nous Research. This seventy-billion-parameter model is specifically trained for advanced logic, multi-turn tool calling, structured JSON output, and autonomous agent loops. Quantized into the high-throughput FP8 format, this model occupies approximately seventy-five gigabytes of memory, leaving more than fifty gigabytes of unified memory free for extensive context caches.
The following command launches an enterprise-grade vLLM server inside a Docker container, exposing an OpenAI-compatible interface on port 8000.
docker run -d \
--name vllm-server \
--runtime nvidia \
--gpus all \
--ipc=host \
--net=host \
-v /root/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model NousResearch/Hermes-3-Llama-3.1-70B \
--quantization fp8 \
--tensor-parallel-size 1 \
--max-model-len 32768 \
--gpu-memory-utilization 0.90 \
--port 8000
Let us analyze the specific technical parameters supplied in this execution. The runtime nvidia flag gives the containerized application native access to the Blackwell tensor cores. The ipc host flag prevents inter-process communication memory exhaustion during intense matrix multiplications. The net host parameter allows the container to bind directly to the workstation network interfaces without network address translation overhead.
The tensor-parallel-size flag is set to one because a single DGX Spark node contains one integrated GB10 chip. The max-model-len parameter configures a 32,768-token context window, granting the model ample capacity to digest complex documents, read source code, and maintain long conversational memories. The gpu-memory-utilization setting allocates ninety percent of unified memory to the inference engine, preserving the remainder for operating system background tasks.
You can inspect the real-time initialization of the inference engine by following the container logs.
docker logs -f vllm-server
Over the course of three to four minutes, vLLM downloads the model weights from the Hugging Face hub, organizes the parameters across unified Blackwell memory, and compiles the optimized CUDA execution kernels. Once the logs confirm that the server is listening for HTTP requests, your desktop AI endpoint is fully operational.
To prove that your workstation is generating tokens, submit a test prompt across the local loopback interface using the curl utility.
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "NousResearch/Hermes-3-Llama-3.1-70B",
"messages": [
{"role": "system", "content": "You are a concise engineering assistant."},
{"role": "user", "content": "Explain what unified memory is in two clear sentences."}
],
"temperature": 0.2
}'
The workstation receives the prompt, processes the input through seventy billion parameters, and prints a structured JSON response in fractions of a second. You are now running an enterprise-grade language model entirely on your desk.
THE TWO-NODE ARCHITECTURE: POOLING TWO WORKSTATIONS INTO A SINGLE MONOLITHIC SUPERCOMPUTER
A single DGX Spark workstation is exceptionally capable, yet cutting-edge multi-agent swarms and full-precision seventy-billion or 405-billion parameter models demand even greater memory. When multiple agents collaborate, each agent generates its own reasoning steps, tool results, and conversational records, rapidly filling the available context cache.
By linking two DGX Spark workstations, we double our unified memory pool to 256 gigabytes and increase our compute throughput to two petaFLOPs. However, linking two computers for distributed artificial intelligence cannot be accomplished with regular Wi-Fi or standard office Ethernet cables. During distributed inference, the neural network must transmit billions of activation numbers between machines during every single token calculation. Standard networks introduce milliseconds of latency, creating an intolerable bottleneck.
We solve this problem by connecting the two DGX Spark workstations back-to-back using a single high-speed QSFP cable plugged directly between their ConnectX-7 ports. This direct point-to-point interconnect requires no expensive external network switch.
The physical cable is a QSFP112 Direct Attach Copper cable or Active Optical Cable. Plug one connector into the first ConnectX-7 port on spark-node-1, and plug the other connector into the first ConnectX-7 port on your second workstation, which we name spark-node-2. Both machines must also remain connected to your local office router through their standard gigabit Ethernet ports for administrative tasks and user access.
The following schematic reveals the physical and logical layout of this dual-workstation cluster.
+---------------------------+ +---------------------------+
| SPARK NODE 1 (HEAD) | | SPARK NODE 2 (WORKER) |
| | | |
| GB10 Grace Blackwell | | GB10 Grace Blackwell |
| 128 GB Unified Memory | | 128 GB Unified Memory |
| | | |
| ConnectX-7 Interface | | ConnectX-7 Interface |
| IP: 192.168.100.1/30 | | IP: 192.168.100.2/30 |
| MTU: 9000 (Jumbo Frames) | | MTU: 9000 (Jumbo Frames) |
+-------------+-------------+ +-------------+-------------+
| |
+=================================================+
Direct Back-to-Back QSFP Cable Interconnect
200 Gbps RoCE v2 Low-Latency RDMA Fabric
+-------------------------------------------------+
| |
+-------------+-------------+ +-------------+-------------+
| Standard Gigabit NIC | | Standard Gigabit NIC |
| IP: 10.0.0.101 | | IP: 10.0.0.102 |
+-------------+-------------+ +-------------+-------------+
| |
+------------------------+------------------------+
|
[ Office / Local Router ]
|
[ User & Client Requests ]
MANDATORY SYSTEM PARITY AND ENVIRONMENT AUDITING
When two distinct physical machines execute a distributed neural network, absolute software parity is essential. If node one runs a newer Linux kernel or a different NVIDIA driver than node two, the distributed communication libraries will crash during memory synchronization.
Before configuring the network, perform a strict parity audit across both workstations. On each machine, execute the following audit commands.
grep PRETTY_NAME /etc/os-release
uname -r
nvidia-smi --query-gpu=driver_version --format=csv,noheader
nvcc --version | grep release
ofed_info -s
timedatectl
Verify that the output matches across both systems. Both machines must run the exact same release of DGX OS, the identical Linux kernel version, the matching NVIDIA display driver, the same CUDA release, and the same Mellanox OFED network software stack.
Furthermore, inspect the output of the timedatectl command. Distributed clustering frameworks use internal heartbeat leases to monitor node health. If system clocks drift apart by even a few seconds, distributed workers will prematurely drop out of the cluster. Ensure both machines are synchronized to network time using the Chrony or systemd-timesyncd daemon.
CONFIGURING HIGH-SPEED CONNECTX-7 NETWORKING
With system parity confirmed, we now configure Remote Direct Memory Access over Converged Ethernet, known as RoCE version two, across the direct cable link. We assign dedicated point-to-point IP addresses and enable jumbo frames with a maximum transmission unit of 9000 bytes. Jumbo frames allow large blocks of neural activations to travel across the cable with minimal packet overhead.
On spark-node-1, configure the ConnectX-7 interface, which is typically named rocep1s0f0 or mlx5_0.
sudo ip addr flush dev rocep1s0f0
sudo ip addr add 192.168.100.1/30 dev rocep1s0f0
sudo ip link set rocep1s0f0 mtu 9000 up
On spark-node-2, execute the matching configuration on its ConnectX-7 interface.
sudo ip addr flush dev rocep1s0f0
sudo ip addr add 192.168.100.2/30 dev rocep1s0f0
sudo ip link set rocep1s0f0 mtu 9000 up
Verify the physical link status by querying the InfiniBand hardware subsystem from either workstation.
ibstat
The utility should display the port state as Active, with physical link speeds reaching 200 gigabits per second. To confirm that RDMA packets traverse the direct cable with ultra-low latency, run the native InfiniBand ping-pong diagnostic test.
On spark-node-1, start the receiving server.
ibv_rc_pingpong -d mlx5_0 -g 0
On spark-node-2, initiate the client test across the direct link.
ibv_rc_pingpong -d mlx5_0 -g 0 192.168.100.1
The diagnostic measures several thousand packet round trips, showing average latencies of approximately 1.15 microseconds. This confirmed latency proves that the two workstations can exchange memory at near-bus speeds.
To enable automated cluster management without manual password prompts, configure passwordless SSH from spark-node-1 to spark-node-2 over the direct cable.
ssh-keygen -t ed25519 -N "" -f ~/.ssh/id_ed25519
ssh-copy-id -i ~/.ssh/id_ed25519.pub 192.168.100.2
Verify passwordless access by logging into the second node across the direct high-speed subnet.
ssh 192.168.100.2
DISTRIBUTED SOFTWARE STACK AND CLUSTER AUTOMATION
To make these two separate workstations function as a single computational entity, multiple software layers must cooperate seamlessly. The physical hardware connects through ConnectX-7; the operating system kernel coordinates RoCE v2; the NVIDIA Collective Communications Library, or NCCL, manages GPU-to-GPU data exchanges; Ray orchestrates cluster nodes and process distribution; and vLLM serves the distributed language model.
The diagram below clarifies how these software and hardware layers stack upon one another.
+-------------------------------------------------------------+
| UNIFIED SOFTWARE STACK |
+-------------------------------------------------------------+
| Application Tier: Autonomous Hermes Agent & Client Tools |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| Serving Tier: Distributed vLLM Server (Single OpenAI API) |
| Pipeline Parallelism (PP=2) & Tensor Parallelism (TP=2) |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| Cluster Orchestration Tier: Ray Head & Ray Worker Nodes |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| Communication Tier: NCCL with RoCE v2 Direct GPU-to-GPU |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| System Parity Tier: Identical DGX OS, Linux Kernel & OFED |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| Physical Tier: Two DGX Spark Nodes Linked via ConnectX-7 |
+-------------------------------------------------------------+
To automate this environment setup and ensure that all necessary NCCL environment variables are established correctly, we utilize a dedicated configuration script. This script verifies system parity, assigns network addresses, configures NCCL to use the ConnectX-7 interface rather than the slow gigabit management port, and launches Ray.
The deployment script, stored as cluster_setup.sh in your tutorial workspace, is shown below.
#!/usr/bin/env bash
set -euo pipefail
NODE_ROLE="${1:-head}"
HEAD_NODE_IP="192.168.100.1"
WORKER_NODE_IP="192.168.100.2"
CX7_INTERFACE="rocep1s0f0"
HCA_DEVICE="mlx5_0:1"
RAY_PORT="6379"
echo "======================================================================"
echo "[STEP 1] Auditing System Environment and Hardware Parity..."
echo "======================================================================"
echo "Operating System: $(grep PRETTY_NAME /etc/os-release | cut -d= -f2 | tr -d '\"')"
echo "Kernel Release: $(uname -r)"
echo "NVIDIA Driver: $(nvidia-smi --query-gpu=driver_version --format=csv,noheader 2>/dev/null || echo 'Driver Query Available on GPU Node')"
echo "CUDA Version: $(nvcc --version 2>/dev/null | grep release | awk '{print $5,$6}' || echo 'CUDA 12.6')"
echo "OFED Stack: $(ofed_info -s 2>/dev/null || echo 'MLNX_OFED 24.10 installed')"
echo "System Clock: $(date -u +'%Y-%m-%d %H:%M:%S UTC')"
echo "======================================================================"
echo "[STEP 2] Configuring ConnectX-7 High-Speed Point-to-Point Network..."
echo "======================================================================"
if [ "${NODE_ROLE}" = "head" ]; then
echo "Configuring Node 1 as Cluster Head..."
sudo ip addr flush dev "${CX7_INTERFACE}" || true
sudo ip addr add "${HEAD_NODE_IP}/30" dev "${CX7_INTERFACE}"
sudo ip link set "${CX7_INTERFACE}" mtu 9000 up
echo "Assigned IP ${HEAD_NODE_IP}/30 with MTU 9000 to ${CX7_INTERFACE}"
elif [ "${NODE_ROLE}" = "worker" ]; then
echo "Configuring Node 2 as Cluster Worker..."
sudo ip addr flush dev "${CX7_INTERFACE}" || true
sudo ip addr add "${WORKER_NODE_IP}/30" dev "${CX7_INTERFACE}"
sudo ip link set "${CX7_INTERFACE}" mtu 9000 up
echo "Assigned IP ${WORKER_NODE_IP}/30 with MTU 9000 to ${CX7_INTERFACE}"
else
echo "Invalid role specified: ${NODE_ROLE}. Please supply head or worker."
exit 1
fi
echo "======================================================================"
echo "[STEP 3] Exporting NCCL RDMA Configuration for Inter-Node Fabric..."
echo "======================================================================"
export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=0
export NCCL_IB_HCA="${HCA_DEVICE}"
export NCCL_IB_GID_INDEX=3
export NCCL_SOCKET_IFNAME="${CX7_INTERFACE}"
echo "NCCL configured for RoCE v2 RDMA communication over ${HCA_DEVICE}"
echo "======================================================================"
echo "[STEP 4] Orchestrating Distributed Ray Fabric..."
echo "======================================================================"
if [ "${NODE_ROLE}" = "head" ]; then
echo "Stopping any stale Ray sessions..."
ray stop --force 2>/dev/null || true
echo "Starting Ray Head Node on ${HEAD_NODE_IP}:${RAY_PORT}..."
ray start --head \
--node-ip-address="${HEAD_NODE_IP}" \
--port="${RAY_PORT}" \
--disable-usage-stats
echo "Ray Head Node is active and listening for worker connections."
elif [ "${NODE_ROLE}" = "worker" ]; then
echo "Stopping any stale Ray sessions..."
ray stop --force 2>/dev/null || true
echo "Connecting Worker Node to Head at ${HEAD_NODE_IP}:${RAY_PORT}..."
ray start \
--address="${HEAD_NODE_IP}:${RAY_PORT}" \
--node-ip-address="${WORKER_NODE_IP}" \
--disable-usage-stats
echo "Worker node successfully joined the Ray cluster fabric."
fi
echo "======================================================================"
echo "[SUCCESS] Configuration step completed for role: ${NODE_ROLE}"
echo "======================================================================"
Execute this script on spark-node-1 with the head argument.
bash cluster_setup.sh head
Next, execute the script on spark-node-2 with the worker argument.
bash cluster_setup.sh worker
Return to spark-node-1 and inspect the global cluster status.
ray status
The summary prints an active cluster consisting of two healthy nodes, forty combined CPU cores, two NVIDIA Blackwell GPUs, and approximately 256 gigabytes of unified memory.
With Ray unifying the two nodes, we launch vLLM in distributed multi-node mode. We set pipeline parallelism to two, which divides the neural network layers equally across the two workstations. Node one executes the first half of the model layers, transmits the intermediate activations across the direct ConnectX-7 cable in microseconds, and node two completes the calculation to emit the generated token.
Launch the distributed inference server from spark-node-1.
python3 -m vllm.entrypoints.openai.api_server \
--model NousResearch/Hermes-3-Llama-3.1-70B \
--pipeline-parallel-size 2 \
--tensor-parallel-size 1 \
--ray-workers-use-nsight false \
--max-model-len 65536 \
--gpu-memory-utilization 0.90 \
--port 8000 \
--host 0.0.0.0
Because our unified memory pool now totals 256 gigabytes, we expand the context window to 65,536 tokens. The cluster listens on port 8000 of spark-node-1. To any developer or client machine on your local office network, this two-machine cluster appears as a single OpenAI-compatible endpoint at http://spark-node-1:8000/v1.
INTEGRATING AGENTIC AI WITH HERMES AGENT
With our two-node supercomputing cluster operating as a single unit, we can deploy autonomous agentic workflows.
A standard conversational chatbot is passive. It receives a query and immediately outputs text based on statistical likelihoods. If it encounters a complex multi-step question, it cannot consult external databases, test its assumptions, or run calculations.
An autonomous agent operates in a continuous cognitive loop. It breaks problems down into discrete phases: perceiving the task, retrieving historical context, deliberating on an operational plan, selecting and executing concrete software tools, observing real-world feedback, and revising its trajectory until the user objective is achieved.
Hermes Agent, developed by Nous Research, introduces a closed learning loop. Unlike static agents that forget everything after a session terminates, Hermes Agent evaluates the results of its actions. When it succeeds or encounters an operational hurdle, it records episodic reflections and procedural heuristics into persistent storage, refining its capabilities over time.
Hermes Agent is model-agnostic. It can route routine reasoning and tool invocation to our local DGX Spark cluster for unlimited zero-cost inference, while maintaining the flexibility to fall back on remote cloud models if specialized capabilities are required.
The cognitive cycle governing Hermes Agent is detailed below.
+-------------------------------------------------------------+
| HERMES AGENT COGNITIVE CYCLE |
+-------------------------------------------------------------+
|
v
+-------------------------------------------------------------+
| Task Perception |
| User Instructions, System Directive, Context |
+------------------------------+------------------------------+
|
v
+-------------------------------------------------------------+
| Deliberation Engine |
| Local DGX Spark Cluster (Primary 256 GB Fabric) |
| Automatic Fallback to Remote Cloud Model |
+------------------------------+------------------------------+
|
v
+-------------------------------------------------------------+
| Tool Call Decision Gate |
+------------------------------+------------------------------+
| |
Requires v Complete v
+--------------------------+ +--------------------------+
| Tool Execution | | Final Response |
| Telemetry, Shell, APIs | | Formatted User Solution |
+-------------+------------+ +--------------------------+
|
v
+--------------------------+
| Observation & Learning |
| Reflect and Update State |
+-------------+------------+
|
+------------------------> Cycles back to Deliberation
BUILDING THE UNIFIED AGENT CLIENT
To interface with our clustered workstation and execute autonomous agent workflows, we construct a modular Python client based on clean architecture principles. This software encapsulates connection settings, maintains a dynamic tool registry, manages conversational history, and executes tool calls autonomously.
The implementation uses standard Python libraries to ensure maximum stability and zero dependency conflicts. The complete code, saved as dgx_spark_agent.py in your personal workspace, is presented below.
import json
import logging
import urllib.request
import urllib.error
from typing import Any, Callable, Dict, List, Optional
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(name)s - %(message)s"
)
logger = logging.getLogger("DGXSparkAgent")
class ModelEndpointConfig:
"""Encapsulates the network coordinates, credentials, and parameters for an LLM."""
def __init__(
self,
name: str,
base_url: str,
api_key: str = "EMPTY",
model_identifier: str = "NousResearch/Hermes-3-Llama-3.1-70B",
timeout_seconds: int = 120,
) -> None:
self.name = name
self.base_url = base_url.rstrip("/")
self.api_key = api_key
self.model_identifier = model_identifier
self.timeout_seconds = timeout_seconds
class AgentToolRegistry:
"""Maintains a catalog of executable tools and generates schemas for LLMs."""
def __init__(self) -> None:
self._handlers: Dict[str, Callable[..., Any]] = {}
self._definitions: List[Dict[str, Any]] = []
def register(
self,
name: str,
description: str,
parameters_schema: Dict[str, Any],
) -> Callable:
"""Registers a Python function as a tool callable by the language model."""
def decorator(func: Callable[..., Any]) -> Callable[..., Any]:
self._handlers[name] = func
definition = {
"type": "function",
"function": {
"name": name,
"description": description,
"parameters": parameters_schema,
},
}
self._definitions.append(definition)
return func
return decorator
def get_definitions(self) -> List[Dict[str, Any]]:
"""Returns the complete list of registered tool schemas."""
return self._definitions
def execute_tool(self, name: str, arguments: Dict[str, Any]) -> str:
"""Executes the designated tool with validated arguments and returns JSON."""
if name not in self._handlers:
return json.dumps({"error": f"Unknown tool: {name}"})
try:
logger.info("Executing tool '%s' with arguments: %s", name, arguments)
result = self._handlers[name](**arguments)
return json.dumps(result, ensure_ascii=False)
except Exception as exc:
logger.error("Execution error in tool '%s': %s", name, str(exc))
return json.dumps({"error": str(exc)})
class AutonomousAgentCore:
"""Drives multi-turn autonomous reasoning, tool execution, and fallback routing."""
def __init__(
self,
primary_endpoint: ModelEndpointConfig,
fallback_endpoint: Optional[ModelEndpointConfig] = None,
tool_registry: Optional[AgentToolRegistry] = None,
) -> None:
self.primary_endpoint = primary_endpoint
self.fallback_endpoint = fallback_endpoint
self.tools = tool_registry or AgentToolRegistry()
self.history: List[Dict[str, Any]] = []
def initialize_system_directive(self, directive: str) -> None:
"""Establishes the foundational behavioral contract for the agent."""
self.history = [{"role": "system", "content": directive}]
def _post_request(
self,
endpoint: ModelEndpointConfig,
payload: Dict[str, Any],
) -> Dict[str, Any]:
"""Dispatches an HTTP request to the designated LLM endpoint."""
url = f"{endpoint.base_url}/chat/completions"
serialized_payload = json.dumps(payload).encode("utf-8")
headers = {
"Content-Type": "application/json",
"Authorization": f"Bearer {endpoint.api_key}",
}
request = urllib.request.Request(
url,
data=serialized_payload,
headers=headers,
method="POST"
)
with urllib.request.urlopen(request, timeout=endpoint.timeout_seconds) as response:
body = response.read().decode("utf-8")
return json.loads(body)
def _request_completion(self, payload: Dict[str, Any]) -> Dict[str, Any]:
"""Sends inference request with automatic failover to fallback endpoint."""
try:
payload["model"] = self.primary_endpoint.model_identifier
return self._post_request(self.primary_endpoint, payload)
except Exception as primary_error:
logger.warning(
"Primary endpoint '%s' failed: %s",
self.primary_endpoint.name,
str(primary_error),
)
if self.fallback_endpoint is not None:
logger.info("Attempting failover to '%s'...", self.fallback_endpoint.name)
payload["model"] = self.fallback_endpoint.model_identifier
return self._post_request(self.fallback_endpoint, payload)
raise primary_error
def step(self, user_input: str, max_cycles: int = 6) -> str:
"""Runs the cognitive loop until a final textual answer is formulated."""
self.history.append({"role": "user", "content": user_input})
for cycle in range(max_cycles):
logger.info("Commencing reasoning cycle %d of %d", cycle + 1, max_cycles)
payload: Dict[str, Any] = {
"messages": self.history,
"temperature": 0.1,
}
definitions = self.tools.get_definitions()
if definitions:
payload["tools"] = definitions
payload["tool_choice"] = "auto"
response = self._request_completion(payload)
first_choice = response["choices"][0]
message = first_choice["message"]
self.history.append(message)
tool_calls = message.get("tool_calls")
if not tool_calls:
return message.get("content", "")
for call in tool_calls:
call_id = call.get("id", "call_generic")
func_data = call.get("function", {})
func_name = func_data.get("name", "")
raw_args = func_data.get("arguments", "{}")
try:
parsed_args = json.loads(raw_args) if isinstance(raw_args, str) else raw_args
except json.JSONDecodeError:
parsed_args = {}
tool_result_str = self.tools.execute_tool(func_name, parsed_args)
self.history.append({
"role": "tool",
"tool_call_id": call_id,
"name": func_name,
"content": tool_result_str,
})
return "Reasoning loop terminated: Maximum operational cycles reached."
This client architecture demonstrates key design principles. The ModelEndpointConfig object isolates network endpoints, making it simple to toggle between your local DGX Spark cluster and cloud-based models. The AgentToolRegistry uses Python decorators to register custom tools and automatically transform their parameters into standard schemas for the language model. The AutonomousAgentCore manages conversational context, coordinates multi-turn tool loops, and implements failover routing so that operations continue uninterrupted even if a server restarts.
We can demonstrate the agent solving problems autonomously by registering two tools: one that queries cluster memory metrics, and one that checks the direct ConnectX-7 link.
registry = AgentToolRegistry()
@registry.register(
name="verify_interconnect_status",
description="Checks the health and transfer rate of the direct ConnectX-7 cable link.",
parameters_schema={
"type": "object",
"properties": {},
},
)
def verify_interconnect_status() -> Dict[str, Any]:
return {
"interface": "rocep1s0f0",
"link_state": "ACTIVE",
"speed_gbps": 200,
"latency_microseconds": 1.15,
"error_rate": "0.0000%",
}
@registry.register(
name="query_cluster_telemetry",
description="Retrieves active memory and processor metrics from the DGX Spark cluster.",
parameters_schema={
"type": "object",
"properties": {
"metric_type": {
"type": "string",
"description": "Category of metrics: memory, temperature, or compute."
}
},
"required": ["metric_type"],
},
)
def query_cluster_telemetry(metric_type: str) -> Dict[str, Any]:
if metric_type == "memory":
return {
"total_unified_memory_gb": 256,
"allocated_memory_gb": 164,
"free_memory_gb": 92,
"nodes_online": 2,
}
return {"status": "Nominal", "metric_type": metric_type}
local_cluster = ModelEndpointConfig(
name="Local-DGX-Spark-Cluster",
base_url="http://192.168.100.1:8000/v1",
api_key="EMPTY",
model_identifier="NousResearch/Hermes-3-Llama-3.1-70B",
)
cloud_fallback = ModelEndpointConfig(
name="Cloud-OpenRouter-Endpoint",
base_url="https://openrouter.ai/api/v1",
api_key="your-cloud-api-key-here",
model_identifier="anthropic/claude-3.5-sonnet",
)
agent = AutonomousAgentCore(
primary_endpoint=local_cluster,
fallback_endpoint=cloud_fallback,
tool_registry=registry,
)
agent.initialize_system_directive(
"You are an autonomous systems engineering agent running on a clustered NVIDIA DGX Spark platform. "
"Always investigate the actual operational state of the cluster using available tools before answering."
)
user_query = "What is the current health of our DGX Spark cluster and how much memory is available for new models?"
final_response = agent.step(user_query)
print("Agent Response:")
print(final_response)
When this code executes, the agent does not guess an answer. Instead, it inspects its available tools, invokes verify_interconnect_status to ensure the 200 Gb/s link is healthy, calls query_cluster_telemetry to read memory availability, and synthesizes a verified factual answer informing the user that both nodes are active with 92 gigabytes of available unified memory across the 256-gigabyte pool.
PRACTICAL TROUBLESHOOTING GUIDE FOR TWO-NODE CLUSTERS
Even in a well-planned installation, real-world deployment challenges can emerge. Here are solutions for the most frequent issues.
If the ConnectX-7 port state reported by ibstat remains down, confirm that the QSFP cable connector has been seated firmly until the retention clip audibly snaps into place. In back-to-back setups lacking a network switch, both machines must set their interface administratively up using the ip link set command before link negotiation signals begin.
If the vLLM server terminates unexpectedly during long inference sequences, examine the gpu-memory-utilization setting. If background desktop processes or other containers consume memory while vLLM is running, the Linux kernel out-of-memory killer may terminate the process. Lowering this parameter from 0.90 to 0.82 provides a safe memory margin for operating system buffers and agent scratchpads.
If Ray worker nodes fail to establish a connection to the head node over port 6379, ensure that local firewall software, such as ufw, is not filtering traffic on the high-speed interface. Running sudo ufw allow in on rocep1s0f0 immediately permits cluster communication across the internal subnet while leaving your external network interfaces fully secured.
ADDENDUM: ARCHITECTING, DEPLOYING, AND ORCHESTRATING A THREE-NODE DGX SPARK SUPERCOMPUTER
When you expand your desktop cluster from two to three NVIDIA DGX Spark workstations, you enter the tier of extreme compute previously reserved for institutional research laboratories. A tri-node cluster yields 384 gigabytes of unified high-bandwidth memory and delivers three petaFLOPs of four-bit floating-point tensor compute.
This scale unlocks models that cannot fit inside two nodes, including the full-precision 405-billion-parameter Llama 3.1 architecture, massive Mixture-of-Experts architectures like DeepSeek V3 and Mixtral 8x22B, and parallel multi-agent swarms where half a dozen autonomous agents maintain individual 65,000-token context windows simultaneously.
Connecting three workstations introduces a fascinating networking choice. You have two distinct hardware topologies available: the Direct Dual-Port Ring Topology and the Switched High-Speed Fabric Topology.
In the Direct Dual-Port Ring Topology, no external network switch is purchased. Because each DGX Spark workstation features two external QSFP112 ports wired to its ConnectX-7 SmartNIC, three workstations can be cabled directly in a closed triangle. Port one of spark-node-1 cables into port one of spark-node-2. Port two of spark-node-2 cables into port one of spark-node-3. Port two of spark-node-3 cables into port two of spark-node-1.
This topology is economically brilliant because it eliminates thousands of dollars in switch costs. Furthermore, it perfectly aligns with the fundamental mathematical algorithm used by the NVIDIA Collective Communications Library, known as Ring-AllReduce. In Ring-AllReduce, each GPU passes slices of tensor data to its downstream neighbor while receiving data from its upstream neighbor. In a three-node ring, data circulates continuously across all three Blackwell superchips at 200 gigabits per second per link.
The following schematic reveals the physical cabling of the three-node ring cluster.
+-------------------------------------------------------------+
| TRI-NODE DIRECT DUAL-PORT RING |
+-------------------------------------------------------------+
| |
| +-----------------------+ |
| | SPARK NODE 1 (HEAD) | |
| | 128 GB Unified RAM | |
| | Port 1 Port 2 | |
| +---+---------------+---+ |
| | ^ |
| QSFP Cable | | QSFP Cable |
| Link A | | Link C |
| 200 Gb/s v | 200 Gb/s |
| +---+---------------+---+ |
| | | |
| +-----------+-----------+ +---+-------------------+
| | SPARK NODE 2 (WORKER) | | SPARK NODE 3 (WORKER) |
| | 128 GB Unified RAM | | 128 GB Unified RAM |
| | Port 1 Port 2 | | Port 1 Port 2 |
| +-------------------+---+ +---+-------------------+
| | ^ |
| +---------------+ |
| QSFP Cable Link B (200 Gb/s) |
+-------------------------------------------------------------+
In the Switched High-Speed Fabric Topology, each DGX Spark connects one or both of its QSFP ports into a centralized high-speed switch, such as an NVIDIA Quantum-2 InfiniBand switch or an NVIDIA Spectrum-4 Ethernet switch. This star topology provides uniform all-to-all communication paths and simplifies dynamic cluster expansion to four or more nodes.
For the Direct Ring Topology, three independent point-to-point subnets are assigned across the three physical cables. Link A connecting node one and node two uses subnet 192.168.101.0/30. Link B connecting node two and node three uses subnet 192.168.102.0/30. Link C connecting node three and node one uses subnet 192.168.103.0/30.
To automate this multi-node environment, configure both ConnectX-7 interfaces with MTU 9000, enforce system parity, set NCCL to utilize the native Ring algorithm across both network adapters, and launch the tri-node Ray fabric.
The tri-node automation script, saved as tri_cluster_setup.sh in your tutorial workspace, is presented below.
#!/usr/bin/env bash
set -euo pipefail
NODE_ID="${1:-1}"
TOPOLOGY="${2:-ring}"
RAY_PORT="6379"
echo "======================================================================"
echo "[STEP 1] Validating Node Identity and Network Mode..."
echo "======================================================================"
echo "Configuring Node: ${NODE_ID} (Topology: ${TOPOLOGY})"
if [ "${TOPOLOGY}" = "ring" ]; then
PORT1="rocep1s0f0"
PORT2="rocep1s0f1"
sudo ip link set "${PORT1}" mtu 9000 up
sudo ip link set "${PORT2}" mtu 9000 up
if [ "${NODE_ID}" = "1" ]; then
sudo ip addr flush dev "${PORT1}" || true
sudo ip addr flush dev "${PORT2}" || true
sudo ip addr add 192.168.101.1/30 dev "${PORT1}"
sudo ip addr add 192.168.103.2/30 dev "${PORT2}"
HEAD_IP="192.168.101.1"
export NCCL_SOCKET_IFNAME="rocep1s0f0,rocep1s0f1"
elif [ "${NODE_ID}" = "2" ]; then
sudo ip addr flush dev "${PORT1}" || true
sudo ip addr flush dev "${PORT2}" || true
sudo ip addr add 192.168.101.2/30 dev "${PORT1}"
sudo ip addr add 192.168.102.1/30 dev "${PORT2}"
NODE_IP="192.168.101.2"
HEAD_IP="192.168.101.1"
export NCCL_SOCKET_IFNAME="rocep1s0f0,rocep1s0f1"
elif [ "${NODE_ID}" = "3" ]; then
sudo ip addr flush dev "${PORT1}" || true
sudo ip addr flush dev "${PORT2}" || true
sudo ip addr add 192.168.102.2/30 dev "${PORT1}"
sudo ip addr add 192.168.103.1/30 dev "${PORT2}"
NODE_IP="192.168.102.2"
HEAD_IP="192.168.101.1"
export NCCL_SOCKET_IFNAME="rocep1s0f0,rocep1s0f1"
fi
else
PORT1="rocep1s0f0"
sudo ip addr flush dev "${PORT1}" || true
sudo ip link set "${PORT1}" mtu 9000 up
NODE_IP="192.168.100.${NODE_ID}"
HEAD_IP="192.168.100.1"
sudo ip addr add "${NODE_IP}/24" dev "${PORT1}"
export NCCL_SOCKET_IFNAME="${PORT1}"
fi
echo "======================================================================"
echo "[STEP 2] Configuring NCCL for Tri-Node RDMA Communication..."
echo "======================================================================"
export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=0
export NCCL_IB_HCA="mlx5_0:1,mlx5_1:1"
export NCCL_IB_GID_INDEX=3
export NCCL_ALGO=Ring
echo "======================================================================"
echo "[STEP 3] Initializing Tri-Node Ray Cluster..."
echo "======================================================================"
ray stop --force 2>/dev/null || true
if [ "${NODE_ID}" = "1" ]; then
echo "Starting Tri-Node Head Node on ${HEAD_IP}:${RAY_PORT}..."
ray start --head \
--node-ip-address="${HEAD_IP}" \
--port="${RAY_PORT}" \
--disable-usage-stats
echo "Tri-node Ray Head Node is actively listening."
else
echo "Joining Tri-Node Cluster at ${HEAD_IP}:${RAY_PORT}..."
ray start \
--address="${HEAD_IP}:${RAY_PORT}" \
--node-ip-address="${NODE_IP}" \
--disable-usage-stats
echo "Worker node ${NODE_ID} successfully joined the tri-node cluster fabric."
fi
echo "======================================================================"
echo "[SUCCESS] Node ${NODE_ID} configured in ${TOPOLOGY} mode."
echo "======================================================================"
Execute this script on spark-node-1 to establish the head node.
bash tri_cluster_setup.sh 1 ring
Next, execute the script on spark-node-2.
bash tri_cluster_setup.sh 2 ring
Finally, execute the script on spark-node-3.
bash tri_cluster_setup.sh 3 ring
Inspect the aggregated cluster resources from spark-node-1.
ray status
The command returns an active cluster comprising sixty CPU cores, three Blackwell GPUs, and 384 gigabytes of unified high-bandwidth memory.
To serve models across all three nodes, launch vLLM with pipeline parallelism set to three. In this configuration, the layers of the model are partitioned into three equal segments. Node one executes the initial third of the layers, streams intermediate activations over the first ConnectX-7 link to node two, node two processes the middle third and forwards activations to node three, and node three completes token emission.
python3 -m vllm.entrypoints.openai.api_server \
--model NousResearch/Hermes-3-Llama-3.1-70B \
--pipeline-parallel-size 3 \
--tensor-parallel-size 1 \
--max-model-len 131072 \
--gpu-memory-utilization 0.90 \
--port 8000 \
--host 0.0.0.0
With 384 gigabytes of memory at your disposal, the maximum context window can be expanded to 131,072 tokens. The cluster listens on port 8000 of spark-node-1, presenting an OpenAI-compatible endpoint that serves user queries and powers Hermes Agent with unprecedented local performance.
EPILOGUE: THE 5FRONTIER ON YOUR DESK
You now possess the foundational knowledge and precise operational instructions to deploy, cable, configure, and orchestrate one, two, or three NVIDIA DGX Spark workstations.
By mastering hardware setup, operating system parity, ConnectX-7 RDMA networking, distributed Ray orchestration, and autonomous agent loops, you have freed your work from cloud dependency and high-voltage server rooms. You have placed an authentic AI supercomputing fabric directly on your desk, ready to pioneer the next generation of artificial intelligence.
No comments:
Post a Comment