node-transfer

High-speed, memory-efficient file transfer between OpenClaw nodes using native Node.js streams.

📋 Table of Contents

• Problem Solved
• Architecture
• Requirements
• Installation
• Usage
• API Reference
• Troubleshooting

---

🎯 Problem Solved

The Original Problem

When transferring large files between OpenClaw nodes using the standard nodes.invoke mechanism, we encountered several critical issues:

The Solution

node-transfer uses native HTTP streaming with Node.js streams, providing:

• ✅ Zero memory overhead - Files stream directly from disk to network
• ✅ No Base64 encoding - Raw binary transfer
• ✅ Speed - Line-speed limited only by network bandwidth
• ✅ Install Once, Run Many - Scripts persist on nodes after first deployment

Performance Comparison

---

🏗️ Architecture

How It Works

┌──────────────┐     HTTP Stream      ┌──────────────┐
│  send.js     │ ◄──────────────────► │ receive.js   │
│  (Source)    │   (Token-protected)  │ (Destination)│
└──────────────┘                      └──────────────┘
       │                                     │
       ▼                                     ▼
┌──────────────┐                      ┌──────────────┐
│  Read Stream │                      │ Write Stream │
│  (fs.create  │                      │ (fs.create   │
│   ReadStream)│                      │  WriteStream)│
└──────────────┘                      └──────────────┘
       │                                     │
       ▼                                     ▼
┌──────────────┐                      ┌──────────────┐
│  File on     │                      │  File on     │
│  Disk        │                      │  Disk        │
└──────────────┘                      └──────────────┘

Security Model

1. One-time Token: 256-bit cryptographically random token (64 hex chars)
2. Single Connection: Only one download allowed per token
3. Auto-shutdown: Server closes after transfer completes or disconnects
4. Token Validation: Every request must include the correct token

Data Flow

1. Sender (send.js):

   • Generates random port and security token
   • Starts HTTP server on ephemeral port
   • Streams file directly from disk to HTTP response
   • Auto-shutdown after transfer or timeout (5 min default)

2. Receiver (receive.js):

   • Connects to sender URL with token
   • Streams HTTP response directly to disk
   • Reports progress, speed, and completion status
   • Validates received bytes match expected size

---

📦 Requirements

• Node.js: 14.0.0 or higher
• Network: TCP connectivity between nodes (any port 1024-65535)
• Firewall: Must allow outbound connections and inbound on ephemeral ports
• Disk Space: Sufficient space on destination for received files

---

🚀 Installation

The "Install Once" Pattern

Instead of deploying scripts on every transfer, we deploy them once per node and use a fast version check for subsequent transfers.

Method 1: Using deploy.js (Recommended)

# Generate deployment script for a target node
node deploy.js E3V3

# This outputs a PowerShell script that you can execute via nodes.invoke()

Method 2: Manual Deployment

On each target node, create the directory and copy files:

# Create directory
mkdir C:/openclaw/skills/node-transfer/scripts -Force

# Copy these files (ensure UTF-8 without BOM encoding):
# - send.js
# - receive.js
# - ensure-installed.js
# - version.js

Method 3: Via OpenClaw Agent

// 1. Check if already installed (< 100ms)
const check = await nodes.invoke({
    node: 'E3V3',
    command: ['node', 'C:/openclaw/skills/node-transfer/scripts/ensure-installed.js', 
              'C:/openclaw/skills/node-transfer/scripts']
});

const checkResult = JSON.parse(check.output);

if (!checkResult.installed) {
    // 2. Deploy if needed (one-time, ~30 seconds)
    // Use the deploy.js output or manually copy files
    console.log('Deploying node-transfer to E3V3...');
    // ... deployment code ...
}

---

💡 Usage

Basic Transfer Workflow

const INSTALL_DIR = 'C:/openclaw/skills/node-transfer/scripts';
const SOURCE_NODE = 'E3V3';
const DEST_NODE = 'E3V3-Docker';

// Step 1: Check installation on both nodes (fast!)
const [sourceCheck, destCheck] = await Promise.all([
    nodes.invoke({
        node: SOURCE_NODE,
        command: ['node', `${INSTALL_DIR}/ensure-installed.js`, INSTALL_DIR]
    }),
    nodes.invoke({
        node: DEST_NODE,
        command: ['node', `${INSTALL_DIR}/ensure-installed.js`, INSTALL_DIR]
    })
]);

// Deploy if needed (usually only once per node ever)
// ... deployment code if not installed ...

// Step 2: Start sender on source node
const sendResult = await nodes.invoke({
    node: SOURCE_NODE,
    command: ['node', `${INSTALL_DIR}/send.js`, 'C:/data/large-file.zip']
});

const { url, token, fileSize, fileName } = JSON.parse(sendResult.output);

// Step 3: Start receiver on destination node
const receiveResult = await nodes.invoke({
    node: DEST_NODE,
    command: ['node', `${INSTALL_DIR}/receive.js`, url, token, '/incoming/file.zip']
});

const result = JSON.parse(receiveResult.output);
console.log(`Transferred ${result.bytesReceived} bytes in ${result.duration}s at ${result.speedMBps} MB/s`);

Using the Command Line

Sender

node send.js /path/to/file.zip

Output:

{
  "url": "http://192.168.1.10:54321/transfer",
  "token": "a1b2c3d4e5f6789...",
  "fileSize": 1073741824,
  "fileName": "file.zip",
  "sourceIp": "192.168.1.10",
  "port": 54321,
  "version": "1.0.0"
}

Options:

node send.js /path/to/file.zip --port 8080 --timeout 10
node send.js --help
node send.js --version

Receiver

node receive.js "http://192.168.1.10:54321/transfer" "token-here..." /path/to/save.zip

Output:

{
  "success": true,
  "bytesReceived": 1073741824,
  "totalBytes": 1073741824,
  "duration": 8.42,
  "speedMBps": 121.5,
  "outputPath": "/path/to/save.zip"
}

Options:

node receive.js <url> <token> <output> --timeout 60 --no-progress
node receive.js --help
node receive.js --version

---

📚 API Reference

send.js

Starts an HTTP server to stream a file.

Usage: node send.js <filePath> [options]

Arguments:

• filePath (required): Path to the file to send

Options:

• --port <n>: Use specific port (default: random ephemeral)
• --timeout <n>: Timeout in minutes (default: 5)

Output (JSON):

Exit Codes:

• 0: Success (transfer completed or info displayed)
• 1: Error (check stderr for JSON error details)

Error Output (JSON):

{
  "error": "ERROR_CODE",
  "message": "Human-readable description"
}

Error codes: FILE_NOT_FOUND, NOT_A_FILE, SERVER_ERROR, TIMEOUT, READ_ERROR, RESPONSE_ERROR

---

receive.js

Connects to a sender and downloads a file.

Usage: node receive.js <url> <token> <outputPath> [options]

Arguments:

• url (required): URL from send.js output
• token (required): Security token from send.js output
• outputPath (required): Path to save the received file

Options:

• --timeout <n>: Connection timeout in seconds (default: 30)
• --no-progress: Suppress progress updates

Output (JSON):

Progress Updates (when not using --no-progress):

{
  "progress": true,
  "receivedBytes": 536870912,
  "totalBytes": 1073741824,
  "percent": 50,
  "speedMBps": 125.4
}

Exit Codes:

• 0: Success
• 1: Error (check stderr for JSON error details)

Error codes: INVALID_ARGS, INVALID_URL, CONNECTION_ERROR, HTTP_ERROR, TIMEOUT, WRITE_ERROR, SIZE_MISMATCH, FILE_EXISTS, NO_DATA

---

ensure-installed.js

Fast check if node-transfer is installed on a node.

Usage: node ensure-installed.js <targetDir>

Arguments:

• targetDir (required): Directory to check

Output (JSON):

Installed:

{
  "installed": true,
  "version": "1.0.0",
  "message": "node-transfer is installed and up-to-date"
}

Needs installation:

{
  "installed": false,
  "missing": ["send.js"],
  "mismatched": [],
  "currentVersion": null,
  "requiredVersion": "1.0.0",
  "action": "DEPLOY",
  "message": "Installation needed: 1 missing, 0 outdated"
}

Exit Codes:

• 0: Already installed and up-to-date
• 1: Needs installation/update
• 2: Error (invalid directory, etc.)

---

deploy.js

Generates deployment scripts for the main agent.

Usage: node deploy.js <nodeId> [targetDir]

Output: JSON with:

• script: PowerShell script to deploy files
• escapedScript: Escaped version for command-line use
• usage: Example code for JavaScript and CLI usage

---

🔧 Troubleshooting

"Connection timeout"

Cause: Network connectivity issue or firewall blocking connection.

Solutions:

• Verify both nodes can reach each other
• Check firewall rules allow outbound connections
• Try specifying a specific port with --port
• Increase timeout with --timeout

"403 Forbidden: Invalid or missing token"

Cause: Token mismatch or URL manipulation.

Solutions:

• Use the exact token from send.js output
• Don't modify the URL
• Ensure the token hasn't expired (sender times out after 5 minutes)

"409 Conflict: Transfer already in progress"

Cause: Multiple connections attempted with same token.

Solutions:

• Each sender URL/token can only be used once
• Start a new sender if you need to retry

"FILE_NOT_FOUND" or "NOT_A_FILE"

Cause: Invalid file path on sender.

Solutions:

• Use absolute paths
• Verify file exists
• Check file permissions

"SIZE_MISMATCH"

Cause: Connection interrupted or network error.

Solutions:

• Retry the transfer
• Check network stability
• The partial file is automatically cleaned up

"Hash mismatch" during ensure-installed

Cause: Files were modified or corrupted.

Solutions:

• Re-deploy scripts using deploy.js
• Ensure files are copied without modification
• Check encoding (must be UTF-8 without BOM)

Slow transfers on subsequent runs

Cause: Not using ensure-installed.js check pattern.

Solutions:

• Always check installation first (< 100ms)
• Only deploy if installed: false
• Follow the "Install Once, Run Many" pattern

---

📄 Files

---

🤝 Contributing

See CONTRIBUTING_PROPOSAL.md for information on how this could be integrated into OpenClaw core.

---

Built for OpenClaw - No Base64, No OOM, No Waiting.