name: api-benchmark description: Benchmark token generation speed across multiple LLM API providers. Measures TTFT (Time To First Token), tokens-per-second throughput, and total generation time. Use when comparing performance of different API providers, models, or testing API connectivity. Requires OpenCLAW config with provider definitions. compatibility: Requires Python 3 with requests library. Reads target configuration from ~/.openclaw/openclaw.json. Supports anthropic-messages, openai-completions, and openai-responses API formats. metadata: author: Polar version: "1.0.2" requires: config: - ~/.openclaw/openclaw.json env: - OPENCLAW_CONFIG

API Token Speed Benchmark

This skill benchmarks token generation speed across multiple LLM API providers.

When to use this skill

Use this skill when you need to: - Compare token generation speed across different API providers - Measure latency and throughput of LLM models - Verify API connectivity and authentication - Test new API endpoints or models

How to run benchmarks

List available targets

python3 main.py --targets

Run benchmark on a specific target

python3 main.py run --label <target-label>

Run benchmark on all targets

python3 main.py run --all

Run preflight check (verify API connectivity)

python3 main.py check --label <target-label>
python3 main.py check --all

Options

• -l, --label: Specific target label to benchmark
• -a, --all: Run on all available targets
• -r, --repeat: Number of runs per prompt level (default: 1)
• -c, --category: Run specific prompt category (can repeat: -c short -c medium). Options: short, medium, long
• -q, --quiet: Quiet mode - suppress progress output
• --timeout N: Request timeout in seconds (default: 120)
• --table: Output as formatted table (default: JSON)

Configuration

The tool reads configuration from ~/.openclaw/openclaw.json. Targets are defined in the models.providers section with:

• baseUrl: API base URL
• apiKey: Authentication key (or ${ENV_VAR} to read from environment variable)
• api: API format (anthropic-messages, openai-completions, openai-responses)
• models: List of model configurations

Security Note: Instead of hardcoding API keys in the config file, use environment variable placeholders: - "apiKey": "${ANTHROPIC_API_KEY}" will read from the ANTHROPIC_API_KEY environment variable

Example provider config:

{
  "models": {
    "providers": {
      "my-provider": {
        "baseUrl": "https://api.example.com",
        "apiKey": "sk-xxx",
        "api": "openai-completions",
        "models": [
          { "id": "model-name", "api": "openai-completions" }
        ]
      }
    }
  }
}

Output Metrics

• TTFT (Time To First Token): Latency before first token arrives (seconds)
• TPS (Tokens Per Second): Generation throughput
• Total Time: Full generation duration (seconds)
• Input/Output Tokens: Token counts from API usage data (or estimated at 4 chars/token if not provided by API)

Note: Token counts are reported by the API when available. If the API doesn't return token counts, they are estimated at 4 characters per token.

Example Usage

# Check if a specific target is reachable
python3 main.py check --label my-provider

# Benchmark a single target
python3 main.py run --label my-provider --repeat 3

# Compare all targets
python3 main.py run --all --table

这是一款用于比较不同 AI 模型响应速度的工具，文档清晰、使用方法明确，支持同时测试多个服务商并自动处理网络异常。功能较为完整，提供了连接预检查确保测试顺利。但目前仅支持非流式响应测试，缺少速度结果的可视化图表，配置过程稍显繁琐。整体质量良好，能满足基本的性能对比需求。