Promptfoo + Volcengine Ark Coding Plan: Zero-Cost Prompt Evaluation

If you've shipped anything with LLMs, you've been here: you tweak a prompt, and now you're guessing whether it got better or worse. No data. Just vibes.

Promptfoo is the most active open-source prompt evaluation framework right now. It does matrix testing (multiple prompts × multiple models × multiple test cases), has a built-in assertion engine, and tells you exactly what each change improved or broke. But most tutorials assume you're using OpenAI's paid API — and for individual developers, tens of dollars a month in evaluation costs isn't great.

Volcengine Ark's Coding Plan includes free quota with DeepSeek V4 Pro and other reasoning models. This article covers the full setup process and three real pitfalls I hit along the way.

1. What Is Promptfoo

In one sentence: Promptfoo is a YAML-driven LLM evaluation tool. You define providers (models), prompts (templates), and tests (cases + assertions). It runs everything and gives you a visual report.

Core concepts:

Concept	Description	Example
`providers`	Model APIs to test	DeepSeek V4 Pro, GPT-4o-mini
`prompts`	Prompt templates to compare	Simple translation vs. professional translation
`tests`	Test cases + assertion rules	Input "Hello" → output should contain "你好"
`assert`	Assertion types	`contains`, `not-contains`, `is-json`, `llm-rubric`

Component	Layer	Responsibility
`promptfooconfig.yaml`	Config	Define providers, prompts, tests
Promptfoo CLI	Execution	Concurrent API calls, result collection
Promptfoo View	Display	Web UI visualization
Assertion Engine	Verification	Run assertions on each output

2. Installation — Brew Is 10× Faster Than npm

The first pitfall hit immediately.

Attempt 1: npm global install (failed)

npm install -g promptfoo@latest

Timed out at 120 seconds. Retried with proxy — 180 seconds, still timed out. Local install with npm install --save-dev promptfoo — 600 seconds, 508 packages downloaded, but the .bin directory never materialized. Installation interrupted.

Root cause: Promptfoo's dependency tree is massive (23,830 files, 172.6MB after npm install). npm's download and linking phases are fragile on large trees.

Attempt 2: brew install (success)

brew install promptfoo

30 seconds. Done. Homebrew uses pre-compiled bottles — no dependency resolution or linking overhead.

$ promptfoo --version
0.121.19

Verdict: Use brew on macOS.

3. Configuration — Wiring Up Ark Coding Plan

3.1 Getting API Credentials

Volcengine Ark Coding Plan console → API Key Management → grab the ark-* format key.

Critical: the Coding Plan endpoint is https://ark.cn-beijing.volces.com/api/plan/v3, not the standard Ark /api/v3. Wrong endpoint = 401.

3.2 Config File

# promptfooconfig.yaml
description: "Promptfoo Demo - Translation comparison (Ark Coding Plan)"

providers:
  - id: openai:chat:deepseek-v4-pro
    label: "DeepSeek V4 Pro"
    config:
      apiBaseUrl: https://ark.cn-beijing.volces.com/api/plan/v3
      apiKey: ark-xxx
      temperature: 0.0
      max_tokens: 16384    # Reasoning models need 3-5× output tokens

  - id: openai:chat:deepseek-v4-flash
    label: "DeepSeek V4 Flash"
    config:
      apiBaseUrl: https://ark.cn-beijing.volces.com/api/plan/v3
      apiKey: ark-xxx
      temperature: 0.0
      max_tokens: 16384

prompts:
  - label: "Simple translation"
    raw: |
      Translate the following English into Chinese. Output only the translation:
      {{text}}

  - label: "Professional translation"
    raw: |
      You are a professional translator. Translate the following English into
      natural Chinese, with cultural adaptation. Output only the translation:
      {{text}}

tests:
  - vars:
      text: "Hello, how are you today?"
    assert:
      - type: contains
        value: "你好"

  - vars:
      text: "The quick brown fox jumps over the lazy dog."
    assert:
      - type: contains
        value: "狐狸"
      - type: contains
        value: "狗"

  - vars:
      text: "Machine learning is transforming every industry."
    assert:
      - type: contains
        value: "机器"
      - type: contains
        value: "学习"

  - vars:
      text: "Break a leg at your performance tonight!"
    assert:
      - type: contains
        value: "演出"

  - vars:
      text: "It's raining cats and dogs outside."
    assert:
      - type: contains
        value: "大"

3.3 Three Critical Config Details

① Provider ID format: openai:chat:<model-name>

Promptfoo's openai: prefix defaults to OpenAI's official API. You must use openai:chat: to specify a custom apiBaseUrl. Plain openai:deepseek-v4-pro will throw "API key is not set."

② max_tokens ≥ 16384

DeepSeek V4 Pro is a reasoning model. The API returns:

{
  "message": {
    "content": "你好，你今天好吗？",
    "reasoning_content": "We need to understand the user's request... translate directly."
  }
}

reasoning_content tokens count against max_tokens. At 1024 tokens, reasoning consumes the entire budget and content comes back empty. Translation tasks need 4096–16384 in practice.

③ Model names are lowercase

Ark Coding Plan expects deepseek-v4-pro, not DeepSeek-V4-Pro. Case mismatch = "model not found."

4. Running the Evaluation — Three Iterations to 100%

First run: 0/20 (API key error)

Provider ID was openai:deepseek-v4-pro. Promptfoo tried OpenAI's API, got "API key is not set."

Fix: Changed to openai:chat:deepseek-v4-pro.

Second run: 14/20 (reasoning content contamination)

Six failures fell into two patterns:

Failed test	Assertion	Why it failed
"Break a leg..."	`not-contains: "段腿"`	Reasoning explained the literal meaning "break a leg"
"raining cats and dogs"	`not-contains: "猫和狗"`	Reasoning explained the literal meaning "cats and dogs"

The model's reasoning content explains idioms literally — and those explanations contain the very words we were trying to exclude. not-contains assertions are fundamentally unreliable with reasoning models.

Third run: 20/20 (all passing)

Removed all not-contains assertions, kept only positive contains checks.

┌────────────────────────┬────────────────────────┬────────────────────────┬────────────────────────┬────────────────────────┐
│ text                   │ DeepSeek V4 Pro        │ DeepSeek V4 Pro        │ DeepSeek V4 Flash      │ DeepSeek V4 Flash      │
│                        │ Simple translation     │ Professional trans.     │ Simple translation     │ Professional trans.     │
├────────────────────────┼────────────────────────┼────────────────────────┼────────────────────────┼────────────────────────┤
│ Hello, how are you     │ PASS                   │ PASS                   │ PASS                   │ PASS                   │
│ The quick brown fox... │ PASS                   │ PASS                   │ PASS                   │ PASS                   │
│ Machine learning...    │ PASS                   │ PASS                   │ PASS                   │ PASS                   │
│ Break a leg...         │ PASS                   │ PASS                   │ PASS                   │ PASS                   │
│ It's raining cats...   │ PASS                   │ PASS                   │ PASS                   │ PASS                   │
└────────────────────────┴────────────────────────┴────────────────────────┴────────────────────────┘

✓ 20 passed (100%)
0 failed (0%)
0 errors (0%)
Duration: 20s (concurrency: 4)

Token consumption: 2,894 total, with 2,179 reasoning tokens (75%). The reasoning models "think" a lot, but the translation quality holds up.

5. Pitfall Retrospective

#	Pitfall	Symptom	Root Cause	Fix
1	npm install timeout	Repeated timeouts, broken downloads	Large dependency tree (172MB), slow npm linking	Use brew
2	API key error	"API key is not set"	`openai:` prefix routes to OpenAI's API	Use `openai:chat:` prefix
3	Empty content	Output shows only "Thinking:..."	Reasoning tokens exhausted max_tokens	Set max_tokens ≥ 16384
4	not-contains false positives	Chinese output fails English exclusion	Reasoning content includes original text/explanations	Use positive `contains` only
5	doubao model 404	`doubao-1.5-pro-32k` returns "not supported"	Not included in Coding Plan	Stick to deepseek models

6. Assertion Selection Guide for Reasoning Models

Based on real testing, assertion reliability with reasoning models:

Assertion Type	Works?	Notes
`contains`	✅	Positive inclusion, stable and reliable
`icontains`	✅	Case-insensitive, same as above
`not-contains`	❌	Reasoning content contaminates output with original text
`equals`	❌	Reasoning mixed into output, exact match impossible
`is-json`	✅	Structural validation, unaffected by reasoning
`llm-rubric`	⚠️	Requires grading provider; reasoning may interfere with scoring
`cost`	✅	Cost assertion, independent of content
`latency`	✅	Latency assertion, independent of content

Recommendation: With reasoning models, stick to contains, is-json, cost, and latency. For negative semantics, use positive contains to verify expected behavior rather than not-contains to exclude unwanted behavior.

7. What You Can Build on Top

This setup is trivial to extend:

1. Compare more models

providers:
  - id: openai:chat:deepseek-v4-pro
  - id: openai:chat:deepseek-v4-flash
  - id: openai:chat:gpt-4o-mini          # Needs OpenAI key
  - id: openai:chat:claude-3.5-haiku    # Needs Anthropic key

2. Add more prompt variants

prompts:
  - label: "Zero-shot"
    raw: "Translate: {{text}}"
  - label: "Few-shot"
    raw: |
      EN: Hello → ZH: 你好
      EN: {{text}} → ZH:

3. CI/CD integration

# .github/workflows/prompt-eval.yml
- name: Run promptfoo eval
  run: npx promptfoo eval --no-cache

Every PR that changes a prompt automatically runs regression tests. Failed assertions block the merge.

8. Conclusion

Promptfoo + Ark Coding Plan is a zero-cost prompt evaluation pipeline. Individual developers and small teams can establish prompt regression testing without any paid API.

But reasoning models introduce a unique challenge: reasoning_content bleeds into the output, breaking traditional assertion strategies. This is fundamentally a mismatch between prompt evaluation tools' design assumption (output = final answer) and reasoning models' actual behavior (output = reasoning process + final answer).

Our recommendations:

Use brew on macOS. Don't fight npm's dependency resolution.
Use openai:chat: prefix for provider IDs. Otherwise the API key never reaches your endpoint.
Set max_tokens ≥ 16384 for reasoning models. Give the reasoning process room to breathe.
Use positive contains assertions, not not-contains. Reasoning content will contaminate negative assertions.
Put promptfoo eval in CI. Stop guessing whether your prompt changes helped.

References

Promptfoo Documentation — Configuration guide
Promptfoo GitHub — Open-source repository
Volcengine Ark Platform — Coding Plan console
Ark API Documentation — API reference
Homebrew promptfoo formula — Brew package info