Skip to main content

Documentation Index

Fetch the complete documentation index at: https://apidoc.cometapi.com/llms.txt

Use this file to discover all available pages before exploring further.

Promptfoo is a framework for prompt evaluation, regression testing, and AI red teaming. Promptfoo includes a cometapi provider that calls CometAPI through OpenAI-compatible endpoints.

​
Prerequisites

​
Configure the integration

1

Set your CometAPI API key

Set COMETAPI_KEY before you run evaluations:
read -rsp "CometAPI API key: " COMETAPI_KEY
printf '\n'
export COMETAPI_KEY
2

Choose a provider format

Promptfoo uses the following provider ID format for CometAPI:
providers:
  - cometapi:<type>:<model>
Use these provider types:
TypeUse case
chatChat completions, vision, and multimodal prompts
completionText completion models
embeddingText embedding evaluations
imageImage generation evaluations
You can also use cometapi:your-model-id for the default chat mode.
3

Run a CLI evaluation

The following command runs a one-prompt chat evaluation:
npx promptfoo@latest eval \
  --prompts "Write a concise release note for {{feature}}" \
  --vars feature="a unified API dashboard" \
  -r cometapi:chat:your-model-id
4

Create a Promptfoo config file

The following promptfooconfig.yaml evaluates the same prompt against a CometAPI model:
prompts:
  - "Classify this support request: {{message}}"

providers:
  - id: cometapi:chat:your-model-id
    config:
      temperature: 0.2
      max_tokens: 256

tests:
  - vars:
      message: "The API key works locally but fails in production."
    assert:
      - type: contains-any
        value:
          - authentication
          - configuration
Run the config file with Promptfoo:
npx promptfoo@latest eval -c promptfooconfig.yaml

​
Provider examples

Use a provider type that matches the model behavior that you want to evaluate: 
providers:
  - cometapi:chat:your-model-id
  - cometapi:completion:your-model-id
  - cometapi:embedding:your-model-id
  - cometapi:image:your-model-id
For multimodal tests, keep the chat provider type and pass image variables through Promptfoo test variables.

​
Troubleshooting

Confirm that COMETAPI_KEY is exported in the same shell session that runs promptfoo eval.
Use chat for conversational and multimodal models, embedding for embedding models, and image for image generation models.
Replace your-model-id with an exact model ID from the CometAPI Models page.