Skip to main content
Promptfoo is a framework for prompt evaluation, regression testing, and AI red teaming. Promptfoo includes a cometapi provider that calls CometAPI through OpenAI-compatible endpoints.

​
Prerequisites

​
Configure the integration

1

Set your CometAPI API key

Set COMETAPI_KEY before you run evaluations:
2

Choose a provider format

Promptfoo uses the following provider ID format for CometAPI:
Use these provider types:You can also use cometapi:your-model-id for the default chat mode.
3

Run a CLI evaluation

The following command runs a one-prompt chat evaluation:
4

Create a Promptfoo config file

The following promptfooconfig.yaml evaluates the same prompt against a CometAPI model:
Run the config file with Promptfoo:

​
Provider examples

Use a provider type that matches the model behavior that you want to evaluate:
For multimodal tests, keep the chat provider type and pass image variables through Promptfoo test variables.

​
Troubleshooting

Confirm that COMETAPI_KEY is exported in the same shell session that runs promptfoo eval.
Use chat for conversational and multimodal models, embedding for embedding models, and image for image generation models.
Replace your-model-id with an exact model ID from the CometAPI Models page.
Last modified on July 2, 2026