DigitalOcean API (unofficial showcase)
POST/v2/gen-ai/model_evaluation_runs

Create a model evaluation run

Creates a model evaluation run to score a candidate model against a dataset using selected metrics and a judge model. Configure the candidate, dataset, metrics, and judge with the corresponding fields; candidate_model_source defaults to the serverless model source, and epochs defaults to 1 and is capped at 3. A successful response returns the created run's eval_run_uuid.

  • IdempotentThe SDK sends Idempotency-Key, so a retried request is only applied once.

15 body fields

Optional configuration for creating a model evaluation run; all fields are optional.

candidate_inference_configobjectoptional
Inference configuration for the candidate model during evaluation.
candidate_model_namestringoptional
Model slug used to call the candidate model API. For dedicated inference, this is the model slug from the deployment. For serverless, this should match the model's internal name.
candidate_model_sourcestringoptional
Whether the candidate is a served model (serverless platform, a dedicated deployment, or a model router) or an OHS-hosted agent config.
Allowed:CANDIDATE_MODEL_SOURCE_SERVERLESSCANDIDATE_MODEL_SOURCE_DEDICATEDCANDIDATE_MODEL_SOURCE_ROUTERCANDIDATE_MODEL_SOURCE_AGENTDefault:CANDIDATE_MODEL_SOURCE_SERVERLESS
candidate_model_uuidstringoptional
UUID of the candidate model to evaluate.
dataset_uuidstringoptional
UUID of the dataset to use for evaluation.
epochsintegeroptional
Number of times to evaluate each dataset row (n-pass/epochs), so the result reports avg@k/pass@k/cons@k instead of a single score. Defaults to 1 when unset. Capped at 3 until throughput/sharding work lands, since each extra epoch roughly multiplies wall-clock run time. Not persisted on presets.
eval_preset_uuidstringoptional
UUID of an evaluation preset to use for the run.
judge_model_uuidstringoptional
UUID of the judge model used to score responses.
metric_uuidsarray<string>optional
UUIDs of metrics to evaluate (selected from ListModelEvaluationMetrics).
namestringoptional
Name for the evaluation run.
preset_namestringoptional
Name for a reusable preset, if saving one.
preset_save_sectionsarray<string>optional
Which sections of this run's resolved configuration to persist as a reusable preset. Each selected section saves only its own fields; the remaining sections stay empty on the preset and must be supplied inline on future runs that reference it. Empty means do not save a preset (unless the deprecated `save_as_preset` boolean is true, in which case all sections are saved). Ignored when `eval_preset_uuid` is set. Use `preset_name` to label the saved preset.
save_as_presetbooleanoptional
Deprecated: use `preset_save_sections`. When `true` and `preset_save_sections` is empty, all five sections of the resolved configuration are saved as a reusable preset (legacy behavior). Ignored when `eval_preset_uuid` is set.
sourcestringoptional
Source of the run creation (api, sdk, cli).
star_metricobjectoptional
Optional star metric configuration, including the metric UUID, name, and either a numeric `success_threshold` or percentage `success_threshold_pct`.

6 status codes
200Returns an object containing the UUID of the created evaluation run in `eval_run_uuid`.
eval_run_uuidstringoptional
UUID of the created evaluation run.
401Returned when authentication fails because the credentials are invalid.
idstringrequired
A short identifier corresponding to the HTTP status code returned. For example, the ID for a response returning a 404 status code would be "not_found."
messagestringrequired
A message providing additional information about the error, including details to help resolve it when possible.
request_idstringoptional
Optionally, some endpoints may include a request ID that should be provided when reporting bugs or opening support tickets to help identify the issue.
404Returned when a requested resource cannot be found.
idstringrequired
A short identifier corresponding to the HTTP status code returned. For example, the ID for a response returning a 404 status code would be "not_found."
messagestringrequired
A message providing additional information about the error, including details to help resolve it when possible.
request_idstringoptional
Optionally, some endpoints may include a request ID that should be provided when reporting bugs or opening support tickets to help identify the issue.
429Returned when the API rate limit has been exceeded.
idstringrequired
A short identifier corresponding to the HTTP status code returned. For example, the ID for a response returning a 404 status code would be "not_found."
messagestringrequired
A message providing additional information about the error, including details to help resolve it when possible.
request_idstringoptional
Optionally, some endpoints may include a request ID that should be provided when reporting bugs or opening support tickets to help identify the issue.
500Returned when a server error occurs.
idstringrequired
A short identifier corresponding to the HTTP status code returned. For example, the ID for a response returning a 404 status code would be "not_found."
messagestringrequired
A message providing additional information about the error, including details to help resolve it when possible.
request_idstringoptional
Optionally, some endpoints may include a request ID that should be provided when reporting bugs or opening support tickets to help identify the issue.
defaultThere was an unexpected error.
idstringrequired
A short identifier corresponding to the HTTP status code returned. For example, the ID for a response returning a 404 status code would be "not_found."
messagestringrequired
A message providing additional information about the error, including details to help resolve it when possible.
request_idstringoptional
Optionally, some endpoints may include a request ID that should be provided when reporting bugs or opening support tickets to help identify the issue.

Error handling

A 401 is returned when authentication credentials are invalid, and a 404 is returned when a referenced resource cannot be found. A 429 indicates that the API rate limit has been exceeded; a 500 indicates a server error. If supplied, candidate_model_source must be one of CANDIDATE_MODEL_SOURCE_SERVERLESS, CANDIDATE_MODEL_SOURCE_DEDICATED, CANDIDATE_MODEL_SOURCE_ROUTER, or CANDIDATE_MODEL_SOURCE_AGENT; epochs defaults to 1 and is capped at 3.