Metric
Create metrics bottom-up with plain-object factory APIs.
Metric
Factory APIs are the preferred way to create metrics. These functions provide type-safe construction of metric definitions.
Import
import {
defineBaseMetric,
withNormalization,
withMetadata,
defineSingleTurnCode,
defineSingleTurnLLM,
defineMultiTurnCode,
defineMultiTurnLLM,
} from '@tally-evals/tally';defineBaseMetric()
Creates the foundational metric definition with name, value type, and optional metadata. This is the starting point for all metrics.
Unique metric name (used as identifier in reports).
The type of value this metric produces.
Human-readable description for UI/reporting.
Custom metadata attached to the metric.
Normalization configuration for converting raw values to 0–1 scores.
Normalizer specification object or custom function.
Static context or async function to derive calibration data from the dataset.
withNormalization()
Attaches a normalization strategy to an existing metric definition. Use this to convert raw values to a 0–1 score for consistent comparison and scoring.
The metric or base metric to attach normalization to.
Normalizer specification or custom function. See Normalizers reference for available specs.
Static calibration context or async function to derive context from raw values.
withMetadata()
Adds or merges metadata into an existing metric definition. Useful for tagging metrics with custom properties for reporting or debugging.
The metric or base metric to annotate.
Metadata to merge into the metric definition.
defineSingleTurnCode()
Creates a single-turn metric that uses TypeScript code to compute values. Best for deterministic checks like JSON validation, keyword presence, or response length.
Base metric definition (from defineBaseMetric).
Function that computes the metric value from preprocessed data.
Optional function to prepare the target before compute. Default exposes { input, output }.
Other metrics this metric depends on (for execution ordering).
Whether results can be cached for identical inputs.
Normalization configuration (overrides base if provided).
Additional metadata (merged with base metadata).
Custom aggregators for summarizing results. Default aggregators are added based on valueType.
defineSingleTurnLLM()
Creates a single-turn metric that uses an LLM to evaluate quality. Best for semantic judgments like relevance, tone, helpfulness, or toxicity detection.
Base metric definition (from defineBaseMetric).
AI SDK language model instance or factory function.
Prompt template for LLM evaluation.
Template string with {{variable}} placeholders.
List of variable names used in the template.
Few-shot examples for the LLM.
Optional function to prepare the target before LLM evaluation.
Optional rubric to guide LLM scoring consistency.
Scoring criteria description.
Scale description (e.g., "1-5").
Example scores with reasoning.
Optional post-processing of the LLM output.
Whether to normalize the output.
Transform raw LLM string to metric value.
Normalization configuration (overrides base if provided).
Additional metadata (merged with base metadata).
Custom aggregators for summarizing results. Default aggregators are added based on valueType.
defineMultiTurnCode()
Creates a multi-turn metric that uses TypeScript code to analyze entire conversations. Best for deterministic checks across conversation history.
Base metric definition (from defineBaseMetric).
Prepares the conversation for downstream compute. Returns a serializable payload.
Function that computes the metric value from the prepared data.
Other metrics this metric depends on.
Whether results can be cached.
Normalization configuration.
Additional metadata.
defineMultiTurnLLM()
Creates a multi-turn metric that uses an LLM to evaluate entire conversations. Best for assessing goal completion, role adherence, topic drift, and other multi-step behaviors.
Base metric definition (from defineBaseMetric).
Prepares the conversation for LLM evaluation. Returns data to include in the prompt.
AI SDK language model instance or factory function.
Prompt template for LLM evaluation.
Template string with {{variable}} placeholders.
List of variable names used in the template.
Few-shot examples for the LLM.
Optional rubric to guide LLM scoring.
Scoring criteria description.
Scale description.
Example scores with reasoning.
Optional post-processing of the LLM output.
Whether to normalize the output.
Transform raw LLM string to metric value.
Normalization configuration.
Additional metadata.
Example
import { defineBaseMetric, defineSingleTurnLLM, withNormalization } from '@tally-evals/tally';
import { createMinMaxNormalizer } from '@tally-evals/tally/normalization';
import { google } from '@ai-sdk/google';
// Create base metric
const base = defineBaseMetric({
name: 'answerRelevance',
valueType: 'number',
description: 'Measures how relevant the answer is to the query',
});
// Create LLM-based single-turn metric with normalization
const answerRelevance = defineSingleTurnLLM({
base: withNormalization({
metric: base,
normalizer: createMinMaxNormalizer({ min: 0, max: 5, clip: true }),
}),
provider: google('models/gemini-2.5-flash-lite'),
prompt: {
instruction: `Score the relevance of the response to the query on a scale of 0-5.
Query: {{input}}
Response: {{output}}`,
variables: ['input', 'output'] as const,
},
rubric: {
criteria: '0 = completely irrelevant, 5 = perfectly relevant',
scale: '0-5',
examples: [
{ score: 5, reasoning: 'Directly answers the question with accurate information.' },
{ score: 0, reasoning: 'Response has nothing to do with the query.' },
],
},
});