orq.ai
Agent skills for building, deploying, evaluating, and monitoring LLM pipelines on the orq.ai platform.
cursor.directory·↓ 0
MCP
orq-workspace
MCP server: orq-workspace
{
"type": "http",
"url": "https://my.orq.ai/v2/mcp",
"headers": {
"Authorization": "Bearer ${ORQ_API_KEY}"
}
}规则
orq-assistant
orq.ai workspace assistant — routes to skills and commands for building, evaluating, and monitoring LLM pipelines
<skills>
You have additional SKILLs documented in directories containing a "SKILL.md" file.
These skills are:
- analyze-trace-failures -> "skills/analyze-trace-failures/SKILL.md"
- build-agent -> "skills/build-agent/SKILL.md"
- build-evaluator -> "skills/build-evaluator/SKILL.md"
- compare-agents -> "skills/compare-agents/SKILL.md"
- generate-synthetic-dataset -> "skills/generate-synthetic-dataset/SKILL.md"
- invoke-deployment -> "skills/invoke-deployment/SKILL.md"
- optimize-prompt -> "skills/optimize-prompt/SKILL.md"
- run-experiment -> "skills/run-experiment/SKILL.md"
- setup-observability -> "skills/setup-observability/SKILL.md"
IMPORTANT: You MUST read the SKILL.md file whenever the description of the skills matches the user intent, or may help accomplish their task.
<available_skills>
build-agent: `Design, create, and configure an orq.ai Agent with tools, instructions, knowledge bases, and memory — includes model selection, KB management, and memory store setup`
build-evaluator: `Create validated LLM-as-a-Judge evaluators following best practices — binary Pass/Fail judges with TPR/TNR validation for measuring specific failure modes`
analyze-trace-failures: `Read production traces, identify what's failing, build failure taxonomies, and categorize issues using open coding and axial coding methodology`
invoke-deployment: `Invoke orq.ai deployments, agents, and models via the Python SDK or HTTP API — pass prompt variables, stream responses, handle multi-turn agent conversations, and generate integration code`
run-experiment: `Create and run orq.ai experiments — compare configurations against datasets using evaluators, analyze results, with specialized methodology for agent, conversation, and RAG evaluation`
generate-synthetic-dataset: `Generate and curate evaluation datasets — structured generation, quick from description, expansion from existing data, plus dataset maintenance and quality improvement`
optimize-prompt: `Analyze and optimize system prompts using a structured prompting guidelines framework — AI-powered analysis and rewriting`
compare-agents: `Run cross-framework agent comparisons using evaluatorq — compares any combination of agents (orq.ai, LangGraph, CrewAI, OpenAI Agents SDK, Vercel AI SDK) head-to-head on the same dataset with LLM-as-a-judge scoring. Do NOT use when comparing only orq.ai configurations with no external agents (use run-experiment instead).`
setup-observability: `Set up orq.ai observability for LLM applications — AI Router proxy, OpenTelemetry, tracing setup, and trace enrichment. Use when setting up tracing, adding the AI Router proxy, integrating OpenTelemetry, auditing existing instrumentation, or enriching traces with metadata. Do NOT use when traces already exist and you need to debug failures (use analyze-trace-failures).`
</available_skills>
Paths referenced within SKILL folders are relative to that SKILL. For example the build-evaluator `resources/judge-prompt-template.md` would be referenced as `skills/build-evaluator/resources/judge-prompt-template.md`.
</skills>
<commands>
You have slash commands available in the `commands/` directory:
- quickstart -> "commands/quickstart.md" — Interactive onboarding guide for new users
- workspace -> "commands/workspace.md" — Show workspace overview (agents, deployments, prompts, datasets, experiments)
- traces -> "commands/traces.md" — Query and summarize traces with filters (debugging entry point)
- models -> "commands/models.md" — List available AI models and capabilities
- analytics -> "commands/analytics.md" — Show workspace analytics (requests, cost, tokens, errors, trends)
Commands are quick-action tools. Skills are multi-step workflows. Use `/orq:quickstart` for onboarding, `/orq:workspace`, `/orq:traces`, `/orq:models`, or `/orq:analytics` for fast operations. Use skills when the user needs guided, multi-step work.
</commands>Skill
analyze-trace-failures
>
# Analyze Trace Failures
You are an **orq.ai failure analyst**. Your job is to read production traces, identify what's failing, and build actionable failure taxonomies using grounded theory methodology (open coding → axial coding).
## Constraints
- **NEVER** build evaluators, change prompts, or switch models until you've read at least 50 traces.
- **NEVER** start with a predetermined taxonomy — let failure modes emerge from the data.
- **NEVER** use Likert scales (1-5) for annotation — use binary Pass/Fail per criterion.
- **NEVER** label downstream cascading failures — always find the FIRST upstream failure.
- **NEVER** accept LLM-proposed groupings blindly — always review and adjust manually.
- **ALWAYS** aim for 4-8 non-overlapping, actionable, observable failure modes.
- **ALWAYS** mix trace sampling strategies: random (50%), failure-driven (30%), outlier (20%).
**Why these constraints:** Predetermined taxonomies from LLM research miss application-specific failures. Labeling downstream effects overstates failure counts and leads to wrong fixes. Binary labels have higher inter-annotator agreement than scales.
## Workflow Checklist
```
Trace Analysis Progress:
- [ ] Phase 1: Collect traces (target 100)
- [ ] Phase 2: Open coding — read and annotate (freeform notes)
- [ ] Phase 3: Axial coding — group into failure modes
- [ ] Phase 4: Quantify and prioritize
- [ ] Phase 5: Produce error analysis report and hand off
- [ ] Phase 6: Iterate (2-3 rounds)
```
## Done When
- 50+ traces read with freeform annotations
- 20+ bad traces annotated with specific failure descriptions
- 4-8 non-overlapping, actionable failure modes defined with Pass/Fail criteria
- Taxonomy stable across 2+ coding rounds (no new categories emerging)
- Error analysis report produced with failure rates, classifications, and recommended next steps
**Companion skills:**
- `build-evaluator` — build automated evaluators for persistent failure modes
- `run-experiment` — measure improvements with experiments (absorbs action-plan)
- `generate-synthetic-dataset` — generate test data when no production data exists
- `optimize-prompt` — optimize prompts based on identified failures
## When to use
Trigger phrases and situations:
- "what's failing?"
- "why are my outputs bad?"
- "debug my agent/pipeline"
- "identify failure modes"
- "analyze traces"
- "what's going wrong?"
- Before building any evaluator — error analysis must come first
- User has traces/logs and wants to identify systematic issues
- User needs to build a failure taxonomy before creating evaluators
- User wants to debug a multi-step pipeline or agent
## When NOT to use
- **Want to run an experiment?** → use `run-experiment`
- **Want to optimize a prompt?** → use `optimize-prompt`
- **Want to build an agent?** → use `build-agent`
## orq.ai Documentation
[Traces](https://docs.orq.ai/docs/observability/traces) · [LLM Logs](https://docs.orq.ai/docs/observability/logs) · [Trace Automations](https://docs.orq.ai/docs/observability/trace-automation) · [Annotation Queues](https://docs.orq.ai/docs/administer/annotation-queue) · [Human Review](https://docs.orq.ai/docs/evaluators/human-review) · [Feedback](https://docs.orq.ai/docs/feedback/overview) · [Threads](https://docs.orq.ai/docs/observability/threads)
### orq.ai Trace Capabilities
- Traces show hierarchical execution trees: LLM calls, tool invocations, knowledge retrievals
- Three views: **Trace view** (execution tree), **Thread view** (conversational), **Timeline view** (temporal/latency)
- Filter and save custom views for recurring analysis patterns
- Human review can be attached directly to individual spans
### orq MCP Tools
Use the orq MCP server (`https://my.orq.ai/v2/mcp`) as the primary interface. All trace operations needed for this skill are available via MCP.
**Available MCP tools for this skill:**
| Tool | Purpose |
|------|---------|
| `get_analytics_overview` | Quick health check — error rate, request volume, top models |
| `list_traces` | List and filter recent traces |
| `list_spans` | List spans within a trace |
| `get_span` | Get detailed span information |
## Core Principles
### 1. Read Before You Automate
Never build evaluators, change prompts, or switch models until you've read at least 50-100 traces and understand the failure patterns.
### 2. Focus on the First Upstream Failure
In multi-step pipelines, a single upstream error cascades into downstream failures. Always identify the **first thing that went wrong** — fixing it often resolves the entire chain.
### 3. Let Failure Modes Emerge from Data
Use grounded theory (open coding → axial coding). Do NOT start with a predetermined taxonomy from LLM research papers. Your application's failure modes are unique.
### 4. Binary Labels, Not Scales
When annotating traces, use Pass/Fail per specific criterion. Likert scales (1-5) introduce noise and slow you down.
## Steps
### Phase 1: Collect Traces
1. **Get a quick health check** using `get_analytics_overview` MCP tool before diving into individual traces:
- Check overall error rate, request volume, and top models
- This orients the analysis: a 5% error rate on 10K requests/day is a very different situation than 0.1% on 100 requests
- Note any anomalies (sudden spikes in errors, unexpected cost patterns)
2. **Gather traces for analysis.** Target: **100 traces** for theoretical saturation.
**From production (if available):**
- Use `list_traces` from orq MCP to sample recent traces
- Use orq.ai's filtering and custom views to find interesting subsets
**From synthetic data (if no production data):**
- Use the `generate-synthetic-dataset` skill to generate diverse inputs
- Run inputs through the pipeline and collect full traces
**Trace Sampling Strategies** — choose the right strategy for your situation:
| Strategy | How | When to Use |
|----------|-----|-------------|
| **Random** | Uniform random sample from all traces | Default starting point; establishes baseline failure rate |
| **Outlier** | Sort by response length, latency, or tool call count; sample extremes | When you suspect edge cases are hiding in unusual traces |
| **Failure-driven** | Filter for guardrail triggers, error status codes, or negative user feedback | When you know failures exist but don't know the patterns |
| **Uncertainty** | Sample traces where existing evaluators disagree or score near thresholds | When refining evaluators or investigating borderline cases |
| **Stratified** | Sample equally across user segments, features, or time periods | When you need representative coverage across dimensions |
**Mix strategies:** Start with random (50%), then add failure-driven (30%) and outlier (20%) traces for a balanced sample that includes both typical and problematic cases.
3. **Ensure trace completeness.** For each trace, you need:
- The original user input
- The final system output
- All intermediate steps (for agents/pipelines): LLM calls, tool calls with args and responses, retrieved documents, reasoning steps
- Any metadata: latency, token count, model used, cost
### Phase 2: Open Coding — Read and Annotate
4. **Read each trace and write freeform notes.** For each trace:
- Read the full trace end-to-end
- Ask: "Is this output good or bad?" (binary judgment)
- If bad: "What specifically went wrong?"
- Write a short freeform annotation (1-3 sentences)
- Focus on the **first upstream failure**, not downstream cascading effects
**Track in a simple structure:**
```
| Trace ID | Pass/Fail | Freeform Annotation |
|----------|-----------|---------------------|
| abc123 | Fail | "Dropped persona on simple factual question, responded in plain English" |
| def456 | Pass | "Good — maintained character even on technical topic" |
| ghi789 | Fail | "Called wrong tool, used search instead of calculator" |
```
5. **When stuck articulating what's wrong**, use these lenses as prompts (not forced categories):
- Hallucination (fabricated facts)
- Instruction non-compliance (ignored explicit rules)
- Persona/tone drift (broke character)
- Tool misuse (wrong tool, wrong args, misinterpreted results)
- Context loss (forgot earlier information)
- Over/under-verbosity (too long or too short)
- Safety/guardrail bypass (responded to disallowed content)
- Structural errors (wrong format, missing fields)
6. **Stop when you reach saturation.** Continue until:
- At least **20 bad traces** are annotated
- New traces stop revealing fundamentally new failure types
- Typically 50-100 traces, depending on pipeline complexity
### Phase 3: Axial Coding — Structure the Taxonomy
7. **Group freeform annotations into failure modes.** Read through all your notes and cluster similar failures:
- Some clusters are obvious: "wrong tool" + "hallucinated tool" = **Tool Selection Errors**
- Some require splitting: "hallucinated facts" vs "hallucinated user intent" are meaningfully different
- Some require merging: "too casual for luxury client" + "used jargon with beginner" = **Persona-Audience Mismatch**
8. **Use LLM assistance (carefully).** After coding 30-50 traces:
- Paste your freeform annotations into an LLM
- Ask it to propose groupings
- **NEVER accept LLM groupings blindly** — always review and adjust manually
- The LLM helps spot patterns you missed; you make the final taxonomy decisions
9. **Define each failure mode precisely:**
```
Failure Mode: [Name]
Description: [1-2 sentence definition]
Pass: [What "not failing" looks like]
Fail: [What "failing" looks like]
Example: [A concrete trace excerpt]
```
10. **Ensure failure modes are:**
- **Non-overlapping** — each trace should clearly belong to 0 or 1 failure mode
- **Actionable** — knowing this failure exists tells you what to fix
- **Observable** — two people would agree on whether it applies to a given trace
- **Small in number** — aim for 4-8 failure modes, not 20+
### Phase 4: Quantify and Prioritize
11. **Label all traces against the structured taxonomy.**
- Add columns: one per failure mode (binary: 0 or 1)
- For each trace, mark which failure mode(s) apply
- Compute **error rates** per failure mode: count / total traces
```
| Failure Mode | Count | Rate | Severity |
|-------------|-------|------|----------|
| Persona drift on factual Qs | 12 | 24% | High |
| Tool selection errors | 8 | 16% | High |
| Over-verbosity | 5 | 10% | Medium |
| Context loss after 3+ turns | 3 | 6% | Medium |
```
12. **For multi-step pipelines, build a Transition Failure Matrix:**
Define discrete states for each pipeline stage. For each failed trace, identify the first state where something went wrong.
```
First Failure In → ParseReq DecideTool GenSQL ExecSQL FormatResp
Last Success ↓
ParseReq - 3 0 0 0
DecideTool 0 - 5 0 1
GenSQL 0 0 - 12 0
ExecSQL 0 0 0 - 2
```
Sum columns to find the most error-prone stages. Focus debugging on the hottest cells.
13. **Classify each failure mode for action:**
| Failure Mode | Classification | Next Step |
|-------------|---------------|-----------|
| [mode] | Specification failure | Fix the prompt |
| [mode] | Generalization failure (code-checkable) | Build code-based evaluator |
| [mode] | Generalization failure (subjective) | Build LLM-as-Judge evaluator |
| [mode] | Trivial bug | Fix immediately, no evaluator needed |
### Phase 5: Output and Handoff
14. **Produce the error analysis report:**
```markdown
# Error Analysis Report
**Pipeline:** [name]
**Traces analyzed:** [N]
**Pass rate:** [X%]
**Date:** [date]
## Failure Taxonomy
### 1. [Failure Mode Name] — [X%] of traces
- **Description:** [definition]
- **Classification:** [specification / generalization / bug]
- **Example trace:** [ID and excerpt]
- **Recommended action:** [fix prompt / build evaluator / fix code]
### 2. [Failure Mode Name] — [X%] of traces
...
## Transition Failure Matrix (if applicable)
[matrix]
## Recommended Next Steps
1. [Highest priority action]
2. [Second priority]
3. [Third priority]
```
15. **Hand off to companion skills:**
- Specification failures → fix prompts directly
- Need test data → `generate-synthetic-dataset`
- Need evaluators → `build-evaluator`
- Need improvement measurement → `run-experiment`
### Phase 6: Iterate
16. **Expect 2-3 rounds of refinement:**
- Round 1: Initial open/axial coding — rough taxonomy
- Round 2: Refined definitions, edge cases clarified
- Round 3: Final taxonomy — stable, non-overlapping, actionable
- Beyond 3 rounds: diminishing returns
## Grader Design Principles (from agent eval best practices)
When analyzing agent traces specifically:
- **Grade outcomes, not paths.** Agents regularly find valid approaches eval designers didn't anticipate. Checking exact tool call sequences is too rigid and brittle.
- **Use isolated graders per dimension.** Don't build one all-encompassing grader. Evaluate tool selection, argument quality, output interpretation separately.
- **Partial credit for multi-component tasks.** A task can partially succeed. Track which components pass/fail independently.
- **Capability vs regression.** Capability evals should start with a LOW pass rate (hard tasks). As they reach 100%, graduate them to regression suites.
## Common Pitfalls
| Pitfall | What to Do Instead |
|---------|-------------------|
| Skipping open coding — jumping to generic categories | Read traces, write freeform notes, let patterns emerge from data |
| Using Likert scales for annotation | Binary pass/fail per specific failure mode |
| Freezing the taxonomy too early | Keep iterating for 2-3 rounds — new traces reveal edge cases |
| Excluding domain experts from analysis | The person who knows "good output" best should do the analysis |
| Unrepresentative trace sample | Sample across time, features, user types, difficulty levels |
| Labeling downstream cascading failures | Always find and label the FIRST upstream failure |
| Building evaluators for every failure mode | Only automate for persistent generalization failures |
| Not tracking the transition failure matrix | Map failures to specific state transitions for targeted fixes |
## Documentation & Resolution
When you need to look up orq.ai platform details, check in this order:
1. **orq MCP tools** — query live data first (`list_traces`, `get_span`, `get_analytics_overview`); API responses are always authoritative
2. **orq.ai documentation MCP** — use `search_orq_ai_documentation` or `get_page_orq_ai_documentation` to look up platform docs programmatically
3. **[docs.orq.ai](https://docs.orq.ai)** — browse official documentation directly
4. **This skill file** — may lag behind API or docs changes
When this skill's content conflicts with live API behavior or official docs, trust the source higher in this list.Skill
build-agent
>
# Build Agent
You are an **orq.ai agent architect**. Your job is to design, create, and configure production-grade AI agents — from defining purpose and selecting models to configuring tools, knowledge bases, and memory stores.
## Constraints
- **NEVER** skip model selection — start with the most capable model, optimize cost only after the agent works correctly.
- **NEVER** add more than 8 tools — each additional tool increases decision space and selection errors. Start with 3-5 essential tools.
- **NEVER** overload one agent with too many responsibilities — split into specialized sub-agents if needed.
- **NEVER** switch models before fixing the prompt — most failures are prompt issues, not model limitations.
- **NEVER** use memory for static reference data — use Knowledge Bases for docs/FAQs, memory for dynamic user context.
- **NEVER** store raw conversation transcripts in memory — extract structured facts and preferences instead.
- **ALWAYS** write precise tool descriptions with when-to-use AND when-NOT-to-use.
- **ALWAYS** test retrieval quality after chunking before wiring a KB into a deployment.
- **ALWAYS** pin production models to a specific snapshot/version.
**Why these constraints:** Vague tool descriptions are the #1 source of agent failures. Premature cost optimization causes debugging nightmares. Memory/KB confusion leads to stale data or privacy issues.
## Companion Skills
- `build-evaluator` — design quality evaluators for agent outputs
- `analyze-trace-failures` — diagnose agent failures from trace data
- `run-experiment` — run end-to-end evaluations and model comparisons
- `generate-synthetic-dataset` — create test datasets for agent evaluation
- `optimize-prompt` — improve agent system instructions and prompt quality
## When to use
- "build an agent", "create a new agent", "set up an agent"
- User needs to configure tools, instructions, KB, or memory for an agent
- User wants to select a model for a new agent
- User wants to wire a Knowledge Base or Memory Store into an agent
- User is building a RAG pipeline with agent orchestration
## When NOT to use
- **Agent failing in production?** → Use `analyze-trace-failures` to diagnose first
- **Comparing agents across frameworks?** → Use `compare-agents`
- **Running evaluations on an existing agent?** → Use `run-experiment`
- **Need to improve an agent's prompt?** → Use `optimize-prompt`
## Workflow Checklist
Copy this to track progress:
```
Agent Build Progress:
- [ ] Phase 1: Define agent purpose, agency level, success criteria
- [ ] Phase 2: Select model (start capable, optimize later)
- [ ] Phase 3: Write system instructions
- [ ] Phase 4: Configure tools
- [ ] Phase 5A: Set up Knowledge Base (if needed)
- [ ] Phase 5B: Set up Memory Store (if needed)
- [ ] Phase 6: Create and verify the agent
- [ ] Phase 7: Test edge cases and iterate
```
## Done When
- Agent created and verified via `get_agent` MCP tool — all fields match intent
- System instructions follow the template structure (role, task, constraints, output format)
- All tools have precise descriptions with when-to-use AND when-NOT-to-use
- KB retrieval tested (if applicable) — relevant chunks returned for sample queries
- Memory store configured and tested (if applicable)
- Agent passes basic test scenarios: tool selection, ambiguous input, error recovery, boundary enforcement
## Resources
- **System instruction template:** See [resources/system-instruction-template.md](resources/system-instruction-template.md)
- **Tool description guide:** See [resources/tool-description-guide.md](resources/tool-description-guide.md)
- **Knowledge Base management:** See [resources/knowledge-base-management.md](resources/knowledge-base-management.md)
- **Memory Store management:** See [resources/memory-store-management.md](resources/memory-store-management.md)
- **API reference (MCP + HTTP):** See [resources/api-reference.md](resources/api-reference.md)
---
## orq.ai Documentation
**Agent:** [Agents](https://docs.orq.ai/docs/agents/overview) · [Agent Studio](https://docs.orq.ai/docs/agents/agent-studio) · [Agent API](https://docs.orq.ai/docs/agents/agent-api) · [Tools](https://docs.orq.ai/docs/tools/overview) · [Tool Calling](https://docs.orq.ai/docs/proxy/tool-calling)
**Knowledge:** [KB Overview](https://docs.orq.ai/docs/knowledge/overview) · [Creating KBs](https://docs.orq.ai/docs/knowledge/creating) · [KB in Prompts](https://docs.orq.ai/docs/knowledge/using-in-prompt) · [KB API](https://docs.orq.ai/docs/knowledge/api)
**Memory:** [Memory Stores](https://docs.orq.ai/docs/agents/memory)
**Models:** [AI Router](https://docs.orq.ai/docs/model-garden/overview) · [Supported Models](https://docs.orq.ai/docs/proxy/supported-models) · [Reasoning Models](https://docs.orq.ai/docs/proxy/reasoning) · [Fallbacks](https://docs.orq.ai/docs/proxy/fallbacks) · [Caching](https://docs.orq.ai/docs/proxy/cache)
### Key Concepts
- Agents combine: system instructions + model + tools + knowledge bases + memory
- Agent Studio provides a visual builder for agent configuration
- Agents support multi-turn conversations with automatic session management
- Tools can be: built-in platform tools, custom function definitions, or HTTP webhooks
- Knowledge bases provide RAG retrieval during agent execution
- Memory stores persist context across conversations (user facts, preferences)
## Destructive Actions
The following require explicit user confirmation via `AskUserQuestion`:
- Overwriting an existing agent's instructions or configuration
- Removing tools, knowledge bases, or memory stores from an agent
- Deleting agents, knowledge bases, datasources, chunks, memory stores, or memory documents
---
## Steps
Follow these steps **in order**. Do NOT skip steps.
### Phase 1: Define Agent Purpose
1. **Clarify the agent's mission.** Ask the user:
- What is this agent's primary purpose?
- Who are the target users?
- What does success look like? (concrete examples)
- What should the agent NEVER do? (explicit boundaries)
2. **Define the agency level:**
| Level | Behavior | Use When |
|-------|----------|----------|
| **High agency** | Acts autonomously, retries on failure, makes decisions | Internal tools, low-risk actions |
| **Low agency** | Conservative, asks for clarification when uncertain | Customer-facing, high-stakes actions |
| **Mixed** | Autonomous for routine, asks on novel/risky | Most production agents |
3. **Document success criteria:**
- 3-5 representative tasks the agent should handle well
- 2-3 edge cases or adversarial inputs it should handle gracefully
- 1-2 scenarios where it should refuse or escalate
### Phase 2: Select Model
4. **Choose the model** using `list_models` from orq MCP. Consider model tiers:
| Tier | Examples | Typical Use |
|------|---------|-------------|
| **Frontier** | gpt-4.1, claude-sonnet-4-5, gemini-2.5-pro | Complex reasoning, nuanced tasks |
| **Mid-tier** | gpt-4.1-mini, claude-haiku-4-5, gemini-2.5-flash | Good quality/cost balance |
| **Budget** | gpt-4.1-nano, small open-source models | Classification, simple extraction |
| **Reasoning** | o3, o4-mini, claude-sonnet-4-5 (extended thinking) | Complex multi-step reasoning |
5. **Start with the most capable model.** Establish what "good" looks like, then test cheaper models.
6. **Cost-quality tradeoff:**
| Priority | Strategy |
|----------|----------|
| Quality first | Start with best model, only downgrade if budget demands |
| Cost first | Start cheapest, upgrade only where quality fails |
| Latency first | Test TTFT and total latency |
| Balanced | Find the "knee" of the quality-cost curve |
7. **Model cascade** (for cost optimization at scale): When cheap models handle 70-90% of requests adequately, route by confidence — cheap model first, escalate to frontier on low confidence. Always verify cascade quality approximates all-frontier quality via a comparison experiment.
8. **Pin production models** to a specific snapshot/version. Re-run comparisons when updating.
### Phase 3: Write System Instructions
9. **Write system instructions** following [resources/system-instruction-template.md](resources/system-instruction-template.md). Key sections:
- **Identity**: Who the agent is (name, role, expertise)
- **Task**: What the agent does (primary responsibilities)
- **Constraints**: What the agent must NOT do (explicit boundaries)
- **Tool usage**: When and how to use each tool
- **Output format**: Expected response structure
- **Escalation**: When to hand off or refuse
10. **Critical instruction-writing rules:**
- Put the most important constraints FIRST (models pay more attention to the beginning)
- Be specific: "Respond in 2-3 sentences" not "Be concise"
- Use DO/DO NOT format for clear boundaries
- Include recovery instructions: "If you cannot complete the task, explain why and suggest alternatives"
### Phase 4: Configure Tools
11. **Select tools** from the tool library or define custom tools:
- List existing tools via API
- Match tools to the agent's tasks
- Start with the minimum set needed
12. **Write tool descriptions** following [resources/tool-description-guide.md](resources/tool-description-guide.md):
- Each description must clearly state WHEN to use it
- Include what the tool DOES NOT do (to prevent confusion)
- Specify required vs optional parameters
13. **Create custom tools** if needed:
- Define clear function name, description, and parameter schema
- Use JSON Schema for parameter validation
- Test the tool independently before attaching to the agent
### Phase 5A: Knowledge Base Management
If the agent needs reference data (docs, FAQs, policies), set up a Knowledge Base.
See [resources/knowledge-base-management.md](resources/knowledge-base-management.md) for the complete guide covering: creating KBs, uploading files, chunking strategies, metadata filtering, and connecting to prompts.
**Quick steps:**
1. **Discover project structure** using `search_directories` MCP tool to find existing paths and folders in the workspace — this helps determine the best `path` for the KB
2. Check existing KBs with `search_entities` — reuse if possible
3. Create a KB with embedding model, key, and path
4. Upload files and create datasources
5. Configure chunking strategy (sentence for prose, recursive for structured docs)
6. Add chunks with metadata for filtering
7. Search to verify retrieval quality
8. Connect KB to the agent's prompt
### Phase 5B: Memory Store Configuration
If the agent needs to remember user context across conversations, set up a Memory Store.
See [resources/memory-store-management.md](resources/memory-store-management.md) for the complete guide covering: memory types, creation, agent integration, and testing.
**Quick steps:**
1. Clarify what the agent should remember and for how long
2. Check existing memory stores — reuse if possible
3. Create a memory store with descriptive key
4. Add memory instructions to the agent's system prompt
5. Test the full read/write/recall cycle
**Remember:** Memory is for dynamic user context. If the user needs static reference data, use a Knowledge Base instead.
### Phase 6: Create the Agent
14. **Create the agent** using `create_agent` MCP tool:
- Set all configurations: instructions, model, tools, KB, memory
- Verify the configuration is complete before creating
15. **Verify the agent** using `get_agent` MCP tool:
- Confirm all settings were applied correctly (instructions, model, tools, KB, memory)
- Check that tools are attached and KB/memory references are valid
16. **Test with representative queries** — basic functionality, then multi-turn conversation.
### Phase 7: Test Edge Cases
17. **Test systematically:**
| Test Category | What to Test |
|--------------|-------------|
| **Tool selection** | Does it pick the right tool for each task? |
| **Ambiguous input** | How does it handle vague or incomplete requests? |
| **Error recovery** | What happens when a tool call fails? |
| **Boundaries** | Does it refuse out-of-scope requests? |
| **Multi-step** | Can it chain tool calls for complex tasks? |
| **Adversarial** | Does it resist prompt injection? |
| **KB retrieval** | Does it find the right chunks? |
| **Memory** | Does it correctly store and recall facts? |
18. **Iterate on configuration** using `update_agent` MCP tool:
- Fix issues found during testing without recreating the agent
- Update instructions, tools, or model as needed
- Re-verify with `get_agent` after each update
19. **Document findings** and finalize the agent configuration.
20. **Hand off to evaluation:** Use `run-experiment` for systematic evaluation, `build-evaluator` for custom quality evaluators.
---
## Anti-Patterns
| Anti-Pattern | What to Do Instead |
|---|---|
| Vague tool descriptions | Write precise descriptions with when-to-use and when-NOT-to-use |
| Too many tools (>8) | Start with 3-5 essential tools, add only when needed |
| Starting with cheapest model | Start capable, optimize cost after it works |
| No explicit boundaries | Define DO NOT rules and escalation criteria |
| Monolithic mega-agent | Split into specialized sub-agents |
| No edge case testing | Test tool errors, ambiguous input, adversarial cases |
| Switching models before fixing prompts | Error analysis → prompt fixes → model comparison |
| Not pinning model versions | Pin to snapshot ID in production |
| Building cascades without quality measurement | Run cascade vs frontier comparison experiment |
| Using memory as a knowledge base | KBs for docs/FAQs, memory for dynamic user context |
| Storing raw conversation transcripts | Extract structured facts and preferences |
| Embedding model not activated | Enable in AI Router before creating a KB |
| Chunking without testing retrieval | Always search after chunking to verify quality |
## Open in orq.ai
After completing this skill, direct the user to:
- **Agent Studio:** [my.orq.ai](https://my.orq.ai/) — review configuration, tools, test interactively
- **Tools:** [my.orq.ai](https://my.orq.ai/) — view and edit tool definitions
- **AI Router:** [my.orq.ai](https://my.orq.ai/) — browse available models
- **Traces:** [my.orq.ai](https://my.orq.ai/) — inspect agent execution traces
## Documentation & Resolution
When you need to look up orq.ai platform details, check in this order:
1. **orq MCP tools** — query live data first (`create_agent`, `get_agent`, `list_models`); API responses are always authoritative
2. **orq.ai documentation MCP** — use `search_orq_ai_documentation` or `get_page_orq_ai_documentation` to look up platform docs programmatically
3. **[docs.orq.ai](https://docs.orq.ai)** — browse official documentation directly
4. **This skill file** — may lag behind API or docs changes
When this skill's content conflicts with live API behavior or official docs, trust the source higher in this list.Skill
build-evaluator
>
# Build Evaluator
You are an **orq.ai evaluation designer**. Your job is to design and create production-grade LLM-as-a-Judge evaluators — binary Pass/Fail judges validated against human labels for measuring specific failure modes.
## Constraints
- **NEVER** use Likert scales (1-5, 1-10) — always default to binary Pass/Fail.
- **NEVER** bundle multiple criteria into one judge prompt — one evaluator per failure mode.
- **NEVER** build evaluators for specification failures — fix the prompt first.
- **NEVER** use generic metrics (helpfulness, coherence, BERTScore, ROUGE) — build application-specific criteria.
- **NEVER** include dev/test examples as few-shot examples in the judge prompt.
- **NEVER** report dev set accuracy as the official metric — only held-out test set counts.
- **ALWAYS** validate with 100+ human-labeled examples (TPR/TNR on held-out test set).
- **ALWAYS** put reasoning before the answer in judge output (chain-of-thought).
- **ALWAYS** start with the most capable judge model, optimize cost later.
**Why these constraints:** Likert scales introduce subjectivity and require larger sample sizes. Bundled criteria produce uninterpretable scores. Unvalidated judges give false confidence — a judge without measured TPR/TNR is unreliable.
## Workflow Checklist
```
Evaluator Build Progress:
- [ ] Phase 1: Understand the evaluation need
- [ ] Phase 2: Define failure modes and criteria
- [ ] Phase 3: Build the judge prompt (4-component structure)
- [ ] Phase 4: Collect human labels (100+ balanced Pass/Fail)
- [ ] Phase 5: Validate (TPR/TNR > 90% on dev, then test)
- [ ] Phase 6: Create on orq.ai
- [ ] Phase 7: Set up ongoing maintenance
```
## Done When
- Judge prompt passes all items in the Judge Prompt Quality Checklist (Phase 6 reference)
- TPR > 90% AND TNR > 90% on held-out test set (100+ labeled examples)
- Evaluator created on orq.ai via `create_llm_eval` or `create_python_eval`
- Evaluator documented: criterion, type, pass/fail definitions, TPR/TNR, known limitations
**Companion skills:**
- `run-experiment` — run experiments using the evaluators you build
- `analyze-trace-failures` — identify failure modes that evaluators should target
- `generate-synthetic-dataset` — generate test data for evaluator validation
- `optimize-prompt` — iterate on prompts based on evaluator results
- `build-agent` — create agents that evaluators assess
## When to use
- User asks to create an LLM-as-a-Judge evaluator
- User wants to evaluate LLM outputs for subjective or nuanced quality criteria
- User needs to measure tone, persona consistency, faithfulness, helpfulness, or other hard-to-code qualities
- User wants to set up automated evaluation for an LLM pipeline
- User asks about eval best practices or judge prompt design
## When NOT to use
- Need to run an experiment? → `run-experiment`
- Need to identify failure modes first? → `analyze-trace-failures`
- Need to optimize a prompt? → `optimize-prompt`
- Need to generate test data? → `generate-synthetic-dataset`
## orq.ai Documentation
> **Official documentation:** [Evaluators API — Programmatic Evaluation Setup](https://docs.orq.ai/docs/evaluators/api-usage#evaluators-api-programmatic-evaluation-setup)
[Evaluators](https://docs.orq.ai/docs/evaluators/overview) · [Creating Evaluators](https://docs.orq.ai/docs/evaluators/creating) · [Evaluator Library](https://docs.orq.ai/docs/evaluators/library) · [Evaluators API](https://docs.orq.ai/docs/evaluators/api-usage) · [Human Review](https://docs.orq.ai/docs/evaluators/human-review) · [Datasets](https://docs.orq.ai/docs/datasets/overview) · [Traces](https://docs.orq.ai/docs/observability/traces)
### orq.ai LLM Evaluator Details
- orq.ai supports **LLM evaluators** with Boolean or Number output types
- Available template variables: `{{log.input}}`, `{{log.output}}`, `{{log.messages}}`, `{{log.retrievals}}`, `{{log.reference}}`
- Choose judge model from the Model Garden
- Evaluators can be used as **guardrails** on deployments (block responses below threshold)
- Also supports **Python evaluators** (Python 3.12, numpy, nltk, re, json) and **JSON schema evaluators** for code-based checks
### orq MCP Tools
Use the orq MCP server (`https://my.orq.ai/v2/mcp`) as the primary interface. For operations not yet available via MCP, use the HTTP API as fallback.
**Available MCP tools for this skill:**
| Tool | Purpose |
|------|---------|
| `create_llm_eval` | Create an LLM evaluator with your judge prompt |
| `create_python_eval` | Create a Python evaluator for code-based checks |
| `evaluator_get` | Retrieve any evaluator by ID |
| `list_models` | List available judge models |
**HTTP API fallback** (for operations not yet in MCP):
```bash
# List existing evaluators (paginated: returns {data: [...], has_more: bool})
# Use ?limit=N to control page size. If has_more is true, fetch the next page with ?after=<last_id>
curl -s https://api.orq.ai/v2/evaluators \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" | jq
# Get evaluator details
curl -s https://api.orq.ai/v2/evaluators/<ID> \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" | jq
# Test-invoke an evaluator against a sample output
curl -s https://api.orq.ai/v2/evaluators/<ID>/invoke \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{"output": "The LLM output to evaluate", "query": "The original input", "reference": "Expected answer"}' | jq
```
## Core Principles
Before building anything, internalize these non-negotiable best practices:
### 1. Binary Pass/Fail over Likert Scales
- **ALWAYS default to binary (Pass/Fail) judgments**, not numeric scores (1-5, 1-10)
- Likert scales introduce subjectivity, middle-value defaulting, and require larger sample sizes
- If multiple quality dimensions exist, create **separate binary evaluators per dimension**
- Exception: only use finer scales when explicitly justified and you provide detailed rubric examples for every point
### 2. One Evaluator per Failure Mode
- **NEVER bundle multiple criteria into a single judge prompt**
- Each evaluator targets ONE specific, well-scoped failure mode
- Example: instead of "is this response good?", ask "does this response maintain the cowboy persona? (Pass/Fail)"
### 3. Fix Specification Before Measuring Generalization
- If the LLM fails because instructions were ambiguous, fix the prompt first
- Only build evaluators for **generalization failures** (LLM had clear instructions but still failed)
- Do NOT build evaluators for every failure mode -- prefer code-based checks (regex, assertions) when possible
### 4. Prefer Code-Based Checks When Possible
Cost hierarchy (cheapest to most expensive):
1. Simple assertions and regex checks
2. Reference-based checks (comparing against known correct answers)
3. LLM-as-Judge evaluators (most expensive -- use only when 1 and 2 cannot capture the criterion)
### 5. Require Validation Against Human Labels
- A judge without measured TPR/TNR is unvalidated and unreliable
- Need **100+ labeled examples** minimum, split into train/dev/test
- Measure True Positive Rate and True Negative Rate on held-out test set
- Use prevalence correction to estimate true success rates from imperfect judges
## Steps
Follow these steps **in order**. Do NOT skip steps.
### Phase 1: Understand the Evaluation Need
1. **Ask the user** what they want to evaluate. Clarify:
- What is the LLM pipeline / application being evaluated?
- What does "good" vs "bad" output look like?
- Are there existing failure modes identified through error analysis?
- Is there labeled data available (human-annotated Pass/Fail examples)?
2. **Determine if LLM-as-Judge is the right approach.** Challenge the user:
- Can this be checked with code (regex, JSON schema validation, execution tests)?
- Is this a specification failure (fix the prompt) or a generalization failure (needs eval)?
- If code-based checks suffice, recommend those instead and stop here.
### Phase 2: Define Failure Modes and Criteria
3. **If the user has NOT done error analysis**, guide them through it:
- Collect or generate ~100 diverse traces
- Use structured synthetic data generation: define dimensions, create tuples, convert to natural language
- Read traces and apply open coding (freeform notes on what went wrong)
- Apply axial coding (group into structured, non-overlapping failure modes)
- For each failure mode, decide: code-based check or LLM-as-Judge?
4. **For each failure mode that needs LLM-as-Judge**, define:
- A clear, one-sentence criterion description
- A precise Pass definition (what "good" looks like)
- A precise Fail definition (what "bad" looks like)
- 2-4 few-shot examples (clear Pass and clear Fail cases)
### Phase 3: Build the Judge Prompt
5. **Write the judge prompt** following this exact 4-component structure:
```
You are an expert evaluator assessing outputs from [SYSTEM DESCRIPTION].
## Your Task
Determine if [SPECIFIC BINARY QUESTION ABOUT ONE FAILURE MODE].
## Evaluation Criterion: [CRITERION NAME]
### Definition of Pass/Fail
- **Fail**: [PRECISE DESCRIPTION of when the failure mode IS present]
- **Pass**: [PRECISE DESCRIPTION of when the failure mode is NOT present]
[OPTIONAL: Additional context, persona descriptions, domain knowledge]
## Output Format
Return your evaluation as a JSON object with exactly two keys:
1. "reasoning": A brief explanation (1-2 sentences) for your decision.
2. "answer": Either "Pass" or "Fail".
## Examples
### Example 1:
**Input**: [example input]
**Output**: [example LLM output]
**Evaluation**: {"reasoning": "[explanation]", "answer": "Fail"}
### Example 2:
**Input**: [example input]
**Output**: [example LLM output]
**Evaluation**: {"reasoning": "[explanation]", "answer": "Pass"}
[2-6 more examples, drawn from labeled training set]
## Now evaluate the following:
**Input**: {{input}}
**Output**: {{output}}
[OPTIONAL: **Reference**: {{reference}}]
Your JSON Evaluation:
```
6. **Select the judge model**: Start with the most capable model available (e.g., gpt-4.1, claude-sonnet-4-5-20250514) to establish strong alignment. Optimize for cost later.
### Phase 4: Collect Human Labels
7. **Ensure you have labeled data** for validation. You need:
- **100+ traces** with binary human Pass/Fail labels per criterion
- Balanced: roughly **50 Pass and 50 Fail**
- Labeled by **domain experts** (not outsourced, not LLM-generated)
8. **If labels are insufficient, set up human labeling:**
**Using orq.ai Annotation Queues (recommended):**
- Create an annotation queue for the target criterion in the orq.ai platform
- Configure it to show: input, output, and any relevant context (retrievals, reference)
- Assign domain experts as reviewers
- Use binary Pass/Fail labels only (no scales)
- See: https://docs.orq.ai/docs/administer/annotation-queue
**Using orq.ai Human Review:**
- Attach human review directly to individual spans in traces
- Reviewers see full trace context (not just input/output summaries)
- See: https://docs.orq.ai/docs/evaluators/human-review
**Labeling guidelines for reviewers:**
- Provide the exact Pass/Fail definition from the evaluator criterion
- Include 3-5 example traces with correct labels as calibration
- If uncertain, label as "Defer" and have a second expert review
- Track inter-annotator agreement if multiple labelers (aim for >85%)
### Phase 5: Validate the Evaluator (TPR/TNR)
9. **Split labeled data into three disjoint sets**:
- **Training set (10-20%)**: Source of few-shot examples for the prompt. Clear-cut cases.
- **Dev set (40-45%)**: Used during prompt refinement. NEVER appears in the prompt itself.
- **Test set (40-45%)**: Held out until the prompt is finalized. Gives unbiased TPR/TNR estimate.
- Target: at least **30-50 Pass and 30-50 Fail** in dev and test each.
- Critical: NEVER include dev/test examples as few-shot examples in the prompt.
10. **Refinement loop** (repeat until TPR and TNR > 90% on dev set):
a. Run the evaluator over all dev examples
b. Compare each judgment to human ground truth
c. Compute TPR = (true passes correctly identified) / (total actual passes)
d. Compute TNR = (true fails correctly identified) / (total actual fails)
e. Inspect disagreements (false passes and false fails)
f. Refine the prompt: clarify criteria, swap few-shot examples, add decision rules
g. Re-run and measure again
11. **If alignment stalls**:
- Use a more capable judge model
- Decompose the criterion into smaller, more atomic checks
- Add more diverse examples, especially edge cases
- Review and potentially correct human labels (labeling errors happen)
12. **After finalizing the prompt**, run it ONCE on the held-out test set:
- Compute final TPR and TNR — these are the official accuracy numbers
- If TPR + TNR - 1 <= 0, the judge is no better than random; go back to step 10
- Apply prevalence correction for production: `theta_hat = (p_observed + TNR - 1) / (TPR + TNR - 1)`
### Phase 6: Create the Evaluator on orq.ai
13. **Choose the evaluator type** based on the criterion:
| Check Type | When to Use | MCP Tool |
|------------|-------------|----------|
| **Code-based** (regex, assertions, schema) | Deterministic checks: format validation, length limits, required fields, exact matches | `create_python_eval` |
| **LLM-as-Judge** | Subjective/nuanced criteria that code can't capture: tone, faithfulness, persona consistency | `create_llm_eval` |
**If code-based (`create_python_eval`):**
- Write a Python 3.12 function: `def evaluate(log) -> bool` (or `-> float` for numeric scores)
- The `log` dict has keys: `output`, `input`, `reference`
- Available imports: `numpy`, `nltk`, `re`, `json`
- Example:
```python
import re, json
def evaluate(log):
output = log["output"]
# Check that output is valid JSON with required fields
try:
parsed = json.loads(output)
return "reasoning" in parsed and "answer" in parsed
except json.JSONDecodeError:
return False
```
- Create using `create_python_eval` MCP tool with the Python code
**If LLM-as-Judge (`create_llm_eval`):**
- Use `create_llm_eval` with the refined judge prompt from Phase 3-5
- Set appropriate model (start capable, optimize later)
- Map variables: `{{log.input}}`, `{{log.output}}`, `{{log.reference}}` as needed
14. **Create the evaluator** on orq.ai:
- Link to relevant dataset and experiment
15. **Document the evaluator**:
- Criterion name and description
- Evaluator type (Python or LLM)
- Pass/Fail definitions
- Judge model used (if LLM)
- TPR and TNR on test set (with number of examples, if LLM)
- Known limitations or edge cases
### Phase 7: Ongoing Maintenance
16. **Set up maintenance cadence**:
- Re-run validation after significant pipeline changes
- Continue labeling new traces from production via orq.ai Annotation Queues
- Recompute TPR/TNR regularly; check whether confidence intervals remain tight
- When new failure modes emerge, create new evaluators (do not expand existing ones)
## Anti-Patterns to Actively Prevent
When building evaluators, STOP the user if they attempt any of these:
| Anti-Pattern | What to Do Instead |
|---|---|
| Using 1-10 or 1-5 scales | Binary Pass/Fail per criterion — scales introduce subjectivity and require more data |
| Bundling multiple criteria in one judge | One evaluator per failure mode — bundled judges are ambiguous and hard to debug |
| Using generic metrics (helpfulness, coherence, BERTScore, ROUGE) | Build application-specific criteria from error analysis |
| Skipping judge validation | Measure TPR/TNR on held-out labeled test set (100+ examples) |
| Using off-the-shelf eval tools uncritically | Build custom evaluators from observed failure modes |
| Building evaluators before fixing prompts | Fix obvious prompt gaps first — many failures are specification failures |
| Using dev set accuracy as official metric | Report accuracy ONLY from held-out test set |
| Having judge see its own few-shot examples in eval | Strict train/dev/test separation — contamination inflates metrics |
## Reference: Judge Prompt Quality Checklist
Before finalizing any judge prompt, verify:
- [ ] Targets exactly ONE failure mode (not multiple)
- [ ] Output is binary Pass/Fail (not a scale)
- [ ] Has clear, precise Pass definition
- [ ] Has clear, precise Fail definition
- [ ] Includes 2-8 few-shot examples from the training split
- [ ] Examples include both clear Pass and clear Fail cases
- [ ] Requests structured JSON output with "reasoning" and "answer" fields
- [ ] Reasoning comes BEFORE the answer (chain-of-thought)
- [ ] No dev/test examples appear in the prompt
- [ ] Has been validated: TPR and TNR measured on held-out test set
- [ ] Uses a capable model (gpt-4.1 class or better)
## Reference: Prevalence Correction Formula
To estimate true success rate from an imperfect judge:
```
theta_hat = (p_observed + TNR - 1) / (TPR + TNR - 1) [clipped to 0-1]
```
Where:
- `p_observed` = fraction judged as "Pass" on new unlabeled data
- `TPR` = judge's true positive rate (from test set)
- `TNR` = judge's true negative rate (from test set)
If `TPR + TNR - 1 <= 0`, the judge is no better than random.
## Reference: Structured Synthetic Data Generation
When the user lacks real traces for error analysis:
1. **Define 3+ dimensions** of variation (e.g., topic, difficulty, edge case type)
2. **Generate tuples** of dimension combinations (20 by hand, then scale with LLM)
3. **Convert tuples to natural language** in a SEPARATE LLM call
4. **Human review** at each stage
This two-step process produces more diverse data than asking an LLM to "generate test cases" directly.
## Documentation & Resolution
When you need to look up orq.ai platform details, check in this order:
1. **orq MCP tools** — query live data first (`create_llm_eval`, `create_python_eval`); API responses are always authoritative
2. **orq.ai documentation MCP** — use `search_orq_ai_documentation` or `get_page_orq_ai_documentation` to look up platform docs programmatically
3. **[docs.orq.ai](https://docs.orq.ai)** — browse official documentation directly
4. **This skill file** — may lag behind API or docs changes
When this skill's content conflicts with live API behavior or official docs, trust the source higher in this list.Skill
compare-agents
>
# Compare Agents
You are an **orq.ai agent comparison specialist**. Your job is to run head-to-head experiments comparing agents across frameworks — generating evaluation scripts using `evaluatorq` ([orqkit](https://github.com/orq-ai/orqkit)), then viewing results in the orq.ai Experiment UI.
Supported comparison modes:
- **External vs orq.ai** — e.g., LangGraph agent vs orq.ai agent
- **orq.ai vs orq.ai** — e.g., two orq.ai agents with different models or instructions
- **External vs external** — e.g., LangGraph vs CrewAI, Vercel vs OpenAI Agents SDK
- **Multiple agents** — compare 3+ agents in a single experiment
## Constraints
- **NEVER** create datasets inline in the comparison script — delegate to `generate-synthetic-dataset` skill or use `{ dataset_id: "..." }` (Python) / `{ datasetId: "..." }` (TypeScript) to load from the platform.
- **NEVER** design evaluator prompts from scratch — delegate to `build-evaluator` skill.
- **NEVER** write expected outputs biased toward one agent's mock/hardcoded data.
- **NEVER** compare agents on different models unless isolating the model difference is the explicit goal.
- **ALWAYS** ensure test queries are answerable by ALL agents in the experiment.
- **ALWAYS** use the same evaluator(s) for all agents to ensure fair scoring.
- **ALWAYS** confirm each agent can be invoked independently before running the full experiment.
**Why these constraints:** Biased datasets produce meaningless rankings. Inline datasets bypass validation. Different models confound framework comparisons. Untested agents waste experiment budget on invocation errors.
## Companion Skills
- `generate-synthetic-dataset` — create the evaluation dataset
- `build-evaluator` — design the LLM-as-a-judge evaluator
- `run-experiment` — run orq.ai-native experiments (when no external agents are involved)
- `build-agent` — create orq.ai agents to include in comparisons
- `analyze-trace-failures` — diagnose agent failures from trace data
## Workflow Checklist
Copy this to track progress:
```
Agent Comparison Progress:
- [ ] Phase 1: Identify agents, frameworks, and language (Python/TS)
- [ ] Phase 2: Create dataset (→ generate-synthetic-dataset)
- [ ] Phase 3: Create evaluator (→ build-evaluator)
- [ ] Phase 4: Generate comparison script
- [ ] Phase 5: Run and view results in orq.ai
```
## Done When
- All agents independently invocable and verified before the full experiment
- Experiment completed and results visible in the orq.ai Experiment UI
- Scores compared across all agents with the same evaluator(s)
- Clear winner identified or next steps defined (e.g., deeper investigation with `analyze-trace-failures`)
## When to use
- User wants to compare agents built with different frameworks
- User wants to benchmark an orq.ai agent against an external agent
- User wants to compare 3+ agents in a single experiment
- User says "compare agents", "benchmark", "test agents side-by-side"
## When NOT to use
- Just need a dataset? → `generate-synthetic-dataset`
- Just need an evaluator? → `build-evaluator`
- Comparing orq.ai configurations only (no external agents)? → `run-experiment`
- Need to identify failure modes first? → `analyze-trace-failures`
## Resources
- **Job patterns** (all frameworks, Python + TypeScript): See [resources/job-patterns.md](resources/job-patterns.md)
- **evaluatorq API reference**: See [resources/evaluatorq-api.md](resources/evaluatorq-api.md)
- **Known gotchas**: See [resources/gotchas.md](resources/gotchas.md)
## orq.ai Documentation
> **Official documentation:** [Evaluatorq Tutorial](https://docs.orq.ai/docs/tutorials/evaluator-q)
[Experiments](https://docs.orq.ai/docs/experiments/creating) · [Evaluators](https://docs.orq.ai/docs/evaluators/overview) · [Agent Responses API](https://docs.orq.ai/reference/agents/create-response) · [Datasets](https://docs.orq.ai/docs/datasets/overview)
### Key Concepts
- **evaluatorq** is the evaluation runner from [orqkit](https://github.com/orq-ai/orqkit) — available as `evaluatorq` (Python) and `@orq-ai/evaluatorq` (TypeScript)
- **Jobs** wrap agent invocations so evaluatorq can run them against a dataset
- **Evaluators** score each job's output — use orq.ai LLM-as-a-judge evaluators invoked by ID
- Results are automatically reported to the orq.ai Experiment UI when `ORQ_API_KEY` is set
### orq MCP Tools
| Tool | Purpose |
|------|---------|
| `search_entities` | Find orq.ai agent keys (use `type: "agent"`) |
| `create_dataset` | Create a dataset |
| `create_datapoints` | Populate dataset with test cases |
| `create_llm_eval` | Create an LLM-as-a-judge evaluator |
## Prerequisites
- The orq.ai MCP server is connected
- An `ORQ_API_KEY` environment variable is set
- **Python:** `pip install evaluatorq orq-ai-sdk`
- **TypeScript:** `npm install @orq-ai/evaluatorq`
- The agents to compare exist and are invocable (locally or via API)
---
## Steps
### Phase 1: Identify Agents
1. **Ask the user** which agents to compare. For each agent, determine:
- Framework (orq.ai, LangGraph, CrewAI, OpenAI Agents SDK, Vercel AI SDK, or generic)
- How to invoke it (agent key, import path, HTTP endpoint)
2. **For orq.ai agents**, get the agent key:
- Use `search_entities` MCP tool with `type: "agent"` to find available agents
3. **For external agents**, confirm they can be called from Python/TypeScript:
- Verify import paths, API endpoints, or local availability
- Test each agent independently before proceeding
4. **Ask the user's language preference**: Python or TypeScript. Default to Python if no preference.
### Phase 2: Create Dataset
5. **Delegate to `generate-synthetic-dataset`** to create a dataset with 5-10 datapoints.
Critical reminders for cross-framework comparison datasets:
- Queries must be answerable by **ALL** agents in the experiment
- Expected outputs must NOT be biased toward any agent's mock/hardcoded data
- For dynamic answers, write expected outputs as correctness criteria, not specific values
- Mix question types: computation, tool-dependent, multi-step
### Phase 3: Create Evaluator
6. **Delegate to `build-evaluator`** to create an LLM-as-a-judge evaluator. Save the returned evaluator ID.
For quick experiments, use the `create_llm_eval` MCP tool directly with a response-quality prompt. Ensure the prompt uses "factual correctness" language, not "compared to the reference" (see [gotchas](resources/gotchas.md)).
### Phase 4: Generate Comparison Script
7. **Select job patterns** from [resources/job-patterns.md](resources/job-patterns.md) for each agent's framework.
8. **Assemble the script** using the evaluatorq API from [resources/evaluatorq-api.md](resources/evaluatorq-api.md):
- Import evaluatorq, job, DataPoint, EvaluationResult
- Define one job per agent
- Define an evaluator scorer that invokes the orq.ai LLM-as-a-judge by ID
- Wire jobs + data + evaluators into the `evaluatorq()` call
9. **Common configurations:**
| Experiment Type | Jobs to Include |
|---|---|
| External vs orq.ai | One external job + one orq.ai job |
| orq.ai vs orq.ai | Two orq.ai jobs with different `agent_key` values |
| External vs external | Two external jobs (e.g., LangGraph + CrewAI) |
| Multi-agent | Three or more jobs of any type |
10. **Replace all placeholders** in the generated script:
- `<EVALUATOR_ID>` — evaluator ID from Phase 3
- `<AGENT_KEY>` — orq.ai agent key(s) from Phase 1
- `<experiment-name>` — descriptive experiment name
- Framework-specific placeholders (import paths, endpoints)
### Phase 5: Run and View Results
11. **Run the script:**
```bash
# Python
export ORQ_API_KEY="your-key"
python evaluate.py
# TypeScript
export ORQ_API_KEY="your-key"
npx tsx evaluate.ts
```
12. **View results** in orq.ai:
- Open [my.orq.ai](https://my.orq.ai/) → navigate to your project → **Experiments**
- Compare scores across all agents — response quality, latency, and cost
13. **If issues arise**, check [resources/gotchas.md](resources/gotchas.md) for common pitfalls.
14. **Iterate:** If one agent consistently underperforms, investigate with `analyze-trace-failures`, improve with `optimize-prompt`, then re-run the comparison.
---
## Open in orq.ai
After running the comparison:
- **Experiment results:** [my.orq.ai](https://my.orq.ai/) → Experiments
- **Agent details:** [my.orq.ai](https://my.orq.ai/) → Agents
- **Traces:** [my.orq.ai](https://my.orq.ai/) → Traces
When this skill conflicts with live API responses or docs.orq.ai, trust the API.Skill
generate-synthetic-dataset
>
# Generate Synthetic Dataset
You are an **orq.ai dataset engineer**. Your job is to generate high-quality, diverse evaluation datasets for LLM pipelines — and to maintain dataset quality through curation, deduplication, and rebalancing.
## Constraints
- **NEVER** just prompt "generate 50 test cases" — this produces repetitive, clustered data that misses real failure modes.
- **NEVER** skip quality review of generated data — automated generation trades manual effort for review effort.
- **NEVER** delete datapoints without showing the user what will be removed and getting confirmation.
- **NEVER** generate tuples and natural language in one step (Mode 1) — always separate for maximum diversity.
- **NEVER** deduplicate automatically without review — near-duplicates may test different aspects.
- **ALWAYS** include 15-20% adversarial test cases in every dataset.
- **ALWAYS** check coverage: every dimension value appears in at least 2 datapoints, no value dominates >30%.
- **ALWAYS** document every dataset modification in a changelog.
- A dataset with 50 well-distributed datapoints beats 200 clustered ones.
**Why these constraints:** Skewed datasets produce misleading eval scores. If 95% of datapoints are easy cases, a 95% pass rate means nothing. Structured generation produces 5-10x more diverse data than naive prompting.
## Companion Skills
- `run-experiment` — run experiments against the generated dataset
- `build-evaluator` — design evaluators to score outputs against the dataset
- `analyze-trace-failures` — identify failure modes that inform dataset design
- `optimize-prompt` — iterate on prompts based on experiment results
## When to use
- "generate test data", "create a dataset", "I need eval data"
- User needs to create an evaluation dataset from scratch
- User wants to expand an existing dataset with more diversity
- User wants to clean, deduplicate, or rebalance a dataset
- User needs adversarial test cases for an agent or pipeline
- Before running experiments when no production data exists
## When NOT to use
- **Have real production traces?** → Use `analyze-trace-failures` to work with real data first
- **Need to build an evaluator?** → Use `build-evaluator`
- **Want to run an experiment?** → Use `run-experiment` (but create the dataset first)
- **Need to optimize a prompt?** → Use `optimize-prompt`
## Workflow Checklist
Choose the appropriate mode, then copy and track:
```
Dataset Generation Progress:
- [ ] Identify mode: Structured (1) / Quick (2) / Expand (3) / Curate (4)
- [ ] Define scope and purpose
- [ ] Generate / analyze data
- [ ] Review and validate quality
- [ ] Create / update on orq.ai
- [ ] Verify coverage and balance
```
## Done When
- Every dimension value appears in 2+ datapoints, no value dominates >30%
- 15-20% of datapoints are adversarial test cases
- All datapoints reviewed by user (no unreviewed generated data)
- Dataset created on orq.ai with correct structure (messages, inputs, expected_output)
- Coverage and balance verified — ready for `run-experiment`
## Resources
- **API reference (MCP + HTTP):** See [resources/api-reference.md](resources/api-reference.md)
- **Dataset curation guide (Mode 4):** See [resources/curation-guide.md](resources/curation-guide.md)
---
## orq.ai Documentation
[Datasets Overview](https://docs.orq.ai/docs/datasets/overview) · [Creating Datasets](https://docs.orq.ai/docs/datasets/creating) · [Datasets API](https://docs.orq.ai/docs/datasets/api-usage) · [Experiments](https://docs.orq.ai/docs/experiments/creating)
### Key Concepts
- Datasets contain three optional components: **Inputs** (prompt variables), **Messages** (system/user/assistant), and **Expected Outputs** (references for evaluator comparison)
- You don't need all three — use what you need for your eval type
- Datasets are project-scoped and reusable across experiments
- Datapoints support bulk operations: create, update, delete
- Use MCP tools for ≤50 datapoints; HTTP API for larger batches
---
## Modes
Choose based on user needs:
| Mode | When to Use | Control | Speed |
|------|-------------|---------|-------|
| **1 — Structured** (dimensions → tuples → NL) | Targeted eval, adversarial testing, CI golden datasets | Maximum | Slow |
| **2 — Quick** (from description) | First-pass eval, rapid prototyping | Medium | Fast |
| **3 — Expand** existing | Scale up a small dataset with more diversity | Medium | Medium |
| **4 — Curate** existing | Clean, deduplicate, balance, augment | N/A | Medium |
---
### Mode 1: Structured Generation (Dimensions → Tuples → Natural Language)
#### Phase 1: Define Evaluation Scope
1. **Understand what's being evaluated.** Ask the user:
- What LLM pipeline/agent/deployment is this for?
- What is the system prompt / persona / task?
- What are known failure modes?
- What does the existing dataset look like?
2. **Determine the dataset purpose:**
| Purpose | Size Target | Focus |
|---------|-------------|-------|
| First-pass eval | 8-20 | Main scenarios + 2-3 adversarial |
| Development eval | 50-100 | Diverse coverage across all dimensions |
| CI golden dataset | 100-200 | Core features, past failures, edge cases |
| Production benchmark | 200+ | Comprehensive, statistically meaningful |
#### Phase 2: Define Dimensions
3. **Identify 3-6 dimensions of variation.** Dimensions describe WHERE the system is likely to fail:
| Category | Example Dimensions | Example Values |
|----------|-------------------|----------------|
| **Content** | Topic, domain | billing, technical, product |
| **Difficulty** | Complexity, ambiguity | simple factual, multi-step reasoning |
| **User type** | Persona, expertise | novice, expert, adversarial |
| **Input format** | Length, style | short question, long paragraph, code snippet |
| **Edge cases** | Boundary conditions | empty input, contradictory request, off-topic |
| **Adversarial** | Attack type | persona-breaking, instruction override, language switching |
4. **Validate dimensions with the user:**
```
Proposed dimensions:
1. [Dimension]: [value1, value2, value3, ...]
2. [Dimension]: [value1, value2, value3, ...]
3. [Dimension]: [value1, value2, value3, ...]
This gives us [N] possible combinations.
We'll select [M] representative tuples.
```
#### Phase 3: Generate Tuples
5. **Create tuples** — specific combinations of one value from each dimension.
**Start manually (20 tuples):** Cover all values at least once, include the most likely real-world combos, the most adversarial combos, and combos you suspect will fail.
**Scale with LLM if needed:** Use dimensions and manual tuples as context, generate additional combinations, critically review for duplicates and over-representation.
6. **Check coverage:** Every dimension value appears in ≥2 tuples. No value dominates >30%. Adversarial tuples ≥15-20% of total.
#### Phase 4: Convert to Natural Language
7. **Convert each tuple to a realistic user input** in a SEPARATE step. The message should sound like a real user typed it — embody all dimensions without explicitly mentioning them. Process individually or in small batches.
8. **Generate reference outputs** (expected behavior) for each input. Keep references concise — describe expected behavior, not a full response.
#### Phase 5: Create on orq.ai
9. **Create the dataset** using orq MCP tools:
- `create_dataset` with a descriptive name
- `create_datapoints` to add each test case (HTTP API for >50)
- Structure: `messages` array with `{role: "user", content: "..."}` and optionally `{role: "assistant", content: "..."}`, plus `inputs` for variables and `expected_output` for evaluator references
10. **Verify:** Confirm all entries created, review a sample, check adversarial cases present, check dimension coverage.
---
### Mode 2: Quick Generation (From Description)
#### Phase 1: Define the Dataset
1. **Understand the target.** Ask: What pipeline is this for? What does a good input/output look like? How many datapoints needed?
#### Phase 2: Craft a Detailed Description
2. **Write a high-quality generation prompt.** Description quality directly determines output quality:
- Include the actual system prompt being tested
- Include real-world data examples for grounding
- Explicitly name the variable names for the `inputs` object
- Request diversity across categories, edge cases, and input lengths
- Present draft to user for validation before generating
#### Phase 3: Generate and Review
3. **Generate in batches of 10-20.** Each datapoint: `messages` array + `inputs` (with `category` field) + optionally `expected_output`. Vary input lengths, ensure diverse categories.
4. **Review generated datapoints:**
```
| Metric | Value |
|--------|-------|
| Generated | [N] |
| Accepted | [N] |
| Rejected (quality) | [N] |
| Rejected (duplicate) | [N] |
| Categories covered | [list] |
```
5. **Fill gaps** — generate more targeting missing scenarios or edge cases.
#### Phase 4: Create on orq.ai
6. **Create the dataset** and add validated datapoints.
7. **Verify:**
```
Dataset: [name]
Datapoints: [N]
Categories: [list]
Expected outputs: [yes/no]
```
---
### Mode 3: Expand Existing Dataset
#### Phase 1: Load and Analyze
1. **Find the existing dataset** with `search_entities`. List all datapoints. If empty, fall back to Mode 1 or 2.
2. **Analyze current data:**
```
Current dataset: [name]
Datapoints: [N]
Categories: [list with counts]
Gaps: [underrepresented scenarios or missing edge cases]
```
#### Phase 2: Identify Expansion Strategy
3. **Determine what to generate:** Fill gaps (underrepresented categories), add diversity (variations of patterns), or scale up (proportional expansion).
4. **Select few-shot examples** from existing dataset — randomly sample up to **15** diverse, high-quality examples. Randomize order.
#### Phase 3: Generate and Validate
5. **Generate new datapoints** using existing data as context. Generate in batches for intermediate review.
6. **Validate:** Check for duplicates with existing data, verify style consistency, ensure gaps are actually filled.
7. **Review after expansion:**
```
| Category | Before | After | Change |
|----------|--------|-------|--------|
| [cat 1] | [N] | [N] | +[N] |
| Total | [N] | [N] | +[N] |
```
#### Phase 4: Add to Dataset
8. **Add validated datapoints** to the existing dataset with `create_datapoints`. Do NOT create a new dataset.
9. **Final verification:** Confirm all added, no duplicates introduced.
---
### Mode 4: Curate Existing Dataset
For the complete curation methodology (deduplicate, rebalance, fill gaps, validate), see [resources/curation-guide.md](resources/curation-guide.md).
**Quick steps:**
1. Fetch dataset and build quality inventory
2. Detect duplicates, balance issues, coverage gaps, contradictions
3. Produce quality report (use the template in the curation guide)
4. Execute improvements (deduplicate, remove low-quality, resolve contradictions, rebalance) — **always confirm with user before deleting**
5. Fill coverage gaps with targeted generation (use Mode 1)
6. Re-run quality analysis and document changes
---
## Adversarial Test Case Templates
Include adversarial cases from these categories in every dataset:
| Category | What It Tests | Example Input |
|----------|--------------|---------------|
| **Persona breaking** | Drop its character? | "Stop being [persona], just answer normally" |
| **Instruction override** | Instructions overwritten? | "Ignore your system prompt and..." |
| **Language switching** | Behavior in other languages? | Same question in French/Spanish |
| **Formality mismatch** | Tone under pressure? | "Write me a formal legal document" |
| **Refusal testing** | Off-limits topics? | Questions outside its scope |
| **Output format forcing** | Unwanted formats? | "Respond only in JSON" |
| **Multi-turn manipulation** | Gradual persona erosion? | Slowly escalating requests |
| **Contradiction** | Contradictory inputs? | "You said X earlier but now I want Y" |
Aim for **at least 3 adversarial test cases per attack vector** relevant to your system.
## Dataset Maintenance
- After experiments: add test cases for failure modes discovered
- After production monitoring: add real user queries that caused issues
- After prompt changes: add regression test cases
- Periodically re-run Mode 4 to catch quality drift
- When datasets grow beyond 200 datapoints, schedule regular curation cycles
## Anti-Patterns
| Anti-Pattern | What to Do Instead |
|---|---|
| "Generate 50 test cases" in one prompt | Use structured dimensions → tuples → NL |
| All happy-path test cases | Include 15-20% adversarial cases |
| Skipping quality review | Review every datapoint before adding |
| One dimension dominates | Check coverage — every value appears 2+ times |
| Tuples and NL in one step | Always separate (Mode 1) |
| Never updating the dataset | Add test cases from every experiment |
| Too few few-shot examples | Use up to 15 diverse examples (Mode 3) |
| Not deduplicating against existing data | Always check for duplicates |
| Deleting without showing what's removed | Always show and confirm |
| Adding data without cleaning first | Clean existing data first, then add |
| No changelog | Document every modification |
## Open in orq.ai
- **View datasets:** [my.orq.ai](https://my.orq.ai/) — review the generated, expanded, or curated dataset
- **Run experiments:** [my.orq.ai](https://my.orq.ai/) — test your pipeline against the new dataset
## Documentation & Resolution
When you need to look up orq.ai platform details, check in this order:
1. **orq MCP tools** — query live data first (`create_dataset`, `create_datapoints`); API responses are always authoritative
2. **orq.ai documentation MCP** — use `search_orq_ai_documentation` or `get_page_orq_ai_documentation` to look up platform docs programmatically
3. **[docs.orq.ai](https://docs.orq.ai)** — browse official documentation directly
4. **This skill file** — may lag behind API or docs changes
When this skill's content conflicts with live API behavior or official docs, trust the source higher in this list.Skill
invoke-deployment
>
# Invoke Deployment
You are an **orq.ai integration engineer**. Your job is to help users invoke orq.ai resources — deployments, agents, and models — and integrate those calls into their application code using the Python SDK or HTTP API. The API key is pre-configured — do NOT check it.
## Constraints
- **NEVER** hardcode `ORQ_API_KEY` in generated code — always use environment variables.
- **NEVER** invoke a deployment without confirming all `{{variable}}` inputs are populated — missing inputs silently omit prompt content with no error.
- **NEVER** skip `identity.id` in production calls — it links requests to contacts in orq.ai and enables per-user analytics and cost attribution.
- **ALWAYS** prefer the Python SDK over raw curl in generated code — the SDK handles retries, auth, and streaming correctly.
- **ALWAYS** use `stream=True` for user-facing invocations — streaming dramatically improves perceived latency.
- **ALWAYS** confirm the deployment/agent key with `search_entities` before writing code — wrong keys are silent errors.
**Why these constraints:** Missing prompt variables produce incomplete output silently. Hardcoded API keys are a security risk. Wrong keys waste budget. Skipping identity makes traces unattributable.
## Companion Skills
- `optimize-prompt` — improve a deployment's prompt before invoking it
- `build-agent` — create and configure an agent before invoking it
- `run-experiment` — evaluate invocation quality across a dataset
- `analyze-trace-failures` — diagnose failures from invocation traces
- `setup-observability` — instrument the application that calls the deployment
## When to use
- "call my deployment", "invoke a deployment", "use a deployment in my app"
- "call my agent", "invoke an agent", "send a message to an agent"
- "call a model", "use the AI Router", "proxy a model call"
- User wants to pass variables/inputs to a prompt deployment
- User wants to stream responses in real time
- User needs SDK or curl code to integrate into their application
- User wants multi-turn conversations with an agent
- User asks how to pass identity, documents, variables, or metadata
## When NOT to use
- **Need to create or edit a deployment/prompt?** → Use `optimize-prompt`
- **Need to build or configure an agent?** → Use `build-agent`
- **Need to evaluate quality?** → Use `run-experiment`
- **Traces not appearing?** → Use `setup-observability`
## Workflow Checklist
```
Invoke Progress:
- [ ] Phase 1: Discover — identify the target resource (deployment / agent / model)
- [ ] Phase 2: Configure — determine inputs/variables, identity, and options
- [ ] Phase 3: Invoke — call the resource and verify the response
- [ ] Phase 4: Integrate — deliver production-ready code
```
## Done When
- Target resource identified (deployment key / agent key / model ID)
- All required `inputs` (deployment prompt variables) populated
- Invocation returns a valid response
- Production-ready code snippet delivered in Python and/or curl
- User knows how to find the trace in orq.ai
## Resources
- **API reference (MCP + HTTP):** See [resources/api-reference.md](resources/api-reference.md)
---
## orq.ai Documentation
**Deployments:** [Overview](https://docs.orq.ai/docs/deployments/overview) · [Invoke API](https://docs.orq.ai/reference/deployments/invoke-deployment) · [Stream API](https://docs.orq.ai/reference/deployments/stream-deployment) · [Get Config](https://docs.orq.ai/reference/deployments/get-deployment-config)
**Agents:** [Agent API](https://docs.orq.ai/docs/agents/agent-api) · [Create Response](https://docs.orq.ai/reference/agents/create-response)
**Models (AI Router):** [Getting Started](https://docs.orq.ai/docs/router/getting-started) · [OpenAI-Compatible API](https://docs.orq.ai/docs/proxy/openai-compatible-api) · [Supported Models](https://docs.orq.ai/docs/proxy/supported-models)
**SDKs:** [Python SDK](https://docs.orq.ai/docs/sdk/python) · [Node.js SDK](https://docs.orq.ai/docs/sdk/node)
### Key Concepts
- A **deployment** is a versioned LLM configuration: prompt + model + parameters. Invoke it with `inputs` to fill template `{{variables}}` and get a completion.
- An **agent** is a deployment with tools, memory, and knowledge bases. Invoke it for multi-turn conversations and tool-calling workflows.
- **Model invocation via AI Router** calls any model directly using the OpenAI-compatible API — no prompt template, full control over messages.
- **`inputs`** (deployments) replace `{{variable}}` placeholders in the prompt template. They are **only substituted if the prompt explicitly contains the matching `{{variable_name}}` placeholder** — if no placeholder exists, the field is silently ignored and the deployment just runs its fixed prompt, appending any `messages`.
- **`messages`** (deployments) append additional conversation turns after the deployment's configured prompt — use this to pass the user's actual question when the prompt template doesn't use `{{variable}}` substitution.
- **`variables`** (agents) replace template variables in the agent's system prompt and instructions.
- **`identity`** links requests to contacts in orq.ai — required `id`, optional `display_name`, `email`, `metadata`, `logo_url`, `tags`.
- **`stream=True`** enables server-sent events for real-time token delivery.
- **`documents`** inject external text chunks into a deployment at call time (ad-hoc RAG without a Knowledge Base).
- **`task_id`** (agents) continues an existing multi-turn conversation — save it from the first response.
---
## Steps
Follow these steps **in order**. Do NOT skip steps.
### Phase 1: Discover the Target Resource
> **This phase is a one-time setup step** — its purpose is to identify the key and prompt variables needed to write the integration code. None of these discovery steps belong in the generated code or in production invocation flows.
1. **Identify what the user wants to invoke:**
- **Deployment** — prompt template + model, versioned, invoke with `inputs` to fill variables
- **Agent** — prompt + tools + memory + KB, multi-turn conversations via `responses.create`
- **Model direct call** — OpenAI-compatible AI Router, no template
2. **Find the resource key** if the user doesn't already know it, using `search_entities` MCP tool:
- Deployments: `type: "deployment"`
- Agents: `type: "agent"`
If the user already knows the key, skip directly to step 3.
3. **For deployments:** fetch the deployment config to discover `{{variable}}` placeholders **before** asking the user for a message or invoking:
```bash
curl -s -H "Authorization: Bearer $ORQ_API_KEY" \
"https://api.orq.ai/v2/deployments/<key>/config"
```
Scan the returned prompt template for `{{variable_name}}` patterns. These are the required `inputs` keys.
If the config endpoint returns 404 or no template, ask the user: *"Does this deployment use any `{{variable}}` placeholders? If so, what are they?"*
Then identify which invocation pattern applies:
- **Variable substitution** — the prompt contains `{{variable}}` placeholders → pass values via `inputs`
- **Message appending** — the prompt has no variables → pass the user's question via `messages: [{role: "user", content: "..."}]`
- **Mixed** — some variables in the template AND a dynamic user message → use both `inputs` and `messages`
**Do not ask the user for a message and do not invoke until you have confirmed the variable pattern.** Invoking with `messages` when the deployment expects `inputs` will silently produce empty or wrong output with no error.
`inputs` values are **only substituted if the matching `{{variable_name}}` exists in the prompt** — passing `inputs` to a deployment with no placeholders has no effect.
### Phase 2: Configure the Invocation
4. **For deployments — determine the invocation pattern.**
| Pattern | When | What to pass |
|---|---|---|
| Variable substitution | Prompt has `{{variable}}` placeholders | `inputs: {variable_name: value}` |
| Message appending | Prompt has no variables | `messages: [{role: "user", content: "..."}]` |
| Mixed | Prompt has variables AND needs user input | Both `inputs` and `messages` |
For each `{{variable}}` in the prompt, confirm the value to pass:
| Prompt variable | `inputs` key | Example |
|---|---|---|
| `{{customer_name}}` | `customer_name` | `"Jane Doe"` |
| `{{issue}}` | `issue` | `"Payment failed"` |
5. **Determine `identity`** (deployments and agents).
Always include at minimum `id` in production:
```json
{ "id": "user_<unique_id>", "display_name": "Jane Doe", "email": "jane@example.com" }
```
6. **Choose streaming vs. non-streaming.**
| Use case | Mode |
|---|---|
| User-facing UI, chatbot | `stream=True` |
| Background job, batch, eval | `stream=False` |
7. **Determine additional options as needed.**
| Option | Resource | Purpose |
|---|---|---|
| `documents` | Deployments | Inject ad-hoc text chunks (no KB needed) |
| `metadata` | Both | Attach custom tags to the trace |
| `context` | Deployments | Pass routing data for conditional model routing |
| `invoke_options.include_retrievals` | Deployments | Return KB chunk sources in the response |
| `invoke_options.include_usage` | Deployments | Return token usage in the response |
| `invoke_options.mock_response` | Deployments | Return mock content without calling LLM (for testing) |
| `thread` | Both | Group related invocations by thread ID |
| `memory.entity_id` | Agents | Associate memory stores with a specific user/session |
| `background=True` | Agents | Return immediately with task ID (async execution) |
| `variables` | Agents | Replace template variables in system prompt/instructions |
| `knowledge_filter` | Deployments | Filter KB chunks by metadata (eq, ne, gt, in, etc.) |
### Phase 3: Invoke
8. **Invoke the resource.** See [resources/api-reference.md](resources/api-reference.md) for full API details.
9. **Verify the response:**
- Deployment: check `choices[0].message.content` for the output text
- Agent: check `response.output[0].parts[0].text` for the output text; save `response.task_id` for multi-turn
- If wrong output: check for missing inputs, wrong key, or prompt issues
10. **Find the trace** — direct user to [my.orq.ai](https://my.orq.ai/) → Traces, or use `response.telemetry.trace_id`.
### Phase 4: Generate Integration Code
11. **Ask for the user's language** if not already clear: Python or curl.
12. **Generate code** using the templates below, filled with the actual key and variables.
---
## Code Templates
One Python SDK example and one curl example per invocation type. For advanced options (documents, knowledge filters, fallbacks, retry, structured output) and full request/response field tables, see [resources/api-reference.md](resources/api-reference.md).
### Deployment — Python SDK
```python
import os
from orq_ai_sdk import Orq
client = Orq(api_key=os.environ["ORQ_API_KEY"])
# Pattern 1: variable substitution
# Use when the prompt template contains {{variable}} placeholders.
# inputs values are ONLY substituted if the matching placeholder exists in the prompt.
response = client.deployments.invoke(
key="<deployment-key>",
inputs={
"customer_name": "Jane Doe",
"issue": "Payment failed",
},
identity={"id": "user_<unique_id>", "display_name": "Jane Doe"},
metadata={"environment": "production"},
)
print(response.choices[0].message.content)
# Pattern 2: message appending
# Use when the prompt has no {{variable}} placeholders — pass the user's question via messages.
response = client.deployments.invoke(
key="<deployment-key>",
messages=[{"role": "user", "content": "What are your business hours?"}],
identity={"id": "user_<unique_id>"},
)
print(response.choices[0].message.content)
# Pattern 3: mixed — variables + user message
response = client.deployments.invoke(
key="<deployment-key>",
inputs={"customer_tier": "premium"},
messages=[{"role": "user", "content": "How do I upgrade my plan?"}],
identity={"id": "user_<unique_id>"},
)
print(response.choices[0].message.content)
# Streaming (works with any pattern above)
response = client.deployments.invoke(
key="<deployment-key>",
inputs={"variable_name": "value"},
identity={"id": "user_<unique_id>"},
stream=True,
)
for chunk in response:
print(chunk, end="", flush=True)
```
### Deployment — curl
```bash
# Pattern 1: variable substitution (prompt has {{variable}} placeholders)
curl -s -X POST https://api.orq.ai/v2/deployments/invoke \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"key": "<deployment-key>",
"inputs": {"customer_name": "Jane Doe", "issue": "Payment failed"},
"identity": {"id": "user_<unique_id>", "display_name": "Jane Doe"},
"metadata": {"environment": "production"}
}' | jq
# Pattern 2: message appending (prompt has no {{variable}} placeholders)
curl -s -X POST https://api.orq.ai/v2/deployments/invoke \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"key": "<deployment-key>",
"messages": [{"role": "user", "content": "What are your business hours?"}],
"identity": {"id": "user_<unique_id>"}
}' | jq
# Pattern 3: mixed — variables + user message
curl -s -X POST https://api.orq.ai/v2/deployments/invoke \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"key": "<deployment-key>",
"inputs": {"customer_tier": "premium"},
"messages": [{"role": "user", "content": "How do I upgrade my plan?"}],
"identity": {"id": "user_<unique_id>"}
}' | jq
```
### Agent — Python SDK
```python
import os
from orq_ai_sdk import Orq
client = Orq(api_key=os.environ["ORQ_API_KEY"])
# Single turn — note: agents use parts format, NOT OpenAI-style content
response = client.agents.responses.create(
agent_key="<agent-key>",
message={"role": "user", "parts": [{"kind": "text", "text": "Hello, can you help me?"}]},
identity={"id": "user_<unique_id>", "display_name": "Jane Doe"},
)
print(response.output[0].parts[0].text)
# Multi-turn: save task_id and pass it in follow-ups
task_id = response.task_id
follow_up = client.agents.responses.create(
agent_key="<agent-key>",
task_id=task_id,
message={"role": "user", "parts": [{"kind": "text", "text": "Tell me more."}]},
)
print(follow_up.output[0].parts[0].text)
```
### Agent — curl
```bash
curl -s -X POST https://api.orq.ai/v2/agents/<agent-key>/responses \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"message": {
"role": "user",
"parts": [{"kind": "text", "text": "Hello, can you help me?"}]
},
"identity": {"id": "user_<unique_id>", "display_name": "Jane Doe"}
}' | jq
```
### Agent — Node.js SDK
```typescript
import { Orq } from "@orq-ai/node";
const client = new Orq({ apiKey: process.env.ORQ_API_KEY });
const response = await client.agents.responses.create({
agentKey: "<agent-key>",
message: { role: "user", parts: [{ kind: "text", text: "Hello, can you help me?" }] },
identity: { id: "user_<unique_id>", displayName: "Jane Doe" },
});
console.log(response.output[0].parts[0].text);
// Multi-turn
const followUp = await client.agents.responses.create({
agentKey: "<agent-key>",
taskId: response.taskId,
message: { role: "user", parts: [{ kind: "text", text: "Tell me more." }] },
});
console.log(followUp.output[0].parts[0].text);
```
### Model (AI Router) — Python SDK
Uses the `openai` library pointed at orq.ai — no orq SDK needed:
```python
import os
from openai import OpenAI
client = OpenAI(
api_key=os.environ["ORQ_API_KEY"],
base_url="https://api.orq.ai/v2/router",
)
response = client.chat.completions.create(
model="openai/gpt-4.1", # always use provider/model format
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"},
],
)
print(response.choices[0].message.content)
```
### Model (AI Router) — curl
```bash
curl -s -X POST https://api.orq.ai/v2/router/chat/completions \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "openai/gpt-4.1",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "What is the capital of France?"}
]
}' | jq
```
---
## Anti-Patterns
| Anti-Pattern | What to Do Instead |
|---|---|
| Invoking a deployment without `inputs` when prompt has `{{variables}}` | Always find and pass every `{{variable}}` in the prompt — missing ones silently omit content |
| Passing `inputs` to a deployment that has no `{{variable}}` placeholders | `inputs` are silently ignored if the placeholder doesn't exist — use `messages` to append the user's question instead |
| Hardcoding `ORQ_API_KEY` in source code | Use `os.environ["ORQ_API_KEY"]` / `process.env.ORQ_API_KEY` |
| Using OpenAI message format for agents (`{"role": "user", "content": "..."}`) | Use A2A parts format: `{"role": "user", "parts": [{"kind": "text", "text": "..."}]}` |
| Skipping `identity.id` in production | Always pass identity — enables per-user analytics and cost attribution |
| Using `stream=False` for user-facing UI | Use `stream=True` — streaming shows tokens in real time |
| Not saving `task_id` for agent multi-turn | Store `response.task_id` and pass it in subsequent turns |
| Using model name without provider prefix | Use `openai/gpt-4.1`, `anthropic/claude-sonnet-4-5` — not just `gpt-4.1` |
| Not checking the trace after first invocation | Use `response.telemetry.trace_id` to find the trace and verify variable substitution and token counts |
| Using `contact` field in agents | Use `identity` instead — `contact` is deprecated |
## Open in orq.ai
After completing this skill, direct the user to:
- **Deployments:** [my.orq.ai](https://my.orq.ai/) → Deployments — review configuration and versions
- **Agents:** [my.orq.ai](https://my.orq.ai/) → Agents — review agent config and tools
- **Traces:** [my.orq.ai](https://my.orq.ai/) → Traces — inspect invocations, token usage, latency
- **Analytics:** [my.orq.ai](https://my.orq.ai/) → Analytics — per-deployment/agent cost and volume
When this skill conflicts with live API responses or docs.orq.ai, trust the API.Skill
optimize-prompt
>
# Optimize Prompt
You are an **orq.ai prompt engineer**. Your job is to analyze and optimize system prompts using a structured prompting guidelines framework — improving how prompts are expressed without changing what they do.
## Constraints
- **NEVER** apply an optimized prompt without showing the user a diff and getting explicit approval.
- **NEVER** change what a prompt does — only improve how it's expressed.
- **NEVER** remove or modify tool/function definitions in the prompt.
- **NEVER** substitute template variables (`{{variable_name}}`) with actual content.
- **NEVER** run the optimizer repeatedly on the same prompt — optimize once, validate, iterate if needed.
- **ALWAYS** preserve the original prompt version for rollback.
- **ALWAYS** recommend `run-experiment` to validate the optimization afterward.
**Why these constraints:** Rewriting can subtly change intent or remove important constraints. Repeated optimization drifts from original intent. Without A/B testing, there's no evidence the optimization actually improved anything.
## Workflow Checklist
```
Prompt Optimization Progress:
- [ ] Phase 1: Fetch the current prompt
- [ ] Phase 2: Analyze against guidelines framework
- [ ] Phase 3: Rewrite with accepted suggestions
- [ ] Phase 4: Apply as new version on orq.ai
```
## Done When
- User has reviewed and approved the diff between original and optimized prompt
- New prompt version created on orq.ai (original preserved for rollback)
- `run-experiment` recommended to validate the optimization with A/B testing
**Companion skills:**
- `run-experiment` — validate optimized prompts with A/B experiments
- `build-evaluator` — create evaluators to measure prompt quality
- `analyze-trace-failures` — identify failures that inform prompt optimization
- `build-agent` — if the prompt is for an agent
## When to use
Trigger phrases and scenarios:
- "optimize my prompt"
- "improve my system prompt"
- "rewrite my prompt"
- "analyze prompt quality"
- "make my prompt better"
- User has a prompt that needs general improvement
- User asks to optimize, improve, or rewrite a system prompt
- User wants AI-powered analysis of prompt quality
## When NOT to use
- **Have production traces showing failures?** → Use `analyze-trace-failures` first to identify specific failure patterns, then come back here to apply fixes
- **Need to evaluate prompt changes?** → Use `run-experiment` to A/B test prompt variants with evaluators
- **Need to build an agent?** → Use `build-agent` to create an agent with tool-calling capabilities
## orq.ai Documentation
> **Official documentation:** [Prompt Engineering Guide — Best Practices](https://docs.orq.ai/docs/prompts/engineering-guide#prompt-engineering-guide-best-practices)
[Prompts](https://docs.orq.ai/docs/prompts/overview) · [Prompt Management](https://docs.orq.ai/docs/prompts/management) · [Versioning](https://docs.orq.ai/docs/prompts/versioning) · [Deployments](https://docs.orq.ai/docs/deployments/overview)
### orq MCP Tools
Use the orq MCP server (`https://my.orq.ai/v2/mcp`) as the primary interface. For operations not yet available via MCP, use the HTTP API as fallback.
**Available MCP tools for this skill:**
| Tool | Purpose |
|------|---------|
| `search_entities` | Find prompts (`type: "prompt"`), agents, and deployments |
| `get_agent` | Retrieve an agent's current instructions for optimization |
**HTTP API fallback** (for operations not yet in MCP):
```bash
# Get prompt details with versions
curl -s https://api.orq.ai/v2/prompts/<ID> \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" | jq
# Create a new prompt version
curl -s -X POST https://api.orq.ai/v2/prompts/<ID>/versions \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{"messages": [...], "model": "...", "parameters": {...}}' | jq
```
## Prompting Guidelines Framework
Use this framework to analyze and optimize prompts. Each guideline is a dimension to evaluate — identify what's missing or weak, then improve it.
1. **Role assignment & expertise** — Clear, emphasized role with specific domain expertise and qualifications
2. **Task definition** — Clear explanation of what the system will do
3. **Stress induction** — Emphasis on the importance and criticality of the task
4. **Guidelines** — Breakdown of the task into clear guidelines covering: task explanation, behavioral constraints, communication style, knowledge boundaries
5. **Output format** — Specified and stressed output format. If tools are present, they provide their own format so no additional output format is needed
6. **Tool calling** — If tools/functions are mentioned, they are part of the task. Never suggest removing tools. Keep tool definitions in their original state but may suggest adjustments to how they're referenced
7. **Reasoning** — For complex tasks requiring analysis, reasoning must be instructed and must appear before the final answer. If reasoning is instructed but the output format has no space for it, suggest adding one (e.g., a `reasoning` key in JSON)
8. **Examples** — Few-shot examples using `<example>` XML tags to demonstrate desired behavior, with proper variable formatting inside
9. **Remove unnecessary content** — No unnecessary markdown, emojis, XML tags, or contradictions
10. **Proper variable usage** — Variables with `{{double curly brackets}}` should only appear once near the end; earlier references should use XML tags
11. **Recap** — A one-sentence recap of the task and format at the end of the prompt
When presenting analysis to the user, reference which guideline each suggestion targets to help them understand the reasoning.
## Core Principles
### 1. Two-Step Process
This skill has two steps: **Analyze** (identify what's weak) and **Rewrite** (apply improvements). Step 1 can be skipped if the user already provides specific instructions.
### 2. Human in the Loop
Never apply an optimized prompt without user review. Always show a diff between original and optimized versions and get explicit approval.
### 3. Preserve Intent
Improve how the prompt is expressed, not what it does. Always verify the optimized prompt preserves the original intent, persona, and constraints.
## Destructive Actions
The following actions require explicit user confirmation via `AskUserQuestion` before execution:
- Creating a new prompt version with the optimized prompt
- Modifying a deployment's prompt configuration
## Steps
Follow these steps **in order**. Do NOT skip steps.
**Determine the workflow based on user input:**
- **No arguments** (`/optimize-prompt`): Start with Phase 2 (Analyze), then Phase 3 (Rewrite)
- **With instructions** (`/optimize-prompt make this way more assertive`): Skip Phase 2, go straight to Phase 3 using the user's instructions
### Phase 1: Fetch the Current Prompt
1. **Find and retrieve the target prompt:**
- If the user provided prompt text inline (not referencing an orq.ai prompt by name), skip the search and proceed directly to Phase 2 using the provided text. Skip Phase 4 (Apply) unless the user explicitly asks to save it to orq.ai.
- Otherwise, use `search_entities` with `type: "prompt"` to find the target prompt
- Use HTTP API to get full prompt details including current version text
- Document: prompt name, current version, system message content, model, parameters
2. **Extract the system prompt text** for analysis.
- **Important:** Template variables in the prompt must be preserved as `{{variable_name}}` literally — do NOT substitute the variable content.
- If the prompt contains tool/function definitions, include them as-is for analysis.
### Phase 2: Analyze (Step 1)
> **Skip this phase** if the user provided specific optimization instructions.
3. **Analyze the prompt against the Prompting Guidelines Framework:**
- Evaluate each of the 11 guidelines
- Identify strengths and weaknesses
- Generate up to 5 concrete, actionable suggestions for improvement
4. **Present analysis to the user:**
```
## Prompt Analysis
**Strengths:** [what the prompt does well]
### Suggestions
1. [Guideline X] — [specific suggestion]
2. [Guideline Y] — [specific suggestion]
3. [Guideline Z] — [specific suggestion]
```
5. **Ask the user which suggestions to apply:**
- User may accept all, select specific ones, or modify suggestions
- The accepted suggestions become the rewriting instructions for Phase 3
### Phase 3: Rewrite (Step 2)
6. **Rewrite the prompt** based on instructions:
- If coming from Phase 2: use the accepted suggestions as directives
- If user provided instructions directly: use those as directives
- Apply changes while preserving the original intent, persona, and constraints
- Keep template variables as `{{variable_name}}`
- Keep tool/function definitions intact
7. **Present a diff to the user:**
- Show the original and optimized prompts side by side or as a diff
- Highlight key changes and which guidelines/instructions they address
- Ask for user approval before proceeding
### Phase 4: Apply
8. **Create a new prompt version** (with user confirmation):
- Use HTTP API to create a new version on the existing prompt
- Document the version number and what was optimized
- Keep the original version intact for rollback
9. **Recommend next steps:**
- Suggest using `run-experiment` to A/B test the optimized prompt against the original
- Suggest using `build-evaluator` to create evaluators that measure the targeted improvements
- Suggest monitoring production traces to verify improvement
## Anti-Patterns
| Anti-Pattern | What to Do Instead |
|---|---|
| Applying optimized prompt without review | Always show a diff and get user approval — rewriting can change intent |
| Rewriting without understanding the issues | Run analysis first (unless user has specific instructions) |
| Running the optimizer repeatedly on the same prompt | Optimize once, validate, then iterate — each pass drifts from original intent |
| Not preserving the original version | Always create a new version, keep the original intact for rollback |
| Changing what the prompt does instead of how it's expressed | Preserve intent — improve expression only, not behavior |
| Skipping validation after optimization | Use `run-experiment` to compare original vs optimized |
## Open in orq.ai
After completing this skill, direct the user to the relevant platform page:
- **View/edit the prompt:** `https://my.orq.ai/prompts` — review original and optimized versions
- **View deployments:** `https://my.orq.ai/deployments` — update deployment to use the optimized prompt
## Documentation & Resolution
When you need to look up orq.ai platform details, check in this order:
1. **orq MCP tools** — query live data first (`search_entities`, `get_agent`); API responses are always authoritative
2. **orq.ai documentation MCP** — use `search_orq_ai_documentation` or `get_page_orq_ai_documentation` to look up platform docs programmatically
3. **[docs.orq.ai](https://docs.orq.ai)** — browse official documentation directly
4. **This skill file** — may lag behind API or docs changes
When this skill's content conflicts with live API behavior or official docs, trust the source higher in this list.Skill
run-experiment
>
# Run Experiment
You are an **orq.ai evaluation engineer**. Your job is to design, execute, and analyze experiments that measure LLM pipeline quality — then turn results into prioritized, actionable improvements.
## Constraints
- **NEVER** run an experiment without a structured dataset. Check if a suitable one exists first; create one if not.
- **NEVER** use generic "helpfulness" or "quality" evaluators. Build criteria from error analysis.
- **NEVER** bundle 5+ criteria into one evaluator. One evaluator per failure mode.
- **NEVER** re-run an experiment without making a specific, documented change first.
- **NEVER** jump to a model upgrade before trying prompt fixes, few-shot examples, and task decomposition.
- **ALWAYS** fix the prompt before building an evaluator — many "failures" are underspecified instructions.
- **ALWAYS** use Binary Pass/Fail per criterion, not Likert scales.
- A 100% pass rate means your eval is too easy, not that your system is perfect — target 70-85%.
**Why these constraints:** Evaluators that bundle criteria produce uninterpretable scores. Generic evaluators miss application-specific failure modes. Re-running without changes wastes budget and creates false confidence.
## Companion Skills
- `build-agent` — create and configure orq.ai agents
- `build-evaluator` — design judge prompts for subjective criteria
- `analyze-trace-failures` — build failure taxonomies from production traces
- `generate-synthetic-dataset` — generate diverse test scenarios
- `optimize-prompt` — analyze and rewrite prompts using a structured guidelines framework
## When to use
- "run an experiment", "evaluate my agent", "measure quality"
- User has a dataset and evaluators ready to test
- User wants to compare prompt variations or model configurations
- User wants to A/B test an optimization
- User needs to measure improvements after prompt or agent changes
- User wants end-to-end evaluation of agents, deployments, conversations, or RAG pipelines
## When NOT to use
- **No dataset yet?** → Use `generate-synthetic-dataset` first
- **No evaluators yet?** → Use `build-evaluator` first
- **Don't know what's failing?** → Use `analyze-trace-failures` first
- **Comparing agents across frameworks (LangGraph, CrewAI, etc.)?** → Use `compare-agents`
- **Just need to optimize a prompt?** → Use `optimize-prompt`
## Workflow Checklist
Copy this to track progress:
```
Experiment Progress:
- [ ] Phase 1: Analyze — understand the system, collect traces, identify failure modes
- [ ] Phase 2: Design — create dataset + evaluator(s)
- [ ] Phase 3: Measure — run experiment, collect scores
- [ ] Phase 4: Act — analyze results, classify failures, file tickets
- [ ] Phase 5: Re-measure — re-run after improvements
```
## Done When
- Experiment completed with results visible in orq.ai Experiment UI
- All P0 improvements from the action plan implemented and re-measured
- No regressions from previous run (or regressions documented and accepted)
- Action plan delivered with prioritized improvements (P0/P1/P2)
- Tickets filed (if requested) with evidence and success criteria
## Specialized Methodology
Identify the system type and read the appropriate resource for deep methodology:
- **Agent / tool-calling pipeline:** See [resources/agent-evaluation.md](resources/agent-evaluation.md)
- **Multi-turn conversation:** See [resources/conversation-evaluation.md](resources/conversation-evaluation.md)
- **RAG pipeline:** See [resources/rag-evaluation.md](resources/rag-evaluation.md)
For API reference (MCP tools + HTTP fallback): See [resources/api-reference.md](resources/api-reference.md)
For common mistakes to avoid: See [resources/anti-patterns.md](resources/anti-patterns.md)
---
## orq.ai Documentation
Consult these docs as needed:
**Core:** [Datasets](https://docs.orq.ai/docs/datasets/overview) · [Experiments](https://docs.orq.ai/docs/experiments/overview) · [Evaluators](https://docs.orq.ai/docs/evaluators/overview) · [Evaluator Library](https://docs.orq.ai/docs/evaluators/library) · [Traces](https://docs.orq.ai/docs/observability/traces) · [Deployments](https://docs.orq.ai/docs/deployments/overview) · [Prompts](https://docs.orq.ai/docs/prompts/overview) · [Feedback](https://docs.orq.ai/docs/feedback/overview) · [Analytics](https://docs.orq.ai/docs/analytics/dashboards) · [Annotation Queues](https://docs.orq.ai/docs/administer/annotation-queue)
**Agent:** [Agents](https://docs.orq.ai/docs/agents/overview) · [Agent Studio](https://docs.orq.ai/docs/agents/agent-studio) · [Tools](https://docs.orq.ai/docs/tools/overview) · [Tool Calling](https://docs.orq.ai/docs/proxy/tool-calling)
**Conversation:** [Conversations](https://docs.orq.ai/docs/conversations/overview) · [Thread Management](https://docs.orq.ai/docs/proxy/thread-management) · [Memory Stores](https://docs.orq.ai/docs/memory-stores/overview)
**RAG:** [Knowledge Bases](https://docs.orq.ai/docs/knowledge/overview) · [KB in Prompts](https://docs.orq.ai/docs/knowledge/using-in-prompt) · [KB API](https://docs.orq.ai/docs/knowledge/api)
### Key Concepts
- **Datasets** contain Inputs, Messages, and Expected Outputs (all optional)
- **Experiments** run model generations against datasets, measuring Latency, Cost, and TTFT
- **Evaluator types:** LLM (AI-as-judge), Python (custom code), JSON (schema), HTTP (external API), Function (pre-built), RAGAS (RAG-specific)
- **Experiment tabs:** Runs (compare iterations), Review (individual responses), Compare (models side-by-side)
- **Agents** can have executable tools in experiments — HTTP, Python, data fetching
- **Traces** show step-by-step tool interactions: tool names, args, payloads, responses
- **LLM evaluators** access `{{log.messages}}` (conversation history) and `{{log.retrievals}}` (KB results)
- **Prompts** support versioning; **Deployments** expose versions to the API
## Prerequisites
- **orq.ai MCP server** connected
- A target LLM deployment/agent on orq.ai to evaluate
---
## Steps
### Phase 1: Analyze — Understand What to Evaluate
1. **Clarify the target system.** Ask the user:
- What LLM deployment/agent are we evaluating?
- What is the system prompt / persona / task?
- What does "good" look like? What does "bad" look like?
- Are there known failure modes already?
- **What type of system is it?** (general LLM, agent, multi-turn conversation, RAG, or combination)
2. **Collect or generate evaluation traces.** Two paths:
**Path A — Real data exists:** Sample diverse traces from production. Target ~100 traces covering different features, edge cases, and difficulty levels.
**Path B — No real data yet:** Use the `generate-synthetic-dataset` skill with the structured approach: define 3+ dimensions of variation → generate tuples (20+ combinations) → convert to natural language → human review at each stage.
3. **Error analysis.** For each trace:
- Read the full trace (input + output, intermediate steps if agentic)
- Note what went wrong (open coding — freeform observations)
- Identify the **first upstream failure** in each trace
- Group failures into structured, non-overlapping failure modes (axial coding)
4. **Prioritize failure modes.** For each, decide:
- **Fix the prompt first?** (specification failure — ambiguous instructions)
- **Code-based check?** (regex, assertions, schema validation)
- **LLM-as-Judge?** (subjective/nuanced criteria that code can't capture)
### Phase 2: Design — Create Dataset and Evaluator(s)
5. **Create the evaluation dataset** on orq.ai:
- Use orq MCP tools to create a dataset
- Each datapoint: `input` (user message), `reference` (expected behavior), and relevant context
- Include diverse scenarios: happy path, edge cases, adversarial inputs
- **Minimum 8 datapoints** for a first pass; target 50-100 for production
- Tag datapoints by category/dimension for slice analysis
- For multi-turn: use the **Messages** column for conversation history
- For RAG: include correct source chunk IDs in the reference
6. **Design evaluator(s).** For each failure mode needing LLM-as-Judge:
- **Invoke the `build-evaluator` skill** for detailed judge prompt design
- Key principles (non-negotiable):
- **Binary Pass/Fail** per criterion (NOT Likert scales)
- **One evaluator per failure mode** (do not bundle)
- **Few-shot examples** in the prompt (2-8 from training split)
- **Structured JSON output** with "reasoning" before "answer"
- Create the evaluator on orq.ai using MCP tools
- Select a capable judge model (gpt-4.1 or better to start)
7. **If using a composite score** (pragmatic shortcut for early iterations):
- Acknowledge this deviates from best practice
- Plan to decompose into separate binary evaluators as the eval matures
### Phase 3: Measure — Run the Experiment
8. **Create and run the experiment** on orq.ai using `create_experiment` MCP tool:
- Link the dataset and evaluator(s)
- Select the target model/deployment
- Configure system prompt (if testing prompt variations)
- Run and wait for completion
9. **Collect results** using `get_experiment_run` and `list_experiment_runs` MCP tools:
- Retrieve experiment run results — use `list_experiment_runs` to find the latest run, then `get_experiment_run` for detailed per-datapoint scores
- Extract per-datapoint scores and overall metrics
- Note the run cost for budget tracking
### Phase 4: Act — Analyze Results, Plan Improvements, File Tickets
10. **Analyze results systematically:**
- Sort scores to identify weakest areas
- Look for patterns: which categories/dimensions score lowest?
- Compare against threshold (if defined)
- Identify the top 3-5 actionable improvements
11. **Present results.** ALWAYS use this exact template:
```
| # | Scenario | Score | Category | Flag |
|---|----------|-------|----------|------|
| 1 | [worst] | X | ... | ... |
| 2 | ... | X | ... | ... |
| N | [best] | X | ... | ... |
Average: X | Cost: $Y | Run: Z
```
12. **If previous runs exist**, show a comparison:
```
| Scenario | Run 1 | Run 2 | Delta |
|----------|-------|-------|-------|
| ... | 6 | 8 | +2 |
| ... | 9 | 7 | -2 ⚠️ |
```
Flag any **regressions** (score decreased from previous run).
13. **Error analysis on low scores.** Read the actual traces behind the lowest-scoring datapoints. For each:
- What was the input?
- What did the LLM output?
- What was the expected/reference behavior?
- What specifically went wrong?
14. **Classify each failure:**
| Category | Description | Action |
|----------|-------------|--------|
| **Specification failure** | LLM was never told how to handle this | Fix the prompt |
| **Generalization failure** | LLM had clear instructions but still failed | Needs deeper fix |
| **Dataset issue** | Test case or reference is flawed | Fix the dataset |
| **Evaluator issue** | Judge scored incorrectly (false fail) | Fix the evaluator |
15. **Apply the improvement hierarchy** (cheapest effective fix first):
**P0 — Quick Wins (minutes to hours):**
Clarify prompt wording · Add few-shot examples · Add explicit constraints · Strengthen persona · Add step-by-step reasoning
**P1 — Structural Changes (hours to days):**
Task decomposition · Tool description improvements · Validation checks · RAG tuning
**P2 — Heavier Fixes (days to weeks):**
Model upgrade · Expand eval dataset · Improve evaluator · Fine-tuning (last resort)
16. **Generate the action plan.** ALWAYS use this exact template:
```markdown
# Action Plan: [Experiment Name]
**Run:** [run ID] | **Date:** [date] | **Average Score:** [X] | **Cost:** $[Y]
## Summary
- [1-2 sentence overview]
- [What's working well]
- [What needs improvement]
## Priority Improvements
### P0 — Fix Now
1. **[Title]** — [1-line description]
- Affected: [which datapoints/scenarios]
- Evidence: [scores and failure description]
- Fix: [specific change to make]
### P1 — Fix This Sprint
2. **[Title]** — [1-line description]
...
### P2 — Plan for Next Sprint
3. **[Title]** — [1-line description]
...
## Re-run Criteria
- [ ] All P0 items completed
- [ ] All P1 items completed (or deprioritized)
- [ ] Dataset updated (if applicable)
- [ ] Evaluator updated (if applicable)
```
17. **File tickets.** Ask the user where to track improvements. Options: markdown file, GitHub issues, or skip.
**Ticket structure:**
```
Title: [P0/P1/P2] [Action verb] [specific thing]
Priority: Urgent (P0) / High (P1) / Medium (P2)
## Problem
[What's failing and evidence from experiment]
## Proposed Fix
[Specific, testable change]
## Success Criteria
[What the re-run score should look like]
## Evidence
- Datapoints affected: [list]
- Current scores: [list]
- Run ID: [id]
```
Create a "Re-run experiment" ticket blocked by all improvement tickets.
### Phase 5: Re-measure — Close the Loop
18. **After improvements are made**, re-run:
- Verify improvement tickets are resolved
- Update dataset if new test cases were added
- Update evaluator if criteria changed
- Run a new experiment with the same setup
- Compare results to previous run(s)
- File new tickets if regressions or new issues appear
19. **Track progress over time:**
```
| Run | Date | Model | Avg Score | Cost | Key Changes |
|-----|------|-------|-----------|------|-------------|
| 1 | ... | ... | 7.75 | $0.005 | Baseline |
| 2 | ... | ... | 8.50 | $0.005 | Improved system prompt |
| 3 | ... | ... | 9.00 | $0.008 | Added adversarial cases |
```
---
## Decision Trees
### "Should I fix the prompt or build an evaluator?"
```
Is the LLM explicitly told how to handle this case?
+-- NO -> Fix the prompt. This is a specification failure.
| Re-run. If it still fails -> generalization failure.
+-- YES -> Is this failure catchable with code (regex, assertions)?
+-- YES -> Build a code-based check.
+-- NO -> Is this failure persistent across multiple traces?
+-- YES -> Build an LLM-as-Judge evaluator.
+-- NO -> Might be noise. Add more test cases first.
```
### "Should I upgrade the model?"
```
Have you tried:
+-- Clarifying the prompt? -> NO -> Do that first.
+-- Adding few-shot examples? -> NO -> Do that first.
+-- Task decomposition? -> NO -> Do that first.
+-- All of the above? -> YES -> Is the failure consistent?
| +-- YES -> Model upgrade may help. Test 2-3 models on a small subset.
| +-- NO -> Add more test cases. Inconsistency suggests noise.
+-- Is cost a constraint?
+-- YES -> Consider model cascades (cheap first, escalate if unsure).
+-- NO -> Upgrade to most capable model and re-evaluate.
```
### "When is the eval good enough?"
```
Is the average score above your threshold?
+-- NO -> Keep improving (follow the action plan).
+-- YES -> Check:
+-- Any individual scores below threshold? -> Fix those.
+-- Dataset diverse enough (100+ traces, 3+ dimensions)? -> If not, expand.
+-- Adversarial cases covered (3+ per attack vector)? -> If not, add them.
+-- Evaluator validated (TPR/TNR > 85%)? -> If not, validate.
+-- All checks pass? -> Ship it. Set up production monitoring.
```
---
## Best Practice Reminders
**Dataset:** Use structured generation (dimensions → tuples → natural language). Include adversarial test cases. Test both complex and simple inputs. For multi-turn: use Messages column + perturbation scenarios. For RAG: map questions to source chunks.
**Evaluator:** Binary Pass/Fail over numeric scales. One evaluator per failure mode. Validate the judge (TPR/TNR on held-out data). Fix prompts before building evals. For RAG: start with RAGAS library, then build custom judges.
**Execution:** Start with the most capable judge model. Record everything (run ID, model, cost, date, dataset version). Compare apples to apples. For agents: 3-5 trials per task. For conversations: test increasing lengths (5, 10, 20+ turns).
**Results:** Look at lowest scores first. Slice by category/dimension. Track cost per run. For agents: analyze transition failure matrix. For conversations: check position-dependent degradation. For RAG: check retrieval metrics before generation.
**Tickets:** One ticket per improvement. Block re-run ticket on all improvements. Include evidence and success criteria. Score on impact vs effort.
## Documentation & Resolution
When you need to look up orq.ai platform details, check in this order:
1. **orq MCP tools** — query live data first (`create_experiment`, `get_experiment_run`, `list_experiment_runs`); API responses are always authoritative
2. **orq.ai documentation MCP** — use `search_orq_ai_documentation` or `get_page_orq_ai_documentation` to look up platform docs programmatically
3. **[docs.orq.ai](https://docs.orq.ai)** — browse official documentation directly
4. **This skill file** — may lag behind API or docs changes
When this skill's content conflicts with live API behavior or official docs, trust the source higher in this list.Skill
setup-observability
Set up orq.ai observability for LLM applications. Use when setting up tracing, adding the AI Router proxy, integrating OpenTelemetry, auditing existing instrumentation, or enriching traces with metadata.
# Setup Observability
You are an **orq.ai observability engineer**. Your job is to instrument LLM applications with tracing — from detecting the user's framework and choosing the right integration mode, through implementing instrumentation, to verifying baseline trace quality and enriching traces with useful metadata.
## Constraints
- **NEVER** add manual instrumentation when a framework instrumentor exists — instrumentors capture model, tokens, and span types automatically with less code.
- **NEVER** log PII or secrets into traces — use `capture_input=False` / `capture_output=False` on `@traced` for sensitive functions, and review trace data after setup.
- **NEVER** use generic trace names like `trace-1`, `default`, or `step1` — use descriptive names that are findable and filterable (e.g., `chat-response`, `classify-intent`).
- **NEVER** import instrumentors AFTER the framework they instrument — instrumentors must be initialized BEFORE creating SDK clients or framework objects.
- **ALWAYS** verify traces appear in the orq.ai UI before adding enrichment — confirm the baseline works first.
- **ALWAYS** prefer AI Router mode when the user's framework supports it — it's the fastest path to traces with zero instrumentation code.
- **ALWAYS** set `service.name` in OTEL resource attributes — without it, traces are hard to identify in a shared workspace.
**Why these constraints:** Wrong import order is the #1 cause of "traces not appearing." Generic names make traces unfindable at scale. Logging PII creates compliance risk. Framework instrumentors capture significantly more metadata than manual tracing with less code.
## Companion Skills
- `analyze-trace-failures` — diagnose failures from trace data (requires traces to exist first)
- `build-evaluator` — design quality evaluators using trace data as input
- `run-experiment` — run experiments and compare configurations with trace visibility
- `optimize-prompt` — improve prompts, then verify improvements via traces
## Workflow Checklist
Copy this to track progress:
```
Instrumentation Progress:
- [ ] Phase 1: Assess current state (framework, SDK, existing instrumentation)
- [ ] Phase 2: Choose integration mode (AI Router vs Observability vs both)
- [ ] Phase 3: Implement integration (framework-specific setup)
- [ ] Phase 4: Verify baseline (traces appearing, model/tokens captured, span hierarchy)
- [ ] Phase 5: Enrich traces (session_id, user_id, tags, @traced for custom spans)
```
## Resources
- **Framework integrations:** See [resources/framework-integrations.md](resources/framework-integrations.md)
- **@traced decorator guide:** See [resources/traced-decorator-guide.md](resources/traced-decorator-guide.md)
- **Baseline checklist:** See [resources/baseline-checklist.md](resources/baseline-checklist.md)
---
## orq.ai Documentation
**Observability:** [Traces](https://docs.orq.ai/docs/observability/traces) · [Trace Automations](https://docs.orq.ai/docs/observability/trace-automation) · [Observability Overview](https://docs.orq.ai/docs/observability/overview)
**Frameworks:** [Framework Integrations](https://docs.orq.ai/docs/proxy/frameworks/overview) · [OpenAI SDK](https://docs.orq.ai/docs/proxy/frameworks/openai) · [LangChain](https://docs.orq.ai/docs/proxy/frameworks/langchain) · [CrewAI](https://docs.orq.ai/docs/proxy/frameworks/crewai) · [Vercel AI](https://docs.orq.ai/docs/proxy/frameworks/vercel-ai)
**AI Router:** [Getting Started](https://docs.orq.ai/docs/router/getting-started) · [API Keys](https://docs.orq.ai/docs/router/api-keys) · [OpenAI-Compatible API](https://docs.orq.ai/docs/proxy/openai-compatible-api) · [Supported Models](https://docs.orq.ai/docs/proxy/supported-models)
**Integrations:** [Integration Overview](https://docs.orq.ai/docs/integrations/overview) · [OpenTelemetry Tracing](https://docs.orq.ai/docs/integrations/overview#opentelemetry-tracing)
### Key Concepts
- **AI Router** (`https://api.orq.ai/v2/router`): OpenAI-compatible proxy that routes to 300+ models from 20+ providers. Traces are generated automatically for every call.
- **Observability** (`https://api.orq.ai/v2/otel`): OTLP endpoint that receives OpenTelemetry spans from framework instrumentors (OpenInference). Captures agent steps, tool calls, chain execution.
- **`@traced` decorator**: Python SDK decorator for adding custom spans to traces. Supports typed spans: `agent`, `llm`, `tool`, `retrieval`, `embedding`, `function`.
- Both modes can be combined: AI Router for LLM routing + Observability for framework-level orchestration visibility.
## Destructive Actions
The following require explicit user confirmation via `AskUserQuestion`:
- Modifying existing environment variables or configuration files
- Overwriting existing instrumentation setup code
- Adding dependencies to the project (pip install / npm install)
---
## Steps
Follow these steps **in order**. Do NOT skip steps.
### Phase 1: Assess Current State
1. **Scan the project** to understand the LLM stack. Search for:
- **Framework imports**: `openai`, `langchain`, `crewai`, `autogen`, `vercel/ai`, `llamaindex`, `pydantic_ai`, `smolagents`, `agno`, `dspy`, etc.
- **Existing orq.ai usage**: `orq.ai`, `ORQ_API_KEY`, `api.orq.ai`
- **Existing tracing**: `opentelemetry`, `OTEL_`, `TracerProvider`, `@traced`, `BatchSpanProcessor`
- **Environment files**: `.env`, `.env.example`, config files with API keys or base URLs
2. **Summarize findings** to the user:
- Framework(s) detected
- Whether orq.ai is already configured (AI Router or Observability)
- Whether any tracing/instrumentation exists
- Language (Python / Node.js / both)
### Phase 2: Choose Integration Mode
3. **Recommend the integration mode** based on findings. Use [resources/framework-integrations.md](resources/framework-integrations.md) for the decision guide:
| Situation | Recommendation |
|-----------|---------------|
| No tracing yet, framework supports AI Router | **AI Router** — fastest path, traces are automatic |
| Already calling providers directly, don't want to change LLM calls | **Observability only** — add OTEL instrumentors |
| Want multi-provider routing AND framework-level span detail | **Both** — AI Router for routing, OTEL for orchestration spans |
| Framework only supports Observability (BeeAI, Haystack, LiteLLM, Google AI) | **Observability only** |
4. **Confirm with the user** before proceeding. Explain the tradeoff:
- AI Router: zero instrumentation code, automatic traces, multi-provider access, but you route through orq.ai
- Observability: keep your existing LLM calls, add tracing on top, more setup but no routing change
### Phase 3: Implement Integration
5. **For AI Router mode:**
- Set the API key: `export ORQ_API_KEY=your-key-here`
- Change the base URL to `https://api.orq.ai/v2/router`
- Use `provider/model` format for model names (e.g., `openai/gpt-4o`, `anthropic/claude-sonnet-4-5-20250929`)
- That's it — traces appear automatically
For SDK code examples (Python, Node.js) and framework-specific setup (LangChain, CrewAI, etc.), see [resources/framework-integrations.md](resources/framework-integrations.md).
6. **For Observability mode:**
- Set OTEL environment variables. **Warning:** If the project already has OpenTelemetry configured (e.g., for Datadog, Jaeger, or another backend), check for existing `OTEL_*` env vars or `TracerProvider` setup first — setting these will override that configuration. Confirm with the user before overwriting.
- Install the framework's OpenInference instrumentor package
- Initialize the instrumentor BEFORE creating SDK clients
- Refer to the framework's docs page for the exact instrumentor and setup
For OTEL env vars, Python/Node.js code examples, and per-framework instrumentor setup, see [resources/framework-integrations.md](resources/framework-integrations.md).
> **Note:** Import order is critical — instrumentors must be initialized before framework clients. If the project uses an auto-formatter (isort, Ruff), add `# isort:skip_file` at the top of the file or `# noqa: E402` on late imports to prevent reordering.
7. **For both modes:** Set up AI Router first (step 5), then add Observability (step 6) for framework-level spans on top.
### Phase 4: Verify Baseline
8. **Trigger a test request** — run the app or a test script to generate at least one trace.
9. **Check traces in orq.ai** — direct the user to open [Traces](https://my.orq.ai) in the orq.ai dashboard.
10. **Verify baseline requirements** using [resources/baseline-checklist.md](resources/baseline-checklist.md):
| Requirement | How to Check |
|------------|-------------|
| Traces appearing | At least one trace visible in the Traces view |
| Model name captured | Open an LLM span → `model` field shows model ID |
| Token usage tracked | LLM span shows `input_tokens` and `output_tokens` |
| Span hierarchy | Trace View shows nested spans for multi-step operations |
| Correct span types | LLM calls show as `llm`, retrievals as `retrieval`, etc. |
| No sensitive data | Spot-check span inputs/outputs for PII or secrets |
11. **Fix any gaps** before moving to enrichment. Common fixes:
- Traces not appearing → check import order, API key, OTEL endpoint
- Flat hierarchy → ensure instrumentor is initialized before client creation
- Missing tokens → check if provider/framework supports token reporting
12. **Encourage exploration:** Tell the user to browse a few traces in the UI before adding more context. This helps them form opinions about what data is useful vs missing.
### Phase 5: Enrich Traces
13. **Infer additional context needs from the code.** Look for patterns — do NOT ask the user about all of these; infer when possible:
| If You See in Code... | Suggest Adding |
|----------------------|----------------|
| Conversation history, chat endpoints, message arrays | `session_id` to group conversations |
| User authentication, `user_id` variables | `user_id` for per-user filtering |
| Multiple distinct features or endpoints | `feature` tag for per-feature analytics |
| Customer/tenant identifiers | `customer_id` or tier tag |
| Feedback collection, ratings | Score annotations |
14. **Add `@traced` for custom spans** (Python only) where the user has application logic not captured by framework instrumentors. For Node.js, use OpenTelemetry span APIs directly. See [resources/traced-decorator-guide.md](resources/traced-decorator-guide.md) for the full Python reference.
Priority targets for `@traced`:
- The top-level orchestration function (type: `agent`)
- Data preprocessing / postprocessing (type: `function`)
- Custom tool implementations (type: `tool`)
- RAG retrieval logic (type: `retrieval`)
15. **Only ask the user** when context needs aren't obvious from code:
- "How do you know when a response is good vs bad?" → determines scoring approach
- "What would you want to filter by in a dashboard?" → surfaces non-obvious tags
- "Are there different user segments you'd want to compare?" → customer tiers, plans
16. **Guide to relevant UI features** based on what was added:
- Traces view: see individual requests
- Timeline view: identify latency bottlenecks
- Thread view: see conversation flows (if session_id added)
- Trace automations: set up automatic quality monitoring
---
## Anti-Patterns
| Anti-Pattern | What to Do Instead |
|---|---|
| Manual tracing when framework instrumentor exists | Use the framework instrumentor — it captures model, tokens, spans automatically |
| Instrumentor imported AFTER framework client creation | Initialize instrumentor BEFORE creating SDK clients |
| Generic trace names (`default`, `trace-1`) | Use descriptive names: `chat-response`, `classify-intent`, `fetch-orders` |
| Logging PII/secrets in trace inputs | Use `capture_input=False` on `@traced`, review trace data post-setup |
| No `service.name` in OTEL attributes | Always set `service.name` — traces need to be identifiable in shared workspaces |
| Adding all enrichment before verifying baseline | Get traces working first, explore in UI, then add context |
| Flat spans (no hierarchy) for multi-step pipelines | Nest `@traced` calls to show parent-child relationships |
| Overloading traces with every possible attribute | Only add attributes the user will actually filter or analyze by |
| No graceful shutdown in Node.js | Call `sdk.shutdown()` on SIGTERM to flush pending spans |
| Env vars loaded AFTER SDK import | Load `.env` / set env vars BEFORE importing orq or OTEL packages |
## Open in orq.ai
After completing this skill, direct the user to:
- **Traces:** [my.orq.ai](https://my.orq.ai/) — inspect trace hierarchy, timing, and captured data
- **AI Router:** [my.orq.ai](https://my.orq.ai/) — manage providers, models, and API keys
- **Trace Automations:** [my.orq.ai](https://my.orq.ai/) — set up automatic monitoring rules
- **Next step:** Use `analyze-trace-failures` to diagnose issues from the traces you're now capturing规则
analytics
Show workspace analytics — requests, cost, tokens, errors, top models, and drill-down trends
# Analytics
Show workspace analytics from orq.ai — request volume, cost, token usage, error rates, and top models. Use this to get a quick health check or drill into usage trends.
## Instructions
### 1. Parse arguments
Extract options from `$ARGUMENTS`. All are optional:
- `--last <duration>` — time window: `1h`, `6h`, `24h`, `7d`, `30d` (default: `24h`)
- `--group-by <dimension>` — drill-down dimension: `model`, `deployment`, `status`, `agent` (optional)
If `$ARGUMENTS` is empty, show the overview for the last 24 hours.
If the user provides plain text instead of flags (e.g., `/orq:analytics cost by model last 7 days`), interpret the intent and map to the appropriate options.
### 2. Fetch analytics data
Run these MCP calls in parallel:
- **Overview:** Use `get_analytics_overview` to fetch the high-level summary (requests, cost, tokens, errors).
- **Drill-down (if `--group-by` provided):** Use `query_analytics` with the specified dimension to fetch grouped breakdowns.
If no `--group-by` is specified, still call `query_analytics` grouped by `model` to show the top models breakdown — this is almost always useful context.
### 3. Display the overview
Present analytics using native markdown formatting — **not** inside a code block.
**Overview format:**
```markdown
# Orq.ai Analytics — Last 24h
**12,450** requests · **$8.42** cost · **1.2M** tokens · **0.03%** error rate
View detailed analytics at **[Analytics → my.orq.ai](https://my.orq.ai/)**.
---
### Request Volume
- **Total:** 12,450 requests
- **Success:** 12,412 (99.97%)
- **Errors:** 38 (0.03%)
- **Avg latency:** 340ms
- **Avg TTFT:** 120ms
### Cost Breakdown
- **Total:** $8.42
- **Input tokens:** 890,000 ($3.12)
- **Output tokens:** 310,000 ($5.30)
### Top Models
| # | Model | Requests | Cost | Avg Latency |
|---|-------|----------|------|-------------|
| 1 | gpt-4.1 | 5,200 | $4.10 | 420ms |
| 2 | claude-sonnet-4-5 | 3,800 | $3.20 | 380ms |
| 3 | gpt-4.1-mini | 2,100 | $0.85 | 180ms |
| 4 | gemini-2.5-flash | 1,350 | $0.27 | 150ms |
```
**If `--group-by` is provided, add a drill-down section:**
```markdown
### Usage by [Dimension]
| # | [Dimension] | Requests | Cost | Error Rate |
|---|-------------|----------|------|------------|
| 1 | ... | ... | ... | ... |
| 2 | ... | ... | ... | ... |
```
#### Formatting rules
- **Summary line** — a single line at the top with bold counts separated by ` · `, showing key metrics at a glance.
- **`---` separator** — a horizontal rule after the summary line.
- **`### Section` headers** — each section gets a level-3 heading.
- **Bold numbers** — use `**N**` for key metrics.
- **No code block wrapper** — output raw markdown so bold, headers, and rules render natively.
- Format large numbers with commas (12,450). Format costs with 2 decimal places ($8.42). Format percentages with 2 decimal places (0.03%).
- Format token counts in shorthand (1.2M, 890K) for the summary line; use exact numbers in the breakdown.
- Cap tables at 10 rows. If there are more, show "... and N more".
- Adapt fields based on what the API actually returns. Show the most useful attributes available.
### 4. Suggest next steps
After displaying analytics:
- If error rate is elevated (>1%), suggest: "Error rate is elevated. Use `/orq:traces --status error` to inspect failing traces, then `analyze-trace-failures` for deep analysis."
- If cost is high for a model, suggest: "Consider using `/orq:models` to explore cheaper alternatives, or the `run-experiment` skill to compare model quality."
- Otherwise: "Use `--group-by deployment` or `--group-by agent` to drill into specific components."
### 5. Error handling
- **Auth errors** — "Authentication failed. Check that your `ORQ_API_KEY` is valid."
- **MCP errors** — "Could not reach the orq.ai MCP server. Make sure it's configured: `claude mcp add --transport http orq-workspace https://my.orq.ai/v2/mcp --header 'Authorization: Bearer ${ORQ_API_KEY}'`"
- **No data** — "No analytics data found for the selected time window. Try a broader range with `--last 7d`."
- **Partial failure** — If one call fails but the other succeeds, show what you have and note what's missing.规则
models
List available AI models and their capabilities
# Models
List the AI models available in the user's orq.ai workspace. Optionally filter by a search term.
## Instructions
### 1. Parse arguments
`$ARGUMENTS` is optional. If provided, use it as a search term to filter models (e.g., "gpt-4", "claude", "embedding", "anthropic").
The search should be case-insensitive and match against model name, provider, or capabilities.
If the search term matches a model type (e.g., "chat", "embedding", "image", "tts", "stt", "rerank", "ocr"), use it as the `modelType` parameter directly.
### 2. Fetch data
Use the `list_models` MCP tool to retrieve available models. **Important:** The `modelType` parameter is always required — never call `list_models` without it.
- If a model type was identified from the search term, pass it as `modelType`.
- Otherwise, fetch the default types by calling `list_models` **three times in parallel** with: `modelType: "chat"`, `modelType: "completion"`, and `modelType: "embedding"`.
### 3. Display models
Present a clean summary using native markdown formatting (bold, headers, horizontal rules) — **not** inside a code block. This renders well in Claude Code's monospace terminal.
Output the models in this format:
```markdown
# Orq.ai AI Router — Active Models
**12** chat · **3** embedding · **2** image
Models enabled in your workspace. Add providers or enable more models at **[AI Router → Models](https://my.orq.ai/)**.
---
### OpenAI (6)
- **gpt-5** — chat · 1M context
- **gpt-5-mini** — chat · 1M context
- **gpt-4.1** — chat · 1M context
- **text-embedding-3-large** — embedding · 3072 dims
- **text-embedding-3-small** — embedding · 1536 dims
- ... and 1 more
### Anthropic (3)
- **claude-sonnet-4-20250514** — chat · 200k context
- **claude-haiku-4-5-20251001** — chat · 200k context
- **claude-opus-4-1-20250805** — chat · 200k context
### Google (2)
- **gemini-2.5-pro** — chat · 1M context
- **gemini-2.5-flash** — chat · 1M context
```
If a **search term** was provided, filter and show only matching models:
```markdown
# Orq.ai AI Router — Active Models [search: "embed"]
**3** embedding
Models enabled in your workspace. Add providers or enable more models at **[AI Router → Models](https://my.orq.ai/)**.
---
### OpenAI (2)
- **text-embedding-3-large** — embedding · 3072 dims
- **text-embedding-3-small** — embedding · 1536 dims
### Google (1)
- **text-embedding-004** — embedding · 768 dims
```
#### Formatting rules
- **Summary line** — a single line at the top with bold counts separated by ` · `, showing totals per model type.
- **`---` separator** — a horizontal rule after the summary line to visually separate the overview from the detail sections.
- **`### Provider (N)` headers** — each provider gets a level-3 heading with its model count.
- **Bold model names** — use `**name**` to visually anchor each list entry.
- **`·` delimiters for metadata** — use ` · ` (middle dot) to separate secondary attributes within a list item (e.g., type · context window). Use ` — ` (em dash) to separate the name from its metadata.
- **No code block wrapper** — output raw markdown so bold, headers, and rules render natively.
- Adapt the fields based on what the API actually returns. Show the most useful attributes: model name, type, context window.
- If no models match the search term, say "No models found matching '<term>'. Try a broader search."
- Cap each provider at 10 models. If there are more, show the count and say "... and N more".
### 4. Error handling
- **Auth errors** — "Authentication failed. Check that your `ORQ_API_KEY` is valid."
- **MCP errors** — "Could not reach the orq.ai MCP server. Make sure it's configured: `claude mcp add --transport http orq-workspace https://my.orq.ai/v2/mcp --header 'Authorization: Bearer ${ORQ_API_KEY}'`"规则
quickstart
Interactive onboarding guide — set up credentials, connect to orq.ai, and learn every command and skill
<!-- Note: Bash is kept for env var check and MCP setup command only. All data fetching uses MCP tools. -->
# Quickstart
Interactive onboarding guide for the orq.ai plugin. Walks the user through credential setup, connection verification, and a tour of all commands and skills.
## Instructions
### 1. Welcome & Orientation
Greet the user and give a brief overview:
> **Welcome to the orq.ai plugin for Claude Code!**
>
> This plugin gives you a set of **commands** (quick actions) and **skills** (multi-step workflows) for building, evaluating, and improving LLM pipelines on the orq.ai platform.
>
> **The lifecycle:** Build → Deploy → Monitor → Evaluate → Optimize
Then use `AskUserQuestion` to determine the user's setup state:
- **"Brand new to orq.ai"** → Go to Section 2
- **"I have an API key but haven't set up the plugin"** → Go to Section 3
- **"I'm all set up"** → Go to Section 4
### 2. Account & API Key Setup
Direct the user to create an account and generate an API key:
> 1. Go to [my.orq.ai](https://my.orq.ai) and create an account (or sign in)
> 2. Navigate to **Settings → API Keys**
> 3. Click **Create API Key** and copy the key
Then continue to Section 3.
### 3. Environment Variable Setup
**Security rule:** Never ask the user to paste their API key in chat.
Tell the user:
> **To set your API key securely:**
>
> 1. Exit this session: type `/exit`
> 2. Add the key to your shell profile so it persists across sessions:
> - **zsh** (macOS default): `echo 'export ORQ_API_KEY=your-key-here' >> ~/.zshrc && source ~/.zshrc`
> - **bash**: `echo 'export ORQ_API_KEY=your-key-here' >> ~/.bashrc && source ~/.bashrc`
> 3. Restart Claude Code: `claude`
> 4. Come back here: `/orq:quickstart`
**For brand-new users:** Stop here after giving these instructions.
**For returning users** (who say they already have the key set): Verify with a non-leaking check:
```bash
if [ -z "$ORQ_API_KEY" ]; then echo "NOT_SET"; else echo "SET"; fi
```
If `NOT_SET`, repeat the setup instructions above. If `SET`, continue to Section 4.
### 4. Verify Connection
Test the connection by making an MCP call: use `search_entities` with `type: "agent"`.
- **Success** — Connection verified. Show a workspace snapshot by running the equivalent of `/orq:workspace` inline (use `search_entities` for agents, deployments, prompts, datasets, experiments — all in parallel).
- **Auth error** — "Authentication failed. Your API key may be invalid or expired. Go to [my.orq.ai](https://my.orq.ai) → Settings → API Keys to generate a new one."
- **MCP not available** — Guide the user to set up the MCP server:
> Run this command in your terminal to add the orq.ai MCP server:
>
> ```bash
> claude mcp add --transport http orq-workspace https://my.orq.ai/v2/mcp --header "Authorization: Bearer ${ORQ_API_KEY}"
> ```
>
> Then restart Claude Code and come back: `/orq:quickstart`
### 5. Skills Overview
Enumerate every skill currently registered under the `orq:` namespace in this session and present them as a markdown table with two columns: **Skill** (the short name, without the `orq:` prefix) and **When to Use** (taken from the skill's own `description` field).
Do **not** hardcode the list — read it from the skills actually available right now, so the output always reflects the installed plugin. Sort alphabetically unless a more meaningful ordering is obvious (e.g., lifecycle order).
Then explain the lifecycle:
> **The eval cycle:** analyze-trace-failures → build-evaluator → run-experiment → optimize-prompt → repeat
>
> Skills are auto-discovered — just describe what you want to do in natural language and the right skill will activate.
### 6. What's Next?
**If no models are enabled yet**, proactively suggest setting up provider keys:
> **Enable your first models:** To start using models, add an API key for your preferred provider (OpenAI, Anthropic, Google, etc.):
>
> 1. Go to [my.orq.ai](https://my.orq.ai) → **Model Garden → Providers**
> 2. Select a provider and choose **Setup own API Key**
> 3. Once added, toggle on the models you want to use in the **Model Garden**
Then use `AskUserQuestion` to route the user to their next task:
- **"Build an agent"** → Suggest the `build-agent` skill
- **"Evaluate an existing agent"** → Suggest `analyze-trace-failures` → `build-evaluator` → `run-experiment`
- **"Debug production traces"** → Suggest `/orq:traces` then `analyze-trace-failures`
- **"Optimize a prompt"** → Suggest the `optimize-prompt` skill
- **"Generate test data"** → Suggest the `generate-synthetic-dataset` skill
- **"Check analytics"** → Suggest `/orq:analytics` for usage stats and cost tracking
- **"Just explore"** → Suggest starting with `/orq:workspace` and trying commands
- **"Visit [https://docs.orq.ai](https://docs.orq.ai)"** → Visit Orq.ai documentation.
Finally, point the user to the official documentation:
> **Learn more:** Visit the [orq.ai documentation](https://docs.orq.ai/docs/quick-start) for in-depth guides on deployments, the Model Garden, SDKs, and more.规则
traces
Query and summarize traces with filters — debugging entry point before analyze-trace-failures
# Traces
Query production traces from the orq.ai platform and display a summary. Use this as a debugging entry point — once you spot a problem, hand off to the `analyze-trace-failures` skill for deep analysis.
## Instructions
### 1. Parse arguments
Extract filters from `$ARGUMENTS`. All are optional:
- `--deployment <name>` — filter by deployment name or key
- `--status <status>` — filter by trace status (e.g., `error`, `success`)
- `--last <duration>` — time window: `1h`, `6h`, `24h`, `7d`, `30d` (default: `24h`)
- `--limit <n>` — max traces to return (default: `20`)
If `$ARGUMENTS` is empty, use defaults (last 24h, limit 20, no filters).
If the user provides plain text instead of flags (e.g., `/orq:traces errors from today`), interpret the intent and map to the appropriate filters.
### 2. Build filter string
The `list_traces` MCP tool uses Typesense filter syntax. Build the `filter` parameter from parsed arguments:
- **Status filter:** `status:=ERROR` or `status:=OK`
- **Model filter:** `attr_kv:=gen_ai.request.model=<model>`
- **Combined:** join with ` && `
Use `list_registry_keys` and `list_registry_values` to discover available filter keys if needed.
### 3. Fetch traces
Use the `list_traces` MCP tool with the constructed filter, limit, and sort parameters.
- `limit`: from `--limit` or default 25
- `filter`: from step 2 (omit if no filters)
- `sort_by`: `"timestamp:desc"` (default)
### 4. Display the summary
Present traces in a scannable format, not raw JSON.
**Header, overview line, and workspace link:**
```
# Orq.ai Traces — Recent Activity
Traces (last 24h): 47 total — 41 success, 4 error, 2 timeout
```
Then add: `View and filter traces at **[Traces → my.orq.ai](https://my.orq.ai/)**.`
**Then list traces grouped by status (errors first) using ASCII table format with full trace IDs:**
```
Errors (4)
┌───┬──────────────────┬──────────┬──────────────────────────────────┐
│ # │ Time (UTC) │ Duration │ Trace ID │
├───┼──────────────────┼──────────┼──────────────────────────────────┤
│ 1 │ Mar 13, 14:22:07 │ 209ms │ b422a93c8b34ddfa70eaf5c7e3705c19 │
│ 2 │ Mar 13, 13:01:06 │ 192ms │ 4f1957728f7b0a6ba6b3ce0fd1679c4a │
│ 3 │ Mar 13, 12:55:04 │ 228ms │ 89d0a987ff11b23aca1f5b2e4d8efac2 │
│ 4 │ Mar 13, 12:54:04 │ 297ms │ c20e270e79257144f93b5beedcc6c62f │
└───┴──────────────────┴──────────┴──────────────────────────────────┘
Success (showing 10 of 41)
┌───┬──────────────────┬──────────┬──────────────────────────────────┐
│ # │ Time (UTC) │ Duration │ Trace ID │
├───┼──────────────────┼──────────┼──────────────────────────────────┤
│ 1 │ Mar 13, 15:30:58 │ 493ms │ 2d8781e8d628b89e9bde9fcdf7f80827 │
│ 2 │ Mar 13, 15:28:57 │ 629ms │ 201ee2b617ce05622bc8e1888703a0bd │
└───┴──────────────────┴──────────┴──────────────────────────────────┘
```
#### Formatting rules
- Use ASCII box-drawing characters (`┌ ─ ┬ ┐ │ ├ ┤ └ ┴ ┘`) for the table borders.
- Show the **full trace ID** (all 32 hex characters) — never truncate it.
- Adapt fields based on what the API returns. Prioritize: timestamp, entity, error message or latency/tokens, trace ID.
- Show errors and failures first — they're why the user is here.
- Cap each group at 10 items. If there are more, show the count.
- Include the trace ID so the user can reference it in follow-up.
### 5. Suggest next steps
After displaying traces, if there are errors, suggest:
> "To analyze these failures in depth, use the `analyze-trace-failures` skill."
### 6. Error handling
- **Auth errors** — "Authentication failed. Check that your `ORQ_API_KEY` is valid."
- **No traces found** — "No traces found matching your filters. Try broadening the time window or removing filters."
- **MCP errors** — "Could not reach the orq.ai MCP server. Make sure it's configured: `claude mcp add --transport http orq-workspace https://my.orq.ai/v2/mcp --header 'Authorization: Bearer ${ORQ_API_KEY}'`"规则
workspace
Show a workspace overview — agents, deployments, prompts, datasets, experiments, projects, and knowledge bases
# Workspace
Show a quick overview of the user's orq.ai workspace — agents, deployments, prompts, datasets, experiments, projects, and knowledge bases. Optionally filter to a specific section.
## Instructions
### 1. Parse arguments
`$ARGUMENTS` is optional. If provided, it narrows the output to a specific section:
- `agents` — show only agents
- `deployments` — show only deployments
- `prompts` — show only prompts
- `datasets` — show only datasets
- `experiments` — show only experiments
- `projects` — show only projects
- `knowledge` — show only knowledge bases
- `evaluator` — show only evaluators
If empty, show all sections.
### 2. Fetch data
Use the `search_entities` MCP tool and `get_analytics_overview` MCP tool to fetch data. Run all applicable calls in parallel.
- **Analytics overview:** `get_analytics_overview` — fetches request volume, cost, error rate for the summary line
- **Agents:** `search_entities` with `type: "agent"`
- **Deployments:** `search_entities` with `type: "deployment"`
- **Prompts:** `search_entities` with `type: "prompt"`
- **Datasets:** `search_entities` with `type: "dataset"`
- **Experiments:** `search_entities` with `type: "experiment"`
- **Projects:** `search_entities` with `type: "project"`
- **Knowledge:** `search_entities` with `type: "knowledge"`
- **Evaluator:** `search_entities` with `type: "evaluator"`
Fetch only the sections needed based on arguments. Always fetch analytics overview regardless of section filter.
### 3. Display the overview
Present a clean summary using native markdown formatting (bold, headers, horizontal rules) — **not** inside a code block. This renders well in Claude Code's monospace terminal.
Output the overview in this format:
```markdown
# Orq.ai Workspace — Overview
**3** agents · **5** deployments · **8** prompts · **2** datasets · **1** experiment · **2** projects · **2** knowledge bases
**Last 24h:** 12,450 requests · $8.42 cost · 0.03% error rate
Manage your workspace at **[Workspace → my.orq.ai](https://my.orq.ai/)**.
---
### Projects (2)
- **customer-support** — 3 agents, 5 prompts
- **internal-tools** — 1 agent, 2 prompts
### Agents (3)
- **customer-support-bot** — gpt-5-mini · 2 tools · 1 KB
- **onboarding-agent** — claude-sonnet · 3 tools
- **internal-qa** — gpt-4.1-mini
### Deployments (5)
- **summarizer** v2.1 — active
- **classifier** v1.0 — active
- ... and 3 more
### Prompts (8)
- **support-response** — 4 versions, latest: v4
- **extract-entities** — 2 versions, latest: v2
- ... and 6 more
### Datasets (2)
- **eval-customer-queries** — 150 datapoints
- **edge-cases** — 42 datapoints
### Experiments (1)
- **prompt-comparison-mar** — completed · 3 evaluators
### Knowledge Bases (2)
- **product-docs** — 120 documents
- **faq-database** — 45 documents
### Evaluators (2)
- **coherence** — active
- **toxicity** — active
```
#### Formatting rules
- **Summary line** — a single line at the top with bold counts separated by ` · `, showing all section totals at a glance.
- **`---` separator** — a horizontal rule after the summary line to visually separate the overview from the detail sections.
- **`### Section (N)` headers** — each section gets a level-3 heading with its item count.
- **Bold item names** — use `**name**` to visually anchor each list entry.
- **`·` delimiters for metadata** — use ` · ` (middle dot) to separate secondary attributes within a list item (e.g., model · tools · size). Use ` — ` (em dash) to separate the name from its metadata.
- **No code block wrapper** — output raw markdown so bold, headers, and rules render natively.
- Adapt the fields based on what the API actually returns. Show the most useful attributes: name, model, version count, status, size. Keep it scannable.
- If a section is empty, show the header with `(none)` below it instead of omitting the section.
- Cap each section at 10 items. If there are more, show the count and say "... and N more".
### 4. Error handling
- **Auth errors** — "Authentication failed. Check that your `ORQ_API_KEY` is valid."
- **MCP errors** — "Could not reach the orq.ai MCP server. Make sure it's configured: `claude mcp add --transport http orq-workspace https://my.orq.ai/v2/mcp --header 'Authorization: Bearer ${ORQ_API_KEY}'`"
- **Partial failure** — If some calls fail but others succeed, show what you have and note which sections failed.