orq.ai

<skills>

You have additional SKILLs documented in directories containing a "SKILL.md" file.

These skills are:
 - analyze-trace-failures -> "skills/analyze-trace-failures/SKILL.md"
 - build-agent -> "skills/build-agent/SKILL.md"
 - build-evaluator -> "skills/build-evaluator/SKILL.md"
 - compare-agents -> "skills/compare-agents/SKILL.md"
 - generate-synthetic-dataset -> "skills/generate-synthetic-dataset/SKILL.md"
 - invoke-deployment -> "skills/invoke-deployment/SKILL.md"
 - optimize-prompt -> "skills/optimize-prompt/SKILL.md"
 - run-experiment -> "skills/run-experiment/SKILL.md"
 - setup-observability -> "skills/setup-observability/SKILL.md"

IMPORTANT: You MUST read the SKILL.md file whenever the description of the skills matches the user intent, or may help accomplish their task.

<available_skills>

build-agent: `Design, create, and configure an orq.ai Agent with tools, instructions, knowledge bases, and memory — includes model selection, KB management, and memory store setup`

build-evaluator: `Create validated LLM-as-a-Judge evaluators following best practices — binary Pass/Fail judges with TPR/TNR validation for measuring specific failure modes`

analyze-trace-failures: `Read production traces, identify what's failing, build failure taxonomies, and categorize issues using open coding and axial coding methodology`

invoke-deployment: `Invoke orq.ai deployments, agents, and models via the Python SDK or HTTP API — pass prompt variables, stream responses, handle multi-turn agent conversations, and generate integration code`

run-experiment: `Create and run orq.ai experiments — compare configurations against datasets using evaluators, analyze results, with specialized methodology for agent, conversation, and RAG evaluation`

generate-synthetic-dataset: `Generate and curate evaluation datasets — structured generation, quick from description, expansion from existing data, plus dataset maintenance and quality improvement`

optimize-prompt: `Analyze and optimize system prompts using a structured prompting guidelines framework — AI-powered analysis and rewriting`

compare-agents: `Run cross-framework agent comparisons using evaluatorq — compares any combination of agents (orq.ai, LangGraph, CrewAI, OpenAI Agents SDK, Vercel AI SDK) head-to-head on the same dataset with LLM-as-a-judge scoring. Do NOT use when comparing only orq.ai configurations with no external agents (use run-experiment instead).`

setup-observability: `Set up orq.ai observability for LLM applications — AI Router proxy, OpenTelemetry, tracing setup, and trace enrichment. Use when setting up tracing, adding the AI Router proxy, integrating OpenTelemetry, auditing existing instrumentation, or enriching traces with metadata. Do NOT use when traces already exist and you need to debug failures (use analyze-trace-failures).`

</available_skills>

Paths referenced within SKILL folders are relative to that SKILL. For example the build-evaluator `resources/judge-prompt-template.md` would be referenced as `skills/build-evaluator/resources/judge-prompt-template.md`.

</skills>

<commands>

You have slash commands available in the `commands/` directory:

 - quickstart -> "commands/quickstart.md" — Interactive onboarding guide for new users
 - workspace -> "commands/workspace.md" — Show workspace overview (agents, deployments, prompts, datasets, experiments)
 - traces -> "commands/traces.md" — Query and summarize traces with filters (debugging entry point)
 - models -> "commands/models.md" — List available AI models and capabilities
 - analytics -> "commands/analytics.md" — Show workspace analytics (requests, cost, tokens, errors, trends)

Commands are quick-action tools. Skills are multi-step workflows. Use `/orq:quickstart` for onboarding, `/orq:workspace`, `/orq:traces`, `/orq:models`, or `/orq:analytics` for fast operations. Use skills when the user needs guided, multi-step work.

</commands>