How to build Harbor eval tasks from traces with LangChain's eval-engineering skill

One-off notebooks do not let you hill-climb harness changes. LangChain’s eval-engineering flow is a two-step pipeline: first turn traces, code, and human input into a detailed Task Spec; then run Spec2Task to produce a Harbor task and the environment that task runs in. A World Spec holds the shared knowledge so you are not rewriting schemas, mock APIs, and rubric rules for every example.

Harbor, in LangChain’s glossary, is the framework for defining tasks and datasets and running evals. A task is an input, an environment, and a test script. The agent runs the instructions inside the environment; a rubric and test script score the result. The skill packages that pipeline so a coding agent can inspect your repo and traces, draft a spec you can actually review, and only then write the Harbor files.

Install the skill, then inspect first

Add the skill to the current project:

npx skills add langchain-ai/langchain-skills --skill eval-engineering --yes

You can use any coding agent. Agents that support Agent Skills can install eval-engineering directly; for others, put the skill instructions in the session.

The skill contract is strict. Inspect all inputs before proposing work: the repository, harness, optional traces, existing tasks and runs, existing world knowledge, and the human goal. Identify the source files and skill references that apply. Then create or update a small project World Knowledge skill from reusable facts in those inputs, and use it to propose one grounded task.

If you are evaluating a Managed Deep Agent, LangSmith’s docs add a workspace step. You need a project created with mda init (or an existing project with an agent entry), uv, Docker, and a coding agent. From the project root:

mda evals init -i

The interactive handoff lists detected coding agents, starts a session in the project directory, and initializes Harbor files under evals/ and .mda/evals/. Ask the agent to follow the skill’s review flow and the Managed Deep Agents layout:

Use the eval-engineering skill to develop Harbor evals for this Managed
Deep Agent. Inspect the project and existing evals first. Draft the Task
Spec and wait for my review before implementing the approved task directly
under evals/<task>/.

LangChain’s process guide uses a similar first prompt against traces and the current repo: load the skill, pull traces from a LangSmith project, and create an eval task for a named agent. Expect back-and-forth. The agent should surface a separate skill for the world spec, then create a task after you agree.

Keep world knowledge out of the individual task

A spec is a markdown file that describes the task in natural language: input, environment, and graders. That split is the point. Figuring out what each task should look like often needs human iteration and alignment with what the team actually cares about. Building the Harbor task from an approved spec can be more automated, versioned, and parallelized. Humans review markdown much more easily than raw task code and data.

World knowledge is not task-specific. It lives in the World Spec and covers facts that apply across the dataset: guidance on what to store (size and shape of data), scripts for parsing traces, how to write good rubrics for this domain (programmatic versus LLM-as-a-judge), scripts that generate data to populate the environment, APIs and schemas for backend services you need to mock, and common questions mined from existing traces.

LangChain’s internal examples include a prompt-optimization benchmark (domain, input shape, output classes, a standard scoring function) and a GTM agent benchmark (Salesforce and Notion schemas, questions mined from traces).

A Task Spec still has to cover three parts unless the World Spec already owns them: what the agent environment looks like, what the inputs should be, and how outputs should be scored. For a general QA chatbot the environment may be identical across tasks, so it can live in the World Spec. For a GTM research task the inputs may stay fixed in the World Spec while the environment and rubric stay in the Task Spec.

The skill tells the agent to draft the Task Spec and World Skill together, show both to you, and keep exact task truth only in Task.md. Refine both until you approve them.

World spec generation is iterative. LangChain’s process is to build the first task hand-in-hand with a coding agent, then have it write up what it learned as a general world spec. You may want two or three tasks before that spec is complete. In practice the agent scans the repo for prompts, tools, and skills; groups traces to find real user patterns; maps credentials and whether live APIs should be called or simulated; catalogs service schemas; and plans synthetic data generation. That first-task loop is how you gather the user feedback the World Spec needs.

Spec2Task into Harbor format

Spec2Task takes an approved spec and generates a task in Harbor format. Pass the coding agent the task spec, the World Spec as a skill, and eval-engineering, which includes guidance on Harbor layout.

On Managed Deep Agents, each direct child of evals/ that contains an instruction and tests is a Harbor task. Beside those folders sits evals/harbor-job.json. Inside a task directory you typically find Task.md, instruction.md, task.toml, an environment/ directory, and tests/ with test.sh plus a verifier.

Task.md is the human-reviewed spec. instruction.md tells the agent what to do. Harbor builds the environment, runs the managed agent, then runs tests/test.sh. The verifier writes a numeric reward to /logs/verifier/reward.txt or numeric metrics to /logs/verifier/reward.json. evals/harbor-job.json is user-owned; Managed Deep Agents writes it only when it is missing, so later edits are preserved. Files under .mda/evals/ are generated.

Do not ship the first generated task as the benchmark. LangChain’s Spec2Task notes: have the pipeline refine tasks by running them with real agents and reading trajectories. That is how you catch leaky environments—overly specific instructions, or a table entry that literally says “Answer placeholder.” Calibrate difficulty by running weaker versus stronger models. That tells you whether a task is too easy, too hard, or broken because a stronger model found a reward hack. Agents are also weak at choosing data-generation methods, so the pipeline gets explicit guidance: LLMs with rubrics for free-text data, scripts with sqlite and specified schemas for tabular data.

After the first task, review the world spec skill, switch threads so you can validate it, and create a second task with both skills loaded. Repeat until you trust the world spec. Then scale: ask the agent to look at recent traces, generate multiple different task specs for review, and run each approved spec through Spec2Task.

Where you still have to judge

The process is not autonomous. Refining specs needs several rounds of feedback on whether the spec matches the real domain, user behavior, and requirements. Agents also tend to create tasks that are too easy—useful for proving the environment works, not useful as a benchmark. Calibrating difficulty means running each task multiple times, reviewing trajectories, and asking the agent to make the task easier or harder.

Once tasks exist, environments become a continuous loop against production data: prompt tuning, harness tuning, post-training, and questions like whether a cheaper model is enough for a subset of work. Update the task or verifier when the eval does not measure the intended behavior; update the agent when it exposes a product failure. Then run the same Harbor job again.

Next step: Install eval-engineering with the command above, then walk one real LangSmith project through LangChain’s first-task prompt. If you are on Managed Deep Agents, run mda evals init -i and do not let the agent implement files until you have approved Task.md.