Discover errors and usage patterns with the Insights Agent

The Insights Agent automatically analyzes your traces to detect usage patterns, common agent behaviors and failure modes — without requiring you to manually review thousands of traces. Insights uses hierarchical categorization to make sense of your data and highlight actionable trends.

Prerequisites

• An OpenAI API key (generate one here) or an Anthropic API key (generate one here)
• Permissions to create rules in LangSmith (required to generate new Insights Reports)
• Permissions to view tracing projects LangSmith (required to view existing Insights Reports)

Generate your first Insights Report

Auto configuration flow for Insights Agent

From the LangSmith UI:

1. Navigate to Tracing Projects in the left-hand menu and select a tracing project.
2. Click +New in the top right corner then New Insights Report to generate new insights over the project.
3. Enter a name for your job.
4. Click the icon in the top right of the job creation pane to set your OpenAI (or Anthropic) API key as a workspace secret. If your workspace already has an OpenAI API key set, you can skip this step.
5. Answer the guided questions to focus your Insights Report on what you want to learn about your agent, then click Run job.

This will kick off a background Insights Report. Reports can take up to 30 minutes to complete.

Generating insights over 1,000 threads typically costs $1.00-$2.00 with OpenAI models and $3.00-$4.00 with current Anthropic models. The cost scales with the number of threads sampled and the size of each thread.

Understand the results

Once your job has completed, you can navigate to the Insights tab where you’ll see a table of Insights Report. Each Report contains insights generated over a specific sample of traces from the tracing project.

Insights Reports for a single tracing project

Click into your job to see traces organized into a set of auto-generated categories. You can drill down through categories and subcategories to view the underlying traces, feedback, and run statistics.

Common topics of conversations with the https://chat.langchain.com chatbot

Top-level categories

Your traces are automatically grouped into top-level categories that represent the broadest patterns in your data.

The distribution bars show how frequently each pattern occurs, making it easy to spot behaviors that happen more or less than expected.

Each category has a brief description and displays aggregated metrics over the traces it contains, including:

• Typical trace stats (like error rates, latency, cost)
• Feedback scores from your evaluators
• Attributes extracted as part of the job

Subcategories

Clicking on any category shows a breakdown into subcategories, which gives you a more granular understanding of interaction patterns in that category of traces.

In the Chat Langchain example pictured above, under “Data & Retrieval” there are subcategories like “Vector Stores” and “Data Ingestion”.

Individual traces

You can view the traces assigned to each category or subcategory by clicking through to see the traces table. From there, you can click into any trace to see the full conversation details.

Configure a job

You can create an Insights Report three ways. Start with the auto-generated flow to spin up a baseline, then iterate with saved or manual configs as you refine.

Autogenerating a config

1. Open New Insights and make sure the Auto toggle is active.
2. Answer the natural-language questions about your agent’s purpose, what you want to learn, and how traces are structured. Insights will translate your answers into a draft config (job name, summary prompt, attributes, and sampling defaults).
3. Choose a provider, then click Generate config to preview or Run job to launch immediately.

Providing useful context

For best results, write a sentence or two for each prompt that gives the agent the context it needs—what you’re trying to learn, which signals or fields matter most, and anything you already know isn’t useful. The clearer you are about what your agent does and how its traces are structured, the more the Insights Agent can group examples in a way that’s specific, actionable, and aligned with how you reason about your data.

Describing your traces

Explain how your data is organized—are these single runs or multi-turn conversations? Which inputs and outputs contain the key information? This helps the Insights Agent generate summary prompts and attributes that focus on what matters. You can also directly specify variables from the summary prompt section if needed.

Choose a model provider

You can select either OpenAI or Anthropic models to power the agent. You must have the corresponding workspace secret set for whichever provider you choose (OPENAI_API_KEY or ANTHROPIC_API_KEY).

Note that using current Anthropic models costs ~3x as much as using OpenAI models.

Using a prebuilt config

Use the Saved configurations dropdown to load presets for common jobs like Usage Patterns or Error Analysis. Run them directly for a fast start, or adjust filters, prompts, and providers before saving your customized version. To learn more about what you can customize, read the section below.

Building a config from scratch

Building your own config helps when you need more control—for example, predefining categories you want your data to be grouped into or targeting traces that match specific feedback scores and filters.

Select traces

• Sample size: The maximum number of traces to analyze. Currently capped at 1,000
• Time range: Traces are sampled from this time range
• Filters: Additional trace filters. As you adjust filters, you’ll see how many traces match your criteria

Categories

By default, top-level categories are automatically generated bottom-up from the underlying traces. In some instances, you know specific categories you’re interested in upfront and want the job to bucket traces into those predefined categories.

The Categories section of the config lets you do this by enumerating the names and descriptions of the top-level categories you want to be used. Subcategories are still auto-generated by the algorithm within the predefined top-level categories.

Summary prompt

The first step of the job is to create a brief summary of every trace — it is these summaries that are then categorized. Extracting the right information in the summary is essential for getting useful categories. The prompt used to generate these summaries can be edited.

The two things to think about when editing the prompt are:

• Summarization instructions: Any information that isn’t in the trace summary won’t affect the categories that get generated, so make sure to provide clear instructions on what information is important to extract from each trace.
• Trace content: Use mustache formatting to specify which parts of each trace are passed to the summarizer. Large traces with lots of inputs and outputs can be expensive and noisy. Reducing the prompt to only include the most relevant parts of the trace can improve your results.

The Insights Agent analyzes threads - groups of related traces that represent multi-turn conversations. You must specify what parts of the thread to send to the summarizer using at least one of these template variables:

Variable	Best for	Example
run.*	Access data from the most recent root run (i.e. final turn) in a thread	{{run.inputs}} {{run.outputs}} {{run.error}}

You can also access nested fields using dot notation. For example, the prompt "Summarize this: {{run.inputs.foo.bar}}" will include only the “bar” value within the “foo” value of the last run’s inputs.

Attributes

Along with a summary, you can define additional categorical, numerical, and boolean attributes to be extracted from each trace. These attributes will influence the categorization step — traces with similar attribute values will tend to be categorized together. You can also see aggregations of these attributes per category.

As an example, you might want to extract the attribute user_satisfied: boolean from each trace to steer the algorithm towards categories that split up positive and negative user experiences, and to see the average user satisfaction per category.

Filter attributes

You can use the filter_by parameter on boolean attributes to pre-filter traces before generating insights. When enabled, only traces where the attribute evaluates to true are included in the analysis.

This is useful when you want to focus your Insights Report on a specific subset of traces—for example, only analyzing errors, only examining English-language conversations, or only including traces that meet certain quality criteria.

Using filter attributes to generate Insights only on traces with agent errors

How it works:

• Add "filter_by": true to any boolean attribute when creating a config for the Insights Agent
• The LLM evaluates each trace against the attribute description during summarization
• Traces where the attribute is false or missing are excluded before insights are generated

Save your config

You can optionally save configs for future reuse using the ‘save as’ button. This is especially useful if you want to compare Insights Reports over time to identify changes in user and agent behavior.

Select from previously saved configs in the dropdown in the top-left corner of the pane when creating a new Insights Report.

---

Edit the source of this page on GitHub.

Connect these docs programmatically to Claude, VSCode, and more via MCP for real-time answers.