October 21, 2026

How we built the Langfuse Assistant

How we built the Langfuse Assistant on our existing MCP server, why the agent runs outside its sandbox, and how we handle background runs.

Lotte

Ben

Back in June we launched Halo, the Langfuse Assistant. You can ask it to interpret your traces, build a dashboard or dataset, or help set up evals without leaving the Langfuse UI.

It's somewhat ironic: as a company focused on helping others build agents, we didn't have an in-product agent of our own. We've invested in building the infrastructure and tools for agent integration, such as the Langfuse skill, MCP server, and CLI, for using Langfuse with agents like Claude Code or Cursor. Now we wanted to bring those capabilities into Langfuse itself.

When we first discussed whether we should write this post, I wasn't sure there was much to write about. The first version was simply a Mastra agent, an MCP server, and some code to glue them together. We didn't invent some super-efficient algorithm for agent execution, and we didn't rewrite anything in Rust (although there might be some other posts here coming soon). Building the in-product agent was mostly about integrating existing pieces and making them work smoothly together, as well as creating the foundations for future agentic features.

From the start, we knew we wanted to reuse as much as possible and share the tools and instructions behind Halo with people using Langfuse through Claude Code or Codex. We didn't want a second set of things to maintain just for the in-product agent. If we improved a tool or fixed some guidance for Halo, that should help people using their own agents too. That meant building on our existing MCP server as well as bringing the shared Langfuse skill into the Assistant.

Langfuse back-end

• API
• MCP
• CLI
• Docs
• Skill

Langfuse UI

Halo

Coding agents

How the Assistant is put together

The first step was expanding the MCP server. The May 29 release added tools for observations, metrics, scores, comments, datasets, and annotation queues, making most Langfuse project data available to agents. We needed these tools for Halo, but they were also useful to anyone connecting their own agent.

In June, we launched the first beta version of the Assistant: a Mastra agent that ran behind an HTTP endpoint, streaming responses to the browser. It handled one conversation at a time, with no parallel work or background execution. In early July, we gave it write access with human-in-the-loop approvals, and later that month added a sandbox. We added background execution in August. We announced all these capabilities together in September.

Here's the high-level architecture that we ended up with:

Users can ask Halo to investigate something, close the tab, and come back to its progress and results. The worker keeps going independently of the browser, while the saved event log lets the UI catch up when you return. MCP tools let Halo work with user's project data. The optional sandbox lets Halo process larger results with scripts instead of reading everything into the model's context. Neither gives it more access than you have, and actions that change data still need approval.

Reusing the MCP server and skill

Halo uses our public MCP server, the same one you can connect to. The only tools it gets besides MCP are for interacting with the Langfuse UI, such as proposing a redirect to another page, or working in the sandbox. Almost all customer-data access goes through MCP, though we also give it some context about the user and the page they're on: their name, timezone, browser languages, and the current URL. By relying on MCP and tools, we can control which tools the Assistant gets to call without having to secure arbitrary network requests.

For each run, the worker creates a temporary API key scoped to the project. By default, it can only call read-only MCP tools. We maintain our own mapping of tool permissions rather than trusting the MCP server's read-only annotations. We also filter tools by the signed-in user's role before exposing them to the model. If the user's role doesn't allow an action, the agent doesn't get the tool for it.

Tools that change data require approval. The worker adds the approved tool to an allowlist in the MCP request headers, then removes that permission before continuing the agent loop. Choosing Always approve saves the approval for that conversation. We still recheck it against the user's current role on each run, so an approval can't give the agent permissions the user doesn't have.

If Halo needs another way to access project data, we have to add an MCP tool first. I actually like that constraint, though, because the same tool is then available to people using their own agents.

We do the same with instructions. Halo's system prompt explains how to handle approvals, link to Langfuse pages, and keep answers short. For guidance on using Langfuse, such as setting up evals or migrating prompts, we load the open source Langfuse skill as a Mastra skill. It's the same one you can install in Claude Code or Cursor.

TODO: I think the text below does not belong in this section. Separate section on observability?

When an organization opts into AI feature telemetry, we can use traces of Assistant runs to see where it struggles. We also see people trying to break it. These attempts started as soon as we announced Halo and still turn up regularly. One example is asking it to download an executable from a link. With no web-fetch tool and no sandbox network access configured, it can't make that request.

When we see Halo struggle with a tool or give poor guidance, we want to fix the MCP tool or the shared skill. Maintaining a separate set of fixes just for Halo would defeat much of the point of sharing them.

Some of those fixes are fairly small. Agents kept asking to sort metrics by totalCost_sum when the tool expected sum_totalCost, so we made the MCP adapter accept both. When dashboard creation failed validation, the error told the agent which field was wrong but not what it could use instead. We added the allowed values and guidance on looking up the schema, so it had enough information to correct the call. Both changes help anyone using the MCP server, not just Halo.

The agent does not run in a sandbox

We added the sandbox because we knew some tasks could be handled more efficiently by writing scripts for processing data than by putting raw tool outputs into the model's context. But giving the agent bash and file system tools didn't mean the agent itself had to move into a sandbox.

Many conversations don't need a sandbox at all. Answering a docs question or looking up yesterday's traces might only take a few MCP calls. Starting a VM for those conversations would cost us memory and disk, but not actually provide value in these cases.

There's also self-hosting. Langfuse is open source, and I don't think setting up an external sandbox provider should be a requirement for using the Assistant. Without one, it still works with MCP tools.

The downside is that each sandbox operation has to cross that process boundary, adding latency. For interacting with the sandbox, we give the agent tools: read, write, edit, and bash, backed by an AWS Lambda MicroVM. These are created when the agent first uses a sandbox tool. It runs as an unprivileged user, with no network egress configured by default. We configure it to suspend after 60 seconds idle and expire after four hours.

TODO: the things below should either be removed or put under a separate section

If the session has expired, the next sandbox operation creates a new one. Files the agent wrote are gone, and we tell it that. Saved MCP results survive because we store them outside the sandbox.

Without network access, scripts can't use the Langfuse SDK or CLI. That's a fairly large limitation, but they can still process data the agent fetched through MCP. Keeping data access in MCP meant we could add the sandbox without also having to secure arbitrary network calls from scripts.

Processing tool results with code

A lot of the work Halo does consists of reading large amounts of traces, and some Langfuse projects have large amounts of data stored in their traces, such as large JSON structures. We had to provide a way to access this data without blowing up Halo's context window.

Halo can scan through large amounts of data using the sandbox and tools like grep, head, jq, or a short Python script. Say you ask it to group the errors in a few hundred failed observations. It doesn't need to read every field of every observation to do that. It can inspect the files, group the errors, and read only the output it needs.

When a sandbox is available, we add an optional silent parameter to every MCP tool so the agent can fetch data without putting the full result into the model's context:

// Simplified: when a sandbox is available, every MCP tool gets an optional `silent` flag.
tool.execute = async ({ silent, ...input }, context) => {
  const result = await execute(input, context);
  persistToolCall({ toolName, toolCallId, result }); // stored with the run

  if (!silent || isError(result)) return result; // failures are never silenced
  return `Output available at /workspace/tool_calls/${toolName}_${toolCallId}.json`;
};

With silent: true, the model gets a file path instead of the full result. The worker saves the result outside the VM. Before the next sandbox operation, we write the saved calls into the sandbox as JSON files containing the request and response.

Lotte: I would replace this paragraph with some screenshots showing this instead. Can we do that safely?

Looking through runs for this post, I saw Halo use short Python scripts to count observations, combine results from several files, and count errors by environment or model. Sometimes it just printed part of a file to see what was there. The agent decides what to inspect rather than having to read the whole response. Failed tool calls still return their errors to the model, even with silent: true.

Lotte: I would remove this paragraph, it does not sound super important.

Our first implementation of silent also hid some tool errors from the model. Some tool failures came back as structured responses rather than exceptions, so our wrapper saved them to a file just like successful results. The model got a file path instead of the validation error it needed to correct its call. We fixed the failure detection so those errors go straight back to the model instead.

Background runs

We didn't want Halo's runs to be interrupted when someone closed the browser.

We were already using BullMQ, so we added functionality to deliver run IDs to workers. Postgres stores Halo's run state. A worker claims a queued run and sends a heartbeat every five seconds. When someone reads or watches the conversation, we check that heartbeat. If it's more than a minute old, we consider the worker lost and mark the run as failed.

We also had to take into account waiting times for action approvals, which could be minutes to hours. To avoid workers being idle for too long, a run always stops when an approval step is presented to the user, and when the user approves or denies a proposed step, it creates a new run that continues from the saved conversation in Postgres.

We decided not to automatically retry failed runs. This was a trade-off between keeping it simple and user experience. Since Halo's runs are relatively short, this was an acceptable decision to make. If we automatically retried runs with our current architecture, there's the risk that a worker makes changes to the user's project, then fails before the run could be saved to Postgres. That's something we wanted to avoid.

If your runs take hours, cost a lot, or have side effects you can't check afterwards, I'd probably make a different choice and use a durable workflow engine.

What's next

We're exploring giving sandbox scripts access to the Langfuse SDK and the project's configured LLM connections. Evaluator calibration is one possible use case: testing an evaluator against human-labeled examples, changing its prompt, and checking whether the results improve.

We'd need to restrict scripts to the requests the user is allowed to make and has approved. We're still exploring how to do that, so neither SDK access nor LLM connections are available to sandbox scripts today.

The Assistant is live on Langfuse Cloud and in public beta for self-hosted deployments. If one of these decisions gets in your way, tell us in the feedback discussion.

If you like working on this kind of problem, we are hiring.