How Pinterest Improved Analytics Agent Compliance with Loop Engineering

Looking down Spiral Staircase representing loops that are essential to Loop engineering for tuning agents.

The Compliance Problem

Compliance means following the right steps at the right time

Pinterest’s Analytics Agent helps teams answer questions such as “How did query latency for the ads reporting pipeline change last week?” It searches the company’s data ecosystem, analyzes the relevant information, and recommends an answer or course of action.

Not every question requires the same steps. We call the agent compliant when it identifies and follows the steps that apply. Improving compliance is intended to make answers more consistent and reduce the risk of incomplete or inaccurate results. Examples include:

Read the data-discovery skill when it needs help finding the right tools or sources.
Inspect table metadata before writing SQL to learn what the columns mean, which filters are required, and whether the table has important usage notes.
Search every relevant source type, including tables, dashboards, and metrics, rather than stopping at the first useful result.

These are three examples from a broader set of compliance rules, and whether they apply depends on the question. If a user provides the appropriate table, the agent may not need the data-discovery skill, while an open-ended question may require a broader search. The agent must decide which rules apply and follow them when needed.

That flexibility also makes compliance difficult. When the agent skips a required step, it may stop at the first table it finds, miss an important filter in the metadata, or overlook a certified metric with a more authoritative definition. It could then confidently report that “latency is down 4.2%.”[1] The answer may sound reasonable while using the wrong definition, pipeline stage, or reporting window.

Why manually revising the agent’s instructions was not enough

Measuring noncompliance showed us where the agent failed, but not how to fix it. After parsing existing chat logs, we found that table searches occurred more than ten times as often as chart and dashboard searches.[2] For a separate rule requiring the agent to read the data-discovery skill, compliance reached only 64% across a broader set of chats even after we manually revised the agent’s instructions.[3]

It was often difficult to predict which change would improve the agent because the model could interpret reasonable instructions in unexpected ways. One guardrail designed to keep the agent within its intended scope seemed reasonable:

Scope: Only answer questions within your domain. For unrelated requests, politely decline and suggest that the user try a general-purpose assistant.

The guardrail produced an unexpected response:

User, paraphrased: “What are the cloud storage costs for this table, and how long is its data retained?”

Agent, paraphrased: “This is outside my domain. Please try a general-purpose assistant.”

Although the topic sounded like specialized infrastructure, it was an in-scope data question because the agent had access to relevant storage data. The agent had misclassified the question and declined without searching. We identified the cause only after trying and evaluating several changes.

Manually revising the agent’s instructions therefore became a process of trial and error, with no reliable way to know whether a change would generalize or break something else. We needed a repeatable process that could propose a change, test it, and use the results to guide the next attempt. That cycle of proposing, testing, and learning became the foundation of what we call loop engineering. We explain the full approach later in this article. Before building it, however, we first needed test suites that defined success and protected behavior that already worked.

Build the Evaluation Framework

Start with paired test suites

We built a separate pair of test suites for each compliance rule we wanted to improve, such as reading the data-discovery skill or searching all required source types. A focused LLM labeler answered two questions for each chat, and humans reviewed the labels:

Does this compliance rule apply to this chat?
If it applies, did the agent follow this compliance rule?

We used the first answer to assign each chat to a suite. If the rule applied, the chat went into the required suite. If it did not, the chat went into the skip suite. We then compared the agent’s behavior with that expectation. A chat passed if the agent followed the rule when required or skipped it when it was not needed. These pass and fail labels gave us a baseline for comparing future versions.

Figure 1: One labeled set of chats produces two suites that pull in opposite directions. The skip suite stops a fix from turning into a new bug.

Both suites are necessary. If we tested only required cases, a system prompt telling the agent to “always search dashboards” could earn a perfect score even when a dashboard was irrelevant. We saw this tradeoff in testing. A change that made the agent search dashboards more aggressively fixed one chat where a dashboard was required, but it also triggered dashboard searches in two chats where they were not. The required suite showed the improvement; only the skip suite revealed the two new regressions.[2]

Together, the suites gave the loop a balanced goal: follow the rule when a question requires it, and avoid unnecessary actions when it does not.

Protect unrelated behavior with regression tests

These two suites covered only the rule we were tuning. We also needed to protect everything else the agent already did well. For that, we used a much larger regression suite and rejected any change that pushed its performance below the required level.

Once we could measure progress, the next challenge was helping each iteration learn from the ones before it.

Design the Tuning Loop

Why persistent sessions and Ralph Wiggum loops both fell short

Our first design kept one agent session open across every iteration. The full history helped the agent make better decisions early, but the context grew with every attempt. Later iterations became more expensive and less reliable.

The second design used a memoryless pattern often called a Ralph Wiggum loop. Every iteration started with a fresh context window, reducing the context sent to the model and lowering the estimated model cost by about fivefold.[4] The tradeoff was that it often repeated earlier mistakes because it had no memory of what had already failed.

One design remembered too much; the other remembered nothing. The final design needed fresh context without losing the lessons from earlier iterations.

Fresh context with structured memory

Every iteration now starts a fresh session and reads a compact, structured handoff. The handoff records:

Current results: Scores for every test suite, including validation cases not used for tuning.
What is failing: The cases that still fail and their likely causes, helping the agent address a broader pattern instead of one example at a time.
The best change so far: The highest-performing change from earlier iterations, which becomes the starting point for the next one.
What not to repeat: Previous approaches that reduced performance or failed to solve the problem.
Possible test problems: Cases that no candidate has passed, which may indicate an issue with the test rather than the agent.
What must remain stable: Test suites that are not allowed to regress, along with the file and line limits for the next change.

The handoff preserves decisions rather than conversation history. Each new session receives the evidence it needs, while the context stays small enough to remain useful.

How the tuning loop works

For each rule, we split the required and skip cases into tuning and validation sets. The tuning cases guide improvements to the agent, while the validation cases test those improvements on examples that did not guide them. A final holdout set stays untouched until the end. Alongside these rule-specific cases, the larger regression suite checks that the agent’s other capabilities still work.

Get Pinterest Engineering’s stories in your inbox

Join Medium for free to get updates from this writer.

Each iteration follows the same process:

Draft up to four changes to the Analytics Agent’s system prompt, skills, context layer, or tools.
Evaluate every candidate on the same small sample and select the highest-scoring one. If the scores are effectively tied, prefer the change that adds less prompt context and modifies fewer lines.
Evaluate the winner on the full tuning and validation suites.
Apply the acceptance checks, then keep or revert the change.

The loop repeats until it reaches its targets, with a preset budget serving as a safety limit. When the run ends, it evaluates the final result on the holdout set.

Figure 2: One iteration. Edits are raced on a small sample, and only the winner is scored on the full suites and sent to the checks in Figure 3. The holdout opens once, at the end.

Reject changes that improve the wrong thing

A higher tuning score is not enough to keep a change. It must also pass three independent safety checks:

Repair/break gate: Did the change fix more cases than it broke?
Acceptance critic: Does the change address a general pattern, or does it hard-code an answer from the test set?
Regression guard: Did the change reduce performance on a test suite that was not allowed to regress?

Figure 3: Three independent checks can reject an edit, and only the tuning suites can accept one. Any single check sends the edit back.

The checks make acceptance deliberately conservative. A higher score is not useful if the change overfits the test cases or damages another capability.

We also limit what the loop can change. Each proposal may modify only the agent’s system prompt, skills, context layer, and tools. It is capped at a few files and a few dozen lines, with the limit shrinking over the run. Changes outside those boundaries are reverted. This prevents the loop from modifying application code to make its own tests easier to pass.

The shrinking limit also favors small, targeted changes over unnecessary rewrites. Without that constraint, an agent may rewrite a prompt that already works. Amazon’s Promptimus reports the same tendency in its edit mode.

Measure Real Improvement

The run that fooled us

With the test architecture and safety checks in place, the design appeared ready. Our first large run reached 89.3% compliance across its tuning and validation cases.[4] The iteration history told a less impressive story. Most of the gain came from the first accepted change, and iterations 21 through 30 added nothing.

Figure 4: Thirty iterations, four accepted edits. The first did most of the work, and the last ten produced nothing.

The plateau raised a more fundamental question: when a score moved by one or two cases, was the change real or simply normal model variation?

Figure 5: The same code runs twice over the same cases. The strands that cross are the 23 that changed verdict.

Despite similar totals, 10.3% of cases changed verdict between runs.[2] Because gains and losses nearly canceled each other out, the aggregate score hid substantial case-level variation.

Learn from individual cases, not only the score

Evaluating every candidate on all 223 cases would take too long, so we first run each one on the same sample of about a dozen cases. With a 10% flip rate, roughly one result in a sample that size may change by chance. That is why we do not rely on a small score increase alone.

The overall score tells us whether performance moved, but individual cases tell us what changed. The loop records which cases changed from fail to pass and which changed from pass to fail. The next iteration uses those results to identify what improved, what broke, and what to address next. Validation cases and the broader regression suite provide additional evidence before the loop accepts a change.

Results: small changes backed by strong evidence

The loop improved compliance in chats where each rule was needed while preserving performance in the skip and regression suites.[3] In other words, a change had to fix existing failures without breaking cases the agent already handled correctly.

Figure 6: Before/after compliance on each rule’s two suites, with the expected effect on answer quality.

The improvement in searching every relevant source type came from a backend change to the Analytics Agent’s search process. Reaching it required two labeled datasets, a dozen loop iterations, and two verification runs. Most of the work went into identifying a change that improved compliance without introducing regressions.

From Prompt Engineering to Loop Engineering

The Analytics Agent now interprets experiment results, respects restricted-data access, builds dashboards, writes reports, and continues to take on new responsibilities. As these capabilities expand, a change intended to improve one workflow can affect how the agent handles others, sometimes introducing new failures.

That is why we moved from one-off instruction changes to loop engineering. People define the goals, evaluation criteria, and safety boundaries. The loop proposes changes, tests them against real failures, and rejects those that introduce regressions. The same process can target other goals, including consistency, latency, and accuracy, as the agent evolves.

Loop engineering turns failures from real user chats into tests that guide changes to the agent. By keeping only changes that improve compliance without damaging existing behavior, it gives us a repeatable way to make answers more reliable as the Analytics Agent evolves.

Acknowledgements

None of this would have been possible without the people who shaped and supported this work. We thank Attila Dobi for design reviews that strengthened the loop; Jason Coffman for the Analytics Agent capabilities and infrastructure; Bryant Xiao for sponsorship and engineering guidance; Yi Wu and Laura Palmer for product vision and direction; and J.C. Zhong, Yufei Dong for the real-world evaluation data and infrastructure.

Footnotes

[1] Illustrative example; not actual Pinterest data.

[2] Pinterest internal evaluation data, global, July to September 2026. Search figures count tool calls, not successful retrieval. One trial per case, and the LLM labeler is not calibrated against human labels. All runs used one base model at one reasoning setting (Claude 4.7 Opus Fast), so results may not hold across models.

[3] Same source as [2]. The 64% figure is the compliance rate across a broader population after the agent’s instructions were manually revised; it is not a before-and-after comparison. The skill-read suite was a separate 41-case dataset built entirely from known failures, so its baseline was 0% by design. The expected-benefit column describes what we expect each rule to improve; we have not measured answer quality directly.

[4] Same source as [2]. The 89.3% is a pooled pass rate over the run’s tuning and validation cases, with most of the gain landing at the first change and nothing kept after iteration 20. The cost comparison is an internal estimate of model inference cost, not a controlled benchmark.