LinkedIn’s self-improving agent first had to fix its judges

Ziwei Guo and Saloni Gandhi describe an improvement loop for LinkedIn’s user-understanding agent. The goal is to learn from production traces, identify failure patterns and propose changes that improve how the agent represents a member’s career interests.

The user-understanding agent supports a broader career agent intended to help LinkedIn members pursue career goals. Developers collect problems from daily work and production traffic, evaluate the traces and use the resulting feedback to guide the next iteration.

Guo stresses that the evaluator must be objective and aligned with user feedback. Otherwise the loop may optimize the wrong thing: the quality signal is what identifies a bottleneck and determines which change the system tries next.

The surrounding agent system has several possible targets: planning logic, context and memory, interfaces to tools and data, and storage for outputs and logs. Evaluation sits across these components; the underlying model powers the system but is not currently being optimized in this project.

Guo describes a ladder of increasing complexity. The current work concentrates on prompts, the most direct place to make a change. Later possibilities include structured context, workflow design, harness code and eventually the optimizer itself; these are future directions rather than current results.

The loop has seven steps: run the agent and collect samples and traces; score them; diagnose causes; propose edits; validate candidates; evolve the prompt; and evaluate the new baseline on held-out data. The final test is intended to check that the change generalizes beyond the examples used to make it.

The reflection step asks why an output scored poorly. It may be an agent, model, prompt or tool problem, but the evaluator or its scoring rubric may also be at fault. Proposed edits are tested using the same evaluation engine and criteria before a candidate proceeds.

A final held-out evaluation compares the updated system with the previous baseline. If the change is adopted, it becomes the starting point for another iteration; the team plans to run the cycle on a regular cadence.

The target agent starts with a member identifier and retrieves information such as search history, profile, location and stated preferences. It synthesizes those signals into a structured understanding of the member.

One output field is preferred job titles. Alongside values such as network manager or account manager, the agent supplies a confidence level, citations to supporting information and a short explanation. Repeated saved searches, for example, can provide evidence for a title preference.

An evaluation judge scores the agent across task dimensions using product-defined rubrics. The example shows an overall score as well as per-dimension scores and rationales; the scoring scale runs from one for poor quality to three for very good quality.

Reflection begins by grouping failed examples by dimension without an additional model call. A second-level judge then diagnoses each example and tags its suspected root cause, which can include the agent, code, context, judge or rubric.

Because similar problems can be described in different words, the system merges near-duplicate diagnoses into canonical issues. Missing profile data and an incomplete profile picture, for example, can point to the same underlying lack of supporting information.

It then groups and ranks issues by frequency without another model call, generating a scorecard for humans and retaining the diagnoses for future iterations. Prior diagnoses can inform the second-level judge on later runs.

The proposal stage uses the judge’s evidence and reflection artifacts to suggest edits. The team uses GEPA to propose prompt changes in mini-batches, selects candidates, validates them and, when a candidate is accepted, updates the production baseline for subsequent cycles.

The team describes three areas of work: quality, automation and production updates. Human annotations improve both levels of judging, while analysis of judge variance addresses the instability that comes from using language models as evaluators.

The workflow can be scheduled weekly, every two weeks or monthly according to need. For now, humans remain involved in production updates to protect quality. The team hopes to reduce that effort gradually as its confidence in the process improves.

Gandhi says the team deployed its first improvement produced through the loop and GEPA after human review and quantitative measurement. It also invested in comparing the evaluator’s judgments with human annotations before trusting it to steer further changes.

The slide reports 180 completed human reviews. For outputs the judge rated one or two, full agreement with reviewers was 55.7% and 57.5%; for a rating of three it reached 80.4%. The judge was better at recognizing strong outputs than distinguishing weaker ones.

Agreement also varied across dimensions. The slide reports preferred job titles as the weakest observed category at 33.3%, primary domain expertise at 46.7% and target employment type at 50%. These differences pointed to dimensions needing clearer definitions and evidence rules.

One recurring problem was plausibility being mistaken for completeness: a returned job title could sound reasonable while omitting an important signal of the member’s intent. Other disagreements arose because domain, function and skill were not clearly separated or partial credit was underspecified.

The judge sometimes penalized harmless formatting differences and made inconsistent assumptions about what employment information allowed it to infer. Of 37 judgments identified as too harsh, 31 concerned outputs the judge had assigned the lowest score.

The disagreements often reflected ambiguous evaluation policy, not simply a model malfunction. The team clarified rules for capitalization, title normalization, concurrent employment, seniority, geographic preferences and the boundary between poor and partially correct outputs.

After the rubric changes, reported human alignment rose from 57% to 64%, while mean error fell from 0.96 to 0.74. Gandhi attributes the gain to making the specification of a correct evaluation clearer, not merely adjusting the model.

The team conducted a similar review of the second-level reflection agent. Identifying the cause of a failure proved harder than noticing the failure itself: human reviewers agreed more often when the agent blamed the model than when it attributed a problem to other causes, particularly the rubric.

The original diagnosis categories had no explicit way to say the first-level judge was wrong. The reflection agent could therefore blame the rubric for a mistaken judgment. It also sometimes reused the first judge’s rationale without independently checking it against the agent’s output.

A convincing explanation is not the same as a correct diagnosis. If an optimizer uses a wrong root cause to produce a prompt edit, it may change the wrong part of the system. Improving diagnosis therefore became a priority alongside improving the first-level judge.

Gandhi says the team aims to stabilize first-level scores and make root-cause diagnosis more reliable. Majority voting and held-out evaluations help mitigate variance from individual language-model judgments, with adoption thresholds and confidence intervals informing decisions.

The broader measures are not just how many prompt edits the system produces. The team wants to track which changes can be generated and validated reliably, whether downstream response quality improves, and whether engineers save time maintaining agents.

To scale this approach across teams, Gandhi describes a stack: useful agents at the base, repeatable evaluation above them, systematic reflection above that, and the improvement loop at the top. Each layer depends on the reliability of the layers below it.

The proposed shared infrastructure would provide reusable evaluation frameworks and human-calibrated judges, collect traces and diagnose failures, automate candidate generation, and validate changes with held-out data and regression tests. Self-service components would let other teams adopt the loop without rebuilding it all.

Gandhi distinguishes the target agent from the improvement engine. The engine evaluates performance, diagnoses failures and proposes changes; its ambition is not only to raise performance once, but to increase the rate at which the agent improves over time.

The team closes by emphasizing trust in the changes themselves. The project brought together evaluation, engineering and other collaborators because a useful improvement loop depends on the quality of both the proposed fix and the feedback behind it.