Pamela Fox: Test Model Swaps Against Your Own Application

Pamela Fox leads a guided workshop comparing model behavior across concrete application scenarios. The examples use models deployed on Microsoft Foundry and are demonstrations from the event, not a permanent ranking.

The goal is to make model substitution practical: services and access can change, so teams should know how another model behaves in their application and what adjustments it needs.

The workshop compares GPT-5.5, Kimi K2.6, DeepSeek V4 Flash and Mistral Large 3 in the configurations available to the presenter. Exact versions, prompts and settings matter.

Benchmarks, cost and throughput offer useful clues, but a benchmark alone cannot tell you whether a model meets your own task requirements. Test the scenario you actually intend to deploy.

The first example counts a letter in a sentence. Fox runs the same request across models and observes different answers; a single run is illustrative rather than a statistically robust comparison.

She changes the prompt to count word by word and sum the results. In that demonstration, the revised instruction helps one model answer correctly, showing why prompts may need retesting after a swap.

The next example tests grounded retrieval. A question asks about two types of bees, but the retrieved documents describe only one; a grounded answer should identify the missing information rather than invent a source.

Fox shows one response explicitly saying the source lacks the requested details. She then compares other models using the same question and context.

One model supplies information outside the provided source. Fox strengthens the instruction to identify unsupported claims, but the revised answer still includes extra information in the demonstration.

Her practical question for RAG is whether an answer stays within the available evidence and admits gaps. Prompt changes may help, but they do not replace evaluation.

The vision examples include an animal image, crocodiles and a table with small text. She asks models to identify the image or extract a price, comparing outputs against the intended answers.

The selected models differ on these examples; one deployment is text-only and is not included in the image test. Results depend on model capability, the specific image and the setup.

Fox argues that prompting cannot always repair a model’s visual limitations, so teams should test the actual kinds of images they expect to receive. This is her workshop interpretation, not a universal rule.

She changes the language of an image question and sees a different answer. Multilingual applications need examples in each supported language, including ambiguous terms and local usage.

For tool calling, the application supplies function names, descriptions and JSON Schema. The model proposes a tool and arguments; application code, not the model alone, executes the function.

Function and argument descriptions are part of the model-facing interface. Ambiguous names or missing constraints can produce an unexpected attendee, location or date in the calendar example.

Fox compares proposed tool calls and expected arguments. When a result differs, she suggests clarifying the schema and instructions, then rerunning a range of cases.

A calculator loop can solve the same word problem using one combined expression or several calls. Both may reach the right answer, but call count affects latency and cost.

She observes that a model may also make extra calls to check itself. Evaluation should inspect both the final answer and the path taken, while recognizing that some checks can improve reliability.

The workshop then builds a travel-planning agent with tools for flights, hotels, budgets and activities. The example logs tool calls so behavior is visible.

Independent flight and hotel searches can run in parallel. Budget checking depends on their results, and activity selection can then use the remaining budget.

Fox compares both models and agent frameworks, since the tool-calling loop and model choice can each change the trace. In her demonstration, some models use more turns or omit a step.

Manual inspection helps discover failure modes, but production comparisons need repeatable bulk evaluations. A simple first test can check whether the right function and arguments were returned.

Build cases for date arithmetic, ambiguous durations, different languages and colloquial requests. Add new examples when real users reveal gaps, with appropriate handling of their data.

Model evaluations can be slow and expensive, so decide how often to run them rather than assuming every test belongs on every commit.

For outputs without a single exact answer, an LLM judge can apply a rubric and return pass or fail with a reason. It may need the question, answer, source context and tool traces.

A judge is itself fallible. Fox recommends calibrating it with human or domain-expert labels and distinguishing errors in the application from errors in the evaluator.

She demonstrates ASSERT, a Microsoft Research tool that generates evaluation scenarios and rubrics from a specification. The generated cases are a starting point that still needs review.

The travel specification can include dimensions such as ample, tight or infeasible budgets. Stratifying cases helps test behavior across different conditions.

The displayed comparison reports violations for specific model runs and scenarios. Fox treats it as a way to inspect the application, not proof of a model’s performance on all workloads.

Finally, she shows DSPy optimizing a prompt against declared constraints and checks. The optimizer searches candidate instructions; the resulting prompt still needs independent evaluation.

Her example asks for a structured list satisfying multiple constraints. She supplies a student model, a reflection model and an optimizer, then tests candidate prompts against the specification.

She shows a prompt that worked for the chosen model in this task. A lower-cost model may be sufficient for a specific workload after optimization, but this is not guaranteed for every workload.

Different providers can also expose different APIs and structured citation features. Model substitution may therefore require adapter changes as well as prompt and evaluation changes.

The workshop code remains available, but the temporary demonstration credentials are not a persistent resource. Use your own authorized configuration and test cases when reproducing it.