Most adversarial machine learning research assumes white-box access — the attacker can see the model's architecture, weights, and gradients, and can use them to directly compute a perturbation that maximizes error (e.g. via gradient ascent). That assumption almost never holds for a real surrogate model deployed inside plant instrumentation: an attacker, or an auditor probing for weaknesses, typically only gets to submit an input and observe an output.
This goal asks whether an LLM agent can still succeed under that much harder constraint: given only black-box query access to a surrogate model — an input vector in, a prediction out, nothing else — can it construct a sequence of queries that reliably surfaces a high-error region, without ever seeing a gradient?
Without gradients, the agent cannot follow the direction of steepest error ascent analytically. It has to build an implicit model of where the surrogate is weak using indirect evidence: physical intuition about where a surrogate model is likely to have been under-trained (distribution edges, discontinuities, rare parameter combinations), sequential experimentation (propose an input, observe the output, refine), and reasoning about the ground-truth physics well enough to recognize when a surrogate's prediction has quietly diverged from it.
Our ReAct-style agent (see System Overview) operates entirely through the same input/output tool interface a black-box attacker would have: it forms a hypothesis about a weak region, submits a probe, observes the resulting prediction error against the ground-truth physics solver, and updates its strategy — iterating reason → act → observe across each simulator.