Useful AI Coding Tools Are Mostly Harness, Not Model

Today’s notes came from John Crepezzi’s talk, AI Engineering at Jane Street, surfaced alongside coverage of Jane Street’s record $16.1B first-quarter trading revenue.

The practical lesson: in a non-standard engineering environment, the model is only one part of the product. Jane Street cannot simply drop in generic coding assistants and expect great results because its developer stack is unusually custom: OCaml-heavy code, a large monorepo, Mercurial, internal build and review tools, and multiple editor workflows.

What Made the System Interesting

Jane Street’s approach sounds less like “pick the best LLM” and more like building an internal AI engineering harness:

• Collect workflow-specific data from real developer environments.
• Snapshot workspaces and observe build-state transitions like green → red → green.
• Fine-tune or adapt models on examples that resemble actual code changes.
• Integrate with editors developers already use.
• Evaluate generated diffs by applying them to pre-warmed builds and checking parsing, typechecking, compilation, and tests.

That evaluation loop matters. A coding assistant is only useful if it can survive the local repo, build system, language constraints, and feedback mechanisms that real developers use every day.

Key Takeaway

For serious AI coding tools, harness > model. The durable advantage comes from context capture, eval infrastructure, editor integration, and tight feedback loops around the team’s actual workflow—not from calling a generic model in isolation.