John Stecher and Scott Worland on Building AI Systems That Can Change

Scott Worland, CTO of Norm AI, introduces John Stecher, CTO of Blackstone. Norm works in legal and regulatory AI and supports an affiliated law firm. Stecher oversees technology at Blackstone, advises on investments and serves as a technology resource for portfolio companies.

Worland asks how Blackstone is approaching the rush to deploy agents. Stecher says longer context windows and the ability to run tasks longer have opened opportunities, but the foundation is a cohesive data strategy that makes information accessible and manageable for agents.

Blackstone combines proprietary and public data. Stecher says organizations that leave essential information scattered in transactional systems will struggle to extract value from agents. Moving it into forms that can be accessed and recalled is a major part of the work.

Straight-through processing still matters, Stecher adds. He uses agents to help understand workflows and write code that automates deterministic steps, adding AI where intelligence or a human judgment was previously needed. The aim is to free investment, finance and engineering professionals for higher-value work.

He names software testing, performance tests and library upgrades as examples of work agents can assist with. For him, having an agent help renovate source code has improved delivery speed. Worland then explains how Norm’s own architecture changed as models improved.

Norm began deploying models in production when capabilities and context windows were changing quickly. The team built substantial structure around content selection, sequencing, consistency checks and output generation—a workflow resembling map-reduce. Worland says this produced accurate, consistent results important in regulatory work.

To regression-test that system, Norm assembled tens of thousands of expert-labeled evaluation examples. When more autonomous tool-using agents became practical, the team ran them against the same benchmark and found that much of its carefully written workflow code was getting in the way.

Worland says the engineering leaders left their normal setting and asked what platform they would build if starting from scratch. They produced specifications for delivering agentic experiences to clients and allowing AI to work alongside the development team. The company has been executing against that plan.

Stecher emphasizes the high quality bar in financial services. Model revisions, harness changes and new data can all change answers, so evaluation sets must be run consistently to measure accuracy, consistency and system health. The duty to be accurate remains central.

Worland connects the discussion to the “bitter lesson”: systems that scale with compute can eventually outperform elaborate hand-crafted structures. He asks how Blackstone considers that possibility when choosing projects and planning a roadmap.

Stecher weighs expected return against today’s cost of implementation. Sometimes Blackstone has chosen to wait for a more capable model when the business need is not urgent and a later solution may be cheaper. He describes waiting as a tradeoff made case by case, not a rule to delay every project.

Waiting too long has its own cost. Stecher encourages engineers to use development tools to turn a concept into an interactive proof of concept quickly, show it to stakeholders and discard it if it does not deliver value. A proof of concept is not yet production-ready or security-reviewed.

Worland says investment is a gradient: some capabilities deserve heavy investment, while other work should be built conservatively because better models or third-party tools may replace it. Norm still needs harnesses, sandboxes and controls around network egress, but wants a way to incorporate outside advances.

He says Claude Code changed his view of generalizable systems: an agent can write and execute code and verify results in a computer environment. He presents this as his assessment of the direction of the technology, not a guarantee that agents can perform every task reliably.

Norm invests in data capture, making information available as state and building evaluations with its affiliated lawyers. Instead of a roadmap consisting only of features, Worland wants scorecards describing successful performance on tasks so that general improvements can be measured without hard-coding every workflow.

Stecher says getting a proof of concept is easier than getting it into production. Enterprise infrastructure, cybersecurity, supportability, traceability and integration with site-reliability monitoring remain hard work. Those requirements must be addressed before an application can run reliably at scale.

Worland says Norm requires full human oversight for core primitives: design, code ownership and approval of changes. Applications built on those primitives cannot modify them without human review. In a system handling sensitive data, he says, security matters even more than speed.

The team tries to keep pathways open for improvements from frontier models without allowing sensitive core systems to change unchecked. Worland expects some investment decisions to be wrong and says teams need to remain willing to replace components as capabilities change.

Stecher says many internal applications can now be built for a particular purpose and later thrown away. He encourages engineers to ask whether rebuilding costs less and delivers more value than maintaining and incrementally upgrading an existing application—a question that previously arose far less often.

In closing, Stecher says AI tools have renewed his enthusiasm for engineering. Experienced developers still understand architecture and system design, and agents can help them express that knowledge without spending as much time building initial scaffolding. Worland thanks him and ends the session.