AWS Automated Reasoning Checks: A Sponsored Policy Verification Demo

Automated reasoning uses formal logic to analyze statements, and implementing complicated policies can require extensive specialist work. In this AWS-partnered 2024 video, I examine a preview feature intended to help make that work more accessible.

Language models can assist with code and natural language but can also hallucinate. I ask whether formal checks can help assess their responses against explicit rules, particularly in situations where mistakes matter.

As a simple illustration, if rain implies wet ground and wet ground implies less tire grip, the implications can be chained. Real policy systems contain many more conditions, exceptions and interactions.

An airline refund policy may have many pages of eligibility rules. Access-control policies also have many services and permissions; errors in modeling or implementation can have serious consequences.

I recount Amazon’s description of building reasoning around IAM permissions: it involved logicians, engineers, years of work and significant expense. This is an account of that particular effort, not a measurement of all formal verification projects.

The challenge is to use a probabilistic model to help construct rules while relying on formal logic for a check. A mathematical proof establishes a conclusion relative to its formal rules and assumptions; it does not by itself prove that a translated policy perfectly matches the original document.

AWS announced Automated Reasoning checks in Amazon Bedrock Guardrails as a preview at re:Invent. I describe how a natural-language policy can be used to propose variables and logical rules for later testing.

My example is a paid leave policy with eligibility conditions such as employment status, level and tenure. Although the demonstration document is short, real policies can contain interacting constraints and exceptions.

I upload the policy document and describe the intended use: answering employee questions about eligibility. The demonstration shows a policy-creation interface that extracts candidate rules and variables from the source.

The resulting policy lists rules and typed variables. Reviewing those extracted rules is essential: if a condition is omitted or translated incorrectly, a logically valid conclusion about the formal policy may still misrepresent the source.

I test a proposed answer saying a temporary contractor is eligible. The tool flags the answer as invalid and shows relevant conditions and variables, making the reasoning available for inspection.

The explanation also assumes an employee level that the question did not state. I edit the variable description to request an explicit level rather than making that assumption, then run another test.

A part-time contractor example is flagged against the full-time requirement, and I also show a positive test. These examples illustrate how policy tests and human review can reveal assumptions or deficiencies in a formulation.

I am optimistic that natural-language assistance can reduce the effort required to build formal rules. The video’s broader claims of near-perfect truth should be read cautiously: accuracy depends on the policy, translation, inputs and the scope of the check.

I contrast the approach with contextual grounding or retrieval, which can supply source material but does not itself prove a conclusion from formal rules. The methods serve different purposes and each can fail if its inputs or interpretation are wrong.

I close by pointing viewers to the AWS announcement and thanking AWS for the partnership. The feature shown was a preview at the time; the demonstration should not be taken as a guarantee that all hallucinations are eliminated.