Why the AI never touches the ledger

One of our early testers texted PocketMoney that they had paid for dinner and wanted to split it. PocketMoney replied with a cheerful confirmation and a check mark. The group balance did not move. Nothing had been saved.

This was the most common bug in our first round of testing, and four separate testers ran into it. This post is about why it happened and the rule we made because of it.

A chatbot is a bad bookkeeper

A large language model (a text-prediction system, ChatGPT being the best-known example) is very good at conversation. It is not reliable at keeping records. It writes the reply that sounds most likely, and after a few turns of seeing "Logged ✅" in the chat, the most likely reply to a new expense is another "Logged ✅". Whether anything was actually written down is a separate question, and the model does not really keep track of it.

Our first fix was to give the model a special instruction for writing an expense and forbid it from claiming success without one. That helped. Then in the second round of testing the bug came back, on and off. Sometimes the model said "hang tight" and never came back. Sometimes it copied an earlier confirmation and wrote nothing.

The cause turned out to be us. We were feeding our own confirmations back into the conversation history, and the model learned to imitate them.

One rule for the whole product

So we made a rule. The AI drives the conversation. It never drives the ledger.

In practice, the common money messages, like "I paid $60 for groceries, split with Sam" or "paid Maya back $45 on Venmo", are read by ordinary code and not by the model. That code only acts on clear commands and ignores questions. When it finds one, the expense is written first, and only then does PocketMoney reply. In testing it wrote the expense five times out of five, where before it had been hit or miss.

Everything else has a backstop. If the model ever says something was logged or settled and no write happened, its reply is swapped for an honest question. And our confirmations are kept out of the history the model sees, so there is nothing for it to copy.

Numbers come from code

The same idea covers every figure PocketMoney shows you. Who owes whom, and how much, is worked out in code from the ledger. The model can talk about those numbers, but it is not allowed to do arithmetic on your balances or make up a figure it is not sure of. If it does not know, it asks.

This is a less magical way to build an AI product. But a money app that is right most of the time is a money app you cannot use.