We Built AI Risk Analysis for Carrier Email. Our First Version Flagged a Normal Booking Message.

A risk check that fires on ordinary email is worse than no check at all. Brokers who see flags on normal freight business stop reading the flags — and then the box is wallpaper on the one message that actually matters. That is the failure mode we hit, and it is the reason this post exists.

The test email was as ordinary as freight gets: a rate, a lane, equipment type, a pickup time, and a signature block. No payment language. No urgency. No contact changes. Nothing a fraud analyst would look at twice.

The model came back with two flags. It read "rate confirmation looks good on our end" as urgency pressure combined with an unusually good rate. It read a plain signature block — name, title, company, phone — as an announcement of changed contact details. Both readings were fabricated from text that said neither thing.

This was a control test, not a positive test. We were checking that a clean message stayed clean. That distinction matters more than it sounds: it is easy to build a check that catches suspicious language, and hard to build one that stays quiet the other 99% of the time. If we had only tested it on fraudulent emails, it would have looked like it worked.

The cause was the prompt, not the model. We had given it six pattern labels — "urgency pressure", "changed contact details", and so on — and asked which were clearly present. That framing has two defects, and both push toward false positives.

First, a topic label is not a test. "Changed contact details" describes a subject, and a signature block is about contact details, so a model matching on topic will match. What the pattern actually means is narrower: the message must announce that contact information has changed. Nothing in the label said so.

Second, nothing told the model that matching nothing was an acceptable answer. Language models are built to produce output. Asked which of six categories apply to a piece of text, a model will look for reasons one applies, because returning an empty list feels like failing the task. On ordinary freight email — which is most freight email — the correct answer is an empty list almost every time.

Rewriting the check took three changes. Only one of them was about catching more; the other two were about restraint.

The highest-value pattern to surface is a payment redirect — a message asking that money go somewhere new. It is also the pattern most likely to be legitimate, and that is a freight-specific problem a general-purpose risk model gets wrong.

Roughly 80% of carrier payments route through a factoring company. A Notice of Assignment directing payment to a factor is normal business correspondence, sent constantly, by carriers doing nothing wrong. A check that flags payment redirects without knowing this would fire on a large share of legitimate invoices — and would train brokers to ignore it before it ever saw a real attack.

So the evidence test carries an explicit carve-out. A Notice of Assignment, an established factor named on the rate confirmation, or standard "remit to our factor" language is not the pattern. What is the pattern: changing previously agreed payment routing mid-relationship, especially with urgency, or instructions that bypass a factor already on file.

That last case is worth stating plainly, because it cuts the other way. A message saying "we have left our factor, pay us directly now" is exactly what an impersonator sends to divert payment away from a legitimate factor. Our check flags it — and it should, even when the release is genuine. A broker who receives one should confirm it out-of-band before changing where money goes.

We tested the rebuilt version against 46 labelled freight emails, weighted deliberately toward the negatives — 30 of the 46 were messages that should produce nothing at all.

On those 30 — ordinary booking emails, rate negotiations, detention disputes, breakdown notices, cargo claims, onboarding packets, and four legitimate factoring notices — it returned zero flags. That is the number we care most about, because it is the one that decides whether a broker keeps reading.

On the fraudulent messages it caught 7 of 7 obvious cases and 7 of 8 written specifically to evade the pattern list — polite, well-formatted messages with no urgency words and the payment redirect buried mid-message. The one it missed is instructive: an email that never pressures anyone to skip verification, but politely offers to skip it as a favour to the broker's own busy compliance queue. Deference reads as cooperative, so the test did not fire.

We are publishing that miss because it is the honest shape of this problem. This is a language check, not a verdict. It reads the words in one message; it does not know the carrier, and it cannot tell you whether the person emailing you is who they claim to be. Patterns are not proof.

The lesson generalizes past our own product. When a vendor tells you their AI detects freight fraud, the first thing to note is the verb. Reading an email cannot establish that fraud occurred — the same sentences appear in legitimate business every day. What a language check can honestly offer is a risk signal: a reason to slow down and verify.

The question that then separates a real tool from a demo is not "what does it catch?" It is "what does it stay quiet on, and how do you know?"

Ask what the false-positive rate is on ordinary email, and what it was measured against. Ask whether every flag carries evidence you can read yourself, or just a score. Ask what happens on a legitimate factoring notice. A vendor who has not tested the boring case has not finished building the thing.

And treat any language signal as one input among several. Message language tells you how an email is written. It does not tell you whether the authority is active, whether insurance is on file, or whether the sender's address matches the carrier's federal record. Those are separate checks against separate sources, and the ones that catch an impostor using a real carrier's identity.