Costs
Part of 6 points on choosing fraud detection tools
Fraud detection tools mistakes: three kinds, three victims
Fraud detection tools mistakes sorted by what each one costs you: money, a relationship with somebody who did nothing wrong, or the ability to decide next time.
Fraud detection tools mistakes tend to be discussed as though there were one kind of error, which is missing something real. There are three kinds, and they have different victims.
Some mistakes cost money. Some cost a relationship, often with somebody who did nothing wrong. The third group costs you the ability to decide anything next time, which is the most expensive and the least visible. Sorting them this way shows which ones your process is optimized to avoid, and which it is quietly accepting.
The three victims are identifiable, and naming them is what makes the rest of this page usable. The first is whoever paid: the brand, the client budget, or the agency that carried the cost, losing money it cannot recover. The second is the creator who was wrongly flagged — they did nothing wrong, they are rarely told, and the cost lands on their record with that buyer. The third is your own team, which pays in judgment: it loses the ability to tell a good call from a lucky one.
The products in play — HypeAuditor, Modash, CreatorIQ, Fohr and Upfluence among them — read what a platform exposes and turn it into a score. None of them sees a deal paid for off-platform, and none of them sees the audience a creator reached in a private channel. The mistakes below live in how that output is bought, read and acted on, and they look much the same whichever product produced the number.
What to take away
- Detection conversations focus almost entirely on the first group. The second and third are where the durable damage sits.
- Every threshold encodes a view about what each error costs. If you did not set it, somebody else's view is running your bookings.
- Logging the cases you cleared is what prevents the third group, and almost nobody does it.
Mistakes that cost money
The victim here is the buyer: the brand, the client budget, or the agency that absorbed the loss.
- Screening after the campaign, when a finding tells you what you lost rather than what to decide.
- Buying a product before the agreement gives you a remedy, so a real finding produces a document and nothing else.
- Paying on promised reach rather than on validated outcomes, which leaves the incentive that most inflation exists to serve fully intact.
- Skipping unique links and codes per partner, so every dispute becomes opinion against opinion.
- Treating the platform's own report as independent evidence about itself.
The mistake to fix first is buying a product before the agreement gives you a remedy, because it costs nothing. A remedy written into the agreement is worth more than any score. What you want in the contract is a validation window on delivery, unique links and codes per partner, and a holdback you can apply while a finding is being checked.
Screening is not only a commercial matter. The FTC's Endorsement Guides require creators to disclose material connections, and the FTC's rule on fake reviews and testimonials gives a regulator a direct interest in the same evidence. Keep the record of what was checked, because more than one audience reads it.
Mistakes that cost a relationship
These are the ones nobody counts, because the person you were wrong about does not tell you.
The victim is the creator who was wrongly flagged — and, if the accusation went to a client, the client relationship as well.
Relationship mistakes and who pays
The mistake
- Declining on a score
- Creator who did nothing wrong
- Telling why without components
- You, publicly
- One threshold across categories
- Whole formats and audience sizes
- Sharing score with client
- Every future booking
- Blocking traffic on pattern
- Customers you never hear from
Who pays
- Declining on a score
- Accusation from a machine
- Telling why without components
- An argument you cannot win
- One threshold across categories
- Ruled out by irrelevant rule
- Sharing score with client
- A number becomes the deliverable
- Blocking traffic on pattern
- Nothing, which is the problem
What it looks like
- Declining on a score
- Telling why without components
- One threshold across categories
- Sharing score with client
- Blocking traffic on pattern
A score that starts as one input becomes the thing the client asks for, and then the score is what you are being judged on. That is Goodhart's law arriving in a client report — the measure becomes the target and the underlying judgment quietly stops mattering. What belongs in that report instead is an account of what was checked and what was found, which is a question of measurement layers rather than of a number.
The false positive is the invisible one. It removes real customers, and you never see the ones you turned away.
Work out roughly what each error costs before setting a threshold. Then the decision gets made, not inherited.
Mistakes that cost the ability to decide
The third group is the subtle one. Each of these leaves you unable to improve, which means every future decision is as uninformed as the first.
The victim is your own team, which pays in decisions rather than in cash.
Decision mistakes to avoid
- Logging only rejected cases
- Never establishing your own base rate
- Letting the tool win by default
- Adopting a default threshold
Logging only the cases you rejected. If your record of what a bad case looks like is built entirely from cases you already thought were bad, it confirms whatever you believed at the start. Logging the clears is what breaks the loop, and it is the single most valuable habit in this whole area.
Never establishing your own base rate. Without knowing roughly how common the underlying problem is in your categories, a flag rate is uninterpretable. Twenty creators reviewed slowly by hand tells you more about your population than any product will, because the product does not know your population. This is the core of detection and audience quality: a score cannot be read without a base rate, and the base rate can only come from your own bookings.
Letting the tool win by default when a reviewer disagrees. Record both, and record which one the decision followed. Over a year that record is the only honest evidence about whether the subscription is earning its price.
Adopting a default threshold. Moving a threshold trades one error against the other, which is the whole content of precision and recall and is not a setting anyone can choose for you. A default encodes somebody else's tolerance for the two kinds of error. Theirs is not yours, and the difference is not a technical matter but a commercial one.
Not writing down what was outside a check. A note that says what could not be seen is honest. A note that omits it reads, months later, as a claim that everything was fine.
Where the mistake is not a mistake
Declining a booking on an unexplained pattern, without telling the creator why, is often correct. So is buying nothing at all at low volume, where unique codes and a validation window will find nearly everything a product would. So is deciding that a whole partner type is not worth the screening cost and simply not using it.
The test is whether the choice was made and recorded. An unrecorded decision becomes a habit, and habits in this area are how agencies end up with rules nobody can explain to the person affected by them.
A surprising share of what gets reported as fraud turns out to be a measurement problem instead. Where the counterparty is represented, the conversation about a finding runs through a manager rather than the creator, which changes it considerably and puts talent representation in the middle of the correction.
Common questions
We flagged a creator and were wrong. What is the repair?
Say so directly, say what you looked at, and say what you have changed. The relationship usually survives an admitted error and rarely survives a defended one.
Should the threshold be the same for every client?
Only if their tolerance for a missed case and a wrong accusation is the same, which it is not. Set it per client, and write down why.
Our tool flags a third of a shortlist. Is that plausible?
It tells you where the threshold sits, not what is true. Calibrate in three steps: take twenty partners you already know are clean, run them through the tool, then count how many come back flagged. That count is your false-positive rate on a population you actually understand, and it is the only number from the product you can defend.
Is any of this worth doing without a product?
Yes, and it is where to start. Unique codes, a validation window, a holdback, and twenty creators reviewed by hand will teach a team more than a first subscription does.
