Costs
Part of 6 points on choosing fraud detection tools
Before you buy fraud detection tools, sample twenty creators by hand
A fraud detection tools strategy that starts by hand: sample twenty creators you know, learn your base rate, and keep the known cases to test every product.
Buying a detector before you have looked at a single creator by hand means buying a number you cannot read. A hand sample of twenty creators you already know gives you a base rate, an eye for a real anomaly, and test cases. Then you can pay for a tool to do at volume what you learned once.
What to take away
- Sample by hand until you know what you are looking at. Only then pay a tool to do at volume what you learned to do once.
- The sample gives you a base rate, a trained eye and known cases, and the cases are the only honest way to compare two products.
- Screen before you book. A finding after the money has gone out is a receipt, not a control.
Why the sample comes before the purchase
A detector's output means little until you know what normal looks like in your own roster. The sample tells you that before you have committed to a subscription. Most of what you see will be ambiguous, and that ambiguity is the lesson.
Most agencies buy detection at the wrong point in the process and then wonder why nothing changes. A score that arrives after the money has gone out changes nothing. The sample moves the learning to before the spend, where it can still shape a booking.
Run the sample in five steps
Take twenty creators you have already worked with. Look at their recent activity yourself, slowly, without a tool. Slowness is the point: the aim is to see patterns, not to reach a verdict on any one person.
Sample twenty creators by hand
- Take twenty creators you have worked with
- Review recent activity slowly, no tool
- Note what is ambiguous and what is obvious
- Learn your actual base rate
- Build known cases to test products
- Pick twenty creators you have booked before, so you know how their campaigns actually went.
- Open each profile's recent activity and read it slowly, with no tool running.
- Write one line per creator against the four signals belowwhat looked ordinary, what looked odd, what stayed ambiguous.
- Mark the cases you are confident about in either direction, and keep that list.
- Look again at the ambiguous ones on another day, to see whether your first read held.
Four signals are worth reading on each profile. None proves fraud alone, and each is judged against what you already know of the creator.
- Engagement against audience sizelikes and comments that look far too small or far too large for the follower count.
- Follower growtha sudden jump with no post, campaign or mention that explains it.
- Comment qualitygeneric, repeated or off-topic comments from accounts with no profile of their own.
- Audience placefollowers concentrated in countries the content is not written for.
A few patterns become obvious once you have seen them twice, and the cases you worried about are often not the odd ones. Write down what surprised you, because the surprises are what a tool will later be measured against.
What the sample tells you
It tells you your base rate. That decides whether a detector's output is meaningful at all, because a flag means one thing when odd cases are rare and another when they are common. The guide to detection and audience quality sets out what a score is.
It teaches your team what a real anomaly looks like, so a flag can be judged rather than obeyed. A flag nobody can judge turns into an order, and the sample is the cheapest way to prevent that.
And it produces a small set of known cases you can use to test any product you later consider, whether HypeAuditor, Modash or another. That is the only honest way to compare two of them. Run each product on the same cases and note which ones it misses and which it flags without reason.
Where in the process to screen
Screen at the point where you can still say no. The sample is a hand version of that step, and the tool should later sit in the same place in your process, not at the end of it.
What a finding is worth
- Before shortlisting
- A cheap exclusion, no conversation needed
- Before contracting
- Full value: you can decline, renegotiate or ask
- After booking, before posting
- Partial: you may be able to change the deliverable
- After the campaign
- Almost none for this campaign
- At payment
- A remedy only if the contract gave you one
What it costs you
- Before shortlisting
- A little time, and some false exclusions you will never know about
- Before contracting
- An awkward conversation, occasionally with someone innocent
- After booking, before posting
- A dispute mid campaign, which is expensive in goodwill
- After the campaign
- Nothing now, and you pay for it again next time
- At payment
- The relationship, usually
When you screen vs what it's worth
When you screen
- Before shortlisting
- Cheap exclusion
- Before contracting
- Full value
- After booking
- Partial
- After the campaign
- Almost none
- At payment
- Remedy only
What a finding is worth
- Before shortlisting
- Time, hidden false exclusions
- Before contracting
- Awkward conversation
- After booking
- Mid-campaign dispute
- After the campaign
- Nothing now
- At payment
- The relationship
What it costs you
- Before shortlisting
- Before contracting
- After booking
- After the campaign
- At payment
The middle row is where screening pays, but most agencies screen in the last two rows, when reporting happens and someone asks. Moving it earlier is a process change, not a purchase, and it is worth more than any upgrade. Where the step sits is a question for your campaign workflow tools, not a detector.
Turn the cases into a threshold policy
Write a threshold policy from the sample, before a purchase and not after. Once a score exists, it starts standing in for the judgment it was meant to inform. That is Goodhart's law in your booking process: the measure becomes the target.
Write down four things
- Threshold at which you look
- Who looks, and what they see
- What the creator is told, and when
- What happens when the person disagrees
The sample answers the first of four decisions a finding must trigger: which result earns a closer look and which earns action. The other three are who acts, what they may do, and who is told. Guidance on who decides and who is told covers those for an agency screening on a client's behalf.
Write the known cases down as decisions rather than scores, as worked examples do when they turn numbers into written decisions. Everyone optimizes the number instead of the thing it was meant to measure, and a case written as a decision is harder to game.
What the law does and does not supply
The law will not supply the remedy for you. The Federal Trade Commission's Rule on the Use of Consumer Reviews and Testimonials is 16 CFR Part 465.
It makes it a violation to purchase or procure fake indicators of social media influence that the buyer knew or should have known to be fake and that materially misrepresent their influence or importance for a commercial purpose.
The wording turns on what the buyer knew. A hand sample is how an agency learns what a warning sign looks like in its own roster. It does not decide who bears a loss once an audience turns out to be inflated.
That is decided by what your agreement says. So the warranty about the creator's audience and the clause on what evidence both sides accept come before any tool, and the sample tells you what to put in them.
When the sample says detection is not the answer
Sometimes the answer is not detection at all.
- Paying on delivered outcomes rather than on promised reach, a cost per action basis, removes the incentive that most inflation exists to serve.
- Working repeatedly with people you already know converts a screening problem into a relationship, which is cheaper and more reliable.
- Buying from a source that is party to the deal rather than an introduction changes who carries the risk.
- Fixing your own measurement first, because a good deal of what gets reported as fraud is a reporting problem in the other direction.
None of these are arguments against buying a detector. They are arguments for knowing what you are buying it to replace. If your sample shows few odd cases, one of them may beat the purchase.
What to write down before you shop
One page before you shop
- The decision the tool informs
- Where in the process the answer arrives
- The threshold policy
- The named owner
- Known cases to test against
- The contract remedy
The page has six lines:
- What the tool is meant to replace or speed up, in one sentence.
- The point in your process where it runs, which should be before contracting.
- Your threshold policy, written from the sample.
- The named person who acts on a finding, and the person who recommends.
- The known cases from your hand sample, to test each product against.
- What your agreements let you do with a finding.
An agency with that page will get value out of a modest product. An agency without it will get a subscription and a monthly report that nobody reads past the summary.
Common questions
How many creators does the sample need?
Twenty is enough to see patterns and not enough to prove a rate. Treat the base rate as a rough guide, and widen the sample if your cases look evenly split between ordinary and odd.
Do we need a tool at all if we only run a few campaigns a year?
Probably not yet. At low volume, manual review by someone who has learned what to look for is more accurate than a cheap product, and the contract terms do most of the work either way.
What if we have not worked with enough creators to sample?
Sample whoever you have, and say plainly that the base rate is thin. Then run any product you try on that same small set, and treat its early flags as questions rather than answers.
How often should the threshold policy be revisited?
When something surprises you, and otherwise once a year. A threshold that has never moved is either well chosen or never used, and it is worth knowing which.
