How we test AI companion apps: the GenLovers benchmark
The fixed script, memory exam, media requests and scoring rules we run on every AI companion app, so each GenLovers review is graded the same way.
Benchmark version 1 · Last updated:
The rules every review follows
- Every app gets the same test persona, the same 36-message script, the same recall exam and the same media requests, in the same order.
- Every check is graded pass, partial or fail. Partial is common and expected. The stars are arithmetic on those grades: pass counts 1, partial 0.5, fail 0, scaled to 5. Nobody types a star rating by hand.
- A feature the app does not have, or lists as coming soon, fails every check of that feature. A request the app refuses is a fail too, because our requests are plain ones and a refusal means the app gives you less freedom.
- Every grade has a receipt: a screenshot, a clip, a recording we keep private or a logged number. No receipt, no grade.
- We grade the first result. No re-rolls and no retouching. A retry is allowed only when the app errored, and it is logged with its cost.
- The adult requests are graded like the others, but their results never appear on this site. They live in our private, age-restricted Discord.
- Voice is judged on the app's own voiced replies, not on fixed sentences. We never publish audio, because it is the vendor's output and most apps give no way to download it. We describe the voice in words.
- Times are measured by us from the tap on send to the first usable result. They are shown as numbers but not scored yet: we freeze the speed thresholds after we have measured more apps.
- Prices are the vendor's published claims, with the date we read them, next to what we spent. If a vendor gives us the plan we test on, the review says so.
The nine modules, and what each one tests
The weights add up to 100 and come in three tiers. Conversation, images, video and customization (with the catalog) weigh 15 each, because they make a companion app complete. Memory, voice and scenarios weigh 10 each. Value and privacy weigh 5 each. A module the app does not have scores 0 stars and keeps its weight.
Conversation · weight 15%
We send every app the same 36 messages from our test persona, Chris. 25 of them state a fixed fact about him. The other 11 are free replies to whatever the app just said. We grade the replies on staying in character, repeating phrases, taking initiative, matching the length we ask for, handling a trap question and keeping up when the tone shifts.
What we grade
- Stays in character
- Avoids repeating phrases
- Takes initiative
- Reply length follows the request
- Handles the trap question
- Follows tone shifts
- Speed of text replies
Memory · weight 10%
After the 36 messages we ask 22 questions: 20 about facts we gave and 2 about things we never said. Each answer is graded correct, partial or wrong. An invented answer counts as a hallucination. An old value kept after we corrected it counts as stale.
What we grade
- P01 Name and age
- P02 Where Chris lives
- P03 Job and shift
- P04 Position and team
- P05 League day after the update
- P06 Workout routine
- P07 Dog's name and breed
- P08 Cat's name
- P09 His mom
- P10 His sister and her kids
- P11 What happened to his dad
- P12 Relationship status
- P13 Best friend
- P14 Food he loves and hates
- P15 Allergy
- P16 Favorite drink
- P17 Soccer club
- P18 Camper van savings after the update
- P19 What he drives
- P20 Birthday
- T01 Trap: invented favorite color
- T02 Trap: invented brother-in-law
Scenarios · weight 10%
If the app offers scenes, roleplay modes or story runs, we start one and direct it. We grade whether the scene stays continuous, whether the app follows the direction we give, and whether it can end the scene cleanly.
What we grade
- Scene stays continuous
- Follows our direction
- Ends the scene cleanly
Images · weight 15%
We send three fixed requests: one SFW and two adult. The SFW result is shown on this page. The adult results stay locked and are posted to our age-restricted Discord. Each result gets four checks: follows the request, hands, text and artifacts. If the app refuses a request, every check for that request fails.
What we grade
- I1 follows the request
- I1 hands and fingers
- I1 text is legible
- I1 body and artifacts
- I2 follows the request
- I2 hands and fingers
- I2 text is legible
- I2 body and artifacts
- I3 follows the request
- I3 hands and fingers
- I3 text is legible
- I3 body and artifacts
- Same character across the images
- All three requests accepted as asked
- Speed of images
Video · weight 15%
We send three fixed video requests: one SFW and two adult. Each clip gets four checks: follows the request, natural motion, same character and artifacts. If the app refuses a request, every check for that clip fails. If the app has no video at all, this module is left out of the total.
What we grade
- V1 follows the request
- V1 natural motion
- V1 same character
- V1 artifacts
- V2 follows the request
- V2 natural motion
- V2 same character
- V2 artifacts
- V3 follows the request
- V3 natural motion
- V3 same character
- V3 artifacts
- All three requests accepted as asked
- Speed of video
Voice · weight 10%
We play at least ten of the app's own voiced replies: some with numbers and place names, a reply to a sad message, a playful one and a one-line answer. We grade clarity, pronunciation, naturalness and whether the voice stays the same. If the app has a voice call, we also time the reply, interrupt it and check the voice stays the same.
What we grade
- Clarity
- Pronunciation
- Naturalness and emotion
- Same voice across replies
- Speed of voice replies
- Call reply latency
- Call interruption
- Same voice on the call
Value and price · weight 5%
We log what is free and what is paid for chat, images, video and voice, what each costs in credits or money, and what we actually spent during the test. We also check whether the renewal price is shown before paying and how many steps it takes to cancel.
What we grade
- Free tier is usable
- Price and renewal shown before paying
- Limits and credit costs are clear
- Cancelling is easy
- What the paid plan adds is worth it
Customization and catalog · weight 15%
We build a companion from scratch and grade how deep the controls go: personality traits, backstory and appearance, and whether the changes show up in the conversation. We also count the ready-made companions by gender and by visual style, because most users pick from the catalog. An app with no way to create a companion fails the four creation checks.
What we grade
- Creation depth
- Personality controls
- Backstory controls
- Appearance controls
- Enough ready companions to choose from
- Women, men and other categories
- More than one visual style
Privacy and safety · weight 5%
We check the age gate, whether an account and its data can be deleted and exported, what the privacy policy says about chats, and whether the app uses pressure tactics to keep you subscribed.
What we grade
- Age gate
- Account and data deletion
- Data export
- Privacy policy read
- No pressure tactics
Versions and corrections
Every review shows the benchmark version it was graded on. Adding a module, changing a weight or changing a check makes a new version, and earlier reviews keep their version label until we re-test them. If you can reproduce a result differently, write to us through the contact page and we fix the page.