Do two people at the same company agree on a tool?
Less than you would expect. When two different people at one company each rate the same product in their own interviews, their scores sit about 1.5 points apart on average, and four pairs in ten are two or more points apart. That is most of the way to the disagreement between buyers at different companies, who sit about 1.8 apart on the same products, with roughly half of pairs two or more points apart. The yardstick is the same person asked again months later: their two answers sit about a point apart, and fewer than a quarter of them are two or more points apart. People are fairly consistent with themselves. Two people who share an employer, a stack and often a contract are not much more consistent with each other than strangers are.
Three terms run through the page. Colleagues are different people at the same company who rated the same product in separate interviews. The same person is one buyer’s ratings of that product across two interviews, usually about six months apart. Different companies are buyers at separate employers who rated the same product. Each figure is the average gap between two 1–10 ratings of one product. These figures describe how much ratings vary in these interviews; they say nothing about who is right, or about how a product’s rating will change.
Each bar is the average absolute gap between two 1–10 ratings of the same product, for each of the three groups. Bars are centred on a typical product average for comparison, so only their length carries data. The shaded band is the middle half of product averages, the 25th to the 75th percentile, among products with at least twenty-five ratings. Where three or more colleagues rated one product, every pairing of them counts, and the different-companies row is matched to the products the colleague pairs rated. The same-person row leaves out repeat interviews in which the interviewer referred back to the earlier rating.
What does a 1.5-point gap mean on a 10-point scale?
It is large next to the differences between products. Among products enough buyers have rated to read with confidence, the middle half have averages between about 6.8 and 7.6, a band under a point wide. A 1.5-point gap is roughly the distance between products at the top-tenth and bottom-tenth marks of that field, and only about one pair of those products in twelve has averages that far apart. These are different comparisons: individual ratings vary widely, while averages of many buyers cluster much more tightly. The point is not that product averages tell you nothing. It is that one employee can sit further from a colleague on a single tool than many competing products’ averages sit from each other.
The gap often crosses the line between satisfied and not. Taking 8 or higher as satisfied and 6 or lower as not, which is our line rather than a standard definition, about one colleague pair in four has one person on each side of it. For the same person asked twice, that happens about one time in nine. For buyers at different companies, it happens about three times in ten, which is barely more often than it does between colleagues.
Why would colleagues rate the same tool so differently?
Often because they are describing different experiences of it. The widest colleague disagreements tend to come with two buyers recounting quite different encounters with the same product. The ratings measure how far apart people are, not why, so the examples below illustrate those gaps rather than break them down. What they suggest are plausible sources: who carries the cost, how far into the rollout someone is, how directly they use the tool and what they expected of it.
One owns the cost, the other gets the benefit. At a small company, an analytics and loyalty lead gave the reviews platform a 10: “They’re pretty good.” The customer-experience director gave it a four. Asked whether it sat with another team, they answered, “No, that was more under my purview,” and explained that the team does not use the platform’s analytics, piping review data into its own analysis instead, and that cheaper tools could capture reviews for a fraction of the cost.
One was asked earlier in the rollout. At another company, a growth marketing leader gave a newly installed media-data tool a six, “only because we’ve just onboarded them.” A colleague on the same team, interviewed about six months later once it was fully in use, rated it nine and called it “the biggest game changer for us.”
One is rating it from somewhere else. At a large enterprise, a personalization technology director rated the web-analytics platform nine and a half while saying that “in my current role, I don’t use much,” drawing on heavy use at a previous employer. A loyalty-insights director who uses it gave it six, and wondered whether that was “a little bit of how we’re using it versus the actual capabilities of the tool.” Elsewhere, an ecommerce leader gave the data warehouse a 10 after saying “I’m not super close to it, so I can’t speak a lot about it,” while the data-products leader whose team works with it put it “in the 7-to-8 range.”
And sometimes it is the rater. A senior IT leader who gave the commerce platform five opened with a warning: “Our CEO calls me the curmudgeon.” A CRM and loyalty colleague at the same company rated the same platform eight.
These accounts do not establish which colleague is more accurate. They show one product drawing different verdicts depending on which costs, capabilities or experiences a person weighs: the bill, the first month, a past job, the part of it their team touches. Hear only one of those perspectives and you can miss much of how the same company’s implementation is going. A disagreement is evidence worth investigating even when it cannot be settled.
Is it seniority, department or company size?
None of them accounts for most of it. A vice president and a manager at the same company sit about 1.5 points apart on a tool, the same as two colleagues at the same level, and across the interviews senior leaders rate the same products less than a tenth of a point below practitioners on average. Two people in the same department sit slightly closer than two in different departments, about 1.5 against 1.6, and both are well above the gap for one person asked twice. And the disagreement appears at every company size, as the table shows.
| Company size | Same person | Two colleagues | Different companies |
|---|---|---|---|
| Under 200 employees | 1.0 | 1.3 | 1.6 |
| 200 to 999 | 1.0 | 1.4 | 1.7 |
| 1,000 to 9,999 | 1.1 | 1.6 | 1.8 |
| 10,000 and above | 0.9 | 1.5 | 1.8 |
Colleagues sit closest at companies under 200 employees, about 1.3 points apart, against about 1.5 to 1.6 from a thousand employees up. In every band, two colleagues are further apart than one person asked twice and closer than buyers at different companies. People at smaller organizations may simply share more of their experience of a tool, but size alone does not explain why.
The counter-current: a person’s own score moves too
The same-person yardstick is not fixed. Products change, contracts change and people’s jobs change, and one buyer’s answers drift further apart the longer the gap between interviews: asked a year or more later, they sit about 1.4 points from where they started. So timing contributes to the variation. Elapsed time alone does not appear to account for the colleague gap, though: two colleagues interviewed within three months of each other still sit about 1.5 points apart, similar to one person checked a year or more later.
Want this checked against your own stack?
Start my checkup →What this means when you check references
Our recommendation, based on these findings rather than on a reference-checking practice buyers report following: treat any individual rating as one perspective, whether it comes from a vendor reference, a peer at another company or the internal champion. Colleagues here differ by about 1.5 points on average, so agreeing with one person at a company does not show how others there experience the tool. Look for people with different responsibilities, not several people doing the same job. These findings do not set a number of references that is enough; the aim is to see which experiences are represented and which are missing. In practice:
- Talk to more than one person per company, and pick them by job, not title. The person who runs the tool day to day, the person who owns its budget or contract, and someone who uses what it produces.
- Ask how close they are and how long they have had it. A score from the first month of a rollout and one from a year in answer different questions.
- Ask which parts they use. Two people can rate the same product name while describing different modules of it.
- Do the same inside your own company. Before a renewal or a replacement, collect ratings from several roles rather than taking the owner’s, or the loudest critic’s, as the company’s view.
- Read the spread, not just the average. If three people at one company say 9, 6 and 4, the disagreement is the finding, and the reasons behind it are what to test in a demo.
What not to read into this
Three inferences go further than the data. The first is that the lower-rating colleague is the more accurate one, or the higher one: the page measures how far apart two ratings fall, not which is right. The second is that published ratings are therefore useless. An average across many buyers holds more perspectives than any single reference, though how far it can be relied on depends on how many ratings sit behind it, who is represented and whether they are evaluating the same thing. Here, well-sampled averages sit in a much narrower range than individual ratings. That does not remove the limits of sampling or comparison, but it is a reason to treat one opinion and a well-supported aggregate differently. The third is that a person’s role causes their rating. The interviews show who rated what, not why, and the explanations above are our reading of the widest gaps. One structural caveat applies too: at large enterprises, two employees may work in different business units, brands or environments under one vendor name, so their ratings may describe different configurations. Similar disagreement appears at small companies, but the interviews do not establish how much of the gap comes from differences in implementation.
Common questions
Do people at the same company rate software the same way?
Not closely. In Alium's interviews, two different people at the same company rating the same product sit about 1.5 points apart on a 10-point scale on average, and four pairs in ten are two or more points apart. Buyers at different companies rating the same products sit about 1.8 apart. One person asked again months later sits about 1 point from their earlier answer. So a colleague's rating is only a little closer to a coworker's than a stranger's is.
What does a 1.5-point difference in a software rating mean?
More than it sounds. The middle half of well-sampled products in Alium's interviews have average ratings between about 6.8 and 7.6, a band less than a point wide, and 1.5 points is roughly the distance between products at the top-tenth and bottom-tenth marks of that field. Averages of many buyers cluster far more tightly than individual ratings, so this does not make product averages meaningless; it means one colleague can differ from another by more than many competing products differ on average. In about one colleague pair in four, one person rates the tool 8 or higher and the other rates it 6 or lower.
Why do colleagues rate the same software differently?
The interviews do not measure the cause. Read side by side, the widest gaps tend to come with different jobs around the same tool: one person runs it and another benefits from it, one uses a different part of it, one is asked early in the rollout and the other a year in, or one rates it from experience at a previous company. Seniority and department explain little: a vice president and a manager at the same company are no further apart than two people at the same level.
How many references should you talk to before buying software?
Our recommendation, not a practice buyers describe: there is no fixed number that is enough, but talk to more than one person per company, chosen by job rather than seniority, such as the person who runs the tool day to day, the person who owns its budget or contract, and someone who uses what it produces. Ask each how close they are to it, how long they have had it and which parts they use. Two people at the same company disagree almost as much as strangers do, so the aim is to see which experiences are represented and which are missing.
Does seniority change how people rate software?
Very little. In Alium's interviews, vice presidents and C-level leaders rate the same products less than a tenth of a point below managers and directors on average. Between colleagues, a senior leader and a practitioner are no further apart than two people at the same level, about 1.5 points either way.
For how the same product’s rating shifts with the size of the company rating it, see does company size change how buyers rate software; for where shortlists and references come from in the first place, how software buyers actually find vendors; and for what happens when a buyer takes their opinions to a new employer, do marketing tools follow people.
Get a checkup on your tech stack
A checkup for your marketing stack, built from hundreds of verified buyer interviews: what matters most to you, a shortlist for your stack, and your next steps.
Start my checkupFree · no password · your answers are anonymized before they inform research like this.