Do colleagues rate the same software the same way?

Not closely. Two people at the same company, rating the same product in separate interviews, are about 1.5 points apart on a 10-point scale. Buyers at different companies are about 1.8 apart, and one person asked again months later lands about a point from their first answer. Knowing where someone works tells you only a little about how they will rate its software.

Based on verified interviews with the marketing and ecommerce leaders who check references before they buy software, at DTC and enterprise brands. Buyers are anonymized before publication; vendor names and views are reported as given. No vendor paid to appear or could edit this page.

1.5
Points: the average gap between two colleagues’ ratings of the same product. Buyers at different companies are about 1.8 apart.
1 in 4
Colleague pairs where one person rates the tool 8 or higher and the other rates it 6 or lower.
~1
Point: how far one person’s rating moves when they are asked again months later. Colleagues disagree half as much again.

Do two people at the same company agree on a tool?

Less than you would expect. When two different people at one company each rate the same product in their own interviews, their scores sit about 1.5 points apart on average, and four pairs in ten are two or more points apart. That is most of the way to the disagreement between buyers at different companies, who sit about 1.8 apart on the same products, with roughly half of pairs two or more points apart. The yardstick is the same person asked again months later: their two answers sit about a point apart, and fewer than a quarter of them are two or more points apart. People are fairly consistent with themselves. Two people who share an employer, a stack and often a contract are not much more consistent with each other than strangers are.

Three terms run through the page. Colleagues are different people at the same company who rated the same product in separate interviews. The same person is one buyer’s ratings of that product across two interviews, usually about six months apart. Different companies are buyers at separate employers who rated the same product. Each figure is the average gap between two 1–10 ratings of one product. These figures describe how much ratings vary in these interviews; they say nothing about who is right, or about how a product’s rating will change.

Average distance between the two ratings Where the middle half of well-sampled products’ averages sit

Each bar is the average absolute gap between two 1–10 ratings of the same product, for each of the three groups. Bars are centred on a typical product average for comparison, so only their length carries data. The shaded band is the middle half of product averages, the 25th to the 75th percentile, among products with at least twenty-five ratings. Where three or more colleagues rated one product, every pairing of them counts, and the different-companies row is matched to the products the colleague pairs rated. The same-person row leaves out repeat interviews in which the interviewer referred back to the earlier rating.

What does a 1.5-point gap mean on a 10-point scale?

It is large next to the differences between products. Among products enough buyers have rated to read with confidence, the middle half have averages between about 6.8 and 7.6, a band under a point wide. A 1.5-point gap is roughly the distance between products at the top-tenth and bottom-tenth marks of that field, and only about one pair of those products in twelve has averages that far apart. These are different comparisons: individual ratings vary widely, while averages of many buyers cluster much more tightly. The point is not that product averages tell you nothing. It is that one employee can sit further from a colleague on a single tool than many competing products’ averages sit from each other.

The gap often crosses the line between satisfied and not. Taking 8 or higher as satisfied and 6 or lower as not, which is our line rather than a standard definition, about one colleague pair in four has one person on each side of it. For the same person asked twice, that happens about one time in nine. For buyers at different companies, it happens about three times in ten, which is barely more often than it does between colleagues.

This is the average buyer's picture. What does your stack need?A free, peer-powered checkup returns a shortlist for your stack and your next steps.
Start my checkup →

Why would colleagues rate the same tool so differently?

Often because they are describing different experiences of it. The widest colleague disagreements tend to come with two buyers recounting quite different encounters with the same product. The ratings measure how far apart people are, not why, so the examples below illustrate those gaps rather than break them down. What they suggest are plausible sources: who carries the cost, how far into the rollout someone is, how directly they use the tool and what they expected of it.

One owns the cost, the other gets the benefit. At a small company, an analytics and loyalty lead gave the reviews platform a 10: “They’re pretty good.” The customer-experience director gave it a four. Asked whether it sat with another team, they answered, “No, that was more under my purview,” and explained that the team does not use the platform’s analytics, piping review data into its own analysis instead, and that cheaper tools could capture reviews for a fraction of the cost.

One was asked earlier in the rollout. At another company, a growth marketing leader gave a newly installed media-data tool a six, “only because we’ve just onboarded them.” A colleague on the same team, interviewed about six months later once it was fully in use, rated it nine and called it “the biggest game changer for us.”

One is rating it from somewhere else. At a large enterprise, a personalization technology director rated the web-analytics platform nine and a half while saying that “in my current role, I don’t use much,” drawing on heavy use at a previous employer. A loyalty-insights director who uses it gave it six, and wondered whether that was “a little bit of how we’re using it versus the actual capabilities of the tool.” Elsewhere, an ecommerce leader gave the data warehouse a 10 after saying “I’m not super close to it, so I can’t speak a lot about it,” while the data-products leader whose team works with it put it “in the 7-to-8 range.”

And sometimes it is the rater. A senior IT leader who gave the commerce platform five opened with a warning: “Our CEO calls me the curmudgeon.” A CRM and loyalty colleague at the same company rated the same platform eight.

These accounts do not establish which colleague is more accurate. They show one product drawing different verdicts depending on which costs, capabilities or experiences a person weighs: the bill, the first month, a past job, the part of it their team touches. Hear only one of those perspectives and you can miss much of how the same company’s implementation is going. A disagreement is evidence worth investigating even when it cannot be settled.

Is it seniority, department or company size?

None of them accounts for most of it. A vice president and a manager at the same company sit about 1.5 points apart on a tool, the same as two colleagues at the same level, and across the interviews senior leaders rate the same products less than a tenth of a point below practitioners on average. Two people in the same department sit slightly closer than two in different departments, about 1.5 against 1.6, and both are well above the gap for one person asked twice. And the disagreement appears at every company size, as the table shows.

Average distance, in rating points, between two ratings of the same product, split by the number of employees at the buyers’ company. Different companies compares each band’s colleague pairs with buyers elsewhere rating the same products.
Company size Same person Two colleagues Different companies
Under 200 employees 1.0 1.3 1.6
200 to 999 1.0 1.4 1.7
1,000 to 9,999 1.1 1.6 1.8
10,000 and above 0.9 1.5 1.8

Colleagues sit closest at companies under 200 employees, about 1.3 points apart, against about 1.5 to 1.6 from a thousand employees up. In every band, two colleagues are further apart than one person asked twice and closer than buyers at different companies. People at smaller organizations may simply share more of their experience of a tool, but size alone does not explain why.

The counter-current: a person’s own score moves too

The same-person yardstick is not fixed. Products change, contracts change and people’s jobs change, and one buyer’s answers drift further apart the longer the gap between interviews: asked a year or more later, they sit about 1.4 points from where they started. So timing contributes to the variation. Elapsed time alone does not appear to account for the colleague gap, though: two colleagues interviewed within three months of each other still sit about 1.5 points apart, similar to one person checked a year or more later.

Want this checked against your own stack?

Start my checkup →

What this means when you check references

Our recommendation, based on these findings rather than on a reference-checking practice buyers report following: treat any individual rating as one perspective, whether it comes from a vendor reference, a peer at another company or the internal champion. Colleagues here differ by about 1.5 points on average, so agreeing with one person at a company does not show how others there experience the tool. Look for people with different responsibilities, not several people doing the same job. These findings do not set a number of references that is enough; the aim is to see which experiences are represented and which are missing. In practice:

  1. Talk to more than one person per company, and pick them by job, not title. The person who runs the tool day to day, the person who owns its budget or contract, and someone who uses what it produces.
  2. Ask how close they are and how long they have had it. A score from the first month of a rollout and one from a year in answer different questions.
  3. Ask which parts they use. Two people can rate the same product name while describing different modules of it.
  4. Do the same inside your own company. Before a renewal or a replacement, collect ratings from several roles rather than taking the owner’s, or the loudest critic’s, as the company’s view.
  5. Read the spread, not just the average. If three people at one company say 9, 6 and 4, the disagreement is the finding, and the reasons behind it are what to test in a demo.

What not to read into this

Three inferences go further than the data. The first is that the lower-rating colleague is the more accurate one, or the higher one: the page measures how far apart two ratings fall, not which is right. The second is that published ratings are therefore useless. An average across many buyers holds more perspectives than any single reference, though how far it can be relied on depends on how many ratings sit behind it, who is represented and whether they are evaluating the same thing. Here, well-sampled averages sit in a much narrower range than individual ratings. That does not remove the limits of sampling or comparison, but it is a reason to treat one opinion and a well-supported aggregate differently. The third is that a person’s role causes their rating. The interviews show who rated what, not why, and the explanations above are our reading of the widest gaps. One structural caveat applies too: at large enterprises, two employees may work in different business units, brands or environments under one vendor name, so their ratings may describe different configurations. Similar disagreement appears at small companies, but the interviews do not establish how much of the gap comes from differences in implementation.

Common questions

Do people at the same company rate software the same way?

Not closely. In Alium's interviews, two different people at the same company rating the same product sit about 1.5 points apart on a 10-point scale on average, and four pairs in ten are two or more points apart. Buyers at different companies rating the same products sit about 1.8 apart. One person asked again months later sits about 1 point from their earlier answer. So a colleague's rating is only a little closer to a coworker's than a stranger's is.

What does a 1.5-point difference in a software rating mean?

More than it sounds. The middle half of well-sampled products in Alium's interviews have average ratings between about 6.8 and 7.6, a band less than a point wide, and 1.5 points is roughly the distance between products at the top-tenth and bottom-tenth marks of that field. Averages of many buyers cluster far more tightly than individual ratings, so this does not make product averages meaningless; it means one colleague can differ from another by more than many competing products differ on average. In about one colleague pair in four, one person rates the tool 8 or higher and the other rates it 6 or lower.

Why do colleagues rate the same software differently?

The interviews do not measure the cause. Read side by side, the widest gaps tend to come with different jobs around the same tool: one person runs it and another benefits from it, one uses a different part of it, one is asked early in the rollout and the other a year in, or one rates it from experience at a previous company. Seniority and department explain little: a vice president and a manager at the same company are no further apart than two people at the same level.

How many references should you talk to before buying software?

Our recommendation, not a practice buyers describe: there is no fixed number that is enough, but talk to more than one person per company, chosen by job rather than seniority, such as the person who runs the tool day to day, the person who owns its budget or contract, and someone who uses what it produces. Ask each how close they are to it, how long they have had it and which parts they use. Two people at the same company disagree almost as much as strangers do, so the aim is to see which experiences are represented and which are missing.

Does seniority change how people rate software?

Very little. In Alium's interviews, vice presidents and C-level leaders rate the same products less than a tenth of a point below managers and directors on average. Between colleagues, a senior leader and a practitioner are no further apart than two people at the same level, about 1.5 points either way.

For how the same product’s rating shifts with the size of the company rating it, see does company size change how buyers rate software; for where shortlists and references come from in the first place, how software buyers actually find vendors; and for what happens when a buyer takes their opinions to a new employer, do marketing tools follow people.

Get a checkup on your tech stack

A checkup for your marketing stack, built from hundreds of verified buyer interviews: what matters most to you, a shortlist for your stack, and your next steps.

Start my checkup

Free · no password · your answers are anonymized before they inform research like this.

Want to come back to this?We'll email you a link to this page, so it's easy to find later or forward to your team.
Methodology. Alium conducts verified interviews with software buyers — the people who select and operate these platforms. Each figure on this page is the average absolute difference between two 1–10 ratings of the same product given in two separate interviews; where a buyer rated one product more than once within an interview, those ratings are averaged first. Colleagues are two different people at the same company; the same person is one buyer interviewed twice at the same company on different days, usually about six months apart; a buyer who rated a product at two different employers is left out of every group. Every pair is counted: where three or more colleagues rated one product, each pairing of them is included, and a buyer appears once for each colleague and each product they both rated. The pairs are therefore not independent of one another, and the figures describe these interviews rather than estimate a margin for all buyers. Each pair counts equally. Weighting each company equally instead gives the same colleague figure, and weighting each product equally narrows the colleague gap somewhat without changing the order of the three groups. The different-companies figure takes, for each product, the average gap across every pair of buyers at different companies who rated it, then weights each product by how many colleague pairs rated it, so the two groups are compared on the same products. In some repeat interviews the interviewer reminded the buyer of their earlier score, which pulls the two answers together, so the same-person figure leaves out every repeat interview in which the interviewer referred to the earlier conversation while asking for ratings. Including them gives about 0.9 rather than 1.0; the colleague figure does not move. Restricting every group to ratings stated in the buyer's own words near the product name gives the same three figures. Products recorded under more than one name are combined. Company size is drawn from the buyer's company record. Buyer identities are verified at interview time and anonymized before publication; vendor names are reported as given. No vendor paid to appear or was able to edit this page.