What happens after a company evaluates software?
Alium interviews some companies more than once. This page sets the evaluations buyers described at one interview, whether talking to vendors, comparing options, piloting or making the business case, against the same company's next interview, about ten months later on median. It covers evaluations active at the first interview where the follow-up said enough to classify what had happened since; evaluations the second interview did not address are left out. About half, 53%, had reached a new-tool outcome: an option named at the first interview was live or being implemented in 37% of cases, one had been selected but not started in 3%, and a product not named at first had been selected, was being implemented or was in use in 12%, a few of them in-house builds. Another 23% were still being evaluated, 22% still had the incumbent running with no active evaluation reported, and 2% had shelved a project for a capability they did not yet have. These are statuses at the next available interview, not final outcomes: a selected product may never go live, and an open search may close later. Nor are they vendor win rates, migration success rates or the share of all evaluations that end in a purchase.
Status of evaluations at the next interview with the same company, about ten months later on median. The three left-hand segments are evaluations where a new product had been selected, was being implemented or was in use, not all completed implementations. Each company contributes at most one evaluation per category of software, evaluations without enough follow-up information are left out, and shares are rounded. The stage and respondent rows are cuts of the same followed evaluations, not separate samples: the two stage rows and the two interviewee rows split them two ways, and the few already decided at the first interview sit in neither stage row.
The six outcomes
Each evaluation lands in exactly one of six stages at the second interview, taken in the order below: the first that fits is the one recorded. They are what the two interviews say, not a record of contracts.
| Outcome | What it means here | Share |
|---|---|---|
| Live or implementing | A tool the buyer named as an option at the first interview is in use, or the migration onto it is under way. | 37% |
| Selected, not live | An option named at the first interview has been chosen or signed, and implementation has not started. | 3% |
| Off the shortlist | The company chose, is implementing or runs a tool that was not among the options named at the first interview, or built its own. In-house builds are a few of these. | 12% |
| Still evaluating | The buyer says the search is still open: still looking, an RFP under way, a pilot running with no decision, or waiting on budget or approval. | 23% |
| Kept the incumbent | The company's existing tool was still in operation at the follow-up, and no replacement was reported as selected, being implemented or live. Some buyers said outright they had decided against replacing it; in other cases continued use was observed without confirmation that the evaluation had ended. | 22% |
| Dropped | There was no existing tool, and the need has been shelved or is no longer wanted. | 2% |
Two distinctions matter here. Evaluating counted only when a buyer actively described considering software at the first interview; a passing interest or a product only the interviewer raised did not count. And kept the incumbent does not always mean an affirmative decision to stop: in about six in ten such cases the follow-up showed the old software still running but did not mention the evaluation, which may include searches still under way or decisions the respondent did not know about. The category describes the recorded status at follow-up more reliably than a final retention decision. None of these shares is a vendor win rate, a churn rate or a final outcome.
Does the shortlist pick the winner?
Mostly. Most new-tool outcomes were a product already named at the first interview; about a quarter were absent from the options recorded then. That is a comparison with the options named at the first interview, not necessarily the company's full formal shortlist: a product may have been considered internally without being mentioned, or entered the search later. It shows the first interview did not capture every eventual destination, not that every unmentioned product was a late entrant.
The options named early are less predictive. When the first interview caught the evaluation at the research stage, with interest and a few names but no substantive vendor talks, about 18% later produced a tool not named then, and 29% still had the incumbent running at follow-up. Where the buyer was already talking to vendors, comparing named options or piloting, those figures were about 7% and 16%, and the share with a named option live or being implemented rose from 28% to 44%. The share reaching any new-tool outcome was about the same in both groups, at around half. That is consistent with an early list being less settled, but it may also reflect how completely buyers described their alternatives, and it does not show that a formal shortlist was overturned each time.
Two buyer histories illustrate how a company can end up choosing a product it didn't name in its earlier interview. A mid-sized services company's digital and technology chief built a shortlist of personalization vendors from research, sent out a capabilities list, and named VWO as the favorite. By the next interview the company had switched off a personalization module in its content platform to save hosting cost, “and we put in place of it AB Tasty, an A/B test tool with some personalization capabilities.” A global footwear brand had spent two years in conversations with Dynamic Yield, including “some kind of piloting, proof of concept, demos, but leadership is hesitant to pull the trigger.” At the next interview, a different person explained that the brand needed “another tool for more localized sorting, site merchandising, and recommendation. That's why we moved to Algolia.”
Pilots go with more advanced outcomes. Evaluations already in a pilot at the first interview more often had a named option live or being implemented at follow-up: about seven in ten, against fewer than four in ten for evaluations with vendor conversations but no pilot. That is an association with the stage at the first interview, not a measured effect of piloting: companies that pilot may already have stronger intent, clearer requirements, funding or approval. And the footwear brand above shows that extensive testing does not guarantee the company picks the vendor it piloted.
What keeps an evaluation open?
About a quarter of evaluations were still open at the next interview. The median interval was about ten months across the whole analysis and about nine among the evaluations still open. When buyers say why, the reason is more often the company's own calendar than the vendors. The search has been queued behind another project, the budget or the approval has not arrived, or the company is waiting for the incumbent's contract to end. An ecommerce replatform is the project other evaluations wait for more than once.
A small footwear brand had held calls with Smile.io and called it the front-runner for a loyalty program. Six months later the program was still unbought: “So we're still in the market looking for loyalty solutions. Like I know I said Smile.io, but I'm still open to looking for others.” The reason was sequencing, not the vendor: “We really wanted to get the loyalty program, but that's shifted because of the whole product recommendations.”
Evaluations observed after longer intervals had more often progressed. Where the second interview came more than a year after the first, about 46% had a named option live or being implemented and 14% were still open; within a year, those figures were 34% and 27%. These are different groups of evaluations with different intervals, not the same evaluations watched over time, so they do not show a standard buying-cycle length, and time alone is not shown to explain the difference.
When does a company keep the tool it has?
Among evaluations to replace an existing product, about a third were classified at follow-up as incumbent still in use with no confirmed new tool. Some of those buyers said they had chosen to keep it or that the move was called off; others were inferred from continued use. So the figure is not the share of replacements definitively abandoned in favor of the existing tool. Evaluations for a capability the company did not have yet reached a new-tool outcome more often, about 65% of the time against 48% for replacements, and they are almost the only place a project is dropped outright. Replacing a tool means beating a system that already works, a migration, and a contract with a date on it.
When buyers explain staying, the reasons are mostly about the move rather than the alternative: the engineering time a migration would take, the revenue work it would displace, a budget that would rather keep a cheap tool, an incumbent that improved in the meantime, or a renewal that arrived before a decision did. Sometimes the decision is made above the evaluator.
The clearest case is a DTC fashion brand's retention director, who had been meeting Iterable as a replacement for the company's email platform at the first interview. At the second: “Yeah, we actually almost signed with Iterable last year. Had the contract, had a better price, had the whole thing. The leadership team decided not to sign it at the last moment.” The company was still on its incumbent platform, and the buyer still wanted to replace it.
Why does the answer depend on who you ask?
The reported outcome differs a lot by whether the same person took part both times. When the same person was interviewed twice, about 30% of evaluations were still open and 17% had the incumbent still running; when a different person at the company answered, it was 15% and 28%. The share reaching a new-tool outcome was similar in both, and the contrast between open and incumbent held when only follow-ups within a year of the first interview are compared. That is consistent with respondent continuity affecting how status gets reported; it does not show that the choice of respondent caused the difference, or that one account is always more accurate.
One possible explanation, ours rather than something buyers state, is that a new respondent may know what software is running without knowing whether an earlier evaluation is still live. Another is that different people define an active evaluation differently, or see different teams' decisions. A different respondent also left the original evaluation unaddressed about a quarter of the time, against about 6% for the same person; those cases are excluded, which may affect how the two groups are made up. So treat the 22% with the incumbent still running with caution: some of it may be unresolved evaluations nobody described as active, especially when the respondent changed. The interviews do not show how many were misclassified or which group is closer to the real status, and the same-person figure of 17% is a useful comparison, not a verified correction.
Does company size change the outcome?
Little observed difference. About half of evaluations had produced a new tool at every size of company. The largest companies show slightly more evaluations with the incumbent still running and slightly fewer still open, but the gaps are within what these sample sizes would produce by chance.
| Company size | Produced a new tool | Still evaluating | Kept the incumbent |
|---|---|---|---|
| Under 1,000 employees | 53% | 25% | 20% |
| 1,000 to 9,999 | 53% | 23% | 21% |
| 10,000 or more | 53% | 18% | 28% |
The counter-current: going live is not the end of the decision
A new-tool outcome does not establish implementation success or lasting satisfaction. A specialty outdoor retailer's marketing head named a session-recording tool as one of two platforms to bring on. By the next interview, within a year, it was in place, and the buyer was weighing cutting or downgrading it because it was delivering too little value for what it cost: “That's honestly one that we're looking at just cutting from the stack.” In the buyer's account, much of the cost came from the setup: a former team member had pushed to record every session, and the buyer now calls it “just an underutilized tool for us.” The outcomes on this page record reported decision and implementation status, not sustained adoption, return on investment or retention.
Want a review of your own stack?
Get my review →What this means if you are running an evaluation
These are our recommendations: implications of the histories above, not a tested process for raising purchase or implementation success. The follow-ups show how decisions evolve, not which practices cause a purchase or whether the tool meets the company's needs. Five questions to settle:
- Was the candidate list assembled before substantive evaluation began? Early interviews less consistently named the eventual tool. Revisit the list as requirements firm up, check whether other teams have relevant experience, and document why options are added or dropped. The interviews do not show that every product absent from an early list was found later.
- What would keeping the incumbent mean? For a replacement, weigh the benefits of changing against migration effort, contract timing, training, implementation risk and the cost of staying. About a third of replacement evaluations here still had the incumbent in use with no confirmed replacement at follow-up, which does not necessarily mean the company had stopped evaluating. Treat keeping it as an option to assess, not a failed evaluation.
- Who must approve before signing? Identify the budget, technical, procurement and final-approval stakeholders, and confirm their requirements before final negotiation. One buyer above reached the contract stage, with a better price, before leadership declined, which shows that the evaluator's preference and the organization's decision can differ.
- What evidence must a pilot produce before the company decides? Define the capabilities, performance, implementation effort and commercial terms to test, who assesses the results and when the decision will be reviewed. The footwear brand above had piloted and held demos with one vendor across two years. Pilots here were associated with more advanced outcomes, but the research does not show that piloting causes a purchase or that a deadline prevents delays.
- Can the evaluation's status be verified across stakeholders? If responsibility has changed hands, confirm the original objectives, the options named, the decisions made and any pending approvals with the previous evaluator or team. Continued use of the incumbent does not, on its own, show the search ended.
What not to read into this
The shares are not a close rate for any vendor, not the share of all evaluations in the market, and not a churn rate for any category. They describe evaluations Alium caught in one interview and could follow to the same company's next one, excluding the ones the second interview did not address. Still evaluating is not a failure: many of those evaluations may close after the second interview. The stage of an evaluation is what the buyer described at the first interview, not an audit of the process. Kept the incumbent includes explicit decisions and cases inferred from continued use; because follow-up respondents differed in what they knew, it may include unresolved searches, so do not read its percentage as a confirmed retention rate. And the pilot, stage and incumbent comparisons are associations between groups of companies, not effects: they do not show that piloting or contacting vendors earlier would change any one company's outcome.
Common questions
What usually happens after a company evaluates new software?
In Alium's interviews, about half of evaluations had reached a new-tool outcome by the same company's next interview, about ten months later on median: an option named at the first interview live or being implemented in about 37% of cases, one chosen but not yet started in 3%, and a product not named at first in 12%, a few of them in-house builds. Another 23% were still being evaluated, 22% still had the incumbent running with no active evaluation reported, and 2% had been shelved. These are statuses at the next interview, not final outcomes, and they cover evaluations the second interview addressed, not every evaluation a company starts.
How often do companies buy a tool that wasn't on their shortlist?
About one in four of the new tools chosen in these evaluations was not among the options the buyer named at the first interview, a few of them in-house builds. That is a comparison with the options named then, not necessarily a full formal shortlist. It happened most when the first interview caught the evaluation early, at the stage of interest and research: about 18% of those evaluations later produced a tool not named then, against about 7% where the buyer was already talking to vendors or running a pilot.
How often does a software evaluation end with the company keeping its current tool?
Where an existing tool did the job, about a third of evaluations to replace it still had that tool in use, with no confirmed replacement, at the next interview. Across all evaluations it was about one in five. Most of those cases were inferred rather than stated: the old tool was still running and the evaluation was not mentioned again, so the share is a recorded status, not a confirmed retention rate. When a different person answered the second time, incumbent outcomes were more common and still-open ones less common, so some of that share may be evaluations a new respondent did not know were still open.
Does running a pilot make a software purchase more likely?
Evaluations that were already in a pilot at the first interview had a named option live or being implemented about seven times in ten at the next interview, against fewer than four in ten for evaluations that were talking to vendors without a pilot. That is an association, not proof that pilots cause purchases: a company that agrees to a pilot may already have stronger intent, clearer requirements, funding or approval. A pilot can also stall: one footwear brand piloted and held demos with a personalization vendor over two years and then moved to a different vendor.
Do larger companies take longer to finish a software evaluation?
In these interviews there was little observed difference by company size. About half of evaluations had produced a new tool by the next interview at companies under 1,000 employees, between 1,000 and 10,000, and over 10,000. The largest companies showed slightly more evaluations with the existing tool still in use and slightly fewer still open, but the gap is small enough to be noise at these sample sizes.
Related reading: for where a shortlist comes from, see how software buyers find vendors; for when owners start looking at all, at what rating shopping begins; for what a first purchase asks that a replacement does not, replacing or buying first; and for the tools that stay despite poor ratings, why buyers keep software they rate poorly.
A tech stack review, powered by the buyers who've been there
What matters most to you, a shortlist for your stack, and the next steps to take — built from hundreds of verified buyer interviews.
Get my stack reviewFree · no password · your answers are anonymized before they inform research like this.