Ask the teams who run an A/B testing tool what frustrates them and the most common answer is about who can use it: many tests need a developer, an agency or someone who knows CSS and JavaScript, which slows how many tests a team runs. Below it sit results that don't reconcile with the analytics tool, implementations that took quarters rather than weeks, enterprise pricing, AI that is slower to arrive than buyers hoped, and — a complaint buyers often say is not the tool's fault — too little traffic to detect meaningful differences. What differs is which tool draws which. And the response is strikingly passive: none of the buyers here describes a completed move off a named testing tool for one of these complaints. For the ratings and the pre-purchase view, see what experimentation buyers wish they'd known.
The six complaints — graded
Each complaint is classified by how buyers in these interviews typically respond: tolerating the issue, working around it, waiting for the vendor, or reconsidering the contract. These are observed responses, not fixed severity ratings; the same limitation may be manageable for one team and unacceptable for another. Every tool here draws complaints, and the question is which of them matter for your team.
Building a test takes a specialist
The most common complaint, and it lands on the enterprise tools and the developer-led ones alike. On Adobe Target “we always need external developers to build the tests on,” one buyer says, and another that “some of the more complicated tests require some work from somebody with knowledge of CSS or knowledge of JavaScript.” On Optimizely, “it does take someone who's almost an expert or very experienced in this field to be able to set up a test,” and on Eppo “the marketing team is not able to set up the tests, and we have to rely on our tech and engineering team to set up A/B tests in the platform.” Teams staff around it with agencies, contractors or a central testing team — at a cost: “we end up relying on external agencies, and that makes everything more costly and slow.” Self-serve tools are praised for the opposite; one LaunchDarkly user says “We don't need to go to our dev team.”
Named for this: Adobe Target · Optimizely · VWO · Intelligems · AB Tasty · Eppo
Results don't match the analytics
The complaint about reconciling experiment measurements with the rest of the business. Buyers find that testing platforms and their analytics tools can report different results, making an experiment's outcome harder to understand and communicate. Some differences come from measurement definitions or tracking setups; others may reveal implementation or data-quality problems. On Adobe Target the sharpest version is reconciliation: “The data discrepancy between the Adobe Target platform and Adobe Analytics is huge,” says one buyer, and another that “Adobe knows that, but they have not fixed that problem.” An Optimizely user finds “the attribution and measurement is not always exactly aligned with internal data or revenue-based data,” and on Intelligems “the analytics is pretty basic.” Most buyers add a second tool to explain the result — “Statsig alone is not enough,” one says of pairing it with product analytics and session replay — and the Adobe users wait for Adobe to close the gap.
Named for this: Adobe Target · Optimizely · Statsig · Intelligems · GrowthBook
Implementation was slower and harder than sold
The complaint about getting started. One Adobe Target user says “the implementation has been a whole lot more difficult than we expected”; another describes a tool bought months before it was used — “we acquired the tool for many months now, but we are starting to use it just a few months ago.” For a third, “It's taken contractors, a couple of quarters, and heavy engineering expertise to try to get this thing up and working.” Buyers usually describe it in the past tense, sorted out with their own engineers or the vendor: “Those have been addressed by our technical teams through a good partnership with Adobe.” Client-side tools add one technical issue of their own — page flicker while a variant loads — raised by a few Adobe Target and VWO users and by none on Optimizely.
Named for this: Adobe Target · Optimizely · VWO
Cost
Buyers name the price and mostly pay it. On Adobe Target, “It's enterprise pricing again, through the roof, and it's premium”; on Optimizely, “sometimes it can be expensive, particularly for small teams.” Some accept it knowingly — “we were willing to pay the premium for the testing integration,” one Optimizely buyer says — and an Intelligems user calls it “kind of expensive, but nothing overly alarming.” Cost features in the replacement decisions buyers describe, even though these interviews do not establish a completed complaint-driven departure: one company in the process of moving toward AB Tasty weighed feature robustness against price, while another planned a move to Intelligems partly for cost reasons. Others praise their tool's value — one Optimizely user calls it “a good tool that we use at a good price point.” For the mechanics, see what experimentation buyers actually pay.
Named for this: Optimizely · Adobe Target · Intelligems · AB Tasty
AI lags, or works as a black box
The complaint about what comes next. Buyers want AI to make testing faster and personalization smarter, and describe it as limited or opaque: on Adobe Target, “the AI capability is a little bit limited, especially when it comes to personalization,” and the models “can feel a bit of a black box.” Others frame it as a watch-item — “A frustration would be if quickly they don't make it easier to scale using AI,” says one. It can cut the other way: an AB Tasty user steered to its AI module as a workaround finds “it takes a lot more time to develop the right queries for the AI rather than being able to just drag and drop a module.” Buyers describe it as a wish rather than a reason to act.
Named for this: Adobe Target · Optimizely · AB Tasty · Intelligems
Too little traffic to detect meaningful differences
The complaint buyers most often attribute to their own traffic rather than the software. When experiments have too few eligible visitors or conversions, it takes longer to detect a meaningful change with adequate statistical confidence; the sample needed depends on the baseline conversion rate, the size of effect the team wants to detect and the statistical approach used. One Optimizely user says “sometimes it can take a while to get statistical significance”; on Intelligems, “it requires quite a bit of traffic to really get that data sample size up.” An AB Tasty user says “volume of traffic is something that we've struggled with,” and adds that “this is less of an AB Tasty issue.” For some the statistics themselves are the barrier — GrowthBook “can be quite hard to use for people that don't have a lot of knowledge on these mathematical models.” Buyers often live with the constraint, because switching tools does not create additional eligible traffic; teams may instead need longer tests, larger detectable effects or a more selective testing program.
Named for this: Optimizely · AB Tasty · Intelligems · GrowthBook · Adobe Target
Want this checked against your own stack?
Start my checkup →The stories behind the complaints
The counter-current: what testing teams don't complain about
Ease of use draws more praise than complaint for Optimizely and VWO: “it's very easy for me to go in and set up a feature flag or launch an experiment,” says one Optimizely user, and a VWO user calls it “Incredibly easy to use.” Self-serve tools are praised for exactly what Adobe Target is faulted for — “now it's more self-serve,” says a Statsig user. Adobe Target's critics mostly still rate it well and value how it fits the rest of Adobe's stack: “overall, the benefits definitely outweigh the challenges.” Almost nobody complains about support, contract terms, outages or whether their vendor will last. Before you conclude the tool is the problem, separate what the vendor controls — pricing, AI, analytics integration — from what depends on your traffic and your team's skills; the second will follow you to any tool.
What this means — whether you run one or you're buying one
If you already run an A/B testing tool, the common complaints are mostly worked around: buyers bring in agencies or a central team to build tests, add analytics tools to explain results, sort out implementation with engineers or the vendor, and accept that low traffic slows testing. Cost and feature fit enter some active or planned switching decisions, but no completed departure was explicitly attributed to one of these complaints in this interview set. If you're evaluating one, our recommendations — not practices buyers describe — are to have the people who will build tests try the editor on your real pages, to compare the tool's experiment populations, conversion definitions and attribution rules with your analytics setup, and investigate material discrepancies before relying on a reported winner, and to estimate how many tests your traffic can support before paying for capacity you cannot use. For what triggers a move, see why teams switch A/B testing tools.
Common questions
What are the most common complaints about A/B testing tools?
Six recur across Adobe Target, Optimizely, VWO, AB Tasty and others. The most common is that building a test takes a specialist — developers, an agency or someone with CSS and JavaScript skills. Next come results and data that don't reconcile with the analytics tool, implementations that are slower and harder than sold, cost, AI features that lag or work as a black box, and too little traffic to detect meaningful differences quickly. Most buyers live with them or work around them rather than switch.
What are the biggest complaints about Adobe Target?
Adobe Target users in these interviews most often describe tests that need developers or agencies to build, data that does not match Adobe Analytics, implementations that took longer than expected, AI and personalization models that feel like a black box, and enterprise pricing. Many still rate it well overall and value how it fits with the rest of Adobe's tools.
What are the biggest complaints about Optimizely?
Optimizely's experimentation users in these interviews most often raise cost, the expertise needed to set up a test, results that do not line up with internal revenue data, and the time it takes to reach statistical significance on limited traffic. Others praise it as easy to use and fairly priced, and none here complains about page flicker.
Do companies switch A/B testing tools over these complaints?
Rarely, in these interviews. No buyer describes a completed move off a named testing tool for a stated complaint. The few moves buyers do describe — one underway toward AB Tasty and one planned toward Intelligems — weigh price against features, and one company evaluated other vendors over a missing release-management feature. Most buyers staff around the tool, add a second tool for analysis, or wait for the vendor to improve.
Are these complaints reasons to switch A/B testing tools?
Mostly not, in how existing customers respond. Buyers bring in agencies or a central team to build tests, add analytics tools to explain results, tolerate cost, and accept that low traffic slows testing whatever the tool. That is how current customers behave, not a judgment that these limitations are acceptable for a new buyer.
Get a checkup on your tech stack
A physical for your marketing stack, built from hundreds of verified buyer interviews: what matters most to you, a shortlist for your stack, and your next steps.
Start my checkupFree · no password · your answers are anonymized before they inform research like this.