Participating in beta testing offers big cost savings
Published on July 12, 2002
The tech department of Mindbridge CIO/COO Scott Testa had just completed its evaluation of several CRM providers and whittled the favorites down to a short list. Meanwhile, the company's PR director was attending a conference and met with a VP of sales for one of Testa's short-list CRM vendors, who was seeking a client for a beta test environment.
Subsequent conversations between Testa and the vendor resulted in a beta contract that included a nondisclosure agreement-requiring that Mindbridge not release the vendor's name. In a short time, Mindbridge was provided a free copy of the CRM application-which it estimated to be a $400,000 savings.
The two beta approaches
There are essentially two types of beta approaches-engineering betas and betas aimed for sales and marketing venues. Sales and marketing betas typically involve less direct contact with the vendor and are generally used to generate sales leads or customer references for a new product. Mindbridge's was an engineering beta, which is significantly more time- and labor-intensive than one aimed at marketing.
To prepare for the CRM beta test, Testa's tech unit spent nearly $60,000 to set up equipment. Once the CRM software was installed, the Mindbridge beta team met weekly to gather feedback on the product's functionality. The team relayed the information to the CRM vendor, who then implemented some suggestions in the software's final release. Problems or bugs were addressed on a case-by-case basis directly with the vendor's engineering department.
Beta testing requires committed resources
No matter which type of beta test is offered to an enterprise, CIOs need to be certain that they have dedicated staff resources to participate.
Initially, Testa devoted a 40-person team (which eventually grew to 95) to participate in the four-month beta test. The team included salespeople, sales support, administrators, and sales departmental managers. A separate tech team was created to maintain the CRM application that ran on a Windows 2000 server.
Testa said it's also crucial to make sure the vendor provides sufficient resources. "When it comes to any of these installations, you have to [find out] what kind of support comes with the beta," advised Testa. "[Is the vendor] just going to ship the CDs and say, 'You're on your own'?"
He made sure the vendor put a technical beta team in place so that Mindbridge could call on the vendor's team when the beta experience hit hurdles. The effort, Testa said, went a long way toward making the work go smoothly.
But glitches during a beta test are inevitable. Testa recalled an earlier beta of a software development package tested by Mindbridge engineers. He'd pulled engineers from other projects and dedicated equipment to the effort, only to have the software vendor demonstrate incredibly poor response times to beta challenges.
"It was like pulling teeth to get anyone knowledgeable," recalled Testa. "We e-mailed, we made phone calls." Four days later, the vendor finally responded, he said.
That earlier beta experience left such a bad feeling that Testa not only won't ever beta test another product from that vendor, but he will also likely never buy its product.
Negotiate everything ahead of time
Testa says that the response-time issue could have been addressed in prebeta negotiations, which play a key factor in a smooth beta experience. The tech leader advises that companies and the vendor spell out specifics-including expectations for response times, uptime, installation support, and expected outcomes.
Companies need to ensure that they have a strong enough research and development team to participate in the beta, and should demand the same of the software vendor as well, said Theresa Lanowitz, research director of applications development analysis at Gartner Research.
An important aspect of the negotiations involves incentives-such as the free application Mindbridge received for participating-and the incentives will differ from one beta contract to another. For engineering betas, it's not unreasonable to expect a free copy of the software. For sales and marketing betas, some kind of discount or preferential service contract could be negotiated.
Finally, tech leaders should expect to sign a nondisclosure agreement and demand the same from the vendor. During a beta test, a company may reveal sensitive business processes or data that it doesn't want leaked to competitors.
Betas can be a sweet deal for all
Keep in mind that not all beta programs are created equal. Nor do they always run as smoothly as Mindbridge's, even if a tech leader is familiar with the process. To avoid possible pitfalls, it's critical to know exactly what kind of beta is being offered and to ask as many questions as needed.
For Mindbridge and the vendor, the recent beta experience was a mutually beneficial exercise. Not only did Mindbridge get free CRM software, but it also purchased one of the vendor's other products.
"We got some free software, and now we've become a paying customer," Testa said. That's just the kind of win-win outcome you can expect from a well-negotiated and well-executed beta.
xhtml + css