From frustrating interfaces to glitches in functionality that stall production, engineering software problems can sometimes cause users to roll their eyes in dismay. But, say software vendors, beta testing can avoid that.
Through alpha and beta testing, both software developers and users have a chance to please everyone. "One of the most important things we do is have input into the development of the software," says Kim Spinsby, design manager at Siemens Energy & Automation (SEA), and four-time beta tester for UGS PLM Solutions software.
These alpha and beta testers, both new and experienced, provide the developers with raw feedback on potential changes and improvements. And while the software industry obviously benefits from the immediate customer feedback that beta testing provides, the incentive for potential beta testers may seem less apparent.
So why would software users want to take up to two weeks away from work in some cases to test software, often at their own expense? Spinsby sees beta testing as a necessary cost-saver and training opportunity for SEA. "If we can catch something in beta and get it fixed, it might save us hundreds of thousands of dollars," he says. The two weeks he spends in Cincinnati testing UGS PLM Solutions' I-deas NX software are two weeks of vested training in the program. Furthermore, Spinsby believes the benefit of having insider knowledge from working in development helps him give his coworkers an idea of what's coming and what functionality they can plan on for future software releases.
Commited to testing
Many software developers look for an on-site time commitment of one to two weeks, in order to receive immediate feedback. Solid Edge arranges for five days of testing at their development site in Huntsville, AL. "Then we ask that they continue to test subsequent builds as time allows and continue to provide feedback," says Laura Watson, Solid Edge testing manager. And PTC's Mike Campbell, vice president of product management for Pro/ENGINEER, says that PTC seeks a one-week commitment, either at the customer's site or at PTC's site. Matt Rodenkirch, beta test coordinator for UGS PLM Solutions, explains the rationale for the company's general two-week standard commitment. "In beta tests, you lose one day to introductions and one day to interactions, so that's three days of testing in one-week scenarios," he says. "For two weeks, you get eight days of testing rather than three, and testers actually find deeper problems." UGS PLM Solutions actually tests »users simultaneously in four different locations worldwide. So, while Spinsby modeled a design in Cincinnati (home to I-deas NX software), he would then send the parts to Japan, England, or another location to test its overall functionality.
Problems generally center around the software functionality, and how the beta testers find these problems varies as much as the software programs they use. Both PTC and SolidEdge ask testers to exercise the software's new and existing functionality on their own files, doing workflow simulations of the production process. MSC.Software and Autodesk also want users to test the software on their own designs. UGS PLM Solutions asks its engineering users to reverse engineer a design such as a bicycle or gas grill. Any problems or suggestions they come across in design must be forwarded either via e-mail, through "incident reports", or within beta surveys. The software developers then prioritize changes in terms of the time frame and overall impact on users.
Off-site testing, on the other hand, may require as few as a couple of days, or even hours, or as much as six weeks. It generally involves testing the small upgrades made throughout the production process. Rodenkirch explains that UGS PLM Solutions provides new "phases" of the testing software every two weeks for beta testers to download throughout a period of three months. However, he adds, "The issue with off-site testing is that the later in the release cycle, the less likely that any issues the customers have will be addressed."
To each their own test
Functionality aside, software companies are eager to test usability in order to achieve the desired "ease-of-use" description. UGS PLM Solutions brings in customers to their usability testing facility in Cypress, CA, where they study the number of mouse clicks and shifts of the eyes to measure the intuitiveness of the program's interface. Rodenkirch describes the pull-down icon panel as one example of a change instigated by usability testing. The pull-down icon panel takes up less space than listing the series of icons in the interface, as was done in the past.
Other companies feature their own unique style of testing software. »Autodesk has what it calls a "Gunslinger event." " 'Gunslingers' is our term for multiple-day, hands-on meetings where customers and resellers can 'shoot' through the products and provide feedback directly to developers and product managers about what they would like to see next in the software," says Misty Pesek, senior quality assurance manager for Autodesk's Manufacturing Solutions Division. Autodesk has recently expanded the number of locations and invitations to these meetings to get more global feedback.
A few years back, when money was flowing freely, beta testing expenses were the concern of the software company doing the testing, not the customers. UGS PLM's Rodenkirch describes their lavish testing event as both a marketing and engineering investment. "We got their input, found out critical problems, and then built good relationships," he says. "And since we invested heavily into it, we would use it as a selling platform by treating the customers well and showing them what's new in the product." Times have changed though, and more software companies are banking on their customers' realization of the benefits of beta testing, regardless of immediate expenses. Kim Spinsby, the I-deas NX beta tester, says that while UGS PLM Solutions now pays for some of the expenses, beta testing is too important to him and to his company to miss.
SolidWorks invests heavily in beta testing, and it requires no recruiting on its part. "For the release of SolidWorks 2004, we took the traditional beta program and expanded it to ten times the normal number of testers," says Fielder Hiss III, product manager. Ultimately, more than 3,300 participants signed up for the beta program, providing more feedback than the company has ever received for a release. The high number of participants enabled SolidWorks to get the broadest spectrum of users. "We don't look for any particular qualities in a beta tester," says Hiss. "Customers don't use software the way you design it to be used. It's impossible for us to think of every scenario."
PTC's Mike Campbell backs the need for a spectrum of users. "We look for a broad range of customers—large and small companies, in various industries, and with various workflows and design methodologies," he says. Often, the beta testers that work out best are the ones with a vested interest in the software. "I want to find problems before my users find it," says beta tester, Spinsby.
Becoming a bug hunter with early access to key software upgrades is not a right however. Most potential beta testers approach the software companies about being involved in the testing rather than vice-versa, and even then, many of the companies still weed through the potential users to come up with the final roster. In most instances, they choose a combination of experienced users from major companies who have been involved in testing for a while, as well as some new users or new testers.
Spinsby, however, got involved with beta testing when he joined a UGS PLM user group, from where the company often takes its beta testers. "I've been testing for four betas, and it's gotten better on every one of those," he says. "After tasting the problems of not beta testing, you never want those problems again."