Design of Experiments in R&D

Research & development (R&D) can be the ultimate black box. Start with smart people. Add a pile of money. Wait for the next cool technology to pop out. When will it happen? No one can tell. Will it happen? We sure hope so. Executing R&D in a more predictable manner is possible and is a reasonable expectation for companies. Why is this a challenge?

R&D projects have shown a few common themes. First, lots of trial and error disguised as testing. Second, lots of frustration from potential customers. What’s going on? Are you on track? Are we ever going to see a return on our investment? Finally, the smart people are thinking, “Will you just leave me alone and let me work?”

What tools are used in R&D? Computer models are good when the science is well understood. Prototype testing is used when the science is not as well proven. How do we create knowledge with a limited amount of testing during R&D, and make the best decisions to quickly move forward? Only one tool has shown the ability to do that, and that is Design of Experiments (DOE).

What Is Design of Experiments (DOE)?

Simply put, DOE is a method to quickly develop empirical equations for multiple inputs and multiple outputs simultaneously. It is statistically based and best when practically applied. Do not let pure stats people set them up. It can waste tons of money and materials. Equally, do not let smart people just tinker and test either. Lack of structure implies no predictability and no end date. Instead, leverage the knowledge of your team and use the structure of DOE to maximize the learning.

Is DOE Just for Manufacturing?

DOE use exploded during the 1970s in the automotive industry. As a problem-solving tool, it could be used by anyone when certain assumptions are made. This provided a view of DOE as a manufacturing tool. However, the early DOE applications were in agriculture and similar complex situations, which is really like what is faced in R&D environments. Bottom line, DOE is applicable in many areas including design, manufacturing, and marketing.

What If I Don’t Think I Need DOE?

If you do not need it, then you should have a proven answer to your design problem within a short time. Longer than a week may indicate the situation is not as well understood as first believed. If trial and error is continued when progress is stuck, then you really should be using DOE instead. Equally if your process has not been proven and is not working right away, you should also be using DOE. If you can guess and you guess right, then go for it. Adding structure within this flexible approach (control with freedom) is actually much faster and cost efficient than anything else I have seen in R&D.

Ok – I Need It. How Do I Do It?

There are a couple of key differences between simple problem solving and R&D application. First, solve the right problem. It needs to be scoped properly. Not too broad, but also not too narrow. And it needs to be defined for something before we have failed a test. Sometimes we don’t even have a clear and stable set of requirements. Taking time to define the test well is important. In problem solving, we can quickly say, “We failed the customer’s acceptance test, so we need to get that fixed.” Things are well-defined, and the scope is relatively straightforward. During R&D, we need to decide on everything to define a test.

Next, we need to think about how we measure our performance. It is easy to record pass/fail data or something directly related to our specification. The problem is our specification can be at a high level, which requires a fully assembled product to measure. Instead, we may need surrogate measures at a lower level. We also need variables data so we can sense progress and understand a direction toward “better” operation. Using discrete or pass/fail data requires much more testing to gain a similar level of confidence compared to variables data.

The final key is brainstorming all potential sources for input variables. Not that all will be changed in the DOE, but it helps to know where we are drawing the boundaries of our problem. Typically, either things are scoped too narrow or only directed at a specific vendor. List everything first, then prioritize. Variables not in the DOE can be controlled in the test protocol.

These three steps may sound simple, but experience has shown they are often avoided. Why? The smart people already know the answer. Management wants the answer yesterday. Realistically, it can be hard to think about these “what if” situations. A style of asking simple questions first may appear silly, until the team tries to answer them out loud. This approach tends to expose the true problem. Other times, it helps us scope things so our first test has the answer “surrounded” and will yield a confident direction.

Three Questions Before You Test

Before someone does a DOE, there are three questions they should be able to answer. If they are not answered, it could lead to test issues and less learning from the available data.

What Is an Expected Level of Use?

What would be considered an active level for DOE use? Each person performing one per quarter minimum is a reasonable expectation. How is this possible?

Regardless of R&D role, it is easy to see the wide variety of testing possible. This does not mean every test is needed, but when considering risk, it is easy to find areas that would benefit from a DOE test as described above. It would also create a confidence that characterization testing has been complete enough to move forward.

Two R&D DOE Case Studies

In the first case, our first meeting talked about a design failure of an existing product. They were being returned because they were not working during early use. The desire was to determine what changed, and how to resolve the customer requirement. Most importantly, we needed to stop the expense of high warranty costs. At one point I asked, “So how are they using it in the field?” The immediate answer was, “We did not know.” The eventual answer was, “It was not even close to how we imagined it.” No wonder it was failing far earlier than expected! It was being used with a much higher duty cycle. Thus, a complete redesign was needed to create the significant improvement needed for the application. A DOE could have made a step improvement, but not on the order required for this case. No DOE was done because we were not even talking about the real problem at the initial meeting. Being structured in our approach allowed the right questions to be asked, thus exposing the situation.

In the second case, a new technology product had been in development for several years. As this technology was applied to an existing product, it was creating several design and manufacturing challenges. We first used DOE to understand our measurement system. We needed a quick yet representative approach to understand performance in use. Then, we used DOE to understand our design parameters using the new test. Which parameters were key, and where should they be set for a robust design solution. Prior efforts tightened all tolerances to improve control, however some of these items were not key variables. Opening up the design where feasible allowed cost to be reduced, without impacting performance in our new measurement system. This modified design simplified our manufacturing process, which previously was forced to assume everything was “critical.” The resultant manufacturing process still had one complex operation. This became our final stage of DOE application. These phases of test took place over a few months and put the team in position for immediate success in the final validation testing. After launch, field acceptance was high without unnecessary costs built into the product.

Conclusion

In 2001, we presented “DOE as a Tool in the Development Process” at a conference. Using DOE in R&D was not being done on the design side or on the process side based on a survey of attendees. Additionally, only 30% were aware of DOE. Over the years, people have slowly been adding DOE as a tool within several industries. Many of the applications fall short in one of two ways. First, they claim a DOE but the truth is they simply planned a test. They did not use the DOE structure. Second, they define the problem poorly. This either proves something that is already obvious or confuses the situation worse than it was before. In both cases, people would do better if they could. They simply have not been trained or mentored on how to do this.

I am fortunate to have been mentored by experts at the General Motors Research Labs. I was able to understand how they thought, but then also to see the full power of DOE in practice. Translating our paint process lab work into an automotive assembly plant without start up issues was unheard of. Gaining the appropriate understanding of the right problem made development much quicker and easier. It is time for industry to fully embrace the power of DOE in the R&D process, gaining efficiency, predictability, and higher performance.