Rough Guide to Outsourcing in China

An engineer's highly practical and entertaining account of two weeks he spent in China ironing out issues with a subcontractor, scouring electronics markets, dealing with schedule slips, and much, much, more

Al Mudrow, Contributing Writer -- Design News, September 24, 2006

WiLife's Outsourcing Project

WiLife develops and markets an innovative line of consumer digital surveillance cameras that use power-line networking to stream video to a PC. We successfully introduced our first product in November of 2005 after nearly a year's worth of product and manufacturing engineering efforts. The strategy was to manufacture our products domestically initially until volumes dictated going off-shore. After receiving several quotes from Chinese manufacturers, it was evident that even at modest production volumes of two to three thousand units per month the move would be justified.

Prior to the trip I describe in this article, WiLife had produced engineering samples of the surveillance camera. In WiLife's terminology, engineering samples use, for the most part, hand-assembled printed circuit assemblies and prototype plastic. These samples had gone through a series of engineering tests including voltage range and power line impairment tests, temperature tests and emissions tests. Our contract manufacturer (CM) in China was provided assembly drawings, printed circuit board artwork files, a BOM, plastic mold files and instructions for manufacturing test. CM's responsibilities were to source the components and set up the production line so the production run could be observed and evaluated by WiLife personnel.

In the days leading up to the trip, there was a barrage of e-mails back and forth between CM and WiLife, resolving issues with the BOM, component shortages and clarification of procedures. In addition to observing and evaluating the EV production build, VP of Manufacturing John Chase and I planned on visiting several other vendors on this trip, some to resolve outstanding issues with the earlier product and others to evaluate their capabilities.

Sunday 3/26/2006

We arrive at Guangzhou Airport at 6:00 a.m. and are met by one of CM's drivers. He shows me an English text message on his mobile phone indicating that John and I are to be driven to CM's factory to work until we can check into our hotel.
Upon arrival at CM's factory in Dongguan, we are directed to the demo room where we check e-mail. The CM team members will arrive in a few days from Taiwan so John and I plan to meet with suppliers until the full Taiwanese team has assembled in China.

After a few hours, Summer, the plant manager, meets with us. He is part of the permanent Taiwanese staff who reside full-time here. John and I give Summer the parts we brought with us. He passes them to the production setup staff for preparation of the EV run.

Summer invites us to eat lunch with his plant managers. Over lunch, John and I comment on the fact that WiLife had worked with several project managers in the past year. The CM people respond that most have "graduated" to Taiwanese system companies — indicative of the changing dynamics of the Chinese market.
Later in the afternoon we check into the Dongguan Sheraton Hotel near the factory.

Monday 3/27/2006

John and I discuss our plans to visit two suppliers. The first, Camstar, is relatively unknown to us. I came across them on the Internet and have exchanged a few e-mails with their salesperson, Anton. Camstar claims to be the leading analog surveillance camera producer in China. We are interested in them to see if they have any synergistic elements that we might benefit from. WiLife's cameras are all digital and networked, so at this point there is little competitive overlap, but Camstar might produce accessories or mechanisms of interest. We know the second company, RDI, and have already specified some of their components. Our motive for meeting with them is to solidify our business relationship, access their capabilities, and ascertain if they have any components in development of interest to us.

Developing business relationships in China are much more personal then in the U.S., where it is quite common to conduct significant amounts of business without physically meeting your suppliers. Chinese prefer to develop personal relationships. This preference means it usually takes longer to develop a strong business contact but it also means they are more apt to act as a true partner.

The Camstar salesperson, Anton, meets us at the hotel and we drive to his factory in Shenzhen. We are greeted by the bleak sameness typical of many medium-size factories here. We talked about what we do and what types of components and accessories we need. At first he expresses somewhat of a disconnect — WiLife markets digital cameras and Camstar makes analog cameras. Finally, he sees the bigger picture and starts bringing out samples of enclosures, pan and tilt mechanisms and mounting brackets. We find several of interest and negotiate for samples. The low-cost items are given to us, but he asks for payment for a higher-end motorized pan and tilt mechanism. We pay in cash on the spot.

We then take a somewhat disappointing factory tour. It seems to be arranged as more of a job shop than a production line. Quality does not seem to be a priority. Their claim of being China's largest surveillance manufacturer seems to be a stretch.

After visiting Camstar, we take a taxi to RDI. Rather than bleak surroundings, however, the RDI factory is bright and well-landscaped. We meet their sales person, Wiley, and are taken to their demo room. RDI also manufactures consumer surveillance cameras but seems to focus more on the baby monitor market. Wiley shows us several innovative monitors complete with a small RF linked LCD television display. Since we already have established a business relationship, we spend our time discussing more ways to do business with each other. We then take a factory tour and are impressed with its cleanliness and organization. They seem more oriented toward international OEMs.

We conclude our discussions by giving Wiley a purchase order for two types of RDI's illuminators and drive back to the hotel.

Tuesday 3/28/2006

We start out at CM's factory to observe the surface mount portion of an indoor camera build currently going through production. By this time, CM has enough experience with the product that we anticipate no issues. I spend a few hours giving a detailed inspection of several assembled PCAs. Everything looks great.After lunch, we travel to Lucky Light Electronics, an LED fabricator I came across on the Internet. They looked like an interesting source because their specified efficiency was three times that of the best U.S. supplier and their cost only half. We meet Freya, Lucky Light's salesperson, and receive a cool reception. It seems Freya is annoyed with us for having her meet us at the factory instead of the more comfortable sales office in Shenzhen. But, the purpose of our visit is to gain confidence with their operations and quality. We discuss the specifications with Freya and tour the factory. Lucky Light assembles LEDs using third party LED chips. Their factory is clean and well-run, their prices are good, so their supply chain is of lower consequence. Freya had some samples for me to test, but sent them to my office in the U.S. She promised additional samples will be sent to CM by Saturday.

Wednesday 3/29/2006

Today we travel to our second contract manufacturer (CM2), the source of USB/powerline bridges we use with our cameras. We spend the entire day here working on a production issue. We have received some of their bridges programmed with an invalid configuration.

CM2 does not manufacture these components — outsourcing them to another company. CM2 has a crew of four to five people in China who overview the manufacturing process on its behalf.

Thursday 3/30/2006

John and I are back at CM's factory. The Taiwanese project manager May, and her staff, have arrived. We review the BOM for the new camera to verify all of the parts were ready for the production run. We also review the production line test procedures, the system test procedures and CM's QA processes. During this review, we receive some bad news. First, CM has misplaced the pre-programmed EEPROMs I brought from the U.S. — a problem because now we need to find a local source for the parts and program them at the factory. The second bit of news is more troubling, though. CM has failed to give the design files for the plastic enclosure to their plastic sub contractor (PSC). We hadn't planned on using these parts for this particular test build, but it pushes back the schedule for the next test run. I wish that CM had informed us of this earlier. I tell CM to give the design files to PSC immediately.

To source the missing EEPROMs, we drive to the Dongguan Electronics Market, a large, multi-story building filled with small vendors, each focusing on a limited range of components. You simply walk around until you find someone who specializes in what you want. They either have the part on location, at a nearby location, or, as in our case, at their affiliate at the Shenzhen Electronics Market. We leave empty-handed, but with a lead on where we can find the parts in Shenzhen. We send James, a Taiwan EE to Shenzhen to retrieve the parts.

Friday 3/31/2006

This morning we review the outdoor camera schedule in detail. According to the revised schedule, we should have the first plastic for review from the vendor by May 13.

CM seems to be having problems programming the EEPROMs we bought yesterday, claiming the TSSOP package is too small to be reliably programmed. They ask us to redesign the PCB to accommodate a larger SOIC package. I ask to see their procedure because we had no problem programming the parts in the U.S. After observing their procedure, I understand their difficulty. Rather than using a standard programming socket, they use what is essentially a bare board with exposed pads for the electrical connection. The operator holds the part against the pads with her finger during the programming process.

With a TSSOP package, any minute movement would cause the programming process to fail. I recommend they get a TSSOP head for their programmer. A few hours later they come up with one (another trip to Shenzhen). They are still having problems programming the parts because the setup instructions are for a different brand of programmer. I try to help them, but the user interface is a combination of Chinese and cryptic English menu items like "Motorola way." I assume this has something to do with big endian/little endian but I am not sure which one they need to use. After an hour of tweaking with the user interface, we set it up how we think is best, but, in the end the parts were misprogrammed anyway.

Saturday 4/1/2006

The first thing this morning, I go to the factory to make sure the reprogramming process is working. It seems to be progressing, but at a very slow pace. We had expected a camera cycle time of about 20 seconds, but it's taking about one minute per camera. I discuss an improvement plan with the production line manager. We are able to improve the cycle time to about 30 seconds.

This afternoon we meet with PSC, the plastic vendor. PSC informs us they need a test fixture to verify the optical quality of the lens cover. I tell them I will create the fixture and bring it to them on my next trip to China.

While meeting with PSC, the rubber gasket vendor shows up. They give me several samples and data sheets. I select the material that will meet our specifications and describe our requirements. They seem very anxious to work with us. I am very impressed with their technical knowledge and understanding. I give them our design files and they promise samples in about a week.

Sunday 4/2/2006

Today we take a break and go with several factory people on an excursion to "Waterfall Mountain," located about six miles east of the factory. Later, we shop for DVDs.

Monday 4/3/2006

Today John meets with the suppliers of our wall mount bracket, and I travel to our Ethernet cable vendor, "Yes Cable" to review their test process. On my last trip we had talked to them about a manufacturing issue we had detected with some of their cables. The cables did not appear to have any open or shorted connections, but the cables were failing our tests. In that meeting, we discovered that they did not perform a high-frequency test on the cable, only a dc test. We proposed a more complete test process which they agreed to implement.

They tell me the proposed test process was not implemented because the test has failed. Digging into the cause, I discover their tester has an MDI/MDIX problem. I propose they use a hub to connect the cable being tested between the two PCs of their test setup. They propose downloading a file from their test server to test the cable. I convinced them that TCP automatic retransmissions would mask potential problems. After poking around a bit on the Internet I find a "raw" cable test program they will use instead.

CM calls and reports they are having problems with the camera test software we use on the product line. I get the information on what is not working and send it in an e-mail to our test programmer in the U.S. to see if he can duplicate the issue and provide a fix.

I return to CM and begin evaluating the first six test units of the new outdoor camera. This is a stressful time for the whole team. With about 12 people looking over my shoulder I connect each camera to the test fixture and start the test program. All six cameras fail the test. I ask for a volt meter to see if the voltages are at least correct. I find that on all cameras, the 1.2V supply is only putting out 0.6V. It is late and the factory is shutting down for the night so I have to wait until tomorrow to find out what's going on.

Tuesday 4/4/2006

This morning I go to the factory to determine why the new outdoor camera PCAs are not working. After a few minutes I discover the 1.2V regulator is the wrong part. This is one of the parts we hand-carried from the U.S. The correct part was listed on the BOM, but somehow the wrong part was ordered. Checking the PCA in greater detail, I discovered that two changes we noted on the engineering samples had not been implemented on the new units. This is an easy problem to fix.

I also notice issues with hand-soldered components. WiLife and CM had elected to construct this PCA with a lead-free process. Several of the parts were not lead free at the time of the test run so they had to be added manually. These particular parts are extremely difficult to solder by hand so I was not surprised to detect several shorts and opens on their leads.

I receive updated camera test software for the camera from the U.S. I test it on my laptop before installing it on the production line test machines.

Wednesday 4/5/2006
We go to the Shenzhen Electronics Market to find the correct 1.2V regulators. After an hour's search, we locate a vendor who will sell us 50 units at a reasonable price. We wait another half-hour for the parts to arrive and consummate the transaction. Then we wander around the floors a bit.
We then return to the factory and spend the rest of the evening replacing the regulators.

Thursday 4/6/2006

Today we continue troubleshooting the new outdoor camera PCAs. We receive 20 more PCAs from manufacturing. As I dig into the details, I find a few more manufacturing errors. The Taiwan team is getting anxious. They are scheduled to fly to Taiwan early tomorrow morning.

Tonight we hold the "outdoor camera production evaluation meeting." I begin by giving my report. The cameras aren’t passing the tests, but I believe the issues are production-related, not engineering-related. The topic then turns to the plastic enclosure. The files are being held up by CM's mechanical design team in Taiwan for evaluation. They have proposed a redesign they would make that would add an additional two weeks to the schedule. I inform the project manager this is absolutely unacceptable. Our design has been tested and the industrial design approved by WiLife management.

At this point the project manager loses control and begins screaming at us in Chinese. The intended effect is to intimidate CM's team into backing her position. She's probably not aware that I have been taking Mandarin language courses and am able to understand much of what she is saying.

The meeting ends and I tell the CM team that I plan to return to the factory tomorrow and Friday to continue testing the cameras. They ask who will accompany me as they are all returning to Taiwan early tomorrow morning. I respond that I know my way around the factory and if I need something I can ask one of the Chinese production engineers who I've worked with in the past. They are incredulous that I would make such a request after such a rancorous meeting, but since my flight isn't until Sunday, I think I might as well do something useful until I leave.

The Taiwanese are loath to let their customers interact directly with the factory staff. If customers become comfortable dealing directly with mainland Chinese, of what value then are the Taiwanese? They quickly huddle and one of the Taiwanese people, Jimmy, volunteers to remain in China until I leave.

Friday 4/7/2006

Today I spend the entire day working on debugging the camera PCAs. I am heartened later in the day by getting the camera to work. This confirms my suspicion that there is not a design problem in the camera.

In CM's defense, this is the most complex PCA they have manufactured to date. It pushes the limits of their capabilities. The main PCA contains three BGAs and several high pin density surface mount devices. The bulk of CM's output is fairly simple mice and game controllers. CM top management wants to work with WiLife because it forces their factory to enhance their capabilities.

Saturday 4/8/2006

Knowing the design is correct, I perform a quick check on the remaining units. I check each unit and record its ability to generate dc power, clocks, proper reset, access the Flash ROM, network across the power-line and generate a video stream. Of the working units, I leave two with Jimmy and keep the rest to hand-carry back to the U.S. The non-functional units we package and express ship back to WiLife. I thank Jimmy for staying and helping me get the cameras working.

Sunday 4/9/2006

Heading home at last!

WiLife Confronts a QA Breakdown The scenario below illustrates some of the difficulties of making sure Chinese manufactured products have good quality. Chinese manufacturers truly want to achieve world-class quality standards. They seem more devoted, though, to following assumed standard procedures rather than having a focus on an end-user quality experience. In this example, if one of the quality engineers had stood up and said "something is not right here," the issue could have been easily resolved. A subcontractor to CM2 uses a properly configured WiLife Production Test System (PTS) to program the WiLife bridges. This system works as designed. After programming, CM2 operators ran the systems through an additional "throughput" test. If this test did not pass, the operator is given the option to reset the configuration to the default, non-WiLife configuration. The configuration supplier QA engineers perform a sampling test process on units before they are delivered to CM2. Unfortunately, instead of using the WiLife PTS "QA mode" as specified in WiLife's production specification, they elect to use "production mode." QA mode checks the bridge and would have noticed the problem. Production mode reprograms the bridge, setting the configuration back to the WiLife standard. CM2 QA would then re-run the throughput test which would set the configuration back to a non-WiLife configuration. CM2 QA engineers perform a similar sampled QA test on a separate PTS system located in their office. When they performed the sample test, running the PTS in "QA mode" as specified, all of the samples failed, as expected. Rather than flagging this as a production issue, the CM2 engineers made the assumption that the problem was with the PTS system, not the units under test. They returned the units to the factory and tested the units using CM2's flawed QA procedure, which passed the units. CM2 then dismantled their PTS test station and tried replacing the motherboard and other components in an attempt to "repair" it. Of course, they were unsuccessful because there was nothing wrong with the system in the first place. CM QA engineers perform a system-level sampled QA test on all of the components in a starter kit. This test performs a complete installation of the system using the components inside the box and verifies functionality of all components of the unit under test. CM short-cutted this procedure to test only the components they manufacture, not the complete system. Al Mudrow’s tips on How to Search for and Select a Supplier in China Most of the business activity in China revolves around individual trust networks based on personal contacts. The best starting point for finding a new Chinese supplier is with your existing contacts. These contacts can be other suppliers, customers, personal relationships, school acquaintances, etc. For example, one of WiLife’s customers, Radio Shack, gave us a list of contract manufacturers which they have had good relations in the past. Additional contacts may come from companies further up your supply chain, i.e. the suppliers of your suppliers. A printed circuit board supplier of one of WiLife’s contract manufacturers introduced us to other contract manufacturers to whom they supply. Don’t overlook non-obvious sources for contacts. A neighbor of mine produces sports drink bottles in China. He asked me to find additional contacts for him through my contact list. If you’re going to China, talk to the person sitting next to you on the airplane. Chances are, they can give you some starting points. They key point is Chinese prefer to have personal relationship with their supply chain. The second best approach is to do a web search. Sites like Alibaba (www.alibaba.com), Global Sources (www.globalsources.com), China Sources (http://www.exports.cn/china-sources) can act as starting points, but these need to followed up by personal visits. Other approaches include trade fairs and electronics markets. I have found these sources to be less productive because of the volume of traffic. Many suppliers and customers mean each one only has little time to exchange information. Once a list of potential suppliers has been identified, make a trip to China to evaluate their capabilities. At the factory tour look at the overall cleanliness of the work stations, the age and maintenance status of the equipment, the quality assurance processes and personnel and the level of professionalism exhibited by the management and staff. Always ask for and follow up with reference suppliers, customers and partners. There are often only blurry lines drawn between social and business relationships. Chinese business men and women spend large amounts of time on their businesses so Chinese social interaction is most often with business/work associates. After visiting the factory, invite the factory managers to eat lunch or dinner with you. Share some things about your personal life, such as your family, your parents, how and why you got into business, etc. Chinese place higher importance on the value you have in your group rather than your individual strengths. Keep in mind Chinese supplier-customer relationships are also often strengthened by cross ownership of companies. It is common for a contract manufacturer to own stock in or be in joint developments with several of their key suppliers. It’s not necessary that you buy stock in your supplier’s companies, but it will strengthen your relationship if exhibit a commitment to a long term relationship. Al Mudrow’s Tips For Traveling in China: Plan a trip to China the same way you would plan an extended backpacking trip into the wilderness. Start with a backpacking checklist. Leave off the obvious items of tent, sleeping bag and sleeping pad. Pretty much everything else on your list is useful. China is a cash-based economy. Credit cards can be used in high-end western hotels but nowhere else. You can get cash through your hotel, banks or “grey-market� foreign exchange shops. China is mostly BYOTP (bring your own toilet paper). Always carry tissue paper with you when you venture out of the hotel. Don’t drink the tap water, no matter how fancy or western your hotel. Use provided bottled water. I carry water purification tablets if I want additional water. Be very aware of traffic as you are walking around. Cars and motorcycles come in all directions. Motorcycles especially seem to obey no traffic rules. Vehicles commonly travel on sidewalks as well. Be especially careful of trucks made from converted farm roto-tillers. They don’t have brakes. Take a GPS receiver with you. Mark the important waypoints like your hotel, your CM factory and vendor locations. You can use it guide your way when your CM forgets to pick you up, or when guiding your taxi driver, who will inevitably get lost. Always carry a mobile phone with you. Find out before you leave if your mobile phone will work in China. Get a Skype account and purchase Skype-out minutes. Phone calls back to the U.S. and within China are about 3 cents a minute. Quality is better than both mobile phones and even local telephones. In restaurants, use the tea they pour you to sanitize your eating utensils and dinnerware. Only eat food that has been thoroughly cooked. If you go out for a massage, which are common in China, make sure you specify to the receptionist that you want a “foot massage�. I’ve been told a regular massage involves more intimate contact than you may be comfortable with. Always remember that you are a guest in China. Be respectful of Chinese customs and people, especially the elderly. Chinese view their country with pride and reverence. Past interactions with foreigners have left them somewhat wary. The opium wars are a particular sore point. From their perspective, it is the same as if Columbia invaded the United States, annexed San Diego and forced the US government to allow the importation of cocaine and heroin. China is an atheist country. Expressions of religion are illegal. Avoid religious symbolism and any discussions of religion. Do not enter China illegally. Make sure you have a proper visa and do not overstay the limit of your visa, which is typically only 30 days. This is just common courtesy.