|
This is a story about industrial espionage. At the outset, I must say that some of what I report here is heresay. The technical parts are what I have added, and
I've got a 40 year technical background, including three patents, that I hope will satisfy you
that I'm at least somewhat qualified to talk about this. It's the chronology of events that I can't guarantee.
Much of the info here has been corroborated by more than one manufacturer or other source.
I wouldn't be including anything if it didn't make sense and fit together with everything else I've
picked up together with what I do know to be fact. If you know something more, or have a correction you're certain of, I would appreciate
hearing from you.
The first signs of a problem surfaced in our shop in 2003. Motherboards that were well within their one year warrranty were being returned
at an alarming rate, all for the same reason: frequent reboots, lock-ups, accompanied by the physical symptoms of swollen, leaking electrolytic capacitors. In one instance an
electrolytic capacitor actually exploded and filled the area around the motherboard with tiny shreds of plastic.
Multiple motherboard manufacturers were seeing/hearing the same scenario. I was told that Abit, at one point, had a 40% return rate and it just about killed them. I also heard
that Acer had a significant return rate due to this problem. I'm sure there were many, many manufacturers that were victimized in the same way.
TECHNICAL BACKGROUND:
Motherboards use a lot of electrolytic capacitors to filter out noise on the power-supply lines, i.e. the +12, -12, +5, -5, +5VSB, and +3.3V lines. Motherboards
also have on-board switching regulators to provide the additional lower voltages used by CPUs. The on-board regulators need the highest quality capacitors because the
current variation for their voltages is very high and the CPUs require the tightest regulation.
There is a range of quality for manufacturers to choose from in aluminum electrolytic capacitors. The highest quality capacitors are used in places like servers that run 24/7 and are expected to be reliable for several years. Desktop motherboards are not held to such a high standard, so they don't get the best capacitors. Typical desktop motherboards can reasonably be expected to last 3 to 6 years and the capacitors are chosen accordingly.
So, these electrolytic capacitors, so important to reliable operation, are used all over the motherboard. These are the things that look like tiny aluminum beer
cans, in various sizes, standing up, scattered around the motherboard. The biggest, most critical ones are clustered near the CPU.
Aluminum electrolytic capacitors are made with two layers of metal physically separated by a dielectric layer of material that is bathed in a liquid electrolyte. The surface
area of the two metal layers is so great that the thickness of the metals and the separator are microscopically thin. The electrical properties of the electrolyte facilitate the microscopic size
while still getting useful capacity values. None of the materials are particulary exotic, rare or expensive. The metal is aluminum. It's the manufacturing
process that is most amazing by being able to make these tiny things so precisely and at such low cost.
THE SCAM:
The story I heard was that in 2002 two guys in Taiwan, who were working for a chemical company that made electrolyte for sale to capacitor manufacturers, decided to start their
own company. They took the recipe from their former employer and set up shop. They sold at lower prices and were able to capture business from multiple capacitor manufacturers,
three that I was specifically told about, by name, by a motherboard manufacturer. These three brands of capacitors were being bought and used by many, many motherboard
manufacturers. And not just on motherboards, also in lots of other consumer electronics products. But motherboards are somewhat unique in the demands that they place on capacitors,
and this story is about motherboards.
The capacitors with the suspect electrolyte work fine initially. After about six months of use, the capacitors start failing, as I described before. There's no clue to the impending failure mode that any
testing engineer can predict. It takes a few months for the mode to develop, and with the computer industry racing new generations of technology to market every
three months, no-one is willing to test their product for six months to watch for unexpected failure modes. NASA can wait this long. The military can wait. Communication Satellite
manufacturers can. Not Consumer electronics!
With motherboards dropping like flies, the engineers now start researching the failure mode. It turns out that the guys who stole the electrolyte recipe DIDN'T GET THE WHOLE RECIPE. One or two things, that acted like a preservative in food, were left
out. So the electrolyte, with the "preservative" missing, starts going bad after just a few months instead of a few years. With the
electrolyte going bad those capacitors, the ones that have the hardest work to do, develop excessive internal resistance and internal heating. The heat causes the electrolyte to swell,
and leak out, and go bad even faster. So the process accelerates over a brief time until, in worst-case scenarios, the capacitor literally explodes. Usually, though, the capacitor
harmlessly vents the pressure, as it is designed to do. But also typically the electrical characteristics of the capacitor, i.e. its filtering ability, deteriorates first and allows
noise to grow on the supply lines until the motherboard locks-up and reboots and prompts the owner to replace the motherboard at the time that the capacitor swelling is just beginning.
By the time the problems began manifesting themselves, these motherboards were all over the world. The motherboard manufacturers start seeing very heavy RMA
returns. The solution, replacement of the defective capacitors with good ones, is labor intensive. The manufacturers usually give a one-year warranty with the motherboards,
but as you know most distributors won't help after the first 30 days. So the customer, or VAR must send the board direct to the manufacturer and wait weeks or months for a
repaired or replaced board.
|
|
For a VAR who sold dozens or hundreds of these boards to a client this is a disaster. The labor and logistics of removing and replacing each board is horrendous. No client
wants to be told he has to stop using his computer for six weeks while the motherboard gets repaired free under warranty. The client is mad and doesn't trust the VAR
anymore. So the VAR loses the client(s.) The VAR is mad at the distributor and motherboard manufacturer and vows never to buy that brand ever again. The motherboard
manufacturers are staggering under the financial drain of replacing millions of defective capacitors at the same time new sales revenue is plummeting. It was at this
stage that I heard Abit almost went bankrupt.
Manufacturers typically fullfilled their one-year warranty obligation and that was it. Distributors usually wouldn't lift a finger to help. The VAR had to tough it
out with the manufacturer and their clients. Some VARs had to buy new replacement motherboards, and because Pentium-3's were long gone they also had to buy new CPUs and
RAM for their clients.
Dash's experience was much better than typical. One of our motherboard manufacturers, Jetway, got bit by the bad electrolyte on two Pentium-3 motherboards. We sold a
lot of them. Jetway's warranty is two years. We started getting boards back from our VARs at about six months and the volume was such that we asked for and got a stock of boards
for advance replacement. At two years we were still getting returns with bad capacitors. We asked Jetway if they would continue replacing capacitors beyond the warranty
period AND THEY AGREED! The manager of the US branch is very service-oriented and it is he who deserves the credit for this great treatment, unheard of in this industry.
At almost three years, we are still advance-replacing a trickle of motherboards for our VARs.
There's cruel irony in this. Jetway treated us and our VARS far better than any other motherboard manufacturer I've ever heard of. Yet most of our VARs now refuse
to consider buying a new Jetway motherboard, despite the reality that Jetway was the best company to have in a situation like this.
And Jetway was scammed. Jetway tries to use better-rated capacitors than most motherboard manufacturers. Jetway was using capacitors rated to 105degC when most
manufacturers, including Intel, use 85degC capacitors. The higher rated caps normally have less internal heating and therefore last longer. Jetway was paying a
little more for higher rated caps that should have made their motherboard last longer than average. They were trying to make a better product. They were victimized,
as we all were.
We heard that Dell, too, had this problem in a big way. CRN ran a story in November 2005 saying that Dell took a $300,000,000 charge to their bottom line related to this capacitor problem. Apparently
they kept it quiet for over two years but it finally outgrew their public-relations resources.
There's some useful, practical knowledge that you should take from all this: When you buy a product using electrolytic capacitors you're buying time. You get what you pay for, usually. The more expensive higher
rated caps last longer. And for a motherboard, the caps usually limit its lifespan. In the early days of the Pentium the power levels were so low that the capacitors loafed
and tended to last almost forever. A VAR once told me his client was finally forced to replace the Pentium-III computers he got from us 6 years earlier. The computers were
working perfectly, like new, but a new version of his client's application software required faster computers. We hear this kind of story multiple times per month. The
reason our Pentium-III motherboards lasted so long was that the capacitors were good quality and they weren't working very hard.
Pentium-IV changed all that. They were power hogs that worked the electrolytic capacitors very hard. Since, as I said before, when you buy electrolytic capacitors
you're buying time, you should turn your computer off as much as possible to avoid using up the limited number of hours you paid for. If you're buying servers, that
run 24/7, you should look for only the best quality capacitors on the motherboards and in the power-supplies. Do you know the MTBF of your power-supply? We know ours. Our server supplies are always rated over 90,000hrs MTBF. And the limiting factor in them is the fans, not the capacitors!
The cheapest motherboards will use the cheapest capacitors. As this story should have illustrated, there's more to choosing a motherboard, and motherboard
supplier, than getting the cheapest price.
Many motherboard manufacturers are now offering motherboards with a special type of capacitor using solid electrolyte instead of liquid. These parts are a little
more expensive, but they last longer than the best liquid-electrolyte types because there is nothing to dry out. These capacitors are sold under the trade-names like OC-CON and OS-CON,
depending on the capacitor manufacturer.
For more research, try these links (found Feb 2013):
http://en.wikipedia.org/wiki/Capacitor_plague
http://www.guardian.co.uk/technology/blog/2010/jun/29/dell-problems-capacitors
http://spectrum.ieee.org/computing/hardware/leaking-capacitors-muck-up-motherboards
http://www.badcaps.net/pages.php?vid=4
http://www.deadprogrammer.com/the-capacitor-plague
http://businesspracticeimprovement.com/2011/11/23/bad-capacitors/
http://archive.siliconchip.com.au/cms/A_30328/article.htm
|
