Toshiba & WD NAND Production Hit By Power Outage: 6 Exabytes Lost
by Anton Shilov on June 28, 2019 12:15 PM ESTToshiba Memory and Western Digital on Friday disclosed that an unexpected power outage in the Yokkaichi province in Japan on June 15 affected the manufacturing facilities that are jointly operated. Right now, production facilities are partially halted and they are expected to resume operations only by mid-July.
Western Digital says that the 13-minute power outage impacted wafers that were processed, the facilities, and production equipment. The company indicates that the incident will reduce its NAND flash wafer supply in Q3 by approximately 6 EB (exabytes), which is believed to be about a half of the company’s quarterly supply of NAND. Toshiba does not disclose the impact the outage will have on its NAND wafer supply in the coming months, but confirms that the fabs are partially suspended at the moment. Keeping in mind that Toshiba generally uses more capacity of the fabs than WD, the impact on its supply could be significantly higher than 6 EB with some estimating that it could be as high as ~9 EB.
Both companies are assessing the damage at the moment, so the financial harm of the incident is unclear. Not even counting potential damage to production tools and other equipment used at the fabs, 6 EB of NAND cost a lot of money. Furthermore, analysts from TrendForce believe that a consequence of the outage will be some loss of confidence from clients of both companies, which will have a financial impact as well.
The Yokkaichi Operations campus jointly owned and run by Toshiba and Western Digital produces about 35% of the global NAND output in terms of revenue, according to TrendForce. At present, the manufacturing base has five production facilities (Fab 2, Fab 3, Fab 4, Fab 5, and Fab 6) as well as an R&D center, all of which were affected by the outage. Three fabs within the campus produce 3D NAND flash, whereas another two are used to make special-purpose types of memory.
Considering the gargantuan size of the Yokkaichi Operations, disruptions of its supply will inevitably have an effect on 2D NAND and 3D NAND spot prices in the short-term future. Nonetheless, since contract prices have already been set for Q3 (and possibly Q4), they are not going to change. Meanwhile, it remains to be seen whether large customers will have to go shopping in Q3 or Q4 and affect prices on the spot markets further.
Considering that the Yokkaichi Operations produces at least 1/3 of the global NAND flash output (let’s assume that dollar share more or less corresponds to bit share) and half of its production for the quarter was lost because of the incident, this means that the industry will miss approximately 1/6 (or 16.5%) of the global NAND supply in Q3. Whether or not this will create a deficit on the market that will cause significant price hikes depends on multiple factors and is something that remains to be seen.
Related Reading
- Toshiba Memory and Western Digital Open Fab 6 and New Memory R&D Center
- Toshiba Memory & Western Digital Finalize Fab K1 Investment Agreement
- Toshiba Begins to Construct New BiCS 3D NAND Fab in Iwate Prefecture
- Toshiba Memory to Build New Fab to Produce BiCS 3D NAND
- Toshiba Finalizes Plans for New 3D NAND Fab: Coming Online in 2019
- Toshiba to Build New Fab to Produce BiCS NAND Flash
Sources: Western Digital, Reuters, TrendForce, Blocks & Files
148 Comments
View All Comments
azazel1024 - Tuesday, July 2, 2019 - link
Of course there is. You aren't talking gigawatts of power. You are talking at most several dozen megawatts. I've seen the backup power supplies for many industrial operations that draw far more power than a fab. You are talking a backup power solution that likely at most runs in to the low hundred million range for what is several billion dollars of equipment. Even ignoring the cost of the equipment that might be damaged, 6 exabytes of NAND production is in the ballpark of $150-200 million dollars worth of NAND down the drain (plus Toshiba's losses, which if the article is correct in speculation, is likely another $200-400 million in losses).Example of backup "UPS" on a grand scale.
https://electrek.co/2018/09/24/tesla-powerpack-bat...
$66 million installation cost for 100mw of battery power, 300mwh storage (so 3 HOURS of backup power). Yes, the Tesla project was done well after whatever backup power system the fab complex has would have been installed. The point is, for what is clearly higher output than what the fab would need (again, the power requirements for the entire fab complex are likely to be in the 10-40mw range), it is drop in the bucket compared to the losses they suffered.
I am sure that the fab complex DOES have backup power sufficient to the task. What I am also pretty sure of is it isn't redundant (which it should be) and also very likely wasn't tested sufficiently to catch problems (as that would likely require shutting down the fab temporarily, maybe once a year, to test the system live). Most industries that have/rely on backup power do an annual plugs out test.
Looks like maybe WD/Toshiba didn't adhere to industry standards there.
whyaname - Friday, July 5, 2019 - link
Actually there is, you only need a few seconds of UPS service till the back up generators of such factories kick in.You usually have several protection levels for this stuff.
For example it is not uncommon for such large facillities to have a complete discrete power plant or to be connected to more than one power plant/grid at a time in addition to UPS and generators.
Did you never wonder why industrial hardware has like 2-4 power connections, that is not just a redundant PSU, it enables you to have the hardware on just as many discrete UPS and powergrids to mitigate the likelyhood of failure.
I remember seeing the new some time ago that some flash manufacturers want to reduce their production because or prior over production and lots of flash still in stock at warehouses.
This is 13min failure that could have been prevented, but instead somehow causes 3months of issues.
This sounds like a way to avoid issues with markt price manipulation, but artificialy reducing the available amount of flash.
This should under no scenario have caused an issue similar to the flood back then in south east asia.
If the power goes out you do not suddenly start pumping polluted air into the clean rooms and making stuff unusuable for months.
Dug - Monday, July 1, 2019 - link
Your use of the word dude is a clear indication about your knowledge of power. Aircraft carriers use direct nuclear power for one, so I'm not sure why you included that? And thinking they didn't have a power backup is another indication you have no idea how things work in a fab. Just the fact you have no clue how much power is used or how it's produced is another indication. Or the fact a backup can't produce enough power if you are right in the middle of production.I'm sure they'll be contacting you soon to revamp power distribution sense you know so much about it.
Gunbuster - Monday, July 1, 2019 - link
A high end data-center can run 30kW racks. Why can a data center plan and economically execute full time backup power with that energy density but a fab can not?Samus - Tuesday, July 2, 2019 - link
Gunbuster is correct with his assumption. I worked at Argonne National Laboratory through college in the APS (Advanced Photon Source) building - which is a large synchrotron particle accelerator. It had 32, and recently expanded to 36 sectors. It's actually larger than Fermi lab. But nowhere near as large as the large haldron collider.But collisions it does. The Argonne campus has its own power plant. So do fabs. As anybody who knows anything about power generations knows, power plants produce at capacity in real time, and this capacity is always beyond grid demand to account for spikes or sudden demand surges where power cannot be diverted elsewhere from the grid. This makes them inherently inefficient because the excess power can't be stored, so it is wasted in the form of simply not generating power that could be generated using the supplied pressure\heat\whatever type of fuel is being used. This is especially true with nuclear power production (although no sites like these use nuclear power production on-site, that is overkill) as it takes weeks, and even months with old designs, to roll back fuel processing and scale down power production. During the cooldown, they are super inefficient in the sense they COULD be making a lot more power for the same amount of fuel spent.
ANYWAY. This site clearly has a oil fuel power plant on site. I can spot it on the low-res campus photo in the top right. This plant was built to supply more than enough power to the site EVEN IF grid power seized. But built for, and producing, are two different things.
Something happened at the power processing facility, or something malfunctioned causing a blackout (which seems to be the case considering the length of time there was a power outage...they could divert grid power almost immediately like Samsung did during their recent production woes that lasted in the milliseconds.)
That said, this is hard to wrap my head around. These mistakes just don't happen. But mistakes can be manufactured. And mistakes that take down an entire campus of electricity for minutes almost has to be manufactured...even natural gas\diesel generators warm up and produce power in 20 seconds...
close - Tuesday, July 2, 2019 - link
@Gunbuster - A high end data-center can run 30kW racksProblem is in this case we're talking 2 orders of magnitude higher. In the tens of MW range, maybe even close to 100MW (GloFo Fab 8 needs 80MW *on average* as a reference). And supplying that from backup constantly until the power is restored in not something most industries ever have to deal with so people should stop using their UPS as a reference for anything. There's no real way to save any power during such incidents, there's no "power saving mode", no shutting down part of the line. The backup power has to provide exactly what the main did. Usually you manage to supply enough power to shut down the lines with no damage to the equipment but all the silicon in the pipeline just became trash anyway.
This happened to Samsung, it happened to Toshiba/WD. It's just AT commenters that seem to have the wrong career. AT became home to all of the "I'm an expert dude" type people.
Gunbuster - Tuesday, July 2, 2019 - link
And most data centers don't cost a billion to stand up. You scale up to provide services. Plenty of 10MW gensets to choose from. Sure they are the size of a semi trailer but that's a huge building. Line up 15 of them.whyaname - Friday, July 5, 2019 - link
@closeYou realize that is supposed to mean 30kW per rack?
scale that
Aside from that somebody very well posted already that backup power solutions for this scale exist.
Round - Tuesday, July 9, 2019 - link
Not related to the article, but regarding " how can a whole country be stupid enough to fall for something like a bolshevik revolution".We have almost half the US population teetering on the edge of a similar mental catastrophe right now. Intelligence is not distributed equally...
lazarpandar - Friday, June 28, 2019 - link
..What? Darwin awards are awarded for deaths.