Vapor gate including failure of the entire quality management? Background on the cooler problem of the AMD Radeon RX 7900 XTX (MBA)

Before I mention more numbers about the Radeon RX 7900 XTX and the cooler problem, as well as a few words about the supply chain and its failure in quality management, we also have to talk about quality management in general and quality assurance as well as quality control in particular today. Why? Because it helps to better understand the complex interrelationships and also to classify them, since AMD itself is not a manufacturer, but has them manufactured. And this contract manufacturer, in turn, relies on other suppliers, who in turn have third-party and partner companies work for them. The causal chain therefore seems to be almost endless and if even one link breaks, then good night.

However, this is exactly what seems to have happened with the current problem with AMD’s own MBA cards (Made by AMD). In the end, one can only feel sorry for Scott Herkelman, because to give a live interview at a time when not all details could be clarified for sure, I think is quite explosive and breakneck. Because actually he could only lose here, at least as far as the statements made were concerned. The fact that one uses the publicity of a medium, in order to deflect and/or calm down, is given. After all, it is also the job of PR to do such things. But one should really be careful at this point where the boundaries between reality and wishful thinking run. After all, everything is verifiable, even if unfortunately often only much later. And the customers are not late risers.

Important preliminary remark

To be fair, I have to state that some of my conclusions are based on information from third parties (involved parties), but it seems completely plausible when looking at the usual business processes. I am, however, familiar with the processes at the manufacturers involved, because there are no real secrets there either. But what went wrong in detail must now be worked out and clarified by quality management as quickly and completely as possible. This is neither my task nor that of any third parties or outsiders. I have also clearly broken down the following processes and considerations on quality management so that the normal reader does not fall asleep bored over his morning coffee.

The globality and complexity of a quality management system includes all customer-supplier relationships and acts as a permanent, intensive interrelationship between them. And when we speak of quality here, the quality concept used stands as a unity of condition and quality requirement. The quality demanded is therefore not the maximum feasibility, but the correct realization of the quality requirements defined for a cooler. errors can always happen and there is no such thing as 100% error-free production. But one must always strive to approach this unattainable ideal with a so-called zero-failure strategy. 

Fact check

Let’s therefore get to the fact check first before our quality management investigations. The first statement that is surprising is that the interview talked about a wide availability of exchange cards. Since I am involved in an RMA process myself, I will quote from the letter from AMD support to the customer, which is also from the same period, so it is up-to-date. While the first letter (from the beginning of January 2023) still spoke of around two weeks for the replacement already applied for, this circumstance has now turned out to be no longer tenable. I therefore quote once from the letter now available to me, which came as a response to the exchange already agreed with AMD:

 

This could have been communicated properly in the interview, indeed it should have been. The fact that the PR contradicts the support here is actually no longer comprehensible. In addition, Scott Herkelman speaks in the interview of only a small number of affected cards, which can certainly not be so true. While various board partners (AIB) report 9 to 11 percent of RMA cases now realized or requested, the information from system integrators (SI) and finished PC manufacturers is much more alarming. Because there are now also disassembled or not completely assembled PCs with rates of well over 10% on stockpile or standby. This may also be due to the fact that SIs test much more purposefully than many buyers of single cards (due to lack of knowledge). Selling a clock loss due to thermal throttling as a feature is downright absurd because the memory, whose operating temperature was specified far below 110 °C, is also fried here. You really have to blame the interviewer for not following up on some points here. Unfortunately, that’s soft washing à la carte and was guaranteed to have been agreed upon in advance.

Apart from the fact that some SIs have already run out of storage space (we are also talking about an amount of systems in the higher three-digit range in each case), this unfortunately plays right into the hands of the green competitor. I wouldn’t even be surprised if NVIDIA would ensure that a simple replacement with a GeForce RTX 4080 would be pushed here with special discount promotions. At least I know that PCs have already shipped with cards converted to NVIDIA products. That is at least bitter for the market penetration in the SI sector, which is already a difficult patch for AMD. If I were NVIDIA, Jensen would easily pay for that out of his leather wallet. After the adapter gate from the November event, which was presented with media attention, now the vapor gate as revenge? Karma is a b1tch.

Basic problem: AMD is not a manufacturer, but quality management is a serious science

I always find it difficult (for understandable reasons) to name exact sources. Because this will burn them faster than you can say pug. And it’s also a matter of decency. But we can reveal a few things today that are certainly not familiar to the masses. This already starts with the chain of the entire contract manufacturing, which AMD entrusts to an experienced manufacturer (OEM and ODM) like PC Partner from Dongguan. But this manufacturer is actually only the final manufacturer, who in turn commissions third and fourth parties to supply further components in a long chain.

What does it actually take to make a graphics card or do the finishing work? First of all, you commission board manufacturers to deliver the bare PCBs to the manufacturer, you buy all the components and sometimes even have the belts for the SMD components already assembled by third parties – and you buy a complete cooler. Before I go into this cooler and the other entanglements in the delivery process, let’s take a rough look at this with quality management first so we can also grasp the fundamental problem.

The first problem, despite ISO 9000:2015 , is the exact demarcation of the 3 areas of quality management (QM), quality assurance (QA, Quality Assurance) and quality control (QC, Quality Control), which cannot be answered so precisely at all. This is because the entire supply chains and distributed responsibilities blur the definitions here and create fluid transitions that make exact demarcation difficult to impossible. Furthermore, each manufacturer can also introduce its very own definitions and transitions, respectively adapt these sensitivities to the exact circumstances, conditions and requirements of the respective end product. In any case, it is essential that the higher-level OEM, together with the client, defines and ultimately enforces the meaning, definition and delimitation of QM, QA and QC. 

While quality management is in the hands of AMD and, to a large extent, the contract manufacturer, third-party providers and suppliers must fit in seamlessly here. This is definitely a rocky road in the case of the RX 7900 XTX, as the cooler is supplied by Cooler Master. However, Cooler Master is also only “man-in-the-middle”, because the cooling construction including the vapor chamber is again made by Asia Vital Components Co, Ltd. (also known as AVC for short). So while QM is up to AMD and the contract manufacturer, Cooler Master has to subordinate and ensure QA on their part. The actual manufacturer AVC must in turn subordinate itself to the superordinate QA (and thus also to the general QM) and ensure QC on site during production. Let’s take a quick look at how QA and QC are directly related:

The QA should have already been applied to the bare cooler. You have to assign the responsibility to the Cooler Master product manager in question, even if AVC is the actual culprit, i.e. the guilty party. QA at AVC has to be controlled by the client, that’s just the way it is. In detail, this means that QA always takes a proactive approach. This means that a defect should be prevented by a process or product design, before it even occurs. QC in the production process and final inspection, on the other hand, takes a purely reactive approachand focuses on the detection and verification of defects, that have already occurred.

QA takes a process-oriented approach and focuses on preventing quality problems by implementing processes and methods, while QC takes a product-oriented approach and focuses on identifying quality problems in incoming goods (more on this on the next page), within the manufacturing process, or during final inspection and release (also more on this in a moment). QA’s quality systems include methods and procedures for meeting quality standards and requirements. The quality control systems measure, test and verify before and during the production process. I don’t want to go any further than that here.

So we see that a defective chamber could (and should) have been detected by already testing samples in the finished product from each batch (series), which at AVC is generally 10,000 units. And for those who still remember: AMD, PC Partner, Cooler Master and AVC have not learned anything from mistakes already made. Does anyone still remember the dreadful pump whining of AMD’s FuryX? It took a German editor to find the trapped air bubbles, while the entire Asia Connection fumbled in the dark for weeks. Sometimes it was the PWM controller, sometimes the stupid customer with a wrong installation. All just cheese. Fun fact: AVC later even supplied the defective pumps to Gigabyte for their Waterforce, which I was then allowed to uncover. Well, if the product managers for AMD and NVIDIA don’t talk to each other, something like this will come out. The controller at AVC must have been pleased with the profitable disposal of partially defective pumps.

A quiet pump for Gigabytes GTX 980 Ti Xtreme Gaming Waterforce | Retro 5 years ago

When nothing matches

Exactly at this point, however, I would still have an insider information that actually makes a meaningful QA hardly possible. Well, we’re not on Tinder now, but “match” has a lot of meaning in QA, too. I already wrote that the serial number of a graphics card can be used to infer the series of the cooler, whose QR code provides the necessary information. However, anyone who believes that this will also allow them to locate the series (batch) of the vapor chamber may now be disappointed.

For this to be possible at all, the batches of the vapor chamber and the finished cooler would have to be congruent, i.e. “matched”. Only if a batch of coolers is continuously equipped with only one continuous batch of these vapor chambers can one speak of a traceable process at all. However, according to my information, this is not the case. If this is true, then partially defective vapor chambers can spread wildly over many batches of coolers because different batches were mixed in the factory. However, anyone who dares to do something like this also has the damned duty to thoroughly test and check all components until the end. The word QA contains “Assurance”, and that’s what matters.

I wish AMD (and in the end especially Cooler Master) a lot of fun and nice weather to be able to quantify this at all validly. That’s why it’s so difficult: you just don’t know. So in the end, you will only be able to act reactively and exchange. If the circumstances I mentioned are really true, then a targeted recall would be technically and logistically impossible at all. However, you can read why PC Partner also failed miserably on the next page, where we will talk about the exact QM at the manufacturer, because I have already been to a PC Partner factory and know what I am writing about.