800G Module Thermal Shutdown: DDM Clues and Airflow Fixes
An 800G link that drops at 2 p.m. and comes back at night is usually not a bad module. It is a module doing exactly what its thermal design tells it to do.
Two weeks into an 800G leaf rollout, the pattern shows up: links come up clean at commissioning, then a handful of ports start logging temperature warnings and dropping frames during the afternoon peak. By evening everything is normal again, and every vendor's module tests fine when pulled and benched. The site then spends a month RMA-ing good modules.
That loop exists because 800G changed where the margin lives. At 400G, a module drew 7-10 W and almost any compliant cage kept it cool. At 15-20 W, the heat removed from the cage depends on per-slot airflow, cable bulk, adjacent ASIC exhaust and the room's daily cooling cycle - variables that a module datasheet does not contain. The diagnosis therefore starts with the module's own telemetry, not with a replacement.
What the telemetry shows first
CMIS-based modules report temperature, flags and lane-level telemetry continuously. The order in which the fields move tells you most of the story.
| Observation in DDM | What it suggests | Typical next step |
|---|---|---|
| Case temp high warning, no flags on TX/RX power | Ambient or airflow problem, module still healthy | Check rack inlet and cage airflow before touching optics |
| Temp high alarm and link drops, recovers when cool | Thermal margin exhausted at peak; consistent shutdown path | Fix airflow or reduce module power class |
| TX bias creeping up over weeks at stable temp | Laser working harder; possible end-of-life trend | Log the slope; replace when bias crosses vendor threshold |
| Temp normal, RX power sagging on one lane | Not thermal - connector or fiber path | Check MPO seating and cleanliness first |
| Flags fire on many modules in the same cage row simultaneously | Chassis or room cooling event, not per-module fault | Look at fan trays, blanking panels, CRAC setpoints |
The multi-module simultaneous case is the one that saves the most time. When eight modules in one cage row flag temperature within the same minute, the modules are bystanders. Treat it as a cooling event and correlate with fan speed logs and room temperature history.
Bias current is the leading indicator, not the verdict
On most laser types, bias current rises with junction temperature. So the first thing thermal stress does in your telemetry is not "link down" - it is a slow upward drift in TX bias on the hottest ports, visible weeks before any alarm. That drift is why a DDM baseline taken at commissioning is worth more than any single reading: a bias of 48 mA means nothing on its own, but 48 mA when the port commissioned at 41 mA at the same case temperature is a finding.
The interpretation rule that avoids most false alarms: judge bias against temperature, not against a fixed number. Plot bias vs. case temp per port; a healthy module traces a consistent curve, and a degrading one slowly departs from its own curve. The general not-linking checklist covers where bias fits in the wider diagnostic order.
Why 800G rows hit the limit first
The arithmetic is unforgiving and worth doing once, explicitly.
| Configuration | Optics power | Practical meaning |
|---|---|---|
| 64 ports x 7 W (400G class) | ~450 W | Traditional cage airflow absorbs it |
| 64 ports x 16 W (mainstream 800G) | ~1024 W | Front panel heat approaches the ASIC's own |
| 64 ports x 20 W (Class 8 parts) | ~1280 W | Requires verified per-slot airflow at real inlet temp |
That heat sits at the front panel, in the exhaust direction, behind the fattest cable bundles in the rack. Three field conditions turn a compliant design into a marginal one:
- Cable bulk. An OSFP with an MPO trunk is physically wider than the cage; a dozen of them act as a radiator fin that blocks the intended airflow path. Cables that exit upward, over the cage, are a common silent contributor.
- Recirculation. Missing blanking panels and half-populated cages let hot exhaust loop back to the intake side. The room temperature looks fine; the air reaching the cage is not.
- Daily cycle. Inlet temperature that drifts 5-8 °C between night and afternoon consumes most of a Class 8 part's margin by itself.
The commissioning gap
Modules are usually commissioned in the morning, on a partially populated chassis, at the cool end of the daily cycle. Production peaks happen in the afternoon, on a fully populated chassis, at the warm end. A thermal problem that lives entirely inside that gap will pass every acceptance test and still fail in production. If the rollout schedule allows, re-run the telemetry sweep at the day's thermal peak before sign-off - it is the cheapest hour of preventive work available.
A sequence that isolates the cause
The order below is built so each step either fixes something or eliminates a hypothesis. Skipping steps is how sites end up RMA-ing healthy modules.
- Freeze the facts. Export per-port temperature, bias, TX/RX power and flags for the affected ports at peak and at trough. Without the two-point picture, everything after this is guessing.
- Separate chassis from module. If flags cluster in one cage row or one chassis, work the airflow path: fan tray speeds, blanking panels, cable routing, intake temperature at that row. If flags are scattered across chassis, suspect the module SKU or the room.
- Audit the configuration. Confirm the module's power class against the switch's per-port limit at your inlet temperature, not the datasheet ambient. A Class 8 part in a cage qualified for Class 7 at 45 °C inlet is an unsupported configuration, and the switch may enforce that silently by throttling.
- Swap within type. Move one affected module to a known-cool port on another chassis and one healthy module into the hot slot. If the problem follows the slot, it is thermal. If it follows the module, only then is it a part.
- Re-check seating and connectorization. A module that runs hot can also simply be seated with insufficient engagement, which raises both thermal contact resistance and electrical reflections. Reseat, re-latch, and confirm the cage latch clicks on both sides.
- Log the fix, not just the outcome. Whatever held - rerouted cables, added blanking panels, derated power class - record the before/after telemetry. The next row will have the same physics.
When it is not thermal at all
Two impostors are worth excluding explicitly. A failing fan tray raises every module in a chassis together - that one is caught by step 2. The sneakier one is signal integrity masquerading as heat: a marginal electrical channel shows errors that correlate with traffic, and traffic correlates with temperature, so the two look married. The separator is TX bias: an electrical-path problem does not raise bias, while a genuinely hot laser does. If temp is high but bias is flat and errors correlate with utilization bursts, spend your time on the link budget math instead of the airflow.
Fixes that survive the next review
Ranked by how durable they are in practice:
- Physical airflow correction (cable routing, blanking panels, reseating). Free, permanent, and fixes the actual cause in most field cases.
- Spreading high-power ports. If only some ports need Class 8 parts, distribute them so no single cage section carries the full heat density. Costs nothing but planning.
- Derating the module choice. A Class 7 (15 W) part in place of a Class 8 (20 W) part restores margin, at the price of reach or media options. Compare candidates in the 800G selection guide before assuming the drop-in.
- Raising room cooling or lowering the setpoint. Works, but it is the expensive fix for a front-panel problem, and it usually indicates the cage decision was made for the wrong ambient.
- Changing cage type. Moving a hot row from QSFP-DD to OSFP buys heatsink area. It is the right answer for AI-scale densities and the wrong answer for an already-purchased chassis, which is why it belongs in the next procurement cycle, not this incident.
What does not work: firmware tweaks that lower alarm thresholds. Silencing the warning does not add one watt of heat removal, and the next thing silenced is usually the link.
Sources
- QSFP-DD MSA - CMIS specification - the Common Management Interface Specification behind temperature flags, power classes and lane telemetry referenced throughout.
- OSFP MSA - mechanical and thermal envelope definitions for OSFP cages and heatsink options.
- Ethernet Alliance - multi-vendor interoperability and hot-plug demonstrations, useful context for cage and module qualification practices.
Temperature thresholds, bias limits and power classes are part- and vendor-specific. Nothing in this article substitutes for the qualified-module list and thermal derating matrix of the switch you actually run.
FAQ
- At what case temperature does an 800G module shut down?
- There is no single number. Most modules support 0 to 70 or 75 °C case temperature, and the alarm and shutdown points are set per part through the management interface. Read the module's own temperature flags in DDM rather than assuming a datasheet constant - the host typically warns long before it shuts the lane down.
- Why does the link drop only during the afternoon?
- Rack inlet temperature follows the room's cooling cycle, and optics power is additive to switch ASIC power. A link that drops when traffic and ambient peak, then recovers overnight, is the classic signature of a thermal margin problem rather than a component fault.
- Is a rising TX bias current a sign of a failing module?
- Not necessarily a failing one. Bias current rises with junction temperature on most laser types, so bias creep under load often indicates thermal stress. It becomes a replacement criterion when bias keeps climbing at stable temperature, or when it crosses the module's alarm threshold.
- Can I fix thermal shutdown with firmware?
- Firmware can lower module power or re-check the qualified power class, which buys margin, but it cannot remove heat. If the cage airflow cannot handle the module's power class at your real inlet temperature, the durable fixes are physical: reduce power class, spread high-power ports, or change cage type.
- Do 400G modules have the same problem?
- Far less often. A 400G module at 7-10 W leaves most cages comfortable margin. The 800G step to 15-20 W is what turned per-slot airflow from a datasheet footnote into a qualification item, which is why the problem concentrates in 800G and 1.6T rows.