HPC Testing: How to Validate a High Performance Computing Cluster

High performance computing (HPC) testing covers software that runs on thousands of nodes at once. It breaks the two assumptions a normal test suite rests on. First, that a correct answer exists to check against. Second, that the same input gives the same output twice.

Neither holds on a cluster. A climate model computes results nobody knows in advance. A summed reduction returns other low-order bits once you change the rank count. So the check changes shape. Exact equality gives way to invariants, tolerance bands, and cross-run comparison.

The cost profile is odd too. Most defects announce themselves. These ones do not.

Why the normal QA toolkit misfires

Three staples of app testing carry almost no weight here.

Line and branch coverage. Coverage counts which code ran. Cluster defects live in execution order, not in unvisited lines. The space of possible thread orderings blows up with program size. A test run samples a vanishing slice of it. Surveys of race detection make the point plainly: a harmful data race can hide behind an ordering that testing never schedules. A code can hit 100% line coverage and still deadlock the first time it runs on 4,096 ranks. Coverage still helps on serial logic, and the case for reading it as a signal rather than a gate holds with more force here.

Deterministic unit tests. Mocked, isolated checks cover the parts that were never the risk. The bugs come from behavior across ranks: message order, memory contention, topology-dependent routing.

UI and end-to-end automation. The runtime is a batch queue and a CLI. There is no UI to drive.

Testing without an oracle

When no golden answer exists, test properties of the result instead of the result. Three substitutes cover most cases.

Invariants and conservation laws

Physical models conserve quantities. Mass, energy, and momentum must balance whatever the output file holds. The invariant stands even when the value is unknown. That is the exact property an oracle-free test needs.

The same shape works outside physics. Schema validity, entity counts, and monotonicity are checkable without knowing the right answer.

Metamorphic relations

Change the input in a known way. The output must change to match. Double a dataset and a scale-linked metric should double, while a normalized one holds steady. The check asserts on the link between two runs. It needs no correct value for either one.

This is the standard academic answer to the oracle problem. Metamorphic testing for scientific programs grew out of this exact gap. It is also the most portable idea in the field.

Ensemble consensus

Run one input through several independent code paths. If four cluster tightly and one drifts, the outlier is the finding. The peer group builds the oracle at runtime.

Drift is the baseline, not the bug

Floating-point addition is not associative. Reduction algorithms re-associate operands based on rank count and network topology, so round-off shifts between runs. This is known behavior, not a defect.

The MPI standard says so outright. An implementation may exploit associativity and commutativity to reorder evaluation. Doing so may change the result for operators that are not strictly associative, and floating-point addition is one. For codes that cannot absorb this, the standard’s own fix is to gather the operands and reduce them locally in a fixed order.

Here is the effect in eleven values, run in Python 3:

import math

vals = [0.1] * 10 + [1e8]
forward = sum(vals)
backward = sum(reversed(vals))

print(repr(forward))                                    # 100000001.0
print(repr(backward))                                   # 100000000.99999994
print(forward == backward)                              # False
print(math.isclose(forward, backward, rel_tol=1e-12))   # True

Same eleven numbers, opposite order, new result. Now scale that to a reduction over 100,000 ranks. Three responses are in use. They work together rather than compete.

Tolerance bands

Replace assert a == b with a relative or absolute tolerance. The band should follow from how well-conditioned the algorithm is. It should not follow from whatever number makes the suite green. A band picked to silence a failure is a disabled test with extra steps.

Bitwise repeatability modes

Compilers and libraries can be forced onto deterministic paths for debugging.

Intel’s compilers default to -fp-model=fast. Value-safe behavior needs an explicit -fp-model precise. On NVIDIA GPUs, PyTorch’s guidance calls for torch.use_deterministic_algorithms(True) plus CUBLAS_WORKSPACE_CONFIG=:4096:8 or :16:8. Without it, cuBLAS varies across streams.

Note the ceiling on this move. PyTorch’s own docs state that several backward kernels use atomicAdd, and that no simple way exists to avoid the drift they cause. These paths also cost speed. They belong in debug and test setups, never in production runs.

Distribution matching

Where per-value checks are hopeless, compare distributions.

Climate modeling has formalized this. The E3SM project builds two 30-member ensembles and applies a two-sample Kolmogorov-Smirnov test across more than 100 output variables, then counts how many differ. E3SM’s write-up states the acceptance question plainly: can we switch computers? Bit-for-bit repeatability is off the table, so statistical sameness becomes the pass bar.

Acceptance and regression are different jobs

Merging the two is a common and costly mistake. They hunt different failures.

Acceptance leans on two synthetic workloads. HPL, the portable High Performance Linpack behind the TOP500 ranking, drives the floating-point units and memory hard enough that it doubles as a burn-in test. Fabric faults show up in it as odd performance. On GPU clusters, nccl-tests measures collective bandwidth. The useful read is the curve, not one number. Plot bus bandwidth against GPU count and look for the sudden drop that names a bad link or switch.

The escapes that cost the most

Silent data corruption

Some CPUs compute a wrong answer and report success. Google’s HotOS 2021 paper “Cores that don’t count” named these mercurial cores. It put the rate at a few per several thousand machines. Facebook, now Meta, published its own account of silent data corruption in the same period.

Four traits make this the worst class of escape:

• The errors come and go and depend on the data, so they survive diagnostic loops.
• Their rate shifts with voltage, frequency, and heat.
• Some cores behave for years after install, then stop.
• Nothing crashes. No machine check fires. The job ends and the numbers are wrong.

A long run can therefore sit on a quietly bad core for weeks and void the whole compute window. Catching it needs routine check workloads across the fleet. There is no error to alert on.

Faults that only appear at full scale

The second class passes small-scale acceptance cleanly, then fails in production. A marginal optical transceiver drops packets only when most of the cluster hits peak collective traffic. Congestion, link flaps, and route recompute all scale in a nonlinear way. A 64-node check says little about 4,096 nodes.

The published numbers are stark. Meta’s Llama 3 405B report covers a 54-day pre-training window on 16,384 H100 GPUs. It logged 466 job interruptions, 419 of them unplanned. That is roughly one failure every three hours. Switch and cable faults caused 35 of the unplanned ones. OpenAI lands in the same place from the network side, reporting that at large enough scale even the best network carries a steady background of link and switch failures.

The design upshot: at that failure rate, fault tolerance is a tested feature, not a fallback. Checkpoint and restart paths earn the same scrutiny as the compute kernels.

What HPC testing borrows from ordinary QA

Three techniques carry over intact. Two of them get stronger at scale.

Boundary value analysis. The bounds are memory footprint, message size, and rank count. Test at the edge of each. Test at the scale steps where the algorithm switches strategy.

Combinatorial matrix testing. The HPC stack is a product of compilers, MPI builds, GPU runtimes, and math libraries. Spack was built to manage exactly that. Its stack matrices generate the build combinations that the E4S CI system then validates. This is matrix testing with a package manager attached.

Chaos engineering. Kill nodes, drop links, and throttle power on purpose. Then verify that checkpoint-restart does its job. Given a fault every few hours at scale, this is closer to a functional need than an experiment.

The same problem reappears in AI evals

Every technique above answers one question. How do you test a system whose correct output you cannot state up front? That question now describes most AI systems too. Invariants become schema and structure checks. Metamorphic relations become paraphrase and perturbation tests. Ensemble consensus becomes multi-model agreement scoring.

Teams arriving at model evals from web QA tend to rebuild these ideas from scratch. The practical view of what AI agents can and cannot test and the broader shift in the QA role circle the same gap. HPC solved a version of it decades earlier, and the terms carry over.

Related reading

• Test Coverage Explained. Why coverage is a signal, not a gate.
• Defect Density. The metric that survives contact with real projects.
• QA & SDET Interview Questions. The broader interview guide.