The ET-SoC-1's DVFS loop, and the leakage it never switches off

The yardstick is how Kanter described a mature chip managing its own power. It runs a DVFS loop (dynamic voltage and frequency scaling) that knows and, because voltage and frequency are already known, computes its own power estimate from activity counters. It folds thermal sensors into the same loop, because leakage depends on temperature. And it puts cache data arrays behind leakage-suppression transistors, so that only the ~10% of an array in use is awake. He hedged that this might not hold for Esperanto's part, only for anything from a more mature company. The hedge was right. The table below checks five of his statements, paraphrased, against the firmware source, the open RTL and the card.

every device-management pass: ,
 while a 10 Hz telemetry sampler queries the card, in the SP's own trace
(the board value changes every  in the sampler's stream, §7: the two methods differ by  ms):
    T = pvt_get_minion_avg_temperature()     // integer mean of the 34 minion-shire sensors, each truncated to whole degrees
    P = pmic_read_instantaneous_soc_power()  // MEASURED board power, from the PMIC over I2C
    if T > 65 °C and state < THERMAL_DOWN:                // no test of whether a kernel runs
        state = THERMAL_DOWN, and the power task runs the thermal loop (below)
    elif state < THERMAL_DOWN:                            // the power branch: locked out while the loop runs
        if master minion IDLE:  clock back to the boot point (600 MHz)
        elif P > 65 W:          step DOWN, no delay, until the PMIC average < 1.05 × 65 W or 300 MHz
        elif P < 65 W:          step UP,   no delay, until the PMIC average > 65 W or the top point

thermal loop (the power task; nothing else changes the clock meanwhile):
    repeat: step DOWN one VMIN-LUT point; sleep 1000 ticks ( s of SP time); T = the mean again
    until T <= 65
    then clock back to the boot point, and on the next pass the power branch climbs to the top point in one call

Where each line lives (all "if active power management is on"): the trigger at 668–679, with the threshold TEMP_THRESHOLD_SW_MANAGED 65 at include/thermal_pwr_mgmt.h:46; the mean at driver/pvt_controller.c:1279–1306 (shires 0–33; the I/O shire's sensor is not in it), each reading converted in integer arithmetic, so truncated to a whole degree, at :572–590; the power branch's gate at 844–845, its idle, down and up cases at 848–860, 862–873 and 876–887, reading the instantaneous power at 798–807; the power loops at 2173–2256 (the climb at 2229–2236, the down loop's exit at 2238–2246); the thermal loop at 2316–2361 with its vTaskDelay(pdMS_TO_TICKS(1000)) at 2354, its exit at 2366–2374 and the return to the boot point at 2446–2449 and 1975–1979. The tick is read from the card: the service processor's heartbeat, vTaskDelay(100000), appears in its trace every s of its own clock ( intervals on 27–28 September, §8), so 1000 ticks are s.

• The input is a measurement, not an estimate. The trigger reads the power-management controller's instantaneous wattage, and the loops' exits its own running average. Nothing in the loop counts bus bits, instructions or activity factors, in this source, the older one or the current one. Kanter's is not evaluated anywhere in this firmware, because the chip does not need to infer the left-hand side: it has a meter.
• Temperature is one number: the mean of 34 whole-degree readings. The governor compares the integer mean of the 34 minion-shire sensors, each already truncated to a whole degree, with 65, strictly: it acts at a mean of 66. No per-sensor or hottest-sensor path changes the clock: the firmware keeps a max_temp field and never writes it, enables no temperature interrupt on the sensors, and hands the hottest sensor's reading only to the host. So a hot spot matters only through what it adds to the mean. §8 tests this on the card: every step that fitted a rule fitted the mean, in development and in a frozen validation, but the validation had too few runs to pass its registered test.
• Thermal has priority, and it blocks. Once the mean reads 66, the service processor steps the clock down one table point, sleeps s, re-reads the mean, and repeats until it reads 65 or less. While it loops, the power branch, and with it every step-up, step-down and idle reset, is locked out, and nothing tests whether a kernel is running, so an idle card that rests above 65 °C sits in the loop at the lowest point. aifoundry2 often rests there (73 °C on 22 September, about hours after our last recorded workload, and 73–74 °C after a night of no recorded work on each of the next two days; its residency counter put it in this state for of the days since its last reset, §8), so a kernel launched on aifoundry2 usually starts inside the loop and runs at the lowest operating point however little power it draws: every launch of the strict protocol, at 80 °C, ran at 600 MHz. aifoundry3 rested at 50–58 °C on 22–24 September and at 55–57 °C since 25 September, but its governor is stuck another way (§3).
• There is no hysteresis. The loop starts at a whole-degree mean of 66 and ends at 65, with no band between; the only damping is the loop's own s. The cards' source uses one power guardband, on the way out of a power step-down (it stops below 1.05 × the TDP, 68.25 W); the September 2024 rewrite defines both guardbands and uses neither (§9). Neither would damp hunting at the thermal threshold (§2); that needs a temperature dead band.
• The loop exits to the boot point, then the clock climbs. When the mean re-reads 65, the loop sends the clock to the boot point, 600 MHz, not back up; on the next pass, if a kernel is running and the board draws under 65 W, the power branch climbs to the top point, 800 MHz, in one call, with no delay between points. When the master minion reports idle, the power branch also sets the boot point. On aifoundry2 that came about s after the host saw the last launch end ( s, at the of block ends on 21 and 23 September that found the clock above 600 MHz), and at of those the loop first stepped the clock up. Back-to-back launches rarely trigger it, and a short pause between launches more often: .

No down-step is the power branch acting alone. Of the 18, seven were thermal only (the die above 65 °C while the board drew 45–56 W, well under the limit) and four read above both thresholds on the sample after the step (random data draws up to 88 W at 800 MHz and 69 W at 700, and heats fast). The other seven cannot be attributed from our telemetry: the die read 65 °C (once 64) and the board 34–56 W, so neither test fired on the values we sampled. One of them, a single event (run 1, zeros, at 5.95 s: 800→600 MHz in one step, 171 ms after a launch boundary), looks like the boot-point reset, or a thermal loop that found the mean at 65 °C on its first re-read and exited straight to the boot point. (Launches run back to back every 0.37–0.49 s, so nearness to a boundary alone explains nothing.) The other six are single-point steps in pairs 0.4–0.5 s apart, which is the thermal loop's period (§1): the first of each pair a thermal step, taken on a pass where the service processor read a mean of 66 °C between our 10 Hz samples, and a 700→600 MHz step at a reading of 65 °C the loop's exit to the boot point (§9 gives the split read on the sample before each step). The power branch never fired on its own, because on a cool die only random data at 700–800 MHz exceeds 65 W (at 600 MHz it does too, but only on a die above about 82 °C, where the thermal loop already holds the floor), and random data takes the die through 65 °C within half a second. Two drops at high power need a closer look. Run 2 went 800→600 MHz in one step between two samples at 87.5 W; that fits either the power loop or a thermal episode of zero steps, whose loop finds the mean back at 65 °C on its first read and exits straight to the boot point (the same path as run 1's step above, and the commonest episode on the idle card, §8). Run 7 went 800→700→600 MHz within 0.1 s at 87.8 W, and that one cannot be the thermal loop, which steps once per s: in the cards' source only the power loop steps with no delay, so run 7's two steps are most likely the power branch lowering the clock, on passes where the service processor's mean still read 65 (§3; an inference from the source, since our samples there read above both thresholds).

The loop is also slow to start: the first clock change came a median s after a block of launches began ( s over starts on 21 and 23 September; s in the seven runs charted), about passes of the loop as it runs under our 10 Hz sampling (about ms per pass, §1). Once moving, it changed the clock a median s after the previous change ( s for the middle 80%), close to the thermal loop's own period, s (§1), rather than a count of passes. up-steps charted () go straight from 600 to 800 MHz between two 100 ms samples, faster than one table point per pass: the cards' governor climbs to the top point in one call (§1), where the September 2024 rewrite would step one point per pass (§9).

aifoundry1 could not be measured on 22 September. It holds two ET-SoC-1 cards, and the management library refused both (Error unable to evaluate compatibility!). The cause, found on 25 September, was an empty version string in the kernel module built there, not the srcversion mismatch this page first blamed; a rebuild of the driver fixed it that day. Its card 1 (firmware 1.2.0) then took part in the three-card check; its card 0 (firmware 1.4.1) overheats and was left out.

Power idle state event, current pwr 26220  tdp level 0
Power throttle down event, current pwr 35380  tdp level: 0
Power idle state event, current pwr 26100  tdp level 0
Power throttle down event, current pwr 40200  tdp level: 0
Power idle state event, current pwr 26060  tdp level 0
Power throttle down event, current pwr 48470  tdp level: 0

A second, independent reading agrees. get_power_state() classifies the card as MAX_POWER whenever power exceeds the TDP level and MANAGED_POWER when it is merely above the 30 W safe threshold. aifoundry3 reports max_power while sitting at 23 W and 600 MHz, which is only consistent with a TDP level below 23 W; aifoundry2, drawing more, reports managed_power, and so does aifoundry1's card 1 (every pass of the three-card check read the same). Two different code paths, one cause.

The contrast is not that aifoundry2 is always faster. Launched warm, it too runs at 600 MHz: the thermal branch holds it at the floor, and every one of the strict protocol's launches at 80 °C ran at 600 MHz (§1). The difference is the remedy. aifoundry2 reaches 700 and 800 MHz when it starts cool (§2). aifoundry3 cannot at any temperature, because the branch that would raise it is unreachable.

This also bears on what §2 could not settle: whether the power branch works at all. It never fired on its own in any aifoundry2 run. On aifoundry3, in the 22 September window, it was the only branch that fired, at every kernel start, on a die well below the thermal threshold, as the comparison says it will; the three-card check's windows hold no governor line to repeat that, as the cards' source predicts once the governor is stuck (above). That the branch lowers a clock that is above the floor is not shown alone on either card; aifoundry2's fastest drop, two steps within 0.1 s at 87.8 W, fits only its no-delay power loop, and its one-step drop at 87.5 W fits that loop or a thermal episode of zero steps (§2).

That RTL is the Erbium configuration (one neighbourhood, no shire cache), so it says nothing about the L2 and L3 arrays; for those, the wake-up probe below is the only evidence.

// Power Ctrl - power control handled outside cpu subsystem and this ifce is not used
.pwr_ctrl_glb_nsleepin ('1),   // '1 = never asleep
.pwr_ctrl_min_nsleepin ('1),
.pwr_ctrl_min_isolate  ('0),

And the service-processor firmware, searched both at 353f20e and in the older governor, contains no power-gating control of any kind: its only PWR_CTRL hits are an eMMC controller's bus voltage, and elsewhere in the tree the name appears only in eMMC and PCIe PHY register headers. Whatever drives those pins on the taped-out part, it is not this firmware, and the chip-level netlist is not in the open release.

And no card shows any sign of it. If an array were suppressed once idle, the first access afterwards would pay the wake-up. The probe: put one line at a chosen level, let the minion spin on its cycle counter for a swept interval from zero to 27 ms (0, 1.7 µs, 17 µs … 27 ms) without touching that line, then time a single load of it: same line, same address, only the idle time varying, 20 repeats. It ran once on aifoundry2 on 22 September (the chart), and three times on each of the three cards in the three-card check (the table below it). (The probe counts the idle in cycles; the times assume 600 MHz. On 22 September the clock during the probe was not recorded; aifoundry2 was resting at about 73 °C that morning, and in 26 sessions on five days it never ran above 600 MHz at a die reading of 68 °C or more. In the check it was recorded, and read 600 MHz before and after every probe.)

Checks the fit never saw. The idle law above was fitted on aifoundry2 on 21 September to idle readings at °C (the Horace experiment, §8; tabulated in the energy manual, §1). It predicts . . In that one 22 September session, nearly half of the idle, W, is on no rail sensor (energy manual, §1). .

• Kanter's argument that measured power is too expensive to mandate is weaker for this card, though it does not vanish. His reason MLPerf cannot require measured power is that most submitters will not pay the time and energy to instrument a system. This chip is already instrumented, as a load-bearing part of its own safety loop: a PMIC that meters board power, a service processor that reads it on every pass of its management loop, and a management interface that hands the number to the host. The cost of reporting the card's own power here is one query. But MLPerf's measured power is the whole system (host, power supply, cooling), which this meter does not see, and the meter gives one new board reading per service-processor pass, , with the rail readings behind the PMIC's own running average (a time constant of about one second).
• But the loop's hunting is exactly the kind of thing a provisioned-power metric cannot see. A normalisation by provisioned kilowatts scores a card by its nameplate. On aifoundry2 a governor with no dead band moves the clock, and so the throughput, by up to a third within a single 7-second run, invisibly.

• The watch. Each cycle read the service processor's 4 KB trace ring, which holds its governor lines with their own microsecond timestamps, its throttle-state residency counters and its configuration (the threshold read 65 °C in every cycle), then took one 1 s, 10 Hz sample: the 34-sensor mean the governor compares, the lowest and the hottest sensor (the sensors' peak-hold marks, restarted at the start of the sample), board power and the clock. About 1.5 s of card access, no heat, nothing set.
• The heating sessions, each named by the design's branch for the card's reading at its start (NAT-4 is the coolest): .
• A run. Wait for the mean to fall from 63 to 62 °C (a statistics reset at 63 restarts the peak-hold marks, so the hottest sensor's reading afterwards is the hottest since then), launch one kernel for 8 s (7 s in the first block) that multiplies random data on 128 or 192 minions in one of four placements, stop it 2 s after the first 800→700 MHz step, and record the clock, the mean, the hottest sensor and board power at 10 Hz. The placements: B4C@32, the four central shires with all 32 minions each; INT16@N and PER16@N, sixteen interior or sixteen perimeter shires with N minions each; UNI32@4, all 32 shires with four each. The first block used N = 8 (128 minions); none of its four runs reached a step within the launch, so the rule set in advance moved the design to N = 12 (192 minions) for blocks 2 and 3.
• Nothing was set on the card: no threshold, power limit, clock or voltage, and the service processor's log level was never changed. No reset was attempted, including after the hang below.

• On the idle card (read-only watch cycles, no heat): I1, the governor's state against the mean and the hottest sensor (at least 5 cycles, over at least 3 stretches, in which the governor is out of its loop while the hottest sensor reads 66 °C or more in every sample, and no cycle in the loop with the mean at 64 or less); I2, every ENTER prints 66 and every EXIT 65 or less; I3, every EXIT followed by the idle reset within 0.30 s; I4, every ENTER→EXIT interval within 15 ms of a whole number of 0.4053 s periods; I5, ENTER lines on an idle card; I6, the THERMAL_DOWN residency counter against the episodes the lines record.
• Heating sessions on the same card (NAT-4 only, from a rest of 60 °C or less, every development choice fixed): G1-T and G1-H (the step's timing against the mean and against the hottest sensor, and the hot hold); G4-S (the perimeter-against-interior ratio above zero, at 99% over at least 6 blocks); G2-C, G2-D and G2-U (the one-call climb, the 0.4 s descent, and up-steps only from a reading of 66 °C or less); G3-L and G3-I (launch to 800 MHz and kernel end to 600 MHz).
• Reported, not registered: whether the placement effect is the size a heat-only model predicts (): development ran only two blocks under the final settings, where the rule set in advance asked for four.

• Data. Everything here is in docs/reports/data/2026-09-28-dvfs2-aifoundry2: the night's raw record (raw/), the design, the development pre-registration and the frozen validation plan (plan/), the two reductions (reductions/), the kernel-log excerpt (incident/), the validation's raw record and its verdicts (validation/: raw/, verdicts-dv2val.json from the frozen reducer, and its own README) and the page's own reduction, dv2.json, which tools/ettelem/build_dv2_data.py writes and merges into this page's data. Its README says what ran, when and with which commands.
• Definitions. A run's times are from the launch of its measured kernel: the first sample at 800 MHz; the first sample at 800 MHz with the hottest sensor at 66 °C or more; the first sample with the mean at 66 °C or more; and the first 800→700 MHz step (tools/claims-v3/dv2/dv2obs.py, computed live). A run separates the two rules when the hottest sensor reached 66 °C at least 1.5 s before the mean did; a rule fits a run when the step came between 0.3 s before and 0.6 s after that reading's first 66. Time at 800 MHz is the step's time less the first 800 MHz sample; a run that never stepped counts to its kernel's end and is marked as held to the end; a block whose two runs both held is dropped.
• The hottest sensor is the sensors' peak-hold high since the statistics reset at 63 °C: on a die that is heating it is the hottest shire's reading, but the host cannot tell which shire it is, and a brief spike between two governor passes could raise it without the governor ever seeing it.
• Governor lines are read from every ring dump of the night with the validation reducer's parser (tools/claims-v3/dv2v/reduce_val.py, frozen) and mapped to host time through the host's own requests that the rings record (). Intervals between lines are in the service processor's clock.
• One card, one chassis, one night, at the native threshold of 65 °C. aifoundry3's governor is stuck (§3) and aifoundry1's card 1 does not act (§9), so no other card can replicate the governor results; the plan's answer is a replication on aifoundry2 in a later session, which ran from 20:45 PDT on 28 September to 16:57 PDT on 29 September (above). Its verdicts hold for this card in this chassis; the card's warm rest limited what it could decide.

• Which card. A result stated with no card named holds on both aifoundry2 and aifoundry3 (at least three independent repeats on each, or a readout identical on both). The three-card check of 25–26 September (docs/reports/data/2026-09-25-claims-v3, pre-registered) re-measured this page's claims on aifoundry2, aifoundry3 and aifoundry1's card 1; where the text gives its results, they name the card. aifoundry3's clock is held at 600 MHz (§3), so everything about the clock moving (§1's measurements, all of §2, §7's check) is aifoundry2's by necessity; §5's idle law is aifoundry2's too, while §4's wake-up probe now holds on all three cards. Where fewer than three runs or sessions stand behind a number, the text says how many.
• Firmware claims are read from the et-platform source under device-bootloaders/src/ServiceProcessorBL2/ at ffca4cbb4 (cafe03fc3^, "Close development of version 0.20.0", 17 May 2024), the closest public source to aifoundry2 and aifoundry3: services/thermal_pwr_mgmt.c, driver/pvt_controller.c, driver/pmic_controller.c, rtos_task/dm_task.c, include/thermal_pwr_mgmt.h, include/bl2_pmic_controller.h, config/mgmt_build_config.h; at da192816a (0.18.0) for aifoundry1's card 1; and at 353f20e (the same governor files as the head of the tree, 836a4ab) for the rewrite. The reading, line by line, is plan/review-governor.md in the development night's record (§8).
• The cards run the governor of §1, not the September 2024 rewrite. aifoundry2 and aifoundry3 both report firmware release 1.3.1, with service-processor bootloaders 0.20.0 and minion runtime 0.23.0, which date their build to about mid-May 2024 (firmware.md in the check's record). On 22 September the service processor on aifoundry3 printed log lines that exist only in firmware before commit 60b40c10f (24 September 2024), which rewrote this governor, and on 27–28 September aifoundry2 printed that build's "Thermal idle state event" lines, which the rewrite no longer has (§8). Against the rewrite at 353f20e, which earlier versions of this page described: the rewrite acts only while a kernel runs, the cards' thermal trigger regardless; the rewrite's power test reads the PMIC's average, the cards' its instantaneous reading; the rewrite steps one point per pass with no loop, the cards run the blocking thermal loop, exit to the boot point and climb to the top in one call; the rewrite uses neither power guardband, the cards use one on the way out of a step-down; the rewrite logs a request on every pass it acts, the cards once per change of state; and a zero power limit would merely pin the rewrite's clock at the floor, where it stops the cards' governor (§3). The measurements in §2–§7 are unchanged; §1–§3 now explain them with the cards' own source.
• aifoundry1's card 1 runs 0.18.0 (release 1.2.0; closest source da192816a): the same trigger, loop and thresholds, but fixed 50 MHz steps clamped to 300–700 MHz, a voltage computed from the boot point instead of the table, and no check of the table at boot.
• No firmware path limits the die's temperature at the lowest point. At 600 MHz the thermal loop has no point left to step to, and the only hardware trip is the PMIC's alarm at 75 °C and 75 W (bl2_pmic_controller.h:267, 272; set at pmic_controller.c:480–516), which compares the PMIC's own temperature register, not the die sensors. The service processor reads that register only when it starts or resets its statistics (thermal_pwr_mgmt.c:2995–3001 at ffca4cbb4, under the firmware's own note that the PMIC reports the system temperature as 0); on every other pass it feeds the statistic's minimum and maximum a hard-coded 0 (:663–664). So the statistic, which read 0 on all three cards throughout the check, says only that the PMIC gave 0 at each reset, not what the register held in between. Only a runner's own stop guards the die: this work's long runs and the DV2 watcher stop at a mean of 90 °C, but the version-3 catalogue runner had no upper stop, hence the 103 °C pass.
• RTL claims are read from core-et's Erbium branch at b38a1a3: rtl/shire/neigh/neigh_top.v, neigh_top_pwrstub.v, rtl/cpu_subsystem/cpu_subsystem_top.v. That is the same Minion core lineage in a later MCU-class configuration, not the taped-out ET-SoC-1, which is why "nothing drives the sleep pins" is a statement about the open release and this firmware, corroborated by the card, not a statement about the silicon's chip-level netlist.
• The 36 transitions come from aifoundry2's cool-start runs of 21 September, in docs/reports/data/2026-09-21-horace-aifoundry2/cold1 and cold2. The timings quoted over two days add six cool passes of 23 September (docs/reports/data/2026-09-23-reruns-aifoundry2/, the discarded first attempt at E29); a block is launches less than 2 s apart. tools/ettelem/analyze_dvfs.py classifies the steps mechanically; nothing is fitted. The split depends on which sample is read: , because the board meter lags the clock by . Either way the power branch never fired alone on aifoundry2.
• The wake-up probe is gen_ops.py wakeup in workloads/memprobe, one hart, 20 repeats, raw results in docs/reports/data/2026-09-22-dvfs-aifoundry2/wakeup/. It can only detect a wake-up longer than about ten cycles, and only for arrays idled up to 27 ms, which is the widest the delay op encodes. A gating policy that waits longer than that would not show up. The three-card check ran the same probe, with two shorter idles added (0.17 and 0.5 µs), three times on each card, and recorded the clock before and after each (V3-MEM; raw/<card>/mem/p1–p3/wake/). The kernel corrects the cycle counter's late carry itself (Anatomy of a memory access, A timer bug worth knowing), so the analysis uses the stored latencies as they are (signed 32-bit differences); the loads on which that correction misfired read exactly 128 cycles off, and the paired medians the chart draws are the same whether those loads are kept, corrected or dropped.
• The long-idle sample is docs/reports/data/2026-09-22-dvfs-aifoundry2/idle_20h.jsonl.gz: 300 samples over 60 s. The idle time is computed from timestamps, from the end of the last launch of the previous day's long runs to the first sample: it is the time since our last recorded workload. Use of the card in between by anyone else, or by runs that were not committed, cannot be excluded, and the wake-up probe's start time is only in the lab log. The overnight point is the 3.2 s before the first launch of the cool-start session (cold1); the idle-law fit leaves out the first 20 s of every session and ±4.5 s around every launch, so neither point was in it. The session-by-session check in §5 uses every committed session with idle stretches that the law was not fitted on, on both cards (the fit used cold1, cold2, long and strict), and the three-card check's cooling cycles (V3-IDLE; analyze_dvfs.py --v3 reads them with the same rule, restricted to each cycle's 900 s cooling window, and reproduces the check's per-cycle offsets). The split's range refits the fixed part and the leakage on the law's own idle readings with the leakage temperature scale held at each end of °C; those fits leave an rms error of W against the central fit's W, all well inside what whole-degree readings allow. These rms errors are over the whole-degree bin means, weighted by their samples; over the idle samples one by one, the law's own fit leaves W rms, the figure other pages quote.
• The three-machine comparison uses two read-only tools written for it. tools/etcfg issues ETSOC1_IOCTL_GET_DEVICE_CONFIGURATION to the driver; ettelem config asks the service processor for DM_CMD_GET_MODULE_STATIC_TDP_LEVEL, ..._TEMPERATURE_THRESHOLDS and ..._POWER_STATE. Neither writes anything. The firmware log excerpts are the card's own trace buffer, read with ettelem sptrace and stored verbatim in docs/reports/data/2026-09-22-cards/. The release numbers (1.3.1 / 1.5.0) and the 23 W at which aifoundry3 reported max_power were read by hand from ettelem's output and are not in the stored data. On 26 September the three-card check read the same configuration, the driver's view, the firmware revisions and the trace buffer in three passes per card (V3-TEL; raw/<card>/tel/p<N>/gov/ in its record).
• aifoundry3's session is the Horace experiment's strict protocol, launched at (20 runs); raw data in docs/reports/data/2026-09-22-horace-aifoundry3/.
• Not established: whether the taped-out ET-SoC-1 drives its sleep transistors under some other firmware or in some other power state; whether the cards' build matches ffca4cbb4 line for line (the release numbers, the log strings and every timing in §2 and §8 fit it); whether aifoundry3's governor is stuck as §3 infers, and whether aifoundry1's card 1 has active power management off or a state latched early; what the boot frequency is set to in flash (why aifoundry3's TDP is zero is now known: a boot service sets it, §3); whether aifoundry3's lower switching power (its model factor, 0.93–0.95 at its own launch temperature against 0.96–0.99 on aifoundry2, §6) is silicon, package or board regulator, which neither the third card nor the two-temperature test separated; whether each unattributed down-step of §2 is a thermal step or the loop's exit, decided on readings between our samples (the likeliest reading: §8's governor lines show the loop's period and its exits on the idle card, in development); what §8's frozen validation left untested: whether the step follows the mean or the hottest sensor (TH1), whether placement delays it (Q2) and whether the cards run the 0.20.0 loop (TH2), each short of its registered count; how the idle law's power divides between fixed and leakage beyond the range of §5; and why two relay launches on aifoundry2 returned wrong elements (1,118 and 1,000) on 23 September, in one pass of the discarded first attempt at a cool-card rerun. Both ran while the governor dropped the clock from 800 to 600 MHz inside the launch, and the relay's other launches on both cards were correct. An error at the clock transition and a race in the relay's own completion flags when the clock ratio changes would both fit; one pass on one card cannot tell them apart.

To rebuild this report's data and page from the raw files (the wake-up probe itself needs a card):

python3 workloads/memprobe/gen_ops.py wakeup --out W --reps 20 --seed 11 \
    --delays 0,1000,10000,100000,1000000,4000000,8000000,16000000
build/memprobe/host/memprobe_host --program W/wakeup.ops --out-dir W --budget 40

D=docs/reports/data; H=$D/2026-09-21-horace-aifoundry2; A3=$D/2026-09-22-horace-aifoundry3; C=$D/2026-09-22-cards
R2=$D/2026-09-23-reruns-aifoundry2
python3 tools/ettelem/analyze_dvfs.py --cold $H/cold1 $H/cold2 --wakeup $D/2026-09-22-dvfs-aifoundry2/wakeup \
    --idle $D/2026-09-22-dvfs-aifoundry2/idle_20h.jsonl.gz --since $H/long2/runs.jsonl.gz \
    --model $H/model.json --ablation $H/ablation.json --sptrace $C/sptrace-aifoundry3.bin \
    --cool-passes $R2/hotline-pass2 $R2/hotline-pass3 $R2/hotline-pass4 $R2/relay-pass1 $R2/relay-pass2 $R2/relay-pass4 \
    --idle-sessions $H $D/2026-09-2[234]-* --v3 $D/2026-09-25-claims-v3 \
    --out $D/2026-09-22-dvfs-aifoundry2/dvfs.json
python3 tools/ettelem/build_cards_data.py --cards $A3/cards.json --transfer $A3/transfer.json \
    --leak $A3/leakage_crosscard.json --config $C/config.json --driver $C/driver_config.json \
    --sptrace $C/sptrace-aifoundry3.bin --out $C/cards-report.json \
    --merge $D/2026-09-22-dvfs-aifoundry2/dvfs.json
V=$D/2026-09-28-dvfs2-aifoundry2     # §8: the development night; its two reductions first
python3 tools/claims-v3/dv2/reduce_dv2.py --dev --data $V/raw --out $V/reductions/dv2-dev.json
python3 tools/claims-v3/dv2v/reduce_val.py --data $V/raw --dev --out $V/reductions/dev-idle.json
python3 tools/claims-v3/dv2v/reduce_val.py --data $V/validation/raw --out $V/validation/verdicts-dv2val.json   # the validation
python3 tools/ettelem/build_dv2_data.py --data $V --merge $D/2026-09-22-dvfs-aifoundry2/dvfs.json
python3 scripts/build-report.py dvfs-leakage $D/2026-09-22-dvfs-aifoundry2/dvfs.json \
    docs/reports/2026-09-22-dvfs-leakage.html

analyze_dvfs.py carries over the blocks the two merges add, and stops rather than drop a block its options would not rebuild; build-report.py refuses data without the cards and dv2 blocks.

Versions. 22 September 2026: first published for aifoundry2, then the three-machine comparison. 24 September: the governor's pass is about 133 ms, not 10 ms; it hunts on the thermal threshold, not the power guardband; the open RTL is Erbium, not the chip; seven down-steps are unattributed. 25 September: the rule's frequency check sits inside the thermal test, and the DRAM wake-up counts corrected; then version 3: the leakage shares as ranges, and "every result checked was correct" withdrawn. 26 September (version 4), the three-card check: aifoundry3's zero TDP is set by a boot service, not flashed; the busy leakage share is no longer said to exceed 30%. 27 September: the review's fixes (verdict row 6 cut) and charts. 28 September: §1 rewritten for the governor the cards run (firmware 0.20.0), with §2's steps explained by it; aifoundry3's zero limit stops its governor and aifoundry1's card 1's governor is inactive; no firmware temperature limit at the lowest point; a new §8, the night of 27–28 September; later, the review's cuts; then the card's restore at 08:32, and aifoundry3's query and placement runs pointed to their data. 29 September: §8's validation, started at 20:45 PDT on 28 September, its three heating sessions done and its reduction due after about 16:45 PDT; that evening, its verdicts (TH3, TH4 and TH8 survived, TH7 fell, and the owner's two questions stayed untested), the validation's runs in the charts of the two questions, and "seven complete placement blocks" corrected to four. What each version changed in full: this page's history in the repository.