midi-harbor/specs/008-resilience/research.md

# Research: Resilience

Entries from the research log, under their original numbers. Findings marked **VERIFIED** were
proven in this repository; **ASSUMED** ones rest on the literature.

---

## R-010: Detecting sleep/wake and network changes

**Decision**: Platform-specific event sources behind a single `SystemEvents` trait, **backed by a
platform-independent heuristic**.

| Event | macOS | Linux |
|---|---|---|
| Sleep / wake | IOKit `IORegisterForSystemPower` | `org.freedesktop.login1.Manager.PrepareForSleep` over D-Bus |
| Network change | `SCNetworkReachability` / `nw_path_monitor` | netlink `RTMGRP_IPV4_IFADDR` / `RTMGRP_IPV6_IFADDR` |

**Status**: **ASSUMED**; moderate risk.

**Rationale**: These are the events that drive FR-024 and FR-025. The important finding is that
**the platform APIs are a convenience layer, not a correctness layer** — a machine can suspend
without the API firing, and logind is known not to emit `PrepareForSleep(false)` when resuming from
hibernation. Relying on them alone would leave exactly the silent-dead-connection failure this
product exists to eliminate.

**Therefore**: the authoritative recovery mechanism is the per-session liveness check — missed
clock-synchronisation responses on RTP-MIDI, and characteristic notification silence on BLE. The
platform events are an *optimisation* that makes recovery near-instant instead of waiting for a
timeout. Correctness must not depend on them, and the test suite must prove recovery works with the
platform event source disabled.

A third, cheap signal worth adding: a monotonic-versus-wall-clock gap detector. A task that ticks
every second and observes that substantially more wall-clock time passed than expected has almost
certainly just come back from suspend, on any platform.

**Alternatives considered**: `choreo-power-events` — a convenience wrapper over the same APIs;
reasonable, but the trait boundary is needed regardless and the dependency adds little. Polling
interface lists — too slow, too coarse.

---

## R-031: The comment said it silenced; the code did not

**Status**: **VERIFIED** (2026-09-20) on macOS over CoreMIDI and Linux over ALSA.

`delete_virtual_port` carried the comment *"Silence before tearing down, so nothing is left
sounding on the far side"* directly above the teardown, and the CLI help said "Delete a port,
silencing it first". Neither did. Disabling a port did not either, and stopping the daemon did
not: a held note sounded on whatever received it until something else stopped it, and nothing
else would — the only thing that knew a note was playing had just exited.

`EventKind::NotesSilenced` had been defined, given a wire name, and never emitted.

**Per channel, not all sixteen.** Sending a reset on every channel is the usual approach and is
what the network session path already did, but a virtual port is shared: resetting a channel
nothing was playing clears sustain and cuts sound for whatever else is driving that port.
`core::sounding` keeps a bit per note per channel, updated from what leaves an endpoint, so
silencing touches only the channels that were actually played and an idle port is left alone.

The record is updated from outbound traffic only. A note sounds on whatever *received* it, so the
endpoint that sent it is the one that will have to stop it.

**Measured**: a held note, then `port disable`, produced sustain-off, all-notes-off and
all-sound-off on channel 1 alone, on both platforms. Repeating with `SIGINT` to the daemon
produced the same three, before the socket was removed.

**Not covered**: a device unplugged mid-note. The endpoint that went away cannot be told
anything, and the notes are sounding on its route destinations, which other sources may also be
playing. Stopping only what came from the removed source needs per-source attribution, which is
route-suspension work (FR-032).

---

## R-033: Nothing was watching the machine

**Status**: **VERIFIED** (2026-09-20) by stepping a real machine's wall clock.

The `SystemEvents` seam existed, a fake existed, `Event::Nudge` existed in the connection state
machine with a doc comment describing exactly what it was for, and `on_retry` already reset the
backoff when nudged. Nothing produced a nudge. No backend existed on either platform and nothing
in the daemon consumed the seam, so a machine waking from sleep was invisible.

**The number that makes this matter**: the backoff caps at thirty seconds. A session that was
retrying when the lid closed waits up to thirty seconds after it opens — against SC-003's ten.
The mechanism to avoid that was fully built except for the part that notices.

**Why the clock gap rather than IOKit and logind first.** R-010 already records that the platform
power APIs miss suspends. Comparing the monotonic clock against the wall clock catches every
suspend on both platforms with no bindings and no privileges, because both platforms stop the
monotonic clock while suspended and keep the wall clock running: the difference is how long the
machine was away. The platform backends remain worth adding — they report a suspend *before* it
happens, which this cannot — but they are an addition to this, not a replacement for it.

Both failure modes are benign, which is what makes a heuristic acceptable here. A false positive
costs one reconnection attempt that was going to happen anyway. A missed detection costs the wait
this removes, which is what happens today. Neither can lose MIDI.

Nudges are ignored by sessions that are connected or switched off, so a hint arriving while
everything is fine cannot turn into a dropout.

**Measured**: stepping the wall clock forward twenty seconds on the Linux machine produced
`away_seconds=19` in the log and a `system_resumed` event, through the whole chain from detector
to event log.

**The network half, by the same reasoning.** `NetworkChanged` is produced by comparing the set of
addresses this machine answers on, loopback excluded. An address appearing or disappearing is how
a network comes back without anything announcing it — a cable plugged in, a laptop joining a
different network, a lease renewed on a different address — and a session bound to the old address
never recovers by waiting. `if-addrs` was already in the dependency tree through `mdns-sd`, so
this costs nothing in the headless build.

The platform-native sources are still worth adding, for the same reason as the power ones: they
report the change the instant it happens rather than within a second, and they can say what
changed. They are an addition, not a replacement (T097, T098).

**Measured**: adding an address to a throwaway interface on the Linux machine produced
`addresses=30` and a `system_resumed` event; removing it produced `addresses=28` and another. The
event detail names which change prompted the reconnection, because a reconnection that happens on
its own reads as a spontaneous dropout otherwise.

---

## R-034: A suspended session looks exactly like a working one

**Status**: **VERIFIED** (2026-09-20) between the macOS and Linux machines, measured both ways.

R-033 framed the wake case as a session waiting out a thirty-second backoff. That was the wrong
mechanism, and the nudge built for it did nothing at all in the case it was named after.

After a suspend, a session is still in `Connected`. Nothing ran while the machine was away, so
nothing failed and nothing was scheduled. The nudge only acted on sessions that were retrying, so
it skipped the one case that mattered. What actually notices is `LIVENESS_TIMEOUT`, which is
**thirty-five seconds** — more than twice SC-004's budget, and the backoff never enters into it.

Thirty-five seconds is right for an ordinary quiet session: a link that is merely idle is not a
link that is broken, and a shorter timeout would drop working sessions. What was missing is that
a hint changes the question. After a wake or an address change the link is *suspect*, and the
right move is to ask the peer now and give up in seconds rather than in half a minute.

`Session::probe` sends a clock exchange immediately and sets a two-second deadline. An answer
clears it, so a working link survives any number of hints. Repeated hints do not push the deadline
out, or a machine reporting changes steadily would never let a probe conclude — which is not
hypothetical: adding one interface produced three address changes in three seconds.

**Measured**, by dropping the session's UDP between the two machines so the link died silently:

| | time to notice |
|---|---|
| with a hint to prompt a probe | **3.8 s** |
| with nothing to prompt one | **37.2 s** |

The second number is the behaviour every release before this one had, and it is what SC-004's
fifteen seconds was being missed by.

---

## R-035: A thirty-second retry ceiling cannot meet a ten-second promise

**Status**: **VERIFIED** (2026-09-20) by simulation over the real backoff and measured between the
two machines.

Writing the test for SC-003 was what found this. An interruption that leaves this machine's own
addresses alone — an upstream router rebooting, a switch power-cycling — reports nothing when it
ends. The address watch cannot see it and the peer cannot tell us. The only thing that finds out
is the next scheduled retry, so the ceiling on that delay *is* the recovery time.

The ceiling was thirty seconds, with full jitter. Simulating a hundred interruptions over the real
backoff: **53 of 100 took longer than ten seconds**, some over twenty-five. SC-003 asks for at
most one in a hundred. It was never close.

Thirty seconds is a sensible default for a retry that costs something. A network session's retry
is three small packets, and waiting half a minute between them saves nothing worth having while
being most of the time a user spends wondering why their keyboard stopped working.
`BackoffPolicy::responsive` caps at five seconds and is what network sessions use; the default
stays for everything else.

**Measured** on the two machines, blocking the session's UDP without touching any address: the
session gave up, retried through a twenty-second outage, and reconnected **2.8 seconds** after the
block was lifted.

The division of evidence is deliberate. The distribution — "at least 99 of 100" — is a property of
the jitter and the ceiling, which a simulation over the real `Backoff` can assert every run. That
one interruption recovered in 2.8 seconds is evidence the mechanism works end to end, not evidence
about the ninety-nine.

---

## R-044: A flapping link was retried four times a second, forever

**Status**: **FIXED** (2026-09-21) in the state machine, with the delays tested against an
injected clock.

The spec's rapid connect/disconnect edge case asks two things: the backoff keeps a flapping link
from consuming resources, and the user can see that the link is unstable. The state machine did
neither. It reset the backoff on every successful connection, so a link that came up and dropped
at once went straight back to the shortest delay. Its eight retries were spaced at 250 ms, 250 ms,
250 ms and so on, which is what the regression test prints when the old reset is put back. Nothing
distinguished it from a link that had simply failed once.

A connection now resets the backoff only once it has stayed up for `STABLE_AFTER` (10 s). Until
then the backoff carries on growing, so the same eight flaps end with retries 16 s or more apart.
After `UNSTABLE_AFTER` (3) brief connections in a row the state reports the link as unstable,
until a connection lasts. `ConnectionState.unstable` carries this over the wire (field 6, added
within `midiharbor.v1`). The CLI shows `connected (unstable)`, and the GUI says it keeps dropping
within seconds of connecting.

Two further findings:

- **Bluetooth reconnected on sight.** A remembered device was reconnected every time it was
  heard, whatever its backoff said, so the damping above never reached it. It now waits out the
  backoff. Simply ignoring an early sighting would be worse than no damping, because the radio
  may not report the device again. So an early sighting schedules the reconnect for when the
  backoff ends. The test flaps a device until it is unstable, then brings it back once, and
  requires that single sighting to reconnect it. Dropping early sightings instead fails it.
- **The session supervisor had a second backoff that decided nothing.** It was reset on
  connect, on nudge and on success, and never asked for a delay. Retry timing always came from
  the state machine. It was removed.

**Not covered**: hardware that flaps by being unplugged and replugged gets a fresh state on each
arrival, because the operating system decides when it comes back, so there is nothing to space
out and no history to call it unstable by.

---

## R-045: "No network" and "the peer is not answering" were the same message

**Status**: **VERIFIED** (2026-09-21) on Linux, with one daemon inside `unshare -n`, a network
namespace with no network at all, and one outside it.

The spec's no-network edge case asks that sessions report waiting for a network rather than
repeatedly reporting connection failures. Every unanswered invitation was reported as
`network unreachable`, with a warning for each retry. That happened with no network at all, and
equally on a working network where the peer was simply not running.

**Deciding that there is no network.** Counting the machine's addresses does not work: an
interface that is up carries an IPv6 link-local address with no network behind it, and on macOS
tunnel interfaces carry one even with Wi-Fi off. What is reliable is the kernel's own answer. A
send with no route is refused at once with `NetworkUnreachable`, `HostUnreachable` or
`AddrNotAvailable`, before anything reaches the wire. The session marks itself waiting for a
network when that happens and clears the mark on the next send that goes out.
`ConnectionState.waiting_for_network` carries it (field 7, added within `midiharbor.v1`).

| | Before | After |
|---|---|---|
| No network (`unshare -n`) | `retrying`, `network unreachable`, a warning per retry | `waiting for network`, logged once |
| Network up, peer not running | `network unreachable`, a warning per retry | `peer not responding`, one info line |

The `PeerTimeout` message changed from "peer stopped responding" to "peer not responding", which
also fits a peer that never answered. The slug, which is the contract, is unchanged. A lost
connection still warns every time; only repeated failed attempts went quiet.

---

## R-050: One endpoint being retried stopped MIDI everywhere

**Status**: **FIXED** (2026-09-21), for T071a (FR-029).

FR-029 requires healthy connections to carry on while another is failing or retrying. Network
sessions and Bluetooth links always ran on tasks of their own. Virtual ports and hardware that
failed to open were retried by one loop in `supervisor.rs`, and that loop broke FR-029 three ways:

- **It held the daemon's lock through the platform open.** Every route takes that lock to find
  its destinations, so a device held by another application, retried every few seconds, stopped
  all MIDI in the daemon for as long as each attempt took to answer. With opens that took two
  seconds, a healthy route between two other ports stalled for 1.98 s. The lock is now taken to
  claim the attempt and to record its outcome, and released in between.
- **It ran the open on a runtime worker.** A backend's open waits for its platform thread to
  answer. Waiting on a worker held up whatever task tokio had queued behind it there, which no
  idle worker can take over. With the lock released but the open still on a worker, the
  daemon's own tasks and the test's alike stalled for the full two seconds. Opens now run through
  `spawn_blocking`.
- **It retried one endpoint after another.** Three devices slow to answer took 4.7 s to come
  back, against 1.5 s for one. A fault in one attempt would also have ended retrying for every
  endpoint. Each attempt now runs on its own task.

Releasing the lock opens a window: the endpoint can be switched off, deleted or reopened while
the platform answers. Switching it off or deleting it removes its runtime, and switching it back
on starts a new one. The outcome is therefore recorded only onto the same attempt's runtime,
still connecting with nothing open. Otherwise whatever the attempt opened is closed again.

`crates/daemon/tests/isolation.rs` covers all three defects and the window:

- a healthy route stays under 250 ms throughout two seconds of slow retries;
- a device switched off mid-retry is not left open;
- three slow devices come back in parallel.

The fake platform gained `delay_opens` to make opens slow. Each test fails with its fix reverted.
One case is not induced: a re-enable whose own open finishes before the retry's. A uniform delay
cannot order the two that way, so the guard against it is reasoned rather than tested.

Two other opens are unchanged:

- `reconcile` still opens virtual ports under the lock. It runs when the configuration changes,
  which is a user acting rather than an endpoint failing.
- Hardware plugged in is opened outside the lock but on a worker.

Both happen once per change rather than every backoff step, and a real backend answers in
milliseconds. They matter only if an open ever hangs.

---

## R-051: Releasing held notes before the machine sleeps

**Status**: **VERIFIED** (2026-09-21), for T097 and T098, with a real suspend on each platform in
R-070 and R-072.

R-033 added the polled watcher, which finds a suspend only once the machine is back. That is
enough to reconnect (FR-024), but too late for one thing: a note held on another machine. When
this machine sleeps, the far side hears nothing more. It releases the note only when its own
liveness check gives up, more than half a minute later. Only the platform says a suspend is
coming, and both platforms will wait for a process that asks them to:

| | Announcement | Hold |
|---|---|---|
| Linux | logind `PrepareForSleep(true)` | a `delay` inhibitor lock, released by closing its descriptor, bounded by `InhibitDelayMaxSec` (5 s by default) |
| macOS | IOKit `kIOMessageSystemWillSleep` | answering with `IOAllowPowerChange`; an unanswered message holds the machine for 30 s |

On a suspend still to come, the daemon silences every route, network sessions and Bluetooth
links included. It then calls `SystemEvents::ready_for_sleep`, and the backend lets the suspend
go. Each backend also lets it go after `READY_BOUND` (3 s) on its own: a stalled daemon must not
keep a machine awake. The polled watcher reports a suspend only paired with its resume, after the
fact, and the daemon ignores a suspend that its resume already follows. Releasing notes on waking
could cut one someone has just started playing.

Both backends run beside the polled watcher (`CombinedSystemEvents`), never instead of it, since
R-010 records that both platforms miss suspends. They report sleep and wake only. Network changes
stay with the polled address watch, which finds them within a second. A native source would save
that second and nothing more.

**Measured on Debian 12**, with the real binary and a scratch configuration:

- `systemd-inhibit --list` shows `Midi Harbor … sleep … delay` while the daemon runs, and
  nothing after it exits.
- A forged `PrepareForSleep(true)` sent with `dbus-send` is ignored: only logind's own bus name
  is believed.
- With that sender check removed in a throwaway copy, the same forged signal went through the
  whole chain:
  - the daemon logged `releasing held notes`;
  - the lock was gone within two seconds;
  - `PrepareForSleep(false)` took it again.
- The check restored, the forgery is ignored once more.

**On macOS**, `registers_for_power_notifications` confirms that IOKit accepts the registration,
which is the step that fails if the declarations are wrong. Answering the messages has only been
reasoned through: every `kIOMessageSystemWillSleep` path reaches `IOAllowPowerChange`.

`crates/daemon/tests/suspend.rs` covers the daemon's half. A note held through a loopback
session is released at the far synth on a pending suspend, and the platform is told it may
proceed. A suspend already over releases nothing. Each fails with its rule reverted.

---

## R-067: The 24-hour soak, and a session between the two machines

**Status**: **DONE** (2026-09-22).

**The soak (T148, SC-007)** ran on the Linux desktop for a full day, from `0206fa6`: two daemons in
one process joined by a session, with faults injected in turn. It finished with

```text
done: 24278 faults over 86400.134791996s, every one recovered, slowest 2.558007105s,
11511276 messages played
```

Every fault recovered, and the slowest took 2.56 s against SC-003's ten seconds. The run does not
cover the session work done the same day, which landed after it started.

**A session between this Mac and the desktop**, over the LAN, with today's build on both:

- A 3002-byte dump and a note behind it went from the Mac's port to the desktop's, arriving whole
  and in order, and then the same in the other direction, checked byte for byte (R-066).
- The Mac disconnecting left both sides listening. The desktop did not invite the Mac back, and
  its history reads "MacLink ended its session with PCLink; PCLink is listening again".
- On the desktop, nothing was remembered but the kernel's own Midi Through port: the `aseqdump`
  and BlueZ clients that had opened ports during the day left no entry behind.

**A second soak** (2026-09-23 to 24) covered the session and routing work since. The first
attempt, from before `c4f3c1a`, failed in hour 22: at cycle 22,579 a keyboard replug was not
recovered within ten seconds, and the run printed nothing that said why. `c4f3c1a` made an
unrecovered fault print each side's session phase, attempt and peer, every route, the last 30
events and the keyboard's handle. The run from `c4f3c1a` on the Arch VM then finished clean:

```text
done: 24654 faults over 86402.498228452s, every one recovered, slowest 3.215248496s,
10316356 messages played
```

The hour-22 failure did not recur, so its cause is unknown. If it comes back, the diagnostics
will say what state it was in.

---

## R-069: Roaming between two access points

**Status**: **DONE** (2026-09-22), for T154 scenario 2. The session survived one roam and
reconnected a second after the other. No note was left sounding. The history blamed the peer
for a network this machine had lost, which is fixed.

The owner has two access points with the same network name and authentication. A session ran
from this Mac, on Wi-Fi, to the Linux desktop on the LAN, today's build at each end. The desktop
routed the session to a virtual port whose output `aconnect` fed back into its input, so whatever
the Mac sent came back. A CoreMIDI client on the Mac sent a numbered message every 50 ms and
logged when each returned. A CoreWLAN watcher logged the Wi-Fi channel, since macOS hides the
access point's identity from unprivileged tools. Each roam was forced by restarting the access
point the Mac was on. The Mac kept 192.0.2.10 throughout, so neither roam tested a change of
address.

**First roam, pitch bend.** Messages sent from 14:40:40.95 to 14:40:46.45 were lost, 103 of
them, and the Mac was on the other access point (channel 36 to 157) at 14:40:46.65. The session
never dropped, and messages came back from the moment the Mac rejoined. The round trip, 7 ms
before, ran up to 180 ms for about half a minute on the new access point and then settled. The
counters put almost all the loss on the Mac's side: 4,565 sent, 4,467 received by the desktop.
Every packet in this stream carried a newer pitch bend than its journal, so recovery correctly
applied nothing, and this pass shows only that the session held.

**Second roam, notes.** Each 50 ms tick turned the previous note off and the next one on.
Messages stopped coming back at 14:48:02.86. The Mac left its access point at 14:48:13, lost its
addresses at 14:48:16, and joined on channel 40 at 14:48:35.
The history read:

```text
14:48:16  system_resumed          this machine's addresses changed; reconnecting
14:48:18  endpoint_state_changed  MacLink lost its connection to PCLink: peer not responding
14:48:36  system_resumed          this machine's addresses changed; reconnecting
14:48:37  endpoint_state_changed  MacLink connected to PCLink after 3 failed attempts
```

The session reconnected 1.0 s after the addresses came back, well inside SC-003's ten seconds.
Dropping the session released the notes sounding through it. At the end, of 9,795 notes played
and 9,198 heard back, none was left sounding.

**The lost connection blamed the peer.** At 14:48:15.6 the session logged that it had no network
to reach the peer by. Three seconds later its check on the peer timed out and was recorded as
"peer not responding". A timeout was always reported as the peer's; only unanswered invitations
took the missing network into account. A check that timed out with nothing able to leave the
machine is now "network unreachable" as well.

**Not covered:** a roam that changes the address, which needs two networks, and messages the
network loses during an outage. Those are only as recoverable as the journal makes them, as in
SC-006.

---

## R-070: Sleep and wake, and a session that carried nothing

**Status**: **DONE** (2026-09-22), for T154 scenario 2. A session from this Mac to the Linux
desktop comes back within half a second of the Mac waking, and no note is left sounding. The
first run found a session that said it was connected and carried nothing, and the desktop kept
waking the sleeping Mac. Both are fixed.

The setup was R-069's: the Mac, on Wi-Fi and mains power, invited the desktop, which sent back
whatever arrived. A note played every 50 ms, each turning the previous one off. The owner slept
the Mac from the Apple menu and woke it by hand. `pmset -g log` gave the Mac's own sleep and wake
times.

**The Mac did not stay asleep.** Each time, about 32 s after the Mac went to sleep, the desktop's
liveness check gave up on it and the desktop invited it again. Within a second the Mac woke
briefly, a "DarkWake" that `pmset` puts down to `wifibt`. The session reconnected, the Mac slept
again about 45 s later, and the cycle repeated. The first run woke five times this way, between
15:08:10 and 15:13:27, and the second once, at 15:30:46. Every one came a second after the
desktop's invitation. So a Midi Harbor peer keeps a sleeping Mac on mains power awake for about
45 s in every 80. The desktop keeps inviting because a session whose peer went quiet chases
it, whichever side invited: only a peer that says goodbye is let go.

**A session that said it was connected carried nothing.** After the owner woke the Mac at
15:14:03, both ends read connected. The Mac sent a note every 50 ms, and the desktop's received
count stayed at 3,706. Clock exchanges went both ways, so neither side timed out. The Mac's
packets carried sequence `0x72af` and a journal checkpoint of `0xab65`, which is ahead of
anything that session had sent. At the last dark wake, the desktop's fresh session had taken in
MIDI still coming from the Mac's old session, whose sequence numbers were near `0xab65`, because
a session took MIDI from any source. The Mac then began a new session from a random sequence
14,500 packets behind that. The desktop dropped every one of those packets as late, and would
have gone on dropping them for about twelve minutes. Its feedback acknowledging `0xab65` also
trimmed the Mac's new journal. A session now takes MIDI and feedback only from the source agreed
in the handshake.

**On the fix,** the second run slept at 15:30:13. The desktop re-invited at 15:30:45, and the Mac
dark-woke and reconnected at 15:30:46. MIDI was back 31.5 s after it stopped, and it carried on
through the owner's full wake at 15:31:05. The only note sounding afterwards was the one playing.
SC-004 asks for a reconnect within 15 s of waking, and both runs reconnected within a second of
each wake.

**Saying goodbye before sleep.** The owner chose to have a machine end its sessions as it goes
to sleep, so the peer stops inviting it. The first try sent an ordinary goodbye and reconnected
after five seconds awake without a wake notice. It failed twice on the Mac:

- macOS went on running for five seconds after the notice of sleep, and a slow Bluetooth
  acknowledgement for another one and a half. The fallback counted that as time awake and
  reconnected just as the Mac went down, so the desktop woke it 35 s later. The fallback is now
  sixty seconds awake. Normally the wake notice reconnects at once.
- When the desktop had made the connection, an ordinary goodbye is what a restarting peer sends,
  so the desktop invited again 0.3 s later. The Mac, not yet asleep, accepted. Who made the
  connection also drifts: whichever side re-invites after a timeout becomes the one that made it.

The goodbye before sleep now carries `SLEP` in its token field, which other implementations
ignore on a goodbye. A Midi Harbor peer that receives it goes back to listening, whichever side
made the connection, and the sleeping machine reconnects on waking. Checked with the desktop
having made the connection: the Mac slept at 17:23:43 for 62 s. `pmset` shows no wake in that
time, and the desktop's history reads "MacLink ended its session with PCLink; PCLink is listening
again". The owner woke the Mac at 17:24:45, and the session was up 0.4 s later. Apple and other
implementations see an ordinary goodbye and may still invite a sleeping Mac.

---

## R-072: Sleep on Linux, and NetworkManager going first

**Status**: **DONE** (2026-09-22). A Linux machine going to sleep now tells its peer, as the Mac
does (R-070). Before, NetworkManager had the network down before the daemon looked.

An Arch Linux virtual machine slept through logind with
`systemctl suspend`, into s2idle. An RTC alarm did not wake it from there, and a key press through
`virsh send-key` did. A Midi Harbor session on the VM was connected to one on the Mac, by address,
and `tcpdump` on the Mac recorded what reached it.

**What happened first.** logind announced the suspend at 18:38:58.917, and NetworkManager, reacting
to the same signal, took `eth0` down 0.2 ms later, finishing in 31 ms. The daemon holds a delay
lock, so logind waited for it, but it looked for events only once a second. It saw the suspend at
18:38:59.19 and found no network to send the goodbye by, so the Mac heard nothing. The Mac timed
out 32 s later ("peer not responding") and invited the sleeping machine until it came back. The
notes released before sleep went nowhere for the same reason, so on a machine running
NetworkManager, notes held on the peer sounded until its liveness check gave up.

Worse, NetworkManager's teardown arrived in the same batch as an address change, and the daemon
read that as a reason to reconnect. The session just ended for sleep was reconnected at once, on
the way down.

**The fix.** A backend that reports a suspend now signals the daemon, which acts on it at once
rather than at its next look. The watcher also remembers that the machine is going to sleep and
ignores address changes until it wakes, or until it has been awake as long as the sessions wait
(R-070). On the next suspend the daemon ended its session at 18:46:58.7111, 0.2 ms before
NetworkManager logged the request. The Mac, which had made the connection, recorded "VMSide ended
its session with MacSide; MacSide is listening again", and nothing reached the VM until it woke 50
s later. It reconnected 0.8 s after waking.

This is a race the daemon now wins by a fraction of a millisecond, not an ordering anything
guarantees: NetworkManager does not wait for other holders of the delay lock. A slower machine may
still lose it now and then, and the peer then falls back to its liveness check as before.

---

## R-074: Restoring controller state on a recovered link

**Status**: **DONE** (2026-09-22), for T155 (FR-027, constitution Principle I). A converge pass
found that the connection state machine asked for a restoration on every recovery and nothing
acted on it. A fresh session starts with an empty recovery journal, so a volume moved while a link
was down stayed stale at the receiver, and a Bluetooth synth switched off and on came back at its
defaults.

`midi_harbor_core::controls::Controls` keeps, per channel, the last value of every controller
below the channel-mode range, the program, the pitch bend and the channel pressure an endpoint was
sent. On recovery it produces, per channel, bank select, then the program it qualifies, then the
other controllers in order, then pitch bend and pressure.

Decisions:

- **Where it is kept.** A network session's supervisor records everything routed into it, and
  keeps the record across attempts, since routes to a session stay live while it reconnects. A
  control moved during an outage is therefore restored. A Bluetooth link's routes are suspended
  while it is down, so its record holds what was last delivered: a device that lost its settings
  gets them back, and a change made during the outage does not reach it.
- **What is left out.** Notes, which FR-026 silences and which must never be replayed; the
  channel-mode controllers (120 and up), which are commands, not state; and data entry,
  increment, decrement and the RPN and NRPN selectors (6, 38, 96 to 101), which only mean
  something in sequence. Replayed in numeric order, data entry would land on whichever parameter
  the later selector chose. A stale pitch-bend range is the lesser harm than a wrong one.
- **Reset all controllers** clears the channel's record except its program, which it does not
  reset.

Tests: `crates/core/src/controls.rs` for the rules, `crates/daemon/tests/restoration.rs` for a
fader moved while a session was down, and a Bluetooth test for a device that drops and returns.
Each fails with the restoration switched off.

---

## R-079: Hardware on real Macs, and a MIDI server that dies

**Status**: **CLOSED** (2026-09-25). Recovery on macOS is built and checked (T192). Linux has
nothing to recover from, and its checks with the pad controller turned up four smaller faults, all fixed
(T193 to T195). Replacement under an installed launchd service was checked on 2026-09-25, with
the owner's permission, and works.

Two Macs, then one Linux machine, were checked with real USB MIDI hardware.

A Mac running a presentation program, with a generic USB MIDI interface (maker "Generic", model
"USB Midi ", with a trailing space) that has nothing cabled to it. A scratch daemon with its own
config and socket, built without Bluetooth so no permission prompt reached that screen, found the
interface, read its maker and model, bound routes to it by name despite the trailing space,
refused a two-route loop, carried 8 messages out to it, kept its virtual ports' CoreMIDI ids across
a restart, and paused and resumed its routes across `device disable` and `device enable`. Two
CLI labels were wrong there: the program's Apple network session read "physical", and a
switched-off device read "attached". Both are fixed (they read "provided" and "disabled").

The development Mac, with a USB pad controller.
Pads routed to a virtual port arrived through the route in 0.26 ms median, 6.3 ms worst, with
every note-on and note-off accounted for. A first count seemed to lose note-offs; the listener
read only the first message of each CoreMIDI packet, and a corrected listener saw 5 of 5.

The first unplug crashed Apple's `MIDIServer` (SIGSEGV in `MIDIObject::GetIntegerProperty`,
under `MIDIGetNumberOfSources` and `ObjectTreeCache::GetObjectTree`, serving a client's request
for the object tree while the device was being torn down). The daemon re-reads the device list
the moment a removal is reported, so it may have been the client that asked; the notification
logger that also enumerated inside its callback did not crash the server on a later unplug, so
this is a race, not a certainty. launchd restarted the server, and the daemon never recovered:
the device stayed absent after it was plugged back in, and creating a port failed with
OSStatus -50.

What a client sees when `MIDIServer` dies, measured by killing it (`killall MIDIServer`, which
runs as the user):

- No notification of any kind.
- Every call that reaches the server fails from then on: a property of the client's own endpoint
  and creating a port return -50, finding its own endpoint by unique id returns -10842.
- Sending on its own virtual source returns 0 and goes nowhere.
- Cached counts, such as the number of sources, stay at their old values.
- Creating a new client in the same process also returns -50, and still did 150 s later with a
  new `MIDIServer` running and serving other processes.

So nothing inside the process can recover. The daemon has to notice by asking, since nothing
tells it, and has to be replaced by a new process. The owner accepted recovery taking longer than
the usual five seconds for this case, as long as it comes back without anyone stepping in.

**The recovery (T192).** Every two seconds the CoreMIDI thread creates and drops a private output
port; a failure means the server is gone. The daemon then stops serving, silences what it can and
ends its network sessions, and replaces itself with the same executable and arguments through
`exec`, so the process identifier launchd or systemd knows does not change. The new process
reads the configuration, recreates the ports with their pinned identifiers, reopens the hardware
and routes, and records a warning saying why it started. If the server is not answering yet, it
waits up to thirty seconds for it rather than failing. Killing `MIDIServer` under a scratch
daemon with the pad controller routed:

- The probe noticed 2.8 s after the kill, and the new process was serving 0.4 s later.
- The virtual ports came back with the same CoreMIDI identifiers, and a note crossed a routed
  pair of them.
- The owner's pads came through the route, 5 note-ons and 5 note-offs of 5, and after an unplug
  and replug, 11 of 11. That unplug did not crash the server; the crash is a race, which is why
  killing the server is how this is tested.
- The window showed "cannot be reached" while the daemon was down and filled in again by itself.

**Stopping with the window open never finished.** Found while checking the window: stopping
waited for every connection to close, and a window keeps streams open that never close by
themselves. A daemon asked to stop, by Ctrl-C, by `service stop` or to be replaced, kept running
until it was killed, and a kill skips silencing held notes. Stopping now waits two seconds for
requests already being answered and then closes the remaining connections. With the window open,
a stop request now ends the daemon in 2 s.

**Under launchd.** The app bundle was built from the current code, copied to `~/Applications`,
and its binary ran `service install --start` on the development Mac, with a fresh configuration
holding one port, "Launchd Check". `killall MIDIServer` at 18:45:47 UTC. The probe reported the
server gone at 18:45:51.46, the daemon logged that it was restarting to reach it, and the new
process was listening at 18:45:51.96 with the port open again. launchd listed the same process
identifier, 67582, and a last exit status of 0 before and after: it saw no exit and started
nothing. The daemon started exactly twice in its log, and the history read "restarted because the
MIDI server stopped answering; ports, hardware and routes were set up again". The service,
configuration, log and app copy were removed afterwards.

**Linux, with the pad controller.** On an EndeavourOS machine (kernel 6.18), a headless build
from the Arch VM ran as a scratch daemon with its own config and socket. Nothing was built on
that machine and no kernel module was touched.

- Pads routed to a virtual port arrived identical and in order, 9 of 9 notes, under 1 ms behind
  a listener on the device itself (shell timestamps, which resolve no finer).
- Seven unplugs. Each time the device was reopened about 10 s after it went away, the route went
  to waiting and back, and notes came through afterwards.
- A pad held through an unplug was silenced 1 ms after the daemon saw the removal.
- There is no server to lose. The ALSA sequencer is part of the kernel, not a process that can
  crash and be restarted under a client. Unloading `snd-seq` is the nearest equivalent, and it
  cannot be done while PipeWire holds it, so it was not tried.

Four faults turned up there:

- **`Midi Through` read "physical".** A client counted as hardware when its number was below
  128. `Midi Through` is a kernel client numbered among the sound cards, with no card. Belonging
  to a card is now what makes a client hardware (T193).
- **Other programs' ports were described as hardware.** `aseqdump` was logged as "opened
  attached hardware", and a routed application port that closed would have been recorded as
  "unplugged". Those now read "provided port", "went away" and "is back" (T193).
- **Every replug zeroed the device's counters.** The pads read "received 0" beside a route from
  them that had carried 58 messages. A device's counters now last while it is unplugged and carry
  on when it returns (T194).
- **No note-off was sent for a held note.** Silencing sent sustain-off, all-notes-off and
  all-sound-off, while the comment on it said a synth ignoring all-notes-off would still honour
  explicit note-offs. Nothing sent them, though every held note is tracked. Each held note now
  gets its own note-off, after the pedal release. The same check found the broad resets breaking
  the rule above that only the notes the stopped route carried may be stopped: two controllers on
  the drum channel into one sampler, and unplugging one cut the other's held note. A channel
  another route still has notes sounding on, at the same destination and MIDI Out, now gets only
  the stopped route's note-offs. A channel nobody else is playing still gets the full reset,
  which is what stops a drone or an envelope that ignores note-off (T195). Measured: a held pad,
  then the cable pulled, and the virtual port received sustain-off, note-off 82, all-notes-off
  and all-sound-off, the received count carried from 15 to 17, and the route agreed at 17.