toyos-net-node: a query known not to have reached its resolver is not waited out, an ICMP error ends nothing, and a burst of lookups reaches the wire

issues/toyos-has-its-own-network-stack.md

@@ -17,7 +17,7 @@ netd runs smoltcp. Its replacement is ToyOS's own stack, built clean-room: reade
 - Then multi-core, netring (blocked on the owner's ABI ruling), TCP and IP hardening, IPv6, offloads, soak.
 - This track owns the T14's outbound rows, the machine reaching its router and the internet on the I219, judged from the stick: they arrive on this stack and not on smoltcp (owner: "no smoltcp."), and until they do no T14 row reads the wired card (`issues/the-host-cannot-reach-the-t14-while-it-runs-toyos.md`).
 
-Stage 5 lands in slices, the orchestrator's cut under the owner's words "i want smoltcp out as fast as possible": netd's decisions go into `toyos-net-node` (`userland/netstack/node`), pure and host-tested and shipped in nothing, and then one change moves netd onto it and deletes smoltcp. In the tree: the node, its DHCP lease, its clients' datagram sockets, its mDNS name, its resolver, streams, listeners and the one bound on its clients' streams, listeners and datagram sockets (`userland/netstack/node/src/places.rs`); a lookup's sockets stand outside that bound, under `toyos_dns::MAX_LOOKUPS` and its rounds. Still to build on it: the datagram sockets' broadcast permission; then the move.
+Stage 5 lands in slices, the orchestrator's cut under the owner's words "i want smoltcp out as fast as possible": netd's decisions go into `toyos-net-node` (`userland/netstack/node`), pure and host-tested and shipped in nothing, and then one change moves netd onto it and deletes smoltcp. In the tree: the node, its DHCP lease, its clients' datagram sockets, its mDNS name, its resolver, streams, listeners and the one bound on its clients' streams, listeners and datagram sockets (`userland/netstack/node/src/places.rs`); a lookup's sockets stand outside that bound, under `toyos_dns::MAX_LOOKUPS` and its rounds. Still to build on it: datagram senders that wait on the hop, which is [udp]'s way to [ip] through the node's `lib.rs`, `lease.rs` and `toyos-net-shard/src/lib.rs`; the datagram sockets' broadcast permission; then the move.
 
 What the node does not yet meet:
 
@@ -40,8 +40,8 @@ What the node does not yet meet:
 - A refusal [udp] logs against a client's call waits in the stack until the node's next `receive`, `fire` or `link`, because only those pass over the stack's reports. Exit, at the move and not after it: the datagram calls end in that pass, or netd drains it once per turn of its loop.
 - The machine's name is told to the node twice, as `toyos_dhcp::HostName` in `Node::new` and as `toyos_mdns::Host` in `Node::answer_as`: the first owns its bytes and the second borrows them. Exit: one type carries the label to both; due at the move if handing `answer_as` its `&'static str` takes a leak.
 - The node's name is not probed for or defended (RFC 6762 §8.1, §9), as `toyos-mdns` records of itself. It is not announced again when the link comes back, which §8.3 asks and netd does today, where link-up resets its DHCP client and the address is new again: here the lease outlives the link. Exit: `toyos-mdns` does both, on the node's `link`.
-- A query that does not reach its resolver is waited out for `toyos_dns::WAIT_MS` like one nobody answered, whether [udp] refused it for want of a route (`node.query-unsent`) or the network reported it back, its resolver unreachable or its port refused (`node.query-failed`): `toyos_dns::Lookup` has no word for either. Exit, before the move: `Lookup` is told and asks the next resolver at once, `a_query_udp_refuses_is_counted_holds_no_port_and_is_waited_out` finds its answer at 0 ms, and `a_query_the_network_reports_unreachable_is_counted_and_waited_out` finds its lookup ended at 3 s, [ip]'s first report of its one resolver, and not at `ROUNDS` × `WAIT_MS`.
-- Lookups started together at a resolver whose link address is not yet known lose all but the newest `PENDING_PER_NEIGHBOUR` queries to [ip]'s queue for an unresolved next hop (`nb.pending-overflow`, RFC 4861 §7.2.2), and each lost one costs its lookup a `WAIT_MS`; smoltcp kept each query in its own socket until the neighbour answered. `a_burst_at_a_resolver_not_yet_resolved_leaves_ips_queue_of_queries_and_the_rest_a_wait_later` holds it as it stands. It is a regression against what ships and would ship at the move. Exit, before the move: that test finds every query of the burst on the wire once the resolver answers ARP.
+- A query [ip] refuses at the transmit opportunity that would have carried it, its next hop held failed or its table full, is reported to its socket inside `Node::transmit`, which carries no lookup on: the resolver reads the report at the node's next `receive`, `fire` or `link`, which at the latest is the end of the query's wait, where a query [udp] refuses in the resolver's own call, or one reported in a frame or at a deadline, ends its wait at once. `a_query_ip_refuses_at_its_transmit_opportunity_is_read_at_the_nodes_next_wake` holds it as it stands. It is no regression: what ships waits every such query out. Until that read, an ICMP error that quotes the query's addresses and ports takes [ip]'s word's place on the socket, [udp] holding one pending error a socket, the latest: the query is then waited out and not let go, which costs the early end and never an answer. Owner: the slice in which datagram senders wait on the hop (`Node::transmit` is the node's `lib.rs`). Exit: the node carries its lookups on after the stack's transmit, or [udp] refuses in the call a datagram whose next hop [ip] holds failed, so that no report waits on a socket to be displaced, and that test finds its lookup ended at the 4 s it started.
+- [ip]'s queue for a next hop it is resolving is every sender's: `PENDING_PER_NEIGHBOUR` is 16, a stopgap which the node's resolver asserts is no fewer than `toyos_dns::MAX_LOOKUPS`, so every held lookup's first query waits there whole, but a datagram a client sends to the same next hop in that window, which for everything off the link is the router, takes the oldest one's place (`nb.pending-overflow`, RFC 4861 §7.2.2), and a query lost so costs its lookup a `WAIT_MS`; smoltcp kept each datagram in its own socket until the neighbour answered. `a_clients_datagram_behind_the_burst_takes_the_oldest_querys_place_in_ips_queue` holds it as it stands. It is a regression against what ships for more than 16 datagrams at one next hop not yet resolved, and would ship at the move; and it sizes a limit of [ip]'s by a constant of the resolver's, which leaves four unresolved next hops filling `PENDING_TOTAL` where eight did. Owner: the slice in which datagram senders wait on the hop, where [udp]'s datagram calls already go through `Stack` and `Shard`. Its design: [udp]'s `serve` asks [ip] the next-hop question before it hands a datagram over, as [tcp]'s exit does with `Hop` (NUD-25, IP-D5); a socket whose oldest datagram's hop is pending keeps it in its own queue, which `TX_DATAGRAMS` and `TX_BYTES` already bound and refuse in the call, and is offered again when the hop resolves and told when it fails; [ip] then queues only its own datagrams. Exit, before the move: that test finds every query on the wire once the resolver answers ARP; `PENDING_PER_NEIGHBOUR` is `limits.ip.nud.pending_per_neighbour`'s 8 again; and the resolver's compile-time assertion, the change to `s_ip_nud_004_overflow_drops_the_oldest` and stage 3's departure line about it are deleted.
 - `toyos_dns::Lookup::on_datagram` still takes a reply's source address and port, which the node's connected sockets have already matched and the node hands it from its own record of the query: the two parameters are read only by netd's resolver on smoltcp. Exit, at the move: they go with that resolver.
 - `node.address-refused` has no test: `toyos-dhcp` accepts no address or prefix [ip] refuses, by the same `toyos-net-wire` checks in both, so the refusal cannot be reached from the wire. Exit: the client hands [ip] a type that carries the check, and the counter goes.
 
@@ -54,6 +54,7 @@ What stage 3 departs from its specifications:
 - US-43 expects `ip.not-for-us` for a datagram to an unjoined group in a frame to that group's MAC; the frame filter refuses it first, as `eth.not-for-us` (ETH-23). Exit: the scenario names the frame filter.
 - DH-64 expects an announce request from the client on link-up with a lease held; `toyos-net-ip` announces the held address itself on link-up, by the owner's ruling to announce rather than probe again, so the client asks nothing. Exit: the scenario drops the request, or [ip] stops announcing on its own.
 - The readers' UDP specification lets every datagram a closed socket had accepted leave; `toyos-net-udp` holds at most one socket's queue of them together and refuses the rest `udp.tx-discarded-on-close`, because the readers' stage 4 design holds nothing in the shard without a bound. Exit: `s_udp_us_053_accepted_datagrams_outlive_close` finds `udp.tx-discarded-on-close` at 0, every datagram a closed socket accepted leaving.
+- NUD-04 sends ten datagrams and finds the third to the tenth, two overflowed, under `limits.ip.nud.pending_per_neighbour`, 8; `PENDING_PER_NEIGHBOUR` is 16, so that a burst of every lookup the node holds waits whole for its resolver's link address, and `s_ip_nud_004_overflow_drops_the_oldest` sends two more than the constant: it holds the rule, the oldest dropped and counted, and no longer the scenario's numbers. NUD-05 passes unchanged only because its eight neighbours of eight datagrams are `PENDING_TOTAL`, which did not move. Exit: that of the node's residual on [ip]'s queue above, in the slice in which datagram senders wait on the hop: the constant is 8, and the test sends NUD-04's ten.
 - The readers' IP specification has no scenario for 169.254/16 on an interface that holds a routable address: its routing scenarios are RTE-01 to RTE-17 and its ARP reception scenarios NBR-01 to NBR-18, none names that prefix, and NBR-12's `arp.sender-off-link` is expected of a sender in no prefix at all, which its 198.51.100.9 still is. Its input scenarios are IN-01 to IN-09, of which none names 169.254 or a link-local source: IN-05, the nearest, refuses a source that is no host or an edge of a held prefix, and [ip]'s refusal of a source at an edge of 169.254/16 on every interface, `ip.invalid-source`, is an addition to that step's list. It has none either for two next hops of which one never answers: NUD-04 overflows one neighbour's queue and NUD-05 fills the interface's, and in neither does a second neighbour answer. The tests that hold the two rules therefore name their RFC section and carry no id: the `rfc_3927_…` tests in `toyos-net-ip/tests/addr.rs` and `nbr.rs`, and `rfc_4861_7_2_2_a_silent_next_hop_holds_up_no_other` in `tests/nud.rs`. Exit: the readers add a scenario for each, and each test carries its id.
 
 What stage 3 does not yet meet:

toyos-dns/src/lib.rs

@@ -431,8 +431,12 @@ pub enum Failure {
     NoSuchName,
     /// The name exists and has no `A` record (RFC 2308 §2.2).
     NoAddress,
-    /// Every query went unanswered for [`WAIT_MS`].
+    /// No server answered, and a query that may have reached one was given
+    /// its [`WAIT_MS`].
     TimedOut,
+    /// No query reached a server ([`Lookup::on_unreached`]) and none is left
+    /// to send at once: there is no answer to wait for.
+    Unreachable,
     /// The answer did not fit a UDP reply (TC).
     Truncated,
     /// Every server that answered could not answer, the last with this RCODE.
@@ -478,21 +482,49 @@ struct Sent {
 /// any query this lookup sent for the name now asked is read, so an answer
 /// late past its query's wait still ends the lookup; it is read only on the
 /// query's own port, with the query's ID, from the server the query went to.
+///
+/// **A wait is for an answer that can come.** When the newest query is
+/// answered with a failure, or known or said not to have reached its server,
+/// the next server is asked at once: RFC 1035 §4.2.1's interval of two to
+/// five seconds is between repetitions of a query that may have been lost on
+/// its way, and the same section has the other servers tried first. **The
+/// interval is each server's**: one that has not answered is sent no second
+/// query inside [`WAIT_MS`] of its last, or of the caller's last knowledge
+/// that it was not reached, whatever is said of that query and whatever
+/// another server answers; a server is its address, however often the list
+/// names it. Where the next server cannot be asked yet, the lookup waits
+/// until it can while any answer is still read, and ends at once only with
+/// none.
+///
+/// **What the caller knows and what it was told are two calls.** A query the
+/// caller itself knows never reached its server ([`Lookup::on_unreached`]) is
+/// let go, and a lookup with no query left whose answer is read ends with
+/// [`Failure::Unreachable`]. A report from the network
+/// ([`Lookup::on_report`]) is anyone's word (RFC 8085 §5.2): it lets no query
+/// go and ends nothing, so the answer to a query reported so is still read.
 #[derive(Debug)]
 pub struct Lookup {
     /// The name now asked: the one given, or the last alias followed.
     name: Name,
     /// Aliases followed so far, counted against [`MAX_ALIASES`].
     aliases: usize,
-    servers: Vec<[u8; 4]>,
+    /// Each server, and when it may be asked again: [`WAIT_MS`] after its
+    /// last query or after it was last known not to be reached, and at once
+    /// when it has answered any query, an older one's late failure included.
+    /// An address named twice is one server: every place it has carries the
+    /// same time.
+    servers: Vec<([u8; 4], u64)>,
     /// Queries sent for `name`, oldest first; one answered with a server
-    /// failure is taken out. [`Lookup::waiting`] is this list.
+    /// failure, or known not to have reached its server, is taken out.
+    /// [`Lookup::waiting`] is this list, which holds a query while the
+    /// lookup lasts.
     sent: Vec<Sent>,
     /// The [`Asked`] the next query gets: none is given twice in a lookup.
     next: u32,
     /// Queries sent for `name`, answered or not.
     asked: usize,
-    /// When the newest query is given up on.
+    /// When the newest query is given up on; or, where it is not waited for
+    /// and the next server may not be asked yet, that server's turn.
     due: u64,
     /// The RCODE of the last server failure, which is what a lookup that runs
     /// out of queries reports if any server answered at all.
@@ -509,7 +541,7 @@ impl Lookup {
         let mut lookup = Self {
             name,
             aliases: 0,
-            servers: servers.to_vec(),
+            servers: servers.iter().map(|server| (*server, now)).collect(),
             sent: Vec::new(),
             next: 0,
             asked: 0,
@@ -531,22 +563,41 @@ impl Lookup {
         self.sent.iter().map(|s| s.asked)
     }
 
-    /// The next query for `name`, or the end where every one has been sent.
+    /// The next server's query, if it may be asked at `now`; or the wait for
+    /// the answers still read, until it may or the last query's wait is
+    /// over; or the end.
     fn ask(&mut self, now: u64, id: impl FnOnce() -> u16) -> Step {
-        if self.asked == ROUNDS * self.servers.len() {
-            return Step::Done(Err(match self.failed {
-                Some(rcode) => Failure::ServerFailed(rcode),
-                None => Failure::TimedOut,
-            }));
+        let turn = self.asked % self.servers.len();
+        let (to, free) = self.servers[turn];
+        let spent = self.asked == ROUNDS * self.servers.len();
+        if !spent && now >= free {
+            self.free(to, now + WAIT_MS);
+            self.asked += 1;
+            let asked = Asked(self.next);
+            self.next += 1;
+            let id = id();
+            self.sent.push(Sent { asked, id, to });
+            self.due = now + WAIT_MS;
+            return Step::Ask { asked, to, query: query(id, &self.name) };
+        }
+        if self.sent.is_empty() || (spent && now >= self.due) {
+            // With no query left, each was answered with a failure or never
+            // reached its server.
+            let silence = if self.sent.is_empty() { Failure::Unreachable } else { Failure::TimedOut };
+            return Step::Done(Err(self.failed.map_or(silence, Failure::ServerFailed)));
+        }
+        if !spent {
+            // The newest query is not waited for: the next server's turn is.
+            self.due = free;
+        }
+        Step::Wait
+    }
+
+    /// `server` may next be asked at `at`.
+    fn free(&mut self, server: [u8; 4], at: u64) {
+        for (_, free) in self.servers.iter_mut().filter(|(addr, _)| *addr == server) {
+            *free = at;
         }
-        let to = self.servers[self.asked % self.servers.len()];
-        self.asked += 1;
-        let asked = Asked(self.next);
-        self.next += 1;
-        let id = id();
-        self.sent.push(Sent { asked, id, to });
-        self.due = now + WAIT_MS;
-        Step::Ask { asked, to, query: query(id, &self.name) }
     }
 
     /// The time is `now`: the newest query has had its [`WAIT_MS`] where it
@@ -558,6 +609,33 @@ impl Lookup {
         self.ask(now, id)
     }
 
+    /// The caller knows at `now`, of its own knowledge, that the query
+    /// `asked` did not reach its server: it was never sent. Its answer is
+    /// read no more. `id` draws the ID of the query this sends.
+    pub fn on_unreached(&mut self, asked: Asked, now: u64, id: impl FnOnce() -> u16) -> Step {
+        let Some(which) = self.sent.iter().position(|s| s.asked == asked) else {
+            return Step::Wait;
+        };
+        let unreached = self.sent.remove(which);
+        self.free(unreached.to, now + WAIT_MS);
+        // A newer query still waits for its own answer.
+        if which < self.sent.len() {
+            return Step::Wait;
+        }
+        self.ask(now, id)
+    }
+
+    /// The network said at `now` that the query `asked` did not reach its
+    /// server. Its answer is still read. `id` draws the ID of the query this
+    /// sends.
+    pub fn on_report(&mut self, asked: Asked, now: u64, id: impl FnOnce() -> u16) -> Step {
+        // A newer query still waits for its own answer.
+        if self.sent.last().is_none_or(|s| s.asked != asked) {
+            return Step::Wait;
+        }
+        self.ask(now, id)
+    }
+
     /// `msg` arrived at `now` from `port` of `from`, on the port `asked` was
     /// sent from. `id` draws the ID of the query this sends.
     pub fn on_datagram(
@@ -585,6 +663,9 @@ impl Lookup {
             Verdict::ServerFailed(rcode) => {
                 self.failed = Some(rcode);
                 self.sent.remove(which);
+                // It answered: its interval, which is for a query that may
+                // have been lost, is over.
+                self.free(from, now);
                 // The next server is asked now, unless a newer query is still
                 // waiting for its own answer.
                 if which == self.sent.len() {
@@ -605,6 +686,9 @@ impl Lookup {
                     self.sent.clear();
                     self.asked = 0;
                     self.failed = None;
+                    for (_, free) in &mut self.servers {
+                        *free = now;
+                    }
                     self.ask(now, id)
                 }
             },