Hard crashes on Pico W with high network activiity

[dhalbert]

From #10615 (comment):

The PR seems like a rational fix, so maybe completely unrelated, but under atypically extreme duress (hammering on TCP socket, and hammering on interrupts ...assuming the code is doing what I think), a TCP client will get hardfaults somewhat regularly with PR artifacts on Pico W:

Client code (on a Pico W):

# Adafruit CircuitPython 10.0.0-beta.3 on 2025-08-29; Raspberry Pi Pico W with rp2040
# Adafruit CircuitPython 10.0.0-beta.3-8-g9e7a03cbc2 on 2025-09-05; Raspberry Pi Pico W with rp2040
# "Hard fault: memory access or instruction error." (after "Connecting")

import time
import supervisor
import microcontroller
import os
import random
import traceback
import wifi
import socketpool
import array
import pulseio
import board

# edit host and port to match server
HOST = "192.168.6.57"
PORT = 5000
TIMEOUT = 1
INTERVAL = 0.1
MAXBUF = 8192

def send_pulses():
    # on off on ...
    MAX_TRAIN = 17
    for p_out in range(1, len(pulse_outs)):
        p_train = []
        for p_item in range(0, random.randint(3, MAX_TRAIN)):
            p_train.append(random.randint(100, 10_000))
        pulses = array.array('H', p_train)
        pulse_outs[p_out].send(pulses)

time.sleep(3)  # wait for serial after reset

print("Connecting to Wifi")
while not wifi.radio.connected:
    try:
        wifi.radio.connect(os.getenv('WIFI_SSID'), os.getenv('WIFI_PASSWORD'))
    except Exception as ex:
        traceback.print_exception(ex, ex, ex.__traceback__)
        time.sleep(1 + random.random())
        supervisor.reload()
        # microcontroller.reset()
pool = socketpool.SocketPool(wifi.radio)
print(wifi.radio.ipv4_address)

# 50% duty cycle at 38kHz.
pulse_outs = []
pulse_outs.append(pulseio.PulseOut(board.GP0, frequency=38000, duty_cycle=32768))
pulse_outs.append(pulseio.PulseOut(board.GP1, frequency=38000, duty_cycle=32768))
pulse_outs.append(pulseio.PulseOut(board.GP2, frequency=38000, duty_cycle=32768))
pulse_outs.append(pulseio.PulseOut(board.GP3, frequency=38000, duty_cycle=32768))

buf = bytearray(MAXBUF)
while True:
    try:
        print("Create TCP Client Socket")
        with pool.socket(pool.AF_INET, pool.SOCK_STREAM) as s:
            s.settimeout(TIMEOUT)

            print("Connecting")
            s.connect((HOST, PORT))

            payload = bytearray()
            for _ in range(random.randint(0, MAXBUF)):
                payload.append(random.choice((b'0123456789ABCDEF')))

            send_pulses()  #####

            size = s.send(payload)
            print("Sent", size, "bytes")

            send_pulses()  #####

            size = s.recv_into(buf)
            # occasionally on recv_into:
            # OSError: [Errno 104] ECONNRESET
            # OSError: [Errno 9] EBADF
            # OSError: [Errno 116] ETIMEDOUT

            print('Received', size, "bytes", buf[:size])
    except Exception as ex:
        print(f"⚠️", end=" ")
        traceback.print_exception(ex, ex, ex.__traceback__)

    time.sleep(INTERVAL)

Server code (on a Pico 2W):

import time
import microcontroller
import os
import random
import traceback
import wifi
import socketpool

HOST = ""
PORT = 5000
TIMEOUT = None
BACKLOG = 2
MAXBUF = 8192

time.sleep(3)  # wait for serial after reset

print("Connecting to Wifi")
while not wifi.radio.connected:
    try:
        wifi.radio.connect(os.getenv('WIFI_SSID'), os.getenv('WIFI_PASSWORD'))
    except Exception as ex:
        traceback.print_exception(ex, ex, ex.__traceback__)
        time.sleep(1)
        microcontroller.reset()
pool = socketpool.SocketPool(wifi.radio)
print(wifi.radio.ipv4_address)

print("Create TCP Server socket", (HOST, PORT))
with pool.socket(pool.AF_INET, pool.SOCK_STREAM) as s:
    s.setsockopt(pool.SOL_SOCKET, pool.SO_REUSEADDR, 1)  # 
    s.settimeout(TIMEOUT)

    s.bind((HOST, PORT))
    s.listen(BACKLOG)
    print("Listening")

    buf = bytearray(MAXBUF)
    while True:
        try:
            print("Accepting connections")
            conn, addr = s.accept()
            conn.settimeout(TIMEOUT)
            print("Accepted from", addr)

            size = conn.recv_into(buf, MAXBUF)
            print("Received", buf[:size], size, "bytes")

            conn.send(buf[:size])
            print("Sent", buf[:size], size, "bytes")

        except Exception as ex:
            print(f"⚠️", end=" ")
            traceback.print_exception(ex, ex, ex.__traceback__)
        finally:

            time.sleep(0.1)  # too short and client gets "OSError: [Errno 104] ECONNRESET" in recv

            conn.close()

With beta.3, hardfaults occur under less duress than above, but I hadn't actually replicated it with the prior testing in the previous comment, so just wanted to see how it could be triggered.

There also seems to be a situation where the server closing the connection immediately after sending the response can cause OSError: [Errno 104] ECONNRESET at the client (somehow before the client has read all of the incoming packet... some buffering issue?).

Originally posted by @anecdata in #10615 (comment)

[dhalbert]

@anecdata I've made this a new issue -- thanks for the report.

Is the PulseOut code necessary to cause the problem, or could I get rid of that and just make web requests?

[anecdata]

I didn't see any hard faults on beta.3 or PR artifacts (except for the 1 oddball PANIC reported in the PR), until I added some PulseOut code. The beta.3 client hard faulted easier than the PR artifacts client. I didn't see hard faults with PulseOut on the PR artifacts until I turned up the number of pins / length of pulse sequences / timing range of pulses, and certainly didn't exhaustively test all of those variables... I was after replicating the original issue by introducing interrupts into the socket mix. I didn't know if this was realistic enough to chase down.

* edit, after some additional testing: beta.3 client will hard fault without extra interrupts (PulseOut)

[dhalbert]

If it crashes, it still should be fixed. This might even be a PulseOut bug rather than a networking bug.

[dhalbert]

An interesting test would be to see if you see no crashes in 9.2.4 and you do see crashes in 9.2.5. That would point to the addition of LWIP dynamic storage management in #10027 as the root cause.

[anecdata]

Both clients have the full PulseOut code.

The 9.2.4 client is still running from overnight.

The 9.2.5 client hard faulted around 12:30 AM. Hard faulted again after a couple hours this morning.

[dhalbert]

@anecdata here is another version of the fix, but this makes port_malloc(), port_free(), port_realloc() all do the tlsf (heap) operations in critical sections, not just LWIP operations. Would you mind testing this also? Thanks very much.

picow-guard-all-port-storage.zip

[anecdata]

I have it running on both PicoW clients (with PulseOut), looking good so far... will report if anything happens.

[anecdata]

They both got hard faults within minutes
Adafruit CircuitPython 10.0.0-beta.3-3-geaca0bb751-dirty on 2025-09-07; Raspberry Pi Pico W with rp2040
Hard fault: memory access or instruction error.

And hard fault again on client #2. Could it be due to some needed interrupt that gets blocked.

[dhalbert]

Did these hard faults occur noticeably faster than the testing you described in the original post of this issue, with the #10615 fix?

[anecdata]

Not statistical, but seems to be. These are happening in minutes (twice now each), with the other versions it was more like half hour to several hours (or more).

[dhalbert]

Since you gave us client and server examples, I'll try some other things and see if I can make this better.

[dhalbert]

I am testing this with the tip of main right now. I am seeing ECONNRESETs as you were, about 30-50% of the time. As an experiment, I am running the server on my Linux machine, with the very similar code below. Interestingly , in that case I see EBADF a lot, not ECONNRESET, at least 75% of the time. I've only had one hard crash so far, but was unable to get a stack trace.

I'll also try the more pervasive critical section code I mentioned above in #10618 (comment).

Server code (on Ubuntu 24.04):

import time
import microcontroller
import os
import random
import traceback
import wifi
import socketpool

HOST = ""
PORT = 5000
TIMEOUT = None
BACKLOG = 2
MAXBUF = 8192

time.sleep(3)  # wait for serial after reset

print("Connecting to Wifi")
while not wifi.radio.connected:
    try:
        wifi.radio.connect(os.getenv('WIFI_SSID'), os.getenv('WIFI_PASSWORD'))
    except Exception as ex:
        traceback.print_exception(ex, ex, ex.__traceback__)
        time.sleep(1)
        microcontroller.reset()
pool = socketpool.SocketPool(wifi.radio)
print(wifi.radio.ipv4_address)

print("Create TCP Server socket", (HOST, PORT))
with pool.socket(pool.AF_INET, pool.SOCK_STREAM) as s:
    s.setsockopt(pool.SOL_SOCKET, pool.SO_REUSEADDR, 1)  #
    s.settimeout(TIMEOUT)

    s.bind((HOST, PORT))
    s.listen(BACKLOG)
    print("Listening")

    buf = bytearray(MAXBUF)
    while True:
        try:
            print("Accepting connections")
            conn, addr = s.accept()
            conn.settimeout(TIMEOUT)
            print("Accepted from", addr)

            size = conn.recv_into(buf, MAXBUF)
            print("Received", buf[:size], size, "bytes")

            conn.send(buf[:size])
            print("Sent", buf[:size], size, "bytes")

        except Exception as ex:
            print(f"⚠️", end=" ")
            traceback.print_exception(ex, ex, ex.__traceback__)
        finally:

            time.sleep(0.1)  # too short and client gets "OSError: [Errno 104] ECONNRESET" in recv

            conn.close()