MQTTS connection from MQTTX fails on MQTTnet.Server 5.2.0.1603 (regression from 5.0.1.1416)

[Eviav]

Verification

• The bug is reproduced using the demo code below with MQTTnet.Server version 5.2.0.1603.
• Connect via MQTTX to mqtts://localhost:6017, username/password: admin, SSL secure verification disabled. Connection fails.
• The same setup works correctly with MQTTnet.Server version 5.0.1.1416.

Describe the bug

When using MQTTnet.Server 5.2.0.1603, external clients such as MQTTX cannot establish an MQTTS (TLS) connection to the encrypted endpoint. The TLS handshake fails and the client is unable to connect.

The exact same server code and MQTTX configuration work without issues on MQTTnet.Server 5.0.1.1416, indicating a regression in the server-side TLS handling between these versions.

Which component is your bug related to?

• MQTTnet.Server

To Reproduce

Steps to reproduce the behavior:

1. Using MQTTnet.Server version '5.2.0.1603'.
2. Run the server code provided below.
3. In MQTTX, create a new connection:
   • Protocol: mqtts://
   • Host: localhost
   • Port: 6017
   • Username: admin / Password: admin
   • SSL/TLS: Enabled, Secure verification: Disabled (allow self-signed)
4. See error: Connection fails / TLS handshake does not complete. No ClientConnectedAsync event is raised on the server.

Expected behavior

MQTTX should successfully connect to the MQTTS endpoint on port 6017 with SSL verification disabled, exactly as it did in version 5.0.1.1416.

Screenshots

N/A

Additional context / logging

• Only the encrypted endpoint (port 6017) is affected; the plain TCP endpoint (port 6016) works fine in both versions.
• The certificate is a self-signed cert generated at runtime and exported as PFX without a password. This worked in 5.0.1.1416 but fails in 5.2.0.1603.

Code example

<PackageReference Include="MQTTnet.Server" Version="5.2.0.1603" />

using MQTTnet.Protocol;
using MQTTnet.Server;
using System.Net;
using System.Security.Cryptography.X509Certificates;
using System.Text;

namespace MQTTServerTest
{
    internal class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine("Hello, MQTTServer!");
            var options = new MqttServerOptionsBuilder().WithDefaultEndpoint();
            options.WithDefaultEndpointPort(6016);
            EnableMQTTs(options);
            var server = new MqttServerFactory().CreateMqttServer(options.Build());
            server.ValidatingConnectionAsync += e =>
            {
                // Accept admin/admin credentials
                e.ReasonCode = MqttConnectReasonCode.Success;
                return Task.CompletedTask;
            };
            server.StartAsync().Wait();
            server.ClientConnectedAsync += args =>
            {
                System.Diagnostics.Debug.WriteLine("[Connected] " + args.ClientId);
                return Task.CompletedTask;
            };
            server.ClientDisconnectedAsync += args =>
            {
                System.Diagnostics.Debug.WriteLine("[Disconnected] " + args.ClientId);
                return Task.CompletedTask;
            };
            server.InterceptingPublishAsync += args =>
            {
                var topic = args.ApplicationMessage.Topic;
                var cid = args.ClientId;
                var json = Encoding.UTF8.GetString(args.ApplicationMessage.Payload);
                System.Diagnostics.Debug.WriteLine("[Message received] " + cid + "\nTopic: " + topic + "\n" + json);
                return Task.CompletedTask;
            };
            while (true)
            {
                _ = Console.ReadLine();
            }
        }

        static bool EnableMQTTs(MqttServerOptionsBuilder options)
        {
            var certificate = CreateSelfSignedCertificate();
            options.WithEncryptedEndpoint();
            options.WithEncryptedEndpointPort(6017);
            options.WithEncryptionCertificate(certificate.Export(X509ContentType.Pfx));
            return true;
        }

        public static X509Certificate2 CreateSelfSignedCertificate() => CreateSelfSignedCertificate("1.3.6.1.5.5.7.3.1");
        public static X509Certificate2 CreateSelfSignedCertificate(string oid)
        {
            var sanBuilder = new SubjectAlternativeNameBuilder();
            sanBuilder.AddIpAddress(IPAddress.Loopback);
            sanBuilder.AddIpAddress(IPAddress.IPv6Loopback);
            sanBuilder.AddDnsName("localhost");

            using var rsa = System.Security.Cryptography.RSA.Create();
            var certRequest = new CertificateRequest("CN=localhost", rsa, System.Security.Cryptography.HashAlgorithmName.SHA512, System.Security.Cryptography.RSASignaturePadding.Pkcs1);

            certRequest.CertificateExtensions.Add(new X509KeyUsageExtension(X509KeyUsageFlags.DataEncipherment | X509KeyUsageFlags.KeyEncipherment | X509KeyUsageFlags.DigitalSignature, false));
            certRequest.CertificateExtensions.Add(new X509EnhancedKeyUsageExtension(new System.Security.Cryptography.OidCollection { new(oid) }, false));
            certRequest.CertificateExtensions.Add(sanBuilder.Build());

            using var certificate = certRequest.CreateSelfSigned(DateTimeOffset.Now.AddMinutes(-10), DateTimeOffset.Now.AddMinutes(10));
#pragma warning disable SYSLIB0057
            var pfxCertificate = new X509Certificate2(
                certificate.Export(X509ContentType.Pfx),
                (string)null!,
                X509KeyStorageFlags.MachineKeySet | X509KeyStorageFlags.Exportable);
#pragma warning restore SYSLIB0057

            return pfxCertificate;
        }
    }
}