taosd segfaults on COUNT(DISTINCT col) when the deduplicated set spills to disk (3.4.2.5)

[alexey-milovidov]

Version

• taosd version: 3.4.2.5.community (TDengine TSDB-OSS tarball tdengine-tsdb-oss-3.4.2.5-linux-arm64.tar.gz)
• git: c15925333c9fe385902b153879b812c26bc612f7, build: Linux-arm64 2026-08-16 16:59:49
• Single dnode, stock taos.cfg, Ubuntu 24.04 aarch64

What happens

SELECT COUNT(DISTINCT col) FROM t kills taosd with SIGSEGV once the
deduplicated set outgrows pqSortMemThreshold (16 MB) and the plan switches
from hash deduplication to the spill-to-disk merge sort. The client reports
Unable to establish connection [0x8000000B] after the connection times out,
and systemd restarts the service. It reproduces every time.

Below the threshold the same query is fine, which is what makes it look like a
size-dependent bug rather than a broken function: 1,000,000 distinct BIGINT
values (8 MB) answer in 0.1 s, 5,000,000 (40 MB) crash.

Reproduction

CREATE DATABASE crashtest;
CREATE TABLE crashtest.t (ts TIMESTAMP, v BIGINT);

Generate 5,000,000 rows with all values distinct and load them:

with open('/tmp/crash.csv', 'w') as f:
    for i in range(5000000):
        f.write('%d,%d\n' % (1700000000000 + i, 1000000000000 + i * 7))

taos -d crashtest -s "INSERT INTO t FILE '/tmp/crash.csv'"
Insert OK, 5000000 row(s) affected (9.480796s)

taos -d crashtest -s "SELECT COUNT(DISTINCT v) FROM t"
DB error: Unable to establish connection [0x8000000B] (77.717715s)

VARCHAR reproduces it too — a VARCHAR(40) column needs only 1,000,000
distinct values (52 MB of data) to cross the threshold and crash — so this is
the spill path rather than anything type-specific.

Backtrace

From a core dump, crashing thread vnode-query:

#0  colDataIsNull_t (isVarType=false, row=0, pColumnInfoData=0xfa6e7f0ff660)
        at include/common/tdatablock.h:110
#1  msortComparFn                     at source/libs/executor/src/tsort.c:943
#2  tMergeTreeAdjust                  at source/util/src/tlosertree.c:117
#3  tMergeTreeCreate (numOfSources=6)  at source/util/src/tlosertree.c:69
#4  tsortOpenForBufMergeSort          at source/libs/executor/src/tsort.c:2621
#5  tsortOpen                         at source/libs/executor/src/tsort.c:2964
#6  initSpillSortHandle   at source/libs/executor/src/distinctfilteroperator.c:175
#7  doDistinctFilter      at source/libs/executor/src/distinctfilteroperator.c:705
#8  optrGetNextFnWithExecRecord       at source/libs/executor/src/operator.c:65
...
#11 nextGroupedResult                 at source/libs/executor/src/aggregateoperator.c:348
#12 getAggregateResultNext            at source/libs/executor/src/aggregateoperator.c:454
#14 qExecTaskOpt                      at source/libs/executor/src/executor.c:939
#15 qwExecTask                        at source/libs/qworker/src/qworker.c:132
#16 qwProcessQuery                    at source/libs/qworker/src/qworker.c:1042
        sql = "select count(distinct v) from t"
#18 vnodeProcessQueryMsg              at source/dnode/vnode/src/vnd/vnodeSvr.c:1190

msortComparFn is reached with a SColumnInfoData * that does not describe
the column being sorted. Note isVarType=false even in the VARCHAR variant
of the reproduction, which suggests the comparator is set up from the wrong
column metadata when initSpillSortHandle builds the merge tree.

Workaround

Deduplicating in a subquery is unaffected and returns the correct answer:

SELECT count(*) FROM (SELECT DISTINCT v FROM crashtest.t);

Why it matters to us

We hit this adding TDengine to ClickBench.
Eight of the benchmark's 43 queries use COUNT(DISTINCT ...) and every one of
them deduplicates far more than 16 MB, so all eight took the server down. We
rewrote them to the subquery form to avoid shipping queries that crash the
engine.

[alexey-milovidov]

For context, the ClickBench entry this came out of is now open as ClickHouse/ClickBench#1704. It documents the workaround used for this and links back here, so anyone reading the benchmark entry can see the status of the underlying issue.

The other reports from the same exercise: #35449, #35450, #35451, #35452, #35453.