Test Arrow file reads against gold integration test files

[ianmcook]

Hi chDB devs! I hit some various errors when trying to use chDB to read various Apache Arrow IPC data. Would you consider incorporating the Arrow IPC integration test gold files into your test suite to surface these? https://github.com/apache/arrow-testing/tree/master/data/arrow-ipc-stream/integration

[wudidapaopao]

Thanks for the suggestion, incorporating the Arrow IPC integration test gold files is a great idea. I'll give it a try and see how feasible it is to wire them into our test suite.
Also, just to understand your use case a bit better — were you using the DataStore API or the core chDB API (e.g. chdb.session.Session(...).query(...)) when you hit those errors?

[ianmcook]

Thanks! I was actually using the newly added ADBC interface. I installed chDB using dbc and queried it through databow.

dbc install chdb
databow --driver chdb

Then I ran queries at the databow prompt against local Arrow IPC stream files.

[ShawnChen-Sirius]

Hi @ianmcook , thanks again for the suggestion and the context.

@wudidapaopao is adding the Arrow IPC gold-file tests on the chdb-core side.

After checking the failures, I find most of them appear to come from the underlying ClickHouse Arrow reader. Since chDB embeds the ClickHouse engine, we generally avoid changing ClickHouse kernel behavior directly in chDB unless the issue is chDB-specific.

So I think what we can do is: tracking these cases in chdb-core, reporting the underlying Arrow reader issues upstream to ClickHouse, and picking up the fixes when chdb-core syncs with upstream ClickHouse.

[ShawnChen-Sirius]

An upstream discussion issue is created.