Repository navigation
t_stress.test_clickhouse: incorrect records number calculation #859
Description
# env/bin/python3 run_tests.py t_stress.test_clickhouse
File tests_resume.json not found
Root privileges are required: need access to module loading on localhost.
----------------------------------------------------------------------
Running functional tests...
----------------------------------------------------------------------
t_stress.test_clickhouse.TestClickHouseLogsUnderLoad.test_all_logs_under_load ... FAIL
t_stress.test_clickhouse.TestClickHouseLogsUnderLoad.test_all_logs_with_reload ... FAIL
t_stress.test_clickhouse.TestClickHouseLogsUnderLoad.test_tfw_logger_stop_cont ... FAIL
======================================================================
FAIL: t_stress.test_clickhouse.TestClickHouseLogsUnderLoad.test_all_logs_under_load
----------------------------------------------------------------------
Traceback (most recent call last):
File "/root/tempesta-test/t_stress/test_clickhouse.py", line 86, in test_all_logs_under_load
self.assertEqual(clickhouse_collected_rows, dmesg_logs_count)
AssertionError: 13828 != 1
======================================================================
FAIL: t_stress.test_clickhouse.TestClickHouseLogsUnderLoad.test_all_logs_with_reload
----------------------------------------------------------------------
Traceback (most recent call last):
File "/root/tempesta-test/t_stress/test_clickhouse.py", line 109, in test_all_logs_with_reload
self.assertEqual(clickhouse_collected_rows, dmesg_logs_records)
AssertionError: 18743 != 1
======================================================================
FAIL: t_stress.test_clickhouse.TestClickHouseLogsUnderLoad.test_tfw_logger_stop_cont
----------------------------------------------------------------------
Traceback (most recent call last):
File "/root/tempesta-test/t_stress/test_clickhouse.py", line 132, in test_tfw_logger_stop_cont
self.assertLess(clickhouse_collected_rows, dmesg_logs_records)
AssertionError: 13296 not less than 1
----------------------------------------------------------------------
Ran 3 tests in 54.625s
FAILED (failures=3)
All the 3 assertions are on
self.assertLess(clickhouse_collected_rows, dmesg_logs_records)he problem is that the kernel log is a circular buffer, i.e. it rewrites first messages if there are too much logging, e.g. after the test the first messages are from the access log and I believe a lot of records were rewritten:
# dmesg | head -3
[51427.550693] [tempesta fw] 127.0.0.1 "default" "GET / HTTP/2.0" 201 8 "-" "h2load nghttp2/1.59.0" "ja5t=77178f00a5f5000d" "ja5h=b1c1008c0280"
[51427.550710] [tempesta fw] 127.0.0.1 "default" "GET / HTTP/2.0" 201 8 "-" "h2load nghttp2/1.59.0" "ja5t=77178f00a5f5000d" "ja5h=b1c1008c0280"
[51427.550751] [tempesta fw] 127.0.0.1 "default" "GET / HTTP/2.0" 201 8 "-" "h2load nghttp2/1.59.0" "ja5t=77178f00a5f5000d" "ja5h=b1c1008c0280"
I think it makes sense to get the statistics of sent requests from the benchmark tool and compare it with what we have in the access log. To get the computation reliable we can get message counters (received and early dropped) from /proc/tempesta/perfstat.
After the fix please unamsk in tests_disabled.json:
{
"name": "t_stress.test_clickhouse.TestClickHouseLogsUnderLoad",
"reason": "Disabled by issue #859 (t_stress.test_clickhouse: incorrect records number calculation)"
},
Activity