My consolidation always fails at duration 8.1m

[KonradHEXANT]

Are there any settings that will allow a longer timeout?

[Arthur031221]

There is a configurable consolidation timeout: HINDSIGHT_API_CONSOLIDATION_WALL_TIMEOUT, in seconds. It limits time without progress during a consolidation task; each completed batch restarts the clock. The current default is 7,200 seconds (two hours), so a failure at 8.1 minutes by itself does not indicate that the default wall timeout fired.

I checked the current configuration source:

curl -fsSL https://raw.githubusercontent.com/vectorize-io/hindsight/0cc0f39fc2a6cd889f15353ab3ebc0b3bb383cc3/hindsight-api-slim/hindsight_api/config.py | grep -nE 'ENV_CONSOLIDATION_WALL_TIMEOUT =|DEFAULT_CONSOLIDATION_WALL_TIMEOUT =|ENV_CONSOLIDATION_LLM_TIMEOUT ='
438:ENV_CONSOLIDATION_LLM_TIMEOUT = "HINDSIGHT_API_CONSOLIDATION_LLM_TIMEOUT"
999:ENV_CONSOLIDATION_WALL_TIMEOUT = "HINDSIGHT_API_CONSOLIDATION_WALL_TIMEOUT"
1893:DEFAULT_CONSOLIDATION_WALL_TIMEOUT = 7200  # seconds (2 hours)

Check the failed operation's error or worker log before changing the limit. If it says the task made no progress for a configured number of seconds and names HINDSIGHT_API_CONSOLIDATION_WALL_TIMEOUT, raise that setting on the worker and restart it. If the error names an LLM request timeout, the separate HINDSIGHT_API_CONSOLIDATION_LLM_TIMEOUT setting is relevant. The screenshot shows duration and failure state, but not which timeout or error caused the failure.

[KonradHEXANT]

But there are no error details, just "timeout error"

[Arthur031221]

The screenshot narrows this down. The failing span uses the LiteLLM provider, and its error is TimeoutError:. In the current Hindsight source, the default LLM request timeout is 120 seconds with three retries. The LiteLLM path waits 1, 2, and 4 seconds between those retries. Four timed-out requests plus those waits take 487 seconds, or 8.12 minutes, which closely matches the 8.1 minutes shown. The blank text after TimeoutError: does not give a more specific cause.

Please check the worker logs for this run for LiteLLM call exceeded timeout= and LiteLLM call timed out after. Those lines report the timeout and attempt count actually used. If they confirm a 120-second request timeout and that backend normally needs longer, set HINDSIGHT_API_CONSOLIDATION_LLM_TIMEOUT to a suitable value, such as 300 seconds, on the worker and restart it. That setting falls back to HINDSIGHT_API_LLM_TIMEOUT; HINDSIGHT_API_CONSOLIDATION_WALL_TIMEOUT is a separate limit and will not extend an individual LLM request. If the log messages differ, please share the relevant worker log lines and your Hindsight version, with any credentials removed.

[KonradHEXANT]

Thanks :) That works :)

[sheyoshan]

Glad to see you got your timeout issue resolved!

I recently spent several hours investigating somewhat similar Consolidation behavior with Hindsight v0.10.3 and a local Qwen3.5 4B model running through Ollama.

I also experimented with HINDSIGHT_API_CONSOLIDATION_LLM_TIMEOUT, increasing it from 300 to 600 seconds.

Interestingly, extending the timeout helped some requests complete, but others still failed. One successful request processed approximately 20,340 input tokens and generated just 299 output tokens in about 78 seconds. Another generated over 6,500 tokens without completing before the 600-second limit.

This made me realize that timeout settings are only part of the picture. In my case, unexpectedly long model outputs appear to be a significant contributor, although I couldn't inspect the unfinished responses to determine exactly why.

My setup uses Ollama rather than LiteLLM, so the underlying causes may be different.

I've shared the detailed logs and observations here: [Field Report #5495](#5495).

Hopefully this additional data helps anyone investigating similar behavior!