link-assistant/formal-ai#471 (comment)
When split by 150k exactly and 13k output it looks suspicious, is really split exactly? Usually it is indicator of fake data. Do we really have data about compaction event in log? Do we have ability to calculate actual input and output tokens between compaction events?
If we can - we should do it correctly, not faking, if we cannot, we should not split (if no confirmations for compaction events at all).
If we have partial data, I think we can calculate correctly. Even if Codex does not calculate itself we for example can sum up tokens for each message and so on.
We need to download all logs and data related about the issue to this repository, make sure we compile that data to ./docs/case-studies/issue-{id} folder, and use it to do deep case study analysis (also make sure to search online for additional facts and data), in which we will reconstruct timeline/sequence of events, list of each and all requirements from the issue, find root causes of the each problem, and propose possible solutions and solution plans for each requirement (we should also check known existing components/libraries, that solve similar problem or can help in solutions).
If there is not enough data to find actual root cause, add debug output and verbose mode if not present, that will allow us to find root cause on next iteration.
If issue related to any other repository/project, where we can report issues on GitHub, please do so. Each issue must contain reproducible examples, workarounds and suggestions for fix the issue in code. Also double check to fully apply requirements to entire codebase, so if we have issue in multiple places, it should be fixed in all them.
Please plan and execute everything in this single pull request, you have unlimited time and context, as context auto-compacts and you can continue indefinitely, until it is each and every requirement fully addressed, and everything is totally done.
link-assistant/formal-ai#471 (comment)
When split by 150k exactly and 13k output it looks suspicious, is really split exactly? Usually it is indicator of fake data. Do we really have data about compaction event in log? Do we have ability to calculate actual input and output tokens between compaction events?
If we can - we should do it correctly, not faking, if we cannot, we should not split (if no confirmations for compaction events at all).
If we have partial data, I think we can calculate correctly. Even if Codex does not calculate itself we for example can sum up tokens for each message and so on.
We need to download all logs and data related about the issue to this repository, make sure we compile that data to
./docs/case-studies/issue-{id}folder, and use it to do deep case study analysis (also make sure to search online for additional facts and data), in which we will reconstruct timeline/sequence of events, list of each and all requirements from the issue, find root causes of the each problem, and propose possible solutions and solution plans for each requirement (we should also check known existing components/libraries, that solve similar problem or can help in solutions).If there is not enough data to find actual root cause, add debug output and verbose mode if not present, that will allow us to find root cause on next iteration.
If issue related to any other repository/project, where we can report issues on GitHub, please do so. Each issue must contain reproducible examples, workarounds and suggestions for fix the issue in code. Also double check to fully apply requirements to entire codebase, so if we have issue in multiple places, it should be fixed in all them.
Please plan and execute everything in this single pull request, you have unlimited time and context, as context auto-compacts and you can continue indefinitely, until it is each and every requirement fully addressed, and everything is totally done.