Fix double pendulum env - #643
Merged
Merged
Conversation
zdu863
marked this pull request as ready for review
August 15, 2026 19:29
Physics: replace the single semi-implicit Euler step with RK4 plus configurable substeps. Acceleration is factored out of integrate_physics into dp_accel so it can be evaluated at RK4 trial states; the dynamics themselves are unchanged. Measured energy drift over a 10s episode with zero force falls from 123% to 0.6355% at substeps=1 and 0.0006% at substeps=4. That matters here because a swing-up is entirely energy bookkeeping -- an integrator that leaks energy makes the catch easy for the wrong reason. Reward: replace 0.5*height + hold-streak bonus with a DeepMind Control Suite-style dense blend. The streak bookkeeping is kept because hold_time is derived from it, which keeps that metric reward-independent. On accurate physics the reward choice is decisive. Building on the terminal-signal fix in this branch's parent, at 500M steps from a dead hang, hold_time out of 600: Euler + streak 358 (previous behaviour) RK4 + streak 0 -- never catches RK4 + DeepMind 407 The streak reward pays 0.5100 for the first held step against 0.5000 for flying through the top, a 2% signal that Euler's energy drift was masking by acting as unintended dissipation. The DeepMind blend pays 4.2x at the same decision point. Note this removes the old reward rather than keeping it behind a flag, so existing double_pendulum results are not reproducible from here on. The standalone demo sets substeps and balance_bonus_weight explicitly. Its designated initializer would otherwise zero them, leaving the interactive binary on RK4x1 with w=0 while training runs RK4x4 with w=0.5. Also fix min_lr_ratio = 0 -> 0.0. The CLI infers each flag's type from its config default, so the int made --train.min-lr-ratio reject float values. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
c_step()wrote the terminal transition and thenc_reset()cleared it before vecenv could read it.