Return after rejecting invalid encoder inputs

[gregbenz]

What changed

• Return immediately after rejecting an invalid target display brightness.
• Return immediately after rejecting an invalid compressed-image intent.
• Add regression coverage confirming that rejected values do not mutate encoder state.

Why

Both setters currently construct an error but continue executing. An invalid brightness
is stored in the encoder, and an otherwise valid JPEG with an invalid intent is accepted
under that intent while the API reports success. Returning the validation error keeps
rejected input out of encoder state and makes the public API result trustworthy.

Testing

• Added tests for an invalid compressed-image intent and a non-finite target brightness;
  both fail on current main and pass with this change.
• Full non-HEIF unit suite: 1,292 tests run; 1,068 passed and 224 existing parameter
  combinations skipped.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.