Skip to content

perf(pd): omit unused output logprob metadata - #1486

Open
WANDY666 wants to merge 1 commit into
mainfrom
perf/pd-omit-unused-logprob-metadata
Open

perf(pd): omit unused output logprob metadata#1486
WANDY666 wants to merge 1 commit into
mainfrom
perf/pd-omit-unused-logprob-metadata

Conversation

@WANDY666

@WANDY666 WANDY666 commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Mark OpenAI chat requests, and completions requests without logprobs, as not needing output-token logprob metadata on the PD wire.
  • Preserve that transport-only marker across PD master's ctypes sampling-parameter copies, split blocks, multiple choices, and batched completion prompts.
  • Drop id, logprob, cumlogprob, special, and logprobs only when the caller does not consume them.
  • Keep legacy behavior by default for other APIs and sampling params without the marker.

Why

The common OpenAI path was forwarding per-token metadata that its response builders never read. This PR isolates metadata elision from the tuple wire-format change so its API contract and fallback behavior can be reviewed independently.

Validation

  • Production-code diff passes repository black and flake8 hooks.

@WANDY666
WANDY666 force-pushed the perf/pd-omit-unused-logprob-metadata branch from 788db8c to c5f5eb0 Compare August 24, 2026 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant