core: Add support for playing F4V (MP4) files#24038
Conversation
torokati44
added
A-video
T-fix
Instead, demux AudioSpecificConfig out-of-band, simplifying the decoder. Add `extra_data: Option<Box<[u8]>>` to `SoundStreamInfo` to carry codec-specific init data out-of-band. The FLV handler now stores AAC sequence headers (AudioSpecificConfig) there instead of appending them to the audio substream, and strips the leading packet-type byte from AAC payloads since the enum variant already encodes that distinction. AacSubstreamDecoder reads the config at construction time via `codec_params.with_extra_data`, removing the mid-stream packet-type dispatch that was previously in its decode loop.
torokati44
force-pushed
the
f4v-video-re_mp4
branch
from
0280a38 to
ef8fb8d
Compare
Note for the curious, this is specifically: https://new.weedtowonder.org/ (more specifically, *.weedtowonder.org used to be this site, while the current https://weedtowonder.org/ used to only be accessible on https://old.weedtowonder.org/, but that was changed when it was found https://old.weedtowonder.org/ would work with Ruffle while, for the time being, https://new.weedtowonder.org/ was broken with Ruffle) |
|
Do you have a simple way I could test some representative content from that site with the desktop player? Building and loading the extension after each change is a bit cumbersome. |
|
Also, what exactly is different between the old. and new. sites? |
|
(For testing: I guess just |
2 participants
Description
Take 2 of #14655, superseding it.
Compared to that, this relies on
re_mp4(by the awesome Rerun.io team, who are also the main sponsors ofegui), not on Mozilla'smp4parsecrate. The former is more actively maintained, while the latter hasn't seen a real "release" in years, so we'd have to use it from a git source.It is also a lot nicer in that it directly gives us the "sample" (in MP4 lingo, it's an encoded video frame or audio chunk) offsets and lengths directly, so we don't have to calculate them in convoluted ways from the
moovbox.One downside is that it doesn't provide a BoxIter mechanism, so we'd have to either contribute it, or manually pass in only complete boxes when streaming from a remote file, otherwise the parser errors out. I think an entirely missing
mdatbox would be tolerated though, just not a truncated one. This is one of the TODOs left.This one also adds audio (AAC only) decoding!
Also includes #23985, which was motivated by this.
Now that streams.rs has grown a bit lengthy, in the future it might be worth it to refactor it into a "directory module", with the FLV and MP4 parts in separate submodules.
Testing
There's a basic test included.
#19531 isn't fixed by this alone, at least on desktop - there's a weird H.264 NALU encapsulation artifact in one of the files it plays, which is apparently accepted by WebCodecs but not by OpenH264. Fixing it is out of scope, unrelated to the MP4 format itself.
There's also something fishy on WeedToWonder.org last time I tested (frames not passed in for the video decoder in the right order, or key frames being missed?) - hence why this is also still a draft.
Checklist