mxfdump --dict smpte

[haraldjordan]

This adds a new optional mode to MXFDump --dict smpte that changes how labels are being resolved. It leads to a good result from user perspective but now that its done, from developer perspective, i am not even sure if its a good idea to really push this into MXFDump.cpp. A better fit would be a complete new dumper but this would have been even much wore work. Seriously, going on with SMPTE dict instead of the old ones, i would have needed to replace about 50% of the existing code, making it very dirty to keep both.
If we push this, the general plan to eventually keep only the new smpte dict in future is more or less set, we cannot keep both forever. As using the smpte dict also changes the string label presentation, some users might not be OK with dumping the existing dict but from my perspective it is better to stick to the standard and present the labels as defined by SMPTE.

Considerations/Decisions made:

1. Keep the existing mode default for backward compatibility
2. Change as little existing code as needed
3. Use as much as possible from the SMPTE xml's, avoid hand-maintained whenever possible

Pros:

• Uses the Labels/Names as seen in the SMPTE xmls
• Most things that today are labelled as "Dark" or "Unknown" are being translated now
• Easy to use dict generator (MXFMetaDictionary_smpte.gen.py) auto downloads the dicts from smpte-ra (must be executed before rebuilding MXFDump.cpp)
• Generally working with a single complete SMPTE dict feels cleaner than working with the existing dicts (these are spread around the code and partially superseeded/incomplete)

Cons:

• Adds lots of code
• Uses the Labels/Names as seen in the SMPTE xmls but not for everything, Sub-Items (e.g. content of Indextable) is still printed with old Names
• "Old dict" still there, so its more messy than before
• Users have to specify --dict smpte in order to profit from the functionality
• Some things like Indextablesegment and much more still rely on the old/existing dict, even when --dict smpte is specified

Opinions very welcome.

[haraldjordan]

Here a windows build, again, use --dict smpte to see the new functionality.
MXFDump.zip

[scenarnick]

Just to make sure I get this: The state of SMPTE Label registry is downloaded prior to building and then embedded into the build so that the build is fully offline. Correct? Was this also checked for the MSVC type builds of the release workflow?

[emcodem]

Rebuilding the dict is a manual step, i did not automate that because i really don't like dependencies to the "wild internet" in fully automated workflows for security reasons

[thomasheritage]

A couple of links to MXF libraries powered by the SMPTE Metadata Registers, just in case it's of interest or useful on this topic:

• https://github.com/sandflow/regmxflib
• https://github.com/sandflow/regxmllib

[haraldjordan]

A couple of links to MXF libraries powered by the SMPTE Metadata Registers, just in case it's of interest or useful on this topic:

• https://github.com/sandflow/regmxflib
• https://github.com/sandflow/regxmllib

Thanks a lot Tom, very happy to get additional input especially if its from you :)
Thats what they use in the online-dumper on the SMPTE-RA page, i stepped over this before but after you posted it, i compared what they do and what we do. Whats interesting is that they normalize UL's partly in the code generation phase while we take everything literally and let the code do the normalizing. Other than that of course they create nice structured Java Classes while we only generate a plain list. All of that has its cons and pros and i dont see a clear winner. It would be interesting to discuss with the maker @palemieux if they see any clear disadvantage in the current strategy how we deal with the Dict.
What i definitely like about not storing the already wildcarded UL's is that we leave at least some breadcrumbs this way - usually users do not look up UL's that already have wildcards applied, so at least they have a small chance to find the UL they search for in our dict.
However, i also see that the masking strategy they apply in the cpp part of regxmllib is more easy to read than ours:
MXFFragmentBuilder.cpp
PREFACE_KEY.equals(it->second.getKey(), UL::IGNORE_GROUP_CODING & UL::IGNORE_VERSION)

Very interesting read overall but i would not (yet) apply any of these strategies in MXFDump.cpp because currently we only do "light" changes and try to match the surrounding coding style.

[palemieux]

@haraldjordan Happy to chime-in. Can you provide more context on the short- and long-term goals?

[haraldjordan]

@haraldjordan Happy to chime-in. Can you provide more context on the short- and long-term goals?

Hey @palemieux thanks a lot for joining in, very welcome!
Well the short term goal was to bring MXFDump.cpp to recognize much more UL's than it did until now. This goal is accomplished with the patch. However, i wondered if this strategy to generate the dict directly from the SMPTE xml's could be a future strategy to centralize and clean lots of spread dicts all around the code and maybe parse some things more generically than it's done now.
The only concrete plan i really had when submitting this is in a future patch go on and eventually and change in MXFDump.cpp the explicitly supported items like printPartition, printIndexSegment and co. in a way they are being dumped and validated generically (we resolve what we know, the rest is dumped as unknown item, currently we print only what we know but potentially miss superfluous items in some cases) and also they are being dumped in the exact order as they appear in the file. This would also mean we completely dump the current dicts and users would face the names from smpte xmls instead of the ones they are used to see from MXFDump.cpp.

[palemieux]

Couple of thoughts/questions:

• for groups, types and elements, I would consider using RegXML metadictionaries, such as the ones generated by regxmllib: these metadictionaries remove non-MXF entries and fix longstanding issues with the source registries.
• is the ultimate goal to (a) provide to BMX information about groups, types and elements directly from the SMPTE registers or (b) ultimately replace the BMX core framework with structures generated from the SMPTE registers?

[haraldjordan]

@palemieux thanks for sharing your thoughts!

Couple of thoughts/questions:

• for groups, types and elements, I would consider using RegXML metadictionaries, such as the ones generated by regxmllib: these metadictionaries remove non-MXF entries and fix longstanding issues with the source registries.
• is the ultimate goal to (a) provide to BMX information about groups, types and elements directly from the SMPTE registers or (b) ultimately replace the BMX core framework with structures generated from the SMPTE registers?

Up in front, the goal currently is limited to MXFDump which is a standalone project, not related to BMX. I'll need to see how it goes/where it leads and i'm not yet able to estimate if any part of BMX could really benefit from using similar techniques.

For the rest of this message, please have in mind at least for the modifications i did in MXFDump, i had the following existing principle in mind:

(cite from MXFDump.cpp introduction comment)

// This program is deliberately constructed as one file with no dependencies
// so that it can easily be ported to a new host. You'll be able to make some
// progress understanding an MXF file even if all you have is this source and
// a C++ compiler.

Anyway, executing the code generators to get the C macros header from the xmls is not needed in order to build MXFDump (we ship the latest generated .h files in the repo), so developing MXFDump itself is not influenced by the way how the C Macros are generated. Therefore i adopted the existing generator .py script to be able to work with the xmls generated by RegXML, generated the C Macros and tested MXDFump with it. First thing i saw is that lots of translations is missing, e.g. Partitions and Labels. Using Regxml means either to change MXFDump to work with an xml parser and parse them at runtime (which violates the comment above about no dependencies) and in addition to that, we'd need java/maven to be available when rebuilding the xml's which makes the process tedious. Additionally the Labels are missing, so it would be a mix of using the original Labels.xml and Regxmls output. That all sounds like it's not really worth the efforts to be honest. But maybe i have been blind so far and overlook something very important?

I was especially interested to see if i find the longstanding issues but no luck.. could you clarify what you mean please?