Serialize all audio file info

[Gautzilla]

🐌 wassup?

AudioFile serialization only included the file path plus the begin and end timestamps.
Thus, deserializing an AudioFile (e.g. when using AudioDataset.from_json() still required the audio file manager to read the file metadata to access the sample rate (or deduce it from the number of samples) and the number of chanels.

This led to significant loss of time when trying to deserialize large datasets on slow disks or servers.

💡what's new?

This PR adds the sample_rate and chanels kvps to the serialized AudioFile dictionaries, so that the AudioFile can be fully instantiated from the dictionary without reading anything from the file.

🤷 Is that it??

Not really: since the previously written JSON file didn't include these kvps, we still need to read the metadata for deserializing those old files.
However, it is quite easy to update the JSON once the dataset has been deserialized:

ads = AudioDataset.from_json(r"path_to_the_old_json") # This JSON file doesn't have the sample_rate or chanels kvp
ads.write_json(ads.folder) # This will update the JSON with the missing kvps

🎵 nota bene

I've just tested this with ~15k files locally stored on my computer, so the speed gain is not that obvious (it mostly seems more consistent while repeatedly deserializing the dataset), so I'm all in if anyone wants to test this on remote servers or whatever!

resolves #437

[coveralls]

coverage: 98.981% (+0.007%) from 98.974% — Gautzilla:audiofile-from-dict into Project-OSmOSE:main

[Gautzilla]

@ElodieENSTA This could also speed things up on the APLOSE side! 👏

[mathieudpnt]

I tested this but i get a missing dict key error when trying to deserialize the AudioDataset instance:

file = Path(path\to\json)
ads = AudioDataset.from_json(file)

  File "C:\Users\dupontma2\Documents\Git\OSmOSE\OSEkit\src\osekit\core\audio_data.py", line 659, in _from_base_dict
    if dictionary["instrument"] is None
       ~~~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'instrument'

The issue seem to stem from AudioData._from_base_dict method, particularly on this bit:

instrument = (
            None
            if dictionary["instrument"] is None
            else Instrument.from_dict(dictionary["instrument"])
)

changing from dictionary["instrument"] to dictionary["audio_data"]["instrument"] seem to clear the error though

The logic in the next bits of the method seem broken as well:

butter = (
            None
            if "butter" not in dictionary or dictionary["audio_data"]["butter"] is None
            else Butterworth.from_dict(dictionary["butter"])
        )

The first condition is always True because butter is never a key of dictionary but a key of dictionary["audio_data"]

return cls.from_files(
            files=files,
            begin=begin,
            end=end,
            instrument=instrument,
            sample_rate=dictionary["audio_data"]["sample_rate"],
            normalization=Normalization(dictionary["audio_data"]["normalization"]),
            normalization_values=dictionary["audio_data"]["normalization_values"],
            butter=butter,
        )

Same here none of the returned key here exist in dictionnary, they exist in dictionnary["audio_data"]

Is there something wrong with the way I process the json or is it something to fix @Gautzilla ?

EDIT
Ok, it looks like when I fix _from_base_dict method as mentionned and that I rewrite the json audio_data key no longer appear in the json ???

[Gautzilla]

I tested this but i get a missing dict key error when trying to deserialize the AudioDataset instance:

file = Path(path\to\json)
ads = AudioDataset.from_json(file)

  File "C:\Users\dupontma2\Documents\Git\OSmOSE\OSEkit\src\osekit\core\audio_data.py", line 659, in _from_base_dict
    if dictionary["instrument"] is None
       ~~~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'instrument'

I can't reproduce. Can you share a minimal reproducible example or at least a glimpse of the JSON file?

The issue seem to stem from AudioData._from_base_dict method, particularly on this bit:

instrument = (
            None
            if dictionary["instrument"] is None
            else Instrument.from_dict(dictionary["instrument"])
)

changing from dictionary["instrument"] to dictionary["audio_data"]["instrument"] seem to clear the error though

The AudioData dictionary (created through AudioData.from_dict(), which is the one that should be fed to AudioData.from_base_dict()) should have an "instrument" key, and no "audio_data" key: it looks like AudioData is somehow fed the AudioDataset dict: I need more info on your workflow as it doesn't reproduce on my end:

ads = AudioDataset(...)
ads.write_json(ads.folder)
ads_deserialized = AudioDataset.from_folder(ads.folder / f"{ads.name}.json")

This doesn't raise on my end.

The logic in the next bits of the method seem broken as well:

butter = (
            None
            if "butter" not in dictionary or dictionary["audio_data"]["butter"] is None
            else Butterworth.from_dict(dictionary["butter"])
        )

The first condition is always True because butter is never a key of dictionary but a key of dictionary["audio_data"]

butter is a key of the AudioData dict since #381, it should be in the AudioData dict (which should not have a "audio_data" key)

Is there something wrong with the way I process the json or is it something to fix @Gautzilla ?

I really don't know how you process the JSON, send me a Minimal Reproducible Example and I'll be able to take a deeper look! 😉

[mathieudpnt]

I tested this but i get a missing dict key error when trying to deserialize the AudioDataset instance:

file = Path(path\to\json)
ads = AudioDataset.from_json(file)

  File "C:\Users\dupontma2\Documents\Git\OSmOSE\OSEkit\src\osekit\core\audio_data.py", line 659, in _from_base_dict
    if dictionary["instrument"] is None
       ~~~~~~~~~~^^^^^^^^^^^^^^
KeyError: 'instrument'

I can't reproduce. Can you share a minimal reproducible example or at least a glimpse of the JSON file?

The issue seem to stem from AudioData._from_base_dict method, particularly on this bit:

instrument = (
            None
            if dictionary["instrument"] is None
            else Instrument.from_dict(dictionary["instrument"])
)

changing from dictionary["instrument"] to dictionary["audio_data"]["instrument"] seem to clear the error though

The AudioData dictionary (created through AudioData.from_dict(), which is the one that should be fed to AudioData.from_base_dict()) should have an "instrument" key, and no "audio_data" key: it looks like AudioData is somehow fed the AudioDataset dict: I need more info on your workflow as it doesn't reproduce on my end:

ads = AudioDataset(...)
ads.write_json(ads.folder)
ads_deserialized = AudioDataset.from_folder(ads.folder / f"{ads.name}.json")

This doesn't raise on my end.

The logic in the next bits of the method seem broken as well:

butter = (
            None
            if "butter" not in dictionary or dictionary["audio_data"]["butter"] is None
            else Butterworth.from_dict(dictionary["butter"])
        )

The first condition is always True because butter is never a key of dictionary but a key of dictionary["audio_data"]

butter is a key of the AudioData dict since #381, it should be in the AudioData dict (which should not have a "audio_data" key)

Is there something wrong with the way I process the json or is it something to fix @Gautzilla ?

I really don't know how you process the JSON, send me a Minimal Reproducible Example and I'll be able to take a deeper look! 😉

My bad I was trying to use the function from a SpectroDataset JSON !