Add data dictionary methods

[mcarans]

This can be used like this:

import logging

from hdx.data.dataset import Dataset
from hdx.facades.simple import facade

logger = logging.getLogger(__name__)


def main():
    dataset = Dataset.read_from_hdx("hdx-hapi-funding")
    resource = dataset.get_resource()

    resource.set_hdx_data_dictionary([
        {
            "field": "location_code",
            "label": "Location Code",
            "description": "ISO3 country code",
            "data_type": "text",
            "lalala": "lalala"
        },
        {
            "field": "appeal_code",
            "label": "Appeal Code",
            "description": "Code identifying the humanitarian appeal",
        },
        {
            "field": "funding_usd",
            "label": "Funding (USD)",
            "description": "Amount of funding in US dollars",
        },
    ])
    resource.update_in_hdx()

    dataset = Dataset.read_from_hdx("hdx-hapi-funding")
    resource = dataset.get_resource()
    data_dictionary = resource.get_hdx_data_dictionary()
    logger.info(f"Data dictionary read back from {dataset.get_hdx_url()}:\n{data_dictionary}")


if __name__ == "__main__":
    facade(main, hdx_site="dev", user_agent="test")

[github-actions]

Test Results

158 tests  +1   158 ✅ +1   1m 16s ⏱️ +17s
  1 suites ±0     0 💤 ±0 
  1 files   ±0     0 ❌ ±0 

Results for commit 8e88f88. ± Comparison against base commit 6cb2a55.

♻️ This comment has been updated with latest results.

[coveralls]

coverage: 97.205% (-0.01%) from 97.216% — data_dictionary into main

[mcarans]

@nadineisabel Should I add more validation in the library eg. only allow field, label, description (and data_type?) given that it looks like this won't be added to the CKAN call? If data_type is allowed, should I validate the value (eg. only allow "numeric", "text" and "timestamp without time zone")?

[nadineisabel]

Nice! This is exciting. Thoughts below:

On the keys, no, leave it open. Extra keys are allowed on purpose in the schema so I'd rather the library match that.

On data_type, yes please validate the values restricted to the postgres types datapusher+ produces, so numeric, text and timestamp without time zone.

It may be worth adding to the docs too. @alexandru-m-g confirmed datapusher+ re-infers types on every upload, so for anything going into the datastore that way data_type is regenerated each push, and a pipeline setting it would just get overwritten. The field is really for resources datapusher+ doesn't handle, like the Jenkins datastores and non-datastore resources which is potentially planned for the future.

Cc @danmihaila as well

[mcarans]

@nadineisabel I've added data_type validation

[nadineisabel]

Looks good to me. Cc @alexandru-m-g and @danmihaila.