User Request: Integrating Glider DAC data into CrocoLake

[enrico-mi]

Who is requesting this?

No response

What is being requested?

I'd like to integrate Glider DAC into CrocoLake, a tidy and cloud-optimized database for sparse in-situ ocean observations. It currently includes Argo, OleanderXBT, SprayGliders, OCS Saildrones, and GLODAP observations.

I am interested in particular in QC-ed data for temperature, salinity and some biogeochemical (dissolved oxygen, pH, chlorophyll a, nitrate).

How would this request improve your current workflow?

CrocoLake is a new and growing dataset that is used in oceanographic analyses and for data assimilation in and ocean model verification.
Glider data add very important information for those workflows.

Any alternative approaches or workarounds?

No response

How can we get in contact with you to follow up on this request?

enrico.milanese@whoi.edu

[sarinamann-noaa]

IOOS meeting notes here
Presentation
Poster

Information on delayed-mode data QC:

• Active datasets are real-time submitted files, QARTOD is ran on them.
• Delayed mode files are larger and contain more data overall; providers download this data from gliders and is more dense.
• Currently the process is halted to apply QC to delayed mode datasets - implementing QC on delayed mode data sets #355
• Delayed mode data does not reach the GTS, but delayed mode datasets have the “_QC” suffix variables that have been QC’d by providers. These data should be more reliable.

@mwengren FYI

[benjwadams]

The format for Glider DAC deployments is here https://ioos.github.io/glider-dac/ngdac-netcdf-file-format-version-2.html and data is available on https://gliders.ioos.us/erddap .

You should be able to fetch data from ERDDAP. What is the issue that needs resolving here?

[enrico-mi]

IOOS meeting notes here Presentation Poster

Information on delayed-mode data QC:

• Active datasets are real-time submitted files, QARTOD is ran on them.
• Delayed mode files are larger and contain more data overall; providers download this data from gliders and is more dense.
• Currently the process is halted to apply QC to delayed mode datasets - implementing QC on delayed mode data sets #355
• Delayed mode data does not reach the GTS, but delayed mode datasets have the “_QC” suffix variables that have been QC’d by providers. These data should be more reliable.

@mwengren FYI

Thanks, @sarinamann-noaa . I'll check with my collaborators if the QARTOD approach is enough for their use-cases.

With respect to the delayed mode files, my understanding is that there is no process shared across providers within IOOS to generate them (with #355 aiming to address this), but that still some providers generate delayed mode files. Are the providers required to share these files on the IOOS DAC (even if not standardized and every file's structure is different), or is it up to the provider to decide if/how to make them public?

[enrico-mi]

The format for Glider DAC deployments is here https://ioos.github.io/glider-dac/ngdac-netcdf-file-format-version-2.html and data is available on https://gliders.ioos.us/erddap .

You should be able to fetch data from ERDDAP. What is the issue that needs resolving here?

Thanks for the links, @benjwadams , in particular I was not aware of the first one.

In brief the issues is: how can I pull and keep synced all the delayed mode gliders data?

My goal is to include them, together with other datasets, in a collection of quality-controlled ocean observations that is used in data assimilation, ocean model validation, analyses. E.g. if an ocean model produces a temperature snapshot of the Atlantic when and where a glider also recorded temperature, they'd like to compare the two to evaluate the performance of their model; this is generally done on delayed mode data, not real time.

[benjwadams]

Usually for delayed mode datasets we receive all the files at once. For delayed mode data we follow the convention that the dataset name should end in -delayed and expect the data providers to follow this. I'm not sure if there's a way that you can search on the title only, but this returns a good number of delayed mode datasets: https://gliders.ioos.us/erddap/search/index.html?page=1&itemsPerPage=1000&searchFor=delayed

ERDDAP also allows most html pages to be returned as CSV, so you could filter the returned results with the usual tools capable of reading CSV:

https://gliders.ioos.us/erddap/search/index.csv?page=1&itemsPerPage=1000&searchFor=delayed

It's also possible to watch a dataset for changes and be notified either by an RSS client or email response: https://coastwatch.pfeg.noaa.gov/erddap/information.html#subscriptions

[enrico-mi]

Usually for delayed mode datasets we receive all the files at once. For delayed mode data we follow the convention that the dataset name should end in -delayed and expect the data providers to follow this. I'm not sure if there's a way that you can search on the title only, but this returns a good number of delayed mode datasets: https://gliders.ioos.us/erddap/search/index.html?page=1&itemsPerPage=1000&searchFor=delayed

ERDDAP also allows most html pages to be returned as CSV, so you could filter the returned results with the usual tools capable of reading CSV:

https://gliders.ioos.us/erddap/search/index.csv?page=1&itemsPerPage=1000&searchFor=delayed

It's also possible to watch a dataset for changes and be notified either by an RSS client or email response: https://coastwatch.pfeg.noaa.gov/erddap/information.html#subscriptions

That's great info, thanks.

Fyi if I click on 'files' for any of the results at the first link, I get a '403 Forbidden' error (e.g. https://gliders.ioos.us/erddap/files/blue-20210921T1318-delayed/). It does not seem to be an issue when using the tabledap protocol.

[benjwadams]

That's great info, thanks.

Fyi if I click on 'files' for any of the results at the first link, I get a '403 Forbidden' error (e.g. https://gliders.ioos.us/erddap/files/blue-20210921T1318-delayed/). It does not seem to be an issue when using the tabledap protocol.

This is intentional due to large amounts of flood coming in to /files subdirectory, hopefully #533 will fix. We can arrange to get you access with a static IP off of this GitHub issue.
/files endpoint does not include several metadata modifications so I'd recommend using the ERDDAP query functionality instead when possible.

[benjwadams]

As it stands, I do not feel this is not a good fit for submission of an issue because the action items and issue closure criteria are indeterminate and somewhat dependent on integration of systems outside of our usual maintenance on this project. Please consider putting GDAC usage questions in the discussions section instead. If you feel there are deficiencies in the documentation regarding how to obtain data from the DAC not covered by the current GDAC documentation or ERDDAP's documentation, this may be a better fit for an issue we can take action upon.

[sarinamann-noaa]

@benjwadams I suggested that @enrico-mi start by creating a ticket for this request.
We can shift to email or discussion if that's better.
GDAC support email- glider.dac.support@noaa.gov

[benjwadams]

OK, no problem.

I've discussed this in the past GDAC meetings and broadly speaking, I think it's more appropriate to use GitHub Discussions or the Glider DAC user mailing list for open-ended discussion, usage questions (which may later turn into issues depending if the documentation is determined to be unclear or insufficient), and provider/dataset specific issues not tied to issues within the repository codebase.
GitHub offers some suggestions on when to use one or the other:
https://docs.github.com/en/get-started/using-github/communicating-on-github#introduction

The CF Conventions seem to primarily use GitHub Discussions these days for clarification and questions on various parts of that standard, and often portions of the standard that are determined to need further clarification in the documentation or conformance specs are followed up upon as user submitted issues that end up as changes to the documentation:

https://github.com/orgs/cf-convention/discussions

[leilabbb]

Note: on the right hand side near the bottom you can use "Convert to discussion" to move this issue to discussion.