Adding statewide monitoring workflow under data.remote inst/ccmmf by sarahkanee · Pull Request #3913 · PecanProject/pecan

sarahkanee · 2026-03-31T20:38:37Z

Includes hls, phenology, events, tillage, traits.
Single tree for dependent scripts (events + pool + tillage). Should break this up at some point

Description

Motivation and Context

Review Time Estimate

Immediately
Within one week
When possible

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My change requires a change to the documentation.
My name is in the list of CITATION.cff
I agree that PEcAn Project may distribute my contribution under any or all of
- the same license as the existing code,
- and/or the BSD 3-clause license.
I have updated the CHANGELOG.md.
I have updated the documentation accordingly.
I have read the CONTRIBUTING document.
I have added tests to cover my changes.
All new and existing tests passed.

Includes hls, phenology, events, tillage, traits. Single tree for dependent scripts (events + pool + tillage). Should break this up at some point

…toring-inst

Includes orchestrator, crop/subclass and ADOY gap-fill, product build, QC reporting, and SGE submission scripts. Excludes large runtime artifacts (CDL fractions, trained parquet tables, logs).

Bring lab management code (extract packages, events/phenology/hls/traits, stakeholder docs, Session 0 env template) onto the PR branch. Ship county crop matrices as real files and resolve shared HLS libs via sibling paths so clones work without /projectnb defaults.

Session 1 adds an operator runbook for cadwr-landuse (01-03) while pointing at Alexey's README/docs as canonical; pipeline and Session 0 use user wording and ASCII.

Session 1 adds a CLASS/SUBCLASS lookup check (ignore DWR ** sentinels), spells out management/*-with-YEAR publish dirs, and checks out the cadwr-landuse auto-discover branch until main has it.

These one-off modislc/skanee/neochatt path scripts are lab ops, not part of the portable monitoring stack shown in PR PecanProject#3913.

Sessions 0-4 and pipeline are the curriculum; package READMEs are detail docs linked from sessions, not a parallel TOC.

Lead README, Session 0/1, and pipeline with what the trainee does (add 2024, re-run 2023), portable paths first, and a clearer checklist.

mdietze · 2026-07-23T00:05:24Z

+**You are here to process a new LandIQ year.**
+
+This training walkthrough adds **2024** and re-runs **2023** (the year pair).
+You work on **your own Linux cloud / HPC** (not BU SCC).


This statement is only meaningful to people who ARE working on the BU SCC. No one else will know what that is or consider that the "default"

mdietze · 2026-07-23T00:06:47Z

@@ -0,0 +1,56 @@
+# CCMMF monitoring pipeline - documentation
+
+**You are here to process a new LandIQ year.**


What's LandIQ? what's a LandIQ year? Why would I want to process one and what does that mean?

mdietze · 2026-07-23T00:09:31Z

@@ -0,0 +1,34 @@
+# CCMMF statewide monitoring (`inst/ccmmf`)
+
+California LandIQ → phenology → management-event pipeline scripts for PEcAn.


If I was looking in this folder for the first time, how would I know what this is? You need to start more broadly and explain yourself. Also, I don't think it's fair to assume that the ONLY thing you'll end up putting in the data.remote/insts/ccmmf folder will be phenology

mdietze · 2026-07-23T00:09:53Z

+
+**Documentation (start here):** [documentation/README.md](documentation/README.md)  
+**Environment setup:** [documentation/sessions/00-environment.md](documentation/sessions/00-environment.md)  
+**PR:** https://github.com/PecanProject/pecan/pull/3913


not sure why the documentation would reference a PR. Drop

mdietze · 2026-07-23T00:12:04Z

+
+| Path | Role |
+|------|------|
+| `documentation/` | Stakeholder sessions + `pipeline.md` + `ccmmf_env.example.sh` |


Expand every single "role". Even knowing how this workflow works, I find the descriptions overly terse and cryptic. Any time you read/write documentation or tutorials, you have to put yourself in the shoes of someone who has never heard of what you're doing and is looking at this project for the first time

mdietze · 2026-07-23T00:31:58Z

+export CCMMF_ROOT=/projectnb/dietzelab/ccmmf
+
+export LANDIQ_GAPFILL_ROOT=$CCMMF_ROOT/management/landiq-gapfill
+export CCMMF_LANDIQ_V4=$CCMMF_ROOT/LandIQ-harmonized-v4.1


I don't think it's safe to define variable names with version numbers in them, it will require a lot of code fixes (and potential bugs) when LandIQ jumps to V5.

mdietze · 2026-07-23T00:32:52Z

+A routine run updates only the years you pass and carries the rest over from the
+existing product, so adding a year is cheap.
+
+On a routine run, emission tables should **not** rebuild. If they do, confirm


what are emission tables?

mdietze · 2026-07-23T00:34:07Z

+compute node. Run the whole thing as one batch job:
+
+```bash
+qsub -l buyin -l h_rt=8:00:00 -v 'GAPFILL_ARGS=2023,2024' $LANDIQ_GAPFILL_ROOT/sge/run_gapfill.sge


this won't run for other people. You need to do more to explain concepts so users can understand what something is supposed to do. otherwise they won't know how to configure it for their own use case

mdietze · 2026-07-23T00:35:28Z

@@ -0,0 +1,32 @@
+#!/bin/bash -l
+#
+# SGE job: download CDL GeoTIFF(s) from NASS CropScape (avoids login-node timeout).


unclear what SGE is

these scripts look like they will only run on the BU SCC, not CARB's server or any else's

mdietze · 2026-07-23T00:38:17Z

+})
+
+.fa <- sub("^--file=", "", grep("^--file=", commandArgs(trailingOnly = FALSE), value = TRUE)[1L])
+source(file.path(dirname(normalizePath(.fa, mustWork = FALSE)), "_lib", "pkg_root.R"))


as your code matures towards production quality, you should be using "source" less and relying on the package more. This will mean moving more things into the /R folder and also using system.file more than file.path

Sessions and package README now point at pipeline.md. Package README explains CCMMF broadly without a PR link and clarifies first-time setup from inst/ccmmf.

…sh docs. Fold MSLSP into phenology/extract and NDTI into tillage/extract; move tillage metrics into events; remove SGE/explore leftovers; rewrite pipeline and session docs around the year-pair training path.

…urb.

Removed first-time setup instructions from README.md and streamlined the content.

Added instructions for running the pipeline in the README.

Keep "package" only for real R/Python/PEcAn packages.

Removed duplicate instruction for running the pipeline.

Updated descriptions for various paths and removed first-time setup instructions.

Removed redundant line about following pipeline documentation.

Updated section heading from 'tree' to 'branch' for clarity.

Removed reference to LandIQ crop code lookup table from README.

Document TARGET/PRIOR year roles in the pipeline, share a general session map across Sessions 0-3, and cast Session 1 steps as an input/output flow.

Point harmonized output at cadwr 03-final, require Session 0, and harden first-time operator guidance so annual LandIQ updates are easier to run and QC.

…ntory workflow. Clarify Session 0/1 guidance (including S3 harmonized crops path), slim the tree README and pipeline, and align gap-fill docs/scripts with how crop and ADOY fills actually run.

…k/ paths.

…on 1.

…aths from CCMMF_BASE.

… tree.

…ee README.

…gapfill/outputs. Clones can use these lookups as-is; do not rebuild them on a routine year-pair run.

CropScape is gone, so Session 0 drops CropScapeR and Session 1 treats tile overlays and CDL extract as batch jobs.

Drop leftover QC, coverage CSVs, and the extra lookup parquet so clones match what crop and adoy actually read.

…repeat each other. Also drop extra gap-fill README pointers and say why the existing lookup tables cover later years.

…cond Session 1. One home per idea; year-pair commands stay in the session. Move CLI and env vars to an appendix.

Adding statewide monitoring workflow under data.remote inst/ccmmf

aa12ea4

Includes hls, phenology, events, tillage, traits. Single tree for dependent scripts (events + pool + tillage). Should break this up at some point

github-actions Bot added the modules label Mar 31, 2026

mdietze and others added 9 commits April 17, 2026 14:22

Merge branch 'develop' into feature/ccmmf-statewide-monitoring-inst

f0fd483

Merge branch 'PecanProject:develop' into feature/ccmmf-statewide-moni…

ee61620

…toring-inst

Add LandIQ gap-fill workflow under data.remote inst/ccmmf

c3be9ef

Includes orchestrator, crop/subclass and ADOY gap-fill, product build, QC reporting, and SGE submission scripts. Excludes large runtime artifacts (CDL fractions, trained parquet tables, logs).

Document portable LandIQ year-add ops and two-repo user flow.

46e13d9

Session 1 adds an operator runbook for cadwr-landuse (01-03) while pointing at Alexey's README/docs as canonical; pipeline and Session 0 use user wording and ASCII.

Document legend QC recipe and writable SCC LandIQ product paths.

aa6468e

Session 1 adds a CLASS/SUBCLASS lookup check (ignore DWR ** sentinels), spells out management/*-with-YEAR publish dirs, and checks out the cadwr-landuse auto-discover branch until main has it.

Remove SCC-only HLS ancillary transfer scripts from inst/ccmmf.

dc07510

These one-off modislc/skanee/neochatt path scripts are lab ops, not part of the portable monitoring stack shown in PR PecanProject#3913.

Clarify documentation README as user training path.

4fe952b

Sessions 0-4 and pipeline are the curriculum; package READMEs are detail docs linked from sessions, not a parallel TOC.

Rewrite docs around the 2024 + 2023 user year-pair walkthrough.

98a032e

Lead README, Session 0/1, and pipeline with what the trainee does (add 2024, re-run 2023), portable paths first, and a clearer checklist.

mdietze reviewed Jul 23, 2026

View reviewed changes

sarahkanee and others added 18 commits July 23, 2026 10:58

Drop documentation/README; make pipeline.md the doc spine.

d616245

Sessions and package README now point at pipeline.md. Package README explains CCMMF broadly without a PR link and clarifies first-time setup from inst/ccmmf.

Restructure inst/ccmmf for training: fold extracts, drop attic, refre…

ab611f7

…sh docs. Fold MSLSP into phenology/extract and NDTI into tillage/extract; move tillage metrics into events; remove SGE/explore leftovers; rewrite pipeline and session docs around the year-pair training path.

Update metadata.md

a6cfee1

Update environment documentation

4d44a4e

Drop troubleshooting sections from ccmmf READMEs and trim root doc bl…

d6743d6

…urb.

title

8ff6702

title

0726eab

Update README.md

3e516da

Removed first-time setup instructions from README.md and streamlined the content.

Update README

dd49cf7

Added instructions for running the pipeline in the README.

Give inst/ccmmf README a formal monitoring-pipeline introduction.

8ea66fd

Point CCMMF README at pipeline.md instead of a docs index.

a401599

Refer to management inputs as management events in intro docs.

8204bc5

Call pipeline folders components, not packages.

687f389

Keep "package" only for real R/Python/PEcAn packages.

Update README.md

7f456e5

Removed duplicate instruction for running the pipeline.

Update README.md

02dc5ea

Updated descriptions for various paths and removed first-time setup instructions.

Update README.md

d6cc44a

Removed redundant line about following pipeline documentation.

Update README.md

6952bee

Updated section heading from 'tree' to 'branch' for clarity.

Remove LandIQ crop code lookup table reference

da412a0

Removed reference to LandIQ crop code lookup table from README.

sarahkanee added 22 commits August 7, 2026 16:38

Replace em dash with ASCII in Session 0 setup.

19e0c49

Clarify year-pair inventory concept and unify session Context diagrams.

8044dd7

Document TARGET/PRIOR year roles in the pipeline, share a general session map across Sessions 0-3, and cast Session 1 steps as an input/output flow.

Clarify LandIQ session workflow and separate COVER from gap-fill.

e7bb6b2

Point harmonized output at cadwr 03-final, require Session 0, and harden first-time operator guidance so annual LandIQ updates are easier to run and QC.

Polish CCMMF training docs and LandIQ gap-fill for the year-pair inve…

82ad950

…ntory workflow. Clarify Session 0/1 guidance (including S3 harmonized crops path), slim the tree README and pipeline, and align gap-fill docs/scripts with how crop and ADOY fills actually run.

Walk Session 1 through the S3 year-pair update and flatten LandIQ wor…

09a41e2

…k/ paths.

Tighten Session 0 env checks and show a CDL fraction glimpse in Sessi…

44fa6bd

…on 1.

Split Session 0 into one-time clone vs every-shell setup and derive p…

b13fec1

…aths from CCMMF_BASE.

Remove internal review notes and test env yaml from the training docs…

7b4a3fd

… tree.

Remove the documentation index README from the training tree.

dc82eb0

Remove the stale pipeline.md stub and point remaining links at the tr…

16c517f

…ee README.

Add the CDL x LandIQ probability and ADOY reference tables to landiq-…

4c1b127

…gapfill/outputs. Clones can use these lookups as-is; do not rebuild them on a routine year-pair run.

Download CDL from the NASS national zip and clip to California.

a13c5c6

CropScape is gone, so Session 0 drops CropScapeR and Session 1 treats tile overlays and CDL extract as batch jobs.

Keep only the eight gap-fill lookup tables in landiq-gapfill/outputs.

12006e4

Drop leftover QC, coverage CSVs, and the extra lookup parquet so clones match what crop and adoy actually read.

Tighten Session 1 so the intro, download, and harmonize steps do not …

ec24691

…repeat each other. Also drop extra gap-fill README pointers and say why the existing lookup tables cover later years.

Restructure the gap-fill README as a method reference instead of a se…

9884571

…cond Session 1. One home per idea; year-pair commands stay in the session. Move CLI and env vars to an appendix.

Remove unused landiq-gapfill helpers that no command path called.

3141262

Document full-gap CLASS prediction from neighbor transitions and CDL.

82629b3

Escape pipes in the gap-fill README lookup table.

ae0049f

Fixed wording.

56f2682

Tell Session 1 which years the shipped lookup tables use.

e69d376

Have crop and ADOY read whatever lookup tables are in outputs/.

64c268d

Fix merge when carried LandIQ years disagree on column types.

8c9bb92

mdietze approved these changes Aug 11, 2026

View reviewed changes

Merge branch 'develop' into feature/ccmmf-statewide-monitoring-inst

7b436c2

mdietze enabled auto-merge August 11, 2026 23:46

Keep LandIQ lookup tables in git but out of the R install tarball.

dad57ca

auto-merge was automatically disabled August 12, 2026 03:01
Head branch was pushed to by a user without write access

mdietze added this pull request to the merge queue Aug 12, 2026

Merged via the queue into PecanProject:develop with commit f3744c8 Aug 12, 2026
21 of 22 checks passed

Navigation Menu

Uh oh!

Adding statewide monitoring workflow under data.remote inst/ccmmf - #3913

Adding statewide monitoring workflow under data.remote inst/ccmmf#3913
mdietze merged 75 commits into
PecanProject:developfrom
sarahkanee:feature/ccmmf-statewide-monitoring-inst

sarahkanee commented Mar 31, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

mdietze Jul 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,56 @@
		# CCMMF monitoring pipeline - documentation

		You are here to process a new LandIQ year.

		@@ -0,0 +1,34 @@
		# CCMMF statewide monitoring (`inst/ccmmf`)

		California LandIQ → phenology → management-event pipeline scripts for PEcAn.

Uh oh!

Conversation

sarahkanee commented Mar 31, 2026

Description

Motivation and Context

Review Time Estimate

Types of changes

Checklist:

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants