Make streams configurable at dataset level#510
Make streams configurable at dataset level#510Naomi Parsons (NParsonsMO) wants to merge 18 commits into
Conversation
Naomi Parsons (NParsonsMO)
added
standardise
Naomi Parsons (NParsonsMO)
changed the title
| # Check whether a custom stream map is set to be used | ||
| dataset = os.environ["CYLC_TASK_PARAM_dataset"].strip() | ||
| with open(f"{os.environ['DATASETS_LIST_DIR']}/model_runs.yml") as f: | ||
| content = yaml.safe_load(f) | ||
| dataset_dict = content[dataset] | ||
| logger.debug("Dataset information:\n%s", dataset_dict) | ||
|
|
||
| # Get path to stream mappings | ||
| streams_config = os.environ["STREAM_CONFIG_PATH"] | ||
| if dataset_dict["use_custom_data_streams"] == "true": | ||
| streams_config = dataset_dict["path_to_custom_streams_config"] | ||
| else: | ||
| streams_config = os.environ["STREAM_CONFIG_PATH"] |
There was a problem hiding this comment.
This adds exactly the same code in two different files.
Find a better way.
There was a problem hiding this comment.
We should discuss which runs count as "standard" for writing the streams to this file.
I've written the MJO variables' streams as they are in the not for CMIP model run (u-cw...), as I assume this better represents future uses, but this means we have to provide the custom path for the u-az... and u-bv... runs (in order to run the MJO recipe, not for radiation budget). Eventually I'm guessing users will pull these kinds from CHAMP, as actual CMIP runs.
|
|
||
| # Read the model run information from the model_runs.yml file | ||
| model_runs_yaml = Path(os.environ["DATASETS_LIST_DIR"]) / "model_runs.yml" | ||
| model_runs_yaml = os.environ["MODEL_RUNS_CONFIG"] |
There was a problem hiding this comment.
I found this referenced in a few places.
There was a problem hiding this comment.
Currently this is just a single function that's imported by the two files in this directory so that I didn't write the code twice (and tested it only once), so it doesn't have main().
As it applies only to these two highly related files, I put it here rather than in CMEW/lib/python (it's also much easier to unit test from here).
| - Amon/tas | ||
| - Emon/rls | ||
| - Emon/rss | ||
| - Amon/hfls |
There was a problem hiding this comment.
The spacing worked before, but I think this way is standard.
| yaml.dump(dataset_dict, f) | ||
|
|
||
|
|
||
| def format_as_bools(filepath): |
There was a problem hiding this comment.
Using Rose to set the type to either of "boolean" or "python_boolean" resulted in a quoted string ('true' or 'True'), so this is just to make querying the file easier later (and it makes the file neater to look at, and to reproduce a different way in future if we moved away from the whole namelist thing).
There was a problem hiding this comment.
Tests before were using the actual file, so they started failing. They should have used a mock to begin with.
| DATASETS_LIST_DIR = "${CYLC_WORKFLOW_SHARE_DIR}/dataset_lists" | ||
| DATASETS_LIST_DIR = ${CYLC_WORKFLOW_SHARE_DIR}/dataset_lists | ||
| # This particular file controls behaviours throughout the workflow | ||
| MODEL_RUNS_CONFIG = ${DATASETS_LIST_DIR}/model_runs.yml |
There was a problem hiding this comment.
This is the path I found referenced a few times so I thought it belonged here.
| experiment_id="historical" | ||
| label_for_plots="UKESM1.0 N96ORCA1" | ||
| model_id="UKESM1-0-LL" | ||
| path_to_custom_streams_config="$HOME/my_own_streams.yml" |
There was a problem hiding this comment.
If I know what these "custom" streams are for some styles of model run, should we have that stored somewhere in CMEW?
| description=The calendar used by the run. | ||
| help=Only 'gregorian' and '360_day' are currently supported. | ||
| sort-key=6 | ||
| sort-key=16 |
There was a problem hiding this comment.
The sorting seems to work more like the sub-unitary digits of a decimal than an integer of unspecified number of digits.
|
Naomi Parsons (@NParsonsMO), would it be possible to get a walkthrough of this PR at the surgery on Monday, please? I'm struggling to understand the quantity of changes. My immediate reaction is: how did we end up with so much Python code in CMEW? We might want to evaluate whether any of it can be added back into Rose and / or Cylc, and also consider pulling out all the Python code into a Python package (which can then be properly tested; #228). This would make it easier to replace / share Python code in the future. |
2 participants
Closes #509
This comment may be useful while reviewing:
#509 (comment)
PR creation checklist for the developer
<issue_number>above ☝️ has been replaced with the issue number.mainhas been selected as the base branch.<issue_number>_<short_description_of_feature>.good first issuelabel) have been added to the PR.Climate Model Evaluation Workflow (CMEW)project has been added to the PR.Definition of Done for the developer
docdirectory, including the Quick Start section; select one of the following):Copyright and IPR:
PR creation checklist for the reviewer
<issue_number>above ☝️ has been replaced with the issue number.mainhas been selected as the base branch.<issue_number>_<short_description_of_feature>.good first issuelabel) have been added to the PR.Climate Model Evaluation Workflow (CMEW)project has been added to the PR.Definition of Done for the reviewer
docdirectory, including the Quick Start section; select one of the following):Important
#<pull_request_number>: <pull_request_title>when writing the merge commit message for the pull request, so the pull request number is immediately visible on GitHub, regardless of the length of the pull request title.