Prepare the MERGE

[davide-f]

Copying from #308

Changes proposed in this Pull Request

This PR should clean the scripts, enhance the readability and overall architecture of the model preparing for a future model merge with the power sector model. All open PRs should be reviewed and merged before this one.

Subtasks

This PR consists of sevaral tasks

• 1. Prepare the config files.
  • Clean the config file and remove any duplication. Repo cleaning #350
  • Align with PyPSA-Earth -> Issue 312 use single config #313 and what remains can be done later
  • Use a single config file -> Issue 312 use single config #313
• 2. Improve the script prepare_sector_network.py.
  • Enhance code quality and architecture. -> optional, not spend too much time here for the merge
  • Add sufficient documentation.
• 3. Revise the heat sector.
  • Check the heat scripts. -> secondary priority
• 4. Prepare the snakefile.
  • Add params as PyPSA-Earth #330
  • Add modularity -> TO DO AFTER the merge
  • Enable scenario analysis and link it with PyPSA-Earth -> second priority
• 5. General style improvement
  • Drop warnings from scripts Pandas deprecations - offset warning #353
  • Remove duplication of helper information (using sys.append in the helpers to link scripts from the main repo for example) -> secondary priority, thanks to Merge helpers pypsa-earth#1029 and potentially finalized during the merge
• 6. Other functionalities
  • Support last environment and pypsa version, by adopting linopy Linopy Integration #351 . PyPSA-Earth has a ready PR that supports it and it may be interesting to have this too

[finozzifa]

Use a single config file --> #313

[finozzifa]

Hi @davide-f,
what is it meant exactly with Prepare the snakefile, align with PyPSA-Earth
Thanks : )

[davide-f]

Macrotopics

Usability

• compatibility with previous repos

Procedure

• define procedure to avoid conflicts
  • avoid files in the scripts folder to have the same name across repos
  • do not have file "snakefile" in both repos
  • make sure no other file has the same path and name
• define the repo structure [the -eur one?]
  • have a scripts folder only or more? 1 only is easy but overwhelming; more may be tricky from a coding perspective
  • Maybe define smart naming conventions: e.g. retrieve_power/sec_...
    Add these only if needed
• restructure the repos to make sure to avoid git merge conflicts
• overlapping rules need to be sorted out
• move issues across repos

1. config files
2. snakefile modularity
3. styling convention of python (snakemake params and scenario runs)
4. structure of the repo
   • we keep 1 script folder only
   • do we want the networks folder?
5. how to make sure to satisfy usability?
   • make sure that the commands for both CIs work the same
6. need to revise the documentation folder accordingly

COMMUNICATE TO THE COMMUNITY:

• we go for the merge
• in the next weeks, we will be preparing the repos
• need to close PRs before merge
• once we start the merge, no more merge and it will break existing PRs

Past experience

In -eur there have been misalignment between the repos to be fixed.
A lot of brute force.

First, the two packages have been merged (brute force) and then let's see.
Most snakefile rules were diverging, thus lot of rearrangement, done during the merge-> the snakefile has been modularized into multiple .smk files.
The config: large merge issue: append entries first and then consolidated. Done lots of brute force. [3 intense days]
Not much preparation

Suggestion: make sure that the two git repositories are on the same state: use the latest version.

Concrete steps

1. config files
2. snakefile modularity: secondary priority
3. styling convention of python:
   • drop deprecations
   • update to linopy?
   • snakemake params [not necessary but maybe nice to have]
   • scenario runs [third priority]
4. structure of the repo
   • we keep 1 script folder only
   • do we want the networks folder? -> ask to the community
5. how to make sure to satisfy usability?
   • make sure that the commands for both CIs work the same
6. need to revise the documentation folder accordingly

Timeline

• low availability by Hazem
• fabian, from 2 weeks may have availability

2-4 weeks preparation
weekly update during the -sec model
4-8 weeks target for merge success

[ekatef]

In part of linopy integration: we have a PR opened in pypsa-eath. No problem to keep it opened to after merge. But also happy to support integration of it into pypsa-earth-sec before merge.

[energyLS]

Updates from meeting on 8/7/2024

Two-step-procedure:

1. Replace subworkflow by submodule (make sure compatibility is there). Then git knows, that rules are distinct. Then we can add them into one snakefile at the actual merge
2. Merge-PR: Collecting: Stacking rules and scripts into one repo. Fabian has command lines written down: Both histories should be respected

Post-merge:

1. Split snakefile into .smk files
2. Improve data retrieval process
3. Improve management of config files

General:

1. Scary: let git do the magic
2. Semantic versioning: We may need additional release for PyPSA-Earth-Sec if subworkflows are changed to submodules (0.2)
3. Improve documentation on PyPSA-Earth-Sec (make sure the community is updated and knows how to handle issues) + documentation on scripts (prepare sector network)
4. General style improvements; address pandas deprications

Timeline:

1. Fabian experiments with submodules and reports back. If it works, then he will set up a PR on PyPSA-Earth-Sec. (reports back on Discord by approx. tomorrow)
2. Then: Fabian tries an internal merge on PyPSA-Earth, to get a feeling wether this approach works out (15 or 16th of August)
3. If successful: new release 0.2. Of PyPSA-Earth-Sec
4. Set up the actual merge PR on PyPSA-Earth (with git magic)