efuels-supply-potentials

1. Installation

Clone the repository including its submodules:

git clone --recurse-submodules https://github.com/open-energy-transition/efuels-supply-potentials.git

Install the necessary dependencies using conda or mamba based on the Operating System (OS) of the machine:

mamba env create -f envs/environment.{os}-64-pinned.yaml

• Note! Please check for envs/ directory for names of pinned environment files. Supported OSs are windows, linux and macos.

Activate pypsa-earth-efuels environment:

conda activate pypsa-earth-efuels

• Note! At the moment, head of the PyPSA-Earth submodule points to latest stable commit (0fa2e39) of efuels-supply-potential branch of open-energy-transition/pypsa-earth repository. If OS-specific installation of conda environment does not succeed, it is recommended to install general pypsa-earth environment and activate as mamba env create -f submodules/pypsa-earth/envs/environment.yaml and activate by conda activate pypsa-earth. The detailed instructions are provided in PyPSA-Earth Documentation.

1.1. Configure pre-commit

To ensure that all team members can use the pre-commit hooks and maintain code quality across the project, follow these steps:

1. Navigate to the root directory of the repository, where the .pre-commit-config.yaml file is located. Then, run the following command to install the hooks defined in the .pre-commit-config.yaml file:

pre-commit install

This command will set up the hooks in the .git/hooks directory (specifically the pre-commit hook), ensuring that the hooks are executed automatically before each commit.

2. To ensure that all files comply with the defined hooks, users can run the following command manually to check and fix any issues with all files in the repository:

pre-commit run --a

This step is optional but recommended to make sure that any formatting or linting issues are fixed before starting to commit changes.

3. Once the pre-commit hooks are installed and verified, you can proceed to make changes to the code, add them to your commit, and push them to the repository. From this point forward, pre-commit will automatically run checks on your files whenever you try to commit:

git add .
git commit -m "Your commit message"

If any of the hooks (such as black, isort, or reuse lint) detect issues (for example, formatting issues or missing license headers), the commit will be blocked, and you'll be required to fix the issues before retrying.

4. Some hooks, such as black and isort, will automatically fix code formatting issues during the commit process. If a file is not formatted correctly, the hook will fix it, and the user can reattempt the commit. If a hook fails (for example, the license linting hook or a formatting hook fails), the commit will not proceed. The user will see a message explaining the issue, and they will need to correct it before committing again.

5. Every time a new hook is added, or changes to the configuration in the .pre-commit-config.yaml are performed, it is necessary to run pre-commit install again.

2. Running scenarios

This project utilizes snakemake to automate the execution of scripts, ensuring efficient and reproducible workflows. Configuration settings for snakemake are available in the configs/config.main.yaml file as well as scenario-specific configuration files located in configs/.

2.1. Running the base scenario

To run the power model for the base scenario of the U.S., navigate to the working directory (.../efuels-supply-potentials/) and use the following command:

snakemake -call solve_all_networks --configfile configs/calibration/config.base.yaml

• Note! All following snakemake commands needs to be executed in the working directory (.../efuels-supply-potentials/).

To run the sector-coupled model of the base scenario, execute the command:

snakemake -call solve_sector_networks --configfile configs/calibration/config.base.yaml

2.2. Running scenarios for future years

A. Running an individual horizon

To run the power model for the Reference scenario of the U.S., navigate to the working directory (.../efuels-supply-potentials/) and use the following command:

snakemake -call solve_all_networks --configfile configs/scenarios/config.20**.yaml

• Note! Configuration files for future years are currently available for 2030, 2035 and 2040 (replace the "**" in the command above with one off the mentioned years).

To run the sector-coupled model for the Reference scenario, execute the command substituting the desired year to "**" in the command below:

snakemake -call solve_sector_networks --configfile configs/scenarios/config.20**.yaml

B. Running for multiple horizons

To run the sector-coupled model for the Reference scenario using myopic optimization for the years 2030, 2035, and 2040 consecutively, execute the following:

snakemake -call solve_sector_networks_myopic --configfile configs/scenarios/config.myopic.yaml

3. Snakemake rules

Rule name	Config file	Description
validate_all	config.base.yaml, config.base_AC.yaml	Performs country-level validation comparing with EIA and Ember data
statewise_validate_all	config.base_AC.yaml	Performs statewise validation comparing with EIA data
get_capacity_factors	Any base or scenario config file	Estimates capacity factors for renewables
process_airport_data	-	Performs analysis on passengers and jet fuel consumption data per state and generates plots and table. Also generate custom airport data with state level based demand
generate_aviation_scenario	Any base or scenario config file	Generates aviation demand csv file with different future scenario
modify_aviation_demand	Any base or scenario config file	Switches aviation demand in energy_total to custom demand
preprocess_demand_data	Any base or scenario config file	Preprocess utlities demand data into geojson
build_demand_profiles_from_eia	Any base or scenario config file	Build custom demand data from eia and bypass build_demand_profiles
set_saf_mandate	Any base or scenario config file	Add e-kerosene buses to enable split of aviation demand and sets SAF mandate if enabled
build_custom_industry_demand	Any base or scenario config file	Estimate node-level demands for selected custom industries (e.g. ammonia, ethanol, cement, and steel)
add_custom_industry	Any base or scenario config file	Add selected custom industries into the network
prepare_growth_rate_scenarios	Any base or scenario config file	Allow automatic fetching of correct growth rate files according to the demand_projection scenario name
solve_custom_sector_network	Any base or scenario config file	Allow application of state/country-wise clean/RES electricity generation polices and apply tax credits to selected sectors. The constraints are turned on by default.

Retrieve rules

Rule name	Config file	Description
retrieve_cutouts	Any base or scenario config file	Retrieve US cutouts from google drive
retrieve_osm_raw	Any base or scenario config file	Retrieve resources/{RDIR}/osm/raw/ data from google drive and bypasses download_osm_data rule
retrieve_osm_clean	Any base or scenario config file	Retrieve resources/{RDIR}/osm/clean/ data from google drive and bypasses clean_osm_data rule
retrieve_shapes	Any base or scenario config file	Retrieve resources/{RDIR}/shapes/ data from google drive and bypasses build_shapes rule
retrieve_osm_network	Any base or scenario config file	Retrieve resources/{RDIR}/base_network/ data from google drive and bypasses build_osm_network rule
retrieve_base_network	Any base or scenario config file	Retrieve base.nc data from google drive and bypasses base_network rule
retrieve_renewable_profiles	Any base or scenario config file	Retrieve resources/{RDIR}/renewable_profiles/ data from google drive and bypasses build_renewable_profiles rule
retrieve_custom_powerplants	Any base or scenario config file	Copy data/custom_powerplants.csv to submodules/pypsa-earth/data/ folder
retrieve_ssp2	Any base or scenario config file	Copy data/NorthAmerica.csv to submodules/pypsa-earth/data/ssp2-2.6/. directory
retrieve_demand_data	Any base or scenario config file	Retrieve utility demand data from google drive to data/demand_data/*

• RDIR - scenario folder

4. Updates to the working branch of PyPSA-Earth submodule

4.1. Cherry-picking

Cherry-picking allows applying specific commits from one branch to another. We cherry-picked the important commits from upstream pypsa-earth to our project branch (efuels-supply-potentials). The commits of the following PRs were integrated to project branch:

1. PR #1369: Restore functioning of myopic optimization.
2. PR #1372: Scale temporal loads based on temporal resolution.
3. PR #1381: Remove space in rail transport oil and electricity carriers.
4. PR #1400: Add US-specific demand growth rates and fuel shares (Medium scenario).
5. PR #1410: Fix negative transport demand.
6. PR #1401: Fix H2 pipeline bus names.
7. PR #1422: Fix renamed column in transport related Wikipedia data.
8. PR #1428: Change source for Aluminum production data.
9. PR #1465: Enable power plants filtering using query.
10. PR #1468: Include missing efficiency gains and growth rates for other energy use.
11. PR #1479: Update link for North America cutout.
12. PR #1486: Align prepare_transport_data_input with new structure of the Wikipedia page table.

Please review a short tutorial on cherry-picking in Git to get more familiar with procedure.

4.2. Direct commits to PyPSA-Earth

1. PR #32: Disable implicit calculations and assigning of industry demands for steel and cement industries, because they are added explicitly.
2. PR #36: Introduce custom H2 production technologies.
3. PR #38: Enable correct functioning of myopic optimization.
4. PR #40: Adjust calculation of no_years to properly run 2023 scenario.
5. PR #50: Introduce Universal Currency Conversion to use USD as reference currency.
6. PR #51: Add US cost configurations and split scenarios per technology group.
7. PR #56: Introduce currency conversion in simplify_network.
8. PR #57 PR #58 PR #59: Fix logic for currency conversion to handle past and future years and introduce clear log and warning messages.
9. PR #63: Introduce p_max_pu for nuclear generators (or links, if necessary) to match base year statistics and to apply it to future years.
10. PR #65: Remove lignite from default conventional carriers.
11. PR #69: Extend lifetime of nuclear power plants to 60 years
12. PR #71: Enable selection of custom busmap.
13. PR #76: Add functionality to overwrite cost attributes in sector model.
14. PR #84: Add possibility to overwrite discount rate.
15. PR #86: Use H2 Store Tank costs without compressor and add lifetime.
16. PR #89: Align cost conversion with reference year for costs in input files.
17. PR #91: Include existing batteries from powerplants.csv.

5. Checking techno-economic input data

To check techno-economic input data for selected sectors (power generation, hydrogen and e-kerosene production, CO2 capture) it is not necessarily needed to run the entire workflow and check input cost data files.

To generate Excel files reporting techno-economic input data (for the Moderate and the Advanced cost scenarios adopted in the model), navigate to the working directory (.../efuels-supply-potentials/) and use the following command:

python scripts/non_workflow/fetch_input_costs_multiyear.py

The Excel files will be written to the .../efuels-supply-potentials_emmanuel_fork/notebooks/data/input_costs/ folder.

5. Computational reproducibility

The optimization problems generated by this project become extremely large when high spatial and temporal resolution are used. In such cases, the computational cost is primarily driven by memory-intensive linear algebra operations required during the barrier matrix factorization of the optimisation problem.

Hardware and solver setup

All optimization runs were executed on homogeneous HPC nodes with the following configuration:

• CPU: Intel Xeon Gold 6342 (2.80 GHz)
• Cores: 48 physical cores (96 logical processors)
• Operating system: Debian GNU/Linux 12 (bookworm)

All optimization problems were solved using Gurobi Optimizer 13.0.0.

The following solver parameters were used consistently across all runs:

Method = 2        # Barrier algorithm
Crossover = 0     # Disable crossover to avoid additional memory and runtime overhead
BarConvTol = 1e-4
BarHomogeneous = 1
Threads = 8

No solver parameters were tuned on a per-scenario basis.

Problem size

The resulting linear optimisation problems are extremely large. Typical model sizes are:

Temporal resolution	Rows	Columns	Nonzeros
3-hour	~100–150 million	~50–75 million	~200–350 million
1-hour	~280–440 million	~135–220 million	~630–1020 million

Although presolve procedures eliminate a large fraction of constraints and variables, the resulting presolved systems remain extremely large.

Memory requirements

Memory requirements are characterized using the Estimated Factorization Memory (EFM) reported by the solver. EFM represents the memory required to store the numerical factorization of the linear system solved at each barrier iteration.

Typical EFM values observed:

Temporal resolution	Estimated factorization memory
3-hour	~100–130 GB
1-hour	~500 GB

While EFM does not correspond to total solver memory consumption, it provides a good proxy for peak memory requirements because matrix factorization dominates the memory footprint of barrier-based methods.

Runtime

Typical wall-clock solution times observed in the experiments are:

Temporal resolution	Runtime (per time horizon)
3-hour	~1–3 days (typically 2 days)
1-hour	~7–12 days

Solver logs indicate that 85–95% of runtime is spent in repeated linear solves during barrier iterations, while presolve, matrix ordering and other overheads account for only a small fraction of total runtime.

Implications for temporal resolution

The 3-hour temporal resolution used in the main analysis represents a trade-off between temporal detail and computational feasibility.

At 1-hour resolution, the base-year optimization alone requires approximately one week of runtime and around 500 GB of factorization memory, making multi-scenario analyses computationally very expensive. The 3-hour resolution significantly reduces both runtime and memory requirements while preserving the main temporal dynamics relevant for the analysis.