Synthetic household load profiles for Simona
simonaMarkovLoad generates realistic, synthetic household load curves for the Simona simulation environment. It uses time-inhomogeneous Markov chains to model when state transitions occur and Gaussian Mixture Models (GMMs) to determine how much load is drawn within each state.
The trained model is exported in the PSDM JSON format (simonaMarkovLoad:psdm:1.0) for direct integration with Simona.
- Semantic handling of cumulative energy, interval energy, and power input values
- Global min/max normalisation across all input files (
minmax_global); the kW scale is exported with the model so consumers can de-normalise - Discretisation of continuous power values into 10 states using quadratic thresholds
- 2,304 temporal buckets based on month, weekday/weekend flag, and quarter-hour slot
- Markov transition matrices per bucket with a self-loop fallback for empty rows (handles sparse data gracefully)
- GMM fitting per (bucket, state) pair with automatic component selection via BIC (1-3 components)
- Parallel training with
joblib - Export of the complete model as a PSDM JSON file
- Built-in simulation and diagnostic visualisations
simonaMarkovLoad/
├── src/
│ ├── config.py # YAML configuration loader
│ ├── main.py # Full pipeline: load → train → export → simulate
│ ├── export.py # PSDM JSON export
│ ├── markov/
│ │ ├── buckets.py # Temporal bucket assignment (2,304 buckets)
│ │ ├── gmm.py # GMM fitting and sampling
│ │ ├── transition_counts.py # Raw transition count matrices
│ │ └── transitions.py # Probability matrices with self-loop fallback
│ └── preprocessing/
│ ├── loader.py # CSV ingestion and power computation
│ └── scaling.py # Normalisation and discretisation
├── tests/ # pytest test suite
├── scripts/
│ └── setup_env.py # Pre-commit hook installer
├── data/ # Input CSV files or pool subdirectories
├── out/ # Model output (psdm_model.json)
├── config.yml # Configuration
└── pyproject.toml
- Python >= 3.13
- uv installed:
pip install uv # or, preferably, via pipx: pipx install uv
git clone https://github.com/ie3-institute/simonaMarkovLoad
cd simonaMarkovLoad
# Install all dependencies (including dev dependencies)
uv sync
# Install and activate pre-commit hooks
uv run setupTo run the hooks manually:
uv run pre-commit run --all-filesPlace your raw CSV files in data/ (either loose or inside a single
subdirectory). The CSV format is expected to have:
| Column | Description |
|---|---|
Zeitstempel |
Timestamp (configurable) |
Messwert |
Energy in kWh or power in kW (configurable) |
The first 21 rows are skipped by default (configurable via config.yml).
Edit config.yml to match your data format:
input:
skiprows: 21 # Header rows to skip
timestamp_col: "Zeitstempel"
value_col: "Messwert"
value_representation: "cumulative_energy"
interval_minutes: 15 # Required for energy input
drop_negative_deltas: true # Drop invalid meter resets/corrections
pools: false # Train one model per data subdirectory
model:
n_states: 10 # Number of load states
gmm:
n_jobs: -1 # Parallel workers for GMM fitting
random_state: 42 # Reproducible GMM initialisation
output:
psdm_json: "out/psdm_model.json"
show_plots: true # Set false for headless training runsvalue_representation defines how the configured value column is interpreted:
| Value | Input | Conversion to kW |
|---|---|---|
cumulative_energy |
Cumulative energy meter reading in kWh | Difference consecutive readings, then divide by the interval duration in hours |
interval_energy |
Energy consumed during each interval in kWh | Divide each value by the interval duration in hours |
power |
Power in kW | Use each value directly |
interval_minutes is required for both energy representations and must be
positive. In power mode it is not used in the calculation, but when specified
it is still validated as a positive, finite value. The temporal model currently
assumes fixed 15-minute intervals, so interval_minutes must remain 15; it
controls only the energy to power scaling. drop_negative_deltas is likewise
allowed in every mode but is applied only to cumulative_energy. When omitted
for that mode, negative deltas are dropped by default.
Enabling input.pools: true trains one model per first-level subdirectory of
data/, sequentially. Each pool is written to out/psdm_model_<folder>.json.
Loose CSV files directly in data/ are ignored in pool mode. By default,
pooling is disabled and one model is trained from loose CSV files or the CSVs
in exactly one subdirectory. Multiple subdirectories with pooling disabled are
rejected.
Optional constant-load triples can be stored by CSV file stem in
constant_loads.yml at the repository root:
SM_00001:
- [330.0, 20, 6]
SM_00002:
- [150, 24, 8]
- [80.0, 12, 2]The values are stored for consumption by future preprocessing steps. Their interpretation is TBD.
uv run python -m src.mainThis will:
Negative cumulative meter deltas are dropped by default before normalisation.
- Load and preprocess raw CSV data (input values → kW, global min/max normalisation, discretisation)
- Assign temporal buckets to each observation
- Build Markov transition matrices (2,304 × 10 × 10)
- Fit GMMs for each (bucket, state) pair in parallel
- Export the model to
out/psdm_model.json - Run a short simulation and display diagnostic plots
The model is written to out/psdm_model.json by default.
{
"schema": "simonaMarkovLoad:psdm:1.0",
"generated_at": "...",
"generator": { "name": "simonaMarkovLoad", "config": { ... } },
"time_model": {
"bucket_count": 2304,
"bucket_encoding": { "formula": "bucket = month*192 + is_weekend*96 + quarter_hour" },
"sampling_interval_minutes": 15,
"timezone": "Europe/Berlin"
},
"value_model": {
"value_unit": "normalized",
"normalization": {
"method": "minmax_global",
"max_power": { "value": ..., "unit": "kW" },
"min_power": { "value": ..., "unit": "kW" }
},
"discretization": { "states": 10, "thresholds_right": [ ... ] }
},
"parameters": {
"transitions": { "empty_row_strategy": "self_loop" },
"gmm": { ... }
},
"data": {
"transitions": { "shape": [2304, 10, 10], "dtype": "float32", "values": [ ... ] },
"gmms": { "buckets": [ { "states": [ { "weights": [...], "means": [...], "variances": [...] }, ... ] }, ... ] }
},
"training_data": { "records": ..., "time_range": { ... } }
}
Instantaneous power values from all input files are normalised together using the global minimum and maximum (minmax_global). The kW scale (min_power, max_power) is written to the exported JSON so that consumers such as Simona can map normalised values back to physical power.
Each 15-minute observation is assigned to one of 2,304 buckets:
bucket = month * 192 + is_weekend * 96 + quarter_hour
This allows transition probabilities and load distributions to vary by time of day, day type (weekday vs. weekend), and season.
For each bucket a 10×10 transition matrix is estimated from historical data. Rows with no observed transitions default to a self-loop (stay in the current state), ensuring valid probability distributions in all cases.
Within each (bucket, state) pair a GMM models the distribution of normalised power values. The number of components (1-3) is selected automatically by minimising BIC. At inference time a component is chosen according to its mixture weight, then a value is drawn from the corresponding Gaussian and clamped to [0, 1].
uv run pytestPre-commit hooks enforce:
- black - code formatting (line length 88)
- isort - import ordering
- ruff - linting (E, F, W, I, N, UP, B, C4, PIE, SIM, RET rules)
| Package | Purpose |
|---|---|
| numpy | Numerical arrays |
| pandas | Data manipulation |
| scikit-learn | Gaussian Mixture Models |
| joblib | Parallel GMM fitting |
| matplotlib | Visualisation |
| tqdm | Progress bars |
Version constraints are maintained in pyproject.toml (locked in uv.lock).
If you have any questions, feel free to reach out to:
Philipp Schmelter - philipp.schmelter@tu-dortmund.de