Harden grid loading: whole-grid validation, solver-output checks, trust-boundary docs

.github/workflows/sanitizers.yml

@@ -67,7 +67,9 @@ jobs:
             test_DCSecurityAnlysis \
             test_SecurityAnlysis_cpp \
             test_ContingencyAnalysis_limit_violations \
-            test_ContingencyAnalysis_split
+            test_ContingencyAnalysis_split \
+            test_check_grid \
+            test_LSGrid_out_of_bounds
 
   debug_asserts:
     # Release wheels are compiled with NDEBUG, which silences every Eigen

CHANGELOG.rst

@@ -113,6 +113,48 @@ TODO: Levenberg-Marquardt damping (a.k.a. Tikhonov-regularized Newton) : adding
 
 [0.14.0] 2026-xx-yy
 ---------------------
+- [ADDED] ``GridModel.check_grid()`` (C++ ``LSGrid::check_grid()``): a whole-grid
+  consistency check that verifies every index the grid carries (element bus ids,
+  substation ids, position in the topology vector, generator slack and
+  remote-regulated bus references) is in range. It raises ``IndexError`` /
+  ``RuntimeError`` on an inconsistency.
+- [ADDED] the grid is now validated automatically with ``check_grid()`` when it is
+  loaded (from a pickle or the binary format, via ``set_state``) and by every grid
+  loader (``init_from_pandapower`` / ``init_from_pypowsybl`` / ``init_from_matpower``
+  / ``init_from_powermodels``). A well-formed but inconsistent state (e.g. an
+  out-of-range bus id in a crafted binary file) now raises a clean exception instead
+  of causing an out-of-bounds access during the next powerflow.
+- [BREAKING] solver names are now restricted to ``[A-Za-z_][A-Za-z0-9_.]{0,63}`` (start with
+  an ASCII letter or ``_``; then ASCII letters, digits, ``_`` or ``.``; at most 64
+  characters). Registering any other name is refused. A solver name is written into every
+  serialized grid and is what re-selects the solver on load, so it has to be an identity
+  that cannot be spoofed (a non-ASCII homoglyph of a built-in name), cannot inject content
+  into an error message or a log (control characters), and is bounded in length. Every
+  built-in and every name used by the shipped example plugins already complies; a plugin
+  using an exotic name must rename its solver.
+- [BREAKING] the binary format version is bumped to 4: the AC / DC algorithm is now stored
+  as its registry **name** instead of an ``AlgorithmType`` enum. Files written by an older
+  lightsim2grid are rejected with a clear error (they were already only readable by the
+  same format version).
+- [FIXED] a grid using a solver from a plugin could be saved but never loaded back, and
+  could not even be copied: only the ``AlgorithmType`` was stored, and every external
+  solver collapses onto ``AlgorithmType::Custom``, which is not a concrete solver. The
+  solver name is now stored, so such a grid round-trips (and ``copy()`` works) as long as
+  the plugin is loaded.
+- [ADDED] when a grid is loaded whose solver is not registered here (a plugin that has not
+  been loaded, or a built-in needing an optional KLU / NICSLU / CKTSO backend this build
+  lacks), the error now names the solver, says how to obtain it, and lists the solvers
+  that *are* available -- instead of an internal message about ``AlgorithmType::Custom``.
+- [ADDED] ``LSGrid.load_binary_without_algorithm(path)``: loads the grid data without
+  re-selecting the solver it was saved with (keeping the default solvers), so a grid saved
+  with an unavailable solver can still be loaded. All other checks are unchanged.
+- [ADDED] a sanity check on the voltages returned by an **external** (plugin) solver
+  before they are consumed: a wrong-sized result raises, and a non-finite one is
+  reported as a non-converged solve instead of propagating ``NaN`` / ``Inf``. Built-in
+  solvers skip this check entirely, so they pay nothing for it.
+- [ADDED] a "Security" documentation page describing the trust boundaries (pickle,
+  solver plugins and grid2op environments are trusted-input-only; the binary format is
+  the least dangerous channel, and is validated with ``check_grid`` on load).
 - [BREAKING] For plugin developers (C++ side): solver plugins no longer self-register
   from a static ``AlgorithmRegistrar`` constructor firing during ``dlopen``. Instead a
   plugin writes a ``void(ls2g::PluginRegistrar&)`` registration function and exposes it

docs/binary_serialization.rst

@@ -60,20 +60,42 @@ Individual element containers work the same way:
     corrupted internal sizes (every count in the file is checked against the real file size *before*
     anything is allocated), files that contain an object of a different type (*eg* loading a
     ``LoadContainer`` file with ``StorageContainer.load_binary``), and files with unexpected trailing
-    bytes are all rejected. ``save_binary`` on the other hand is **atomic by default**: it writes to a
+    bytes are all rejected. On top of that byte-level validation, loading a whole grid also runs
+    :py:meth:`LSGrid.check_grid` (the same consistency check used everywhere a grid is loaded): a file
+    that is byte-wise well-formed but carries an out-of-range index (bus / substation /
+    topology-vector) is rejected with an ``IndexError`` (out-of-range index) or a ``RuntimeError``
+    (structural inconsistency) rather than being loaded into a state that would fault on the next
+    powerflow. ``save_binary`` on the other hand is **atomic by default**: it writes to a
     temporary file that only replaces the destination once fully written, so an interrupted save never
     destroys a previously saved file. Pass ``atomic=False`` to write the destination directly instead
     (marginally faster -- it skips one temporary file and rename -- but without that protection).
 
+.. note::
+    **The solver is saved by name.** A grid records the *registry name* of the AC and DC
+    algorithm it was using (``'NR_SparseLU'``, or whatever a plugin registered itself as),
+    and ``load_binary`` re-selects the solver with that name. Two consequences:
+
+    - if that solver is not registered when you load the file -- a plugin you have not
+      loaded in this process, or a built-in needing an optional backend (KLU / NICSLU /
+      CKTSO) this build lacks -- ``load_binary`` raises, naming the solver and how to get
+      it. Use ``LSGrid.load_binary_without_algorithm(path)`` to load the grid *data* and
+      keep the default solvers instead, then pick one with ``change_algorithm``;
+    - a name is **not** a strong identity. If two different plugins register the same
+      solver name, the one loaded when the file is read is the one you get -- it may be a
+      completely different algorithm from the one that was used when the file was written,
+      with no way for lightsim2grid to notice. Give your plugin solvers distinctive names
+      (a project prefix, for instance) if you exchange files between environments.
+
 .. danger::
     Not even pickle is a *portable* format, and this binary format is even less so. Two
     rules to follow:
 
     - **Never load a binary file from a source you do not trust.** ``load_binary``
-      rejects structurally ill-formed input (see above), but it cannot tell a
-      legitimate grid from a maliciously crafted one that happens to be well-formed --
-      loading it still means trusting its content outright, the same way loading a
-      pickle file does.
+      rejects structurally ill-formed input and, via ``check_grid``, a grid with
+      out-of-range indices (see above), but a well-formed, internally-consistent
+      file can still carry adversarial *values* -- loading it
+      means trusting its content, the same way loading a pickle file does. See also
+      :ref:`security`.
     - **Generate the file on the machine that will consume it, with the same
       lightsim2grid version.** The format stores raw native data (same endianness,
       ``real_type``, ...) with no checksum and no cross-platform migration; the

docs/index.rst

@@ -64,6 +64,7 @@ This is a work in progress at the moment
    cpp_library
    solver_plugin
    binary_serialization
+   security
 
 
 Indices and tables

docs/security.rst

@@ -0,0 +1,121 @@
+.. _security:
+
+Security and trust boundaries
+=============================
+
+This page describes what lightsim2grid does, and does **not**, protect you from
+when you load a grid or a solver that you did not produce yourself. It is meant
+to help you decide which inputs are safe to accept from a third party.
+
+The short version
+-----------------
+
+.. list-table::
+   :header-rows: 1
+   :widths: 30 70
+
+   * - Channel
+     - Trust required
+   * - Unpickling a grid (``pickle.load`` / ``joblib`` / ...)
+     - **Trusted input only.** Loading a pickle can execute arbitrary Python
+       code *by design* — this is a property of Python's ``pickle`` module, not
+       of lightsim2grid. Never unpickle a grid (or anything else) that came from
+       an untrusted source.
+   * - Loading a solver plugin (:func:`lightsim2grid.load_algorithm_plugin`)
+     - **Trusted code only.** A plugin is a native shared library that runs
+       in-process; once loaded it can do anything the host process can. Only
+       load plugins you built or otherwise trust.
+   * - Creating a grid2op environment (``grid2op.make``)
+     - **Trusted input only.** An environment is not just data: grid2op reads
+       and executes python code shipped with it (its ``config.py``, the classes
+       it points to, ...). Only use environments you trust.
+   * - Loading a grid from the fast binary format (``save_binary`` /
+       ``load_binary``)
+     - **Least dangerous, still not a safe channel.** See below.
+   * - Building a grid from a source file (pandapower, pypowsybl, matpower,
+       powermodels — including through a grid2op environment)
+     - **Validated on load.** See below.
+
+Pickle, plugins and grid2op environments are trusted-input-only by design
+------------------------------------------------------------------------
+
+Unpickling arbitrary data, loading a native plugin or using a grid2op
+environment are all, fundamentally, ways of running code chosen by whoever
+produced the input. No amount of validation inside lightsim2grid changes that,
+so the only safe rule is: **do not load a pickle or a plugin, nor call**
+``grid2op.make``\ **, on something coming from a source you do not trust.** This
+is the same posture as ``numpy``, ``pandas`` (``read_pickle``), ``torch.load``
+and any other library that supports pickling.
+
+The version string embedded in a pickled grid is a *compatibility* check (it
+refuses to load a grid saved by a different lightsim2grid version); it is **not**
+a security control.
+
+The binary format is the least dangerous channel
+-------------------------------------------------
+
+.. warning::
+    "Least dangerous" is not "safe": loading data you do not trust is a bad
+    idea whatever the format, and none of the validation described here makes
+    it a good one. The point below is only that this format gives an attacker
+    strictly less to work with than pickle does — not that it is meant for
+    untrusted input.
+
+The fast binary format (:ref:`binary-serialization`) was designed as an
+additive alternative to pickle. Loading a binary file never executes code from
+the file, and it is validated at two levels:
+
+* **Byte level.** The reader checks the file's magic number, format version and
+  object type, validates every length field against the actual file size
+  *before* allocating (so a corrupted size cannot trigger a huge allocation),
+  and requires the file to be fully consumed. Malformed, truncated or corrupted
+  bytes raise ``RuntimeError``.
+* **Semantic level.** After the bytes are read back into a grid, ``check_grid()``
+  runs automatically (see below): a file that is byte-wise well-formed but
+  *inconsistent* — for example one with an out-of-range bus id — raises a clean
+  exception instead of leaving the grid in a state that would cause an
+  out-of-bounds memory access on the next powerflow.
+
+``check_grid()``: whole-grid consistency validation
+---------------------------------------------------
+
+:py:meth:`LSGrid.check_grid` verifies that a grid
+is internally consistent and safe to run a powerflow on. It checks that every
+index the grid carries is in range — the bus id of every element, the substation
+id and the position in the topology vector (both optional), and the generator
+slack and remote-regulated bus references. It raises ``IndexError`` (an
+out-of-range index) or ``RuntimeError`` (a structural inconsistency), and returns
+``None`` for a consistent grid.
+
+You normally do not need to call it yourself:
+
+* it runs automatically when a grid is loaded from a pickle or the binary format
+  (via ``set_state``);
+* every grid loader (``init_from_pandapower``, ``init_from_pypowsybl``,
+  ``init_from_matpower``, ``init_from_powermodels``) calls it before returning.
+
+It is exposed so you can validate a grid you built or modified by hand. It runs
+in time proportional to the number of elements in the grid, so it is relatively
+cheap compared to a powerflow.
+
+.. note::
+
+   Release wheels are compiled with ``-O3 -DNDEBUG``, which removes Eigen's and
+   the standard library's own bounds checks. ``check_grid()`` (and the size /
+   finiteness check applied to the voltages an *external* solver returns) are
+   deliberately *not* gated behind that flag: they always run, because they are
+   what turns a malformed input into a clean exception rather than an
+   out-of-bounds access.
+
+A note for solver-plugin authors
+---------------------------------
+
+A solver plugin is trusted code, so lightsim2grid does not try to sandbox it.
+It does, however, sanity-check the voltages your ``compute_pf`` returns before
+using them — for *external* solvers only, so the built-in ones pay nothing for
+it: if you report convergence but return a voltage vector of the wrong
+size, the powerflow raises ``RuntimeError`` (this would otherwise be an
+out-of-bounds read); if you return non-finite values, the solve is reported as
+non-converged rather than propagating ``NaN`` / ``Inf`` into the results. Writing
+a correct plugin therefore means returning ``V`` / ``Va`` / ``Vm`` sized to the
+solver's bus count, and reporting non-convergence yourself when appropriate.

docs/solver_plugin.rst

@@ -53,6 +53,43 @@ The lookup flow is:
            └─ AlgorithmRegistry::instance().make("MySolver")
                 └─ factory()  →  unique_ptr<BaseAlgo>
 
+.. important::
+    **Choose your solver name carefully: it is an identity, and it is persisted.**
+    When a grid is saved (pickle or :ref:`binary-serialization`) it records the
+    *name* of the solver it was using, and loading re-selects the solver
+    registered under that name.
+
+    Because of that, names are restricted to a small ASCII subset — registering
+    anything else is refused, with an error explaining the rule::
+
+        first character  : A-Z a-z _
+        other characters : A-Z a-z 0-9 _ .
+        length           : 1 to 64 characters
+
+    (``lightsim2grid.lightsim2grid_cpp``'s C++ header exposes this as
+    ``ls2g::is_valid_solver_name``.) Non-ASCII characters are excluded so that no
+    plugin can register a name that *looks* identical to another solver's — say
+    ``NR_SparseLU`` with a Cyrillic ``а`` — and masquerade as it in error
+    messages, documentation and ``available_algorithm_names()`` listings while
+    being a different registry key. Control characters are excluded because names
+    are printed into error messages and logs, where a newline or an ESC byte
+    would let a name inject content. The length bound keeps a name from
+    overflowing the diagnostic buffer a plugin is given.
+
+    Beyond the character set:
+
+    - a grid saved while using your plugin can only be loaded where that plugin is
+      loaded. Otherwise loading raises an error naming the missing solver; the
+      reader can fall back to ``LSGrid.load_binary_without_algorithm(path)`` to get
+      the grid data with the default solvers;
+    - the name is the *only* thing stored — there is no version, checksum or
+      identity of the plugin itself. If two plugins register the same name, a grid
+      saved with one will silently be loaded with the other, even though they may
+      be entirely different algorithms. lightsim2grid cannot detect this, so
+      **prefix your solver names** with something specific to your project
+      (``"acme_NR_fast"`` rather than ``"NR_fast"``) if your files travel between
+      environments.
+
 The ``AlgorithmRegistry`` C++ API (defined in ``AlgorithmRegistry.hpp``, installed to ``include/lightsim2grid/``):
 
 .. code-block:: cpp

lightsim2grid/network/from_pandapower/initLSGrid.py

@@ -171,4 +171,8 @@ def init(pp_net: "pandapower.auxiliary.pandapowerNet",
     # deal with slack bus
     _aux_add_slack(model, pp_net, pp_to_ls, pp_orig_file)
 
+    # make sure the grid we just built is internally consistent (bus / substation
+    # / topology-vector indices in range, no NaN/Inf in the physical inputs)
+    model.check_grid()
+
     return model

lightsim2grid/network/from_powermodels/initLSGrid.py

@@ -135,4 +135,10 @@ def init(network: dict,
     # init the HVDC lines, if any
     _aux_add_dc_line(model, network, pm_to_ls, isolated_ls_bus)
 
+    # make sure the grid we just built is internally consistent (bus / substation
+    # / topology-vector indices in range, no NaN/Inf in the physical inputs).
+    # This also covers grids loaded from matpower, which are converted to the
+    # powermodels format and routed through this function.
+    model.check_grid()
+
     return model

lightsim2grid/network/from_pypowsybl/initLSGrid.py

@@ -305,6 +305,11 @@ def init(net : pypo.network.Network,
     model.set_line_to_sub2_id(lex_sub["sub_id"].values)
     model.set_trafo_to_sub1_id(tor_sub["sub_id"].values)
     model.set_trafo_to_sub2_id(tex_sub["sub_id"].values)
+
+    # make sure the grid we just built is internally consistent (bus / substation
+    # / topology-vector indices in range, no NaN/Inf in the physical inputs)
+    model.check_grid()
+
     if not return_sub_id:
         return model
     else:

lightsim2grid/tests/test_binary_serialization.py

@@ -481,11 +481,17 @@ def _check_load(self, corrupted, description):
             f.write(corrupted)
         try:
             self.grid_cls.load_binary(bad_path)
-        except RuntimeError:
-            pass  # corruption detected and rejected cleanly: fine
+        except (RuntimeError, IndexError):
+            # Corruption detected and rejected cleanly. RuntimeError comes from the
+            # byte layer (BinaryArchive: bad magic/version/type, truncation,
+            # corrupted length). IndexError comes from the semantic layer
+            # (check_grid, run on load): a file that is byte-wise well-formed but
+            # carries an out-of-range index (bus / substation / topology-vector).
+            # Either way it is a clean, catchable exception -- never a crash.
+            pass
         except BaseException as exc:
             self.fail(f"{description}: load_binary raised {type(exc).__name__} "
-                      f"instead of RuntimeError: {exc}")
+                      f"instead of RuntimeError / IndexError: {exc}")
 
     def test_single_byte_flips(self):
         """Invert one byte at every offset of the file."""
@@ -574,7 +580,7 @@ def test_converter_station_type(self):
 # bump) with: python -m lightsim2grid.tests.test_binary_serialization regen
 FIXTURE_PATH = os.path.join(os.path.dirname(os.path.abspath(__file__)),
                             "binary_format_fixture",
-                            "case14_sandbox_format3.lsb")
+                            "case14_sandbox_format4.lsb")
 FIXTURE_ENV_NAME = "l2rpn_case14_sandbox"
 FIXTURE_N_SUB = 14
 FIXTURE_N_LOAD = 11