[Core] Support exogenous transforms inside `MLForecast` and allow raw exogenous inputs to be dropped from the model feature set

[simonez-tuidi]

Description

Today MLForecast has two separate paths for time-based feature engineering:

• lag_transforms applies only to the target column (y).
• transform_exog can generate lag-based features for exogenous columns, but it happens outside the main MLForecast pipeline and returns both the original exogenous columns and the derived features.

That split makes a common workflow unnecessarily awkward:

1. Compute transformed exogenous features with transform_exog.
2. Merge those features back into the training dataframe.
3. Manually decide whether to keep or drop the raw exogenous column.
4. Repeat equivalent preparation for future X_df.

This is especially limiting when an exogenous column is only an input to feature engineering and is not intended to remain as a raw model feature.

Example: predict demand using trailing price statistics, without training on raw price directly.

fcst = MLForecast(
    models=models,
    freq="D",
    lags=[1, 7],
    lag_transforms={
        1: [ExpandingMean(), ExpandingStd()],
        7: [RollingMean(window_size=7, min_samples=1)],
    },
    exogenous_transforms={
        "price": {
            1: [
                RollingMean(window_size=30),
                RollingMax(window_size=30),
            ],
        },
    },
    drop_auxiliary_columns=["price"],
)

In this example:

• y keeps using the existing lag_transforms path.
• price is used to build derived features such as price_rolling_mean_lag1_window_size30.
• raw price is not included in the final training matrix as a direct feature.
• during prediction, the user passes future price in X_df, and MLForecast computes the transformed exogenous features internally.

Why this matters

The current behavior conflates two different concepts:

• exogenous columns that should be model inputs as-is
• exogenous columns that are only source data for derived features

Right now, once a dynamic exogenous column is present in the training dataframe, it is treated as part of the fitted feature set unless the user manually preprocesses and reshapes the data outside MLForecast. That adds boilerplate and makes the internal API less expressive than the lower-level helper.

Proposed API

Recommend adding a new constructor argument instead of overloading lag_transforms:

MLForecast(
    ...,
    exogenous_transforms: Optional[Dict[str, Dict[int, List[LagTransform]]]] = None,
)

Recommended semantics:

• lag_transforms remains target-only and keeps its current behavior.
• exogenous_transforms maps a source exogenous column to the same lag-transform structure already used for the target.
• Reuse drop_auxiliary_columns to control whether raw exogenous source columns are kept in the final model matrix.
• drop_auxiliary_columns=True should keep its current behavior and also auto-drop raw exogenous columns that are used only as sources for exogenous_transforms.
• drop_auxiliary_columns=False should keep all raw exogenous columns in the final model matrix.
• drop_auxiliary_columns=["price"] should drop only the listed raw exogenous columns, while still allowing them to be used as source inputs for exogenous_transforms.

This is preferable to adding a separate keep_original_exogenous_features argument because:

• drop_auxiliary_columns already means “needed to build features, but not intended to be model inputs”
• the current implementation already applies it as the final filter over features_order_
• it avoids adding another partially overlapping feature-retention knob

This is also preferable to adding column="price" directly to transform classes because:

• it preserves the current mental model that lag_transforms refers to the target
• it avoids mixing target and exogenous semantics in the same config block
• it keeps transform naming and validation explicit
• it is closer to how transform_exog already works internally

Expected behavior

Fit / preprocess / cross-validation

• Raw exogenous source columns referenced in exogenous_transforms should be accepted in the input dataframe.
• Derived exogenous features should be computed internally using the same naming convention as transform_exog, e.g. price_rolling_mean_lag1_window_size30.
• The final training matrix should include:
  • all target-derived features
  • all exogenous-derived features
  • only the raw exogenous columns retained after applying drop_auxiliary_columns

Predict

• If the fitted model uses exogenous_transforms, predict(..., X_df=...) should accept future raw source columns and compute the derived exogenous features internally.
• A raw exogenous source column that was dropped from the final training matrix may still be required in X_df if it is needed to build derived features.
• Validation errors should distinguish between:
  • exogenous columns required as direct model inputs
  • exogenous columns required only as sources for exogenous_transforms

Backward compatibility

• Existing workflows should remain unchanged when exogenous_transforms=None.
• transform_exog should keep working as a standalone helper.
• Existing users who rely on raw exogenous columns being part of the feature set should see no behavior change unless they explicitly drop those columns or use drop_auxiliary_columns=True with exogenous source columns that are classified as auxiliary-only.

Acceptance criteria

• MLForecast supports target transforms and exogenous transforms in one pipeline.
• Raw exogenous source columns can be kept or dropped independently from derived exogenous features.
• Feature names for exogenous transforms are stable and consistent with transform_exog.
• predict can compute transformed exogenous features from future raw exogenous inputs.
• X_df validation no longer relies only on the final feature matrix to infer all required exogenous inputs.
• drop_auxiliary_columns works consistently for groupby-only helper columns and exogenous-transform source columns.
• pandas and polars behavior stays aligned.
• direct / recursive forecasting paths and cross-validation cover the new behavior with tests.

Implementation notes

The main internal change is likely that MLForecast needs to track two separate sets of exogenous columns:

• final raw exogenous features used by the model
• raw source columns required to materialize exogenous_transforms

Today, required future exogenous inputs are inferred from features_order_, which is not enough for this feature because a dropped raw source column may still be required to compute derived features at prediction time.

Reusing drop_auxiliary_columns does not remove that requirement. It should only control whether a raw column is passed to the model, not whether it is remembered as an input dependency for prediction-time feature generation.

Alternative considered

An alternative is to allow:

lag_transforms={
    1: [
        ExpandingMean(),
        RollingMean(column="price", window_size=30),
    ],
}

This syntax is attractive, but it couples target-based and exogenous-based transforms into the same config surface and would require every transform type to understand column selection. A separate exogenous_transforms argument seems cleaner for a first implementation.

Use case

Common retail / pricing examples:

• use rolling mean / max of price over the last 30 days to predict y
• use expanding aggregates of inventory without training on raw inventory
• use transformed weather signals while excluding noisy raw weather columns from the final model matrix

This removes the need for an extra transform_exog + merge step and makes the public MLForecast API expressive enough for a frequent forecasting workflow.