Skip to content

CI: debug-format regression in aml4td/website (idempotency) #449

Description

@github-actions

Debug-format regression detected

  • Target repository: aml4td/website
  • Failure type: idempotency
  • Failure records in this run: 1
  • Workflow run: https://github.com/jolars/panache/actions/runs/30786596687
  • Panache command: panache debug format --checks all --report <FILE>
  • Target repository commit: b7b36445c605af5999123893bafe7e6e58a9569e
  • Panache version used in scan: panache 3.0.2
  • Panache commit used in scan: 906f2260c8688beb4220a24626aa2200d826c799
  • Sample file: chapters/numeric-predictors.qmd
  • Sample log path: logs/20470ee8aa7fab1aef369a2f8169a241a16832353ee8a6433d6a0a65787af5e9.log
  • Sample report path: logs/20470ee8aa7fab1aef369a2f8169a241a16832353ee8a6433d6a0a65787af5e9.report.md
  • Sample idempotency input artifact: logs/20470ee8aa7fab1aef369a2f8169a241a16832353ee8a6433d6a0a65787af5e9.passes/chapters_numeric-predictors.qmd.idempotency.input.txt
  • Sample idempotency pass1 artifact: logs/20470ee8aa7fab1aef369a2f8169a241a16832353ee8a6433d6a0a65787af5e9.passes/chapters_numeric-predictors.qmd.idempotency.once.txt
  • Sample idempotency pass2 artifact: logs/20470ee8aa7fab1aef369a2f8169a241a16832353ee8a6433d6a0a65787af5e9.passes/chapters_numeric-predictors.qmd.idempotency.twice.txt
  • Approx. diff start line: 353

Sample files:

  • chapters/numeric-predictors.qmd

Reproduce locally:

git clone https://github.com/aml4td/website.git
cd website
git checkout b7b36445c605af5999123893bafe7e6e58a9569e
panache debug format --checks all --report "chapters/numeric-predictors.qmd"
Sample report excerpt
# Debug-format regression report

- Checks: `all`
- Files checked: 1
- Failures: 1

## Failures

### 1. `chapters/numeric-predictors.qmd` (idempotency)

- Approx. diff start line: 353

```diff
 ---
 knitr:
   opts_chunk:
     cache.path: "../_cache/transformations/"
 ---
 
 # Transforming Numeric Predictors {#sec-numeric-predictors}
 
 ```{r}
 #| label: transformations-setup
 #| include: false
 
 source("../R/_common.R")
 source("../R/_themes.R")
 source("../R/_themes_ggplot.R")
 source("../R/_themes_gt.R")
 
 # ------------------------------------------------------------------------------
 
 library(tidymodels)
 library(embed)
 library(bestNormalize)
 library(patchwork)
 
 # ------------------------------------------------------------------------------
 # set options
 
 tidymodels_prefer()
 theme_set(thm_lt)
 set_options()
 ```
 
 ```{r}
 #| label: ames-split
 #| include: false
 
 source("../R/setup_ames.R")
 ```
 
 Data that are available for modeling are often collected passively without the
 specific purpose of being used for building a predictive model. As an example,
 the Ames Housing data contains a wealth of information on houses in Ames, Iowa.
 But this available data may not contain the most relevant measurements for
 predicting house price. This may be due to the fact that important predictors
 were not measured. Or, it may be because the predictors we have collected are
 not in the best form to allow models to uncover the relationship between the
 predictors and the response.
 
 As mentioned previously, feature engineering is the process of representing your
 predictor data so that the model has to do the least amount of work to explain
 the outcome effectively. A tool of feature engineering is predictor
 transformations. Some models also need predictors to be transformed to meet the
 model's mathematical requirements (i.e., pre-processing). In this chapter we
 will review transformations for quantitative predictors.
 
 We will begin by describing transformations that are applied to one predictor at
 a time that yield a revised form of the predictor (one in, one out). After
 these, an example of a *group* transformation is described called the *spatial
 sign*. Later, @sec-embeddings will describe different types of many-to-many
 transformations such as principal component analysis (PCA) and multidimensional
 scaling (MDS). Additionally, in @sec-interactions-nonlinear, we will examine
 techniques for expanding a single numeric predictor to many predictors (one in,
 many out).\
 
 Let's begin by understanding some general data characteristics that need to be
 addressed via feature engineering and when transformations should be applied.
 
 ## What are Problematic Characteristics, and When Should Transformations be Applied? {#sec-general-transformation}
 
 Common problematic characteristics that occur across individual predictors are:
 
 - skewed or unusually shaped distributions,
 - sample(s) that have extremely large or small values, and
 - vastly disparate scales.
 
 Some models, like those that are tree-based, are able to tolerate these
 characteristics. However, these characteristics can detrimentally affect most
 other models. Techniques used to address these problems generally involve
 transformation parameters. For example, to place the predictors on the same
 scale, we would subtract the mean of a predictor from a sample and then divide
 by the standard deviation. This is known as standardizing and will be discussed
 in the next section.\
 
 What data should be used to estimate the mean and standard deviation? Recall,
 the training data set was used to estimate model parameters. Similarly, we will
 use the training data to estimate transformation parameters. When the test set
 or any future data set is standardized, the process will use the estimates from
 the training data set. Any model fit that uses these standardized predictors
 would want new samples being predicted to have the same reference distribution.
 
 Suppose that a predictor column had an underlying Gaussian distribution with a
 sample mean estimate of 5.0 and a sample standard deviation of 1.0. Suppose a
 new sample has a predictor value of 3.7. For the training set, this new value
 lands around the 10th percentile and would be standardized to a value of -1.3.
 The new value is relative to the training set distribution. Also note that, in
 this scenario, it would be impossible to standardize using a recomputed standard
 deviation for the new sample (which means we try to divide with a zero standard
 deviation).
 
 Many transformations that involve a single predictor change the data
 distribution. Most predictive models do not place specific parametric
 assumptions on the predictor variables (e.g., require normality), but some
 distributions might facilitate better predictive performance than others.
 
 TODO some based on convention or scientific knowledge. Others like the arc-sin
 (ref The arcsine is asinine: the analysis of proportions in ecology) or logit?
 See [issue #10](https://github.com/aml4td/website/issues/10).

See artifact debug-format-repo-scan-results for full logs.

Metadata

Metadata

Assignees

No one assigned

    Labels

    ciContinuous integrationdebug-formatPanache debug format scan regressionsidempotencyRegression category from debug format scan

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions