NSCLC Immunotherapy Response Prediction

Machine learning pipeline for classifying responders vs non-responders in NSCLC patients treated with immunotherapy. Uses Random Forest with SHAP (SHapley Additive exPlanations) for model interpretability across five dataset types: clinical, metabolite, cytokine, cell type, and all features combined. Cohort of 33 patients with 70/30 stratified train/test split.

Repository Structure

data/                Raw .xlsx files
  ├── NSCLC_data.xlsx          Clinical, metabolite, and cytokine data
  └── cellular_profile.xlsx    Cellular profiling data
int/                 Intermediate .tsv files
  ├── *_data.tsv / *.tsv       Manually exported from .xlsx (see note below)
  └── concise_*.tsv            Cleaned versions produced by clean_data.py
src/                 Analysis scripts (Python + R)
results/             Outputs organized by dataset
  ├── all/           All features combined
  ├── cells/         Cell type features
  ├── clinical/      Clinical features
  ├── cytokines/     Cytokine features
  └── metabolites/   Metabolite features
      Each contains: RF importances, SHAP plots, and a combinations/ subfolder

Requirements

Python 3.10

• scikit-learn 1.2.0
• shap 0.41.0
• pandas 1.5.3
• numpy 1.24.3
• matplotlib 3.7.1
• seaborn 0.11.2
• openpyxl 3.0.9
• scipy 1.10.1

R

• ComplexHeatmap
• circlize
• ggplot2
• tidyverse
• data.table
• countdata

To set up the Python environment:

conda env create -f environment.yml
conda activate lung_cancer_ml

Pipeline

0. Manual data export

The raw TSV files in int/ (clinical_data.tsv, cytokines_chemokines_pg-ml_t0.tsv, metabolites.tsv, all_cell_types.tsv) were manually exported from the .xlsx files in data/. These are the starting point for the scripts below.

1. Data cleaning — src/clean_data.py

Reads the exported TSVs from int/, cleans and merges them, and writes the cleaned versions (concise_clinical.tsv, concise_cytokines.tsv, concise_metabolites.tsv, concise_cells.tsv). Also produces concise_all.tsv by merging all datasets and filtering to the features listed in src/top_features.txt.

python src/clean_data.py

2. SHAP analysis — src/rf_shap_analysis.py

Trains a Random Forest classifier (hyperparameter tuning via GridSearchCV, 5-fold CV) on a given dataset and generates SHAP explanations (summary, violin, beeswarm, PCA, interaction, and dependency plots). Outputs go to results/<dataset>/.

To run for a different dataset, change name_data in the main() function (e.g., "clinical", "cytokines", "metabolites", "cells", "all").

python src/rf_shap_analysis.py

3. Feature combinations — src/feature_combinations.py

Takes the top 10 features (by RF importance) from a dataset and tests all 1023 possible combinations, evaluating each with accuracy, F1, precision, and recall. Outputs go to results/<dataset>/combinations/.

python src/feature_combinations.py

4. Visualization — src/plots.R

Generates a ComplexHeatmap of the top 100 feature combinations by accuracy. Uses helper functions from src/plots_source.R.

Rscript src/plots.R

Citation

Manuscript in preparation.