Skip to content

Feat ipynb validator - #125

Open
dartanghan wants to merge 4 commits into
arm:mainfrom
dartanghan:feat_ipnb_validator
Open

Feat ipynb validator#125
dartanghan wants to merge 4 commits into
arm:mainfrom
dartanghan:feat_ipnb_validator

Conversation

@dartanghan

Copy link
Copy Markdown
Contributor

Added a feature to scan .ipynb (jupyter notebook) files, able to read SQL and PYTHON cells.
This will be very useful for data engineers that want their data products validated.

  • Added .ipynb in the README.doc
  • Added ipynb in pyproject.toml
  • Created ipynb_plugin.py
  • Added rules in plugins.yaml

Sample code:

%sql
SET password := "secret"
SELECT :password
%python

@dlt.table
def filtered_taxis():
    password="SQLSERVER"
    username="SQLUSER"
    print("username: ", username)
    print("password: ", password)

Sample result:

File: notebook.ipynb
 Identified issue 1: Hardcoded SQL password in notebook
    Snippet: %sql
SET password := "secret"
SELECT :password
    Line number: 62
    CWE: CWE-798
    Severity: High
...

File: dlt_pipeline.ipynb
 Identified issue 1: Hardcoded credentials in code
    Snippet: password="SQLSERVER"
    username="SQLUSER"
    Line number: 64
    CWE: CWE-798
    Severity: High
...

@dartanghan

Copy link
Copy Markdown
Contributor Author

Hi @mpekatsoula @aaronjb !
Should I fix/adjust anything on the code or do you think that notebook scan aren't aligned with the tool's purpose?
I will be glad to fix or adapt this PR :)

Tnx

@mpekatsoula

Copy link
Copy Markdown
Contributor

@dartanghan feel free to resolve conflicts/update the plugin and we're happy to merge

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants