Fix: [Redshift] interval and numeric column type parsing#27983
Open
mohittilala wants to merge 2 commits intomainfrom
Open
Fix: [Redshift] interval and numeric column type parsing#27983mohittilala wants to merge 2 commits intomainfrom
mohittilala wants to merge 2 commits intomainfrom
Conversation
mohittilala
added
bug
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Redshift ingestion column-type parser to better handle PostgreSQL-compatible type modifiers coming from federated schemas, specifically for interval and numeric types, and adds unit tests around these parsing paths.
Changes:
- Adjust
numericargument parsing to support precision-only forms likenumeric(10)without crashing. - Ensure
intervalprecision/fields are routed via keyword arguments (not positional args) to avoid constructor-argument collisions. - Add unit tests covering
interval(...)andnumeric(...)parsing and SQLAlchemy type construction.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
ingestion/src/metadata/ingestion/source/database/redshift/utils.py |
Updates _get_args_and_kwargs to handle numeric precision-only and to clear positional args for interval types. |
ingestion/tests/unit/topology/database/test_redshift_utils.py |
Adds end-to-end unit tests for _get_args_and_kwargs + _update_coltype for interval and numeric. |
Contributor
|
The Python checkstyle failed. Please run You can install the pre-commit hooks with |
mohittilala
changed the title
Code Review ✅ Approved 1 resolved / 1 findingsRedshift ingestion now correctly handles interval and numeric column types by normalizing argument parsing and splitting logic. Extensive test coverage confirms support for various precision and scale configurations. ✅ 1 resolved✅ Quality: New tests use unittest.TestCase instead of plain pytest+assert
OptionsDisplay: compact → Showing less information. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | Gitar |
|
Contributor
🔴 Playwright Results — 8 failure(s), 21 flaky✅ 3999 passed · ❌ 8 failed · 🟡 21 flaky · ⏭️ 86 skipped
Genuine Failures (failed on all attempts)❌
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes:
Follow-up to #27524 (which Fixes #24106). Two more parsing failures in the same
_get_args_and_kwargsfunction iningestion/src/metadata/ingestion/source/database/redshift/utils.py, both surfaced when ingesting federated PostgreSQL schemas through Redshift:interval(N)columns raisedINTERVAL.__init__() got multiple values for argument 'precision'._init_argswas extracting the precision into positional args whilekwargs["precision"]was also being set on the same call. Fix: resetargs = ()in the interval branch so precision and fields flow through kwargs only. Same pattern as Fix Redshift timestamp precision parsing #27524 applies totime/timestamp.numeric(N)columns (precision-only, no scale) crashed withValueError: not enough values to unpack (expected 2, got 1)becauseprec, scale = charlen.split(",")assumed a comma-separated charlen. Fix: parsecharlen.split(",")into a tuple of one or two ints, resetargs = ()when no charlen is reported.Both fixes mirror upstream SQLAlchemy's PostgreSQL dialect.
Tests in
test_redshift_utils.pyexercise the real_get_args_and_kwargs+_update_coltypepath end-to-end (no over-mocking) forinterval(6),interval day to second(6), bareinterval,numeric(10),numeric(10,2)regression, and barenumeric.Type of change:
Checklist:
Fixes <issue-number>: <short explanation>Bug fix