Marekhorst 1614 integrate ec erasmus plus projects mining#1617
Marekhorst 1614 integrate ec erasmus plus projects mining#1617marekhorst wants to merge 4 commits into
Conversation
Integrating mining update and supplementing the integration test suite with the ERASMUS+ matching case.
There was a problem hiding this comment.
Pull request overview
This PR extends the project reference-extraction workflow to recognize and link EC ERASMUS+ grant IDs (e.g., 2021-1-TR01-KA220-VET-000028090) from document text, and updates the end-to-end sample test data accordingly.
Changes:
- Add ERASMUS+ grant-ID matching logic to the SQLite extraction query (
projects.sql). - Adjust grant “normalized acronym” generation for ERASMUS+ projects in the project DB build script (
buildprojectdb.sql). - Update the sample workflow’s test inputs/expected outputs and report counters to include the new ERASMUS+ example.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/project/data/report_funder.json | Updates expected per-funder and total counters for the new extracted reference. |
| iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/project/data/project.json | Adds an ERASMUS+ project record used by the sample workflow test. |
| iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/project/data/document_to_project.json | Adds expected document→project linkage for the new ERASMUS+ grant mention and normalizes one snippet. |
| iis-wf/iis-wf-referenceextraction/src/test/resources/eu/dnetlib/iis/wf/referenceextraction/project/data/document_text.json | Adds a document text containing an ERASMUS+ grant mention used for extraction. |
| iis-wf/iis-wf-referenceextraction/src/main/resources/eu/dnetlib/iis/wf/referenceextraction/project/sqlite_builder/oozie_app/lib/scripts/buildprojectdb.sql | Updates normalized identifier logic for ERASMUS+ projects during grants DB build. |
| iis-wf/iis-wf-referenceextraction/src/main/resources/eu/dnetlib/iis/wf/referenceextraction/project/main_sqlite/oozie_app/lib/scripts/projects.sql | Adds a new UNION branch to extract ERASMUS+ grant IDs from document text and join them to grants. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| select regexprfindall("(\d{4}-? ?\d-? ?[A-Z]{2}\d{2}-? ?[A-Z]{2}\d{3}-? ?[A-Z]{3}-? ?[\d]{9})", prev||middle) as regexpr_prev_middle, docid, middle, prev, next | ||
| from ( setschema 'docid,prev,middle,next' select c1, textwindow2s(regexpr("\n",c2," "), 10, 1, 10, "\d{9}") from pubs ) | ||
| where regexpr_prev_middle is not null | ||
| ) |
There was a problem hiding this comment.
I have run integration tests, the ERASMUS+ case was covered and no runtime error was spotted. In theory it is possible some block of code might not get executed but I doubt this is the case here.
@johnfouf do you think this Copilot comment is relevant or should we discard it because Madis supports this kind of clauses?
There was a problem hiding this comment.
Copilot is wrong, this works in SQLite - although it is undocumented, nonstandard and SQLite author regrets it: https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg94070.html
May also be bad for performance. If adding yet another query nesting level would be better or even worse would have to be tested on real data.
mpol
left a comment
mpol
left a comment
There was a problem hiding this comment.
Looks ok, possibly apart from the nonstandard use of alias in WHERE.
| select regexprfindall("(\d{4}-? ?\d-? ?[A-Z]{2}\d{2}-? ?[A-Z]{2}\d{3}-? ?[A-Z]{3}-? ?[\d]{9})", prev||middle) as regexpr_prev_middle, docid, middle, prev, next | ||
| from ( setschema 'docid,prev,middle,next' select c1, textwindow2s(regexpr("\n",c2," "), 10, 1, 10, "\d{9}") from pubs ) | ||
| where regexpr_prev_middle is not null | ||
| ) |
There was a problem hiding this comment.
Copilot is wrong, this works in SQLite - although it is undocumented, nonstandard and SQLite author regrets it: https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg94070.html
May also be bad for performance. If adding yet another query nesting level would be better or even worse would have to be tested on real data.
|
Merged with the master branch with this rebased and squashed commit: 23f92f3. |
No description provided.