Multi-file batch analysis (+ Parquet, upload limit, Globus file/directory)#142
Draft
jeanbez wants to merge 51 commits into
Draft
Multi-file batch analysis (+ Parquet, upload limit, Globus file/directory)#142jeanbez wants to merge 51 commits into
jeanbez wants to merge 51 commits into
Conversation
Spec for analyzing multiple independent files: a session-free core batch primitive (summarize_files) reused by web/CLI/library, plus a web active-file switcher (compatibility shim over the existing single-file session keys) and a combined cross-file summary. Local upload and Globus feed one source-agnostic file list; per-file type inferred from extension; max 50 files. Globus is remote compute, so the file list/UI is unified but metric execution stays dispatched by source (local routes vs remote_metric_runner); the combined summary shows Globus files with metadata only in phase 1.
Fix completeness to match the existing row-wise metric; correct the MAX_CONTENT_LENGTH/load_dataframe branch attribution; enumerate the Globus shim contract (repopulate globus_file_*); add a source==globus bypass for the local os.path.exists stale-session check; store the file list server-side (cookie-size risk); specify active-file removal, read_file tri-state handling, full cache-key migration, type-key-as-suffix caveat, 413/max-files ordering, and the AJAX upload rewrite.
Combined summary is the default landing view after a multi-file upload: totals strip + per-file table (File/Type/Source/Records/Features/Size/Status), table+totals only, no charts. No metrics in phase 1 — summarize_files does a single light read for records/features/status (local), Globus rows metadata-only. Drops completeness from the summary (and the row-wise/cell-wise concern with it).
Table sorted by file name (case-insensitive). Globus rows now fetch records/features asynchronously via the existing remote _summary_statistics (reusing the globus_summary cache) instead of showing n/a; total records updates as remote counts stream in.
13 TDD tasks across 6 milestones: core batch layer (summarize_files, infer_file_type), server-side file list + active-file shim, cache re-key, file-management endpoints, multi-file upload, limits, Globus unification, and the frontend switcher + batch-overview landing view.
Previously a file whose type couldn't be inferred (e.g. an extension with no registered reader) was saved with an empty type, then silently wiped by the inspector stale-session check (not file_type) — the user saw nothing happen. Now the upload route rejects unsupported files up front with a 400 + message listing the supported formats, so no empty-type file ever enters the list.
Scales to many files (searchable+scrollable, virtualize later); keeps the sidebar focused on metric nav; reuses GET /files + activate.
…totals cards - Add a 'Batch Overview' item to the sidebar nav (shown when >1 file) so the page is reachable after navigating into a metric panel. - Uppercase the Batch Overview page title. - Restyle the totals cards to match the Data Overview stat-card style (p-4, border, rounded-lg, text-3xl) for visual consistency.
The file list lived in in-memory TEMP_RESULTS_CACHE, which is wiped on any app restart. Running a custom metric writes a customDR_*.py into the aidrin package, which triggers the Flask --debug auto-reloader → restart → the list was lost, so switching files afterward returned 'Unknown file'. Persist the list to a per-user JSON on disk (keyed by the cookie-stable user_id, stored outside UPLOAD_FOLDER) so it survives restarts and is shared across worker processes.
The custom-metric .py is written inside the watched aidrin package with open(path, 'w'), which truncates to 0 bytes; the --debug reloader can restart mid-write (or interrupt the load request), leaving an empty file and a blank editor. Write atomically (temp file + os.replace) and fall back to the starter template when the file is missing or empty, so the editor always shows content.
Generated customDR_*.py and remedy_data were written into aidrin/custom_metrics (inside the package the --debug reloader watches), so every save/run/cleanup restarted the dev server — the 'reloads after a while' flicker, and the cause of the earlier lost-file-list and empty-editor bugs. Move CUSTOM_METRICS_FOLDER to data/custom_metrics (project root). base_dr.py/__init__.py stay in the package, so generated metrics still import aidrin.custom_metrics.base_dr; the reloader no longer fires on custom-metric activity.
Changing the file while on the custom-metrics panel reloads with hash #custom-metrics; initWorkspace restores that panel (creating the editor) and batch-mode then shows the overview, hiding it. The /load-custom-metric fetch's setValue() then runs on a hidden CodeMirror, which renders blank. Re-showing the panel skipped init (editor already existed) and never refreshed. Now showPanel refreshes the existing editor so the code renders.
…readable read_file returns None/an error string when the active file is missing (e.g. cleaned up); the metric routes called DataFrame methods on it and crashed (HIPAA: 'str' object has no attribute 'columns'). Guard all read_file sites in metrics.py: on POST return a 400 with a re-upload message, on GET redirect to the inspector (which clears the stale session).
…y on fresh upload Activating a file (top-bar selector or a Batch Overview row) now sets the hash to #data-overview before reloading, so you land on that file's Data Overview. A fresh multi-file upload redirects to #batch-overview. initWorkspace loads the batch overview only when it is the restored landing panel.
…opzone text - clear_file now removes the disk-backed file list before session.clear() (which drops user_id), so it no longer orphans data/filelists/<id>.json. - conftest 'app' fixture isolates UPLOAD_FOLDER/CUSTOM_METRICS_FOLDER (and the derived filelists dir) to tmp_path so tests don't pollute real data/. - Remove the no-op fetchGlobusCount stub; Globus rows render 'n/a' directly. - Dropzone no longer advertises DCAT (no DCAT reader; uploads are rejected).
The execution path already reads via the generic read_file (CSV/JSON/NPZ/HDF5/ Excel -> DataFrame); only the sidebar gated it to CSV. Remove the gate. Fix the remedy output to always be a real .csv (it's written with to_csv; the old name used the reader key as an extension, which is invalid for Excel). Guard the read so a missing file returns a clean error instead of crashing.
…ution Re-review findings: - A URL fragment can't carry the 'land on batch overview' intent: the AJAX upload follows the redirect via fetch, and Response.url strips the fragment, so fresh multi-uploads were landing on Data Overview. Use a one-shot session flag (land_on_batch) consumed on the next inspector render. Add integration tests for multi- vs single-upload landing. - custom.py executed the user metric twice (importlib then runpy); drop the redundant runpy run (and its now-unused import).
… extension The Globus panel previously took a single remote file path + a manual file-type dropdown. Now it accepts a file OR a folder path: a tiny stdlib listing function runs on the endpoint (remote_list_data_files), the server infers each file's type from its extension (decoupled from the endpoint's aidrin version) and adds the supported files to the batch as source:globus entries (unsupported skipped, respecting AIDRIN_MAX_UPLOAD_FILES). Drops the file-type dropdown. The remote metric-execution path (/globus/submit) is unchanged. Mock-tested; needs a real endpoint for live verification.
…new function A live endpoint rejected the separate listing function with a 422 SEMANTICALLY_INVALID / (SystemExit) 73 at submit time. Fold the directory/file listing into the already-registered, proven remote_metric_runner as a '__list_files__' operation so it goes through the same serialization/registration path metrics use. Removes remote_list_data_files/register_list_function; adds a direct test of the list operation.
The function-UUID cache reused a stale registration after remote_metric_runner was edited (e.g. adding __list_files__), so the endpoint ran the old code and reported 'Unknown metric: __list_files__'. Key the cache on a hash of the function source so any change forces a fresh registration once the process has reloaded the new code.
Globus Compute returns 409 RESOURCE_CONFLICT when the endpoint is momentarily busy and asks the caller to retry. Wrap client.run in _run_with_retry with backoff (1s,2s,3s,4s) on 409/RESOURCE_CONFLICT so metric and listing submissions ride out a busy endpoint instead of erroring out.
…_list_files__ Adds a registration log line (func_uuid, src hash, supports __list_files__) so a stale-process situation is diagnosable: if it logs supports=False, the server is running old code and must be fully restarted.
…marize_files Globus batch-overview stats showed as dashes because __summarize_files__ called aidrin.summarize_files on the endpoint, which doesn't exist in the endpoint's released aidrin. Inline the computation with read_file + pandas (present in any aidrin install) so it's version-independent, mirroring summarize_files' output. Verified the remote output matches the local helper exactly.
…ries
Globus per-file stats showed as dashes with no explanation. (1) The 'pending'
guard used 'records not in e', so once an entry was cached as records=None from
a failed fetch it was never retried; use 'records is None' instead. (2) Capture
the failure reason (remote {error} like 'Unknown metric', or an exception) and
return it as overview_error; the UI now shows it as a toast instead of silent
dashes.
…' worker errors The same registered function that listed files succeeded once then failed with 422 'Failed to start ... (SystemExit) 73' minutes later — an intermittent endpoint worker-startup failure, not an invalid request. Treat it as transient and retry with the existing jittered backoff.
…up keys passed!') Running k-anonymity (or l-diversity/t-closeness/entropy) on a Globus file with no quasi-identifiers selected reached pandas' groupby([]) and surfaced the raw 'No group keys passed!'. The local route guards this but the Globus path didn't. Guard all four privacy functions to return a clear 'Please select at least one quasi-identifier.' message, and add a frontend guard so the Globus submit gives instant feedback instead of a slow remote round-trip.
Differential privacy and the single/multiple attribute risk scores have no remote_metric_runner branch, so selecting them on a Globus file silently fell back to k-anonymity (then 'no quasi-identifiers'). Detect those selections in the Globus submit path and show a clear 'not available for remote files yet — use a local file' message instead of misfiring.
# Conflicts: # tests/integration/test_inspector.py # web/__init__.py # web/routes/core.py # web/static/js/main.js # web/templates/_components/upload_panel.html
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| result = status.get("result") | ||
| break | ||
| if status.get("status") == "failed": | ||
| return jsonify({"error": status.get("error", "Remote listing failed")}), 502 |
| if result is None: | ||
| return jsonify({"error": "Timed out listing remote files"}), 504 | ||
| if isinstance(result, dict) and result.get("error"): | ||
| return jsonify({"error": result["error"]}), 400 |
Comment on lines
+436
to
+441
| return jsonify({ | ||
| "added": added, | ||
| "skipped": skipped, | ||
| "active_file_id": first_id, | ||
| "overview_error": overview_error, # why per-file stats are missing | ||
| }) |
# Conflicts: # web/templates/_components/upload_panel.html
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds multi-file batch analysis to AIDRIN, plus supporting features and fixes. Branched from
developand merged up to date with the latestdevelop(29 commits, incl. the Infrastructure rebrand, CLI integration, and the upload-limit/413 work — reconciled with this branch's versions).Highlights
AIDRIN_MAX_UPLOAD_FILES(default 50) files at once; each analyzed independently with an active-file switcher (top-bar selector) and a Batch Overview landing view (records/features/numerical/categorical/size per file).AIDRIN_MAX_UPLOAD_MB(default 1024 = 1 GB), enforced viaMAX_CONTENT_LENGTHwith a clean 413 page + client-side size check.--debugreloads), custom-metric files moved out of the watched package (no reloader thrash), retries on transient Globus 409/SystemExit 73, source-hash function registration, clear error surfacing, purge of cached Globus results on clear/disconnect/remove.No group keys passed!, on every path; frontend guards for instant feedback.Not included here
globus-remaining-metrics).Test Plan
pytest— 410 passed, 8 skipped on the merged treeweb/static/js,web/static/css)create_app()boots with merged config (1 GB limit, 50-file cap, sample-data folder)