Challenge 11 - Interactive map-based dashboard for flood forecasting

[DiegoDomenici]

Stream 1 - Data Visualization

Mentors

• Juan Colonese
• James Varndell
• Dimitar Tasev

Skill Required

Frontend Visualization (Intermediate to Advanced)

• React/TypeScript, Deck.gl/MapLibre GL, Plotly.js, Jupyter ecosystem (ipyleaflet, ipywidgets, Voila)
• HTML/CSS/JavaScript fundamentals
• UX/UI design principles
• Docker containerization

Geospatial Technologies (Intermediate)

• Vector data formats (GeoJSON, Parquet, Shapefiles)
• Web mapping libraries and protocols
• Coordinate systems and projections

Python Development:

• FastAPI backend development
• Xarray/Zarr data manipulation
• GeoPandas or similar for spatial data processing
• Async/await patterns for performance

Goal

Develop a modern, responsive, and highly interactive web-based visualization dashboard that enables users (mostly modellers and analysts) to explore flood forecast data from Zarr datastores efficiently, with intuitive map-based navigation and comprehensive station-level analytics.

Key Outcomes:
• Enable interactive exploration of 10,000+ stations globally, with zoom-level specific visualisation features, like clustering, simpification, overviews and tiling can be used.
• Provide rich contextual visualizations combining forecast data, river network and catchments (vector tile visualisation), and alert thresholds (dynamic styling).
• Support multiple use cases/deployments: operational forecasting, retrospective analysis, and experiments comparisons.
• Modular and reusable visualisation components.
• Be ready for increased scalability and performance, but the user experience is more important in this project.

The challenge is focused on the visualisation and user-facing application but depending how the implementation of the solution progresses some developments may be required in the data provision infra-structure. Therefore, close collaboration and communication with the mentors will be key to achieve the desired outcome.

---

Note: The funding source for this challenge is ECMWF funding. For details on the eligibility, please refer to Article 3 of the Terms and Conditions.

---

Description of the Challenge

Current problem/limitation
The existing Voila-based interface, while functional, has significant UX limitations:

• Slow response times: Kernel restarts and notebook cell execution create delays
• Basic interactions: Limited to simple click events on map markers
• Rigid layout: Difficult to customise views or compare multiple stations
• The current system focuses primarily on point data (stations)
• River networks and catchment boundaries are not visualised
• No spatial context for understanding upstream/downstream relationships
• No animation or time-stepping through forecasts
• Comparisons between multiple forecast runs are only possible for one station at a time.

Data/system
Forecasts and some other initial datasets will be served from an existing REST API connected to a zarr datastore. Alternatively, a serverless bucket-like datastore may be provided. Some geospatial data may be served via public WMS services.

Proposed solution
The proposed solution will be a modern and responsive web-based data visualisation app, accessible only for internal users (authentication or user handling should not be part of the solution), for flood forecasting model results.

Evaluation criteria

• Feasibility
• Easy to maintain / Future-proof approach
• Transferability
• Comprehensibility

[geacomputing]

Hi there,

thank you for sharing this opportunity: it looks very interesting, and I am seriously considering applying.
I’ve put together a list of technical questions below to better understand the scope and requirements.
Thanks in advance for your support.

Technical Questions for ECMWF Mentors — Challenge 11 (Flood Dashboard)

1. Preferred framework or platform

Is there a preferred or recommended framework stack for the project?
I am considering using Django + GeoDjango for the backend because this simplifies geospatial handling and offers built-in support for modular endpoints, multilingual interfaces, and role-based access (even if auth is out of scope).
Is Django acceptable, or is ECMWF expecting FastAPI to be used exclusively?

---

2. Backend resources during development

To run the prototype efficiently, the project will require compute and storage resources (database space, geospatial layers, Zarr access, etc.).
Will ECMWF provide:

• a development API environment,
• a server/VM,
• or backend storage resources?

Alternatively, would it be acceptable to host the prototype during development on a developer sandbox platform such as Heroku, Fly.io, or similar, with the intention of later migrating it to ECMWF infrastructure?
Or should everything be kept strictly local until a later integration phase?

---

3. Asynchronous processing and task orchestration

Given that tasks such as extracting subsets from Zarr, generating tiles, or computing upstream/downstream relationships can be computationally heavy, would ECMWF support the use of an asynchronous task queue (e.g., Celery, RQ, or similar)?
This would allow the application to spin up a temporary “active session” where a worker handles:

• data extraction,
• filtering,
• map layer preparation (e.g., Deck.gl),
• and multi-run comparisons.

Is this approach aligned with expectations, or should the prototype remain synchronous for now?

---

4. Data samples and data model/architecture

Could ECMWF provide example datasets (or documentation) for:

• river network layers,
• catchment boundaries,
• station metadata,
• and any hierarchical encoding (e.g., Pfafstetter indices)?

Clarifications needed:

• Does the river network dataset include Pfafstetter hierarchy or an equivalent connectivity scheme?
• What formats are used (GeoJSON, shapefiles, Parquet, vector tiles, WMS, etc.)?
• What is the approximate size and spatial resolution of these datasets?
• Are these layers delivered through the REST/Zarr API, or via external WMS/WFS services?

This information is essential to design upstream/downstream tracing and station contextualization.

---

5. Expected “on-the-fly analytics”

Could you clarify what types of real-time analytics the dashboard is expected to support?
For example:

• forecast summary statistics (RMSE, bias, exceedance probability),
• comparison between model runs,
• threshold exceedance detection,
• animated time-step exploration,
• hydrological indicators (lag, rise time, cumulative flow).

A minimal required list would help define both the API endpoints and the client-side rendering strategy.

Thanks again for any input you might be willing to give.
Kind regards from sunny Cyprus
Marco

[colonesej]

Dear @geacomputing, thank you for your interest in this challenge. I'll try to address your questions below, but this challenge has an exploratory feel, so feel free to include your ideas of how to better tackle the main goal.

1. We are not attached to a particular framework, but we are looking for a lightweight application and a focus in the user-facing side of it. Of course, the data handling and the processing of some data can affect the overall user experience. So, some development may be required in the data provision, backend and APIs, even though these should not be the focus. We will keep that in mind when assessing the proposals and outcomes.
2. If needed, we will provide access to existing data or infra-structure to test and deploy your solution. This will probably be self-hosted in the European Weather Cloud. Your requirements should be part of your proposal.
3. You are right that some data processing may be computationally heavy, but we do not expect heavy loads in the application, so performance is not a priority at the moment. With that said, any component that you feel is important for your proposal, if based in open-source software, can be provided.
4. This is an example of the river network and catchments dataset that we will want to integrate https://zenodo.org/records/7629908. There are available WMS services for it, but a local copy may be needed for a better experience and customisation in the application. Station metadata are simple tabular data; could be served as GeoJSON or parquet or any other fitting format. Although it can be derived, I'm not sure about topological information at the station level. There are other ways to compute what needs to be computed without it (mainly catchment area boundaries). You can assume this will be made available to you and no heavy river network-based computation should be expected in this application.
5. You can expect them to be dataset that you can retrieve somehow. This project should focus in how to visualise and interact with such dataset. For instance, when we say "model comparison" is about selecting two model "versions" and visualising them side by side, maybe with synched map viewers/windows. "model version" would be a different parameter to an API request that will return some geoJSON of points or a timeseries for a location. Our datasets for this application are simple at the moment, indexed point information mainly.

Hope this clarifies your questions and looking forward to hearing from you again.

[JuanGuerrero09]

Dear @colonesej,

I’m really interested in this challenge, especially because I’ve previously worked with React and Leaflet to build a map-based application for a tourism app in Madrid, and I’ve been using FastAPI for some of my university projects lately.

I have a few questions about the project’s direction. First, will the final outcomes be open-source, or is this intended strictly for internal usage? Regarding the spatial scale, I’m curious if the dashboard is expected to visualize actual inundation extents within the catchments, or if the focus remains primarily on the station points and the river network.

Also, since there is an existing REST API, do you expect us to build directly onto that infrastructure, or should we implement a new middleware API from scratch to handle the communication between the dashboard and the Zarr datastore? Finally, given the requirement for 10,000+ stations, what is the preferred approach for the global view—like vector tiling or clustering—to ensure the page doesn’t crash from the rendering load but is still fast enough to handle all the data requests?

[geacomputing]

Hi @colonesej,

Thank you very much for your valuable input.

I’ve tried to digest the information you so kindly shared, and:

but this challenge has an exploratory feel, so feel free to include your ideas of how to better tackle the main goal.

With that in mind, here is some additional material for brainstorming.

The Data

My understanding is still a bit fuzzy, so please excuse any redundant or basic questions. As I see it, there are two primary datasets that need to be used together:

1. river networks
2. river discharges

Some considerations and questions:

• river network (vector database). This is static, persistent, and has no time dimension. It is downloaded/implemented once and then remains immutable. Is that correct?

• actual discharges (zarr, model-generated). This dataset is time-varying, and this temporal dimension needs to be reflected in the dashboard (e.g., via a time slider), once the user selects a river segment. Given the Zarr format, the implementation should support smooth and responsive interaction.

• important: could you share the link to the Zarr discharge data? I would like to inspect it directly, as it plays a a crucial functional role.

Is this correct?

---

Set-up

Could you confirm whether the following setup makes sense and whether I am on the right track?

1. I downloaded, explored, and visualized the river network on a map using tiling and dynamic refinement for performance. The original files (≈30 GB) were ingested into a Postgres geodatabase. The system performs smoothly: I can click-select individual river segments and, thanks to the relational structure, trace upstream to main channels and delineate catchment areas. Can this be considered a solid foundation for building the web-based dashboard?

Here is a snapshot from an area in Cyprus:

2. Conceptually, once a river segment is selected:

• a dashboard should activate and populate with data corresponding to that selected segment

• one panel could display discharge along that segment, including a time slider

• all available model runs (dimension “model”) should be shown in the dashboard

• IMPORTANT QUESTION: should the visualization focus only on the selected segment, or should it display the entire main river? In the latter case, a crosshair or marker could link the graph position to the geographic location along the river.
  Could you share your thoughts on this design choice? This point is still relatively unclear to me. What is expected to be included/visualized?

Web-page Functionalities

• The current setup uses internal API calls to fetch tiles, roughly of the form:

/api/tiles2/lines/{z}/{x}/{y}

Since this mechanism is already in place, it could be extended to support automated data retrieval as well.

• As shown in the screenshot above, the webpage already supports role-based access and user authentication (“Login”). This could be expanded to provide differentiated access levels, limit monthly data downloads per user type, and integrate API authentication via API keys (as mentioned earlier).

---

Hope this clarifies your questions and looking forward to hearing from you again.

Your previous message was very helpful—thank you again, and I look forward to your feedback.

P.S: The name I came up with for this tool is "Floodviz"

[colonesej]

Hello @JuanGuerrero09, thank you for your interest in this challenge. About your questions:

1. Open-source: Your deliverable will be open-source, meaning the source code will live as a Code4Earth repository in the stage you would eventually deliver it. Access (meaning API access) to the near realtime dataset will remain private. We can think about hosting a subset for the sake of testing and showcasing the final product after the challenge is done. It can be discussed. Please have a look at the old projects repositories hosted in Code4Earth space and, also, bear in mind the project's Terms and conditions, especially Article 5 about Intellectual property.
2. Data to be visualised: At this moment we do not expect to have inundation data as part of the list of datasets we would be showing. For the moment, the focus is on spatially distributed stations, with some styling indicating where floods are forecasted to happen, plus timeseries of discharge forecast. It could include showing forecasts from different models or sources, including different versions of the our AI-based model, other physical-based models, or, even, external forecasts, but this would be addressed in later stages of the project, time-permitting.
3. Communication: The need for a middleware layer will depend on your solution for the front end. For the forecast data, you can assume there is a REST API which you can fetch data from, authentication will be done by a provided token used for the entire application, no user or role management is needed. I'll post an example API call and data forecast data structure in a separate post, so other potential candidates can easily find it as well.
4. Stations view: This should be part of your solution. We have an initial idea that some clustering of points should be done for the lower zoom levels of the map to improve the user experience when first accessing the dashboard in the global view. For this to work some vector tiling may be necessary, or smart use of overviews, or data filtering per zoom level. Those are the technical questions we want the challenge to explore.

[colonesej]

Hello @geacomputing,

no worries about asking questions.

river network (vector database). This is static, persistent, and has no time dimension. It is downloaded/implemented once and then remains immutable. Is that correct?

actual discharges (zarr, model-generated). This dataset is time-varying, and this temporal dimension needs to be reflected in the dashboard (e.g., via a time slider), once the user selects a river segment. Given the Zarr format, the implementation should support smooth and responsive interaction.

R: yes, both are correct.

About the example dataset, I'll post an example API call and data structure in a following comment. I'm afraid I can't share an actual dataset just yet. Your proposal will not be evaluated on any misconceptions about the data itself as we know the candidates cannot have the full picture at this stage. On the other hand, please, make your assumptions clear.

About your initial setup

1. Yes, that's correct and a nice example of using the data.

2. One important clarification, that is related to your following point: The forecast dataset and the river network are not spatially consistent, therefore the forecast query will be done against a points vector data (from the stations locations). The river segments are, for now, background information where you can do the metadata query as you showed in your example but will not be directly used for the forecast data queries. We can think about ways to reconcile them, but this requires a deeper knowledge of each dataset itself and potentially complex spatial operations, metadata handling, and many more potential complications. the river network data is "static" as you said and like a background layer. Time-varying Interactions will all be against points in the map.

3. your suggested frontend structure sounds nice, please add your thoughts and mockups in your proposal. It's nice that you are showing the possibilities about user roles and authentication, but again, this can be an extra feature and not necessarily the focus of the application in the context of this particular challenge. Depending on progress, and by our current view of the challenge, we would focus on integrating other forecast sources (as mentioned in a previous comment) other than creating user management features. Time looks plenty but it runs fast in Code4Earth, so we need to focus on a few set of features.

[colonesej]

How the current global map view looks like

Example forecast dataset and the foundational data API calls:

forecast for a particular stations, date and model version

data/forecasts?gauge_id=lamahice_99&reference_date="2026-01-27H00:00:00&expver=user:real-weir"

{
    "forecast": {
        "valid_time": ["2026-01-28T00:00:00", "2026-01-29T00:00:00", "2026-01-30T00:00:00", "2026-01-31T00:00:00", "2026-02-01T00:00:00", "2026-02-02T00:00:00", "2026-02-03T00:00:00", "2026-02-04T00:00:00", "2026-02-05T00:00:00", "2026-02-06T00:00:00"],
        "data": [{
            "gauge_id": "lamahice_99",
            "values": [1.0339882374, 0.9950551391, 0.9694488049, 0.9279735088, 0.9592425227, 0.9829563498, 0.9601522088, 1.01224792, 1.0439479351, 1.022352457]
        }]
    }
}

returns all stations

data/stations

{
    "features": {
        "type": "FeatureCollection",
        "features": [{
            "id": "24253",
            "type": "Feature",
            "properties": {
                "gauge_id": "lamahice_93",
                "gauge_lat": 65.3297030628193,
                (...)
            },
            "geometry": {
                "type": "Point",
                "coordinates": [-19.0852542844465, 65.3297030628193]
            }
        }, {
            "id": "24258",
            "type": "Feature",
            "properties": {
                "gauge_id": "lamahice_99",
               (...)
            },
            "geometry": {
                "type": "Point",
                "coordinates": [-18.8867554090353, 64.473784342571]
            }
        }]
    }
}

static features of a particular station

data/stations?gauge_id=lamahice_99

{
    "features": {
        "type": "FeatureCollection",
        "features": [{
            "id": "24258",
            "type": "Feature",
            "properties": {
                "gauge_id": "lamahice_99",
                "gauge_lat": 64.473784342571,
                "gauge_lon": -18.8867554090353,
                "gauge_name": "Thjorsa, Thjorsa Nordlingaalda (before 1980)",
                "country": "Iceland",
                (...)

            },
            "geometry": {
                "type": "Point",
                "coordinates": [-18.8867554090353, 64.473784342571]
            }
        }]
    }
}

[geacomputing]

Dear @colonesej,

Thank you! The information you have shared is extremely helpful. The more details you provide, the clearer the picture becomes, allowing me to define a solution and methodology more confidently.

---

Questions about the API endpoint

I’ve split my questions between stations (topology) and forecast (data associated with that station). I understand that they are decoupled, although their coupling would be relatively straightforward given sufficient knowledge of metadata and spatial information.

Forecast

You mentioned that the request:

data/forecasts?gauge_id=lamahice_99&reference_date="2026-01-27H00:00:00&expver=user:real-weir"

returns:

{ "forecast": { "valid_time": ["2026-01-28T00:00:00", "2026-01-29T00:00:00", "2026-01-30T00:00:00", "2026-01-31T00:00:00", "2026-02-01T00:00:00", "2026-02-02T00:00:00", "2026-02-03T00:00:00", "2026-02-04T00:00:00", "2026-02-05T00:00:00", "2026-02-06T00:00:00"], "data": [{ "gauge_id": "lamahice_99", "values": [1.0339882374, 0.9950551391, 0.9694488049, 0.9279735088, 0.9592425227, 0.9829563498, 0.9601522088, 1.01224792, 1.0439479351, 1.022352457] }] } }

My questions regarding forecasts:

1. Will forecast data always be daily values, or could they also be hourly, depending on the station selected on the map?
2. Are there restrictions on the maximum number of stations that can be selected for visualization in the dashboard? Can multiple stations be requested at once, or is it limited to a single station? A single station would simplify the dashboard and avoid overwhelming the user with too much information ("Analysis Paralysis").
3. I see that &expver=user:real-weir is a query key. Can I also query a specific model, or retrieve a list of models available for a given station?
4. I understand that stations and rivers are not “aligned.” However, is there a way to link a station to its river (e.g., "the user clicked station belongs to this or that river"), potentially with 1-to-many relations? Detailed metadata for spatial analysis would be very useful.
5. Can you provide an estimate of how long it would take to retrieve a full dataset—for example, one station, all available models, and all ancillary metadata? If the payload is similar to what you showed, I expect it to be very fast, but I am asking in case there is much more data to be sent.

In essence: I understand you cannot disclose actual data at this stage, but could you provide the full schema or architecture of the dataset? Or is your previous post the most detailed information available regarding the data structure?

---

Stations

You wrote:

returns all stations

data/stations

{
"features": {
"type": "FeatureCollection",
"features": [{
"id": "24253",
"type": "Feature",
"properties": {
"gauge_id": "lamahice_93",
(...) !!! ------------------ Can you include and show all metadata??
},
"geometry": {
"type": "Point",
"coordinates": [-19.0852542844465, 65.3297030628193]
}
}, {
"id": "24258",
"type": "Feature",
"properties": {
"gauge_id": "lamahice_99",
(...)
},
"geometry": {
"type": "Point",
"coordinates": [-18.8867554090353, 64.473784342571]
}
}]
}
}

Static features of a particular station

data/stations?gauge_id=lamahice_99

{
"features": {
"type": "FeatureCollection",
"features": [{
"id": "24258",
"type": "Feature",
"properties": {
"gauge_id": "lamahice_99",
"gauge_lat": 64.473784342571,
"gauge_lon": -18.8867554090353,
"gauge_name": "Thjorsa, Thjorsa Nordlingaalda (before 1980)",
"country": "Iceland",
(...)
},
"geometry": {
"type": "Point",
"coordinates": [-18.8867554090353, 64.473784342571]
}
}]
}
}

This is already very helpful, thank you! However, as mentioned before, is it possible to share the full validation/schema for stations? I understand the actual data may not be disclosed, but perhaps the structure itself could be shared. Or is the example above sufficient? This is relevant for functional constraints and optimization.

---

Evaluation criteria

You mentioned:

Evaluation criteria

Feasibility
Easy to maintain / Future-proof approach
Transferability
Comprehensibility

Could you clarify a bit more about these? For example:

• Transferability: I am developing locally on Python/Django, using Django Ninja for the API and a local Postgres (geodatabase) for fast queries. While everything is on GitHub, some modules (like river data or Postgres) are not portable. How could I demonstrate my prototype given the large amount of static data?
• Feasibility: My current setup is functional, and I am gradually integrating additional components. By the time I submit the proposal, my aim is to have a modular prototype that can accommodate new features easily.
• Comprehensibility: Should I include documentation or unit/integration tests? Would commented code suffice, or is formal documentation expected?

I plan to participate alone. Is there any disadvantage to not forming a team?

[colonesej]

Hi @geacomputing , please find below the answers to your questions:

1. The time frequency would be a model-level parameter, so for this expver the values are daily, for another one it could be hourly for instance. Although the model development is not there yet, so just don't make huge assumptions about daily data but no need to over complicate things right now.
2. This can be part of your proposal. The endpoint currently accepts querying for a list of stations. There is no right answer for this, it will be about the user experience.
3. You can currently retrieve a list of available models, but not subsetting by station. We don't have this use case at the moment but if we get there it can be easily included.
4. As in 2 it could be part of the proposal. There are metadata associated with a river name, for instance, for each station, but there is no guarantee this will align with the river network one as those come from different sources. We can try, I encourage you to add your ideas in the proposal.
5. The payloads are small and we do not expect huge traffic, as it is intended for internal use. Anyway, for this project we will be dealing with sandboxes and internal testing, so don't worry too much about performance.
6. I don't think it would add much more to the proposal knowing the schema. You won't be judge by that. You can make good assumptions based on what I already shared. it's not that it is a particular problem to share, but I think it's too much detail for this stage and I'd rather see your unbiased ideas.

About the selection criteria:

All your points are valid and you can develop them a bit more in the proposal. Basically the question for you to answer is how do you plan to ensure those points will be fullfiled by your proposal?

There is no preference in terms of being alone or a team. But bear in mind that "feasibility" is a criterium, so overpromising can mark you down.

[geacomputing]

Dear @colonesej

Thanks once again for your detailed answers: they’ve been really helpful.

I am working my way through the data and I think I now have enough information to formulate an idea. I do have one last question. You mentioned:

Goal

Develop a modern, responsive, and highly interactive web-based visualization dashboard that enables users (mostly modellers and analysts) to explore flood forecast data from Zarr datastores efficiently, with intuitive map-based navigation and comprehensive station-level analytics.

Could this Zarr endpoint be disclosed at the present time? Is that possible?

I have reached a point where, on my web-based visualizer, I can efficiently overlay points and spatial information (like catchments), for example using a GPU-powered framework for exploratory analysis of large datasets.

Now, to actually populate the dashboard, I would need more information. Given the extremely rich content available on Copernicus datastores, instead of downloading and processing the data asynchronously, I was hoping to “milk the cow directly” by accessing it via Zarr.

Are these infos available at this stage?

Thanks and kind regards,
Marco