Skip to content

Added Legolas with first node/edge schemes. #43

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
visr merged 5 commits into from
Jan 26, 2023
Merged

Added Legolas with first node/edge schemes. #43

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
fcb5cbc
Added Legolas with first node/edge schemes.
evetion Jan 19, 2023
63b8b54
Update src/Ribasim.jl
evetion Jan 19, 2023
7179cf3
Include valdation.jl
evetion Jan 23, 2023
6d9e28d
Working validation.
evetion Jan 24, 2023
d8eb4c0
Addressed review comments.
evetion Jan 26, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Project.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ Dates = "ade2ca70-3891-5945-98fb-dc099432e06a"
Dictionaries = "85a47980-9c8c-11e8-2b9f-f7ca1fa99fb4"
DiffEqCallbacks = "459566f4-90b8-5000-8ac3-15dfb0a30def"
Graphs = "86223c79-3864-5bf0-83f7-82e725a168b6"
Legolas = "741b9549-f6ed-4911-9fbf-4a1c0c97f0cd"
ModelingToolkit = "961ee093-0014-501f-94e3-6117800e7a78"
ModelingToolkitStandardLibrary = "16a59e39-deab-5bd0-87e4-056b12336739"
ModflowInterface = "a348c28d-122a-4d3f-b833-160dba4591a7"
Expand All @@ -21,6 +22,7 @@ PlyIO = "42171d58-473b-503a-8d5f-782019eb09ec"
SciMLBase = "0bca4576-84f4-4d90-8ffe-ffa030f20462"
Serialization = "9e88b42a-f829-5b0c-bbe9-9e923198166b"
TOML = "fa267f1f-6049-4f14-aa54-33bafae1ed76"
Tables = "bd369af6-aec1-5ad0-b16a-f7cc5008161c"

[compat]
Arrow = "2.3"
Expand Down
3 changes: 3 additions & 0 deletions src/Ribasim.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ using Dictionaries
using Graphs
using DataInterpolations: LinearInterpolation
using DiffEqCallbacks
import Legolas
using Legolas: @schema, @version, validate
Comment thread
evetion marked this conversation as resolved.
Outdated
using ModelingToolkit
using ModelingToolkit: getname, renamespace
using ModelingToolkitStandardLibrary.Blocks
Expand All @@ -23,6 +25,7 @@ using Serialization: serialize, deserialize

export interpolator, Register, ForwardFill

include("validation.jl")
Comment thread
evetion marked this conversation as resolved.
include("utils.jl")
include("modflow.jl")
include("lib.jl")
Expand Down
18 changes: 13 additions & 5 deletions src/bmi.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,11 +59,19 @@ end
# Read into memory for now with read, to avoid locking the file, since it mmaps otherwise.
# We could pass Mmap.mmap(path) ourselves and make sure it gets closed, since Arrow.Table
# does not have an io handle to close.
read_table(entry::AbstractString) = Arrow.Table(read(entry))

function read_table(entry)
@assert Tables.istable(entry)
return entry
_read_table(entry::AbstractString) = Arrow.Table(read(entry))
_read_table(entry) = entry

function read_table(entry; schema = nothing)
table = _read_table(entry)
@assert Tables.istable(table)
if !isnothing(schema)
sv = schema()
validate(Tables.schema(table), sv)
R = Legolas.record_type(sv)
foreach(R, Tables.rows(table)) # construct each row
Copy link
Copy Markdown
Member

@visr visr Jan 24, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know roughly how long this takes for the national schematization? Especially the forcing can contain millions of rows, and sometimes only a fraction of them is used for short calculations.

Also, the result of this loop is not kept, that means that we only get errors, but any changes like clamp in the Legolas tour is never done, or not?

Copy link
Copy Markdown
Member Author

@evetion evetion Jan 26, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's correct, at the moment I only check the types and for empty strings. And I rather throw on error than clamp some values, but that's up for debate.

Copy link
Copy Markdown
Member

@visr visr Jan 26, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok yeah right now we don't use something like clamp, and I can't directly think of a use case where it would be better than an error, so let's keep it like this.

Performance was answered here: #43 (comment)

end
return DataFrame(table)
end

"Create an extra column in the forcing which is 0 or the index into the system parameters"
Expand Down
4 changes: 2 additions & 2 deletions src/construction.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
"Load all Arrow input data to SubDataFrames that are filtered for used IDs"
function load_data(config::Dict, starttime::DateTime, endtime::DateTime)
node = DataFrame(read_table(config["node"]))
edge = DataFrame(read_table(config["edge"]))
node = DataFrame(read_table(config["node"]; schema = NodeV1SchemaVersion))
edge = DataFrame(read_table(config["edge"]; schema = EdgeV1SchemaVersion))
state = DataFrame(read_table(config["state"]))
static = DataFrame(read_table(config["static"]))
profile = DataFrame(read_table(config["profile"]))
Expand Down