Handling NetCDF enums as julia enums

docs/src/dataset.md

@@ -31,7 +31,7 @@ Otherwise, we attempt to use standard structures from the Julia standard library
 A NetCDF group is a dataset (with variables, attributes, dimensions and sub-groups) and
 can be arbitrarily nested.
 A group is created with `defGroup` and accessed via the `group` property of
-a `NCDataset`.
+a `NCDataset`. Groups can be nested in arbitrary depth.
 
 ```julia
 # create the variable "temperature" inside the group "forecast"

docs/src/experimental_features.md

@@ -13,10 +13,10 @@ NCDatasets.filter
 ## Experimental MPI support
 
 Experimental MPI support is available as a package extension. It is important to load `MPI` in addition to `NCDatasets` to enable this package extension.
-All metadata operators (creating dimensions, variables, attributes, groups or types) must be done *collectively*.
-Reading and writing data of netCDF variables can be done *independently* (default) or *collectively*. If a variable (or whole dataset) is marked for *collectively* data access, the underlying HDF5 library can enable additional optimization.
+All metadata operations (creating dimensions, variables, attributes, groups or types) must be done *collectively*.
+Reading and writing data of netCDF variables can be done *independently* (default) or *collectively*. If a variable (or whole dataset) is marked for *collective* data access, the underlying HDF5 library can enable additional optimization.
 More information is available in the [NetCDF documentation](https://web.archive.org/web/20240414204638/https://docs.unidata.ucar.edu/netcdf-c/current/parallel_io.html).
-For the MPI IO standard, collective IO means that all MPI processes execute all the same OI functions (calling for example [MPI\_File\_write\_at\_all](https://www.mpich.org/static/docs/v4.1/www3/MPI_File_write_at_all.html)). If this is not the case, then the access is independently (calling for example [MPI\_File\_write\_at](https://www.mpich.org/static/docs/v4.1/www3/MPI_File_write_at.html)).
+For the MPI IO standard, collective IO means that all MPI processes execute all the same IO functions (calling for example [MPI\_File\_write\_at\_all](https://www.mpich.org/static/docs/v4.1/www3/MPI_File_write_at_all.html)). If this is not the case, then the access is independent (calling for example [MPI\_File\_write\_at](https://www.mpich.org/static/docs/v4.1/www3/MPI_File_write_at.html)).
 
 Only the NetCDF 4 format can be currently used for parallel access. On Windows, the MPI interface is [currently unsupported](https://github.com/JuliaPackaging/Yggdrasil/issues/8523). Help from developers with access to Windows would be appreciated.
 
@@ -60,39 +60,62 @@ NCDataset(comm::MPI.Comm,filename::AbstractString,mode::AbstractString)
 NCDatasets.paraccess
 ```
 
-# NetCDF compound types
+# NetCDF 4 user-defined types
+
+NCDatasets.jl supports variable-length arrays, compound types (`struct` in julia) and enums.
+Variable-length arrays can be composed of primitive types, compound types or enums.
+Opaque variables are currently not supported.
+
+## NetCDF compound types
 
 NetCDF 4 allows the users to define their own type, in particular, compound types which correspond to Julia structures.
+Compound types can be composed of primitive types, other compound types or enums or vectors of fixed sizes of these types.
 An array of such structures can be written to and loaded from a NetCDF file. For example:
 
 ```julia
 fname = download("https://raw.githubusercontent.com/Unidata/netcdf-c/refs/tags/v4.8.1/dap4_test/nctestfiles/test_struct_array.nc")
 ds = NCDataset(fname)
 
 array = ds["s"][:,:]
-typeof(array)
+
+propertynames(array[1,1])
 # output
-# Matrix{c_t} (alias for Array{NCDatasets.ReconstructedTypes_123.c_t, 2})
+#
+# (:x, :y)
+
+
+# access individual elements
+array[1,1].x
+# output
+#
+# 1
+
+
+# access all fields x
+getproperty.(array,:x)
+
 
 struct MyCompoundType
     x::Int32
     y::Int32
 end
 
-NCDatasets.usertype!(ds,"c_t",MyCompoundType)
+ds = NCDataset(fname, typemap = Dict("c_t" => MyCompoundType))
 array = ds["s"][:,:]
 typeof(array)
 # output
+#
 # Matrix{MyCompoundType} (alias for Array{MyCompoundType, 2})
 
 ```
 
-It is preferable in fact that the user defines the compound type as a julia struct and register it using the `NCDatasets.usertype!`.
+It is preferable in fact that the user defines the compound type as a julia struct and register it using the `typemap` argument.
 Users should not rely on the type name generated internally by `NCDatasets`. Note also that Julia treats two types as different even if they have
 the same memory layout.
 When defining these structures, avoid using the type `Int` as its size is platform-dependent. Vectors of fixed length can also be used in struct fields.
 They should be declared as `NTuple`s (see
 [Calling C and Fortran Code](https://docs.julialang.org/en/v1/manual/calling-c-and-fortran-code/#Struct-Type-Correspondences) for the manual).
+The corresponding julia type definitions of NetCDF compound types and enums can be automatically generated using the `ncgen` function.
 
 Here is an example to write such a dataset:
 
@@ -115,7 +138,8 @@ NCDataset(fname,"c") do ds
 end
 ```
 
-An important restriction is that the `struct` must be immutable and contain only immutable fields. The memory layout of a `mutable struct` is not compatible with the layout expected by the C library. To update a single field in a struct, the user has to recreate the structure. For example to update the field `x` of the the first element to 10:
+Nested struct/compound types (possibly containing enums) are supported.
+An important restriction is that the `struct` must be immutable and contain only immutable fields. The memory layout of a `mutable struct` is not compatible with the layout expected by the C library. To update a single field in a struct, the user has to recreate the structure. For example to update the field `x` of the first element to 10:
 
 ```julia
 array2[1] = MyCompoundType(10,array2[1].y)
@@ -127,3 +151,44 @@ For large structures, it might be beneficial to use [Accessors](https://github.c
 using Accessors
 @set array2[1].x = 10
 ```
+
+## NetCDF enum type
+
+NetCDF enum types are implemented as Julia enum types. This example shows how to create an enum type and write as an vector of enums to a NetCDF file:
+
+``` julia
+@enum TestEnum::Int8 good=1 bad=2 ugly=3
+
+data = [good, bad, good, ugly]
+fname = tempname()
+NCDataset(fname,"c") do ds
+    # the Julia type name is used by default in the netcdf file
+    # dimension "dim" is created automatically
+    ncv = defVar(ds,"data",data,("dim",))
+end
+
+```
+
+The julia type `TestEnum` must be internally reconstructed unless
+it is provided via the `typemap` parameter (which is preferred).
+Loading the data:
+
+``` julia
+data2 = NCDataset(fname,"r", typemap = Dict("TestEnum" => TestEnum)) do ds
+    ds["data"][:]
+end
+```
+
+The array of enums `data` can be converted to, for example, a `CategoricalArray` of strings using:
+
+```julia
+using CategoricalArrays
+enum_dict = Dict(inst => string(inst) for inst in instances(eltype(data)))
+ca = CategoricalArray([enum_dict[x] for x in data]; levels=collect(values(enum_dict)))
+```
+
+
+```@docs
+NCDatasets.enums
+NCDatasets.typemap!
+```

src/NCDatasets.jl

@@ -64,6 +64,7 @@ include("errorhandling.jl")
 include("lock.jl")
 include("netcdf_c.jl")
 include("compound_types.jl")
+include("enum_types.jl")
 include("dataset.jl")
 include("attributes.jl")
 include("dimensions.jl")

src/attributes.jl

@@ -52,8 +52,11 @@ ds = NCDataset("file.nc")
 title = ds.attrib["title"]
 ```
 """
-attrib(ds::Union{Dataset,Variable},name::SymbolOrString) = nc_get_att(_ncid(ds),_varid(ds),name)
-
+function attrib(ds::Union{Dataset,Variable},name::SymbolOrString)
+    xtype,len = nc_inq_att(_ncid(ds),_varid(ds),name)
+    attribT = _jltype(_ncid(ds),xtype,_dataset(ds).typemap)
+    nc_get_att(_ncid(ds),_varid(ds),name; attribT)
+end
 
 
 """
@@ -81,12 +84,22 @@ ds.attrib["title"] = ["my title"]
 close(ds)
 """
 function defAttrib(ds::Union{Dataset,Variable},name::SymbolOrString,data)
+    if !(data isa Union{String,Enum,NCEnum}) && (ndims(data) > 1)
+        error("Attributes must be a scalar or a vector (while writting attribute '$name' to file $(path(_dataset(ds))).")
+    end
+
     # make sure that the file is in define mode
     defmode(_dataset(ds)) do
-        return nc_put_att(_ncid(ds),_varid(ds),name,data)
+        typeid = nctypeid(_dataset(ds),(data isa AbstractVector ? eltype(data) : typeof(data)))
+        return nc_put_att(_ncid(ds),_varid(ds),name,data; typeid)
     end
 end
 
+function defAttrib(ds::Union{Dataset,Variable},name::SymbolOrString,data::Vector{Any})
+    T = promote_type(typeof.(data)...)
+    @debug "promoted type for attribute $T"
+    defAttrib(ds,name,T.(data))
+end
 
 """
     Base.haskey(a::Attributes,name::SymbolOrString)

src/cfvariable.jl

@@ -109,21 +109,7 @@ function defVar(ds::NCDataset,name::SymbolOrString,vtype::DataType,dimnames;
     defmode(ds) do
         dimids = Cint[nc_inq_dimid(ds.ncid,dimname) for dimname in dimnames[end:-1:1]]
 
-        typeid =
-            if vtype <: Vector
-                # variable-length type
-                typeid = nc_def_vlen(ds.ncid, typename, ncType[eltype(vtype)])
-            elseif haskey(ncType, vtype)
-                ncType[vtype]
-            elseif length(fieldnames(vtype)) > 0
-                @debug "assume type $vtype is a struct "
-                typename = (isnothing(typename) ? string(vtype) : typename)
-                defCompoundType(ds,vtype,typename)
-            else
-                @warn "unsupported type: class=$(class)"
-                Nothing
-            end
-
+        typeid = nctypeid(ds,vtype; typename)
         varid = nc_def_var(ds.ncid,name,typeid,dimids)
 
         if chunksizes !== nothing