Skip to content

Fix bad reads of nested cmpd/vlen types in JNI #6413

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mattjala wants to merge 16 commits into
base: develop
Choose a base branch
from
Open

Fix bad reads of nested cmpd/vlen types in JNI #6413

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
4581812
Fix H5DreadVL failing for pre-allocate cmpd-of-seq dsets
mattjala May 15, 2026
51b5af6
Fix bad vlen of cmpd with null slot read
mattjala May 18, 2026
b9e13b9
Fix bad cmpd of cmpd read in java
mattjala May 18, 2026
a87d1f8
Add exception checks
mattjala May 20, 2026
cde96a0
Update NULL checks in translate_wbuf
mattjala May 20, 2026
a49509f
Correct potentially bad array length check
mattjala May 20, 2026
3bbed2b
Clang format
mattjala May 26, 2026
51d453b
Fix readVL/writeVL crash on malformed buffer
mattjala May 27, 2026
e177c33
Committing clang-format changes
github-actions[bot] May 29, 2026
a2b7679
Add bufSize checks to wbuf/rbuf translation
mattjala Jun 4, 2026
94a77e2
Remove vlen pre-allocation support
mattjala Jun 4, 2026
8c5c5f4
Harden JNI buffer interface
mattjala Jun 4, 2026
17dcc9d
Merge branch 'develop' into java_vlen_read_220
mattjala Jun 16, 2026
a834df9
Merge branch 'develop' into java_vlen_read_220
mattjala Jun 17, 2026
f3778e6
Merge branch 'develop' into java_vlen_read_220
mattjala Jun 18, 2026
16631fd
Merge branch 'develop' into java_vlen_read_220
mattjala Jun 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 25 additions & 4 deletions java/src-jni/hdf/hdf5lib/H5.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,21 @@
* and the parameter <i>data</i> can be any multi-dimensional array of numbers, such as float[][], or
* int[][][], or Double[][].
* <p>
* <b>Buffer data model</b>

@jhendersonHDF jhendersonHDF Jun 23, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also leave a similar header comment on the translate_wbuf() / translate_rbuf() functions so that the data model is documented right where the data is transformed? For those working in the JNI that will probably be easier to find.

* <p>
* Read/write buffers must match the memory datatype. The JNI verifies this at the API boundary and throws
* IllegalArgumentException on a mismatch instead of crashing. The expected Java buffer per datatype class is:
* <ul>
* <li>integer/enum/bitfield, float: a primitive array (byte/short/int/long/float/double[]) or byte[]; it must
* be large enough to hold one element of the memory type per selected point.</li>
* <li>fixed- or variable-length string, reference: a String[] (byte[] for references) with one slot per
* selected point.</li>
* <li>compound, variable-length sequence, array, complex: an Object[] of nested java.util.ArrayLists. Each
* element is an ArrayList; a compound is an ArrayList of its members in order, a VLEN/array/complex is an
* ArrayList of its elements, and scalar leaves are the boxed type (Integer, Double, String, ...). Slots are
* not pre-allocated by the caller on read.</li>
* </ul>
* <p>
* <b>@ref HDF5CONST</b>
* <p>
* The HDF5 API defines a set of constants and enumerated values. Most of these values are available to Java
Expand Down Expand Up @@ -1681,7 +1696,8 @@ public synchronized static int H5Aread_short(long attr_id, long mem_type_id, sho
* @param mem_type_id
* IN: Identifier of the attribute datatype (in memory).
* @param buf
* Buffer of variable-lenght to store data read from the file.
* Object[] (one slot per element) to store the data read; each slot is filled with a nested
* ArrayList structure matching mem_type_id (see "Buffer data model" in the class description).
*
* @return a non-negative value if successful
*
Expand Down Expand Up @@ -2263,7 +2279,8 @@ public synchronized static native int H5Awrite_string(long attr_id, long mem_typ
* @param mem_type_id
* IN: Identifier of the attribute datatype (in memory).
* @param buf
* IN: Buffer of variable-lenght with data to be written to the file.
* IN: Object[] (one slot per element) holding the data to write; each slot must be a nested
* ArrayList structure matching mem_type_id (see "Buffer data model" in the class description).
*
* @return a non-negative value if successful
*
Expand Down Expand Up @@ -3425,7 +3442,9 @@ public synchronized static int H5Dread_short(long dataset_id, long mem_type_id,
* @param xfer_plist_id
* Identifier of a transfer property list for this I/O operation.
* @param buf
* Buffer of variable-lenght to store data read from the file.
* Object[] (one slot per selected point) to store the data read; each slot is filled with a
* nested ArrayList structure matching mem_type_id (see "Buffer data model" in the class
* description). Slots need not be pre-allocated.
*
* @return a non-negative value if successful
*
Expand Down Expand Up @@ -4145,7 +4164,9 @@ public synchronized static native int H5Dwrite_string(long dataset_id, long mem_
* @param xfer_plist_id
* Identifier of a transfer property list for this I/O operation.
* @param buf
* Buffer of variable-length with data to be written to the file.
* Object[] (one slot per selected point) holding the data to write; each slot must be a nested
* ArrayList structure matching mem_type_id (see "Buffer data model" in the class description).
* A structural mismatch raises IllegalArgumentException.
*
* @return a non-negative value if successful
*
Expand Down
122 changes: 115 additions & 7 deletions java/src-jni/jni/h5aImp.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,12 @@ Java_hdf_hdf5lib_H5_H5Aread(JNIEnv *env, jclass clss, jlong attr_id, jlong mem_t
if ((vl_data_class = h5str_detect_vlen(mem_type_id)) < 0)
H5_LIBRARY_ERROR(ENVONLY);

/* For fixed-length data the byte buffer must cover the attribute. */
if (!vl_data_class &&
h5a_validate_raw_buf(env, (hid_t)mem_type_id, (hid_t)attr_id, ENVPTR->GetArrayLength(ENVONLY, buf),
sizeof(jbyte), "H5Aread") < 0)
goto done;

if (vl_data_class) {
/* Get size of data array */
if ((vl_array_len = ENVPTR->GetArrayLength(ENVONLY, buf)) < 0)
Expand Down Expand Up @@ -195,7 +201,8 @@ Java_hdf_hdf5lib_H5_H5Aread(JNIEnv *env, jclass clss, jlong attr_id, jlong mem_t
if ((type_class = H5Tget_class((hid_t)mem_type_id)) < 0)
H5_LIBRARY_ERROR(ENVONLY);

translate_rbuf(env, buf, mem_type_id, type_class, vl_array_len, readBuf);
translate_rbuf(env, buf, mem_type_id, type_class, vl_array_len, readBuf,
(size_t)vl_array_len * typeSize);
}

done:
Expand Down Expand Up @@ -254,6 +261,12 @@ Java_hdf_hdf5lib_H5_H5Awrite(JNIEnv *env, jclass clss, jlong attr_id, jlong mem_
if ((vl_data_class = h5str_detect_vlen(mem_type_id)) < 0)
H5_LIBRARY_ERROR(ENVONLY);

/* For fixed-length data the byte buffer must cover the attribute. */
if (!vl_data_class &&
h5a_validate_raw_buf(env, (hid_t)mem_type_id, (hid_t)attr_id, ENVPTR->GetArrayLength(ENVONLY, buf),
sizeof(jbyte), "H5Awrite") < 0)
goto done;

if (vl_data_class) {
/* Get size of data array */
if ((vl_array_len = ENVPTR->GetArrayLength(ENVONLY, buf)) < 0) {
Expand Down Expand Up @@ -281,7 +294,8 @@ Java_hdf_hdf5lib_H5_H5Awrite(JNIEnv *env, jclass clss, jlong attr_id, jlong mem_
if ((type_class = H5Tget_class((hid_t)mem_type_id)) < 0)
H5_LIBRARY_ERROR(ENVONLY);

translate_wbuf(ENVONLY, buf, mem_type_id, type_class, vl_array_len, writeBuf);
translate_wbuf(ENVONLY, buf, mem_type_id, type_class, vl_array_len, writeBuf,
(size_t)vl_array_len * typeSize);
}

if ((status = H5Awrite((hid_t)attr_id, (hid_t)mem_type_id, writeBuf)) < 0)
Expand Down Expand Up @@ -338,6 +352,12 @@ Java_hdf_hdf5lib_H5_H5Aread_1short(JNIEnv *env, jclass clss, jlong attr_id, jlon
if ((vl_data_class = h5str_detect_vlen(mem_type_id)) < 0)
H5_LIBRARY_ERROR(ENVONLY);

/* Verify the buffer is large enough for the attribute. */
if (!vl_data_class &&
h5a_validate_raw_buf(env, (hid_t)mem_type_id, (hid_t)attr_id, ENVPTR->GetArrayLength(ENVONLY, buf),
sizeof(jshort), "H5Aread_short") < 0)
goto done;

if (vl_data_class) {
/* Get size of data array */
if ((n = ENVPTR->GetArrayLength(ENVONLY, buf)) < 0) {
Expand Down Expand Up @@ -408,6 +428,11 @@ Java_hdf_hdf5lib_H5_H5Awrite_1short(JNIEnv *env, jclass clss, jlong attr_id, jlo
H5_BAD_ARGUMENT_ERROR(ENVONLY, "H5Aread: readBuf length < 0");
}

/* Verify the buffer is large enough for the attribute. */
if (h5a_validate_raw_buf(env, (hid_t)mem_type_id, (hid_t)attr_id, n, sizeof(jshort), "H5Awrite_short") <
0)
goto done;

dims[0] = (hsize_t)n;
if ((sid = H5Screate_simple(1, dims, NULL)) < 0)
H5_LIBRARY_ERROR(ENVONLY);
Expand Down Expand Up @@ -467,6 +492,12 @@ Java_hdf_hdf5lib_H5_H5Aread_1int(JNIEnv *env, jclass clss, jlong attr_id, jlong
if ((vl_data_class = h5str_detect_vlen(mem_type_id)) < 0)
H5_LIBRARY_ERROR(ENVONLY);

/* Verify the buffer is large enough for the attribute. */
if (!vl_data_class &&
h5a_validate_raw_buf(env, (hid_t)mem_type_id, (hid_t)attr_id, ENVPTR->GetArrayLength(ENVONLY, buf),
sizeof(jint), "H5Aread_int") < 0)
goto done;

if (vl_data_class) {
/* Get size of data array */
if ((n = ENVPTR->GetArrayLength(ENVONLY, buf)) < 0) {
Expand Down Expand Up @@ -537,6 +568,10 @@ Java_hdf_hdf5lib_H5_H5Awrite_1int(JNIEnv *env, jclass clss, jlong attr_id, jlong
H5_BAD_ARGUMENT_ERROR(ENVONLY, "H5Aread: readBuf length < 0");
}

/* Verify the buffer is large enough for the attribute. */
if (h5a_validate_raw_buf(env, (hid_t)mem_type_id, (hid_t)attr_id, n, sizeof(jint), "H5Awrite_int") < 0)
goto done;

dims[0] = (hsize_t)n;
if ((sid = H5Screate_simple(1, dims, NULL)) < 0)
H5_LIBRARY_ERROR(ENVONLY);
Expand Down Expand Up @@ -596,6 +631,12 @@ Java_hdf_hdf5lib_H5_H5Aread_1long(JNIEnv *env, jclass clss, jlong attr_id, jlong
if ((vl_data_class = h5str_detect_vlen(mem_type_id)) < 0)
H5_LIBRARY_ERROR(ENVONLY);

/* Verify the buffer is large enough for the attribute. */
if (!vl_data_class &&
h5a_validate_raw_buf(env, (hid_t)mem_type_id, (hid_t)attr_id, ENVPTR->GetArrayLength(ENVONLY, buf),
sizeof(jlong), "H5Aread_long") < 0)
goto done;

if (vl_data_class) {
/* Get size of data array */
if ((n = ENVPTR->GetArrayLength(ENVONLY, buf)) < 0) {
Expand Down Expand Up @@ -666,6 +707,10 @@ Java_hdf_hdf5lib_H5_H5Awrite_1long(JNIEnv *env, jclass clss, jlong attr_id, jlon
H5_BAD_ARGUMENT_ERROR(ENVONLY, "H5Aread: readBuf length < 0");
}

/* Verify the buffer is large enough for the attribute. */
if (h5a_validate_raw_buf(env, (hid_t)mem_type_id, (hid_t)attr_id, n, sizeof(jlong), "H5Awrite_long") < 0)
goto done;

dims[0] = (hsize_t)n;
if ((sid = H5Screate_simple(1, dims, NULL)) < 0)
H5_LIBRARY_ERROR(ENVONLY);
Expand Down Expand Up @@ -725,6 +770,12 @@ Java_hdf_hdf5lib_H5_H5Aread_1float(JNIEnv *env, jclass clss, jlong attr_id, jlon
if ((vl_data_class = h5str_detect_vlen(mem_type_id)) < 0)
H5_LIBRARY_ERROR(ENVONLY);

/* Verify the buffer is large enough for the attribute. */
if (!vl_data_class &&
h5a_validate_raw_buf(env, (hid_t)mem_type_id, (hid_t)attr_id, ENVPTR->GetArrayLength(ENVONLY, buf),
sizeof(jfloat), "H5Aread_float") < 0)
goto done;

if (vl_data_class) {
/* Get size of data array */
if ((n = ENVPTR->GetArrayLength(ENVONLY, buf)) < 0) {
Expand Down Expand Up @@ -795,6 +846,11 @@ Java_hdf_hdf5lib_H5_H5Awrite_1float(JNIEnv *env, jclass clss, jlong attr_id, jlo
H5_BAD_ARGUMENT_ERROR(ENVONLY, "H5Aread: readBuf length < 0");
}

/* Verify the buffer is large enough for the attribute. */
if (h5a_validate_raw_buf(env, (hid_t)mem_type_id, (hid_t)attr_id, n, sizeof(jfloat), "H5Awrite_float") <
0)
goto done;

dims[0] = (hsize_t)n;
if ((sid = H5Screate_simple(1, dims, NULL)) < 0)
H5_LIBRARY_ERROR(ENVONLY);
Expand Down Expand Up @@ -854,6 +910,12 @@ Java_hdf_hdf5lib_H5_H5Aread_1double(JNIEnv *env, jclass clss, jlong attr_id, jlo
if ((vl_data_class = h5str_detect_vlen(mem_type_id)) < 0)
H5_LIBRARY_ERROR(ENVONLY);

/* Verify the buffer is large enough for the attribute. */
if (!vl_data_class &&
h5a_validate_raw_buf(env, (hid_t)mem_type_id, (hid_t)attr_id, ENVPTR->GetArrayLength(ENVONLY, buf),
sizeof(jdouble), "H5Aread_double") < 0)
goto done;

if (vl_data_class) {
/* Get size of data array */
if ((n = ENVPTR->GetArrayLength(ENVONLY, buf)) < 0) {
Expand Down Expand Up @@ -923,6 +985,11 @@ Java_hdf_hdf5lib_H5_H5Awrite_1double(JNIEnv *env, jclass clss, jlong attr_id, jl
H5_BAD_ARGUMENT_ERROR(ENVONLY, "H5Awrite_double: buf length < 0");
}

/* Verify the buffer is large enough for the attribute. */
if (h5a_validate_raw_buf(env, (hid_t)mem_type_id, (hid_t)attr_id, n, sizeof(jdouble), "H5Awrite_double") <
0)
goto done;

dims[0] = (hsize_t)n;
if ((sid = H5Screate_simple(1, dims, NULL)) < 0)
H5_LIBRARY_ERROR(ENVONLY);
Expand Down Expand Up @@ -984,6 +1051,10 @@ Java_hdf_hdf5lib_H5_H5Aread_1string(JNIEnv *env, jclass clss, jlong attr_id, jlo
H5_BAD_ARGUMENT_ERROR(ENVONLY, "H5Aread_string: read buffer length <= 0");
}

/* The buffer must have one slot per element in the attribute. */
if (h5a_validate_slot_buf(env, (hid_t)attr_id, n, "H5Aread_string") < 0)
goto done;

if (!(str_len = H5Tget_size((hid_t)mem_type_id)))
H5_LIBRARY_ERROR(ENVONLY);

Expand Down Expand Up @@ -1049,6 +1120,10 @@ Java_hdf_hdf5lib_H5_H5Awrite_1string(JNIEnv *env, jclass clss, jlong attr_id, jl
H5_BAD_ARGUMENT_ERROR(ENVONLY, "H5Awrite_string: write buffer length <= 0");
}

/* The buffer must have one slot per element in the attribute. */
if (h5a_validate_slot_buf(env, (hid_t)attr_id, n, "H5Awrite_string") < 0)
goto done;

if (!(str_len = H5Tget_size((hid_t)mem_type_id)))
H5_LIBRARY_ERROR(ENVONLY);

Expand Down Expand Up @@ -1107,6 +1182,7 @@ Java_hdf_hdf5lib_H5_H5AreadVL(JNIEnv *env, jclass clss, jlong attr_id, jlong mem
size_t typeSize;
H5T_class_t type_class;
jsize vl_array_len = 0;
hssize_t npoints;
htri_t vl_data_class;
herr_t status = FAIL;
htri_t is_variable = 0;
Expand All @@ -1124,6 +1200,12 @@ Java_hdf_hdf5lib_H5_H5AreadVL(JNIEnv *env, jclass clss, jlong attr_id, jlong mem
if ((is_variable = H5Tis_variable_str(mem_type_id)) < 0)
H5_LIBRARY_ERROR(ENVONLY);

/* The buffer must hold at least one slot per point in the attribute. */
if ((npoints = h5a_io_npoints(env, (hid_t)attr_id)) < 0)
goto done;
if ((hssize_t)vl_array_len < npoints)
H5_BAD_ARGUMENT_ERROR(ENVONLY, "H5AreadVL: read buffer is smaller than the attribute");

if (!(typeSize = H5Tget_size(mem_type_id)))
H5_LIBRARY_ERROR(ENVONLY);

Expand All @@ -1135,7 +1217,7 @@ Java_hdf_hdf5lib_H5_H5AreadVL(JNIEnv *env, jclass clss, jlong attr_id, jlong mem
if ((type_class = H5Tget_class((hid_t)mem_type_id)) < 0)
H5_LIBRARY_ERROR(ENVONLY);

translate_rbuf(env, buf, mem_type_id, type_class, vl_array_len, readBuf);
translate_rbuf(env, buf, mem_type_id, type_class, vl_array_len, readBuf, (size_t)vl_array_len * typeSize);

done:
if (readBuf) {
Expand Down Expand Up @@ -1173,6 +1255,7 @@ Java_hdf_hdf5lib_H5_H5AwriteVL(JNIEnv *env, jclass clss, jlong attr_id, jlong me
size_t typeSize;
H5T_class_t type_class;
jsize vl_array_len = 0;
hssize_t npoints;
htri_t vl_data_class;
herr_t status = FAIL;
htri_t is_variable = 0;
Expand All @@ -1193,16 +1276,27 @@ Java_hdf_hdf5lib_H5_H5AwriteVL(JNIEnv *env, jclass clss, jlong attr_id, jlong me
if ((is_variable = H5Tis_variable_str(mem_type_id)) < 0)
H5_LIBRARY_ERROR(ENVONLY);

/* Verify the buffer holds at least one element per point in the attribute. */
if ((npoints = h5a_io_npoints(env, (hid_t)attr_id)) < 0)
goto done;
if ((hssize_t)vl_array_len < npoints)
H5_BAD_ARGUMENT_ERROR(ENVONLY, "H5AwriteVL: write buffer is smaller than the attribute");

if (!(typeSize = H5Tget_size(mem_type_id)))
H5_LIBRARY_ERROR(ENVONLY);

if (NULL == (writeBuf = calloc((size_t)vl_array_len, typeSize)))
H5_OUT_OF_MEMORY_ERROR(ENVONLY, "H5Awrite: failed to allocate raw VL write buffer");

if ((type_class = H5Tget_class((hid_t)mem_type_id)) < 0)
H5_LIBRARY_ERROR(ENVONLY);

translate_wbuf(ENVONLY, buf, mem_type_id, type_class, vl_array_len, writeBuf);
/* Verify the buffer structure matches mem_type_id before converting it. */
if (h5validate_wbuf(ENVONLY, buf, mem_type_id, type_class, vl_array_len) < 0)
goto done;

if (NULL == (writeBuf = calloc((size_t)vl_array_len, typeSize)))
H5_OUT_OF_MEMORY_ERROR(ENVONLY, "H5Awrite: failed to allocate raw VL write buffer");

translate_wbuf(ENVONLY, buf, mem_type_id, type_class, vl_array_len, writeBuf,
(size_t)vl_array_len * typeSize);

if ((status = H5Awrite((hid_t)attr_id, (hid_t)mem_type_id, writeBuf)) < 0)
H5_LIBRARY_ERROR(ENVONLY);
Expand Down Expand Up @@ -1250,6 +1344,11 @@ Java_hdf_hdf5lib_H5_H5Aread_1VLStrings(JNIEnv *env, jclass clss, jlong attr_id,
if (NULL == buf)
H5_NULL_ARGUMENT_ERROR(ENVONLY, "H5Aread_VLStrings: read buffer is NULL");

/* The buffer must have one slot per element in the attribute. */
if (h5a_validate_slot_buf(env, (hid_t)attr_id, ENVPTR->GetArrayLength(ENVONLY, buf),
"H5Aread_VLStrings") < 0)
goto done;

if ((isStr = H5Tdetect_class((hid_t)mem_type_id, H5T_STRING)) < 0)
H5_LIBRARY_ERROR(ENVONLY);

Expand Down Expand Up @@ -1455,6 +1554,11 @@ Java_hdf_hdf5lib_H5_H5Awrite_1VLStrings(JNIEnv *env, jclass clss, jlong attr_id,
if (NULL == buf)
H5_NULL_ARGUMENT_ERROR(ENVONLY, "H5Awrite_VLStrings: write buffer is NULL");

/* The buffer must have one slot per element in the attribute. */
if (h5a_validate_slot_buf(env, (hid_t)attr_id, ENVPTR->GetArrayLength(ENVONLY, buf),
"H5Awrite_VLStrings") < 0)
goto done;

if ((isStr = H5Tdetect_class((hid_t)mem_type_id, H5T_STRING)) < 0)
H5_LIBRARY_ERROR(ENVONLY);

Expand Down Expand Up @@ -1690,6 +1794,10 @@ Java_hdf_hdf5lib_H5_H5Aread_1reg_1ref(JNIEnv *env, jclass clss, jlong attr_id, j
H5_BAD_ARGUMENT_ERROR(ENVONLY, "H5Aread_reg_ref: buf length < 0");
}

/* The buffer must have one slot per element in the attribute. */
if (h5a_validate_slot_buf(env, (hid_t)attr_id, n, "H5Aread_reg_ref") < 0)
goto done;

if (NULL == (ref_data = (H5R_ref_t *)calloc(1, (size_t)n * sizeof(H5R_ref_t))))
H5_OUT_OF_MEMORY_ERROR(ENVONLY, "H5Aread_reg_ref: failed to allocate read buffer");

Expand Down
Loading
Loading