Bug (CSVHelper) - negative thresholds cause invalid ingest behavior

[arthurianresolve]

Issue

CSVHelper accepts negative field and multi-value thresholds.

Problem

CSVHelper.setup(Configuration) reads both threshold values without checking whether they are negative:

datawave/warehouse/ingest-core/src/main/java/datawave/ingest/data/config/CSVHelper.java

Lines 234 to 240 in 02d6867

	this.fieldSizeThreshold = config.getInt(this.getType().typeName() + FIELD_SIZE_THRESHOLD, this.fieldSizeThreshold);
	this.thresholdAction = ThresholdAction.valueOf(config.get(this.getType().typeName() + THRESHOLD_ACTION, this.thresholdAction.name()).toUpperCase());
	this.thresholdReplacement = config.get(this.getType().typeName() + THRESHOLD_FIELD_REPLACEMENT, this.thresholdReplacement);
	this.truncateField = config.get(this.getType().typeName() + TRUNCATE_FIELD, this.truncateField);
	this.dropField = config.get(this.getType().typeName() + DROP_FIELD, this.dropField);
	this.multiFieldSizeThreshold = config.getInt(this.getType().typeName() + MULTI_VALUED_THRESHOLD, this.multiFieldSizeThreshold);

this.fieldSizeThreshold =
        config.getInt(this.getType().typeName() + FIELD_SIZE_THRESHOLD, this.fieldSizeThreshold);

this.multiFieldSizeThreshold =
        config.getInt(this.getType().typeName() + MULTI_VALUED_THRESHOLD, this.multiFieldSizeThreshold);

A negative field-size threshold makes every field exceed the limit. With the TRUNCATE action, the code can call substring(0, -1) and throw StringIndexOutOfBoundsException:

datawave/warehouse/ingest-core/src/main/java/datawave/ingest/data/config/ingest/CSVIngestHelper.java

Lines 199 to 213 in 02d6867

	protected void applyThresholdAction(Multimap<String,String> fields, String fieldName, String value, int sizeLimit) {
	switch (helper.getThresholdAction()) {
	case DROP:
	processField(fields, helper.getDropField(), aliaser.normalizeAndAlias(fieldName));
	break;
	case REPLACE:
	processField(fields, fieldName, helper.getThresholdReplacement());
	break;
	case TRUNCATE:
	processField(fields, fieldName, value.substring(0, sizeLimit));
	processField(fields, helper.getTruncateField(), aliaser.normalizeAndAlias(fieldName));
	break;
	case FAIL:
	throw new IllegalArgumentException("A field : " + fieldName + " was too large to process");
	}

A negative multi-value threshold is never reached because the value counter starts at zero and is only compared for equality. This silently disables the limit:

datawave/warehouse/ingest-core/src/main/java/datawave/ingest/data/config/ingest/CSVIngestHelper.java

Lines 174 to 190 in 02d6867

	protected void processPreSplitField(Multimap<String,String> fields, String fieldName, String fieldValue) {
	if (fieldValue != null) {
	if (helper.isMultiValuedField(fieldName)) {
	// Value can be multiple parts, need to break on semicolon
	String singleFieldName = helper.usingMultiValuedFieldsDisallowlist() ? fieldName : helper.getMultiValuedFields().get(fieldName);
	int limit = helper.getMultiFieldSizeThreshold();
	int count = 0;
	for (String value : StringUtils.splitIterable(fieldValue, helper.getEscapeSafeMultiValueSeparatorPattern())) {
	value = helper.clean(singleFieldName, value);
	if (value != null) {
	if (count == limit) {
	applyMultiValuedThresholdAction(fields, fieldName, singleFieldName);
	break;
	} else {
	processField(fields, singleFieldName, value);
	count++;
	}

Expected behavior

CSVHelper.setup(Configuration) should reject threshold values below zero.

A value of zero should remain valid.

Actual behavior

Negative values are accepted.

This can disable the multi-value limit or cause field processing to fail with an unrelated index error.

Affected helpers

Ingest helpers that use CSVHelper threshold processing may also be affected, including:

• CSVIngestHelper
• ExtendedCSVIngestHelper

Suggested fix

Validate both threshold values during CSVHelper.setup(Configuration).

Throw IllegalArgumentException when either value is below zero, and include the invalid key and value in the message.

Test

The current upstream tests do not cover negative values for:

• <datatype>.data.field.length.threshold
• <datatype>.data.multivalued.threshold

Add setup tests that confirm negative values are rejected and zero remains valid.