Bug (CSVReaderBase) - missing required fields are not detected in name-value mode

[arthurianresolve]

Issue

A CSV record can leave out a required field without getting a MISSING_DATA_ERROR when the record uses name=value fields.

Problem

CSVHelper.REQUIRED_FIELDS says that .data.fields.required lists fields that a record must have:

datawave/warehouse/ingest-core/src/main/java/datawave/ingest/data/config/CSVHelper.java

Lines 118 to 121 in 02d6867

	/**
	* Partial configuration key for specifying CSV fields that a record must have.
	*/
	public static final String REQUIRED_FIELDS = ".data.fields.required";

However, CSVReaderBase.checkField(...) checks a required field only after that field has been read:

datawave/warehouse/ingest-csv/src/main/java/datawave/ingest/csv/mr/input/CSVReaderBase.java

Lines 218 to 225 in 02d6867

	@Override
	protected void checkField(final String name, final String value) {
	super.checkField(name, value);
	if (csvHelper.isFieldRequired(name) && StringUtils.isEmpty(value)) {
	event.addError(RawDataErrorNames.MISSING_DATA_ERROR);
	log.error("Missing required field: {}", name);
	}

if (csvHelper.isFieldRequired(name) && StringUtils.isEmpty(value)) {
    event.addError(RawDataErrorNames.MISSING_DATA_ERROR);
    log.error("Missing required field: {}", name);
}

When the header is disabled and extra fields are enabled, getEvent() processes only the name=value fields found in the record:

datawave/warehouse/ingest-csv/src/main/java/datawave/ingest/csv/mr/input/CSVReaderBase.java

Lines 157 to 184 in 02d6867

	final String[] rawEventFields = _tokenizer.getTokenArray();
	final String[] header = csvHelper.getHeader();
	// If the event date field name is not specified in the configuration, then set the event date to the file modification time.
	if (StringUtils.isEmpty(eventDateFieldName))
	event.setTimestamp(fileModificationTime);
	// We still try to process the event record.
	final int fields = Math.min(rawEventFields.length, header.length);
	String field, fieldName;
	int i;
	for (i = 0; i < fields; i++) {
	field = StringEscapeUtils.unescapeCsv(rawEventFields[i]);
	fieldName = header[i];
	field = csvHelper.clean(fieldName, field);
	if (field != null)
	processPreSplitField(fieldName, field);
	}
	// Check to see if we have data beyond the header specification that should be processed. This is the case for the CSV logs
	if (csvHelper.processExtraFields() && rawEventFields.length > header.length) {
	while (i < rawEventFields.length) {
	processExtraField(rawEventFields[i]);
	i++;
	}
	}

If a required field is not present, checkField() is never called for that field. There is no later check that compares the configured required fields with the fields read from the record.

For example:

example.data.header.enabled=false
example.data.process.extra.fields=true
example.data.fields.required=REQUIRED

Input:

OPTIONAL=value

Expected behavior

The event should contain MISSING_DATA_ERROR because REQUIRED is missing.

Actual behavior

OPTIONAL is processed, but the missing REQUIRED field is not detected.

Affected readers

Classes that use or extend CSVReaderBase may be affected, including:

• CSVRecordReader
• NYCTLCReader

Suggested fix

Track the field names found while reading each event. After all fields are processed, compare them with the configured required fields and add MISSING_DATA_ERROR when any required field is missing.

Test

The current warehouse/ingest-csv tests do not test .data.fields.required or MISSING_DATA_ERROR.

Add a focused reader test with:

• header processing disabled
• extra-field processing enabled
• REQUIRED configured as a required field
• input containing only OPTIONAL=value

Expected result: the event contains MISSING_DATA_ERROR.