verify_state_save and power loss

[micha-yonin-sndk]

i'm running fio 3.34, libaio engine, qd 128, 4k rw, with verify_state_save=1, as well as latency logs
during the test, my machine is performing a warm reboot, allowing fio to save the verify states and latency logs
after the reboot - i'm performing a read using the states file, and getting miscompares.

for example -
in verify file i see, numberio = 16515
then when comparing completions from verify file to to end of latency log
of last 256 lines in latency log:
first 128 are the 128 from completion log
and then another 128 - most of them have miscompares.

the way it makes sense to me is that when last 128 were sent - warm reboot has started and link to controller was lost, so some commands didn't get a chance to transfer.

the problem with this is that i see in the io_completed function that completions and latency log are added togther:

static void io_completed(struct thread_data *td, struct io_u **io_u_ptr,
			 struct io_completion_data *icd)
{
...
	if (!io_u->error && ddir_rw(ddir)) {
...
		if (ddir == DDIR_WRITE)
			file_log_write_comp(td, f, io_u->offset, bytes);

		if (should_account(td))
			account_io_completion(td, io_u, icd, ddir, bytes);

any ideas?

[sitsofe]

Hi @micha-yonin-sndk,

• Can you try the latest master version of fio (you will have to compile it by hand)
• How are you triggering the saving of verify state file?
• What command line are you using to do verification?

[micha-yonin-sndk]

thanks @sitsofe !
i will try to compile and repeat with master. do you mean that commit 935297d is addressing this? or is there another commit?
in general - is this a known issue?
i'm running the write with --trigger_timeout=120 and --trigger='<script that performs a warm reset to host>'
after reset, i'm doing a read script with verify_state_load=1

[sitsofe]

@micha-yonin-sndk:

do you mean that commit 935297d is addressing this? or is there another commit?

It might do but I don't know for sure. It's quicker to let you find out :-)

in general - is this a known issue?

I've done a search and let's just say yours is the first to articulate it clearly.

Before I forget what is the full jobfile / command line that you're using?

[micha-yonin-sndk]

thanks again @sitsofe !!
running with latest master (including 1e6ca39) seems to fix the issue
seems to me to be related to the inflight handling, but can't tell for sure

the command line for write is:
fio --output-format=json --output=%s --trigger_timeout=120 and --trigger='ipmitool chassis power reset'
with this config file:

[global]
group_reporting
time_based
blocksize=4k
iodepth=128
direct=1
numjobs=1
ioengine=libaio
verify=md5
verify_backlog=10000000
verify_async=4
verify_fatal=1
verify_dump=1
rw=randwrite
runtime=10m
verify_state_save=1

similarly for read (without timeout trigger)

[global]
group_reporting
blocksize=4k
iodepth=128
direct=1
numjobs=1
ioengine=libaio
verify=md5
verify_backlog=10000000
verify_async=4
verify_fatal=1
verify_dump=1
rw=randread
verify_state_load=1

[sitsofe]

@micha-yonin-sndk:
(I've done some light reformatting of your comment - for code blocks you use ``` on GitHub)

Good to know your problem is solved - I'll go ahead and close this issue. As you say, it's hard to know for sure why (without testing individual commits) but it's coincidental someone else improved code in that area.

[sitsofe]

@micha-yonin-sndk FYI: you may get verification failures if your workload wraps around and fio ends up sending two writes (one from the original pass and the other from the following pass) for the same offset at the same time. The likelihood of this happening depends on a number of factors but the race can be prevented by setting serialize_overlap=1 at the cost of some overhead.