Bug in handling of gzipped input files

[Waino]

The command line help indicates that gzipped input files are supported. However, if a gzipped training data file or validation data file is given, training fails with UnicodeDecodeError.

File "/l/sgronroo/scratch/theanopy3/bin/theanolm", line 12, in
exec(compile(open(file).read(), file, 'exec'))
File "/l/sgronroo/scratch/theanopy3/theanolm/bin/theanolm", line 46, in
main()
File "/l/sgronroo/scratch/theanopy3/theanolm/bin/theanolm", line 41, in main
args.command_function(args)
File "/l/sgronroo/scratch/theanopy3/theanolm/theanolm/commands/train.py", line 303, in train
trainer.train()
File "/l/sgronroo/scratch/theanopy3/theanolm/theanolm/trainers/basictrainer.py", line 132, in train
self._validate(perplexity)
File "/l/sgronroo/scratch/theanopy3/theanolm/theanolm/trainers/localstatisticstrainer.py", line 57, in _validate
perplexity = self.scorer.compute_perplexity(self.validation_iter)
File "/l/sgronroo/scratch/theanopy3/theanolm/theanolm/scoring/textscorer.py", line 130, in compute_perplexity
for word_ids, _, mask in batch_iter:
File "/l/sgronroo/scratch/theanopy3/theanolm/theanolm/iterators/batchiterator.py", line 93, in next
sequence = self._read_sequence()
File "/l/sgronroo/scratch/theanopy3/theanolm/theanolm/iterators/batchiterator.py", line 168, in _read_sequence
for word in utterance_from_line(line)]
File "/l/sgronroo/scratch/theanopy3/theanolm/theanolm/iterators/batchiterator.py", line 20, in utterance_from_line
line = line.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte

It appears that the problem is caused by the mmap access to the file (SentencePointers in iterators/shufflingbatchiterator:66) failing for gzipped files. The transparent unzipping (implemented in TextFileType filetypes.py:95) has no effect when using mmap.

[senarvi]

I'm thinking about this kind of solution:

• Provide a temporary directory as a command line argument.
• Store two arrays using numpy.memmap(): one that contains the word IDs and one that contains the mask of each sequence.
• Iterate the data by accessing these arrays in random order.