Any upcoming update to your repository?

[ashim95]

Hi,

Thanks for releasing the code.

Can you please answer a few questions?

1. Does this code reproduce results from your EMNLP paper? And when do you plan to release all the preprocessing scripts along with readme?

2. I tried preprocessing the reddit l2 dataset the way you described in the paper (posts greater than 50 words and balanced dataset for all classes). But I did not get the numbers of the dataset you reported (260k for training, 32k for test, valid). Are these numbers for 10 most frequent classes or all 23 of them?

3. In your arxiv version, the numbers for a linear classifier (LR) in tables 1 and 4 do not match (52.5 vs 21.1 for in-domain). I assume it is because the num_classes in former is 10 while it is 23 in the latter?

4. Can you share the exact splits (train, test, oodtest) you used in your paper?

Thanks again for releasing the code,
Ashim

[Sachin19]

Hi Ashim,

Thank you for interest in our work!

1. I am currently working on cleaning the code and plan to release the clean version by next week.
2. All the numbers are for most frequent 10 classes. The current version of the code might have some different settings than what we reported in the paper, I'll make sure everything is reproducible by then
3. Table 1 reports results from a BiLSTM model whereas Table 4 has results from a logistic regression model. The number of classes is 10 everywhere
4. The splits you have mentioned in your point 2 are correct. The number of ood examples we used were ~100000 balanced by the classes

[TejaGollapudi]

Hi,
Thank you for the code.

I am facing the same problem as Ashim when I am trying to recreate the reddit l2 dataset split as mentioned in the paper. The number of instances aren't matching.
How was the out of domain test set processed?

Thank you ,
Teja

[Sachin19]

Sorry about the delay in my reply. For each class in the larger OOD test set, I sampled 10000 examples at random to create the test set. I haven't been able to clean up the repository in the past couple months due to other commitments, I have started that process now and it should be ready this week,

Please let me know if you have more questions.