Fine-tuning an ESM pre-trained model

[tomsercu]

This is a recurring question, so let me summarize what @joshim5 has provided in issues #9 #11 #12 and #30 :

• We don't have any near term plans to release pre-training code at this time. We are working with a small team and limited capacity; open sourcing our research codebase would require significant additional investment in splitting out public (properly tested), vs research components. At the other hand, a one-time dump of research code without proper follow-up and maintenance is not acceptable either. So for now we are working with the compromise of having a well-supported small standalone repo that allows fast open sourcing. All details necessary for reproduction are in the papers, and if there are any remaining questions we are happy to answer them here. We hope you will find the pre-trained models useful for your work!
• The ESM model is a subclass of pytorch nn.Module, so you can just use it as a component inside another torch model. There are many generic pytorch tutorials that can show how to train/finetune models, for example this one from Huggingface. Also this tutorial finetuning a torchvision model, and this (similar) tutorial.
• In the examples folder, we provide a tutorial for training a downstream model to predict the fitness of protein variants.
• In our experiments, we usually haven't seen much of an improvement from evolutionary fine-tuning. For example, see Figure 15 in the appendix of our recent paper at ICLR 2021.

Keep the questions coming!

[BioTechTimy]

In the examples folder, we provide a tutorial for training a downstream model to predict the fitness of protein variants.

The link https://github.com/facebookresearch/esm/blob/master/examples/variant_prediction.ipynb seems to be broken:
The 'facebookresearch/esm' repository doesn't contain the 'examples/variant_prediction.ipynb' path in 'main'.

[r-kellerm]

Looks like the correct link is https://github.com/facebookresearch/esm/blob/main/examples/sup_variant_prediction.ipynb. (Or zero-shot variant prediction example under https://github.com/facebookresearch/esm/tree/main/examples/variant-prediction)

[realamirhe]

In our experiments, we usually haven't seen much of an improvement from evolutionary fine-tuning. For example, see Figure 15 in the appendix of our recent paper at ICLR 2021.

Figure 16 was intended

We fine-tune the full
ESM-1b model (which has 50x more parameters than UniRep) on 380 protein sequence families.
We find that after 30 epochs of fine-tuning, long range P@L increases only slightly, with an average
of 1.6 percentage points (Fig. 16).