Is "contact_prediction" unsupervised?

[ddofer]

I'm interested in the unsupervised contact prediction paper and analysis.
The example code shows a function for predicting contacts - is that a head that's been trained on the contact prediction task/dataset? Or does it just show attention per head X layer in an unsupervised way?

e.g. the example notebook here:
https://github.com/facebookresearch/esm/blob/master/examples/contact_prediction.ipynb

(IT's unclear if the method loads a pretrained/fine-tuned model. i.e supervised contact prediction).
Thanks!

[tomsercu]

The contact prediction comes from a sparse linear combination of the attention heads is used to directly predict protein contacts. sparse logistic regression trained on 20 protein structures.
See Rao et al. 2020 https://openreview.net/forum?id=fylclEqgvgd
This corresponds to the table here: https://github.com/facebookresearch/esm#unsupervised-structure-prediction

Thanks for calling this out, we will update the README / notebook to clarify this more.

[ddofer]

So, it is a supervised methods / previously trained/finetuned?
(Albeit with linear probing only? )
Thanks!

[tomsercu]

Not sure what is still unclear, but let me summarize:

• Both ESM-1b and MSA transformer are pretrained on MLM, sequences only.
• After pre-training, the model weights are frozen, not finetuned.
• The logistic regression head is fit on 20 protein structures

On semantics: we use the term unsupervised contact prediction because the contacts are learned directly via the unsupervised language modeling objective. The point we are making is that the logistic regression is not learning the contacts, the language modeling is.

Note that the attention heads predict contacts directly without using the regression weights. Results for averaging the top 1, 5, and 10 heads are shown in the paper, Table 2. Simply averaging the top 5 heads already performs better than training a Potts model using the same sequence database used for training ESM (in Table 2 compare Gremlin on ESM data to the lines for top-5 and top-10 heads). When logistic regression weights are used, they are fit with just 20 proteins. This improves performance further over averaging the heads. Figure 12 shows bootstrap results indicating that any randomly selected 20 proteins produce similar results. In the Low-N supervision section we show that the regression can be fit with even a single example.

All of this is evidence that the contacts are learned by the unsupervised pre-training -- which makes Potts models the natural comparison.

Hope that makes sense!

[Leo-T-Zang]

Is there any example code how to finetune the Contact Head module or the Logistic Regression?

Thanks a lot!