No, DeepMind has not solved protein folding

(Please note that this post was updated on 12th Dec 2020 – see below)

This week DeepMind has announced that, using artificial intelligence (AI), it has solved the 50-year old problem of ‘protein folding’. The announcement was made as the results were released from the 14th and latest competition on the Critical Assessment of Techniques for Protein Structure Prediction (CASP14). The competition pits teams of computational scientists against one another to see whose method is the best at predicting the structures of protein molecules – and DeepMind’s solution, ‘AlphaFold 2’,  emerged as the clear winner.

201202-DeepMind-blog

Don’t believe everything you read in the media

There followed much breathless reporting in the media that AI can now be used to accurately predict the structures of proteins – the molecular machinery of every living thing. Previously the laborious experimental work of solving protein structures was the domain of protein crystallographers, NMR spectroscopists and cryo-electron microscopists, who worked for months and sometimes years to work out each new structure.

Should the experimentalist now all quit the lab and leave the field to Deep Mind?

No, they shouldn’t, for several reasons.

Firstly, there is no doubt that DeepMind have made a big step forward. Of all the teams competing against one another they are so far ahead of the pack that the other computational modellers may be thinking about giving up. But we are not yet at the point where we can say that protein folding is ‘solved’. For one thing, only two-thirds of DeepMind’s solutions were comparable to the experimentally determined structure of the protein. This is impressive but you have to bear in mind that they didn’t know exactly which two-thirds of their predictions were closest to correct until the comparison with experimental solutions was made.* Would you buy a satnav that was only 67% accurate?

So a dose of realism is required. It is also difficult to see right now, despite DeepMind’s impressive performance, that this will immediately transform biology.

201202-DeepMind-molecules

Impressive predictions – but how do you know they’re correct?

Alphafold 2 will certainly help to advance biology. For example, as already reported, it can generate folded structure predictions that can then be used to solve experimental structures by crystallography (and probably other techniques). So this will help the science of structure determination go a bit faster in some cases.

However, despite some of the claims being made, we are not at the point where this AI tool can be used for drug discovery. For DeepMind’s structure predictions (111 in all), the average or root-mean-squared difference (RMSD) in atomic positions between the prediction and the actual structure is 1.6 Å (0.16 nm). That’s about the size of a bond-length.

That sounds pretty good but it’s not clear from DeepMind’s announcement how that number is calculated. It might be calculated only by comparing the positions of the alpha-Carbon atoms in the protein backbone – a reasonable way to estimate the accuracy of the overall fold of the protein. Or, it might be calculated over all the atomic positions, a much more rigorous test. If it is the latter, then an RMSD of 1.6 Å is an even more impressive result.

But it’s still not nearly good enough for delivering reliable insights into protein chemistry or drug design. To do that, we want to be confident of atomic positions to within a margin of around 0.3 Å. AlphaFold 2’s best prediction has an RMSD for all atoms of 0.9 Å. Many of the predictions contributing to their average of 1.6 Å will have deviations in atomic positions even greater than that. So, despite the claims, we’re not yet ready to use Alphafold 2 to create new drugs.

There are other reasons not to believe that the protein folding problem is ‘solved’. AI methods rely on learning the rules of protein folding from existing protein structures. This means that it may find it more difficult to predict the structures of proteins with folds that are not well represented in the database of solved structures.

Also, as reported in Nature, the method cannot yet reliably tackle predictions of proteins that are components of multi-protein complexes. These are among the most interesting biological entities in living things (e.g. ribosomes, ion channels, polymerases). So there is quite a large territory remaining were AlphaFold 2 cannot take us. The experimentalists, who have been successful in mapping out the structures of complexes of growing complexity, have still a lot of valuable work to do.

While all of the above is supposed to sound a note of caution to counter some of the more hyperbolic claims that have been heard in the media in recent days, I still want to emphasise my admiration for the achievements of the AlphaFold team. They have clearly made a very significant advance.

That advance will be much clearer once their peer-reviewed paper is published (we should not judge science by press releases), and once the tool is openly available to the academic community – or indeed anyone who wants to study protein structure.

Update (02 Dec, 18:43): This post was updated to provide a clearer explanation of the RMSD measures used to compare predicted and experimentally determined protein structures. I am very grateful to Prof Leonid Sazanov who pointed out some necessary corrections and additions on Twitter.

*Update (12 Dec, 15:35): Strictly this is true, but it misses the more important point that the score given to each structure prediction (GDT_TS) broadly correlates with the closeness of its match to the experimental structure. As a result, I have deleted my SatNav crack.

For a deeply informed and very measured assessment of what DeepMind has actually achieved in CASP14, please read this blogpost by Prof. Mohammed AlQuraishi who knows this territory much better than I do. His post is pretty long but you can skip the technical bits explaining how AlphaFold 2 works. He gives a very good account of the nature of DeepMind’s advance; in AlQuraishi’s view, AlphaFold 2 does represent a solution to the protein structure prediction problem, though he is careful to define what he means by a solution. He also acknowledges that there are still some significant improvements to be made to the programme, but regards these as more of an engineering challenge than a scientific one. He agrees that AlphaFold 2 won’t be used any time soon for drug design work. AlQuraishi also gives an excellent overview of the implications of this work for protein folders, structural biologists and biotechnologists in general, and offers some very interesting thoughts on the differences between DeepMind’s approach to research and that of more traditional academic groups.

This entry was posted in Protein Crystallography, Science. Bookmark the permalink.

23 Responses to No, DeepMind has not solved protein folding

  1. Daniel says:

    For sure, if is a future improvements / that one was almost two fold / the next will be from AI also.
    09 to 03 is a hard one but no one see that 2rd fold before. As no one see Alpha Go and Alpha Zero coming before 30 years.

  2. Yínká says:

    “Would you buy a satnav that was only 67% accurate?”

    What a question to ask!

    Of course I will buy the the SatNav if there is no other alternative, hell I’ll take a vaccine that’s only 50% effective if there is no close/other alternative.

    • Stephen says:

      Good luck with your inaccurate satnav. There is a good alternative: a map. For protein structures, there are also good alternatives to a computer that makes incorrect predictions a third of the time. I think you may have missed the point I was making about the excessive claims made for AlphaFold 2, despite the fact that it it still a significant advance.

      • Jim Albert says:

        The problem is that the map will take a year or more to be produced. If you have a 67% accurate SatNav you can use it until you get the map.

  3. David says:

    The accuracy of DeepMind’s prediction is more close to 90% than 60% in Nature article. Isn’t it?

  4. Laurent Bianchetti says:

    The author speaks true when he says Alpha Fold 2 must be made freely accessible to the scientific and academic community. There are a lot of proteins which are refractory to the process of expression, purification and crystallization and a 3D prediction algorithm is required. In addition, crystallographers often mutate polypeptide sequences (loop deletions, residue mutations) or use crystallization additives to enhance crystal formation and that may alter the native protein 3D structure or its properties of oligomerization. I do not share the author’s view on crystal structures resolution accuracy because proteins are dynamic therefore the positions of amino-acid side chains and even the Ca backbone are moving. Therefore, does it make sense to claim a 0.3 resolution of x-ray crystal atomic positions ? The electron densities are sometimes locally so weak that residue atoms cannot be placed correctly and alternative placements of residue side-chains are written in the Protein data bank (PDB) files.
    I would like to known if Alpha fold 2 also predict 3D structures for intrinsically disordered regions ?
    Because of its unprecedented success in the CASP competition, Alpha folds 2 raises a lot of interest and curiosity in the molecular modeling and structural biology community. Of course it will help protein modeling and drug design. The structure is such a wealth of information in addition to the protein primary sequence. A 3D structure prediction is better than nothing, isn’t it ? Does Alpha fold 2 generates predictions for domains that are likely new folds ? We have know for decades about the genetic code and I wish Alpha fold 2 has cracked the structural code of polypeptide sequences. It would be unfair to be exceedingly skeptic about the program capabilities, aren’t X-crystallographers a bit afraid to loose some kind of monopole on 3D structure resolutions ? Let’s evaluate the program together.

    • Stephen says:

      Thank you for the comment even if it is a bit wide of the mark. You will see that I started and ended my piece congratulating DeepMind for having made a very significant advance in the science of predicting protein folding. What I object to is the hype that accompanied the announcement of this advance and the unjustified claim to have solved protein folding.

      Regarding the 0.3Å precision of proteins structures, then yes, of course this is a global average for a typical high-resolution crystal structure. Some parts of the structure will be better defined than this and some, such as surface loops with high B-factors, will be much, much worse. But the same considerations apply to claims of overall RMSD figures of 1.6Å – the match between predicted and actual structures will be better and worse in different parts of the molecule.

      By all means let’s evaluate the programme together, but only if DeepMind is prepared to leave the hype outside the door.

      • General audience says:

        Then keep objecting to hype…you are free to do so…
        Every person have a right to criticise.
        But also one can claim that 90 percent accuracy is achievable…may be this is just a beginning…there will be more models and a healthy competition will lead to 99.9999 percent accuracy….just don’t assume it won’t be achieved because of hype…

Comments are closed.