Nah, this ain’t it. I too wish it was just using an LLM to get a head start on translation, but apparently they’re getting the LLM, which has been trained on mathematical proofs, to generate mathematical proofs from text that the LLM is told contains mathematical proofs.
I need to be clear here: it’s absolutely possible that you’re right. That’s why a huge part of my comment above is emphasizing that this dudes conclusions drawn from the notes as passed through Claude are largely premature, and need to be verified MUCH more rigorously.
But I do think that categorically disregarding this - particularly, without even understanding the nuance of his methodology or doing any sort of additional verification or isolated reproduction of results with different (non-LLM) tools - is also premature.
Nah, this ain’t it. I too wish it was just using an LLM to get a head start on translation, but apparently they’re getting the LLM, which has been trained on mathematical proofs, to generate mathematical proofs from text that the LLM is told contains mathematical proofs.
That’s hallucination city.
I need to be clear here: it’s absolutely possible that you’re right. That’s why a huge part of my comment above is emphasizing that this dudes conclusions drawn from the notes as passed through Claude are largely premature, and need to be verified MUCH more rigorously.
But I do think that categorically disregarding this - particularly, without even understanding the nuance of his methodology or doing any sort of additional verification or isolated reproduction of results with different (non-LLM) tools - is also premature.
I wish The Guardian included links to other more technical sources rather than just replying on their own direct communication with the researcher.
Unfortunately in the current environment, giving people the benefit of the doubt has been very over-extended.
I have not read through the papers they have put out, maybe there’s something more in there:
https://researchportal.helsinki.fi/en/persons/jan-von-plato/publications/
There’s nothing to disregard. They haven’t posted a single result.