I need to be clear here: it’s absolutely possible that you’re right. That’s why a huge part of my comment above is emphasizing that this dudes conclusions drawn from the notes as passed through Claude are largely premature, and need to be verified MUCH more rigorously.
But I do think that categorically disregarding this - particularly, without even understanding the nuance of his methodology or doing any sort of additional verification or isolated reproduction of results with different (non-LLM) tools - is also premature.
I need to be clear here: it’s absolutely possible that you’re right. That’s why a huge part of my comment above is emphasizing that this dudes conclusions drawn from the notes as passed through Claude are largely premature, and need to be verified MUCH more rigorously.
But I do think that categorically disregarding this - particularly, without even understanding the nuance of his methodology or doing any sort of additional verification or isolated reproduction of results with different (non-LLM) tools - is also premature.
I wish The Guardian included links to other more technical sources rather than just replying on their own direct communication with the researcher.
Unfortunately in the current environment, giving people the benefit of the doubt has been very over-extended.
I have not read through the papers they have put out, maybe there’s something more in there:
https://researchportal.helsinki.fi/en/persons/jan-von-plato/publications/
There’s nothing to disregard. They haven’t posted a single result.