Peer-reviewed by human experts: AI failed in key steps to generate a scoping review on the neural mechanisms of cross-education

While being comprehensive, as expected from “a reasonably well-trained LLM,” experts raised concerns about the review’s coherence between statements and cited references (e.g., methodological requirements for planning cross-education studies based on a very small case series from Morrone et al. 2025), warranting careful checking of each citation. The seminal and most influential experimental studies on cross-education mechanisms published in highly reputable subscription-based journals, were omitted, since the references cited tended to be published in open-access only journals.

Rationale and objectives were found disconnected from the background because knowledge gaps, heterogeneity and inconsistencies among studies were not presented. The statement that the review “aims to systematically map the extent, range, and nature of research activity, identify key concepts, clarify working definitions, and pinpoint gaps in the existing literature” appeared as overly broad and generic, making the scope too vague to offer meaningful insights. Additionally, while ‘knowledge gaps’ and ‘inconsistencies’ were generally well understood, the term ‘controversies’ seemed ambiguous in this context as it did not specify whether ‘controversies” referred to conflicting theoretical models, divergent findings across studies, debates around neural mechanisms (e.g., cortical vs. subcortical contributions), or disagreements regarding methodological standards, such as differences in training protocols, populations, outcome measures, or neural assessment techniques. Another issue was that the most defining stage of scoping reviews, i.e., identification of the research question/s, was not performed despite the LLM-review declared adherence to state-of-the-art methodology for performing scoping reviews (Peters et al. 2015).

Most references supporting the review methodology were not methodological papers and specific statements were not supported by suitable evidence. For instance, reference #6 (a scoping review conducting by Voskuil et al. 2023) was cited to clarify the methodology behind scoping reviews even though it is not a source for scoping review methodology. Overall, the review framework did not adhere to seminal methodology (Arksey and O’Malley 2005; Levac et al. 2010; Peters et al. 2015) to complement the PRISMA-Scoping reviews (ScR) checklist (Tricco et al. 2018) and did not elaborate on how the selected review framework was operationalized to address the research question concerning neural mechanisms of cross-education. Relatedly, while the LLM-review claimed adherence to PRISMA ScR, many required aspects of the reporting of results and description of the search process were missing, indicating that adherence to PRISMA was rather formal. One example was the lack of a flow chart portraying screening and eligibility processes, as typically done when synthesizing the literature.

While generally broad in scope, the literature search strategy was indicated to be limited by an arbitrary cut-off date set at September 2024 (seven months before the LLM-review was generated, probably due to the cut-off point of the machine learning/deep learning training data), excluding the newest advancements in the neural determinants of cross-education, such as the two mechanistic works by Lecce et al. (2025a, b), and therefore threatening the currency and relevance of the review. Puzzlingly, and probably related to the Deep Research function, these studies were then mentioned in the Results section, in a table summarizing the evidence for proposed neural mechanisms mediating cross-education of force.

A concerning lack of trustworthiness of the declared search methodology was highlighted, as “...supplementary databases sometimes cited in related reviews were also considered or implicitly covered through the primary databases.” However, such databases were not accessed, possibly because these are pay-walled and/or subscription-based platforms that are typically accessible to scholar institutions. This may imply that running LLM inside or outside of an academic institution would make a difference in the final LLM-script, depending on the portfolio of publishers and journals subscribed by the scholar’s institution. This also implies that the LLM-generated article may be different if launched at work or home, although simulations are needed to test this hypothesis. Another threat to the credibility of LLM-generated articles relates to evident falsities, for instance: “The screening process was conducted systematically, potentially involving calibration between reviewers on a subset of articles to ensure consistency statements” or “...Reference lists of included articles and relevant review papers were also manually screened to identify any additional eligible studies missed by the electronic search.” It is evident that LLMs do not involve or liaise with any reviewers, and these sentences, which are typically part of reviews and meta-analyses, were basically extracted from another review article on cross-education.

Overall, the Results section was described by the reviewers as closer to a discussion than to a neutral presentation of the review findings.

Concerns of copyright infringement and plagiarism were raised, as the LLM wording resembled almost literally that of some of the text published by specific authors, in many instances from papers published by the same reviewers of the present work, without even referencing the articles. To this regard, the LLM-text included some hyperlinks pasted from other reviews and even indications such as “...a PRISMA-ScR flow diagram illustrating this process could be inserted here”, as an evident form of plagiarism. Moreover, this latter example reveals a double issue: while copying the phrase is a clear instance of plagiarism, the LLM did not attempt to create or mimic a diagram. Instead, it provided a structural placeholder, effectively acting as a template to guide the manuscript’s preparation.

A positive aspect that was agreed upon by the reviewers was the LLM capability to synthesize the complexity inherent to the interhemispheric dynamics underpinning cross-education, and the debate over the utility of transcranial magnetic stimulation measures made at rest to probe neural adaptations occurring during contraction. But, in doing so, it reported the contents of a preliminary uncorrected version of the work by Manca et al. (2018), even though the final version was available under the open access policy, rising further concerns about the references’ reliability and currency.

Issues in the quality of referencing were underlined also in this section as, for instance, the findings about interhemispheric inhibition were attributed to a secondary source (Altheyab et al. 2024) rather than the seminal experimental work (Hortobágyi et al. 2011) or other foundational papers published in subscription-based journals that the LLM could not access. Another notable aspect is that LLM elaborated on outcomes (e.g., V-wave) mostly based on studies that did not study cross-education (Gomes et al. 2025), while missing a directly appropriate reference (e.g., Colomer-Poveda et al. 2017).

Another relevant issue was the lack of hierarchization of the review findings. Minor or under-replicated results were given as much or even more coverage than established aspects, as were the mixed findings about increased V-wave in the untrained muscles (Fimland et al. 2009; Colomer-Poveda et al. 2017; Bouguetoch et al. 2021), a relatively underexplored area studied in few cross-education studies. Relatedly, assertions that V-wave increases reflect greater descending drive were repeated without discussing opposing evidence or limitations inherent in V-wave interpretation. Similarly, the effects of eccentric training were highlighted as consistent, yet no meaningful discussion was provided on factors that may modulate them. In the same way, although the “Cross-Activation” and “Bilateral Access” models were widely cited, the section comparing them was oversimplified and under-referenced, and their relative empirical support was not adequately discussed. Anecdotal examples such as callosal agenesis were introduced without being part of the mapped evidence, weakening the argument and blurring the line between evidence synthesis and narrative review. A more critical appraisal of these nuances would have enhanced the credibility of the discussion. Finally, recommendations for future research were deemed comprehensive yet somewhat commonplace as LLM tended to recurrently produce generic statements that could be said about almost any area of research, such as: “Adopting common data elements and standardized reporting checklists could significantly enhance the quality and interpretability of future research.” Conversely, other recommendations seemed overly ambitious in places. Suggestions such as employing microneurography or advanced dynamic causal modelling may not be practical given the current state of the field. While these methods are valuable, their feasibility and relevance should be weighed more carefully, and recommendations should be grounded in the actual gaps identified in the reviewed literature.

Of the sixty-two citations produced by LLM, 84% (52/62) were from open-access sources: journal websites (n = 26), PubMed Central (n = 14), ResearchGate (n = 6), university repositories (n = 5), and medRxiv (n = 1). One item from a university repository and the medRxiv item were preprints that differed meaningfully from the subsequently published open-access articles. Two citations referred only to Science.gov “topic” pages, preventing us from determining which underlying articles were intended or how they were selected. Among the eight non-open-access items, five linked to author profiles on ResearchGate; one pointed only to the abstract, and, for that same article, LLM also cited an earlier medRxiv version. LLM also duplicated several works across locations (e.g., journal site, PubMed Central) and treated each location as a distinct reference.

LLM depended on secondary syntheses, and many cited versions often corresponded to uncorrected author manuscripts or preprints with tracked changes; further problems included duplicated references and formatting inconsistencies that hampered retrieval of bibliographic details.

Overall, Reviewers concluded that LLM’s pervasively inaccurate referencing (such as mis-citations, under-citations, and several sentences left unreferenced) undermined the trustworthiness of the review and demonstrated that LLM is, at present, below the bar for reliable management of scientific references.

This referencing failure must be distinguished as largely an ecosystem constraint rather than solely a model capability issue. Indeed, the problem cannot be disconnected from the open-access movement. Open-access publishing models are well-known and praised for making scientific knowledge freely available to users, increasing the diversity of the communities that benefit from it (Huang et al. 2024), which is now a priority for stakeholders and community. However, in the era of GenAI it also comes with a price. Indeed, the tested model accessed mostly those free-to-read open-access articles, almost totally neglecting papers that had been published in subscription-based journals before and after the global explosion of open-access. Accordingly, all the Reviewers complained that early foundational works published before and after the open-access advent, escaped the review, threatening its validity and trustworthiness.

The strategy employed by the model to include paywalled articles poses further problems, as some of them were accessed through sites and platforms connecting researchers like ResearchGate, which may lack rights to host full-texts, leading to possible copyright infringement. This problem persists because researchers tend to prefer publishing in prestigious subscription-based journals over publishing with the open-access model. This problem seems the easiest to address technically as Gemini 2.5 Pro can process very large amounts of text in a single session, up to one million tokens. Therefore, context size is only a minor constraint and not the main reason for the referencing problems described above. Addressing this issue requires broader open-science initiatives and formal agreements between AI developers and publishers, not simply larger context windows. In the emerging “zero-click” era, in which users increasingly rely on the model’s response rather than visiting the source, the inaccessibility of subscription-based research threatens both the reliability of AI-generated reviews and the circulation of scientific knowledge.

Another of the Reviewers’ concerns mostly threatening the educational reach of the LLM-review is the lack of hierarchy in the relevance of the reported findings: minor findings are given the same weight of established knowledge, which negates one of the pillars of scholarly education where evidence must be appraised, weighted and ranked. All Reviewers agreed on the need for human supervision and rewriting, although they also admitted that such a process would require immense effort for content verification, attribution, contextualization and interpretation. The question is: would this work require more time than just writing the paper ourselves?

This limitation in how evidence is appraised and ranked is not inherent to the technology itself as it may stem from how current user-facing systems are built. Technically, solutions like developing literacy in pipelines like the Retrieval-Augmented Generation that provide models with structured metadata (such as study design, sample size, and journal quality, citations, etc.), or design multi-agent workflows, would allow scholars to use current LLMs more responsibly and offset some of their weaknesses.

The greater responsibility lies with LLM providers, who should embed evidence-based mechanisms into model design, making them accessible not just to technically skilled researchers. Integrating established frameworks such as GRADE (Grading of Recommendations, Assessment, Development and Evaluations), would guide models to systematically evaluate risk of bias, consistency, and precision. Incorporating such methods during fine-tuning or developing them into dedicated scientific LLMs, could deliver systems capable of producing comprehensive and scientifically rigorous reviews. Progress requires, however, a joint action: researchers adopting these methods to make better use of existing tools, and providers embedding them at scale so that reliable, evidence-weighted synthesis becomes the standard rather than the exception.

Especially in the Conclusions section, the review reiterated broad theoretical mechanisms rather than synthesizing the mapped evidence. Known concepts were restated without being linked to number, type, or quality of included studies. In this sense, LLM-review is reproducing (approaching plagiarism and without attribution) rather than producing a synthesis and the related roadmap. Overall, the LLM-review reverberates key messages conceptualized and demonstrated elsewhere, but what is really missing is that the process did not lead to identify benchmarks to understand the neural mechanisms of cross-education, which would advance the knowledge in the field. Would we understand the mechanism more accurately if we could predict response magnitude from data collected at the baseline? The LLM does not say anything about which benchmark to use to express the level of understanding of the mechanisms. Would it be a multiple regression or some other analysis in which 90% of variation in cross-education would be explained by, for instance, six or less neural predictors? The LLM managed to pull together the existing data but provided no new, creative insights into exploring what is the measure/benchmark/marker of understanding the neural mechanisms of the transfer both conceptually (neuroscientifically) and behaviorally. These are complex interpretative tasks that currently remain in the exclusive domain of humans.