Definition
AI research is the practice that produces new machine learning methods, and it does not run the way most people picture science running. There is no journal at the centre of it. A result becomes public as an arXiv preprint the day its authors decide it is finished, circulates for months, gets reimplemented by strangers, and only afterwards — if ever — passes peer review at a conference. NeurIPS 2025 closed submissions on 15 May and notified authors on 18 September: 126 days in which a paper was already being cited, rebuilt and shipped into products with no reviewer verdict attached to it at all. Correctness here is settled by replication and by the community failing to break a result, not by acceptance.
That inversion tells you what a paper is evidence of: a preprint is a claim with a method attached, and its later acceptance at NeurIPS, ICML, ICLR, ACL or CVPR is a lagging signal about something the field has usually already judged for itself.
How It Works
The working unit is the preprint plus the artefact: a group runs experiments, posts the paper to arXiv, and increasingly releases code, weights or an evaluation harness with it. From that moment the paper is in the world. arXiv moderates for topicality and scope; it does not check whether the result is true.
The volume this produces is the first thing that breaks. In June 2025, the cs.LG machine learning
category on arXiv alone listed 4,223 entries — about 141 papers a day, in one subcategory,
before you count cs.CL, cs.CV or cs.AI. A researcher who reads one paper properly every
working day gets through roughly 21 a month: half a percent of cs.LG. Nobody reads this
literature; everybody samples it, and the sample is chosen by what colleagues rebuilt.
Peer review runs alongside, at a scale it was not designed for. NeurIPS 2025 received 21,575 valid main-track submissions and accepted 5,290 (24.52%), up from 9,467 submissions in 2020 — 2.3× in five years. Handling it took 20,518 reviewers, 1,663 area chairs and 199 senior area chairs — almost exactly one reviewer per submission, and about 13 papers per area chair. Because reviewers come from the same pool that submits, the field cannot recruit its way out of growth: each new submission brings roughly one new reviewer, so the ratio holds while the median reviewer gets more junior. The chairs reported reviewers raising scores simply to end rebuttal arguments — a capacity failure, not a character failure.
What actually filters the literature is reproduction: a method survives because a second group ran it and it worked, because it landed in a library people use, or because a lab's next model was built on it. The benchmark number in the abstract is the advertisement; the reimplementation is the review.
Types
The field splits three ways, and the split is now drawn as much by budget as by taste.
Theoretical work asks why methods work: generalization bounds, optimization dynamics, learning theory. It needs a whiteboard and modest compute, which is why it remains healthy in universities.
Empirical scaling work asks what happens at size — scaling laws, pre-training recipes, data mixtures, RLHF and post-training. It needs thousands of GPUs and has largely moved inside industry labs.
Applied and systems work takes existing foundation models and makes them do something — inference kernels, agents, retrieval, domain models for biology or code. It sits in both camps, and it is where most published papers actually live.
Real-World Applications
The transformer is the canonical case of the pipeline working. "Attention Is All You Need" (Vaswani et al.) went up as arXiv:1706.03762 in June 2017 and was presented at NIPS that December. Within about eighteen months it had been rebuilt into BERT and GPT, and essentially every large language model in service today descends from it. Note what did the filtering: not the citation count, which arrived years later, but the fact that other groups could reimplement the attention mechanism from the paper and found it trained faster than the recurrent networks it replaced.
AlphaFold shows the same route, ending in a scientific instrument. DeepMind entered CASP14 in 2020 and scored a median domain GDT_TS of 92.4, an accuracy no method had reached in the competition's history. That protein folding result became infrastructure: the AlphaFold Protein Structure Database held 214 million predicted structures by 2024, against roughly 230,000 experimentally determined structures accumulated in the Protein Data Bank since 1971 — a thousand-fold expansion of what a biologist can look up, and the reason AI for science stopped being a slogan.
For a practitioner the live decision is which preprints to read. Follow artefacts, not abstracts: has anyone outside the authors' lab reproduced this, is there runnable code, did a serving stack or training recipe adopt it. Conference acceptance — the NeurIPS 2025 best-paper awards, say — is a retrospective marker that arrives months after the decision was needed.
Challenges
Compute asymmetry is the defining economic fact, and it is arithmetic. Meta's Llama 3.1 model card reports 7.0 million H100-80GB GPU-hours for the 70B pre-training run and 30.84 million for the 405B. A well-equipped university group with eight H100s running flat out all year produces 8 × 8,760 = 70,080 GPU-hours. The 70B run is therefore about 100 years of that cluster; the 405B run is roughly 440 years; even a 256-GPU cluster would need about 14 years for it. At rental prices in the low single digits of dollars per GPU-hour, one such run is an eight-figure compute bill, before the ablations that preceded it. Stanford's AI Index reports that industry produced nearly 90% of notable AI models in 2024, up from 60% in 2023, while academia remained the leading source of the hundred most-cited publications. Both halves of that sentence describe one constraint: universities kept the ideas and lost the FLOPs.
Much of the literature cannot be independently checked. A result that depends on an undisclosed data mixture, an unreported learning-rate schedule or one lab's serving harness is not reproducible in any meaningful sense — you can read the paper and still be unable to test it.
Benchmark contamination makes headline numbers untrustworthy by default. Test sets live on the public web; pre-training corpora are scraped from the public web. Unless a lab deduplicates against the evaluation set and says so, a score partly measures memorization. Scale AI's GSM1k study (arXiv:2405.00332) built a fresh benchmark matched to GSM8K and found some model families dropping by up to 13 percentage points while frontier models barely moved. This is the mechanism behind state-of-the-art claims that do not survive contact with your own data.
Safety pressure changed what gets published. Staged release, withheld weights, capability evaluations reported without the system that produced them, and model cards in place of methods sections are now normal at the frontier. That is a defensible answer to real AI safety concerns and it also puts the claims beyond outside verification — the exact tension AI governance has to price.
Future Trends
Evaluation is becoming a first-class research output, not a table in someone's appendix. When benchmarks saturate within months of release, building one that resists contamination is a harder contribution than another point on a leaderboard — and it is work an academic group can still afford.
AI is entering its own research loop. Systems now draft literature reviews, propose ablations and produce reviewer-style feedback; Stanford's agentic paper reviewer reports a Spearman correlation of 0.42 with human reviewers, against 0.41 between two humans. That number cuts both ways: the machine is about as good as a reviewer, and two reviewers agree barely more than that. The uncomfortable version is self-improving AI — if models generate and screen research directions faster than people can read them, the bottleneck moves from ideas to verification, the part already overloaded.
Reproducibility is being rebuilt as infrastructure rather than etiquette. Artefact submission, frozen evaluation harnesses, open weights and shared training logs buy more verifiability than any review process at 21,575 submissions can, and they are the only correction that scales with the field's growth.