Industries / Health / Healthcare / Behavioral health / Autism therapy
Measuring progress
Measuring progress is the work of tracking whether a program is helping: recording behavior session by session, scoring skills and everyday abilities on standard checklists, and judging from those numbers whether a change came from the therapy.
6
Milestones achieved, from 1979 to 2022.
2
To watch for: announced, not yet confirmed outside.
2
Open deliverables. Nobody has shipped them.
The road
AchievedWatch forOpen
Achieved
1979
Experts read graphs differently
Behavior analysts usually judge progress by looking at a graph of one person's behavior across sessions before and after a change, a method called visual analysis of single-case data. When A. DeProspero and S. Cohen sent 36 such graphs to 250 reviewers of behavioral journals in 1979, the reviewers' ratings of how clearly each showed an effect agreed only modestly (a mean interrater agreement of 0.61); in 2021 five experts agreed on 75% of 1,024 simulated graphs, and in 2022 five raters agreed on 84% of 198 graphs from clinical data, as often as a machine-learning classifier agreed with them (85%).
Lessonwhether a treatment is judged to have worked can depend on who reads the graph.
pubmed.ncbi.nlm.nih.govpubmed.ncbi.nlm.nih.govpubmed.ncbi.nlm.nih.gov
Credit
A. DeProspero and S. Cohen; M. J. Lanovaz, K. Hranchuk and T. Taylor
2016
A measure built for change
Diagnostic tools such as the ADOS were not designed to show small changes, so a team led by R. Grzadzinski and Catherine Lord built the Brief Observation of Social Communication Change (BOSCC), which scores short videos of a child at play; in 177 videos of 56 young children with little or no speech, its score changed over time where the ADOS score did not. In 2026 a team in France and Belgium with no part in its design scored 414 videos from a trial of 177 preschoolers and found it reliable (internal consistency 0.88 to 0.98) and responsive to the change clinicians saw, though it tracked change on the ADOS only weakly.
Lessonthe French team notes there is still no agreed threshold for how much BOSCC change counts as meaningful.
Credit
R. Grzadzinski, Catherine Lord and colleagues; A. Jay, M.-M. Geoffray and the IDEA Study Group, Le Vinatier hospital, Lyon
2017
A payer demands outcome scores
TRICARE, the US military's health plan, began requiring standard outcome measures in its Autism Care Demonstration, the program that pays for ABA, on 1 January 2017 after Congress directed it to. Providers now submit a parent and teacher rating scale (PDDBI) and a parent stress index every six months and the Vineland Adaptive Behavior Scales and Social Responsiveness Scale every year, and an ABA authorization cannot be processed without them.
Lessonthe Department of Defense reported in 2022 that it had published findings for only one of its three outcome measures, because inconsistent data collection and reporting by providers kept it from analyzing the other two, and in September 2025 a review by the National Academies of Sciences, Engineering, and Medicine, ordered by Congress, recommended stopping the required questionnaires, finding no well-designed evaluation plan that justified the burden they placed on families and providers.
Credit
US Congress; Defense Health Agency
2021
Most-used checklist tested late
In a 2020 survey of 1,428 ABA practitioners worldwide, 76% used the Verbal Behavior Milestones Assessment and Placement Program (VB-MAPP), a skills checklist by Sundberg, now in its 2014 edition, to plan teaching, 45% the ABLLS-R and 34% the Vineland. Kristen Padilla and J. S. Akers wrote in 2021 that research on these tools had been very limited until recently, and found its content moderately to strongly supported by a panel of 13 experts; in a 2024 study of 235 children its milestones score tracked the Vineland, but its "barriers" score showed no link to the Vineland's measures of problem behavior.
Lessona checklist can steer treatment plans for many years before anyone tests what its scores mean.
Credit
Kristen Padilla and J. S. Akers, Baylor University; A. D. Lotfizadeh and colleagues
2021
Harms and conflicts unreported
Of 150 reports on non-drug interventions for young autistic children, 11 mentioned adverse events, Kristen Bottema-Beutel and colleagues found. In a year of autism studies in eight behavioral journals, 84% had at least one author who worked for or consulted to ABA providers and 2% disclosed it; their 2026 repeat found such ties in 93% of 119 studies, still disclosed in 2%. Their definition counts employment as a provider as a conflict of interest, and the lead author discloses consulting for school districts and teaching courses on ABA.
Lessonfamilies and payers cannot weigh a study's results without knowing what went wrong and who stood to gain.
pubmed.ncbi.nlm.nih.govpmc.ncbi.nlm.nih.govpubmed.ncbi.nlm.nih.gov
Credit
Kristen Bottema-Beutel and colleagues, Boston College
2022
Many stop within two years
In a random sample of 334 of the 4,145 children referred for ABA at Kaiser Permanente Southern California, a health system in a state that requires insurers to cover it, 13% never started, 66% were still in ABA after 12 months and 46% after 24 months. Only 28% received a full dose, at least 80% of the hours prescribed, though children with the lowest starting scores still gained a clinically meaningful amount on the Vineland after 24 months.
Lessonpaying for therapy does not mean children receive the amount that was planned, which any comparison of outcomes has to account for.
Credit
Kristen Choi, Karen Coleman and colleagues, Kaiser Permanente Southern California and the University of California, Los Angeles
Watch for
AI-drafted session notes
Rethink Behavioral Health (RethinkBH), which sells record-keeping software for ABA clinics, launched Session Note AI, which drafts each session's note from the data recorded during the session; the company says the notes are accurate, consistent and compliant and cut unbilled hours.
Achieved whensomeone outside the vendor compares a sample of drafted notes with the recorded session data and publishes how often notes add, change or leave out facts.
Claimed by
RethinkBH, part of RethinkFirst, March 2026
Software that times behavior
A. R. Omrani, M. J. Lanovaz and D. Moroni trained machine-learning models on previously published recordings of eight autistic children to measure how long vocal stereotypy, repetitive sounds without a social purpose, lasted; nearly all models matched a human observer's session totals with correlations of 0.90 or more.
Achieved whenanother team runs the models on recordings they did not train on and reports similar agreement, which the authors name as the next step.
Claimed by
Omrani, Lanovaz and Moroni, October 2025
Open
01
AI models read single-case graphs
A public test of whether multimodal AI models, which read images as well as text, judge single-case graphs as well as expert behavior analysts. Give the models the 1,024 simulated graphs from Lanovaz and Hranchuk's 2021 study, ask the question its five experts answered (does the graph show a clear effect), and score every answer against the effect built into the simulation and against the experts.
The briefCredit · unclaimed02
A living harms audit
A public table, updated every month, of each new autism intervention study in the main behavioral and autism journals, recording whether it reported adverse events, how it explained dropouts, and whether authors' ties to service providers were disclosed, coded with the rules Bottema-Beutel and colleagues published. An AI model screens new papers and drafts each row, and a person checks and approves it.
The briefCredit · unclaimed
Open 01 · the brief
AI models read single-case graphs
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A public test of whether multimodal AI models, which read images as well as text, judge single-case graphs as well as expert behavior analysts. Give the models the 1,024 simulated graphs from Lanovaz and Hranchuk's 2021 study, ask the question its five experts answered (does the graph show a clear effect), and score every answer against the effect built into the simulation and against the experts.
Why it is still open
The graphs, the five experts' ratings and the true answers are public on GitHub under an MIT license, but published comparisons so far pit experts against classifiers trained on the numbers, not against general AI models looking at the picture. The one test of a general model found, a conference paper by Isaac Akuoko-Mensah and Li-Ting Chen of the University of Nevada, Reno, at the American Educational Research Association's April 2026 meeting, gave Claude 80 simulated graphs: it found 93% of real effects but called an effect in 33% of graphs that had none, and it was not compared with expert raters or other models, and no data or paper had appeared by November 2026. A PubMed search of the main behavior-analysis journals in November 2026 found no published study of AI models judging graphs.
Built with what exists
The repository's GraphsforBlindVisualAnalysis.pdf, true_values.csv and expert files; Claude, Gemini and GPT models through their APIs; Python with pandas; the conservative dual-criteria method, a rule-based check already coded in the repository, as a baseline.
First step
Download the repository, split the PDF into one image per graph, run the first 100 graphs through one model and compare its answers with true_values.csv.
Done looks like
All 1,024 graphs judged three times by each of at least three models; each model's false-alarm rate (effects called where none was built in) and power (real effects found) set beside the five experts' and the dual-criteria method's; prompts, answers and code published.
Open 02 · the brief
A living harms audit
The first to publish is credited on this line, and the deliverable moves from Open to Achieved.
What to build
A public table, updated every month, of each new autism intervention study in the main behavioral and autism journals, recording whether it reported adverse events, how it explained dropouts, and whether authors' ties to service providers were disclosed, coded with the rules Bottema-Beutel and colleagues published. An AI model screens new papers and drafts each row, and a person checks and approves it.
Why it is still open
The audits so far are snapshots: 150 reports in the 2021 harms study, and one year of journals each in the 2021 and 2026 conflict-of-interest studies. A Europe PMC title search in November 2026 found no living review of harms reporting in autism intervention research.
Built with what exists
The Europe PMC and Crossref APIs and PubMed E-utilities; the published coding rules; an AI model for first-pass extraction, such as the open-weight Mistral Large 4 (see the Note "A trillion open parameters"); a SQLite database; Cohen's kappa, a standard measure of agreement between two coders; a static site on GitHub Pages with the data on Zenodo.
First step
Code 20 recent autism intervention studies from the Journal of Applied Behavior Analysis and Behavior Analysis in Practice by the published rules, and count how often an AI model's draft matches the person's coding.
Done looks like
Twelve monthly updates in a row; two people code 50 random studies independently with a kappa of at least 0.8, and the model's draft rows match their agreed coding at least 90% of the time; yearly reporting rates published as a chart.