Early detection of autism spectrum disorder risk in children from spontaneous home videos using zero-shot vision-language models

UDC: 
616-72.87+159.974+004.827
DOI: 
10.21668/health.risk/2026.2.12.eng
Authors: 

K.О. Gnidko, А.А. Bylinskaya

Organization: 

Scientific Center for Information Technologies and Artificial Intelligence, Sirius University of Science and Technology, 1 Triumfal’nyi drive, Sirius Federal Territory, Krasnodar region, 354340, Russian Federation

Abstract: 

This paper addresses the problem of identifying the risk of autism spectrum disorders (ASD) in children based on the analysis of behavioral patterns in unstructured video data. The problem is inherently complex, weakly formalized, and tradi-tionally requires substantial expert effort, which limits its scalability and applicability in early screening. At present, universal technical approaches enabling automated analysis of such data under limited annotation conditions remain insufficiently developed.

The aim of the study is to test the hypothesis that the ASD risk can be reliably detected in children using spontaneous home video recordings through deep learning methods that do not require additional training on specialized labeled datasets.

We propose a zero-shot approach for recognizing behavioral patterns in video data based on multimodal vision-language models. The method relies on an ensemble of textual prompts, specialized design of behavioral descriptions, and hierarchical aggregation of behavioral markers. The prompt ensemble strategy ensures robust alignment between video data and textual representations by using multiple linguistically diverse formulations for each behavioral category. The proposed hierarchical aggregation scheme accounts for semantic similarity between different types of behavior and improves the robustness of zero-shot classification.

The results demonstrate the proposed approach to be quite applicable for analyzing rare and underrepresented behavioral patterns characteristic of early ASD manifestations. The method is also shown to be able to produce an interpretable temporal profile of behavioral features, enabling not only classification but also temporal localization of behavioral patterns. This extends the applicability of the approach to more detailed analysis of behavioral structure over time.

The scientific novelty of the work lies in the application of the zero-shot paradigm to behavioral pattern analysis in home video data, as well as in the development of a prompt ensemble strategy and hierarchical behavioral marker aggregation. The proposed approach reduces epistemic uncertainty in interpreting behavioral manifestations, decreases reliance on subjective factors, and provides a foundation for scalable and accessible solutions for early ASD risk detection.

Keywords: 
autism spectrum disorders, early detection of autism, motor stereotypies, digital autism diagnosis technologies, Deep Learning, machine learning, zero-shot models, autism screening items, AI-based screening system, home-video protocols, behavioral pattern
Gnidko K.О., Bylinskaya А.А. Early detection of autism spectrum disorder risk in children from spontaneous home videos using Zero-Shot vision-language models. Health Risk Analysis, 2026, no. 2, pp. 125–139. DOI: 10.21668/health.risk/2026.2.12.eng
References: 
  1. Semina I.I., Mukharyamova L.M., Sabirov I.S., Valeeva E.V., Safiullina L.R., Nikitin D.O. The current state of the problem of autism spectrum disorders – some biomedical and socio-humanitarian aspects. Kazan medical journal, 2019, vol. 100, no. 6, pp. 918–929. DOI: 10.17816/KMJ2019-918 (in Russian).
  2. Analiticheskaya spravka o sostoyanii sovremennykh nauchnykh issledovanii rasstroistv autisticheskogo spektra v Rossii [An analytical report on the state of modern scientific research into autism spectrum disorders in Russia]. Autism: Federal Resource Center. Available at: https://autism-frc.ru/ckeditor_assets/attachments/1351/analiticheskaya_s... (March 03, 2026) (in Russian).
  3. Nasledov A., Tkacheva L., Miroshnikov S., Pakhomova Y., Zashchirinskaia O. Development and psychometric vali-dation of a diagnostic method for autism spectrum disorder symptoms in children aged 5–7 years. Vestnik Sankt-Peterburgskogo universiteta. Psikhologiya, 2025, vol. 15, no. 2, pp. 218–239 (in Russian).
  4. Kim D.Y., Do R., Shin Y., Sim H., Kim H., Cho S., Lee G., Park S. [et al.]. Automated AI based identification of au-tism spectrum disorder from home videos. NPJ Digit. Med., 2025, vol. 8, no. 1, pp. 607. DOI: 10.1038/s41746-025-01993-5
  5. Steinhart S., Gilboa Y., Sinvani R.-T., Gefen N. Home videos for remote assessment in children with disabilities: a scoping review. Telemed. J. E Health, 2024, vol. 6, pp. e1629–e1648. DOI: 10.1089/tmj.2023.0437
  6. Nalentzi K., Ioannidis G.S., Bougias H., Bisdas S., Balafouta M., Sgouropoulou C., Klontzas M.E., Marias K., Papa-vasileiou P. Radiomics vs. Deep Learning in Autism Classification Using Brain MRI: A Systematic Review. Appl. Sci., 2025, vol. 15, no. 19, pp. 10551. DOI: 10.3390/app151910551
  7. Leroy G., Andrews J.G., KeAlohi-Preece M., Jaswani A., Song H., Galindo M.K., Rice S.A. Transparent deep learning to identify autism spectrum disorders (ASD) in EHR using clinical notes. J. Am. Med. Inform. Assoc., 2024, vol. 31, no. 6, pp. 1313–1321. DOI: 10.1093/jamia/ocae080
  8. Simeoli R., Rega A., Cerasuolo M., Nappo R., Marocco D. Using Machine Learning for Motion Analysis to Early De-tect Autism Spectrum Disorder: A Systematic Review. Rev. J. Autism Dev. Disord., 2024. vol. 13, pp. 278–297. DOI: 10.1007/s40489-024-00435-4
  9. Proverbio A.M., Calbi M., Manfredi M., Zani A. Comprehending body language and mimics: An ERP and neuroim-aging study on Italian actors and viewers. PLoS One, 2014, vol. 9, no. 3, pp. e91294. DOI: 10.1371/journal.pone.0091294
  10. Aviezer H., Trope Y., Todorov A. Body cues, not facial expressions, discriminate between intense positive and negative emotions. Science, 2012, vol. 338, no. 6111, pp. 1225–1229. DOI: 10.1126/science.1224313
  11. de Gelder B., de Borst A.W., Watson R. The perception of emotion in body expressions. Wiley Interdiscip. Rev. Cogn. Sci., 2015, vol. 6, no. 2, pp. 149–158. DOI: 10.1002/wcs.1335
  12. Reschke P.J., Knothe J.M., Lopez L.D., Walle E.A. Putting “context” in context: The effects of body posture and emotion scene on adult categorizations of disgust facial expressions. Emotion, 2018, vol. 18, no. 1, pp. 153–158. DOI: 10.1037/emo0000350
  13. Yuan T., Wang L., Jiang Y. Cross-channel adaptation reveals shared emotion representation from face and biological motion. Emotion, 2025, vol. 25, no. 1, pp. 158–173. DOI: 10.1037/emo00 01409
  14. Morency L.P. What is multimodal? ICMI '22: Proceedings of the 2022 International Conference on Multimodal Interaction, 2022, pp. 1. DOI: 10.1145/3536221.3565369
  15. Themistocleous C.K., Andreou M., Peristeri E. Autism Detection in Children: Integrating Machine Learning and Natural Language Processing in Narrative Analysis. Behav. Sci. (Basel), 2024, vol. 14, no. 6, pp. 459. DOI: 10.3390/bs14060459
  16. Rathod S., Rathod M. Review on Autism Spectrum Disorder Assessment in Pediatric Neurodevelopment to Identify the Biomarkers: Conference paper. ICT for Intelligent Systems: Proceedings of ICTIS 2025, 2026, vol. 11, pp. 61–72. DOI: 10.1007/978-981-96-8796-1_6
  17. Rajagopalan S.S., Ghosh S. Recent Developments in the Application of Artificial Intelligence and Machine Learning in Early Screening and Diagnosis of Autism. Methods Mol. Biol., 2025, vol. 2952, pp. 233–242. DOI: 10.1007/978-1-0716-4690-8_13
  18. Che Y.M., Zhou H.Y. Difficulties in Emotion Recognition from Body Movements in Autism Spectrum Disorder: A Systematic Review and Meta-Analysis. Review Journal of Autism and Developmental Disorders, 2025. DOI: 10.1007/s40489-025-00524-y
  19. Yurov I.Yu., Vorsanova S.G., Zelenova M.A., Stroganova T.A., Yurov Yu.B. Variomnyi analiz vysokofunktsion-al'nogo autizma (sindroma Aspergera) [Variome analysis of high-functioning autism (Asperger syndrome)]. Rossiiskii vestnik perinatologii i pediatrii, 2015, no. 4, pp. 193–194 (in Russian).
  20. Kaysheva A.L., Kopylov A.T., Galiullin R.A., Anashkina A.S., Archakov A.I., Ivanov Y.D., Yurov I.Y., Vorsanova S.G., Yurov Y.B. Proteomic analysis of blood serum protein profiles in children with autism. Voprosy prakticheskoi pediatrii, 2016, vol. 11, no. 5, pp. 12–17. DOI: 10.20953/1817-7646-2016-5-12-17 (in Russian).
  21. Vorsanova S.B., Yurov Yu.B., Voinova V.Yu., Kolotii A.D., Demidova I.A., Kurinnaya O.S., Zelenova M.A., Yurov I.Yu. Spektr variatsii chisla kopii gena MECP2 v rossiiskoi kogorte detei s narusheniem intellekta, vrozhdennymi porokami razvitiya, ehpilepsiei i autizmom [Spectrum of MECP2 gene copy number variations in a Russian cohort of children with intellectual disabilities, congenital malformations, epilepsy, and autism]. Rossiiskii vestnik perinatologii i pediatrii, 2016, vol. 61, no. 4, pp. 190–191 (in Russian).
  22. Zelenova M.A., Yurov Yu.B., Vasin K.S., Kurinnaya O.S., Vorsanova S.G., Yurov I.Yu. Molekulyarnoe kariotipiro-vanie v gruppe detei s makrotsefaliei, umstvennoi otstalost'yu i/ili autizmom i vrozhdennymi porokami razvitiya [Molecular karyotyping in a group of children with macrocephaly, mental retardation and/or autism and congenital malformations].
    Rossiiskii vestnik perinatologii i pediatrii, 2016, vol. 61, no. 4, pp. 193 (in Russian).
  23. Kolotii A.D., Vorsanova S.G., Yurov Yu.B., Vasin K.S., Kuznetsova S.Yu., Gordeeva M.L., Kravets V.S., Yurov I.Yu. Triplikatsiya dlinnogo plecha khromosomy Y (Yq11.223q11.23) u mal'chika s zaderzhkoi psikhorechevogo razvitiya i autizmom [Triplication of the long arm of chromosome Y (Yq11.223q11.23) in a boy with delayed psychomotor development and autism]. Rossiiskii vestnik perinatologii i pediatrii, 2017, vol. 62, no. 4, pp. 171 (in Russian).
  24. Kurinnaya O.S., Vorsanova S.G., Yurov Y.B., Voinova V.Y., Dorofeeva M.Y., Yurov I.Y. Duplikatsiya dlinnogo plecha khromosomy 1, assotsiirovannaya s autizmom i mikroanomaliyami razvitiya [Duplication of the long arm chromosome 1 associated with autism and developmental microanomalies]. Psikhiatriya, 2015, no. 4, pp. 64a–65 (in Russian).
  25. Zelenova M.A., Vorsanova S.G., Yurov Y.B., Stroganova T.A., Yurov I.Y. Analiz variatsii chisla kopii posledovatel'nostei DNK u 12 mal'chikov s vysokofunktsional'nym autizmom (sindrom Aspergera) [Analysis of copy number variations in 12 boys with high-functioning autism (Asperger’s syndrome)]. Psikhiatriya, 2015, no. 4, pp. 60–61 (in Russian).
  26. Yurov I.Yu., Vorsanova S.G., Zelenova M.A., Vasin K.S., Ratnikov A.M., Yurov Yu.B. Genomika autizma: sov-remennye interpretatsionnye tekhnologii poiska molekulyarnykh mekhanizmov narusheniya psikhiki u detei [Autism Genomics: Modern Interpretive Technologies for Searching for Molecular Mechanisms of Mental Disorders in Children]. Rossiiskii vestnik perinatologii i pediatrii, 2017, vol. 62, no. 4, pp. 173 (in Russian).
  27. Zhukova M.A. Patterns of EEG activity in individuals with autism spectrum disorders. Psikhologicheskaya nauka i obrazovanie, 2016, vol. 21, no. 3, pp. 47–55. DOI: 10.17759/pse.2016210306 (in Russian).
  28. Sorokin A.B., Zotova M.A., Korovina N.Yu. Screening Methods for Identification of the Target Group Autism Spec-trum for Special Education Teachers and Psychologists. Psikhologicheskaya nauka i obrazovanie, 2016, vol. 21, no. 3, pp. 7–15. DOI: 10.17759/ pse.2016210302 (in Russian).
  29. Chereneva E.A., Bogdashina O.B., Casanova F., Li X. Modernization of the Autism Research Ideas and Development of Support Services for People with Autism in Russia: From a Regional Initiative to Globalization of Solutions. Psikholog-icheskaya nauka i obrazovanie, 2016, vol. 21, no. 3, pp. 131–140. DOI: 10.17759/pse.2016210315 (in Russian).
  30. Pereverzeva D.S., Gorbachevskaya N.L., Blagoveсhtchenski E.D. Development of Experimental Protocol for Visual Cognitive Function Evaluation in Children with Autism Spectrum Disorders. Psikhologicheskaya nauka i obrazovanie, 2016, vol. 21, no. 3, pp. 34–46. DOI: 10.17759/pse.2016210305 (in Russian).
  31. Ilchenko N.V. Selection of Psychodiagnostic Tools for Evaluation of Representatives about World by Children with Autism Spectrum Disorders. Defektologiya, 2015, no. 4, pp. 54–62 (in Russian).
  32. Esterbrook S.A., Esterbrook R.L., Orlova E.A., Karpekova T.A. Approach for teaching and therapies ASD children with communication and language difficulties. Psikhologiya obucheniya, 2016, no. 4, pp. 102–115 (in Russian).
  33. Khaustov A.V., Rudneva E.V. Measuring the Level of Socialization in Children with Autism Spectrum Disorder. Psikhologicheskayanauka i obrazovanie, 2016, vol. 21, no. 3, pp. 16–24. DOI: 10.17759/pse.2016210303 (in Russian).
  34. Davydova E.Yu., Sorokin A.B. Life Competencies in the Context of Education Planning in Children with Autism Spec-trum Disorders. Psikhologicheskaya nauka i obrazovanie, 2016, vol. 21, no. 3, pp. 120–130. DOI: 10.17759/pse.2016210314 (in Russian).
  35. Minnigulova A., Karpychev V., Davydova E., Pereverzeva D., Sorokin A., Tyushkevich S., Mamokhina U., Danilina K. [et al.]. Altered thalamotemporal structural connectivity is associated with autistic traits in children with ASD. Behav. Brain Res., 2025, vol. 481, pp. 115414. DOI: 10.1016/j.bbr.2024.115414
  36. Radford A., Kim J.W., Hallacy C., Ramesh A., Goh G., Agarwal S., Sastry G., Askell A. [et al.]. Learning Transferable Visual Models From Natural Language Supervision. arXiv, 2021. DOI: 10.48550/arXiv.2103.00020
  37. Zhai X., Mustafa B., Kolesnikov A., Beyer L. Sigmoid Loss for Language Image Pre-Training. arXiv, 2021. DOI: 10.48550/arXiv.2303.15343
Received: 
09.04.2026
Approved: 
07.05.2026
Accepted for publication: 
26.06.2026

You are here

Home