Chat2Brain: A Method for Mapping Open-Ended Semantic Queries to Brain Activation Maps

Yaonai Wei, Tianyang Zhong, Songyao Zhang, Xiao Li, Tuo Zhang, Lin Zhao, Zhengliang Liu, Muheng Shang, Tianming Liu, Han Zhang, Chong Ma, Lei Du, Junwei Han

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

Over decades, neuroscience has accumulated a wealth of research results in the text modality that can be used to explore cognitive processes. Meta-analysis is a typical method that successfully establishes a link from text queries to brain activation maps using these research results, but it still relies on an ideal query environment. In practical applications, text queries used for meta-analyses may encounter issues such as semantic redundancy and ambiguity, resulting in an inaccurate mapping to brain images. On the other hand, large language models (LLMs) like ChatGPT have shown great potential in tasks such as context understanding and reasoning, displaying a high degree of consistency with human natural language. Hence, LLMs could improve the connection between text modality and neuroscience, resolving existing challenges of meta-analyses. In this study, we propose a method called Chat2Brain that combines LLMs to basic text-2-image model, known as Text2Brain, to map open-ended semantic queries to brain activation maps in data-scarce and complex query environments. By utilizing the understanding and reasoning capabilities of LLMs, the performance of the mapping model is optimized by transferring text queries to semantic queries. We demonstrate that Chat2Brain can synthesize anatomically plausible neural activation patterns for more complex tasks of text queries.

Original languageEnglish
Title of host publicationProceedings - 2023 2023 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2023
EditorsXingpeng Jiang, Haiying Wang, Reda Alhajj, Xiaohua Hu, Felix Engel, Mufti Mahmud, Nadia Pisanti, Xuefeng Cui, Hong Song
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1523-1530
Number of pages8
ISBN (Electronic)9798350337488
DOIs
StatePublished - 2023
Event2023 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2023 - Istanbul, Turkey
Duration: 5 Dec 20238 Dec 2023

Publication series

NameProceedings - 2023 2023 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2023

Conference

Conference2023 IEEE International Conference on Bioinformatics and Biomedicine, BIBM 2023
Country/TerritoryTurkey
CityIstanbul
Period5/12/238/12/23

Keywords

  • image generation
  • large language model
  • meta-analysis

Fingerprint

Dive into the research topics of 'Chat2Brain: A Method for Mapping Open-Ended Semantic Queries to Brain Activation Maps'. Together they form a unique fingerprint.

Cite this