ChatMolData : a multimodal agent for automatic molecular data processing

Abstract

In recent years, the development of large language models (LLMs) has revolutionized various fields of natural science. However, their application in dealing with various molecular data remains constrained due to the reliance on single-modality inputs and outputs. ChatMolData, a novel LLM-based multimodal agent designed to handle diverse molecular data forms, including molecular databases, images, structure-specific files, and unstructured and structured documents, is introduced. ChatMolData integrates the capabilities of LLMs (e.g., GPT-4 and GPT-3.5) with the robust toolset that supports data retrieval, structuring, prediction, visualization, and search tasks. The agent employs a systematic cycle of reasoning and action to efficiently process complex tasks in molecular science. The evaluation demonstrates that ChatMolData achieves over 90% accuracy for 128 diverse tasks, effectively bridging the gap between experimenters and computational tools. Moreover, it is anticipated that the multimodal-agent strategy provides a pathway to expand data size and improve data accessibility, ultimately promoting molecular research and innovation.