Weakly-supervised multi-action offline reinforcement learning for intelligent dosing of epilepsy in children

Abstract

Epilepsy in childhood is a common neurological disorder in children. Most cases are benign childhood epilepsy, which can be controlled with medication by adaptive adjustment of the dosage of antiepileptic drugs (AEDs). Recently, reinforcement learning-based intelligent dosing has attracted increasing attention. In clinical practice, patients usually take more than one drug at a time, however, conventional reinforcement learning algorithms do not sufficiently address the combination of two or more active drugs. In this paper, we propose the multi-action offline reinforcement learning (MA-ORL) model to solve this problem. Concretely, MA-ORL inherits the basic framework of the actor-critic network. For the patient’s health and safety concerns, MA-ORL abandons the intrusive and high-risk “trial-and-error” interactions with the environment but directly learns from the offline clinical dataset in the initial phase until the model is sufficiently trained and ready to use. Besides, to choose multiple actions simultaneously, we replace the actor’s output in standard reinforcement learning with a 2D matrix indicating the mixed feature representation of all different actions, then multiply it with separate masks to obtain separated actions. In addition to reaching an optimal return (i.e., the reduction of seizure frequency), MA-ORL also emphasizes the accuracy of the recommended dosage. Therefore, we introduce weak supervision to the learning objective to restrict the range of the learning outcome. It guarantees the recommended dosage by MA-ORL is as close as the one prescribed by experienced physicians. We conduct extensive experiments on clinical medical records containing 245 cases of epilepsy in childhood. Experimental results show MA-ORL reaches the highest cumulative return among all baseline models. Moreover, the suggested amount of medication taken a day by MA-ORL is more accurate than any other benchmark.