Constrained human preference alignment for natural language planning with LLMs

Abstract

Recent advances in large language models (LLMs) have established them as promising candidates for natural language planning tasks. However, existing approaches often fail to address two critical challenges: 1) the effective alignment of LLM-generated plans with human preferences, and 2) the dynamic enforcement of diverse constraints inherent in planning scenarios. To bridge these gaps, we propose a constraint-aware human-preference alignment framework for natural language planning. Our contributions are threefold. First, we design a process reward model that aligns LLM outputs with human preferences through step-by-step feedback, facilitating efficient and interpretable preference learning. Second, we develop a constraint-aware mechanism integrated into the rewriting strategy, which dynamically penalizes violations of task-specific constraints at each reasoning step. Third, we introduce a unified adaptive metric enabling a multifaceted assessment of planning quality. We validate our framework through experiments on planning benchmarks, demonstrating improvements in success rate with constraints and human preference alignment over baselines.