Volume 60

A Data-driven Multi-Step Temporal Difference Pretraining Framework of DeepQ-Network for Boiler Combustion Control Jiaxuan Pu, Jingyu Wu, Yan Jiang, Yaran Wang, Huan Zhang

Abstract

Deep Q-network (DQN) has shown significantpotential in industrial control, yet its high explorationcosts and slow convergence during early deploymenthinder practical adoption. To address this, this studyproposes a data-driven pretraining framework for DQN,based on a multi-step temporal difference (TD)algorithm. Historical operation data and well-designedreward functions are employed to pretrain the agent,significantly reducing post-deployment exploration. Theproposed framework is demonstrated through a casestudy on boiler combustion control system. The agentbuilt on an enhanced recurrent neural network (RNN)architecture regulates coal feeding and air supply tomeet dynamic steam load demands. The effects ofdifferent reward formulations and TD steps onpretraining efficacy are analyzed. Results show thatagents pretrained with linear and nonlinear rewardsboth outperform historical control strategies. The 3-stepTD pretraining strategy outperforms single-step TD,achieving 83.4% consistency with expert actions andrealizing a 5.8% improvement in energy efficiencycompared to manual operation.

Keywords deep reinforcement learning, multi-steptemporal difference algorithm, intelligent control, boilercombustion optimization