| 0 | 0 | 1 |
| 下载次数 | 被引频次 | 阅读次数 |
针对现有深度强化学习算法中存在的探索成本高、训练效率低的问题，文章提出了一种基于两阶段深度强化学习的微电网能量管理策略。首先，在离线阶段，用线性规划获得典型日的最优调度结果，形成专家经验库，进而用模仿学习方法对智能体进行预训练。然后，在在线阶段，智能体与实际环境进行实时交互，学习并优化非典型日的调度策略。另外，在线阶段应用悬崖漫步奖励机制，当智能体探索超出约束范围时立刻停止探索，减少了无效探索带来的训练成本。仿真结果表明，与单阶段深度强化学习算法相比，两阶段深度强化学习策略显著提升了智能体的训练效率和优化性能。
Abstract：[Objective] With the gradual transformation of the global energy structure and the rapid development of renewable energy technologies, microgrid technology has emerged as an important and rapidly growing area in the energy sector. As the core of microgrid operation and control, the energy management system ensures the efficient use of renewable energy and the stable operation of microgrids through precise monitoring and intelligent control. Traditional energy management methods struggle to effectively handle the complex interrelationships among variables in microgrids, whereas deep reinforcement learning(DRL) enables intelligent decision-making by learning through interaction with the environment and adjusting strategies based on feedback signals. To address the high exploration cost and low training efficiency of existing DRL algorithms, this study proposes a microgrid energy management strategy based on a two-stage DRL framework. [Methods] The proposed strategy includes two stages: offline and online. First, in the offline stage, linear programming is used to obtain the optimal scheduling results of typical days to construct an expert experience library, and imitation learning is then used to pretrain the agents. This stage involves extracting key information from historical data, such as photovoltaic power, wind power, load demand, and electricity prices, and transforming the data into state–action pairs, thereby forming the pretraining foundation for the agents. Subsequently, in the online stage, the agents interact with the real environment to learn the optimal scheduling strategies for nontypical days. During this stage, having accumulated sufficient knowledge in the offline stage, the agents can significantly improve the environmental tracking accuracy and operational economy. A cliff-walk reward mechanism is introduced to ensure that the agents immediately stop exploring after making decisions that violate constraints, thereby reducing the training cost associated with invalid exploration. Concurrently, the proximal policy optimization(PPO) algorithm is introduced to meet the requirements of continuous action spaces and further improve the performance of the agents. [Results] The proposed algorithm has been validated in a typical microgrid system with three PV stations, one wind turbine, one storage system, and flexible loads. The simulation results show that the convergence speed is significantly improved, and the average daily operating cost is reduced by approximately 14% compared with that of single-stage PPO. A comparison with Double Deep Q-Network, Dueling Deep Q-Network, and Distributed Dueling Deep Q-Network further demonstrates the advantages of the proposed method in achieving optimal performance. [Conclusions] In the two-stage DRL, pretraining in the offline stage enables agents to learn general features and strategies, enabling them to adapt more quickly to new environments and tasks in subsequent missions. Training in the online stage helps agents avoid overfitting to task-specific training data, reducing reliance on such data, lowering the risk of overfitting, and ultimately endowing the strategies with better generalization capability and robustness. Overall, compared with single-stage DRL-based microgrid energy management algorithms, the proposed two-stage DRL-based strategy significantly improves the training efficiency and optimal performance of the agents.
[1]WANG J X, CHEN L D, TAN Z F, et al. Inherent spatiotemporal uncertainty of renewable power in China[J]. Nature Communications,2023, 14(1):5379.
[2]邓叶恒,杨莉,杨晓辉,等.基于机会约束规划的热电联供微电网低碳经济调度[J].实验室研究与探索, 2023, 42(3):92–97.DENG Y H, YANG L, YANG X H, et al. Low-carbon economic dispatch of combined heat and power microgrids based on opportunity constrained programming[J]. Research and Exploration in Laboratory, 2023, 42(3):92–97.(in Chinese)
[3]李霞林,刘一欣,王中冠,等.新型电力系统综合实验平台设计与教学应用[J].实验技术与管理, 2023, 40(4):192–199.LI X L, LIU Y X, WANG Z G, et al. Design and teaching application of comprehensive experimental platform for new electricity power system[J]. Experimental Technology and Management, 2023, 40(4):192–199.(in Chinese)
[4]SHAFIULLAH M, REFAT A M, HAQUE M E, et al. Review of recent developments in microgrid energy management strategies[J]. Sustainability, 2022, 14(22):14794.
[5]齐志远,王旭.基于LabVIEW的氢电综合供应微能源网监控系统[J].实验技术与管理, 2023, 40(6):42–48.QI Z Y, WANG X. Monitoring system of micro-energy grid integrated hydrogen and electricity based on LabVIEW[J].Experimental Technology and Management, 2023, 40(6):42–48.(in Chinese)
[6]黎静华,谢育天,曾鸿宇,等.不确定优化调度研究综述及其在新型电力系统中的应用探讨[J].高电压技术, 2022, 48(9):3447–3464.LI J H, XIE Y T, ZENG H Y, et al. Research review of uncertain optimal scheduling and its application in new-type power systems[J]. High Voltage Engineering, 2022, 48(9):3447–3464.(in Chinese)
[7]杜刚,赵冬梅,刘鑫.计及风电不确定性优化调度研究综述[J].中国电机工程学报, 2023, 43(7):2608–2627.DU G, ZHAO D M, LIU X. Research review on optimal scheduling considering wind power uncertainty[J]. Proceedings of the CSEE, 2023, 43(7):2608–2627.(in Chinese)
[8]林舜江,冯祥勇,梁炜焜,等.新能源电力系统不确定优化调度方法研究现状及展望[J].电力系统自动化, 2024, 48(10):20–41.LIN S J, FENG X Y, LIANG W K, et al. Research status and prospect of uncertain optimal dispatch methods for renewable energy power systems[J]. Automation of Electric Power Systems,2024, 48(10):20–41.(in Chinese)
[9]潘军,卢彦杉,何彬彬,等.计及风、光出力不确定性的微电网经济调度研究[J].电工电能新技术, 2024, 43(2):56–64.PAN J, LU Y S, HE B B, et al. Research on economic dispatch of microgrid considering uncertainty of wind and solar output[J].Advanced Technology of Electrical Engineering and Energy,2024, 43(2):56–64.(in Chinese)
[10]魏斌,乔森,孟润泉,等.基于数据驱动不确定集的微电网两阶段鲁棒优化调度[J].高电压技术, 2025, 51(2):852–863.WEI B, QIAO S, MENG R Q, et al. Two-stage robust optimization scheduling of microgrids based on data-driven uncertain sets[J].High Voltage Engineering, 2025, 51(2):852–863.(in Chinese)
[11]郭方洪,冯秀荣,杨淏,等.基于数据模型双驱动的新能源微电网分布鲁棒优化调度[J].电力系统自动化, 2024, 48(20):36–47.GUO F H, FENG X R, YANG H, et al. Dual-data-model-driven distributionally robust optimal scheduling of renewable energy microgrid[J]. Automation of Electric Power Systems, 2024,48(20):36–47.(in Chinese)
[12]毕聪博,唐聿劼,罗永红,等.电力系统优化控制中强化学习方法应用及挑战[J].中国电机工程学报, 2024, 44(1):1–22.BI C B, TANG Y J, LUO Y H, et al. Review on critical problems in reinforcement learning methods applied in power system optimization and control scenarios[J]. Proceedings of the CSEE,2024, 44(1):1–22.(in Chinese)
[13]张有兵,林一航,黄冠弘,等.深度强化学习在微电网系统调控中的应用综述[J].电网技术, 2023, 47(7):2774–2788.ZHANG Y B, LIN Y H, HUANG G H, et al. Review on applications of deep reinforcement learning in regulation of microgrid systems[J]. Power System Technology, 2023, 47(7):2774–2788.(in Chinese)
[14]黎海涛,申保晨,杨艳红,等.基于改进竞争深度Q网络算法的微电网能量管理与优化策略[J].电力系统自动化, 2022,46(7):42–49.LI H T, SHEN B C, YANG Y H, et al. Energy management and optimization strategy for microgrid based on improved dueling deep Q network algorithm[J]. Automation of Electric Power Systems, 2022, 46(7):42–49.(in Chinese)
[15]徐钰涵,季天瑶,李梦诗.基于深度强化学习的微电网日前日内协调优化调度[J].南方电网技术, 2024, 18(9):106–116.XU Y H, JI T Y, LI M S. Day-ahead and intra-day coordinated optimal scheduling of microgrid based on deep reinforcement learning[J]. Southern Power System Technology, 2024, 18(9):106–116.(in Chinese)
[16]樊会丛,段志国,陈志永,等.基于多智能体深度策略梯度的离网型微电网双层优化调度[J].中国电力, 2025, 58(5):11–20, 32.FAN H C, DUAN Z G, CHEN Z Y, et al. Two-layer optimization scheduling for off-grid microgrids based on multi-agent deep policy gradient[J]. Electric Power, 2025, 58(5):11–20, 32.(in Chinese)
[17]GOH H H, HUANG Y F, LIM C S, et al. An assessment of multistage reward function design for deep reinforcement learningbased microgrid energy management[J]. IEEE Transactions on Smart Grid, 2022, 13(6):4300–4311.
[18]HU M J, HAN B H, LYU S L, et al. Reward mechanism design for deep reinforcement learning-based microgrid energy management[C]//2023 6th International Conference on Renewable Energy and Power Engineering(REPE). Beijing:IEEE, 2023:201–205.
[19]JI S Y, ZHANG Z Z, YING S H, et al. Kullback–Leibler divergence metric learning[J]. IEEE Transactions on Cybernetics,2022, 52(4):2047–2058.
[20]杨龙飞,高山,蔡新雷,等.基于拉丁超立方采样的小范围分布式光伏出力时空概率分布生成方法[J].南方电网技术, 2023,17(9):38–48.YANG L F, GAO S, CAI X L, et al. Generating method for spatiotemporal probability distribution of distributed PV output in small range based on Latin hypercube sampling[J]. Southern Power System Technology, 2023, 17(9):38–48.(in Chinese)
(1)https://www.ndrc.gov.cn/xwdt/tzgg/202206/t20220601_1326720.html.
(1)Australian Energy Market Operator.Aggregated price and demand data[DB/OL].[2025-10-04].https://www.aemo.com.au/energy-systems/electricity/national-electricity-market-nem/data-nem/aggregated-data.
基本信息:
DOI：10.16791/j.cnki.sjg.2026.06.005
中图分类号:TP18;TM73
引用信息:
[1]陆玲霞,胡明颉,罗玮烨,等.基于两阶段深度强化学习的微电网能量管理策略[J].实验技术与管理,2026,43(06):37-46.DOI:10.16791/j.cnki.sjg.2026.06.005.
基金信息:
国家自然科学基金联合基金项目(U21A20485); 2026年度浙江省自然科学基金资助项目(LMS26F030001); 浙江省“十四五”第二批本科省级教学改革备案项目(JGBA2024014); 教育部产学合作协同育人项目(2501270945); 2024年度浙江大学本科“AI赋能”示范课程建设项目(202424EE2501M); 浙江大学第四批AI For Education系列实证教学研究项目(BKSY20251104)
2025-12-14
2025
2026-01-12
2026-01-28
2026
1
2026-06-20
2026-06-20