Multi-UCAV Air Combat in Short-Range Maneuver Strategy Generation using Reinforcement Learning and Curriculum Learning

Weiren Kong; Deyun Zhou; Kai Zhang; Zhen Yang; Wansha Yang

doi:10.1109/ICMLA51294.2020.00238

Multi-UCAV Air Combat in Short-Range Maneuver Strategy Generation using Reinforcement Learning and Curriculum Learning

Weiren Kong, Deyun Zhou, Kai Zhang, Zhen Yang, Wansha Yang

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

10 Scopus citations

Abstract

We present an approach for learning a reactive maneuver strategy for a UCAV formation involved in a short-range multi-UCAV air combat engagement. Specifically, we define an efficient state representation, which breaks down the complexity caused by the large state space in a multi-UCAV air combat engagement. Then a parameter sharing dueling deep Q-network (PS-DDQN) algorithm is proposed to train the UCAV formation. The learning reactive maneuver strategy is shared among our UCAVs to encourage cooperative behaviors. In addition, curriculum learning and self-play extend the maneuver strategy to more difficult scenarios. Thus, speeding up the training process and improving the learning effect. Finally, the effectiveness of the algorithm and the intelligence degree of maneuver strategy is verified by the simulation test of convergence and maneuver strategy quality.

Original language	English
Title of host publication	Proceedings - 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020
Editors	M. Arif Wani, Feng Luo, Xiaolin Li, Dejing Dou, Francesco Bonchi
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1174-1181
Number of pages	8
ISBN (Electronic)	9781728184708
DOIs	https://doi.org/10.1109/ICMLA51294.2020.00238
State	Published - Dec 2020
Event	19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020 - Virtual, Miami, United States Duration: 14 Dec 2020 → 17 Dec 2020

Publication series

Name	Proceedings - 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020

Conference

Conference	19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020
Country/Territory	United States
City	Virtual, Miami
Period	14/12/20 → 17/12/20

Keywords

air combat
curriculum learning
Multi-UCAV
reinforcement learning
training simulations

Access to Document

10.1109/ICMLA51294.2020.00238

Cite this

Kong, W., Zhou, D., Zhang, K., Yang, Z., & Yang, W. (2020). Multi-UCAV Air Combat in Short-Range Maneuver Strategy Generation using Reinforcement Learning and Curriculum Learning. In M. A. Wani, F. Luo, X. Li, D. Dou, & F. Bonchi (Eds.), Proceedings - 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020 (pp. 1174-1181). Article 9356234 (Proceedings - 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICMLA51294.2020.00238

Kong, Weiren ; Zhou, Deyun ; Zhang, Kai et al. / Multi-UCAV Air Combat in Short-Range Maneuver Strategy Generation using Reinforcement Learning and Curriculum Learning. Proceedings - 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020. editor / M. Arif Wani ; Feng Luo ; Xiaolin Li ; Dejing Dou ; Francesco Bonchi. Institute of Electrical and Electronics Engineers Inc., 2020. pp. 1174-1181 (Proceedings - 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020).

@inproceedings{c1310bec8d5f4e8d8cc54e1de4c25851,

title = "Multi-UCAV Air Combat in Short-Range Maneuver Strategy Generation using Reinforcement Learning and Curriculum Learning",

abstract = "We present an approach for learning a reactive maneuver strategy for a UCAV formation involved in a short-range multi-UCAV air combat engagement. Specifically, we define an efficient state representation, which breaks down the complexity caused by the large state space in a multi-UCAV air combat engagement. Then a parameter sharing dueling deep Q-network (PS-DDQN) algorithm is proposed to train the UCAV formation. The learning reactive maneuver strategy is shared among our UCAVs to encourage cooperative behaviors. In addition, curriculum learning and self-play extend the maneuver strategy to more difficult scenarios. Thus, speeding up the training process and improving the learning effect. Finally, the effectiveness of the algorithm and the intelligence degree of maneuver strategy is verified by the simulation test of convergence and maneuver strategy quality.",

keywords = "air combat, curriculum learning, Multi-UCAV, reinforcement learning, training simulations",

author = "Weiren Kong and Deyun Zhou and Kai Zhang and Zhen Yang and Wansha Yang",

note = "Publisher Copyright: {\textcopyright} 2020 IEEE.; 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020 ; Conference date: 14-12-2020 Through 17-12-2020",

year = "2020",

month = dec,

doi = "10.1109/ICMLA51294.2020.00238",

language = "英语",

series = "Proceedings - 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1174--1181",

editor = "Wani, {M. Arif} and Feng Luo and Xiaolin Li and Dejing Dou and Francesco Bonchi",

booktitle = "Proceedings - 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020",

}

Kong, W, Zhou, D , Zhang, K , Yang, Z & Yang, W 2020, Multi-UCAV Air Combat in Short-Range Maneuver Strategy Generation using Reinforcement Learning and Curriculum Learning. in MA Wani, F Luo, X Li, D Dou & F Bonchi (eds), Proceedings - 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020., 9356234, Proceedings - 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020, Institute of Electrical and Electronics Engineers Inc., pp. 1174-1181, 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020, Virtual, Miami, United States, 14/12/20. https://doi.org/10.1109/ICMLA51294.2020.00238

Multi-UCAV Air Combat in Short-Range Maneuver Strategy Generation using Reinforcement Learning and Curriculum Learning. / Kong, Weiren; Zhou, Deyun ; Zhang, Kai et al.
Proceedings - 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020. ed. / M. Arif Wani; Feng Luo; Xiaolin Li; Dejing Dou; Francesco Bonchi. Institute of Electrical and Electronics Engineers Inc., 2020. p. 1174-1181 9356234 (Proceedings - 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Multi-UCAV Air Combat in Short-Range Maneuver Strategy Generation using Reinforcement Learning and Curriculum Learning

AU - Kong, Weiren

AU - Zhou, Deyun

AU - Zhang, Kai

AU - Yang, Zhen

AU - Yang, Wansha

PY - 2020/12

Y1 - 2020/12

N2 - We present an approach for learning a reactive maneuver strategy for a UCAV formation involved in a short-range multi-UCAV air combat engagement. Specifically, we define an efficient state representation, which breaks down the complexity caused by the large state space in a multi-UCAV air combat engagement. Then a parameter sharing dueling deep Q-network (PS-DDQN) algorithm is proposed to train the UCAV formation. The learning reactive maneuver strategy is shared among our UCAVs to encourage cooperative behaviors. In addition, curriculum learning and self-play extend the maneuver strategy to more difficult scenarios. Thus, speeding up the training process and improving the learning effect. Finally, the effectiveness of the algorithm and the intelligence degree of maneuver strategy is verified by the simulation test of convergence and maneuver strategy quality.

AB - We present an approach for learning a reactive maneuver strategy for a UCAV formation involved in a short-range multi-UCAV air combat engagement. Specifically, we define an efficient state representation, which breaks down the complexity caused by the large state space in a multi-UCAV air combat engagement. Then a parameter sharing dueling deep Q-network (PS-DDQN) algorithm is proposed to train the UCAV formation. The learning reactive maneuver strategy is shared among our UCAVs to encourage cooperative behaviors. In addition, curriculum learning and self-play extend the maneuver strategy to more difficult scenarios. Thus, speeding up the training process and improving the learning effect. Finally, the effectiveness of the algorithm and the intelligence degree of maneuver strategy is verified by the simulation test of convergence and maneuver strategy quality.

KW - air combat

KW - curriculum learning

KW - Multi-UCAV

KW - reinforcement learning

KW - training simulations

UR - http://www.scopus.com/inward/record.url?scp=85102487176&partnerID=8YFLogxK

U2 - 10.1109/ICMLA51294.2020.00238

DO - 10.1109/ICMLA51294.2020.00238

M3 - 会议稿件

AN - SCOPUS:85102487176

T3 - Proceedings - 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020

SP - 1174

EP - 1181

BT - Proceedings - 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020

A2 - Wani, M. Arif

A2 - Luo, Feng

A2 - Li, Xiaolin

A2 - Dou, Dejing

A2 - Bonchi, Francesco

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020

Y2 - 14 December 2020 through 17 December 2020

ER -

Kong W, Zhou D , Zhang K , Yang Z, Yang W. Multi-UCAV Air Combat in Short-Range Maneuver Strategy Generation using Reinforcement Learning and Curriculum Learning. In Wani MA, Luo F, Li X, Dou D, Bonchi F, editors, Proceedings - 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020. Institute of Electrical and Electronics Engineers Inc. 2020. p. 1174-1181. 9356234. (Proceedings - 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020). doi: 10.1109/ICMLA51294.2020.00238

Multi-UCAV Air Combat in Short-Range Maneuver Strategy Generation using Reinforcement Learning and Curriculum Learning

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this