Classification of Heads in Multi-head Attention Mechanisms

Feihu Huang; Min Jiang; Fang Liu; Dian Xu; Zimeng Fan; Yonghao Wang

doi:10.1007/978-3-031-10989-8_54

Classification of Heads in Multi-head Attention Mechanisms

Feihu Huang, Min Jiang^*, Fang Liu, Dian Xu, Zimeng Fan, Yonghao Wang

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Citation (SciVal)

Original language	English
Title of host publication	Knowledge Science, Engineering and Management - 15th International Conference, KSEM 2022, Proceedings
Editors	Gerard Memmi, Baijian Yang, Linghe Kong, Tianwei Zhang, Meikang Qiu
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	681-692
Number of pages	12
ISBN (Print)	9783031109881
DOIs	https://doi.org/10.1007/978-3-031-10989-8_54
Publication status	Published (VoR) - 2022
Event	15th International Conference on Knowledge Science, Engineering and Management, KSEM 2022 - Singapore, Singapore Duration: 6 Aug 2022 → 8 Aug 2022

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13370 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	15th International Conference on Knowledge Science, Engineering and Management, KSEM 2022
Country/Territory	Singapore
City	Singapore
Period	6/08/22 → 8/08/22

Keywords

Attention
Classification
Multi-head attention
NLP
Transformer

Access to Document

10.1007/978-3-031-10989-8_54

Cite this

Huang, F., Jiang, M., Liu, F., Xu, D., Fan, Z., & Wang, Y. (2022). Classification of Heads in Multi-head Attention Mechanisms. In G. Memmi, B. Yang, L. Kong, T. Zhang, & M. Qiu (Eds.), Knowledge Science, Engineering and Management - 15th International Conference, KSEM 2022, Proceedings (pp. 681-692). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13370 LNAI). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-10989-8_54

Huang, Feihu ; Jiang, Min ; Liu, Fang et al. / Classification of Heads in Multi-head Attention Mechanisms. Knowledge Science, Engineering and Management - 15th International Conference, KSEM 2022, Proceedings. editor / Gerard Memmi ; Baijian Yang ; Linghe Kong ; Tianwei Zhang ; Meikang Qiu. Springer Science and Business Media Deutschland GmbH, 2022. pp. 681-692 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{dd04f9ed45d143b69eff0d4a09123c77,

title = "Classification of Heads in Multi-head Attention Mechanisms",

keywords = "Attention, Classification, Multi-head attention, NLP, Transformer",

author = "Feihu Huang and Min Jiang and Fang Liu and Dian Xu and Zimeng Fan and Yonghao Wang",

note = "Publisher Copyright: {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 15th International Conference on Knowledge Science, Engineering and Management, KSEM 2022 ; Conference date: 06-08-2022 Through 08-08-2022",

year = "2022",

doi = "10.1007/978-3-031-10989-8_54",

language = "English",

isbn = "9783031109881",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "681--692",

editor = "Gerard Memmi and Baijian Yang and Linghe Kong and Tianwei Zhang and Meikang Qiu",

booktitle = "Knowledge Science, Engineering and Management - 15th International Conference, KSEM 2022, Proceedings",

}

Huang, F, Jiang, M, Liu, F, Xu, D, Fan, Z & Wang, Y 2022, Classification of Heads in Multi-head Attention Mechanisms. in G Memmi, B Yang, L Kong, T Zhang & M Qiu (eds), Knowledge Science, Engineering and Management - 15th International Conference, KSEM 2022, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13370 LNAI, Springer Science and Business Media Deutschland GmbH, pp. 681-692, 15th International Conference on Knowledge Science, Engineering and Management, KSEM 2022, Singapore, Singapore, 6/08/22. https://doi.org/10.1007/978-3-031-10989-8_54

Classification of Heads in Multi-head Attention Mechanisms. / Huang, Feihu; Jiang, Min; Liu, Fang et al.
Knowledge Science, Engineering and Management - 15th International Conference, KSEM 2022, Proceedings. ed. / Gerard Memmi; Baijian Yang; Linghe Kong; Tianwei Zhang; Meikang Qiu. Springer Science and Business Media Deutschland GmbH, 2022. p. 681-692 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13370 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Classification of Heads in Multi-head Attention Mechanisms

AU - Huang, Feihu

AU - Jiang, Min

AU - Liu, Fang

AU - Xu, Dian

AU - Fan, Zimeng

AU - Wang, Yonghao

PY - 2022

Y1 - 2022

KW - Attention

KW - Classification

KW - Multi-head attention

KW - NLP

KW - Transformer

UR - http://www.scopus.com/inward/record.url?scp=85135058090&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85135058090&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-10989-8_54

DO - 10.1007/978-3-031-10989-8_54

M3 - Conference contribution

AN - SCOPUS:85135058090

SN - 9783031109881

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 681

EP - 692

BT - Knowledge Science, Engineering and Management - 15th International Conference, KSEM 2022, Proceedings

A2 - Memmi, Gerard

A2 - Yang, Baijian

A2 - Kong, Linghe

A2 - Zhang, Tianwei

A2 - Qiu, Meikang

PB - Springer Science and Business Media Deutschland GmbH

T2 - 15th International Conference on Knowledge Science, Engineering and Management, KSEM 2022

Y2 - 6 August 2022 through 8 August 2022

ER -

Huang F, Jiang M, Liu F, Xu D, Fan Z, Wang Y. Classification of Heads in Multi-head Attention Mechanisms. In Memmi G, Yang B, Kong L, Zhang T, Qiu M, editors, Knowledge Science, Engineering and Management - 15th International Conference, KSEM 2022, Proceedings. Springer Science and Business Media Deutschland GmbH. 2022. p. 681-692. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-10989-8_54