AutoBERT-Zero: Evolving BERT backbone from scratch

Jiahui GAO, Hang XU, Han SHI, Xiaozhe REN, Leung Ho Philip YU, Xiaodan LIANG, Xin JIANG, Zhenguo LI

Research output: Chapter in Book/Report/Conference proceedingChapters

11 Citations (Scopus)


Transformer-based pre-trained language models like BERT and its variants have recently achieved promising performance in various natural language processing (NLP) tasks. However, the conventional paradigm constructs the backbone by purely stacking the manually designed global self-attention layers, introducing inductive bias and thus leads to sub-optimal. In this work, we make the first attempt to automatically discover novel pre-trained language model (PLM) backbone on a flexible search space containing the most fundamental operations from scratch. Specifically, we propose a well-designed search space which (i) contains primitive math operations in the intra-layer level to explore novel attention structures, and (ii) leverages convolution blocks to be the supplementary for attentions in the inter-layer level to better learn local dependency. To enhance the efficiency for finding promising architectures, we propose an Operation-Priority Neural Architecture Search (OP-NAS) algorithm, which optimizes both the search algorithm and evaluation of candidate models. Specifically, we propose Operation-Priority (OP) evolution strategy to facilitate model search via balancing exploration and exploitation. Furthermore, we design a Bi-branch Weight-Sharing (BIWS) training strategy for fast model evaluation. Extensive experiments show that the searched architecture (named AutoBERT-Zero) significantly outperforms BERT and its variants of different model capacities in various downstream tasks, proving the architecture's transfer and scaling abilities. Remarkably, AutoBERT-Zero-base outperforms RoBERTa-base (using much more data) and BERT-large (with much larger model size) by 2.4 and 1.4 higher score on GLUE test set. Copyright © 2022 Association for the Advancement of Artifcial Intelligence ( All rights reserved.
Original languageEnglish
Title of host publicationProceedings of the 36th AAAI Conference on Artificial Intelligence
Place of PublicationUSA
PublisherAAAI press
ISBN (Print)9781577358763
Publication statusPublished - Jun 2022


Gao, J., Xu, H., Shi, H., Ren, X., Yu, P. L. H., Liang, X., . . . Li, Z. (2022). AutoBERT-Zero: Evolving BERT backbone from scratch. In Proceedings of the 36th AAAI Conference on Artificial Intelligence (pp. 10663-10671). USA: AAAI press.


Dive into the research topics of 'AutoBERT-Zero: Evolving BERT backbone from scratch'. Together they form a unique fingerprint.