What do they capture?: A structural analysis of pre-trained language models for source code

Keyphrases

• Source Code 100%
• Pre-trained Language Model 100%
• Syntax Structure 75%
• Syntax Tree 50%
• Copyright 25%
• Computing Machinery 25%
• Training Model 25%
• Attention Analysis 25%
• Transformer 25%
• Word Embedding 25%
• Code Summarization 25%
• Code Representation 25%
• Code Intelligence 25%
• Tree Induction 25%
• Feature Correlation 25%
• Intermediate Representation 25%
• Code Search 25%
• CodeBERT 25%
• Pre-trained Code Models 25%
• Code Completion 25%
• Model Code 25%
• Transformer Layer 25%
• Language Model of Code 25%

Computer Science

• Source Codes 100%
• Pre-Trained Language Models 100%
• Computing Machinery 33%
• Interpretability 33%
• Word Embedding 33%
• Language Modeling 33%
• Intermediate Representation 33%

Social Sciences

• Language Modeling 100%
• Word Embedding 25%