Prompt Engineering Papers
Paper with Code prompt engineering section (opens in a new tab): Provides access to research papers along with the corresponding code.
Overview
-
OpenPrompt: An Open-source Framework for Prompt-learning (opens in a new tab) (2021): Introduces a flexible framework for prompt-based learning that allows prompting and finetuning large PLMs like BERT, GPT-2, T5 etc. Provides implementations for prompt engineering and analysis. (project) (opens in a new tab)
-
Pre-Trained Models: Past, Present and Future (opens in a new tab) (2021): Provides a historical overview and taxonomy of foundation models like BERT. Analyzes tradeoffs in model scaling, data, compute, and transferability.
-
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (opens in a new tab) (2021): Surveys different prompting formulations for NLP including soft prompts, hard prompts, continuous prompts etc. Analyzes prompt tuning objectives and benchmarks performance. (project) (opens in a new tab)
-
Paradigm Shift in Natural Language Processing (opens in a new tab) (2021): Discusses the shift from feature engineering to pretraining large neural models on unlabeled text. Transfer learning has driven progress on many NLP tasks. (project) (opens in a new tab)
Pilot Work
-
Parameter-Efficient Transfer Learning for NLP (opens in a new tab) (2019): Introduces methods to reduce sizes of pretrained models by pruning and distillation to improve parameter efficiency of transfer learning. (project) (opens in a new tab)
-
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (opens in a new tab) (2019): Proposes T5 model pretrained on a unified text-to-text format. Shows strong transfer learning performance on diverse NLP tasks. (project) (opens in a new tab)
-
Language Models as Knowledge Bases? (opens in a new tab) (2019): Investigates using language models like BERT for knowledge base completion and fact retrieval by querying the model. (project) (opens in a new tab)
-
How Can We Know What Language Models Know? (opens in a new tab) (2019): Analyzes methods to probe linguistic knowledge learned by language models, testing capabilities like coreference resolution. (project) (opens in a new tab)
-
Language Models are Few-Shot Learners (opens in a new tab) (2019): Shows LMs can perform well on many NLP tasks with only a small number of training examples. (blog) (opens in a new tab)
-
AdaPrompt: Adaptive Model Training for Prompt-based NLP (opens in a new tab) (2022): Proposes an adaptive prompting method that automatically searches over prompt space during model tuning.
Basics
-
Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference (opens in a new tab) (2020): Uses cloze-style prompts with masked tokens for few-shot text classification and NLI tasks. (project) (opens in a new tab)
-
It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (opens in a new tab) (2020): Shows even small pretrained language models can achieve good performance on few-shot NLP tasks through prompt tuning. (project) (opens in a new tab)
-
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts (opens in a new tab) (2020): Automatically generates prompt templates for querying knowledge from LMs without manual engineering. (website) (opens in a new tab)
-
Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification (opens in a new tab) (2020): Proposes methods to automatically identify words that can serve as class labels for few-shot text classification. (project) (opens in a new tab)
-
Making Pre-trained Language Models Better Few-shot Learners (opens in a new tab) (2021): Introduces new pretraining objectives like masked language modeling to better adapt LMs for few-shot fine-tuning. (project) (opens in a new tab)
-
Prefix-tuning: Optimizing continuous prompts for generation (opens in a new tab) (2021): Introduces continuous prompt tuning approach with trainable prefixes appended to text sequences. (project) (opens in a new tab)
-
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm (opens in a new tab) (2021): Provides a programming framework for specifying prompts to perform complex reasoning tasks.
-
Improving and Simplifying Pattern Exploiting Training (opens in a new tab) (2021): Enhances pattern exploiting training with additional supervisory signals and modeling advances.
-
GPT understands, too (opens in a new tab) (2021): Demonstrates that GPT models can perform reasoning tasks when prompted with appropriate formulations. (project) (opens in a new tab)
-
The Power of Scale for Parameter-Efficient Prompt Tuning (opens in a new tab) (2021): Shows that larger pretrained language models better leverage soft prompt tuning across NLP tasks. (project) (opens in a new tab)
-
Learning How to Ask: Querying LMs with Mixtures of Soft Prompts (opens in a new tab) (2021): Proposes prompting with weighted mixtures of soft prompt templates. (project) (opens in a new tab)
-
Factual Probing Is [MASK]: Learning vs. Learning to Recall (opens in a new tab) (2021): Studies how much factual knowledge is retained in the parameters of language models. (project) (opens in a new tab)
-
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models (opens in a new tab) (2021): Achieves strong few-shot performance with simple prompt tuning approaches without complex formulations.
-
WARP: Word-level Adversarial ReProgramming (opens in a new tab) (2021): Adversarially generates prompts to reprogram undesirable behaviors in LMs. (project) (opens in a new tab)
-
PTR: Prompt Tuning with Rules for Text Classification (opens in a new tab) (2021): Incorporates human-provided rules to improve prompting for text classification.
-
NSP-BERT: A Prompt-based Few-Shot Learner Through an Original Pre-training Task--Next Sentence Prediction (opens in a new tab) (2021): Pretrains BERT model using next sentence prediction as a self-supervised task. (project) (opens in a new tab)
-
Finetuned language models are zero-shot learners (opens in a new tab) (2021): Shows finetuned LMs can perform well on unseen tasks with no gradient updates.
-
PPT: Pre-trained Prompt Tuning for Few-shot Learning (opens in a new tab) (2021): Pretrains prompts on masked language modeling as initialization for few-shot tuning.
-
Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners (opens in a new tab) (2021): Makes prompts differentiable end-to-end for gradient-based optimization. (project) (opens in a new tab)
-
Multitask Prompted Training Enables Zero-Shot Task Generalization (opens in a new tab) (2021): Jointly trains prompts across multiple NLP tasks.
-
P-Tuning v2: Prompt Tuning Can Be Comparable to Finetuning Universally Across Scales and Tasks (opens in a new tab) (2021): Shows prompt tuning can approach finetuning performance with large LMs. (project) (opens in a new tab)
-
Black-Box Tuning for Language-Model-as-a-Service (opens in a new tab) (2022): Enables querying LMs without access to gradients or parameters. (project) (opens in a new tab)
-
Black-box Prompt Learning for Pre-trained Language Models (opens in a new tab) (2022): Learns prompts by maximizing probability of target texts.
-
Binding Language Models in Symbolic Languages (opens in a new tab) (2022): Binds parameters of LMs to symbols to induce reasoning. (project) (opens in a new tab) (website) (opens in a new tab)
-
A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT (opens in a new tab) (2023): Curates prompt patterns to enhance prompt engineering.
Analysis
-
What Makes Good In-Context Examples for GPT-3? (opens in a new tab) (2021)
-
How Many Data Points is a Prompt Worth? (opens in a new tab) (2021) (project) (opens in a new tab)
-
Surface Form Competition-Why the Highest Probability Answer Isn’t Always Right (opens in a new tab) (2021) (project) (opens in a new tab)
-
Natural Instructions: Benchmarking Generalization to New Tasks from Natural Language Instructions (opens in a new tab) (2021) (project) (opens in a new tab)
-
Meta-tuning Language Models to Answer Prompts Better (opens in a new tab) (2021)
-
True Few-Shot Learning with Language Models (opens in a new tab) (2021) (project) (opens in a new tab)
-
Do Prompt-Based Models Really Understand the Meaning of their Prompts? (opens in a new tab) (project) (opens in a new tab)
-
Avoiding Inference Heuristics in Few-shot Prompt-based Finetuning (opens in a new tab) (2021)
-
Towards a Unified View of Parameter-Efficient Transfer Learning (opens in a new tab) (2021)
-
Exploring Low-dimensional Intrinsic Task Subspace via Prompt Tuning (opens in a new tab) (2022)
-
Exploring the Universal Vulnerability of Prompt-based Learning Paradigm. Findings of NAACL (opens in a new tab) (2022) (website) (opens in a new tab)
-
Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? (opens in a new tab) (2022) (project) (opens in a new tab)
-
Parameter-Efficient Prompt Tuning Makes Generalized and Calibrated Neural Text Retrievers (opens in a new tab) (2022) (project) (opens in a new tab)
-
Ignore Previous Prompt: Attack Techniques For Language Models (opens in a new tab) (2022) (project) (opens in a new tab)
Improvements
-
Calibrate Before Use: Improving Few-Shot Performance of Language Models (opens in a new tab) (2021) (project) (opens in a new tab)
-
Text Generation with Efficient (Soft) Q-Learning (opens in a new tab) (2021)
-
Noisy Channel Language Model Prompting for Few-Shot Text Classification (opens in a new tab) (2021)
-
Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification (opens in a new tab) (2021) (project) (opens in a new tab)
-
Revisiting Self-Training for Few-Shot Learning of Language Model (opens in a new tab) (2021)
-
LiST: Lite Self-training Makes Efficient Few-shot Learners (opens in a new tab) (2021)
-
Prototypical Verbalizer for Prompt-based Few-shot Tuning (opens in a new tab) (2022) (project) (opens in a new tab)
-
BBTv2: Pure Black-Box Optimization Can Be Comparable to Gradient Descent for Few-Shot Learning (opens in a new tab) (2022) (project) (opens in a new tab)
Specializations
-
GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation (opens in a new tab) (2021)
-
Constrained Language Models Yield Few-Shot Semantic Parsers (opens in a new tab) (2022)
-
PADA: A Prompt-based Autoregressive Approach for Adaptation to Unseen Domains (opens in a new tab) (2021) (project) (opens in a new tab)
-
Prompt-Learning for Fine-grained Entity Typing (opens in a new tab) (2021)
-
KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction (opens in a new tab) (2021) (project) (opens in a new tab)
-
Exploring Prompt-based Few-shot Learning for Grounded Dialog Generation (opens in a new tab) (2021)
-
Template-free Prompt Tuning for Few-shot NER (opens in a new tab) (2021)
-
Learning to Prompt for Vision-Language Models (opens in a new tab) (2021)
-
CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models (opens in a new tab) (2021)
-
Few-Shot Bot: Prompt-Based Learning for Dialogue Systems (opens in a new tab) (2021)
-
Control Prefixes for Text Generation (opens in a new tab) (2021)
-
The Power of Prompt Tuning for Low-Resource Semantic Parsing (opens in a new tab) (2021)
-
UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (opens in a new tab) (2022) (website) (opens in a new tab) (project) (opens in a new tab)
-
Ontology-enhanced Prompt-tuning for Few-shot Learning (opens in a new tab) (2022)
-
Learning to Prompt for Continual Learning (opens in a new tab) (2021) (project) (opens in a new tab)
-
Relation Extraction as Open-book Examination: Retrieval-enhanced Prompt Tuning (opens in a new tab) (2022) (project) (opens in a new tab)
-
Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction (opens in a new tab) (2022) (website) (opens in a new tab)
-
Chain of Thought Prompting Elicits Reasoning in Large Language Models (opens in a new tab) (2022)
-
Self-Consistency Improves Chain of Thought Reasoning in Language Models (opens in a new tab) (2022)
-
Large Language Models are Zero-Shot Reasoners (opens in a new tab) (2022)
-
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models (opens in a new tab) (2022)
-
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations (opens in a new tab) (2022)
-
On the Advance of Making Language Models Better Reasoners (opens in a new tab) (2022)
-
Learning to Compose Soft Prompts for Compositional Zero-Shot Learning (opens in a new tab) (2022) (project) (opens in a new tab)
-
Decoupling Knowledge from Memorization: Retrieval-augmented Prompt Learning (2022) (project) (opens in a new tab)
-
Exploring Length Generalization in Large Language Models (opens in a new tab) (2022)
-
Ask Me Anything: A simple strategy for prompting language models (opens in a new tab) (2022)
-
Measuring And Narrowing The Compositionality Gap In Language Models (opens in a new tab) (2022)
-
RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learning (opens in a new tab) (2022)
-
Reasoning with Language Model Prompting: A Survey (opens in a new tab) (2022)