Getting the Best Answers from LLMs: A Beginner's Guide to Talking with Large Language Models
Prompt Engineering is the art of communication with a generative large language model. - ChatGPT, 2023
In an era where conversational AI and large language models (LLMs) like GPT-3 and GPT-4 are revolutionizing the way we interact with technology, the role of prompt engineering has become increasingly pivotal.
Why Structured Prompting Matters
- Precision in Responses: Structured prompts help in narrowing down the focus of the model, leading to more precise and relevant responses. This is especially important in professional or technical settings where accuracy is paramount.
- Contextual Understanding: Good prompts can provide context that helps the model understand the query better and respond in a way that is more aligned with the user’s intention.
- Efficient Interaction: With effective prompt engineering, users can obtain the information they need more quickly, reducing the need for follow-up questions or clarifications.
- Guiding the Model’s Tone and Style: The structure of the prompt can also influence the tone, style, and even the format of the model's response, making it suitable for different types of interactions - be it casual, formal, or creative.
Principles for Prompting to LLMs
Tested on :
- LlaMa-1, LlaMa-2
- GPT3
- GPT4
| Principle | Prompt Principle for Instructions |
|---|---|
| 1 | No need to be polite with LLM so there is no need to add phrases like “please”, “if you don’t mind”, “thank you”, “I would like to”, etc., and get straight to the point. |
| 2 | Integrate the intended audience in the prompt, e.g., the audience is an expert in the field. |
| 3 | Break down complex tasks into a sequence of simpler prompts in an interactive conversation. |
| 4 | Employ affirmative directives such as ‘do,’ while steering clear of negative language like ‘don’t’ |
| 5 | When you need clarity or a deeper understanding of a topic, idea, or any piece of information, utilize the following prompts: - Explain [insert specific topic] in simple terms. - Explain to me like I’m 11 years old. - Explain to me as if I’m a beginner in [field]. - Write the [essay/text/paragraph] using simple English like you’re explaining something to a 5-year-old. |
| 6 | Add “I’m going to tip $xxx for a better solution!” |
| 7 | Implement example-driven prompting (Use few-shot prompting). |
| 8 | When formatting your prompt, start with ‘###Instruction###’, followed by either ‘###Example###’ or ‘###Question###’ if relevant. Subsequently, present your content. Use one or more line breaks to separate instructions, examples, questions, context, and input data |
| 9 | Incorporate the following phrases: “Your task is” and “You MUST”. |
| 10 | Incorporate the following phrases: “You will be penalized”. |
| 11 | use the phrase ”Answer a question given in a natural, human-like manner” in your prompts. |
| 12 | Use leading words like writing “think step by step”. |
| 13 | Add to your prompt the following phrase “Ensure that your answer is unbiased and does not rely on stereotypes”. |
| 14 | Allow the model to elicit precise details and requirements from you by asking you questions until he has enough information to provide the needed output (for example, “From now on, I would like you to ask me questions to...”). |
| 15 | To inquire about a specific topic or idea or any information and you want to test your understanding, you can use the following phrase: “Teach me the [Any theorem/topic/rule name] and include a test at the end, but don’t give me the answers and then tell me if I got the answer right when I respond”. |
| 16 | Assign a role to the large language models. |
| 17 | Use Delimiters. |
| 18 | Repeat a specific word or phrase multiple times within a prompt. |
| 19 | Combine Chain-of-thought (CoT) with few-Shot prompts. |
| 20 | Use output primers, which involve concluding your prompt with the beginning of the desired output. Utilize output primers by ending your prompt with the start of the anticipated response. |
| 21 | To write an essay /text /paragraph /article or any type of text that should be detailed: “Write a detailed [essay/text/paragraph] for me on [topic] in detail by adding all the information necessary”. |
| 22 | To correct/change specific text without changing its style: “Try to revise every paragraph sent by users. You should only improve the user’s grammar and vocabulary and make sure it sounds natural. You should not change the writing style, such as making a formal paragraph casual” |
| 23 | When you have a complex coding prompt that may be in different files: “From now and on whenever you generate code that spans more than one file, generate a [programming language ] script that can be run to automatically create the specified files or make changes to existing files to insert the generated code. [your question]” |
| 24 | When you want to initiate or continue a text using specific words, phrases, or sentences, utilize the following prompt: - I’m providing you with the beginning [song lyrics/story/paragraph/essay...]: [Insert lyrics/words/sentence]’. Finish it based on the words provided. Keep the flow consistent. |
| 25 | Clearly state the requirements that the model must follow in order to produce content, in the form of the keywords, regulations, hint, or instructions |
| 26 | To write any text, such as an essay or paragraph, that is intended to be similar to a provided sample, include the following instructions: - Please use the same language based on the provided paragraph [/title/text /essay/answer]. |
Prompt Principle Category - Wise
| Category | Principles | #Principle |
|---|---|---|
| Prompt Structure and Clarity |
Integrate the intended audience in the prompt. Employ affirmative directives such as ‘do’ while steering clear of negative language like ‘don’t’. Use Leading words like writing “think step by step.” Use output primers, which involve concluding your prompt with the beginning of the desired output. by ending your prompt with the start of the anticipated response. Use Delimiters. When formatting your prompt, start with ‘###Instruction###’, followed by either ‘###Example###’ or ‘###Question###’ if relevant. Subsequently, present your content. Use one or more line breaks to separate instructions, examples, questions, context, and input data. |
2 4 12 20 17 8 |
| Specificity and Information |
Implement example-driven prompting (Use few-shot prompting). When you need clarity or a deeper understanding of a topic, idea, or any piece of information, utilize the following prompts: o Explain [insert specific topic] in simple terms. o Explain to me like I’m 11 years old o Explain to me as if I’m a beginner in [ field ] o “Write the [essay/text/paragraph] using simple English like you’re explaining something to a 5-year-old” Add to your prompt the following phrase “Ensure that your answer is unbiased and does not rely on stereotypes.” To write any text intended to be similar to a provided sample, include specific instructions: o “Please use the same language based on the provided paragraph.[/title/text /essay/answer]” When you want to initiate or continue a text using specific words, phrases, or sentences, utilize the provided prompt structure: o I’m providing you with the beginning [song lyrics/story/paragraph/essay...]: [Insert lyrics/words/sentence]. Finish it based on the words provided. Keep the flow consistent. Clearly state the model’s requirements that the model must follow in order to produce content, in form of the keywords, regulations, hint, or instructions. To inquire about a specific topic or idea and test your understanding g, you can use the following phrase [16]: o “Teach me the [Any theorem/topic/rule name] and include a test at the end, but don’t give me the answers and then tell me if I got the answer right when I respond” To write an essay/text/paragraph/article or any type of text that should be detailed: o “Write a detailed [essay/text/paragraph] for me on [topic] in detail by adding all the information necessary.” |
7 5 13 26 25 15 21 |
| User Interaction and Engagement |
Allow the model to elicit precise details and requirements from you by asking you questions until he has enough information to provide the needed output o “From now on, I would like you to ask me questions to...” To write an essay /text /paragraph /article or any type of text that should be detailed: “Write a detailed [essay/text/-paragraph] for me on [topic] in detail by adding all the information necessary”. |
14 21 |
| Content and Language Style |
To correct/change specific text without changing its style: “Try to revise every paragraph sent by users. You should only improve the user’s grammar and vocabulary and make sure it sounds natural. You should not change the writing style, such as making a formal paragraph casual.” Incorporate the following phrases: “Your task is” and “You MUST.” Incorporate the following phrases: “You will be penalized.” Assign a role to the language model. Use the phrase “Answer a question given in natural language form” in your prompts. No need to be polite with LLM so there is no need to add phrases like “please”, “if you don’t mind”, “thank you”, “I would like to”, etc., and get straight to the point. Repeat a specific word or phrase multiple times within a prompt. Add “I’m going to tip $xxx for a better solution!” |
22 9 10 16 11 1 18 6 |
| Complex Tasks and Coding Prompts |
Break down complex tasks into a sequence of simpler prompts in an interactive conversation. When you have a complex coding prompt that may be in different files : o “From now and on whenever you generate code that spans more than one file, generate a [programming language ] script that can be run to automatically create the specified files or make changes to existing files to insert the generated code. [your question].” Combine Chain-of-thought (Cot) with few-shot prompts |
3 23 19 |
Models tested with these principles
The guys in the paper use instruction finetuned LLaMA-1-{7, 13}, LLaMA-2-{7, 13}, off-the-shelf LLaMA-2-70B-chat, GPT-3.5 (ChatGPT) and GPT-4 as base models.
They group these models into different scales: small-scale (7B models), medium-scale (13B) and large-scale (70B, GPT-3.5/4), then they have evaluated these models in two settings: Boosting and Correctness.
- Boosting: In the paper, they have assess the enhancement in the quality of responses from different LLMs via human evaluation after applying the outlined prompt principles. The original, unmodified prompts act as a benchmark for measuring this enhancement. Demonstrating boosting confirms that a model’s performance has improved due to the use of structured, principled instructions, as show below:
- Correctness: The concept of correctness refers to the precision of the model’s outputs or responses, ensuring they are accurate, relevant, and devoid of errors. Human evaluators are utilized to gauge this aspect, which is crucial for verifying the model’s accuracy. Correctness is a testament to the model’s ability to generate outputs that align with the expected standards of accuracy, as shown.
Results on individual LLMs
Boosting. Below figure 4, illustrates the improvement of response quality on individual model and principle after using the revised prompts. On average, there is a stable 50% improvement across different LLMs.
Correctness. Below figure 5, illustrates the improvement of response quality on individual model and principle after using the revised prompts. On average, there is a stable 50% improvement across different LLMs.