Pick one task, make a language model solve it well with carefully designed prompts, then compare at least three prompting techniques and an agent-based approach. The deliverable is a short, honest report.
Prompt engineering involves developing and optimizing prompts to use language models effectively across a wide variety of applications and research topics. These skills help us understand the capabilities and limitations of large language models. Researchers use prompt engineering to improve LLM performance on tasks such as question answering and arithmetic reasoning; developers use it to design prompting techniques that connect LLMs with users and other tools.
Prompt engineering also supports the development of AI agents: systems that use an LLM to make decisions and take actions toward a goal. Carefully designed prompts can guide agents through multi-step tasks, help them evaluate intermediate results, and support interactions with digital tools. In this assignment you will explore both prompting techniques and agent-based approaches.
Teaching team: Minghao Wu, Shunian Chen, Benyou Wang (instructor). Fall 2026 (AY 2026–27, Term 1).
We have prepared a Google Colab notebook to help you get started. It covers:
We strongly recommend reviewing the notebook before starting your experiments. Code and data for Tasks 1 and 2 are in the course repository.
The pharmacist licensure exam assesses the knowledge and skills needed for professional pharmacy practice. LLMs such as DeepSeek can assist with exam preparation by explaining concepts and answering questions, but poorly designed prompts may lead to inaccurate or incomplete answers. Investigate how prompting and agent-based approaches affect performance on pharmacist examination questions.
Input: multiple-choice questions. Output: the selected answer for each question. Criterion: answer accuracy. Code and data: 100 questions sampled from the 2021 National Pharmacist Professional Qualification Examination, in task1 of the course repository.
Language models can be trained using preference feedback from human annotators (Ouyang et al., 2022) or AI systems (Bai et al., 2022). Such feedback can be represented as a triplet of a question, a chosen answer and a rejected answer, and used to train a reward model. Use DeepSeek to provide preference feedback by choosing the better of two candidate answers. Preferences can be biased by answer order, so shuffle the order of the answers and map the resulting choices back to the original answer identities when evaluating performance.
Input: a question Q and two answers A1 and A2. Output: the chosen answer and the rejected answer. Criterion: agreement with the ground-truth preference labels. Code and data: 100 answer pairs sampled from the Anthropic HH-RLHF dataset, in task2 of the course repository.
Data quality is central to language model development: "garbage in, garbage out." Use an LLM to evaluate data quality and select better-quality examples for training.
Input: a dataset D. Output: a cleaner or higher-quality dataset D′. Criteria: define and justify your own quality criteria. Data examples: filter one of the GPT-4-generated instruction datasets from Peng et al. (2023): English Alpaca or Chinese Alpaca.
You may choose another topic that allows you to investigate prompt engineering. The same requirements apply: compare at least three prompting techniques and include a comparison between non-agent and agent-based approaches. Clearly define the task, data and evaluation criteria. A well-motivated original task can earn bonus marks (see below).
Submit a short report using the Overleaf report template. You may adjust the format as needed; keep the report concise and informative. It should cover:
Submit the report as a PDF via Blackboard. Code may be included as supplementary material. Deadline: 11:59 pm, November 6, 2026 (China Standard Time, UTC+8); the late policy applies.
| Component | Marks | What is assessed |
|---|---|---|
| Quality of prompt engineering | 25 | The quality of your prompt designs, their task-specific rationale, and the refinements supported by your experiments. |
| Quality of the report | 75 | 25 for clarity and organization; 25 for experimental results (completeness, experimental rigour, technical insight); 25 for experimental analysis (case studies, sensitivity to individual components, ablation studies and other relevant discussion). |
| Bonus: significance of an original task | up to 5 | A well-motivated research or engineering task beyond Tasks 1–3 may receive a bonus. Choosing a new topic alone does not guarantee bonus marks. |
| Bonus: code quality | up to 5 | Clean, clear and well-documented code may receive a bonus. Attaching code alone does not guarantee bonus marks. |
The final score, including any bonus marks, is capped at 100. Assignment 2 counts for 15% of the course grade. Read the writing tips before drafting: a two-page report with one clear comparison table beats a long one.
Back to: coursework overview · schedule · Assignment 1 · final project guide