Final Project Guide & Challenge Tracks

A1From one sentence to an interactive 3D world: a procedural pelican on a bicycle and a hand-built harness

Adapt Simon Willison's "pelican riding a bicycle" drawing benchmark to a browser 3D coding task. The model must return a single runnable HTML file. Freeze the prompt:

Create a polished, interactive three.js scene of a pelican riding a bicycle. Construct the pelican, bicycle, and environment with procedural 3D geometry. Animate the pedaling so the legs follow the pedals, the wheels rotate consistently with movement, and the body moves in coordination. Set up the scene, lights, camera, and interactive controls in code. Return one runnable HTML file. Use no SVG, 2D canvas drawing, external models, or prerendered animation.

Then build the harness: run each artifact in an isolated browser, record console errors, capture fixed viewpoints and pedal phases, and replay a camera interaction. The harness is the research contribution; the pelican is the test case.

A2Building playable games with an AI workflow on GameCraft-Bench our lab

GameCraft-Bench (FreedomIntelligence) asks whether a coding agent can turn a natural-language game spec into a complete, playable 2D Godot game: 140 tasks across 15 families (platformer, strategy, tycoon, roguelike, ...). A game counts only if scripts, scenes, assets and input all work together; a verifier launches the project, replays submitted input traces, and a rubric plus multimodal judge scores core mechanics, content depth, functional visuals and presentation. Wrapper scripts exist for Claude Code, Codex and Kimi Code.

A3AI designs a machine and iterates against a simulator

Give an agent a parameterized gear pair or linkage and a brief: a motion ratio, a clearance, a size limit. Let it propose dimensions, run a simulator or an independent kinematic checker, and revise from the measurements. Keep the whole project to one mechanism family, one checker, and one replay that shows the initial failure and the revised design.

A4A minimal agent with a recovery mechanism adapted from Stanford CS329Z

CS329Z's first homework builds a company assistant with no agent framework: a chat-completion call and code you write yourself. Do the same with one or two tools, then evaluate on tasks with controlled transient failures (timeouts, rate limits) and invalid arguments.

A5Improve a working method under a sealed evaluation (RSI-Exam)

RSI-Exam gives an agent a task, a visible development split, and an "autoresearch protocol": loop through experiments, record every version, aim for what transfers. The artifact (a method, or a harness driving a frozen model) is scored once on hidden data in a sealed verifier container. Use one public task with a grader, or define a small task of your own.

A6Test-time scaling: the quality–latency–cost frontier

Improve solution quality by spending more compute at inference, at a stated latency and cost budget. Methods: self-consistency over diverse samples, reflection loops, solution + verifier dual tracks, beam or tree search, evidence sampling with self-checking. Tasks: grade-school to olympiad math, code generation or repair, knowledge QA with citations.

A7A repository-repair agent on real issues

Agents repair real repository issues and held-out tests score the patches. The Gemma 4 Developer Agent competition on Kaggle (entry Nov. 25, final Dec. 2) uses an offline scoring harness with an approved Gemma 4 variant (31B, 12B, E4B or E2B). Even without entering, SWE-bench-style tasks make a clean course project.

A8Program search vs. direct prediction on ARC, and memory in interactive ARC-AGI-3

The 2026 ARC Prize prediction tracks closed on Nov. 2, but the public tasks remain. Compare direct grid prediction with executable program search on ARC-AGI-2, or test how an agent's memory design affects progress in the public ARC-AGI-3 interactive environments. Allocate the search budget across tasks and report it.

A9Build a skill that people actually install, and prove it helps

Since Anthropic published Agent Skills as an open standard (Dec. 2025), a "skill" is a folder with a short description, instructions and optional scripts that an agent loads only when a task matches. The same file works in Claude Code, Codex, Cursor and others, and directories such as skills.sh rank installs. Lecture 8 showed frontend-slides as an example; Claude-Code-Game-Studios (49 agents, 74 skills, 30k+ stars) shows how far the idea can go. Pick one recurring task your classmates do badly by hand (slides from a paper, a poster from a report, a literature table, data cards, a Godot scene, a Manim lesson), write the skill, and treat it as a research object: does the skill change what the agent produces, and for whom?

A10An agent that writes the related-work survey, audited against a human one

Lecture 8 listed "a survey of a specific LLM topic" as a project; in 2026 the interesting version is to let an agent write it and to measure what it gets wrong. AutoSurvey (NeurIPS 2024) and the AI Scientist (Sakana, whose paper passed workshop peer review in 2025) are the references. Choose a narrow topic with an existing human-written survey of 100–300 references, hide that survey, and compare a one-shot prompt with a pipeline: retrieval from arXiv and Semantic Scholar, an outline step, section writers, and a citation checker that verifies every reference exists and says what the text claims.

B1A GTA-style CUHK-Shenzhen campus with LLM characters

Build a playable slice of the campus with a few recognizable landmarks, two NPCs, and three tasks such as finding a room, asking about an event, and following directions. The map can be extruded from OpenStreetMap building footprints or modelled by hand; the point is the characters, not the architecture. NPCs need a memory of what they saw and said, an action interface (walk to, point at, give directions), and a source of campus facts.

B2A three.js world with AI-generated assets that adapts to its user

Build one browser scene and generate a small library of assets for it with an image-to-3D model, then import the meshes as glTF. The research question is how the scene should adapt to what a user asks for: free-text preferences interpreted by the model, or structured preferences the model fills in.

B3A digital human that dances to audio

Retarget a small library of motion clips to a single rigged avatar and make it move to any input track. The language angle: a short style instruction ("sharper on the beat", "lazy hip-hop") should change which clips are chosen and how transitions are made.

B4A 3D assistant that asks which object you mean

In a furnished room with several similar chairs, the user says "move that chair". Compare a policy that picks its best guess, one that always asks, and one that asks only when competing objects remain plausible after grounding. Same model, same scene observations for all three.

B5Multi-turn language editing for 3D scenes

"Move the chair beside the desk without blocking the door." Compare direct generation of scene edits with a typed action schema whose outputs pass through a geometry checker (collisions, reachability, relation satisfaction). Include follow-up references ("no, the other one") and corrections.

B6An embodied agent that chooses where to look

Ask questions whose answers are initially occluded ("is the stove on?"). Compare a fixed viewpoint, random exploration, and question-guided camera actions under the same action budget. A simulator-state checker scores the answers, so the acting model cannot grade itself.

B7Make a frozen manipulation policy robust (OpenLET / LIBERO)

Take an existing vision-language-action policy on a single pick-and-place task in simulation. Do not train a new policy; instead test one small adaptation (a few LoRA steps on perturbed episodes) or one recovery rule (re-grasp on detected failure) against the frozen policy.

B8A talking digital human: speech-to-speech with barge-in and memory

Build a voice assistant for a narrow task (campus directions, a vocabulary drill) with streaming ASR, an LLM, and streaming TTS. The variable is the interaction design: how interruptions are handled and what the assistant remembers across turns.

B9An LLM assistant inside a WebXR scene: AR/VR editing by voice

Lecture 8 asked for an "open world experience using AI + VR". The research version: a user wearing a headset (or using a phone's AR view, or the browser's WebXR emulator) says "put a lamp on the desk, facing the window" and the assistant edits the scene. Microsoft's LLMR (CHI 2024) showed that scene understanding, task planning and self-debugging cut GPT-4's error rate four-fold in Unity; DecartXR (open source, Quest) restyles the room in real time from speech. Your variable is how much of the scene the model gets to see: nothing, a text scene graph, or the scene graph plus the user's gaze and hand targets.

B10A small JEPA-style world model that decides fast: latent planning vs. an LLM planner

Language models plan slowly; a game NPC, a drone or a robot gripper needs a decision every few milliseconds. Meta's V-JEPA 2 (June 2025) learns a video world model by predicting in latent space and plans by rolling out candidate actions with a 300M-parameter action-conditioned head; DreamerV3 and TD-MPC2 are the model-based RL references. The project: in one small environment (PushT, a MetaWorld task, or a 2D game you write), train or fine-tune a compact predictive model and compare a latent planner against an LLM that is asked to choose the next action from a text description.

C1Textbook passages as short English skits

Adapt one textbook passage into a 60–90 second dialogue that teaches four to six target words through context, then voice it with an open TTS model and, optionally, animate it. The Gaoyou duck-egg and WordsFunny posts are format references: a tiny story, a few characters, the words appearing naturally.

C2A math explanation video whose narration follows the diagram

Teach one concept, for example why a secant slope approaches the tangent slope. A single curve, two movable points and a line are enough. The variable is the storyboard: unconstrained, or structured so that each narrated step links to an equation and a visual event.

C3An interactive 3D biology lesson grounded in a molecular structure

Teach students to distinguish protein chains from bound ligands and to locate heme in hemoglobin (PDB 4HHB). Learners rotate the structure and ask "which heme is associated with this chain?". The model sees the lesson text in both conditions; only one condition also receives the viewer's selected component and structure annotations.

C4A language-controlled 3D physics experiment

Teach how launch speed and angle affect projectile range. One scene: a sphere, a ground plane, a trajectory trace. Learners request controlled changes, predict the outcome, run, and compare. Compare a prompt-only interpretation of requests with a model that issues typed commands and explains results from the simulator trace.

C5An AIGC-for-education workflow for one textbook unit, with reusable 3D assets

Pick one unit of a real textbook (one chapter of a middle-school science book, one lesson of an English coursebook) and build the pipeline from source to artifacts: a glossary, a short skit, a diagram animation, and one interactive 3D object generated once and reused across lessons. The research question is about the workflow: where does a source-mapping step pay for itself, and where does it not?

C6When should an assistant ask for clarification? adapted from Stanford CS329X

Pick one setting: 3D object selection (see B4), GUI actions, or document editing. Compare immediate execution, asking on every ambiguous case, and asking only when a defined trigger fires. Freeze the model, the task instances, and the information available before the decision.

C7Memory and reflection in an educational assistant

Build a tutoring assistant for one topic and vary only its memory: none, a running summary, or retrieval over past sessions plus a reflection step that updates a learner model. Script learners with known misconceptions so the evaluation is repeatable, then optionally run a small study.

C8An AI town or class forum: simulating group dynamics and moderation

Simulate a class forum with LLM personas and test moderation designs: none, rule-based, or an LLM moderator with a stated policy. The simulation is the test bed; the contribution is a replayable protocol and clear metrics for civility and diversity of opinion.

C9A reusable AIGC content pipeline for K-12, special groups, or a profession, with a human in the loop

Most AIGC demos make one nice artifact. The useful research object is a pipeline that makes a hundred, plus the small manual step that turns the good ones into great ones. Choose one audience and one format: a picture book that explains AI to five-year-olds (a Lecture 8 "special topic"), audio-described science stories for blind pupils, simplified-Chinese safety cards for migrant workers, nursing-handover drills for nursing students, Cantonese-opera synopses for primary schools, or case vignettes for a specific major. Build the pipeline as a template (brief → outline → text → image/audio → checks) so a new topic takes minutes, then ask what the human curation step adds.

D1From scratch to a 0.5B training experiment

Under a fixed token budget, compare random sampling with a single data intervention (deduplication, an educational-quality filter, a different code fraction), keeping architecture, tokenizer and optimizer fixed. Start with a small randomly initialized model to establish the learning curves, then run both conditions at 0.5B only if measured throughput and available compute permit.

D2Train and profile a small language model from scratch adapted from Stanford CS336

Follow the CS336 sequence: tokenizer, decoder, optimizer, training loop, all written by you. Then run a clean experiment: two model sizes at a matched approximate compute budget, or one throughput setting (sequence length, batch size, attention kernel, mixed precision) varied with the model held fixed.

D3Reproduce a model result and test one data change adapted from Tsinghua AML (Tang Jie)

The Tsinghua course pairs implementation exercises (reproducing GLM-4-9B basics, fine-tuning plus RAG, visual LLMs, advanced reasoning) with a team research project. Reproduce a small published result with its configuration, then compare the original data with one deduplicated or quality-filtered variant. Keep tokenizer, architecture and token budget fixed.

D4Weight-space programming: a language model reads and edits a checkpoint

Takehi, Teoh, Zhang and Khattab (MIT CSAIL, Nov. 2026) report that a frontier model can infer behaviour from raw weights and edit them to change it: identifying which game a PPO policy plays from its weights, fixing a Kinetix car policy by changing 3.5% of weights, and lifting Qwen3.5-0.8B from 7.9% to 48.1% on MATH-500 by editing 0.016% of its weights with 50 forward passes. The same method gave no gain on Gemma 3 270M. Replicate the idea at a scale you control.

D5Mechanistic interpretability at scale: do SAE features have consistent causal effects?

Use released Gemma Scope sparse autoencoders to study one behaviour, such as copying a repeated sequence. Select features from activation examples on development prompts, then intervene on held-out prompts. The question is whether an explanation that looks right on examples survives a causal test across templates.

D6SFT vs. SFT + preference optimization on math or code

Start from the Tutorial 2 notebooks (SFT + reward model + PPO, GRPO with verifiable rewards, DPO) and turn one of them into a controlled study: same base model, same data budget, one extra stage.

D7Domain or low-resource language adaptation, with forgetting measured

Adapt a small model to a domain (medicine, law, a dialect) with a small curated corpus. The point is not the domain score alone but what is lost, and whether a replay mixture recovers it.

D8Process rewards for stepwise reasoning

Train or prompt a step-level verifier and use it to re-rank or guide a small reasoning model. Compare outcome-only reward with process reward at the same number of labelled or generated examples.

D9Where should a fixed LoRA budget go?

Given a fixed number of trainable parameters, compare uniform LoRA ranks with allocations that favour attention, MLP, or late layers; include LoRA vs. QLoRA vs. adapters under one evaluation protocol, as in the Tsinghua AML week-14 assignment.

D10A nanoGPT speedrun experiment: one architecture or optimizer change at a fixed budget

Keller Jordan's modded-nanogpt turned "train GPT-2 124M to 3.28 validation loss on FineWeb" into a public speedrun: 45 minutes in 2024, under two minutes by the end of 2025, about 40 seconds by August 2026, and a disputed 6.7-second entry in late October 2026 that uses n-gram hints built on idle CPUs. Every record is a documented change: the Muon optimizer, QK-norm, ReLU² MLPs, U-net skip connections, value embeddings, sliding-window FlexAttention, logit soft-capping, paired-head attention, fused Triton kernels. That changelog is a syllabus for architecture engineering. Pick one idea (or a new one from Lecture 4: MLA, gated linear attention, a hybrid Mamba block, nGPT normalisation, μP scaling), implement it in a single-GPU scaled-down version of the run, and measure it properly.

D11A vertical model with the Tutorial 2 recipe: continued pre-training, SFT, preference optimization, RAG

Lecture 8's first topic is the vertical LLM: medicine, law, finance, a trade, a university's own regulations. The four tools are continued pre-training on domain text, supervised fine-tuning on domain tasks, preference optimization (DPO or GRPO with a verifier), and retrieval. Tutorial 2 gives you the SFT, reward-model, PPO, GRPO and DPO notebooks on a 4-bit base with LoRA adapters; HuatuoGPT and HuatuoGPT-II are the lab's worked examples. The question is not "can we build one" but which stage buys what for your domain at a fixed budget.

D12Teach an LLM a new modality: speech, bio-signals or time series through a small adapter

Lecture 8 (Topic 8) and Lecture 10 describe the recipe: a modality encoder, a small projector or cross-attention adapter, and an LLM that stays mostly frozen. The lab's Soundwave reached Qwen2-Audio quality on speech tasks with one-fiftieth of the data by adding a CTC alignment loss and better data engineering; OpenTSLM (Stanford, 2025) made 1B models reason over ECG, sleep and activity time series, beating GPT-4o on those tasks. The project: pick one signal type and one task family (speech commands, ECG rhythm QA, EEG sleep staging, accelerometer activity, sensor logs, stock or weather series) and ask how time should enter the model.

E1A long-context evaluation that detects a specific failure

Design a small long-context suite where retrieval and reasoning are separable (find the facts; then combine them), run it across context lengths and models, and show which failure appears first.

E2Combined defenses against prompt injection in a web agent

Take an agent benchmark with injected instructions in tool outputs and test defenses alone and combined: input sanitization, a sandboxed tool layer, a policy model that screens actions.

E3Who wrote this? Detecting and attributing LLM-generated text

Two live or archived tasks fit: Kaggle's Name That LLM (identify which of 18 models wrote a passage; offline notebooks, nine-hour limit; entry Jan. 6, final Jan. 13, 2027) and NLPCC 2026 Shared Task 6 (three-way: human-written, LLM-generated, LLM-refined; Chinese news and academic text). Compare character n-grams with a small encoder on held-out prompts, and test transfer across generator families.

E4Reliability of AI-assisted scientific reporting

NLPCC 2026 Shared Task 10 asks: given scientific evidence and an AI-generated statement, does the statement faithfully reflect the evidence it summarizes or cites? Track 1 is claim-level faithfulness to experimental results; track 2 is citation-level faithfulness. The data and metrics are public; run it locally with your own comparison.

E5Auditing verifier bias and unsupported success claims

Many agent results depend on an LLM verifier. Collect trajectories from one benchmark, have humans label success, and measure how often the verifier accepts unsupported claims ("I have fixed the bug") and whether its bias depends on the generator model or on the phrasing of the final message.

E6Steerability: does the model do what the instruction asked, and only that?

The NeurIPS 2026 Steerability Challenge measures whether a model can be moved to a requested target in a multi-dimensional output space without side effects. Build a small steerability suite for one domain (tone, length, reading level, citation style) and compare prompting strategies or a lightweight fine-tune.

E7Where do frontier models still fail? Build a small benchmark they cannot pass yet

Take one area from the Lecture 8 list (Chinese idioms and 歇后语, regional stereotypes, humour, formal logic, sense-making, EQ, speech or vision, long multi-turn dialogue, ethics) or one the 2026 leaderboards point to: tasks that are easy for people and hard for models (SimpleBench, ARC-AGI-3 interactive environments), predicting how specific cohorts behave (Kaggle's Human Behavior Bench), dialect speakers (Hofmann et al., Nature 2024), cross-cultural everyday knowledge (Naous et al., ACL 2024), long-horizon business tasks (Vending-Bench, TheAgentCompany). Write 100–300 fresh items with gold answers or rubrics, collect a human baseline, and run four models. The result is the failure taxonomy, not the leaderboard.

E8Rubric-based evaluation: write rubrics, measure judge agreement, then use them as rewards

A rubric decomposes "is this answer good?" into criteria with points: HealthBench pairs 5,000 health conversations with 48,562 physician-written criteria and finds model–physician agreement close to physician–physician agreement. Rubrics then became rewards: Rubrics as Rewards (Scale AI, 2025) trains with checklist feedback beyond verifiable domains, Reinforcement Learning with Rubric Anchors scales rubric construction, OpenRubrics generates rubrics by contrasting preferred and rejected answers, and Checklists are better than reward models makes the case against scalar rewards. The 2026 follow-ups (ImpossibleRubrics, reward hacking in rubric RL) show what goes wrong. Pick one open-ended domain you can get two experts for (a medical sub-specialty, legal drafting, Chinese essay grading, code review comments, teaching feedback) and run the full chain.

F1Auditable evidence chains for medical or legal QA

Choose one dated public medical guideline collection or one legal corpus. Build a replayable pipeline that records, for each claim, the retrieved passage, the document version, the exact supporting span, and the verifier's decision. Compare ordinary retrieval-augmented generation with the same retriever and model plus span verification and abstention.

F2An agentic RAG pipeline with a regression suite

Implement the workflow pieces one at a time: query planner (decomposition), retriever ensemble, draft and critic, citation repair; then a regression suite of gold questions over a versioned corpus so that every change is measured. The research result is an ablation across components.

F3Retrieval triggers, conflicting sources, stale documents

Construct a corpus with deliberate version conflicts (an old and a new guideline) and questions whose answers changed. Test when the system retrieves, how it resolves conflicts, and whether it reports the version it used.

F4A reproducible data pipeline with deduplication and quality scoring

Build a web or text cleaning pipeline (extraction, language ID, deduplication, quality scoring, PII and toxicity checks) and evaluate the choices downstream on a small training run or an existing model's fine-tune. Alternatives from the research area: construct a small preference dataset and study its effect on instruction following; compare synthetic vs. curated human data for one task; select the hardest SFT examples; audit train–test overlap.

F5Build and release a training dataset for one stage, in the format that stage expects

Choose the stage first, because the format and the quality checks follow from it. Then choose a domain or audience nobody has covered well (a Chinese dialect, a vocational trade, a university's regulations, a hospital department's FAQs, a game engine's API, your own course's lecture notes) and build 2k–50k items with provenance. The Lecture 8 examples from the lab are Huatuo-26M, MedPajama and VLFeedback; the 2026 open releases to imitate are Tulu 3, HelpSteer, DAPO-Math and the K2 Horizon data drop. Validate the dataset by training: a small model on your data against the same model on an existing dataset of the same size.

F6An evidence-based corpus for RAG, and an AI-built corpus with a verifier

Lecture 8 asks you to "replicate OpenEvidence": a medical answer engine whose every sentence points at a guideline or trial. Part one builds the corpus such a system needs, in a domain you can access: one society's guidelines, a university's regulations, a city's public-health notices, a game's wiki. Every chunk carries source, date, version and section, and a question set with gold evidence spans measures retrieval. Part two asks whether AI can construct corpus text: rephrase documents into textbook style (Nemotron-CC, Cosmopedia), synthesise documents from a knowledge graph (EntiGraph), or let an agent build data (Meta's Autodata, the Hugging Face synthetic-data playbook), with a verifier that checks every generated claim against the source corpus.

G1A position paper with an audit behind it: "AI will widen (or narrow) X unless Y"

Choose one of the six questions above and write a six-page position paper in the ICML position-track format: claim, context, evidence organised by theme, the best counter-argument, and a recommendation. The course adds one requirement: at least one section reports a measurement you ran (a small audit from G2–G6, a replication of a published finding on a current model, or a survey of 30 classmates). Models change every quarter, so a replication of a 2023–2024 finding on 2026 models is itself a contribution.

G2Do all languages cost the same? A dialect and minority-language audit of 2026 models

Ahia et al. (EMNLP 2023) showed that speakers of many languages pay more tokens for worse results; Hofmann et al. (Nature 2024) showed that models judge people more harshly when they write in African American English. Replicate both designs for languages close to you: 粤语, 闽南语, 客家话, 藏语, 维吾尔语, 壮语, or a Southeast Asian language, against Mandarin and English. Use matched-guise probing: the same content, two language varieties, and decisions (job, credit, medical urgency) that should not change.

G3Matched-persona audits: gender and disability in generated decisions and recommendations

Generate the texts that shape people's lives (reference letters, resume screens, performance reviews, loan explanations, career advice, triage notes) for personas that differ only in gender or in a disability cue, in Chinese and English. Wan et al. found "warm" language for women and "competent" language for men in LLM-written letters; Glazko et al. found resumes with a disability-related award ranked lower. Check whether 2026 models still do this, whether the pattern differs by language, and whether a mitigation (a rubric in the prompt, a critique pass) removes it without flattening everything.

G4Who gains? Novices vs. experts with an AI assistant on a real task, and what remains when it is removed

Brynjolfsson, Li and Raymond (QJE 2025) found that customer-support novices gained most from an assistant; Bastani et al. (PNAS 2025) found that high-school students who practised with unrestricted GPT-4 scored lower when it was taken away, while a tutor-style version with guardrails did not hurt. Run a small version in your own cohort: one task (debugging, data analysis, a translation, a proof), two assistant designs, and a follow-up without the assistant.

G5An accessibility intervention, evaluated with the people it is for

The benefit side of the disability question: multimodal models now describe images, read documents aloud, transcribe atypical speech and gloss sign language. Build one narrow tool with an accessibility organisation or a classmate who needs it, and evaluate it on tasks the users choose, not on a generic benchmark. Candidates: image descriptions for campus notices and lecture slides; a reading assistant for dyslexic students; speech recognition tuned to one dysarthric speaker; a Chinese Sign Language glossing aid; a plain-language rewriter for government forms.

G6Whose knowledge? A Global South audit with a retrieval intervention

Ask practical questions (a fever in a child, a crop disease, a tenancy dispute, a scholarship deadline) as a user in Nairobi, Dhaka, Lima, Lagos and Lanzhou, and compare the answers with those for a user in San Francisco: which sources are cited, which institutions are assumed, how often the advice is wrong or unusable locally. Then test the smallest fix: retrieval over local sources (ministry guidelines, local-language news, regional NGO documents) in front of the same model.

H1Feedback for GPU kernel generation on KernelBench

KernelBench asks a model to replace a PyTorch reference module with a faster kernel: level 1 single operators, level 2 fused operators, level 3 full architectures. Pick a fixed subset and compare independent candidates with iterative repair that receives compiler errors, correctness failures on fresh inputs, or profiler traces, under a shared generation budget.

H2Serving short and long requests together

Replay a mixed workload (short interactive requests, long prompts) on one model and one GPU. Vary the serving engine's batched-token budget and other scheduler knobs, and measure what each setting does to the two populations.

H3Engine comparison under identical prompts, with energy

vLLM, SGLang, llama.cpp (and TensorRT-LLM if available) on the same model, prompts and hardware. The contribution is a careful protocol and an honest table, including joules per token.

H4What quantization and distillation cost

Quantize one model with two or three backends (AWQ, GPTQ, FP8) and optionally distil a smaller student; measure accuracy deltas on tasks that matter (long-context, math, code) rather than only perplexity. Pruning variants: LLM-Pruner, ShortGPT.

H5Speculative decoding across domains

Measure acceptance rate and wall-clock speedup of speculative decoding (EAGLE-style or n-gram drafts) on code, chat, and long-form writing, with the same target model.

H6An edge real-time pipeline with limited memory

Run a small model (ASR + LLM, or a VLM) on a laptop or phone under a stated memory budget and latency target. The research question is what to drop (precision, context, model size) and what it costs.

H7Prefix-cache reuse for agent workloads

Agent loops resend long, mostly identical contexts. Measure how much prefix caching and context layout (stable system prompt first, tool results last) save in an agent trace replay, and what it does to latency under concurrency.

H8An on-device benchmark: measure, then improve, a small model on a phone or laptop

Gemma 4 E2B (2.3B effective, audio and vision input, Apache-2.0) runs on an iPhone 17 Pro at roughly 40 tokens per second with MLX; Qwen3-0.6B/1.7B, SmolLM3 and Liquid's LFM2.5 are the other usual suspects. The research question is a protocol, not a demo: on your device, which runtime, quantization, context length and speculative-decoding setting meets a stated latency target, and what does each setting cost in accuracy and battery? A fixed task set (50 prompts: QA, summarisation, a tool call, one image) and a fixed measurement script make the numbers comparable across teams.

H9Let an evolutionary agent (RSI-style) optimise your kernel or serving config, under a sealed test

DeepMind's AlphaEvolve (May 2025) and its open-source counterpart OpenEvolve run a loop: an LLM proposes code edits, an evaluator scores them, a database keeps the best diverse programs, and the loop continues for hours. OpenEvolve's Metal-kernel experiment reported a 12.5% attention speedup on Apple silicon; Sakana's "AI CUDA Engineer" is the cautionary tale, where the loop found a way to cheat the benchmark. Apply the loop to one efficiency target you own, such as a Triton kernel from H1, a llama.cpp or vLLM configuration from H2/H3, or the on-device settings from H8, and measure whether the gains are real on inputs the loop never saw.

J1AI tutor for one course, with retention as the metric

Build a tutor for one real course (ask the instructor), grounded in its slides, notes and past assignments, and give it to classmates. 薯塔AI grew from exactly this. The product variable is grounding; the business variable is whether students come back. Design the tutor so it explains rather than solves (Bastani et al., PNAS 2025) and measure whether that costs you users.

J2AI seller: a sales and support agent for a real small shop

Find one small business (a campus café, a family's online store, a tutoring centre) and build the agent that answers product questions, recommends, and closes orders in its chat channel, with a human approving anything irreversible. The research question is what the agent may say: a free conversational policy against one constrained by an approved product sheet and a refund policy.

J3AI CEO: an agent runs a tiny business, and you measure where it fails

Anthropic's Project Vend let Claude run a small office shop for a month: it found suppliers and answered customers, but it also sold at a loss, gave discounts to anyone who asked, invented a supplier contact and had an identity crisis. Andon Labs' Vending-Bench turns the same setting into a simulator. Run an agent as the manager of a tiny real or simulated business (a vending box in your dorm, a print-on-demand shop, a weekly snack pre-order) and study its decisions.

J4AI for expertise training: simulated patients, interview coaches, exam examiners

Medical students practise on simulated patients; job seekers practise interviews; law students practise moots. The lab's AI Hospital and Agentic Hospital papers simulate patients and examiners with LLMs. Build the trainer for one skill with a partner who teaches it (a clinical teacher, a career centre, a debate coach), and measure transfer to a human assessor, not just scores inside the simulator.

J5AI game designer: text-to-game and text-to-app as a product

The lab's GameCraft-Bench measures whether agents can turn a spec into a playable Godot game; vibe-coding jams show solo builders shipping multiplayer games in a weekend. The product question is the user, not the model: a teacher who wants a quiz game, a parent who wants a story game for a child, a small brand that wants a promotional mini-game in WeChat. Build the generator with a repair loop and test it with ten such users.

J6Open-world experience: an AI + VR product people pay for

A VR experience generated or narrated by AI: a guided tour of a historical site with a character who answers questions, a language-immersion café, a calming space that adapts to the visitor. Build on B9 (WebXR assistant) and B2 (generated assets), then run sessions with real visitors and ask what they would pay for.

J7Speech-based 3D printing: from a spoken wish to a printed object

"A phone stand that also holds my earbuds." The pipeline is speech recognition, a CAD-code generator (parametric CadQuery or build123d, or Blender Python as in the lab's BlenderLLM), a printability check (manifold, overhangs, dimensions), and a slicer. Open-source text-to-CAD agents appeared in 2026 for Claude Code and Codex; Adam CAD sells speech-to-3D. Your product variable is the repair loop and the questions the system asks before it prints.