Pre/Post Concept Test & Survey Instruments for the AI-Assisted Teaching Study (Mixed Design: Quasi-Experimental + Crossover RCT)
Share with Rolf for review and coordination
Measure conceptual understanding of data structures and algorithms at the start and end of the semester. The same test is given twice (pre and post) to both sections. This lets us calculate normalized learning gains and control for any baseline differences between sections.
Format: 20 multiple-choice questions, ~25 minutes. No code writing — pure conceptual reasoning.
Key principle: Questions should test understanding, not memorization. Students who memorized a definition but can't apply it should get the question wrong.
| Topic Area | # Questions | What It Tests |
|---|---|---|
| Algorithm Analysis (Big-O) | 3 | Reasoning about time/space complexity, growth rates |
| Linear Structures (arrays, linked lists, stacks, queues) | 4 | Operation costs, choosing the right structure, tracing operations |
| Recursion | 2 | Tracing recursive calls, understanding base cases |
| Trees (BST, heaps) | 4 | Traversals, insertion/deletion effects, heap properties |
| Hash Tables | 2 | Collision resolution, load factor, when to use hashing |
| Graphs | 3 | Traversal strategies, shortest paths, representation tradeoffs |
| Sorting | 2 | Algorithm behavior, stability, best/worst cases |
Below are ready-to-use questions. Each includes the answer and a rationale explaining why the correct answer is correct and why distractors are tempting.
for (int i = 1; i < n; i = i * 2) {
System.out.println(i);
}
Answer: C — O(log n)
The loop variable doubles each iteration (i = i * 2), so it takes log₂(n) iterations to reach n. Distractor A is tempting because students often assume all single loops are O(n).
Answer: B
Big-O describes asymptotic behavior — it guarantees the relationship holds for sufficiently large n, but not for all n (constant factors matter for small inputs). Distractor D is partially true but overstates the case.
Answer: B — O(log n)
Binary search on a sorted array runs in O(log n). Distractor C (linear search) ignores that the array is sorted. Distractor A would only apply to direct index access.
Answer: B
A singly linked list with a head pointer supports O(1) insert and remove at the front. An ArrayList requires O(n) shifting for front insertions. This tests whether students understand the cost tradeoffs, not just definitions.
Answer: B
Stack is LIFO. After pushing 1-5, the stack top-to-bottom is 5,4,3,2,1. Three pops remove 5, 4, 3. Top is now 2. Tests basic LIFO understanding through tracing.
Answer: B — 3 elements
Elements are at indices 3, 4, 0 (wrapping around). Count = (1 - 3 + 5) % 5 = 3. This is a common point of confusion for students learning circular arrays.
Answer: C
Removing from the back of a singly linked list requires traversing to the second-to-last node (O(n)), since there's no previous pointer. A doubly linked list can follow tail.prev in O(1). Both A, B, and D are O(1) for both structures with head+tail pointers.
int mystery(int n) {
if (n <= 1) return 1;
return n * mystery(n - 1);
}
Answer: C — 120
This computes n! (factorial). 5! = 5 × 4 × 3 × 2 × 1 = 120. Tests ability to trace recursion. Distractor B (15) might come from adding instead of multiplying.
Answer: C
Each recursive call adds a frame to the call stack. Without a base case, calls never stop, and the stack eventually overflows. Distractor B is wrong because the stack is finite. Distractor D is wrong because missing a base case is a runtime error, not a compile error.
8
/ \
3 10
/ \ \
1 6 14
Answer: B — 1, 3, 6, 8, 10, 14
In-order traversal of a BST visits nodes in sorted order (left, root, right). Distractor A is pre-order. Distractor D is level-order. This tests both traversal knowledge and BST properties.
Answer: A — Left child of 6
5 < 8 (go left to 3), 5 > 3 (go right to 6), 5 < 6 (go left — empty slot). Tests BST insertion logic by tracing the path.
Answer: C — 2h+1 - 1
A complete binary tree of height h has 2h+1 - 1 nodes (assuming height of root = 0). Distractor A is the minimum for a "stick" tree. Tests understanding of tree structure bounds.
Answer: A
The heap property (parent ≤ children) is different from the BST property (left < parent < right). Distractor B is wrong — BSTs are not always balanced. Distractor C is wrong — heap search is O(n). Tests understanding of structural vs. ordering properties.
Answer: C — 3
12 % 10 = 2, 22 % 10 = 2, 32 % 10 = 2. All three hash to index 2. With chaining, they form a linked list at that slot. Tests collision understanding.
Answer: A — O(1)
With a good hash function and low load factor, collisions are rare, giving average O(1) lookup. The question specifies "average-case" and "good hash function" to be precise. Distractor C is the worst case (all keys in one bucket).
Answer: B — BFS
BFS explores nodes level by level, so the first time it reaches a node, that's the shortest path (in an unweighted graph). DFS may find a path, but not necessarily the shortest. Dijkstra is for weighted graphs. This is a critical conceptual distinction.
Answer: B — A complete graph
A complete undirected graph with n vertices has n(n-1)/2 edges = 6×5/2 = 15. A tree with 6 vertices would have exactly 5 edges. Tests knowledge of graph properties and formulas.
Answer: C — O(V²)
An adjacency matrix is a V × V 2D array, regardless of how many edges exist. O(V + E) is the space for an adjacency list. Tests understanding of representation tradeoffs.
Answer: B — MergeSort
MergeSort is O(n log n) in all cases (best, average, worst). QuickSort's worst case is O(n²) — a critical distinction that students often miss. Bubble Sort and Insertion Sort are O(n²) worst case.
Answer: C — Insertion Sort
Insertion Sort runs in O(n) on nearly-sorted data because almost no shifting is needed. MergeSort and QuickSort still do O(n log n) regardless. This tests understanding of best-case behavior, not just worst-case.
Measure students' baseline engagement, self-efficacy, and AI perceptions before the semester's instruction takes effect. Administered in both sections in the first few weeks.
Format: 24 Likert-scale items + 4 demographic questions. ~8 minutes to complete.
Scale: 1 = Strongly Disagree, 2 = Disagree, 3 = Neutral, 4 = Agree, 5 = Strongly Agree
D1. What is your major? [CS / CE / IS / Math / Other: ___]
D2. What grade did you receive in the prerequisite programming course? [A / B / C / D / Other: ___]
D3. How many programming courses have you taken before this one? [0 / 1 / 2 / 3+]
D4. How often have you used AI tools (ChatGPT, Claude, Copilot, etc.) for coursework before this class? [Never / A few times / Monthly / Weekly / Daily]
Adapted from the Student Course Engagement Questionnaire (Handelsman et al., 2005)
| # | Statement | SD D N A SA |
|---|---|---|
| E1 | I look forward to attending this class. | 12345 |
| E2 | I actively participate in class discussions and activities. | 12345 |
| E3 | I find the topics in this course interesting. | 12345 |
| E4 | I put effort into understanding the material, not just passing. | 12345 |
| E5 | I feel motivated to study for this course outside of class. | 12345 |
| E6 | When I don't understand something, I seek help (office hours, peers, online). | 12345 |
| E7 | I pay attention during the entire lecture, not just parts of it. | 12345 |
| E8 | I connect what I learn in this class to real-world applications. | 12345 |
Adapted from the Computer Science Self-Efficacy Scale (Ramalingam & Wiedenbeck, 1998)
| # | Statement: "I am confident that I can..." | SD D N A SA |
|---|---|---|
| S1 | ...understand the difference between arrays and linked lists and when to use each. | 12345 |
| S2 | ...analyze the time complexity (Big-O) of a simple algorithm. | 12345 |
| S3 | ...implement a stack or queue in Java. | 12345 |
| S4 | ...trace through a recursive method and predict its output. | 12345 |
| S5 | ...explain how a binary search tree insertion works. | 12345 |
| S6 | ...choose the appropriate data structure for a given problem. | 12345 |
| S7 | ...debug a program that uses data structures like trees or hash tables. | 12345 |
| S8 | ...succeed in this course overall. | 12345 |
Custom items informed by recent AI-in-education literature (Kasneci et al., 2023; Wang et al., 2023)
| # | Statement | SD D N A SA |
|---|---|---|
| A1 | AI tools (like ChatGPT) can help me learn course material more effectively. | 12345 |
| A2 | I trust the accuracy of explanations provided by AI tools. | 12345 |
| A3 | Using AI tools for learning feels like cheating. | 12345 |
| A4 | I would prefer a class that integrates AI tools into instruction over one that does not. | 12345 |
| A5 | AI tools can replace the need for human tutors or teaching assistants. | 12345 |
| A6 | I worry that relying on AI tools will prevent me from truly learning the material. | 12345 |
| A7 | I feel comfortable using AI tools as part of my study routine. | 12345 |
| A8 | AI will play an important role in how computer science is taught in the future. | 12345 |
The post-survey repeats all 24 items from the pre-survey (Sections B, C, D) — this enables pre-post comparison on each construct. Additionally, it includes a short Section E with experience-specific questions.
Demographics (Section A) are not repeated.
These items appear only on the post-survey. For the treatment section, items P3-P6 ask about the web app. For the control section, replace those items with P3c-P6c below.
| # | Statement (Treatment Section) | SD D N A SA |
|---|---|---|
| P1 | This course helped me grow as a computer scientist. | 12345 |
| P2 | I would recommend this course to other students. | 12345 |
| P3 | The animations in the web app helped me understand data structure operations. | 12345 |
| P4 | The interactive exercises in the web app helped me learn actively during lectures. | 12345 |
| P5 | The AI hints helped me work through problems I was stuck on. | 12345 |
| P6 | I wish I could use this web app outside of class for self-study. | 12345 |
| P3c | The lecture examples helped me understand data structure operations. | 12345 |
| P4c | I felt actively engaged during lectures (not just passively listening). | 12345 |
| P5c | When I was stuck on a problem, I had access to helpful resources. | 12345 |
| P6c | I wish the course had more interactive or technology-enhanced learning tools. | 12345 |
O1. What aspect of this course helped you learn the most? Why?
O2. What would you change about how this course is taught?
O3. (Treatment section only) Describe a specific moment when the AI hints helped you (or didn't help you) understand something.
| When | What | Who Administers | Duration |
|---|---|---|---|
| ASAP (this week!) | Pre-Test (concept test, 20 MCQs) | Both instructors, in class | ~25 min |
| ASAP (this week!) | Pre-Survey (demographics + 24 Likert items) | Both instructors, in class | ~8 min |
| Week 8-9 | Optional mid-semester pulse (E1-E8 only) | Both instructors | ~3 min |
| Week 14-15 | Post-Test (same 20 MCQs) | Both instructors, in class | ~25 min |
| Week 14-15 | Post-Survey (24 repeated + 6 new + 3 open) | Both instructors, in class | ~12 min |
| Week 16 | Interviews (treatment section only) | PI (or trained RA) | 20-30 min each |
Generated for CS 205 AI Education Research Study — Spring 2026
Share this document with all collaborators. Last updated: February 2026.