Creating Tests and Quizzes

[Philip Coates]

Subject: Testing in Formal Education

I was fortunate to be employed to teach a wide range of classroom courses in a wide range of grades - elementary, middle, high school, college, grad student, and adult. In each subject and at each level, good tests need to be designed differently. Chronologically, I started with adjunct level at UCLA and at the New School for Social Research in Manhattan. Then for gifted students, I did middle school (and some elementary as a substitute). Then a mix of high school, community college, and adult 'continuing education'.

Early on, what I taught and thus what I tested was dictated by the school. Later on, and for the last six or seven years**, I selected the curriculum, texts, and therefore all the test questions are based on what I consider important to teach.

**You can figure out which courses those when there is a date indicated on the top of the test.

---------------

Alicia: My Story

Seventh Grade Test

1. Describe a typical experience in a bunker during an action.

2. How did Alicia secure work in the fields?

3. Why did the Ukrainians help the Germans?

4. Who are partisans? Specifically, which partisans did Alicia encounter, and what were their goals?

5. What was the Jewish guilt?

6. How were the Jews recognizable after the war?

7. What was one of Alicia's moral and emotional turning points in her life?

8. What explanations did Alicia get and give about the role of God in the halocaust?

9. How did Alicia first get the idea of her orphanage?

10. Explain your basic understanding of why the Jews and the Arabs disagreed about Palestine, the British role in the supervision of Eretz Israel, and why the irony (define the irony) remained unsolved for so long (until 1948 when Israel was declared a nation by the United Nations).

------------

[Philip Coates]

Here's another example of a test I created in another subject. I was briefly a middle school humanities (history, literature, english, logic) teacher at one of the best schools in California. I never got less sleep and worked harder in my life. Or perhaps learned more. ==>

History Test - 7th Grade

12/15/03, Mr. Coates

(always be as specific as possible in your answers on any test that I give you / write the answers on this sheet / if in doubt give a more detailed answer rather than a too short one)

1. Define: bicameral, franchise, emancipation

2. Who could vote in America after the American Revolution and who could not vote?

3. Where is the western border of the United States after the Revolution?

4. Who is the "father" of the U.S. Constitution more than any other man? Why?

5. What were the major types of internal disputes in the U.S. after the Revolution was won? Explain briefly.

6. Describe how the issues surrounding the western lands were resolved and the agreements or legislation resolving them.

7. What are the branches of government set up under the Constitution? How does this differ from the Articles of Confederation?

8. What is the most important thing that happened between 1783-1789? Defend or argue for your answer. [Write on the Back.]

,,,,,,,,,,,,,,

{There was a lot of space between questions on the sheet for answers. I prefer essay questions to 'multiple choice' or yes/no or simple names and dates and facts type questions. Takes the teacher much more time to grade them though. And I was -constantly- giving them tests and pop quizzes. Which had to be carefully looked over; carefully prepared. That's one reason why middle school level was the hardest job I ever had - actually harder than teaching high school, which in turn was harder than teaching college.}

[Selene]

Bi-cameral?

Hmmm

A camel that humps both ways?

[Philip Coates]

That's funny! I'm going to have to remember that one.

(When I was a kid we liked to spring 'daffynishuns' on each other. The only one I remember is "incongruous --- where our laws are made".)

[Selene]

That's funny! I'm going to have to remember that one.

(When I was a kid we liked to spring 'daffynishuns' on each other. The only one I remember is "incongruous --- where our laws are made".)

Why did Preston go to jail?

Because Preston Foster...

[Philip Coates]

I've gone back and added the question numbers to the first test on literature (somehow my word processing software stripped the numbers out of this.)

I would not have chosen this dark a book (about the holocaust or the like) for a grade level where you are just trying to get them to love literature and reading. I had to teach it and thus had to come up with good questions.

<> How did Alicia secure work in the fields?

<> Why did the Ukrainians help the Germans?

<> Who are partisans? Specifically, which partisans did Alicia encounter, and what were their goals?

In the first question above, it is a test that the reader retained a simple fact or factual episode. The second is a more interpretive question. The third, starts with the ability to give a clear and precise definition and then it requires more detail, a longer answer than the other two.

This is a good example of my testing philosophy which is "broad spectrum" or ask them about everything. And in a variety of ways. Many teachers get in a rut and only give one type of questions which the students learn to predict -- all 'factual', all why questions, only about characters, focused heavily on the events, etc.

"Define: bicameral, franchise, emancipation" -- this is a definitional and a short answer question.

"Who is the "father" of the U.S. Constitution more than any other man? Why?" -- you can get part credit for a simple fact, but to get full credit you have to give an explanation that shows an ability to -synthesize- all the things James Madison did and why he gets the lion's share of the credit.

"What is the most important thing that happened between 1783-1789? Defend or argue for your answer." -- This is ability to argue from history for a point of view and support your view. There is no single right answer.

Again, I was trying to challenge them, to make sure they *really understand history* when they emerge from my course with as broad a "spectrum" of types of questions as possible. And that was the way I had been teaching to keep them on their toes. (And no goddamn multiple choice.)

I've often gotten a grudging, "hard, but fair" reaction to my tests from my students.

[Selene]

Mr. Coates:

Out of curiosity, what was the correct answer to your question below:

2. Who could vote in America after the American Revolution and who could not vote?

Adam

[Selene]

Mr. Coates:

Well, what was the correct answer to this question in your quiz?

Adam

[caroljane]

Mr. Coates:

Well, what was the correct answer to this question in your quiz?

Adam

My guess - men could vote, except for slaves or Indians. There may have been property or birthplace requirements Everybody else (more than half the population) couldn't.

[Philip Coates]

Subject: the Objectivist principle of an intellectual trade

Adam,

The purpose of the thread is not to 'quiz' people or to have them see if they can answer the specific questions.

I'll make a deal with you:

If you want to contribute to the actual purpose of the thread by posting something substantive related to the topic of testing, what constitutes good tests, etc., then I would be more willing to answer "side" questions" of intellectual curiosity.

But I'm not here to be a servant of people who a) have a past pattern of insulting me or ridiculing me on other threads, and b) want me to do work to provide them with answers, but don't want to lift a finger to participate in or discuss the actual issues I'm trying to talk about.

.

.

.

(PS, To speak quite frankly, based on past repeated run-ins, I no longer trust your good faith... I suspect that in your case and that of Jnth, ND, and maybe one other, there might be some "gotcha malice" forthcoming. Like this: "What's the answer to number 16...no that's not right...what's the answer to number 12...I disagree...ha, ha, ha...I caught Phil..what's the answer to 6, to 7, to 8, to 9......".)

[Selene]

Mr. Coates:

My reason for asking you about that question, and only that question, is specific.

I will not ask you about any other questions.

I have chosen, since your return to OL, not to engage in discussions with you, and I will continue with my decision.

This is a simple question about a specific question. This is not a gotcha, nor is it a discussion.

You can chose nor to answer.

Adam

[Philip Coates]

As I suggested: I don't hold positive feelings or benevolence toward you due to past "run ins". If you want to respect my thread enough to make a substantive post, in return I will give my answer to this specific one time question.

[Selene]

As I suggested: I don't hold positive feelings or benevolence toward you due to past "run ins". If you want to respect my thread enough to make a substantive post, in return I will give my answer to this specific one time question.

Mr. Coates:

Ahh, a negoitiation. Sure. What would constitute a "substantive posr post?" Give me the guidelines that would be satisfactory, please.

Adam

donning his negotiation cap

[caroljane]

Could I intervene with my bullhorn here?

I feel a little like the guy in Best of Show who could not negotiate his own son off the garage roof, but here goes.

You are both longtime, passionate, committed Objedtivists with a wealth of battle scars and hard-earned self-knowledge to share.

You are both deeply weird strange guys I happen to like a lot, so could you please not fight bad enough to drive each other away?Is that to much to ask? I don't ask for much, really.. ask anybody.. just a little consideration... a few dollars here and there...total loyalty and dedication...sorry, wrong thread....

[Selene]

Carol:

I have no intention of fighting at all. Tai chi mode with Mr. Coates.

Adam

[caroljane]

Well said, Adam. Phil, if you say that tai kwon do is your preferred highroad, I will be very very disappointed.

I think we are making progress here.

Maybe I will get Fluffy down off the roof after all. It didn't work with Uncle Theo, but frankly his cognitive skills were inferior to Fluffy's Cats have to act as if they have nine lives, but they know that humans only have three..

[Selene]

Mr. Coates:

I am waiting for your criteria for a "substantive post" regarding this thread. I am not going to guess what would be an "acceptable post" by your standards.

Please provide the criteria.

Adam

[Philip Coates]

Adam, post / don't post. Doesn't matter. But just for info., here are some examples of kinds of engaging with the issues (as opposed to ridicule, personal attacks, trivia, etc.) I would hope a thread like this would raise or responses -- or challenges or disagreements -- I'd like to see so I don't think I waste my time:

"You said 'no goddamn multiple choice'. That should not be taken as an absolute because...."

"Would your idea of 'essay questions' rather than multiple choice apply to math tests?"

"At what age level would a book about the holocaust be appropriate in a literature class? Don't you think kids need to learn what the world is really like, instead of Pollyannish stuff?"

"The best and worst types of tests I remember were...."

"I think there is better way to test intelligence or accumulated knowledge than the College Boards and better ways than I.Q. tests. Here is why. / Here is what should be done differently..."

"My history classes where all about names and dates and places and less about ideas. How to balance these and how to test them..." "What percentage of a course which has a very heavy reliance on facts - science, history - should be about learning the facts and events as opposed to learning approaches, methods or the like?"

"Testing is inherently unfair because...."

"I don't like your two tests because..."

"Here is how I would -improve- your test..."

"History classes often trace the battles in a major war. How important is it to cover this, to learn it? I think it is/is not because......"

"Can over-analysis of the kind your test requires of the 'Alica' book kill the enjoyment for many kids?"

"If an English teacher grades a literature test, should he give a student a C- or D or F for atrocious grammar, syntax, spelling, even if he can see that the student completely understands the book at an "A" level?

[Philip Coates]

"Why should Phil, have all the fun: Here's a test I would write -- on Objectivism, on free markets, on living by reason. What do people think? Can they add some more questions?"

[Philip Coates]

"I think that science quiz you posted on another thread could be much improved as a more down to earth test of what average people know. Here is how I would improve question #7, question #18, question #33...."

[Philip Coates]

"Here's the best general knowledge quiz I've seen on the internet over the last several years...."

[Philip Coates]

Here's a test I wrote for another subject I taught:

Logic

06/09/04, Mr. Coates

[when the question has multiple parts, label them a), b), c), etc.]

[answers on separate piece of paper / turn in question sheet separately]

[pen, not pencil]

< from this past week >

. Is either or both of these an example of poor thinking or good thinking? ( label each one 'poor' or 'good')

(a) We decide on an answer we prefer for emotional reasons and then look for reasons which will support only that answer

(b) We come up with several possible answers and then select the one we prefer because it has the best reasons to support it

. Give one argument for basing part of a course grade on attendance. Give one argument against it. (Both arguments should come from those listed in our logic book, but you may shorten them.)

. Name two situations in the real world where you might at some time in the future find it valuable to look at two sides of an issue (and perhaps try to come up with your own list of the pros and cons or arguments on each side).

Consider the following: "I am angry at crime in our society. Without a law requiring life imprisonment the third time someone is convicted of any crime, many repeat criminals will still get released and we will be less safe. Some people believe that innocent people will be given life imprisonment under my proposal. But our courts work well and seldom convict the innocent. Furthermore, people who have been found guilty three times are almost never innocent. And convictions can be appealed and innocent people set free. Therefore, all things considered, there should be a law requiring life imprisonment upon the third conviction of any crime."

. What is the question being argued and what is the writer's position on it (in favor/ against)? What is the major argument given for the writer's position?

. What opposing argument against his position does he mention? What are his arguments against the opposing argument?

< from previous weeks >

. " Our Logic teacher, Mr. Coates can play any sport well. I've seen him doing well at basketball, touch football, volleyball, and soccer." What is the error in this argument?

. What makes an argument valid? [give the definition of a valid argument]

. Choose one: It is possible / probable / not possible that a dinosaur bone will not be found on the grounds of our school today at 4:05 P.M.

. Substitute letters, symbolize, and write the truth table for the following: "My car has an engine or it is pulled by four pink and polka-dotted giraffes."

. If N=Anita is naughty, and L=I love logic tests, write the logical symbolic form for "It is false that if I don't love logic tests then Anita is naughty."

[Brant Gaede]

Mr. Coates:

Out of curiosity, what was the correct answer to your question below:

2. Who could vote in America after the American Revolution and who could not vote?

Adam

That's right! Adam could vote, if he were a landowner.

[Selene]

But just for info., here are some examples of kinds of engaging with the issues...I would hope a thread like this would raise or responses -- or challenges or disagreements -- I'd like to see so I don't think I waste my time:

I'll make a deal with you:

If you want to contribute to the actual purpose of the thread by posting something substantive related to the topic of testing, what constitutes good tests, etc., then I would be more willing to answer "side" questions" of intellectual curiosity.

Mr. Coates:

This will work as a meeting of the minds.

I have always, having been a teacher, interested, sometimes fascinated with creating a "test" for students. Seems that we share that idea.

Tests can be powerful educational tools when employed in an energetic atmosphere of seeking the truth. Tests allow for evaluation of a students progress as well as the retention of the information being imparted by the teacher.

Moreover, a well designed test can serve to motivate a strudent as well as verify their absorption of the material imparted in the course, or, program.

Additionally, a test serves as a verification of how skillfull and successful you, the teacher, have been in communicating the material.

Finally, testing can be a valuable reinforcement for the student. Testing can also be stressful and anxiety producing. All part of the process of learning.

I believe that your test should match what you have taught. I tend to be suspicious of a teacher who subconsciously, or consciously, inserts a series of questions that were not covered by their instruction.

A test should be reliable, valid, fair and balanced. The shorter the test, the less reliable it is. A teacher, to be successful shall consentrate on creating 1) questions that are clear; 2) instructions that are unambiguous and 3) balanced in proportion to the material and time the material was covered in class.

The "type," or "model" of the test is also critical in terms of the expectation of what it shall measure. Multiple choice models test/short answer questions test a mastery of details and specific knowledge, whereas essay models measure the students ability to comprehend, synthesize, integrate and analyze information to new situations.

Using specific semantics in questions triggers different mental behavior. For example, to lead a student to integrate information from different disciplines and/or solve, using creative thinking, problems is an act of measuring synthesis.

We would use words like "design," "devise," "organize" and "create," to name a few in the instructions and the questions.

Am I being clear so far?

Adam

[Philip Coates]

> I believe that your test should match what you have taught. [Adam]

When I was taking courses Peikoff taught to regular classrooms full of college students at Brooklyn Poly, I remember his saying half in jest that the test he would give, the only fair test for us, would be to ask us about -everything- that had been covered.

Not literally possible, but I've taken part of it to heart. I've always tried to "cover the waterfront". Not leave any major issues or areas from the previous week or chapter or term or book untouched.

> The shorter the test, the less reliable it is.

That's an excellent point - another way to put it is if I ask them five questions and they get one wrong, they could not get above a B-. Wherease if I give them 20 questions..... My tests individually are not always long, but I am able to do this by giving a lot more tests as well as pop quizzes than most teachers. (Of course, if you are always testing it cuts into teaching time. One way around that is the occasional 'open book /take home' test.)

> [should be} balanced in proportion to the material and time the material was covered in class.

I agree. As an example of balance, if I give a final exam for middle school or "general" science, questions might be something like: 1/5 life sci, 1/5 physics, 1/5 chemistry, 1/5 earth science, 1/5 astronomy. I have a mental outline and calendar on hand before the course even starts. So I can be sure not only that the test is not out of balance, but the teaching leading up to it wasn't either, that I didn't spend 1/3 of the year on biology because we got interested or got bogged down.