Qualitative Research: Anita Salem’s Experience

January 16, 2009

Here we print excerpts from an interview with 1999-2000 Carnegie Scholar Anita Salem, Professor of Mathematics at Rockhurst University. VKP’s Cheryl Richardson interviewed Salem about her experiences organizing and coding online discussion board postings and how the knowledge gained from that experience informed her teaching. Salem’s work is familiar to the many VKP participants at the 2002 Summer Institute who attended her informative workshop on using qualitative techniques to understand data about student learning.

Salem’s project, called Calculus Conversations, asked students to work through calculus problems collaboratively in an online discussion space. In this excerpt from the interview, Salem and Richardson discuss the way the discussion space was structured as a preface to understanding the sorts of data Salem had to work with later in her project.

Salem: The idea was to pose problems for our students to solve collectively. Our thought was that using the language of mathematics might help them think about mathematical ideas in a deeper way, and result in increased understanding of important concepts. We wanted to watch their problem-solving process as they worked to find solutions.

The way we did it was we would pose a problem on a web site and then the students interacted with each other using a threaded discussion format, either posing solutions or questions.

Richardson: Did you tell them exactly what they needed to do, that they needed to always pose a question, or was it just free?

Salem: Their instructions were fairly structured. Participation in the problem-solving process was required and graded.

Richardson: How did you grade it?

Salem: We graded it on four levels: One, if they didn’t respond, they received a zero. Two, if they responded and moved the conversation forward, they received full credit for the activity. Three, if they participated in the conversation, and their contribution kept the conversation even they received three-fourths credit. For example, sometimes a student would say, “let me provide an example of something,” and then another student would say, “here is a second example.” The second student didn’t really move the solution process forward. They were just verifying what the first person had said.

Richardson: I see. And if you participated but didn’t help the group?

Salem: The fourth level was if you participated, but you moved the conversation backwards—in other words, either the expression of your thoughts or your thoughts themselves were so convoluted that the conversation took a turn for the worse because of your statement, you still got half credit. So, the four levels were full credit, three-fourths credit, half-credit or no credit.

Holding them accountable for their ideas was important. We wanted them to think about their responses before they expressed them to the larger group and since this was a web-based activity, they had plenty of time to think. They had about a week from the time we posted the problem until we closed the conversation down.

Richardson: And what was your involvement in the conversation?

Salem: At no time during the conversation did the faculty intercede in the conversation and no hints were given. We wanted them to wrestle with the problem by themselves. Typically, the problems we asked did not involve complicated calculations. We were trying to keep the problems centered around very important and fundamental notions. All of the problems we posed were at the very heart of the material. So the problems weren’t just esoteric fringe problems. Being able to think about the problem and solving the problem was at the heart of what the class was doing at that point in the semester and was central to the fundamental ideas in calculus.

In this section of their talk, Salem and Richardson discuss the evidence of student learning that Salem had gathered. Salem explains some of the steps she took in an attempt to organize and understand her data by reading through it and understanding what students were doing on a finer-grained level.

Richardson: What did you do with all the postings you had gathered?

Salem: We captured all the online components of these conversations. We had time of day, who responded to whom and in what order. This meant we had a significant number of artifacts taken from the students’ problem-solving sessions. It was like videotaping a group of students studying mathematics in their dorm room. And then we closely examined each response.

Before I could evaluate the student responses, I needed to characterize what I was seeing in their problem-solving sessions, what sorts of methods were they employing and approaches were they using.

Richardson: I see. And how did you go about doing that– characterizing their problem-solving processes?

Salem: I collaborated with a psychologist (Dr. Renee Michael) who assisted me because I had never done qualitative analysis before, so this was all new to me. I am your basic quantitative person. The reverse of what most people are, I think. I asked my collaborator about best practices in attempting to capture and characterize what I was seeing in the problem solving sessions.

My collaborator instructed me to take the postings home, read them, and cut them out, so that each individual response was on a separate piece of paper. Then I had all these stacks of paper, and she had me put them in piles, looking at what I saw, just reading them, re-reading them, putting them in piles according to what kinds of things I was seeing that I cared about. This process took me a couple of weeks. She kept telling me to keep it simple; she kept reminding me to avoid being too scientific or overly analytical, just look for the key themes.

What I finally came up with is, if what I want to look for is conceptual understanding, I ought to be able to identify a response where a student is taking a conceptual approach to solving the problem — that is, the student is in some observable way stepping back from the details, and thinking about what is actually happening in this problem.

Richardson: What different kinds of responses interested you?

Salem: I had a pile that I had identified as conceptual responses. Unfortunately, this pile was quite small. As I read and re-read through the remaining responses (those not classified as conceptual responses), I was asking myself what is getting in the way of their abilities to think clearly about the solution? I finally noticed that the students were almost always responding at the detail level. They had long conversations with each other about what to label the vertical and horizontal axes. And then equally long conversations about what the unit of measure on each axis was going to be. Usually the problem itself didn’t call for this level of detail, but that is where they went and frequently that is where they stayed. So I created another stack of responses that illustrated attempts to approach the solution from strictly a detail orientation.

And then it occurred to me, well, it has always occurred to me in my teaching that if I could get my students to frame a good question, they could probably find the answer. It is that framing of the question that is always so hard. So, we also gave full credit for those students who could pose a question, the answer to which would lead us in a positive direction. Posing good questions was rewarded at the same level as finding the solution or moving the solution process forward. Now I created a third stack of responses that posed questions — any question. I would watch the responses and begin to hope that the students would pose a question – to stop and think about what they needed to ask before they began to make conjectures about the solution

The other thing that appeared to be blocking the conversation was the use of language. They might have the seed of an idea, but they lacked fluency in the use of mathematical language, so they were having trouble communicating their own understanding to others and also, I think, to themselves. So, I created a fourth stack of responses where the language was convoluted, and created its own set of difficulties.

Richardson: So at this point you had identified the categories of responses.

Salem: Yes. I had conceptual, practical, question and language categories. And it took me quite a while to establish that those were the things I cared about. Then I had to go through and code every response. Individual responses could be coded in more than one way. For example, a response might have a conceptual component and a detail component. Or a response might have posed a question, but the question was really conceptual, the best kind of question, and so it would get coded as a question and as a conceptual response. Or they might have asked a question that was all about details, so it would be double coded as a question and as a practical response.

At that point, the psychologist I was collaborating with advised me to describe my codes in enough detail, so that other people could code the work. I had to come up with very clear descriptions of what I meant by a conceptual response and what I meant by a detailed response.

Salem shared written descriptions with two colleagues who went through and coded the data as a way of testing her own “rater reliability.” Salem then discusses what she learned from the process and how it informed her teaching.

Richardson: So, what did you learn from this?

Salem: In the end it took quite a while to code the responses and then to make sure that I and other raters were arriving at similar codes. I had to be certain that we were reading and drawing the same conclusions. Once the coding was complete and verified by my co-raters, it was pretty clear to me where the students’ comfort level was: in the details and components of the problems. This had important implications for practice.

It meant I needed to go into class and take advantage of the fact that they were very comfortable with details and talk to them by starting at their comfort level with details. That is your starting place, but you eventually have to move them forward, you have to remind them of the need to rise above the details and keep asking what’s the bigger picture.

So, it did have some implications for practice because typically, what we do in mathematics is we go in with a big idea and then provide examples. [Students] are only paying attention to the examples. We need to start with the examples and move to the big ideas. It also means that we have to pay much closer attention to the examples we choose.

We began changing the direction, so we would use examples of data and details to motivate the larger scheme. Then in the end we would always ask questions that are very different from the ones they have practiced because if they understand the larger scheme, they will be able to use that understanding to solve new and less familiar problems.

I think this sort of coding scheme is very reasonable and very revealing. I think it is easy for people who are not social scientists to believe in. It didn’t take me too long to think, yes, this makes sense, we can learn important things about how our students learn from this type of careful and close observations of their work.

By Cheryl Richardson
Originally published March 2003

Assessment, Coding, Discussion Comments (0)

No Comments

Leave a comment