I have had lengthy discussions with my dean and our former vice-provost (his office is next to mine). If you have graduate students reviewing papers and have a say in acceptance or not, then it is not a peer reviewed conference. My dean told me AECT would not count as a peer reviewed conference for our faculty because many divisions allowed graduate students “full review rights.” That is, they can vote accept or decline and it is considered by the program planner. The mentoring process where a graduate student reviews with a faculty member and receives feedback, but has not vote is encouraged.

Let me reiterate my point. Yes, we can redefine peer review. However, there is a norm in the sciences and it seems that many of the provosts and graduate deans come from the sciences and subscribe to the definition used in the sciences. If we redefine peer review to include graduate students or anyone else in the organization, they may question the credibility of AECT for faculty presentations. As a result, we will be driving off faculty members, especially our junior faculty members.

AECT may need to make a distinction between “scholarly” papers and “practitioner” papers. Both would be peer reviewed, but the peer group would be different for each. Even with this distinction, I would encourage going back to requiring every presenter to provide a discussant with a paper in adequate time for the conference. There is a concept (and book) of the practitioner as a researcher. I strongly believe that practitioners can and should make contributions to our knowledge base. These contributions, however, should be subjected to a scholarly rigor that includes self reflection and support for the practice. As a third option, we might have a category titled Showcase that is for projects and project oriented presentations.

Gary

Hi! Just to add my two cents on this.

The issue was raised that AECT (because of students reviewing) is not a peer review conference. But if this is the definition of peer review, then what is AERA? Students review for AERA. They even review for DIVISIONS and not just SIGs. Yes, it's true. I have done so, as have other students I know. No, we did not all have the requisite "5 publications" AERA asks for. In reality, AERA doesn't have a set standard. They just ask for your CV, but they don't make unilateral exclusions or inclusions criteria. They leave it up to each division and SIG to judge whether someone's CV qualifies them to review.

I think that would be wise for AECT as well. Any unilateral decisions are foolish. As Michael points out, some grad students have great practical experience that qualifies them to review, or even significant research experience. Kay pointed this out in the AECT executive board meeting when she said that many "students" in the Teacher Ed division are former teachers and even school administrators. So they could be considered qualified to review, especially for practice-oriented proposals.

I think what AECT should do (or at least our own division) is create a database of reviewers (drawn from the database of members). Require an abbreviated CV of every reviewer the first time. In subsequent years, a note could be made in the database that they have already reviewed once so that they don't have to submit a CV any more.

Then, maybe make a base standard that every proposal gets reviewed by 1 or maybe 2 people with terminal degrees. This would allow people to answer with confidence that it was a "peer" review no matter how you define "peer." If students want to be the remaining 1-2 reviewers on each proposal, then let them submit their CVs and let the program officers make the decision on a case-by-case basis based on their research or practical experience. Once approved, put the mark next to their names in the database as well so future program officers know this student is experienced enough to review.

For those students not quite qualified yet to do it on their own, kindly encourage them to do a mentored reviewing experience with their advisor this first year and to continue to work on their publication record.

I understand the need to satisfy tenure boards, and this kind of process would do it. It would also allow students who are ready to review to do so, which is good for them and necessary for AECT. Maybe some program planners like our wonderful Tony Pina will review every single proposal, but we can't count on that every year--it's unreasonable. So to get enough reviewers, most program planners will need to tap into the student population to some degree.

And that's what every other organization does--including AERA.

BTW, if the database idea is impractical, much of what I have said could still be done without it, although it would duplicate work if you asked for CVs every year. But at least there could be, in the system for submitting a proposal, a simple check box for "do you want to review proposals" with a text box for your most recent publications. People could just cut and paste from their vitas.

Rick

Rick West
http://the-wests.net/rick

Okay, first let's just address your exact questions...

1. What constitutes peer review?

I believe that peer review is a systematic review by ones colleagues.

2. Who are our peers?

It depends on where the question is being asked. In this instance, an organization (i.e., AECT) is asking the question and the response would be our peers are our fellow members of the organization.

3. Where do graduate students stand?

Well, graduate students are members of AECT which means that only the close minded would see them as anything but peers.

4. How should graduate students participate within the review of proposals?

The same way that everyone else participates.

5. Should we evaluate credentials of reviewers?

Personally I would argue yes, but I would argue that of all reviewers regardless if they are faculty or graduate students.

6. Should we have training on how to review proposals?

Training, mentoring, some form of apprenticeship, there does need to be something.

Okay, let's look beyond the questions here. Why does the pursuing of a terminal degree means that someone becomes a second class member, at least in the eyes of SOME of the leadership of the distance learning division? We have "regular" members who only have Master's degrees that are considered peers and are allowed to review. But if one of those people, who may have been reviewing as a "regular" member for years decided to go back and get their Ph.D. they somehow are no longer qualified to review conference proposals. Having said that, I notice that those same people who have this problem have no problem whatsoever allowing graduate students to hold leadership positions in the divisions (including the DDL), so it's okay for graduate students to do the real work that keep the organization going, but when it comes to reviewing.

What is so magical about having a real job and not being a student, because the issue here isn't having a terminal degree or not, as many of our non-academic "regular" members are able to hold their positions with just a Master's degree?

I believe the real issue here comes down to membership. As Mary mentioned a third of our membership are graduate students. We are also losing hundreds of members ever year, many of which are graduate students who simply decide AECT isn't worth their money when they have to pay the regular member dues. How can we bring these graduate students into the fold, mentor them, provide them with experiences that they will see as useful, and give them a reason to stay members of AECT. It certainly won't be because we isolate them.

I think it is also worth noting that we have a hard enough time trying to get three reviews per proposal as it is. I know I submitted four or five proposals this past year, only two received three proposals and I only received one piece of written feedback (beyond just the raw scores). So let's face facts, our current "review" system doesn't provide much in the way of review as it is. Maybe the current reviewers could use some training on how to actually provide feedback?

To expand on some of your questions above, I might point you to some practices from another professional association for IT folks that isn't suffering the same problems that AECT is. In order to become a reviewer for AACE you actually have to apply to be a reviewer (including submitting your CV). Once you are a reviewer, there is a box in the form that reads "Assisted by:" which allows me as a reviewer to indicate to AACE if I have a student or students or junior faculty members shadow reviewig with me. This allows AACE to track who has been mentored into the review process prior to becoming reviewers on their own.

I think this application and the way to recognize mentoring is useful because it would allow AECT to accept graduate students and faculty members as peers for the review process with some level of confidence. The problem with excluding graduate students wholesale with no grandfathering, as happened last year with Gary Morrison's decision in DDL, is that you isolate and push peole like me away from AECT. I shadow reviewed as a first year doctoral student, reviewed as a second and third year doctoral student, and then when I was a fourth year doc student and a division officer, the decision made fairly unilaterally by Gary Morrison meant that I was no longer considered his peer. Even though I had been mentored into the process and had been doing it for two full years.

That just never made sense to me. Another problem that we have is that I was lucky enough to be at a place like Georgia where senior students and faculty were training you for the academy and valued that kind of apprenticeship and mentoring. Not every student has an environment lke that or works with an advisor/chair like that. How do we mentor those students into the process so that when they are in my shoes (as a first year faculty member) they actually know what they are doing. Cause if there is no mentoring in the review process, that first year faculty member is almost as useless as a first year doctoral student. In fact, I would argue that there are a number of students who come out of these focused, apprentice-style programs that have more research experience, methodological training, an content knowledge in their specific areas than many of our first, second, and even third year junior faculty members. Compare the CV of a graduating student from some of these top programs with many junior faculty who matriculated from some of the smaller programs and in many instances the doctoral student would come out on top. But as the curent DDL decision is right now (i.e., what was done for the most recent convention), that more qualified doctoral student is excluded and treated like a second class member.

Just some thoughts to add to the discusson...

MKB