One small step for man

Mike Caulfield writes a nice reality check on vendor ePortfolio packages:

So enough of letting assessment vendors tell us what facilities we will be forced to use in their walled garden, and expecting us to be excited about it. Enough with assessment vendors selling us “environments”. What we should be doing is describing the the environment that might exist – students using Wordpress, Blogger, S3, GDrive, email, messaging, etc. And then we should ask if they have a tool that can evaluate that. How will their tool interface with the learning environment we’ve constructed?

Put it up against The Centre for Learning and Performance Technologies Top 100 list of tools favored by e-learning professionals. Looks like a list of Worldware to me, which is as it should be.

I’ve been thinking that my blog is an ePortfolio (I collect, select, reflect and project in this space). My categories filter for different audiences. But this is also a working space, I’m not always good at telling a single story to one audience, so I can see a case for a more formal presentation. But ideally, such a presentation would be built with open tools, and using the identity I chose, not a drop-box closed tool branded with the identity of an institution.

As for what I’d want to put into my portfolio, it would look more like the products and activities that Stephen Downes suggests in Open Source Assessment than a resume or the product of a typical university course.

In Cheating on online exams I speculated that students might use any of many synchronous communication channels to cheat during a synchronously delivered online exam (I am assuming the typical multiple choice timed exams.) Then today I found It’s not plagiarism it’s an easy essay where the Custom Writing folks purport to offer essays that will not be detected by TurnItIn.com.

So, there are two aspects to this cheating arms race. One is recognizing that interactions among technologies lead to new unexpected exploits, the other is the more traditional development of a better offense to overcome a better defense.

Rather than dispair, perhaps we need to turn to ideas of open source assessment where the activity and the assessment are both more authentic. The wrinkle to this thought is Custom Writing is offering to create Theses and Dissertations which ought to be more authentic tasks than the typical college essay assignment.

It started because the TagCloud on our university-sponsored blog was broken. Since it doesn’t look promising for a fix, I Googled and I found US Presidential Speeches Tag Cloud, which gives some cool insights into what was important in the times of various presidents. (Check out the the appearance of the word “war” relative to other words.)

Since TagCloud died, I went looking for alternatives, and ways we might build them. I’d rather find a free resource that we could feed RSS into, but with TagCloud gone, maybe we need to take it on.

Ideas and thoughts on addressing this are welcome.

I’m following a thread that starts with the upcoming IWMW Conference in July 2007 and a discussion started by Alison Wildish of how the university might think of embracing student-owned learning resources (aka Facebook, gmail accounts, etc). The thread took me to an article in The Register:

Keele University has ordered its students to watch their mouths on Facebook, and asked them not to express dissatisfaction with the institution on social networking sites.

The administration was provoked by a Facebook group called “James Knowles is a Twat”. Professor James Knowles is an English literature academic at the Staffordshire university.

Members of the group were warned that the group was unacceptable and would be dealt with “very severely” if it continued.

These reminded me of a recent piece in InsideHigherEd exploring the reliability of RateMyProfessor.com compared to campus-based course evaluations. (Original article: RateMyProfessors.com versus formal in-class student evaluations of teaching Theodore Coladarci & Irv Kornfield (PDF)) The authors tentatively make a recommendation similar to Alison’s:

“First, and predicated on the belief that RateMyProfessors.com is not going to go away, higher education institutions should consider encouraging their students to post ratings and comments on RMP. If a large proportion of an institution’s student body were to regularly and responsibly contribute to RMP, the potential value of that information to the institution would only be enhanced.”

“Higher education institutions should make their [student evaluations of teaching] data publicly available online. Although students doubtless would applaud this move, many faculty would oppose it because of genuine concerns about privacy and the negative consequences that published SET data may bring (e.g., see Howell & Symbaluk, 2001). But privacy is a thing of the past in the age of RMP, MySpace, and the like.”

Which brings us back to the Keele case. What is a “responsible” (to apply Coladarci and Kornfield’s word to Keele) Facebook group? Perhaps a group that engaged in critical thinking rather than ranting. And what is responsible rating of professors — perhaps not chili peppers but an engagement with the way the course contributed to substantive learning outcomes. Are current student evaluations of teaching as “responsible” as they might be, or are they chili pepper ratings in disguise? Perhaps universities should provide the vehicle (facilitation of critical engagement) that would allow students to judge and develop for themselves the desired “responsible” faculties. Perhaps this is one of the true missions of the institution.

In Open Source Assessment I wrote a reply to Stephen Downes’ ideas about open source assessment and open source course design. One of the advantages of the thread he started is the assessment ideas are relatively “cheat proof.” I was reminded of this issue because of an incident we have been investigating from the close of the last semester - during Finals week our WebCT server suffered a crash and in the logs were many connections to the chat port. I speculated (now refuted) that students where chatting concurrent with working on their exam. (Turns out that the chat port is also used by an innocent “who’s online” function.)

But the speculation stands:

• Facebook offers twelve 3rd party applications under the “Chat” heading including “WalkieTalkie gives each Facebook group a dedicated voice channel. Just push and talk.”
• Facebook facilitates students making groups associated with their class.
• And students are online concurrently during a high-stakes online exam.

Now, maybe using Facebook to cheat with your classmates is too brazen, and so students opt to exchange IM names and to message one-on-one, but they are online concurrently during an exam, and often at unproctored locations, so the potential must have been recognized by at least some students.

In Half an Hour: Open Source Assessment, Stephen Downes commented on what “…the ideal open online course would look like. …[his] eventual response was that it would not look like a course at all, just the assessment.” And he goes on to talk about that assessment in authentic assessment terms.
His reasoning was: “were students given the opportunity to attempt the assessment, without the requirement that they sit through lectures or otherwise proprietary forms of learning, then they would create their own learning resources.”

This ties into two threads of conversation I’m having with Theron. The first is how the university could use Web 2.0 ideas to respond to a Governor’s order to close during a flu pandemic, I wrote an emergency packet for this event. (Theron noted that the skills in my packet are the skills we’d want students to have when they leave the university, so we jokingly re-purposed the packet “Open in case of graduation…”)

The second thread is the project we are kicking off for the our ePortfolio competition. (As the links might change, here are three pointers into the work: the 2006-07 contest and the portfolio of its results and the 2007-08 pre-announcement.) The 2006-07 contest asked students to make a portfolio to document some aspect of their learning, and we got some very interesting (and diverse) results. The idea of the 2007-08 contest is that students are to find a problem facing WSU or their community and develop a solution to that problem, then document their learning in a portfolio.

To help prompt students to join the contest, we are collecting short summaries of students at WSU who have already done similar work in the context of classes. In a distance degree course, Decision Science 470, students were challenged to find a problem within their workplace, form teams around the problem and solve it. (Students saved a dairy plant from closing, found new ways to manage inventory, and improved performance of call centers.) In Human Development 410, distance students were challenged to find an problem in their community that mattered to them and learn about the policy and political aspects of the problem in order to understand and become an “engaged citizen” around the issue.

What is missing in Stephen Dowens analysis above, that we had in the courses, is an idea of how to leverage the group of learners to enhance their learning. In the John Seeley Brown notion of repair technicians always having their radios on and thereby becoming a community of practice, the course can be a hub for a community of practice. The designs for the courses above were a set of prompts that scaffolded some open activities (e.g., interview a member of the community who has expertise in your problem but who’s perspective differs from yours, ask how they organize to advance their political goal.). Students share their work on the activities (we used threaded discussion, but Web 2.0 solutions might have advantages), and feedback among peers is encouraged and structured with a rubric. As Stephen suggests, the final assessment is open and the evidence for the assessment is some form of synthetic response (aka portfolio or essay) to the course’s overarching question.

This design does not depend on a central role of the instructor, in fact, we have a growing body of evidence that students can use a rubric to provide peer evaluations that agree very well with faculty assessment (agree as well as faculty agree among themselves). In the case of pandemic, this means that the course could proceed and succeed in the absence of the instructor.

So, given Stephen’s open assessment model, what is the role of the university? I think its role is to make occasions for learners to form communities of practice and networks among themselves as a collection of experts. It might credential, based on these open assessments, but its graduates would have portfolios of authentic work that would be open to evaluation by employers and others outside. Further, because the graduates would be members of communities of practice, they would have reputation that would help third parties assess their knowledge and skills. Web 2.0 thinking lets us have conversations about how such a university might be decentralized, either in time of crisis, or to serve a distributed community.

Washington State University is going through an exercise to plan for a pandemic and the dispersion of faculty and students without canceling classes or closing the university (we don’t want to refund tuition). The thinking is along the lines of moving all the current face-to-face courses into WebCT and continue online. Presently there are 3500 group instruction course sections/semester (not counting thesis and other individualized directed study classes) and currently ~1000 are being offered or supplemented in WebCT. The question is, how would the university add by ~2500 sections in the run-up to a pandemic outbreak?

If we start the scale up now, moving all sections online, we could develop a deliberate process and given time, move each course, including providing the training, etc. needed. Ideally, we would include course design work in the process with the goal of improving the learning outcomes of the courses while we were at it.
If we wait until the next flu season and an immenent declaration of an emergency, there does not seem to be any way to expect that we could scale up the hardware or the faculty training, especially given that some of the key people might become sick themselves.

So assume the university could decide to, and successfully go down the deliberate scale-up path. We need to consider that WebCT and the WSU campus network are potential single points of failure. Individual students or faculty might also experience single points of failure with their ISPs. Using a traditional model of an online course: readings, PowerPoint, video/audio streaming, and quizzes, etc., we probably need to conclude that because of the multiple single points of failure many students will not be able to complete their course work during the diaspora.

Is there another model of a collaborative, adaptive group that:

• has a clear goal and can recognize (self- & peer-critique) progress toward the goal,
• uses multiple redundant communications channels and has ways of changing communication channels to meet changing circumstances,
• can continue to function with breakdowns in its command structure, or without one,
• where individuals can continue to function when the group is out of communications, and
• can recognize members of the group by some sign without a central authority providing introductions?

Does this sound like a Smart Mob? Or a terrorist cell? Or a military unit? What can we learn from those organizations and how would it apply to designing a university that would function during a pandemic?

Pandemic as teach-in

Rather than an obstacle to overcome, what if we were to say that the pandemic is itself an authentic learning opportunity for our students. Each university course could create a learning goal that tied to the pandemic, i.e., the sociology of pandemic, microeconomic impacts of pandemic, women’s history and pandemic, etc, etc.

Students would be charged with undertaking activities, individually and as collaborative groups relative to the subject and their personal situation. The course assessment would be using a pre-published rubric (such as the critical thinking rubric) and the artifact to assess would be a portfolio chronicling the student’s activity and learning during the pandemic event.

To manage the communications problem, a Web 2.0 approach needs to be designed. Tags and keywords would be agreed in advance (much like secret handshakes or signs) and these would be used to mark items on the web. Since single points of failure might cripple any single system, learners would use multiple systems, such as Wikipedia, Google Groups, Facebook groups, Blogger, del.icio.us, etc and create resources marked with the tags. Users would also be asked to post pointers in one system to resources in another, for example, in the Facebook group a user who found resources in del.icio.us would post a copy of the links found in del.icio.us. That way, if any given system is out, or any given user offline, others have ways to work around the outage.

When the pandemic is over, instructors ask students to complete their portfolios, including copies or links to appropriate resources and a reflection on how those resources give evidence to their deeper understanding of the relationship between the course topic and pandemic. Assessment is by the rubric.

Earlier this afternoon I met with the CTLT Learning Designers to look at some of my daughter Karina’s Montessori work and think about rubrics that I have been writing for Palouse Prairie Charter School that I am helping to create in Moscow, Idaho.

The work that attracted the most attention of the Designers was a bound portfolio from Karina’s first semester and an unbound collection of the same type of work from her second semester. The portfolio is a chronological collection of all the work of this type. I believe the unbound collection to be complete. It also has a few items from her 4th semester added.

From the discussion, I got a homework assignment, which was to ask Karina to select from the unbound collection “some” pieces that showed “her growth.” She and I talked before dinner about this task, and about how she had grown in ability in many ways. I regularly note this idea to her (couldn’t use a spoon, couldn’t get shoes on right feet, etc. and now can and does.)

The work in question is called “Metal Insets.” It is done with colored pencils on 5” square paper. A metal stencil is laid over the paper and its single simple geometric shape is traced. Then the student is to draw lines back and forth from edge to edge in the shape. “Quality Work,” the teacher’s term, looks like teeth in the mouth of a monster and this is called “jaws.” The lines that zig-zag back and forth are to stay within, but reach the perimeter outline.

The teacher showed us her notebook where this exercise is described and its primary purpose is to develop the hand and the skill of holding the pencil with a light touch, preparatory to penmanship. A secondary goal was to create a pleasing design or form, but the teacher stressed that metal insets are not art.

For this study 38 metal insets were examined (the unbound collection). A 39th was discarded during the process because it had a metal inset on front and back and we became confused as to which side we were examining. Karina does many metal insets that do not meet the “Quality Work” criteria as I understand it.

Karina was eager to help with my homework and sorted the pile into two groups while I was absent (it took about 2 minutes). One pile she called “Yes” the other “No.” I presumed that “Yes” was her denotation for “shows growth.” There were 10 Yes and 28 No.

I asked Karina for the rules for each category and wrote them down as she dictated. Words in brackets [] have been added to provide clarity
Yes-
•    I stayed in the lines (mostly)
•    I like the shapes [of the stencil used to create the outline(s)]
•    [I used my (currently)] Favorite colors
•    I did “jaws”

No-
•    Not my favorite colors anymore
•    I scribbled
•    Not my favorite shapes – I don’t like circles

To check how reproducible this classification was, I shuffled the two piles and then with her to check me, I classified each piece “Yes” or “No” giving my reasons from her list (above).  I was not able to do this work and keep records of my scoring. The biggest reason I seemed to be wrong in my classification was “favorite color.” “Not favorite color” was pretty easy (brown and black), but what makes favorite was harder. There must be a hue of purple and pink that are not in favor now which I could not recognize. Karina reclassified three (of 10) of  her “Yes’s” to “No.”  She also reclassified four (of 28) of her “No’s” to “Yes.” The reasoning for these changes was not clear.

I then found two more items that we had overlooked during our classification activities above. I evaluated each by her rules and found each a “No.” She agreed. These were set aside and not considered further. (One might be “Quality Work” the other is not.)

I then asked if she could adopt her teacher’s perspective and classify the 38 items as “Quality Work” or not.  The “Yes” pile now contained 11 items (10-3+4) and the “No” pile 27 items. I asked for the rules for “Quality Work” and wrote them down as she dictated:
* stayed in the lines [of the perimeter figure]
*  jaws
* color does not matter
* no scribbles

The 27 “No” pile was sorted into 5 “Quality work” and 22 not. The “Yes” pile turned out to have all quality work, despite my questioning what seemed to be scribbles and even lack of jaws on several. One of the pieces, dated May 10, 2006, has this annotation on the back from the teacher, “This is one of the most beautiful Metal Inset this whole year [smiley face] Cindy” In the procedure above, this item was first classified a “No,” and then later re-classified a “Yes.”  It was also classified “Quality Work.” It does not look like “Quality Work” to me, so the praise raises a concern to me that the artistic quality of the piece might impact its assessment.

Using the language of the Palouse Prairie School’s Critical Thinking Rubric (adapted from WSU’s Critical Thinking Rubric) I think Karina was demonstrating she could identify and summarize the problem, but did not articulate nuance or embedded issues. She also showed that she could identify her own a perspective, and it has some richness and is not her teacher’s cultural norm. She could also recognize the perspective of others (the teacher). She did not identify if this perspective was right or wrong, or if there might be yet more perspectives. (Another activity would be for me to classify the items with a rubric I create and see if she could understand and implement mine to sort the work.)

I then sorted the 38 items by date of creation (recorded on the back in a woman’s hand). Seven had no dates (six scored “No” and two scored “Quality Work”). The date range is 1/18/06 to 1/19/07. All but the last 4 items are from Spring 06 (ending May, 25) Of the dated items, the “Quality Work” pieces are scattered among the  dates with no obvious pattern, while the “Yes” pieces are mostly recent.

I am going to forego the Morning Reading Group meeting this morning in the interest of finishing some end of term course evaluation work.

If the merit of last session’s reading were to know a common perspective of faculty, then the merit of this week (Moore, Bill Guilty Bystanders WA State Board for Community & Technical Colleges December 2006 (A reprint from June 1998)) is as an antidote to that . Among the quotes I highlighted in that article was this:

…creating a real community—one in which people genuinely depend on each other for enhancing the quality of learning—out of an artificial context requires letting go of some power and control within the classroom…

And I note how that resonates with some of Chuck Pezeshki’s comments in an OpEd piece in yesterday’s Daily News, where he says:

* Students must be allowed some choices on how they spend their time in the classroom. When confronted with no control over their lives, they become submissive and passive thinkers.

Chuck’s whole item below.

HIS VIEW: Whither education in the 21st century?

By Chuck Pezeshki

Tuesday, December 19, 2006 - Page Updated at 10:07:20 AM

It’s all over the news again, like it has been for the past 20 years. This week, it’s on the front of Time magazine. Whither education? And what’s happening to our kids?

It’s exciting to see Moscow residents have been active in the debate themselves. The conference on Nov. 27 regarding vocational education is an important first step in any dialogue the community has in coming up with solutions that affect the future of our young people.

I am a mechanical engineering professor, and have been involved in solutions for progressive education my whole career. I run a design clinic, where senior college students complete real work for sponsor companies that pay thousands of dollars for that work. I’ve directed the clinic for 13 years. Many of my students are hired from my class by the sponsor companies. Students work in teams of four to six, and while I have high standards, I do not grade, other than an “A” or an incomplete. As in the Real World, students must complete real work, defined by a specification that is agreed upon by the company, and benchmarked at the end for performance. I have close to 100 percent delivery rate, with a product that conforms to the specification, and a 70 percent adoption rate. When students leave my class, they are truly “ready to work.”

You might think that either a) I’m lying, or b) I use such advanced technology that students by default produce good work. Neither is true. But I have secrets.

* If you want students to learn, you must believe that they can learn. Judging anyone constantly is not the recipe for anyone’s personal success.

* Students must be allowed some choices on how they spend their time in the classroom. When confronted with no control over their lives, they become submissive and passive thinkers.

* Plenty of hard questions, but no trick questions. Too much of education has been involved with fooling the students. No one likes to be made a fool — a recipe for disaster.

* A classroom must be a safe place, where students can explore, and not fear humiliation. The moment that any person is scared, they are operating out of the same part of the brain we have in common with a hamster — and it’s not the thinking part.

* Teachers must have some discretionary funding, to try new things, and some respect for their discretion as they experiment. Without this, we cannot move forward. I recently sat down with my son’s third-grade teacher. She is engaged, obviously very bright, and still young enough to not be cynical. Because of the WASL, every day she has is scripted by someone else. The same things that disempower students disempower our educators.

* We have to recognize that we are a society in transition from a more verbal/symbol-based literacy to a visual literacy. As I wrap up my class today, I’ll talk to some corporate sponsors on the phone. I’ll use the Internet to deliver an electronic portfolio of student work to another corporation, consisting of animations, video, a PowerPoint presentation, design drawings and a standard report. After that, I’m meeting a group of students to make our own production video of their final, constructed project. Needless to say, this is very different than turning in a standard, typewritten report to end the semester.

* Instead of the notion of rewarding individual students with the status of gifted, we must recognize that all students can be educated. We have to reward teamwork, instead of the current notion of winners and losers. In order to do this, teachers must move to a paradigm of classroom performance ownership. This means, especially in later grades where developmental issues are not such a large factor, we should be shooting for all students to learn the material presented equally — and then give individuals opportunities to express their own creativity in guided ways. Teachers must be accountable as well as students.

* We must recognize the tools students need to use today — from home-produced video, to computer-aided design, to video games — are different from the ones we learned. And while we as parents must be engaged, there are going to be topics we are not familiar with. Support for teachers now is critical in assuring that they have the educational background.

Finally, we have to love our children. Too often, I hear young people being blamed for the world’s problems. Young people haven’t had a chance to mess up the world yet. And blame gets us nowhere. Let’s keep the dialogue going, and get to work.

Chuck Pezeshki is a professor in mechanical and materials engineering at Washington State University, and chairman of the WSU Faculty Senate.

On 12/18/06 5:10 PM, “Ater-Kranov, Ashley” wrote:

Hi Folks,

Please visit the MRG wiki page http://wiki.wsu.edu/wsuwiki/Category:MorningReadingGroup to access the Moore article proposed by Gary Brown.  I’ve included the link and rationale here as well.
Moore, Bill Guilty Bystanders WA State Board for Community & Technical Colleges December 2006 (A reprint from June 1998) http://www.sbctc.ctc.edu/docs/education/assess/eWAG/2006december/ewagdecember06.html
Moore, who coordinates the Washington State Assessment Conference, spouse of a former HEC Board member, National Learning Community Fellow, great softball player and a good guy expresses the need for a new impatience with educators who fail to participate in assessment. As we head into a new push toward accreditation, this article may be useful for sharing with chairs and others even as it re-energizes our commitment to assessment and transformation.
See you Wednesday morning!

Ashley


From: Ater-Kranov, Ashley
Sent: Friday, December 15, 2006 3:24 PM
To: CTLT.Designers; Johnson-Shull, Lisa Ann; Jorgensen, Randy; Peterson, Nils; Weathermon, Karen Lynn
Subject: RE: a call for MRG reading material for 12/20

Hi MRG participants,

I’ve had one suggestion for MRG reading material so far for Wednesday’s session (12/20) – please send me as well as post your suggestion with the rationale behind the choice on the MRG wiki page by Monday afternoon, so we can get the selection decided upon in time to read it.
Thanks!

Ashley

Ashley Ater Kranov
Assistant Director
Center for Teaching, Learning, and Technology
Washington State University
Smith CUE 503N
(509) 335-6212