Archive for the ‘Assessment’ Category

« Older Entries
Newer Entries »

eLearning 2.0 Talk for Educause

Monday, September 17th, 2007

We (Ashley Ater-Kranov, Theron DesRosier, Jayme Jacobson and I) just put in a proposal for the Educause Learning Initiative 2008 Annual Meeting: Connecting and Reflecting: Preparing Learners for Life 2.0, January 28–30, 2008 San Antonio, Texas.

Our proposal is “ePortfolio 2.0: expanding our views of portfolio”

Abstract (50 words max)

George Hotz’ blog chronicling his iPhone hack demonstrates students can collaborate world-wide and create portfolios that make learning visible. Our research suggests students and faculty are equally adept at giving criteria-based feedback. Portfolios capturing learning process combined with criteria-based feedback have implications for teachers, course design and LMS platforms.

Research Results

Ater-Kranov, Ashley and T. Desrosier. Raising the Bar: Communicating High Expectations and Getting Results. Poster. Washington State University Academic Showcase March 2007.

Cho, Yoon Jung, A Ater-Kranov, and G Brown. Faculty Attitudes about ePortfoios: A study for the National Coalition for ePortfolio Research. Poster. Washington State University Academic Showcase March 2007.

Hotz, George. Finding JTAG on the iPhone. Blog. http://iphonejtag.blogspot.com/ accessed Sept 10, 2007

WSU ePortfolio Contest. Making Learning Visible. Website. http://ctlt.wsu.edu/eportgallery accessed Sept 10, 2007

Session Focus

Portfolios have been used in several ways beyond being showcase of best work, including documentation of learning growth and for personal reflection. In the Spring of 2007, the Center for Teaching Learning and Technology at Washington State University hosted an ePortfolio contest that asked students to document their learning growth. The result was a rich array of evidence of learning, and a wide range of portfolio documentation.

More simply, a blog can be understood to be a learning journal, and with suitable summary posts, might serve as a portfolio. George Hotz blog of the hack of the iPhone is one example that illustrates one person’s informal but substantial learning journey enhanced by a collaborative community.

Personal Learning Environments (PLE) integrate both formal and informal learning episodes into a single experience and often have a blog at their heart around which the user assembles a range of resources and systems to create a personally-managed space.

To the extent that users open their PLE space for inspection by others it becomes a multi-faceted journal that makes learning processes and outcomes visible. When the user presents that log of learning evidence the PLE becomes an extended portfolio view.

A key facet of the blog or PLE is that the user seeks critical feedback and collaboration on their learning objectives, which typically involves the creation of social networks that cross institutional boundaries and are intended to place the learner at the central node in a learning community. We have evidence that demonstrates that students are at least as adept at faculty at providing criteria-based feedback, which opens the potential that giving of critical feedback can be scaled much larger than what faculty alone can provide.

This presentation will explore the blurring of the lines between portfolio, blog and personal learning environments and a parallel blurring between novice and expert feedback when novice feedback is appropriately scaffoled and guided. We will invite participants to join in the exploration and the implications they have for teachers, course design, assessment of learning, and IT planning around LMS and other supporting tools.

We are going to be working on this (sketchy) proposal for Active Learning Strategies in the session and welcome feedback:

The audience will collaborate in an analysis and deconstruction George Hotz’ blog (ne portfolio) of the hack of the iPhone. Then the audience will participate in a collaborative criteria-based rating. Audience data about itself will be shared and discussed within the threads of the presentation. Following the session, the audience data will be posed for later review by the audience and others.

Posted in Assessment, Collaborative Tech, Community, Global Competency | No Comments »

Open Source Assessment and iPhone Hacking

Friday, August 31st, 2007

In Half an Hour: Open Source Assessment Stephen Downes wrote:

What we can expect in an open system of assessment is that achievement will be in some way ‘recognized’ by a community. This removes assessment from the hands of ‘experts’ who continue to ‘measure’ achievement. And it places assessment into the hands of the wider community. Individuals will be accorded credentials as they are recognized, by the community, to deserve them.

We have been talking quite a bit the last few days about George Hotz and his iPhone blog.

The important piece in our conversations is that its easy to ‘recognize’ Hotz’ achievement (and a wide community has), and in the way he structured his blog, its easy to ‘recognize’ that he is a thoughtful and collaborative worker, these last two skills being important traits for employers, and his portfolio an interesting example of how students might demonstrate these global competencies in authentic project-based learning.

Posted in Assessment, Global Competency, Observation (surface & evaluate), WhitePaper | No Comments »

Worldware ePortfolios as tools for educational entrepreneurs

Tuesday, August 21st, 2007

Recently John Gardner posted some thoughts on Entre/Intrapreneurs, and what roles especially they play in a university. This sent me to looking for the blog of Clayton Christensen author of Innovator’s Dilemma. What I found was not specifically Christensen’s blog, but an interesting group blog from his consulting organization. I added that to my blog roll because I’ve found ideas in the book shape my thinking about trends around me at Washington State University.

For example, I’ve been thinking about Innovator’s Dilemma in the context of BlackBoard Course Management System and alternatives that may exist to that (increasingly expensive) tool. Alex Slawsby’s post gives me some further insights in applying the ideas of “interdependency” and “modularity” that I think play well with my own Web 2.0 and ePortfolio thinking.

BlackBoard is an “interdependent” system (if I understand Slawsby), with many tightly linked modules. This produces an internally efficient product, but at a cost to the customer. We (WSU) the customer are looking for alternatives that are “good enough” and at lower price points. SharePoint 2007 looks to meet that goal. It also is an interdependent system, but less specialized, it is a collaboration tool used in many business settings. As a course management system, it does not have all the features of BlackBoard, but many faculty don’t use most of the features, so SharePoint may be “good enough.” And for the University, which can amortize the cost of SharePoint over many other collaborative uses, it might be at a lower price point as well. Ehrmann calls tools like SharePoint, developed for other markets and applied to education, Worldware, and argues that they deserve special consideration for being both valuable and viable.

In a previous post, Slawsby discussed a potentially more disruptive, and more modular approach than even SharePoint to challenge BlackBoard’s CMS — online services offering free storage or other free resources (eg Google Docs). These ideas begin to beg the question, what part of the instructional IT should be outsourced completely?

I would have previously said that the University can’t outsource its instructional applications, because the University needs to manage the identity (the login ID) of its students — because it has scores and grades tied to those student identities. I would have said, “You can’t have a student just using Blogger, how would you know who they were or that the work was authentically theirs?”

Enter the student, who is increasingly “swirling” (taking courses from two or more educational institutions concurrently). The student is treating the university programs as modules (Slawsby’s term), mixing and matching courses to make independently concocted programs. The student may use one institution as a home base, bringing in credits toward a degree, or may be jumping around, ultimately looking for someone to credential the melange.

I recently wrote about an electronic portfolio as the core learning platform. In that thinking, the portfolio serves as the place to present to a specific audience the collection of learning experiences and the value and meaning that come from those experiences. Those experiences are probably not test scores or even a transcript, but more authentic products of learning, work, and avocational activities. Such a portfolio should not be a broadcast, but more like a blog, be open to comment, a place for the learner to present her current state of thinking and seek input to evolve understanding.

Which brings me back to my interest in Dr. Gardner’s post on Entre/Intrapeneurship in the University. He says, “It [entre/ intrapreneurship] must be embedded in our WSU culture and our curriculum.” Given that swirling students are already acting like educational entrepreneurs, and Google continues to move in directions that allow those students the potential outsourcing of elements of our instructional IT, I think the time for Dr Gardner’s conversation has already arrived.

Posted in Assessment, Global Competency, My Blogroll, Observation (surface & evaluate), WhitePaper | 1 Comment »

ePortfolio as the core learning application

Friday, August 17th, 2007

Much of this thinking springs from Stephen Downes’ review article, eLearning 2.0. Experiments like ELGG and Dave Cormier’s FeedBook have implemented some of these ideas and added to our (Center for Teaching Learning and Technology at WSU) thinking.

Portfolio thinking/working includes these elements

  • Collect your work
  • Select from your work important examples, annotate what is important (add metadata)
  • Reflect on your work, are you meeting your goals, how do you know
  • Connect your work to that of others (may provide context, support, evidence of success)
  • Project your work into the community to solve problems (provides context and authentic evaluation)

Following these ideas springs our conviction that platform and tools for creating ePortfolios should be Worldware, rather than custom tools purpose built for education.

Bloggers have foreshadowed our ideas about electronic portfolios, where they are collecting their original writings and synthesizing/ reflecting about their readings.

In thinking about Pandemic Flu planning , we have looked at the multiple points of failure and proposed a loosely coupled teach-in, based on an ad-hoc set of tools.

Our 2007 ePortfolio Contest challenged contestants to document their learning growth — we wanted to explore how to gain insight into the learning that is often masked in a ‘showcase’ portfolio.

The more sophisticated blogger uses a blog roll to provide context about what influences them. And that blogger understands they are a “central node” (Resnick) of a (self-assembled) learning community — and the blogger/learner seeks critical input from others via comment and trackback. The blogger is engaged in dialog for the purpose of learning within a community of practice.

We understand the well developed blog to be a portfolio, but find its chronological structure can limit its utility to a would-be portfolio reader. Well developed “review” posts, that link to other posts (supporting evidence) in the blog can serve this synthetic, and demonstrative, role.

Using a portfolio platform allows the blog to continue in the mode where it is strongest, Collection and Reflection, while the portfolio provides a place to make a presentation to a specific audience for a specific purpose. Ideally the portfolio has its own file storage and Authentication/ Authorization structures to supplement the other systems from which it is aggregating.

In our thinking a portfolio (see Pandemic Flu), is a hub that can aggregate (but may not need to contain) artifacts (it might be important to bring the artifacts into the portfolio if issues of AuthZ might keep the portfolio reader from seeing the artifact, or if the artifacts are in locations where they are subject to destruction (an example of the latter might be a page in Wikipedia). Typically, the artifacts lie in native environments most suitable to them (Flickr, Blogger, del.icio.us, etc) and are arranged into the portfolio by tagging and a syndication mechanism (such as RSS).

The piece we are adding with our 2007-08 portfolio contest is the idea to engage with a community (local, national, international) on a problem and its solution. This requires the learner to learn in a multi-disciplinary way in an authentic context.

The portfolio, in this application, likely becomes a “collaba-folio” where the author is collaborating with a community in the work and documenting learning growth. It is not a showcase portfolio of a finished work. In fact, following BioQUEST, we think that authentic learning work is seldom “finished,” rather it is abandoned in favor of new, more important learning pursuits.

The teacher in this model is taking actions symmetric to the learner. The teacher is a more sophisticated learner, providing feedback to novices within a web of teaching-learning relationships. The teacher also understands that, through past reputation, he may have social capital to extend to a learner, and that extension can be done publicly via the teacher’s blog roll or by a blog post that synthesizes some aspect of the work of the learner with other members of the community (who may then provide the learner with feedback or resources). The teacher should be conscious in using social capital, and perhaps earned credentials, to advance the thinking of more novice learners into the communities of practice.

Posted in Assessment, Global Competency, Observation (surface & evaluate), WhitePaper | 7 Comments »

Open Assessment and Mrs. Kohler’s Spelling Lesson

Tuesday, August 7th, 2007

It was ironic that I was at Technorati searching for “Open Assessment” when I stumbled on

Mrs. Kohler’s Lesson Plan

August 3, 2007

Standards: 7C1.3

Objectives: Studnets will take Spelling Lesson #1 test. Students will work on their Vocabulary Squares for Duffy’s Jacket. Students will learn how to create their Weekly Grading Sheet Packet.

DLA-Proofreading

Anticapatory Set: The teacher will explain that the studnets are taking their Spelling Test #1 for Compound Words. She will remind them that some words are closed, and some are open.

Assessment:

Studnets will take the Spelling test, then trade and Grade.

Now, I also make the typo of “studnets” for “students,” but finding Mrs Kohler has me thinking about how I’d want children to go about the task of learning about spelling.

For the record, I care about spelling correctness, but I have a distaste for Mrs Kohler’s method. (perhaps because it didn’t work for me 45 years ago). So, how would I think about “attending to” spelling? – which I think is a question that goes beyond spelling mechanics.

A colleague on the bus I ride, who teaches in the College of Education, was relating a story about his daughter, a first grader, who comes home with perfect scores on her in-class spelling quizzes. She also loves to write and draw and makes greeting cards for family and friends — on which her spelling is poor. He wryly notes that she does not carry her skill from the classroom to the real context. Now– does this mean that he understands that Mrs Kohler’s work is perhaps in vain, students will learn to spell in authentic contexts by the feedback they get in those contexts, and that students may learn to do school spelling for school tests, but that is a different (and perhaps irrelevant) additional skill. If so, how sad for the efforts of Mrs Kohler and the system of No Child Left Behind where she is trapped.

I think the real issue here is not Mrs Kohler’s typos but schoolbook approaches to teaching spelling. The issue is teaching children to be effective communicators and how to work with their mistakes in the process of being communicators. A few typos didn’t deter my understanding of Mrs Kohler. Children should know the same is true. What they really need to understand is when bad spelling and grammar will impede their communication goals. The other thing they need to understand is how to attend to those errors so that the effect is the one they desire. It may not be by becoming good spellers, it may be by learning to use tools as aides. For example, I make both typos and spelling errors, but I use Firefox as my browser to help me avoid publishing them. The unfortunate thing is that children compose with crayons, a very inflexible medium unfriendly to correcting mistakes and they are reminded by their schooling that making mistakes is bad.

I came back to edit this after I found Vivian Cook’s piece The Liter Cide 2 Spelling where she explores some of these ideas in a humorous way to make some of my points above.

Posted in Assessment | No Comments »

Assessment meets WorldWare

Monday, August 6th, 2007

Mike Caulfield writes a nice reality check on vendor ePortfolio packages:

So enough of letting assessment vendors tell us what facilities we will be forced to use in their walled garden, and expecting us to be excited about it. Enough with assessment vendors selling us “environments”. What we should be doing is describing the the environment that might exist – students using WordPress, Blogger, S3, GDrive, email, messaging, etc. And then we should ask if they have a tool that can evaluate that. How will their tool interface with the learning environment we’ve constructed?

Put it up against The Centre for Learning and Performance Technologies Top 100 list of tools favored by e-learning professionals. Looks like a list of Worldware to me, which is as it should be.

I’ve been thinking that my blog is an ePortfolio (I collect, select, reflect and project in this space). My categories filter for different audiences. But this is also a working space, I’m not always good at telling a single story to one audience, so I can see a case for a more formal presentation. But ideally, such a presentation would be built with open tools, and using the identity I chose, not a drop-box closed tool branded with the identity of an institution.

As for what I’d want to put into my portfolio, it would look more like the products and activities that Stephen Downes suggests in Open Source Assessment than a resume or the product of a typical university course.

Posted in Assessment, Collaborative Tech, My Portfolio | No Comments »

Cheating: an arms race between students and faculty

Thursday, July 12th, 2007

In Cheating on online exams I speculated that students might use any of many synchronous communication channels to cheat during a synchronously delivered online exam (I am assuming the typical multiple choice timed exams.) Then today I found It’s not plagiarism it’s an easy essay where the Custom Writing folks purport to offer essays that will not be detected by TurnItIn.com.

So, there are two aspects to this cheating arms race. One is recognizing that interactions among technologies lead to new unexpected exploits, the other is the more traditional development of a better offense to overcome a better defense.

Rather than dispair, perhaps we need to turn to ideas of open source assessment where the activity and the assessment are both more authentic. The wrinkle to this thought is Custom Writing is offering to create Theses and Dissertations which ought to be more authentic tasks than the typical college essay assignment.

Posted in Assessment, Collaborative Tech | No Comments »

Using Tag Clouds for insight into what’s important

Friday, June 29th, 2007

It started because the TagCloud on our university-sponsored blog was broken. Since it doesn’t look promising for a fix, I Googled and I found US Presidential Speeches Tag Cloud, which gives some cool insights into what was important in the times of various presidents. (Check out the the appearance of the word “war” relative to other words.)

Since TagCloud died, I went looking for alternatives, and ways we might build them. I’d rather find a free resource that we could feed RSS into, but with TagCloud gone, maybe we need to take it on.

Ideas and thoughts on addressing this are welcome.

Posted in Assessment, Collaborative Tech, Community | 1 Comment »

Student-owned learning resources – regulate or educate?

Tuesday, June 12th, 2007

I’m following a thread that starts with the upcoming IWMW Conference in July 2007 and a discussion started by Alison Wildish of how the university might think of embracing student-owned learning resources (aka Facebook, gmail accounts, etc). The thread took me to an article in The Register:

Keele University has ordered its students to watch their mouths on Facebook, and asked them not to express dissatisfaction with the institution on social networking sites.

The administration was provoked by a Facebook group called “James Knowles is a Twat”. Professor James Knowles is an English literature academic at the Staffordshire university.

Members of the group were warned that the group was unacceptable and would be dealt with “very severely” if it continued.

These reminded me of a recent piece in InsideHigherEd exploring the reliability of RateMyProfessor.com compared to campus-based course evaluations. (Original article: RateMyProfessors.com versus formal in-class student evaluations of teaching Theodore Coladarci & Irv Kornfield (PDF)) The authors tentatively make a recommendation similar to Alison’s:

“First, and predicated on the belief that RateMyProfessors.com is not going to go away, higher education institutions should consider encouraging their students to post ratings and comments on RMP. If a large proportion of an institution’s student body were to regularly and responsibly contribute to RMP, the potential value of that information to the institution would only be enhanced.”

“Higher education institutions should make their [student evaluations of teaching] data publicly available online. Although students doubtless would applaud this move, many faculty would oppose it because of genuine concerns about privacy and the negative consequences that published SET data may bring (e.g., see Howell & Symbaluk, 2001). But privacy is a thing of the past in the age of RMP, MySpace, and the like.”

Which brings us back to the Keele case. What is a “responsible” (to apply Coladarci and Kornfield’s word to Keele) Facebook group? Perhaps a group that engaged in critical thinking rather than ranting. And what is responsible rating of professors — perhaps not chili peppers but an engagement with the way the course contributed to substantive learning outcomes. Are current student evaluations of teaching as “responsible” as they might be, or are they chili pepper ratings in disguise? Perhaps universities should provide the vehicle (facilitation of critical engagement) that would allow students to judge and develop for themselves the desired “responsible” faculties. Perhaps this is one of the true missions of the institution.

Posted in Assessment, Collaborative Tech, Facebook/MySpace | No Comments »

Cheating on online exams (a speculation)

Friday, June 8th, 2007

In Open Source Assessment I wrote a reply to Stephen Downes’ ideas about open source assessment and open source course design. One of the advantages of the thread he started is the assessment ideas are relatively “cheat proof.” I was reminded of this issue because of an incident we have been investigating from the close of the last semester – during Finals week our WebCT server suffered a crash and in the logs were many connections to the chat port. I speculated (now refuted) that students where chatting concurrent with working on their exam. (Turns out that the chat port is also used by an innocent “who’s online” function.)

But the speculation stands:

  • Facebook offers twelve 3rd party applications under the “Chat” heading including “WalkieTalkie gives each Facebook group a dedicated voice channel. Just push and talk.”
  • Facebook facilitates students making groups associated with their class.
  • And students are online concurrently during a high-stakes online exam.

Now, maybe using Facebook to cheat with your classmates is too brazen, and so students opt to exchange IM names and to message one-on-one, but they are online concurrently during an exam, and often at unproctored locations, so the potential must have been recognized by at least some students.

Posted in Assessment, Collaborative Tech, Flu | 1 Comment »

« Older Entries
Newer Entries »