Archive for the ‘Assessment’ Category

I missed the bus – thoughts on indirect assessment

Tuesday, March 9th, 2010

I missed the bus to work today. I knew time was tight as I was going out the door. As I went along, I gained confidence I would make it, because I saw one of the school buses that I usually meet. And some kids waiting for another school bus (I usually see two school buses). And I saw the city buses at the bus transfer point (but I could not see my bus meeting them).

ALAS, none of that indirect evidence measured where my bus was on its route. As I rounded the Kibbie Dome I saw my bus picking up a rider at my stop and heading away.

I’m noting this as part of the conversation I’m part of about direct and indirect evidence of student learning outcomes. This is an example of the failure of relying on indirect evidence.

Different conversations about what is important

Thursday, July 16th, 2009

In IJ-SoTL – A Method for Collaboratively Developing and Validating a Rubric Allen and Knight discuss their experience validating a rubric with two groups: faculty and industry professionals. They report:

Faculty weights differed markedly from the professionals’ results [see table 4 in the article]. Faculty considered category 2 (In the headline/lead combination, Is the message clear and compelling?) and category 4 (Does the news release use a convincing journalistic style?) the most important. Categories 1 and 5 received the lowest weighting.

We have seen similar results in a course where we asked a group of faculty and a group of industry professionals to rate student work and also to rate the assignment used to assess the student work.

In both examples, the faculty seem more focused on formalisms and the professionals on the aspects of the task that lead to practical success.

Common Reading & Open Learning Communities

Thursday, May 28th, 2009

Thanks Bill Marler for your offer to support Washington State University’s Common Reading program after it got caught in a recent controversy regarding the book Omnivore’s Dilemma. See also developing Facebook action related to the topic.

From his blog, I can tell Marler has some appreciation of Web 2.0 as a life-long collaboration and learning strategy. This whole event is an example of how having a curriculum open to community review can improve learning outcomes. Searching in Google for “WSU Common Reading” shows that the event lit up a problem-solving community with multiple perspectives but overlapping interests in this topic; a community that produced the resources to sustain a learning opportunity.

WSU’s Center for Teaching Learning and Technology has been exploring how to help students learn in, from, and with such communities with projects like the Microsoft co-funded ePortfolio Contest. A variety of lessons can be learned from that project, including thoughts on how to transform the traditional gradebook by extending the idea of grading out into the community and making it a process for collecting community feedback on student work, AND the assignments that created the work, AND the program goals that shaped the assignments. I think this represents the way WSU needs to move forward with a Global Campus concept.

A lesson in driving up readership

Tuesday, May 5th, 2009

On Friday, April 24 the Chronicle’s Wired Campus ran an item on the failure of U. of Michigan’s Online Teaching-Evaluation System. The article was hot news because of the scale of the player and the scale of the failure. I posted this comment near midnight Sunday, April 26:

My comment on the article

This drove a large spike in readership of the associated resources on April 27.

Page views for WSUCTLT blog

And examining how readers got to the site we see they came from several related pages in the Wired Campus article.

pages that referred to WSUCTLT

which brought readers to these pages

pages viewed as a result of the comment

Implementing Obama’s 100 Hours of Service Plan

Monday, November 10th, 2008

At the Obama/Biden transition site, Change.gov, there is description of a universal voluntary service plan. This includes an idea to “establish a new American Opportunity Tax Credit that is worth $4,000 a year in exchange for 100 hours of public service a year.” And a goal to expand service-learning in the nation’s schools with “a goal that all middle and high school students do 50 hours of community service a year.”

That sounds exciting, but the downside in fraud and corruption is easy to see. To see the problem, look at who can certify my 100 hours of service to the Feds for my tax credit and what I might be doing for them to earn that certification. Beyond those limitations, where is the motivation for the 101’st hour of service? Where is the motivation to quality service? How does this service get leveraged into even greater gains?

Its still a great idea and there is a simple, and powerful, extension to it. We have been pointing at examples, and building some ourselves (see below). It requires a partnership between the school (its teachers and curriculum), the problem being addressed, and the community in which the problem is situated. Here is an example of real problem solving in service to community in a high school in Minneapolis, MN. Expeditionary Learning Schools/ Outward Bound are consulting with schools along similar lines.

For example, a school could set a goal to harness the interests and expertise of the school’s community (students, staff, parents and alumni) to address real world problems encountered by communities both locally and globally. Its curriculum would be designed to collaborate with community members – institutional, local, or global – to identify a problem, explore solutions, develop a plan, and then take steps toward implementing that plan. Students would engage these challenges as service learning. The outcomes of their work would be readily documentable using ideas like Gary Brown’s Harvesting Gradebook. But more than just documenting the student work, the process would have transformative impacts on the educational institution also, far more profound than the Collegiate Learning Assessment (CLA) or the No Child Left Behind (NCLB) standardized testing.

The importance of this strategy is its immediacy. It can be launched rapidly (its already started), it can target needs in communities without a large bureaucracy to decide what the needs are. It can tap local resources and world resources at the same time. It scales well.

Examples

Harvard Program in Networked Governance “The traditional notion of hierarchical, top down, government has always been an imperfect match for the decentralized governance system of the US. However, much of what government does requires co-production of policy among agencies that have no formal authority over each other, fundamentally undermining the traditional Weberian image of bureaucracy.”

United Nations Volunteers “The paper argues that volunteering, like social activism, can be purposeful and change-orientated. Volunteering can be directed at influencing agenda-setting, policy-making, decision-making and representation, and is also an important mechanism for promoting empowerment, personal transformation and social inclusion.

The paper also highlights the complementary and supporting roles that volunteering and activism play in fostering participation. For example, social activism plays an important role in providing leadership, defining areas for engagement and mobilising individuals.”

EDUCAUSE Tower and the Cloud “The emergence of the networked information economy is unleashing two powerful forces. On one hand, easy access to high-speed networks is empowering individuals. People can now discover and consume information resources and services globally from their homes. Further, new social computing approaches are inviting people to share in the creation and edification of information on the Internet. Empowerment of the individual — or consumerization — is reducing the individual’s reliance on traditional brick-and-mortar institutions in favor of new and emerging virtual ones.

Land Grant 2.0 “Dramatic shifts in the economy associated with the rise of globalism call into question the traditional ways in which land-grant institutions have defined their roles in contributing to economic and social well-being. Since the assets most needed for global economic viability – a base of innovation, talented people, and ubiquitous connectivity – are core strengths of universities, it is fair to ask how these institutions can more holistically engage with economically distressed regions to build critical innovation economy competencies.” see also University of Illinois Global campus.

WSU ePortfolio Contest “The goal of the 2007 – 08 WSU ePortfolio Contest was to harness the interests and expertise of the WSU community to address real world problems encountered by communities both locally and globally. It called upon contestants to collaborate with community members – institutional, local, or global – to identify a problem, explore solutions, develop a plan, and then take steps toward implementing that plan.” See also specific winners: Margo Tamez, Kayafungo Women’s Water Project

ThinkCycle This is the thesis to study the (now-defunct) ThinkCycle project exploring “How can we create an environment that encourages distributed individuals and organizations to tackle engineering design challenges in critical problem domains? How should we design appropriate online collaboration platforms, support learning, social incentives and novel property rights to foster innovation in sustainable design?” This concept can be broadened out beyond the engineering domain to other problem domains. An example is National Collegiate Inventors and Innovators Alliance, that says of itself, “The NCIIA works with colleges and universities to build collaborative experiential learning programs that help nurture a new generation of innovators and entrepreneurs with strong technical and business skills and the tools and intention to make the world a better place.”

A final example is Talia Leman, at age 10 organized fund raising for Katrina relief, and has since started RandomKid.org to help other children become social entrepreneurs. In this NYTimes OpEd notes “Frankly, these kinds of initiatives have a mixed record in terms of helping the poor in a cost-effective way. But they have a superb record in enlightening and educating the organizers.” which may be exactly the outcome that is most important from some of these efforts. The Times piece also points out some other efforts along similar lines.

Setback for 21st Century Resume

Sunday, July 20th, 2008

I’ve been trying to make headway on understanding a 21st Century Resume, thinking that it should not be your father’s paper resume. But recently I’ve had a couple “send me your resume” requests, so here it is in all its paginated glory: Nils S Peterson CV.

I’m choosing categories for this post. Milestone is obvious for a noted setback. Reflection-in-action seems an appropriate guide to what I should be doing, but it is Assessment that caught my eye. Clearly, it is assessment that is the goal of keeping a document like a CV. In my experience reviewing job candidates, a CV is a cumbersome tool for assessment.

So, I have added a new section to my CV, at the bottom, Community Activism. My wife’s summary: you defeated WalMart, Saved the Bus, and Chartered a school. But the reflection, and I hope your assessment, is what did I learn doing those things?

I joined the NoSuperWalmart effort in December 2005 hoping to get a chance to apply my ideas about blogs to community organizing. There is a blog, but it did not become a key piece in the communication strategy. I found myself in a role of spokesperson and part of the inner leadership circle, but the communications strategies that carried the day were email, newspaper ads, and handbills.

A year later, I got a better chance to explore the nexus of online and local community organizing. The conclusion that I draw from this work is that the web can be integrated into a local community’s political organizing. It requires effort to make the site participatory (with multiple perspectives) and authoritative (with data that has been collected about the problem and is open to inspection, support or refutation).

My most recent activism has revolved around gaining a charter for Palouse Prairie School in Moscow. This work has been conducted mostly offline, but its given me a chance to think about how to merge the content management of two blogs and Google Calendar into a web site. It not right yet, but its a step beyond the NoSuperWalmart effort.

What I think I’ve learned in those efforts is feeding into a series of blog posts I’m co-authoring regarding transforming the grade book. What we are arguing is that a community assesses its members (and a university education should assess students) by applying a set of broadly agreed criteria across a span of the member’s work. We are exploring the mechanisms by which the learner gathers, organizes, learns from, and demonstrates that assessment. The result will not be a CV, it will be a portfolio and the public assessment of that portfolio. Not a showcase portfolio, but a learning portfolio.

The CV is a showcase, and not even the best one, in that it is a catalog of everything without organizing narrative. What would be more interesting it seems is to showcase an accomplishment and the learning history leading up to it, as George Hotz did in his blog/portfolio on cracking the iPhone and Margo Tamez did in her portfolio on her efforts to prevent a US-Mexico border wall.

In Wikipedia they don’t care about your credentials

Monday, July 14th, 2008

In Wikipedia your contribution can be respected even if you are anonymous. The Wikipedia community works because it has a set of public criteria used by the community (and debated within the community) for judging all work. For example, there is a policy around deletion of pages and procedures for implementing it.

In thinking about transforming the grade book, I am concluding that we have reduced the teaching and learning proposition to one core idea: the public presentation of evaluation criteria, the public application of the criteria to a given learner’s work, and a public means for the criteria and their application to be negotiated by a community over time.

I think this is what Stephen Downes is saying in Open Source Assessment, you don’t need a curriculum created by experts, you need this core tool, the assessment.

But, key to making this work is to directly assess what you want learners to be doing. If you use an indirect measure, like a standardized multiple choice exam, what you will get is learners who master that test but may not have the skills and knowledge that are actually desired. Patricia Cross suggests that what you measure and value is what you will get more of from learners.

And, once you have your assessment underway, you need to constantly redefine it by holding the results up against what you would like to have happening. The key is not how well students do on our exams, its how well the exams promote what we want students to do.

FERPA and Learning 2.0

Thursday, July 3rd, 2008

John Thomson, writing in response to Martin Weller’s ideas in SocialLearn finds Web 2.0 an awkward fit for higher education and gives among his reasons for why universities can’t/won’t change:

“FERPA has everybody scared. The purpose of the law was to protect student’s personal and grade information from things like the grade list on the door and prying parents. Yet fear over having students work on external commercial systems, which are largely secure from hacking and violate privacy only on the aggregate level of data, causes hesitation from using Web 2.0 systems or attempting to form partnerships with their owners.”

Having students do learning activities in public seems completely reasonable in theater or music where performance to audiences is part of the authentic activity. And I doubt FERPA has a problem with a school play or concert listing the names of the performers in the program even if they are also the members of a class preparing the performance. The difference for me is the authenticity. Forcing a student to blog on a class discussion or post a paper on the causes of World War II, where the activity is school work and not authentic performance is where I have a problem.

If the instructor is giving a grade for the activity, I believe that is FERPA protected, but in a public venue like those above, the audience did its own assessment (not to be confused with grading) and that too is a required part of the authenticity of the experience.

For me, the interesting implications of Learning 2.0 come in taking a transformative view of the whole learning enterprise. Recently, we have been writing on transforming the grade book. The activities we propose, especially in the second and third variations probably could not be conducted within a closed learning environment — they are authentic engagements with a community and need to be on the open Internet. I see no FERPA issue here.

More on 21st Century Resume

Friday, November 9th, 2007

I’m struggling with the problem of my resume (or now my portfolio) again. Looking back, I like the analysis I did a year ago thinking about a 21st Century Resume. It talks about forward-looking and backward-looking documents that serve as reflections about problems and their solutions. We see those same kinds of documents created by Hotz as he hacked the iPhone.

This kind of reflection captures Gregory Bateson’s notion about information “a difference that makes a difference”. That is, reflections about changes in the world that I made or plan to make — not just random speculation. So in addition to capturing reflections (be they grant proposals, articles (such at the one requested by Innovate), or blog posts, its incumbent on me to show that my reflecting was “reflection-in-action” (from Donald Schön). That is, I’m reflecting with the intent to make a difference.

This thinking helps me move further along the path in the previous post, where I was reorganizing elements on my paper CV. Beyond organizing the content, I need to make presentations of myself that are narrative, that show how these reflections and milestones connect to make patterns.

But collecting the necessary evidence is not easy. For an example of the problem, I joined CTLT in the fall of 2000. They were well into the process of creating an online learning environment called “The Bridge.” One of the key, and incompletely developed, elements of the program was a personal workspace. Unlike previous thinking about OLEs, this one provided a resource for the learner, the learner’s own, private domain, that transcended the temporal constraints of a course. One of the key limitations of The Bridge’s personal space was that it was totally private, the mechanisms to authorize others to parts of your workspace were never realized.

I played a role in the retirement of The Bridge. It was a loss, a defeat, because it successor on our campus was WebCT, which had a teacher-centric, term-limited perspective. Personal space for the learner was lost. At the time I did not place much emphasis on the lost feature.

Enter SharePoint. The university was adopting it for administrative productivity reasons, but it also had a MySite feature that captured The Bridge’s personal space and added all the functionality that was never implemented in The Bridge.

I spearheaded an effort to understand SharePoint enough to get CTLT and ITS to implement it as a resource for the university. The personal portfolio and collaboration space of the MySite is catching fire in multiple places in the university.

OK, so here is the problem — what goes into my resume or portfolio from this story? I didn’t put the personal space into The Bridge, I recognized (but didn’t document) its importance. I recognized that MySites had the same feature. I argued, probably nowhere in print, that the university should implement MySites. I was a cheerleader for the explorations by CTLT staff and their colleagues into SharePoint portfolios, but did not do much of the important technical or intellectual work. I co-authored a white paper about some of these ideas, but it does not capture the difference that I made at WSU, though the white paper might yet be seen to make a difference itself. Closer to the mark is this summary of the history of SharePoint at WSU. It documents that a change was made, but doesn’t show either the seminal role of The Bridge or my agency.

So how do I capture the evidence of my past work, my learning? If I were writing project proposals I would have forward-looking reflections. If I were writing publications of my work I would have backward-looking reflections. We have tried to institutionalize these documents in CTLT, but we tend to rush along with the work and not create them.

eLearning 2.0 Talk for Educause

Monday, September 17th, 2007

We (Ashley Ater-Kranov, Theron DesRosier, Jayme Jacobson and I) just put in a proposal for the Educause Learning Initiative 2008 Annual Meeting: Connecting and Reflecting: Preparing Learners for Life 2.0, January 28–30, 2008 San Antonio, Texas.

Our proposal is “ePortfolio 2.0: expanding our views of portfolio”

Abstract (50 words max)

George Hotz’ blog chronicling his iPhone hack demonstrates students can collaborate world-wide and create portfolios that make learning visible. Our research suggests students and faculty are equally adept at giving criteria-based feedback. Portfolios capturing learning process combined with criteria-based feedback have implications for teachers, course design and LMS platforms.

Research Results

Ater-Kranov, Ashley and T. Desrosier. Raising the Bar: Communicating High Expectations and Getting Results. Poster. Washington State University Academic Showcase March 2007.

Cho, Yoon Jung, A Ater-Kranov, and G Brown. Faculty Attitudes about ePortfoios: A study for the National Coalition for ePortfolio Research. Poster. Washington State University Academic Showcase March 2007.

Hotz, George. Finding JTAG on the iPhone. Blog. http://iphonejtag.blogspot.com/ accessed Sept 10, 2007

WSU ePortfolio Contest. Making Learning Visible. Website. http://ctlt.wsu.edu/eportgallery accessed Sept 10, 2007

Session Focus

Portfolios have been used in several ways beyond being showcase of best work, including documentation of learning growth and for personal reflection. In the Spring of 2007, the Center for Teaching Learning and Technology at Washington State University hosted an ePortfolio contest that asked students to document their learning growth. The result was a rich array of evidence of learning, and a wide range of portfolio documentation.

More simply, a blog can be understood to be a learning journal, and with suitable summary posts, might serve as a portfolio. George Hotz blog of the hack of the iPhone is one example that illustrates one person’s informal but substantial learning journey enhanced by a collaborative community.

Personal Learning Environments (PLE) integrate both formal and informal learning episodes into a single experience and often have a blog at their heart around which the user assembles a range of resources and systems to create a personally-managed space.

To the extent that users open their PLE space for inspection by others it becomes a multi-faceted journal that makes learning processes and outcomes visible. When the user presents that log of learning evidence the PLE becomes an extended portfolio view.

A key facet of the blog or PLE is that the user seeks critical feedback and collaboration on their learning objectives, which typically involves the creation of social networks that cross institutional boundaries and are intended to place the learner at the central node in a learning community. We have evidence that demonstrates that students are at least as adept at faculty at providing criteria-based feedback, which opens the potential that giving of critical feedback can be scaled much larger than what faculty alone can provide.

This presentation will explore the blurring of the lines between portfolio, blog and personal learning environments and a parallel blurring between novice and expert feedback when novice feedback is appropriately scaffoled and guided. We will invite participants to join in the exploration and the implications they have for teachers, course design, assessment of learning, and IT planning around LMS and other supporting tools.

We are going to be working on this (sketchy) proposal for Active Learning Strategies in the session and welcome feedback:

The audience will collaborate in an analysis and deconstruction George Hotz’ blog (ne portfolio) of the hack of the iPhone. Then the audience will participate in a collaborative criteria-based rating. Audience data about itself will be shared and discussed within the threads of the presentation. Following the session, the audience data will be posed for later review by the audience and others.