Archive for the ‘Assessment’ Category

Notes on “tag clouding” Twitter

Friday, August 13th, 2010

I’m working on my HASTAC/P3 presentation. I want a back channel where the audience can provide feedback/ assessment of the session. The idea is to see if the audience can give feedback with a combination of a controlled vocabulary and free tagging. (As opposed to using a big rubric.)

I looked at a couple Twitter-centric tools with the thought that the audience can readily come prepared to Tweet from a range of mobile devices. What is needed is a cloud of the tweets @UserID and some coaching for the audience to tweet with tags.

tweetcloud.com/ embedded in their web page. I used @nilspeterson as a search and it says there isn’t a cloud.

mytweetcloud.com/ will get the hashtags from a user ID. UserID nilspeterson worked, This is getting the content that the user tweets, not what is tweeted @UserID.

So to get around the above problem, you need the RSS of the tweets @UserID and that is protected by the Twitter user’s password. Yahoo Pipes can retrieve the @UserID content by passing in the required authentication. You need to embed username:password in the URL used in Pipes. (not totally secure, but workable).  Pipes will do a reasonable job filtering tweets –for example, I can get them for a date range. Here is the pipe I’ve created for user nilspeterson
http://pipes.yahoo.com/pipes/pipe.run?_id=c0f7b2078b6ad4aa8e835dfdde927644&_render=rss

wordle.net will take the RSS from Yahoo and makes a handsome display (below). The @UserID comes thru big (duh!), but this might not be a problem — it documents who is getting the feedback. The StopList is hardwired and can’t have any additional words added, so blocking the @UserID would need to happen in Pipes. Wordle requires using it on their page (a setup issue and no embed), they say You may not copy or redistribute the Wordle applet itself under any circumstances. Refreshing the page is a pain and not practical. Need another tool that can imbed.

IBM ManyEyes won’t work because you need to upload a static dataset to them.

www.tag-cloud.de can create an embeddable Flash from the feed. It makes a pretty handsome cloud, and in you can link from words in the cloud to web pages., but they process the tags in the URL once so the resulting cloud its static (no auto updates unless you do it on their site).

Diverse Group Tag Cloud (DGTC) is a WordPress plug in. Its not certified in version the version 3.x of WP used by NilsPeterson.com. First attempt with it does not seem to work.

Candidates
TagCrowd.com Will take the RSS output from Yahoo. It has a customizable stop list, which will be needed to prune the junk from Yahoo (if I can’t get Yahoo Pipes to do the pruning). Takes awhile to get a personal stop list to show up in the pick list on the site. Image below is unfiltered by a stop list to show the problems. There is an embed HTML option, which would allow getting the cloud off their page — I assume it updates when the page loads. This is fairly promising.

Google Docs spreadsheet. In the top cell put the function =ImportFeed(“http://news.google.com/?output=atom”).  Then need to use Google’s word cloud gadget to make the rendering and publish the gadget and display on a web page (see below). Need this to refresh on a regular basis.

Alternative (non-Twitter) Method
An alternative would be to skip Twitter and use a Google Docs form. This avoids the need for Yahoo and for stop lists. It would still work with many mobile devices.

Whats up with Google Docs?
Google is moving to a new version of Spreadsheet. The new version does not support Gadgets (even Google’s own). The old version does, but its flaky. For today, the focus needs to be on the non-Google solutions.

Google Workaround
So, what about using Google Forms to fill a spreadsheet, publish it, take Yahoo Pipes to pick it up and feed it to TagCrowd? That seems like a reasonable next experiment.

A Waterloo for Publishing or for the University?

Friday, June 25th, 2010

Cathy Davidson raised a series of issues in her reaction to a lawsuit known as Cambridge University Press, et al. v. Patton et al.

“My larger point?  We are in a confusing and damned-if-you-do-damned-if-you-don’t moment for publishing.  Scholarly publishing loses money.  Scholars who do not publish (at present) lose careers.  How do we balance these complex and intertwined issues in a sane way?  That is our question.”

Jim Groom has some thoughts on one aspect of this question — the issue of credit, or reputation, generated by journal publication:

“And, often times, but not always, that class [of author] is accompanied by three letters after their name and a long list of publications in similar journals which often, but not always, gives them entrè into the journal in the first place. Is this necessarily bad? No. Does it help certain ideas circulate to a particular audience? Yes. Are we putting too much power in the hands of these journals by reacting this way to the idea of credit? Absolutely.”

And as a result of highly valuing publishing in journals, we have created a system that is producing an avalanche of low-quality research.

Cathy’s question makes me think of the work of physicist A. Garrett Lisi, who is working outside the traditional academe system and who’s practice gave me insight to understand other ways of thinking about credit/reputation and also about gathering feedback for learning from a community:

“Lisi is developing social and intellectual capital by his strategy of working in public, and has posted a “pre-print” of some of his work in the highly visible High Energy Physics – Theory section of arXiv entitled ‘An Exceptionally Simple Theory of Everything.’

“The Wikipedia entry on Lisi’s paper gives a picture of how the work has generated social capital and become a focus of theoretical debate. The paper has been accumulating peer reviews (in the form of blog posts) and a number of citations including in refereed Physics journals as well as comments on the social news website Reddit.com.”

So, I think Cathy is pointing us to a multi-faced conversation about moving beyond the University (see John Seely Brown or Charles Ledbetter or Clay Shirkey) each of whom is exploring forces that I think will probably address Cathy’s “damned-if-you-do-damned-if-you-don’t moment” by rendering traditional publishers in academe irrelevant.

In her post Cathy says

“Shouldn’t we be teaching the genre [scholarly monograph] to our undergraduates (because we believe it is intrinsically worthy enough to determine someone’s career in the academy) as an estimable form? … [If we] require at least one scholarly monograph in every English class, … we show respect to the genre we say that we live by and we give back something to the publishers who, right now, are expected to publish our work but who experience abysmal sales of it.”

Here, I think Cathy’s comment brings academic publishing into the national conversation about university accountability to stakeholders (the students and those investing in them). Molly Corbett Broad wrote in the Chronicle about the political landscape for accreditation and accountability “The administration has already indicated a willingness to take action when it believes that higher-education institutions are not adequately serving students’ interests.” (alas it is “premium content” that you may not be able to access) I think Corbett and Shirkey are talking about forces that may render more than just traditional academic publishing irrelevant.

It strikes me that the scholarly monograph, as a discipline for the mind, could be useful, but it might not be a form “worth studying in every English class.” It might be more useful for students to be developing skills in peer-to-peer pedagogies, based in forms like blogs and wikis, that operate in a context of information abundance rather than to be studying a form based on information scarcity and expensive publication; a form that will not be used by most students in their future careers.

Why do I focus on credit/reputation and legitimate peripheral participation rather than the academic monograph in a conversation about accountability for learning outcomes? Because, I think discovering conversations, contributing and getting feedback are important aspects of peer-to-peer learning beyond the university. Good feedback is a tool for growth, both for the author and for the community of lurkers (see John Seely Brown on legitimate peripheral participation.)

As to Cambridge University Press, et al. v. Patton et al., I think it will be a passing blip, swept away by much larger forces transforming learning.

PS. And thinking about feedback and peer-to-peer learning is why I’m posting this in my blog  (  )  and then cross-posting it as a comment in Cathy’s blog at HASTAC. HASTAC’s blogs do not appear to support Trackback, so  I can’t comment to Cathy in my blog, and consequently I need to post a comment in hers. Which means I need to create a HASTAC identity (see these objections to creating accounts everywhere). Further, a HASTAC comment does not track back to the people I cite – making it even harder for them to discover and join the conversation.

I missed the bus – thoughts on indirect assessment

Tuesday, March 9th, 2010

I missed the bus to work today. I knew time was tight as I was going out the door. As I went along, I gained confidence I would make it, because I saw one of the school buses that I usually meet. And some kids waiting for another school bus (I usually see two school buses). And I saw the city buses at the bus transfer point (but I could not see my bus meeting them).

ALAS, none of that indirect evidence measured where my bus was on its route. As I rounded the Kibbie Dome I saw my bus picking up a rider at my stop and heading away.

I’m noting this as part of the conversation I’m part of about direct and indirect evidence of student learning outcomes. This is an example of the failure of relying on indirect evidence.

Different conversations about what is important

Thursday, July 16th, 2009

In IJ-SoTL – A Method for Collaboratively Developing and Validating a Rubric Allen and Knight discuss their experience validating a rubric with two groups: faculty and industry professionals. They report:

Faculty weights differed markedly from the professionals’ results [see table 4 in the article]. Faculty considered category 2 (In the headline/lead combination, Is the message clear and compelling?) and category 4 (Does the news release use a convincing journalistic style?) the most important. Categories 1 and 5 received the lowest weighting.

We have seen similar results in a course where we asked a group of faculty and a group of industry professionals to rate student work and also to rate the assignment used to assess the student work.

In both examples, the faculty seem more focused on formalisms and the professionals on the aspects of the task that lead to practical success.

Common Reading & Open Learning Communities

Thursday, May 28th, 2009

Thanks Bill Marler for your offer to support Washington State University’s Common Reading program after it got caught in a recent controversy regarding the book Omnivore’s Dilemma. See also developing Facebook action related to the topic.

From his blog, I can tell Marler has some appreciation of Web 2.0 as a life-long collaboration and learning strategy. This whole event is an example of how having a curriculum open to community review can improve learning outcomes. Searching in Google for “WSU Common Reading” shows that the event lit up a problem-solving community with multiple perspectives but overlapping interests in this topic; a community that produced the resources to sustain a learning opportunity.

WSU’s Center for Teaching Learning and Technology has been exploring how to help students learn in, from, and with such communities with projects like the Microsoft co-funded ePortfolio Contest. A variety of lessons can be learned from that project, including thoughts on how to transform the traditional gradebook by extending the idea of grading out into the community and making it a process for collecting community feedback on student work, AND the assignments that created the work, AND the program goals that shaped the assignments. I think this represents the way WSU needs to move forward with a Global Campus concept.

A lesson in driving up readership

Tuesday, May 5th, 2009

On Friday, April 24 the Chronicle’s Wired Campus ran an item on the failure of U. of Michigan’s Online Teaching-Evaluation System. The article was hot news because of the scale of the player and the scale of the failure. I posted this comment near midnight Sunday, April 26:

My comment on the article

This drove a large spike in readership of the associated resources on April 27.

Page views for WSUCTLT blog

And examining how readers got to the site we see they came from several related pages in the Wired Campus article.

pages that referred to WSUCTLT

which brought readers to these pages

pages viewed as a result of the comment

Implementing Obama’s 100 Hours of Service Plan

Monday, November 10th, 2008

At the Obama/Biden transition site, Change.gov, there is description of a universal voluntary service plan. This includes an idea to “establish a new American Opportunity Tax Credit that is worth $4,000 a year in exchange for 100 hours of public service a year.” And a goal to expand service-learning in the nation’s schools with “a goal that all middle and high school students do 50 hours of community service a year.”

That sounds exciting, but the downside in fraud and corruption is easy to see. To see the problem, look at who can certify my 100 hours of service to the Feds for my tax credit and what I might be doing for them to earn that certification. Beyond those limitations, where is the motivation for the 101’st hour of service? Where is the motivation to quality service? How does this service get leveraged into even greater gains?

Its still a great idea and there is a simple, and powerful, extension to it. We have been pointing at examples, and building some ourselves (see below). It requires a partnership between the school (its teachers and curriculum), the problem being addressed, and the community in which the problem is situated. Here is an example of real problem solving in service to community in a high school in Minneapolis, MN. Expeditionary Learning Schools/ Outward Bound are consulting with schools along similar lines.

For example, a school could set a goal to harness the interests and expertise of the school’s community (students, staff, parents and alumni) to address real world problems encountered by communities both locally and globally. Its curriculum would be designed to collaborate with community members – institutional, local, or global – to identify a problem, explore solutions, develop a plan, and then take steps toward implementing that plan. Students would engage these challenges as service learning. The outcomes of their work would be readily documentable using ideas like Gary Brown’s Harvesting Gradebook. But more than just documenting the student work, the process would have transformative impacts on the educational institution also, far more profound than the Collegiate Learning Assessment (CLA) or the No Child Left Behind (NCLB) standardized testing.

The importance of this strategy is its immediacy. It can be launched rapidly (its already started), it can target needs in communities without a large bureaucracy to decide what the needs are. It can tap local resources and world resources at the same time. It scales well.

Examples

Harvard Program in Networked Governance “The traditional notion of hierarchical, top down, government has always been an imperfect match for the decentralized governance system of the US. However, much of what government does requires co-production of policy among agencies that have no formal authority over each other, fundamentally undermining the traditional Weberian image of bureaucracy.”

United Nations Volunteers “The paper argues that volunteering, like social activism, can be purposeful and change-orientated. Volunteering can be directed at influencing agenda-setting, policy-making, decision-making and representation, and is also an important mechanism for promoting empowerment, personal transformation and social inclusion.

The paper also highlights the complementary and supporting roles that volunteering and activism play in fostering participation. For example, social activism plays an important role in providing leadership, defining areas for engagement and mobilising individuals.”

EDUCAUSE Tower and the Cloud “The emergence of the networked information economy is unleashing two powerful forces. On one hand, easy access to high-speed networks is empowering individuals. People can now discover and consume information resources and services globally from their homes. Further, new social computing approaches are inviting people to share in the creation and edification of information on the Internet. Empowerment of the individual — or consumerization — is reducing the individual’s reliance on traditional brick-and-mortar institutions in favor of new and emerging virtual ones.

Land Grant 2.0 “Dramatic shifts in the economy associated with the rise of globalism call into question the traditional ways in which land-grant institutions have defined their roles in contributing to economic and social well-being. Since the assets most needed for global economic viability – a base of innovation, talented people, and ubiquitous connectivity – are core strengths of universities, it is fair to ask how these institutions can more holistically engage with economically distressed regions to build critical innovation economy competencies.” see also University of Illinois Global campus.

WSU ePortfolio Contest “The goal of the 2007 – 08 WSU ePortfolio Contest was to harness the interests and expertise of the WSU community to address real world problems encountered by communities both locally and globally. It called upon contestants to collaborate with community members – institutional, local, or global – to identify a problem, explore solutions, develop a plan, and then take steps toward implementing that plan.” See also specific winners: Margo Tamez, Kayafungo Women’s Water Project

ThinkCycle This is the thesis to study the (now-defunct) ThinkCycle project exploring “How can we create an environment that encourages distributed individuals and organizations to tackle engineering design challenges in critical problem domains? How should we design appropriate online collaboration platforms, support learning, social incentives and novel property rights to foster innovation in sustainable design?” This concept can be broadened out beyond the engineering domain to other problem domains. An example is National Collegiate Inventors and Innovators Alliance, that says of itself, “The NCIIA works with colleges and universities to build collaborative experiential learning programs that help nurture a new generation of innovators and entrepreneurs with strong technical and business skills and the tools and intention to make the world a better place.”

A final example is Talia Leman, at age 10 organized fund raising for Katrina relief, and has since started RandomKid.org to help other children become social entrepreneurs. In this NYTimes OpEd notes “Frankly, these kinds of initiatives have a mixed record in terms of helping the poor in a cost-effective way. But they have a superb record in enlightening and educating the organizers.” which may be exactly the outcome that is most important from some of these efforts. The Times piece also points out some other efforts along similar lines.

Setback for 21st Century Resume

Sunday, July 20th, 2008

I’ve been trying to make headway on understanding a 21st Century Resume, thinking that it should not be your father’s paper resume. But recently I’ve had a couple “send me your resume” requests, so here it is in all its paginated glory: Nils S Peterson CV.

I’m choosing categories for this post. Milestone is obvious for a noted setback. Reflection-in-action seems an appropriate guide to what I should be doing, but it is Assessment that caught my eye. Clearly, it is assessment that is the goal of keeping a document like a CV. In my experience reviewing job candidates, a CV is a cumbersome tool for assessment.

So, I have added a new section to my CV, at the bottom, Community Activism. My wife’s summary: you defeated WalMart, Saved the Bus, and Chartered a school. But the reflection, and I hope your assessment, is what did I learn doing those things?

I joined the NoSuperWalmart effort in December 2005 hoping to get a chance to apply my ideas about blogs to community organizing. There is a blog, but it did not become a key piece in the communication strategy. I found myself in a role of spokesperson and part of the inner leadership circle, but the communications strategies that carried the day were email, newspaper ads, and handbills.

A year later, I got a better chance to explore the nexus of online and local community organizing. The conclusion that I draw from this work is that the web can be integrated into a local community’s political organizing. It requires effort to make the site participatory (with multiple perspectives) and authoritative (with data that has been collected about the problem and is open to inspection, support or refutation).

My most recent activism has revolved around gaining a charter for Palouse Prairie School in Moscow. This work has been conducted mostly offline, but its given me a chance to think about how to merge the content management of two blogs and Google Calendar into a web site. It not right yet, but its a step beyond the NoSuperWalmart effort.

What I think I’ve learned in those efforts is feeding into a series of blog posts I’m co-authoring regarding transforming the grade book. What we are arguing is that a community assesses its members (and a university education should assess students) by applying a set of broadly agreed criteria across a span of the member’s work. We are exploring the mechanisms by which the learner gathers, organizes, learns from, and demonstrates that assessment. The result will not be a CV, it will be a portfolio and the public assessment of that portfolio. Not a showcase portfolio, but a learning portfolio.

The CV is a showcase, and not even the best one, in that it is a catalog of everything without organizing narrative. What would be more interesting it seems is to showcase an accomplishment and the learning history leading up to it, as George Hotz did in his blog/portfolio on cracking the iPhone and Margo Tamez did in her portfolio on her efforts to prevent a US-Mexico border wall.

In Wikipedia they don’t care about your credentials

Monday, July 14th, 2008

In Wikipedia your contribution can be respected even if you are anonymous. The Wikipedia community works because it has a set of public criteria used by the community (and debated within the community) for judging all work. For example, there is a policy around deletion of pages and procedures for implementing it.

In thinking about transforming the grade book, I am concluding that we have reduced the teaching and learning proposition to one core idea: the public presentation of evaluation criteria, the public application of the criteria to a given learner’s work, and a public means for the criteria and their application to be negotiated by a community over time.

I think this is what Stephen Downes is saying in Open Source Assessment, you don’t need a curriculum created by experts, you need this core tool, the assessment.

But, key to making this work is to directly assess what you want learners to be doing. If you use an indirect measure, like a standardized multiple choice exam, what you will get is learners who master that test but may not have the skills and knowledge that are actually desired. Patricia Cross suggests that what you measure and value is what you will get more of from learners.

And, once you have your assessment underway, you need to constantly redefine it by holding the results up against what you would like to have happening. The key is not how well students do on our exams, its how well the exams promote what we want students to do.

FERPA and Learning 2.0

Thursday, July 3rd, 2008

John Thomson, writing in response to Martin Weller’s ideas in SocialLearn finds Web 2.0 an awkward fit for higher education and gives among his reasons for why universities can’t/won’t change:

“FERPA has everybody scared. The purpose of the law was to protect student’s personal and grade information from things like the grade list on the door and prying parents. Yet fear over having students work on external commercial systems, which are largely secure from hacking and violate privacy only on the aggregate level of data, causes hesitation from using Web 2.0 systems or attempting to form partnerships with their owners.”

Having students do learning activities in public seems completely reasonable in theater or music where performance to audiences is part of the authentic activity. And I doubt FERPA has a problem with a school play or concert listing the names of the performers in the program even if they are also the members of a class preparing the performance. The difference for me is the authenticity. Forcing a student to blog on a class discussion or post a paper on the causes of World War II, where the activity is school work and not authentic performance is where I have a problem.

If the instructor is giving a grade for the activity, I believe that is FERPA protected, but in a public venue like those above, the audience did its own assessment (not to be confused with grading) and that too is a required part of the authenticity of the experience.

For me, the interesting implications of Learning 2.0 come in taking a transformative view of the whole learning enterprise. Recently, we have been writing on transforming the grade book. The activities we propose, especially in the second and third variations probably could not be conducted within a closed learning environment — they are authentic engagements with a community and need to be on the open Internet. I see no FERPA issue here.