How can we be sure the speakers we’ve accepted will deliver? How can we ensure they get the feedback they deserve (positive or negative)? Would using a site like SpeakerRate improve the situation?

Scott Berkun's Confessions of a Public Speaker

Berkun’s book isn’t really in the “how to become a better speaker” genre, though I think committed speakers who read it will undoubtedly get better. I’d have subtitled it “What Makes Consistently Successful Public Speaking Nearly Impossible.” Berkun covers all kinds of interferences which prevent talks from being successful, only some of which are under the direct control of the speaker. Room configuration, human physiology and instincts, poor alignment between speaker and occasion, the disconnect between what organizers need and what audiences want, and the sheer difficulty of teaching anyone anything (let alone teaching several hundred people the same thing all at once) all figure into what makes talks go poorly.

(This not incidentally makes the book great reading for conference organizers, managers who send employees to seminars, and people who attend talks as well as potential or frequent speakers).

In the chapter on “things people say” Berkun has a whole section on “Why most speaker evaluations are useless.” He points out that:

Most organizers never bother to collect feedback from the attendees, and of those who do, often it doesn’t get passed on to the speakers. It’s a shame because it’s most appropriate for the organizers to share feedback with the speakers; after all, they invited them to speak, so technically the speakers work for the hosts. But being as busy as they are, the organizers don’t always communicate the data gathered back to the speakers. They ask the good speakers to come back and leave the rest to figure out life for themselves.

Even on occasions where feedback is gathered and shared with speakers, it’s still not very useful as it provides no context. Berkun shares a real example in which his talk was rated by attendees on a scale from “very dissatisfied” to “very satisfied.” But, he points out, just knowing the distribution of the attendees who bothered to fill out the form (129 out of 500 or so in this case) along that scale doesn’t do any good without a comparison:

But the single most valuable data point is how my scores compare to other speakers. Without it, this feedback is useless. Perhaps my scores are the worst of all score in the history of presentations at this organization. Or perhaps they’re the best. There is no way to know.

Berkun points out the feedback speakers really need:

• How did my presentation compare to the others?
• What one change would have most improved my presentation?
• What questions did you expect me to answer that went unanswered?
• What annoyances did I let get in the way of giving you what you needed?

Giving speakers feedback based on this set of questions would be much more likely to improve their performance.

(Aside #1: something about the annoyances question still bugs me. When I was teaching I used to talk about “productive frustration” – which naturally comes from learning something new – as opposed to “non-productive frustration” – which comes from poorly written assignments, badly planned logistics, and other stuff not directly related to the hard work of learning. I think this is the sense Berkun’s after for “annoyances” here but I don’t think it quite gets across. Maybe something more like “What did I do or fail to do that got in the way you getting what you needed?”)

Which brings me to SpeakerRate. It’s a site which lets users give direct feedback on speakers and their talks. It feels to me mostly aimed at speakers themselves, though the site says:

SpeakerRate is a community site for event organizers, attendees, and speakers.

• Event organizers can find speakers, learn about talks they’ve given in the past, and determine who would be a good match for the event they’re organizing.
• Event attendees can provide constructive feedback to speakers, track the talks they’ve attended, and research upcoming talks that they might attend.
• Event speakers can get valuable constructive feedback directly from attendees and find out how they can improve their content and delivery for their next talk. They can also establish a SpeakerRating, which will help them earn future speaking opportunities.

On the surface, this seems to me a great thing: let those who attend talks provide feedback directly to speakers, cutting out the need for event organizers to collect and manage feedback. Thinking about it as a speaker, frankly, it’s a bit frightening. In much the same way that I’m grateful there was no RateMyProfessors when I was teaching, I worry that this might encourage or facilitate the worst kinds of superficial feedback and speaker trashing. What if someone with an axe to grind starts leaving negative comments? Would other attendees come to the rescue of a speaker thus trashed?

SpeakerRate.com Homepage

(Aside #2: As a consultant working with companies who worry about what will be said about their products in social media, it’s quite easy to dismiss their concerns. The conversation will happen anyway, and you can’t be so invested in your belief that your products are superior that you ignore real feedback from real people. Funny how difficult it is to apply this same line of thought when the prospect of being rated on speaker rate arises. Why does this scare the crap out of me, while SlideShare doesn’t?)

Speakers (or, apparently, organizers) set up a page for each talk at SpeakerRate. Users of the site are then given the opportunity to rate that talk and leave comments.

SpeakerRate form showing Delivery and Content

SpeakerRate comment form

The problem is that I’m not sure how effective a simple rating on “Content” and “Delivery” plus a box for unprompted, free text comments is at conveying useful feedback. Wouldn’t it be better to offer a prompt other than “Leave a comment”? Maybe even allow speakers to ask specific questions of their own?

I suppose that the “speaker rating” which comes from some aggregate measure across multiple events would give you some rough sense of how different speakers compare.

But Is the “Speaker Rating” (a single number, which is presented to two decimal places implying a fair degree of precision if not accuracy), enough to really validate a speaker’s abilities?

Isn’t much more context necessary to really understand what a speaker has to offer an audience? Speakers who might be great in one context (a highly technical demo or how-to in front of 30 experienced developers) might be horrible in another context (a keynote to an audience of varied levels of experience).

Have you used SpeakerRate? Have you found it useful, as a speaker, an event organizer, or event a prospective event attendee? Would you recommend it to your neighborhood event organizer?

(A brief aside on “writing in the disciplines” or “interdisciplinary writing” programs: Most college composition courses take one of two approaches: the either ask the students to write about literature or they take a topical approach, choosing topics in which they believe the students will be interested. The former approach assumes the students are interested in what the instructor is interested in, as many of these courses are taught by graduate students or professors whose real interest is something literary. The latter creates an environment in which the ostensible topic of the writing is an artificial academic context usually dealt with very superficially, since the real purpose of the course is the writing, not the topic. IWP and programs like it try to solve that by situating the students and the instructor in a real academic context: an existing undergraduate course in another discipline. The students’ writing tasks are situated in an authentic environment, where they are actually trying to understand and enter an ongoing academic discourse.)

I was reminded of the importance of context (and my love for the insights of the social sciences broadly) this weekend as I watched two videos from an event Microsoft Research held at MIT, to celebrate the launch of their new lab in Cambridge:

• danah boyd on socio-technical practices (streaming video)
• Bill Buxton on “Designing Experience” (streaming video)

(Sorry for the mms links – you can rip them via mplayer if you need to watch in offline mode, but I think reposting them here would be considered a copyright violation).

Both really celebrate / argue for what we might call the situatedness of technology design: the ways in which an understanding of the cultural context of technology use needs to be brought back into the design of those technologies and how non-engineering approaches (from the social sciences in danah’s talk and from Design in Buxton’s talk) can help to provide that context.

danah boyd (capitalization hers) has built a (well-deserved) reputation for being a smart ethnographic observer of teen culture as it intersects with what we now call social networking, having spent many years embedding herself in both the online networks and (importantly) the social contexts in which real teens engage with those networks.

In this video, she talks about the situatedness of what the industry calls “Web 2.0″ within a broader social and cultural history which includes moral panic about teens and adult strangers and changing political geographies which eliminated / privatized traditional public spaces.

She outlines several factors which are inflecting teen behavior (ways in which the new technology both has an impact on and is impacted by the behavior):

1. persistence
2. replicability
3. scalability
4. searchability

And some dynamics which result from these factors:

• invisible audiences
• collapsed contexts
• public == private

For me the key in the video is less the specific issues she discusses (which if you’ve followed her work aren’t necessarily new) but the broader context in which she places the work: how technology creation and design needs to take into account the social contexts in which technology use is always necessarily embedded.

In other words, technology designers and makers can’t really hope to be fully successful without engaging the uses to which their technologies are put. Not that they’ll know in advance what all social uses will be (in fact the most interesting ones are generally those least anticipated) but that they need to remain engaged and active in the kinds of understanding on which social sciences have traditionally focused.

Eminent researcher, designer, and teach Bill Buxton‘s talk, which followed danah’s, actually ends up complimenting it well. He basically makes an argument for bringing “design thinking” earlier and more consistently in the design process for technology products. He also makes a compelling case for doing a different kind of “usability testing,” with two key additions:

1. Showing users multiple prototypes/sketches. Users recruited for testing will rarely be critical of a prototype when shown only one solution, but will provide stronger critiques when shown multiple solutions. This is due in part to a reluctance to criticize the team running the tests, who are presumably invested in the solution. When users were shown three alternative approaches they were much more forthcoming in their criticisms, as they recognize the design team haven’t “solved” the problem.
2. Ask users to sketch a solution. It’s long been a truth universally accepted that users don’t provide solutions: they know the problem, but don’t know how to solve it. Buxton shows that by giving users a vocabulary and toolset which enables them to communicate design solutions, they can and will produce more innovations.

As with boyd’s talk, though, the importance for me of what Buxton’s talking about isn’t a specific set of changes to usability testing, but a broader focus on the kinds of skill sets teams need to encourage, facilitate, and perhaps even require. It’s about what he calls “design thinking” and collaboration among researchers and designers with heterogeneous specialties. He talks specifically about bringing together cognitive psychologists, sociologists, graphic designers, interaction/industrial designers, and software engineers on teams to really cultivate the kind of productive discussion necessary to fundamentally change how technology solutions are imagined.