CALL @ BU

TESOL and the Unconference

2008-06-12T10:37:00.009-04:00

Conventional conferences are models in poor knowledge transfer

The 2008 conference of Teachers of English to Speakers of Other Languages (TESOL) in New York City should have sounded the death knell of the conventional educational conference. Sadly, it didn't, and there promises to be more debacles to come as the membership dues bill in my mailbox seems to indicate.

Such bloated, sponsor-driven conferences—packed as this one was, again, into the undersized New York Hilton and cramped, labyrinthine meeting rooms of the Sheraton across 54th Street—provide gasping testimony to what models of ineffective instructional design and inefficient knowledge transfer such meetings have become.

Evidence of bad design

Logistics. Venues that do more to restrict free movement of people than a lifetime of bacon double cheeseburgers restricts arterial blood flow—too many non-essential elements clotting together.

Lack of subjectivity in planning. Meeting room assignments seemingly made at random (that is, by hotel staff) without regard for reasonable expectations of the popularity of one session over another.

Hype. Session abstracts, written as a part of proposals and printed in conference programs, invariably overstate the goals of the session, representing an unrealistically ambitious agenda for the allotted time and all but guaranteeing disappointment.

Ignoble motivation. Presenters may be motivated by careerism more than having something to say. Such sessions are little more than, for example, a notch on a grad student's c.v. before she hits the job market. Perhaps serving the interests of enlightenment even less are the teachers presenting to justify reimbursed travel—the pseudo-scholarly junket. (In all fairness, considering the relative low pay of language teachers and those working for academic publishers, this is perhaps the most understandable motivation.)

Poor instructional design. The irony, nay, tragedy, of teachers perpetuating bad instructional design seems lost on no one but the presenters themselves.

What might make conference attendance all worthwhile, however, are a few unscripted conversations with peers or counterparts who may have sat through the same disappointing sessions, combined with a little time to reflect on fresh possibilities courtesy of being removed from the quotidian demands of one’s job, not to mention eating well on an expense account.

Power to the people

Imagine a conference free of bloated administration (and its like-sized fee), free of pretension, hype, and poorly designed and delivered bullet-point recitations that rarely risk enlightenment or stimulate thought or interaction. Imagine the TESOL unConference. This "people over process" approach, not wedded to a pre-determined agenda, can be seen in action at any number of technical conferences.

RedMonk’s Free Conference at JavaOne CommunityDay
BarCamp
BrainJams
Wikipedia entry

A description of the unconference from Digital Web Magazine:

Unconferences are gaining popularity in the high-tech community as self-organizing forums for idea sharing, networking, learning, speaking, demonstrating, and generally interacting with other geeks. The unconference format is based on the premise that in any professional gathering, the people in the audience—not just those selected to speak on stage—have interesting thoughts, insights, and expertise to share. Everyone who attends an unconference, such as those put together by organizations like BarCamp or BrainJams, is required to participate in some way: to present, to speak on a panel, to show off a project, or just to ask a lot of questions. As an event, the character of the unconference falls somewhere between that of a bazaar and that of an intellectual salon. It is, to borrow a phrase, a free “marketplace of ideas.” There are no themes or tracks to guide you, as in a typical conference; the whole event is centered on what might be called the discussion group. The ad hoc nature and the low cost of this forum (they’re usually free, compared to the hundreds of dollars needed to attend some industry gatherings) make the unconference accessible to many.

Separating Content from Form

2008-03-25T09:37:00.014-04:00

One critical mistake teachers and trainers make involves the failure to separate content from form. This fundamental error in approach to complex topics frustrates learners and impedes understanding. Teachers and trainers are at times so narrowly caught up in their subject area that they lose sight of perspectives outside the field and are unwilling or too intellectually lazy to take the time to step back and address the challenges of someone for whom this subject is completely new or perplexing.

Avoiding cognitive overload

In an analogy that writing teachers will appreciate, process writing separates the focus on form from the focus on content. Content—what students say, their ideas, examples, and organization—is, initially, more important than form—how it's said, the particular structure and words they use to express them. Thus first drafts of papers should receive few or no comments on form (e.g., word choice, grammar, punctuation, spelling), because this would distract the student from concentrating on content. Furthermore, why correct form in a draft when that form will change to express reworked content anyway? It would be like touching up aesthetic blemishes in a primer coat of paint. It's a strategy of divide and conquer, of consciously avoiding cognitive overload, which is when your brain cries "uncle!" after trying to do too many things at once, and doing none very well.

Take another common example from classes across the college campus: making web pages. Teachers in many disciplines, from management to education, have students create web pages. Outside of a course in web development or design, why assign such a project? Besides keeping students up into the small hours IMing their frustration with the ambiguity and lack of structure, what's the benefit of such a task? True, it's challenging and requires problem solving and teamwork. Unfortunately, such assignments typically follow nothing but the most cursory introduction to the nature of document design on web pages, information structure of web sites, and how different this is from page design and the organization of printed matter. Then there is the enormous technical challenge. Was the tool designated to carry out the mechanics of this creation, such as Adobe Dreamweaver or Microsoft FrontPage, similarly introduced and demonstrated? Not likely.

Editing a page in Dreamweaver

Editing a post in Blogger

Manipulating form does not imply content knowledge

The web is a complex medium, and development tools like Dreamweaver are technical and deep. As with another content tool common in college, PowerPoint, the ability to mechanically manipulate appearance and objects (the how) does not confer or imply knowledge about the information product (the what). Students come to appreciate this quickly. Why don't the teachers assigning these projects? (See related post on the failure to teach presentation design and delivery skills.)

This oversight, in addition to its risk of squandering student time and energy and achieving no expressed learning objective (or at least one billed on the syllabus), fails to exploit a teaching opportunity to provide a proper grounding in some basic concepts.

Rational web design counts as prerequisites many such concepts:

Design

basic instructional design (pedagogically sound approach to presenting multimedia content to teach or inform):
user interface (navigation, accessibility)
interactivity (prompts, feedback)
linear vs. random access (paths to information)
basic information design (layout):

organization (hierarchy)
ordering and embedding informative elements
graphic design (color, iconography, typography)

Technical

page elements (text, graphics, links)
layout tools (tables, alignment, images, CSS)
file types (GIF, JPG, PNG, PDF, HTML, CSS, JS, etc.)
file characteristics (e.g., color palette and transparency of graphics)
integration of dynamic elements (RSS, SSI)
file naming conventions (illegal characters, use of extensions)
file management conventions (e.g., relative vs. absolute paths)
file transfer (FTP)
file hosting options (sponsored vs. fee-based)
accommodating a variety of user capabilities (different connection speeds, operating systems, browsers, plug-ins, displays)

These and other relevant issues are rarely given their due outside of courses devoted to web development. And no wonder. This is time-consuming stuff, and many people have a great deal of difficulty with it.

Web 2.0: Content breaks free of form

One of the tangible benefits of what has become known as "Web 2.0" is the ability of non-technical users to easily create content on the web, most notably through wikis and blogs, like this one, but also photo sharing sites like Flickr and Picasa, collaborative/online writing tools like Google Docs or Buzzword, social networking sites like MySpace or FaceBook, and—to a lesser extent because of a different business model (fee-based not ad-driven)—ePortfolios. Web 2.0 is the term collectively, sometimes loosely, applied to a range of capabilities non-technical users can acess through a web browser alone. These "web apps" take the place of, in many cases, complicated and expensive software installed on the user's computer ("local apps," such as Microsoft Office and Adobe Dreamweaver). For the seminal blog post expounding Web 2.0, see Tim O'Reilly's "What is Web 2.0: Design patterns and business models for the next generation of software."

One important implication of Web 2.0 capabilities, then, is how it facilitates the separation of content from form. The vast proliferation of blogs, wikis, and MySpace pages testifies to the impact on content development of removing issues of form from the process of publishing content. Anyone comfortable with word processing and web browsing, such as shopping online, can readily publish content with a professional look. In Web 1.0, we did it old school: publishers had to possess as much technical wherewithal as content knowledge—they were inseparable.

Using the prêt-à-por·ter, template-based approach to web publishing offered by Web 2.0 frees one to concentrate on the message, the content, rather than the arcane technical minutae constraining earlier custom approaches, which clearly explains the explosion in non-technical forums: web publishing has thus been democratized. What, then, explains the persistence of some teachers to compel students to use the means of an earlier development model in creating content on the web for class? I suspect it's twofold:

The teacher gets more mileage out of her own technical knowledge;
She passes on the pain of her own learning experience (the "why should today's kids have it easy" syndrome).

A more troubling possibility, which I won't dare include in this list of probable explanations, relates to the need some teachers have to make the student's challenge imposing, fed by the fiction that pain=gain is an effective learning strategy.

As teachers, if we recognize that our students have a limited amount of time to engage our subject—competing, as we are, with other classes and the seduction of countless time sinks like Facebook, YouTube, Xboxes, and the like—then we might take more care in constructing their assignments and keeping them focused on what's important.

The Blended Learning Bandwagon

2008-03-12T15:00:00.021-04:00

How elements of distance education are counter-productive
to some language learning environments.

Find an issue in educational technology these days more hyped than "blended learning." A challenging assignment.

Blended learning, with its delicious allusion to those over-priced energy smoothies at the gym and its more utilitarian yet hip sounding hybrid learning, with its allusion to being green, represent the use of some elements of distance education (DE) technology in a traditional class, that is, one that meets together in the same physical space. The most common element of DE to incorporate is computer mediated communication (CMC), taking the form of either a synchronous chat (like IM) or an asynchronous discussion board, where students post messages around a topic (threaded discussions) and responses to the posts of others.

The CMC options in the virtual space of Blackboard courseware.

Student authoring of more elaborately formatted input, including embedded multimedia, may more effectively employ a class wiki or blog (see previous post, The Thinking Person's Wiki). Discussion boards, wikis, or blogs represent the virtual classroom space vs. the physical space of face-to-face interaction.

A pedagogical not technical challenge

Creating this stew of virtual and physical elements for a class, blending ingredients that cook at different rates, makes the task of teaching far more complicated, although potentially more interesting and effective. We are only beginning to address the plethora of skills needed to design and conduct such blended learning courses. What either intimidates or excites teachers initially, and the skills many focus on, however, involve the mechanics of using courseware rather than sound pedagogy and instructional design. The irony of this oversight seems to occur to too few educators.

Language courses present additional challenges to justifying the blended proposition. Teachers have to evaluate what is sacrificed, because, ultimately, in a zero-sum game, hybrid learning is about trade-offs—what's sacrificed for what's gained. The true measure is not merely what's added but the net gain. Class time and time available to students to study outside of class are finite and more-or-less fixed at a ratio determined by credit hours. The challenge, therefore, is to find the most effective use of this given time.

Why they're here and not there

Holed up in their dorm rooms alone, back lit by a blinking cursor, is not something SL students need to travel to another country to experience.

A second language class is a different animal from most other courses on campus. It seems sometimes that no one on campus realizes this (including, sadly, a few of those SL teachers). In a second language learning environment, especially where students live in the target language country temporarily (i.e., not immigrants), the key ingredient to acquiring communicative competency (the principle objective of living amidst the TL) is authentic communicative interaction. The post-behaviorist SL teacher creates an environment where students need to interact with each other to accomplish authentic tasks, ones that in some way intersect with their interests or needs with a deliberate though not necessarily overt focus on form.1 These teachers give students, in other words, what they can't get back home. Holed up in their dorm rooms alone, back lit by a blinking cursor, is not something SL students need to travel to another country (at great expense, by the way) to experience.

They'd be far better served, linguistically and culturally, spending their time outside of class on a ball field, in a mall, at a party, in a bar or a church, anywhere, as long as the environment demands authentic interactions to put into practice what they're studying in class. Thus, spending significant amounts of time performing DE tasks, as with working with drill-and-kill tutorial CALL software (see previous post), fails to make the best use of an SL student's limited time in the TL environment. SL students would be best served maximizing their real not virtual interactions.

Teaching writing presents a genuine opportunity for blended learning or DE as an effective device and not merely an opportunity for a teacher to dabble in technology because it's there. The needs of process writing can be addressed textually with comments on exchanged drafts, such as with Microsoft Word's Comments and Track Changes features, also supported in limited fashion in more collaborative environments, such as Google Docs, Adobe's Buzzword, or wikis (all free and less buggy than MS locally installed applications).

Entering comments in the collaborative Google Docs space.

Teacher feedback on student writing can very effectively be accomplished through virtual rather than physical interaction—something that started long before computer-based DE (think correspondence course). There's little debate about this among language teachers. Still, is this what SL learners should be spending their time on, given what is generally a brief stay in the TL country to study? Probably not, unless these students are also preparing to enter an academic degree program in the country or score well on writing tests for other purposes.

And so we return to a consideration of net gain in adopting one learning approach over another. What do students gain from spending more time in interactions in a virtual space vs. interactions in a physical space? It's been demonstrated by educators studying input in online chats, for example, that shy students who contribute little in a physical class—whether for reasons stemming from culture, personality, or lower language proficiency—may compensate for this deficiency in an online environment with significant input. In fact, such students may shine online, seemingly finding an appropriate outlet for their otherwise pent-up ability or desire to contribute. Such scenarios are widely seen as benefits to online interaction. Nonetheless, such benefits may accrue to only a small number of students in a class, where others may shine in physical rather than virtual environments.

In sum, the justifications for a low-fat blended learning concoction in a SL environment:

1. To bring quiet students out of their shells,
2. To facilitate process writing.

But perhaps the greatest justification for blended learning, if not a pure DE course:

3. To create a learning environment when a physical meeting is not possible for logistical reasons.

There are many reasons why it may be impossible or impractical for a class to meet physically, as this may occasionally be the case in a traditional course due to inclement weather, teacher or student illness, travel commitments, etc. The caution against the use of virtual space interactions at the expense of face-to-face interactions does not discount the use of blended learning or DE as a remedy for logistical issues.

But we need to be careful about seeing our SL class as a laboratory to experiment with blended learning, even if everyone else seems to be doing it. As an instructional designer friend, John M., suggested to me recently over a pint, teachers should, at the very least, have to learn something themselves via DE before they employ these tools in their own classes, either in whole or in part.

1. See Chapelle, C. (2001). Computer applications in second language acquisition: Foundations for teaching, testing and research. Cambridge, UK: Cambridge University Press.

Why Presentation Skills Aren't Taught

2008-02-20T11:38:00.002-05:00

Presentation Series—Part I

In the Information Age, everyone is involved in information design.

I have asked thousands of Boston University students over the last decade three questions:

1. Have you ever been required to give a presentation?
2. Were you told what form it would take?
3. Did you get any explicit instruction in presentation design or delivery?

Overwhelmingly, responses from these students are

1. Yes.
2. Yes.
3. No.

These American undergraduate and graduate students as well as international (ESL) students studying in the U.S. express some frustration with being given a heady task (1), having the tool to accomplish this task prescribed (2), yet given no meaningful instruction (3). At most, they receive some kind of evaluation rubric in advance.

Something fundamental to our lives today is missing from this common scenario: any mention of much less attention to the considerable skills required to design information for presentation and deliver it—which are, by the way, very different skill sets. Why is it that in the world of commerce salespeople receive extensive training in message formulation and pitch, while in education, the need goes unmet if not unacknowledged?

I think there are three primary reasons for this omission:

1. Time. There isn't time in the course schedule to devote to presentation design.
2. Assumptions. Older teachers who never received such training are inclined to think that young students are technology savants.
3. Training. Few teachers in any discipline have formal exposure to principles of instructional design.

Time. Student presentations consume a lot of class time—instruction time that students pay for. The technical and logistical overhead of presentations inevitably result in even more time devoted to student presentations than scheduled, which results in reduced instruction time. Furthermore, in large classes, students must present in groups to reduce the number of presentations, which requires more time, though generally outside of class, for coordination, planning, and preparation.

With this scheduling commitment for presentation delivery alone, few teachers may feel they have the luxury of burdening the schedule further, except in the most cursory or superficial manner, with instruction in presentation design and delivery.

Assumptions. Teachers who did not have computers as a part of their educational experience, as students, tend to see students who grew up with computers as "comfortable" with technology. This comfort, however, does not necessarily impart competency or conceptual understanding. That is, the mechanics of building slides in slideware is a simple skill and one independent of understanding the logic or effectiveness of a particular design. Any sampling of slideshow presentations created by college faculty or students would demonstrate this idea, what with the abundance of gimmicks (animations, transitions, sound effects, clip art) and arbitrary bullet point fragmentation of data masquerading as analysis, all at the expense of thoughtful design and presentation of data in a manner that suits the data first. Such tools in the hands of the uninitiated, not surprisingly, guarantee the triumph of form over content. In an analogy to languages, we might observe that a person with facility in several languages may not necessarily have anything intelligent to say in any one.

Training. Why is it that in the Information Age so few people have any exposure to formal information design? Teachers, or anyone involved in providing instruction, design their information package. Again, to draw on an analogy, academics in higher education who spend their lives teaching often have scant explicit training in the art of teaching—an absence painfully felt by countless uninspired undergraduates. In higher education, one's demonstrated subject knowledge, measured by academic degrees and publications, forms the basis of one's qualifications to teach—at least in the eyes of hiring committees. The closest thing to a teaching apprenticeship, serving as a graduate assistant or teaching fellow in a doctoral program, is more OJT than structured training.

All of this suggests that sometimes skills critical to performance are assumed rather than verified because recognizing a need, addressing it, and following up with higher expectations takes time and resources. Presentations would be a much more effective learning task and a more efficient model for information delivery with some attention paid to information design, suggestions for which I'll address next.

Related links:

Intelligent Use of PowerPoint
Tips for Using PowerPoint for Academic Presentations

PowerPoint Is Evil: Power Corrupts. PowerPoint Corrupts Absolutely (Edward Tufte, Wired Magazine)
Edward Tufte interviewed on NPR (Weekend Edition Sunday, 8/20/06)
Constructivist Educational Theory

Using Presentation Software to Enhance Language Learning (Internet TESL Journal)

The Peril and Promise of Online Video

2008-01-31T14:56:00.002-05:00

Language teachers have been using film and video in class as long as these media have been available. Lately, YouTube, videojug, Google Video, blip.tv and other Web 2.0 inspired, user-created content sites have appeared in the repertoire of language teachers with access to computers and the Internet in class.

Let's assume that most trained language teachers see the pedagogical merit, or potential, in such content, including variety, currency, and appeal to students as a familiar resource. At least four practical concerns quickly emerge when using online video:

Access
In planning on the use of online video in class (streamed at time of use, not previously saved as a local file), we make certain assumptions that we would not likely have made just a few years ago:

1. Continuous broadband (high-speed) Internet access (as opposed to occasional dial-up modem access).
2. Ubiquitous Internet access available at all learning locations via wired or wireless (WiFi) connections.
3. Computer hardware capable of playing compressed video (fast processor, sufficient RAM).
4. Appropriate media player software installed and configured to play Web video (e.g., Flash, QuickTime).

Quality
In order to host a tremendous number of videos, video sharing sites must further compress video files uploaded by users, sometimes appreciably degrading quality. Someone has said that the blurry quality of these videos makes them appear as if they are being viewed through Vaseline rubbed in your eyes. So image quality is an issue.

For language learners, this may blur the lip/sound correspondence that aids in comprehension.

The image on the left below is from the original, uncompressed video file (.dv). The image on the right is from YouTube, which shows only a little image degradation. This might be because I uploaded a large, high-quality version of the video to begin with (.mp4).

Original videos of lower quality might not fare so well, as with the popular Parkour video below.

Permanence
When you link to online content, you always run the risk of losing access to that content without warning if the owner moves or removes it. The ephemeral nature of popular user-generated video increases this risk. Contributors can remove their videos at any time.
To avoid this risk, you can capture some streaming media to locally saved files, access to which you control, but this raises ethical if not legal issues. Applications like iRecordMusic (Mac) or Super MP3 Professional Recorder, Freecorder, or WM Recorder (PC) allow you to capture streaming audio and TubeSock (PC and Mac) captures streaming video. The free online tool KeepVid allows you to easily capture any streaming video in any browser.

Audio or video podcast versions of content, where available, are readily downloadable files that play locally from your computer and skirt the permanence and copyright issues.

Context
At many sites offering video content, a selected video is displayed in a busy and, to some, a confusing context of other content, links, and ads. Especially on sites of user-generated content, such as YouTube or videojug.com, some of this context may be offensive, inappropriate, or, at the least, distracting.

On many video sites the "Embed" information solves this problem by allowing you to display only the video of interest to you in your own context. By pasting that embed code into your own web page, blog, or wiki, you essentially strip this content from its originating context, allowing your students to focus on the video, as in the example below.

How To Open A Bottle Of Champagne With A Sword

Using software in foreign vs. second language environments

2007-11-27T22:29:00.001-05:00

Tutorial software used in a foreign language environment can benefit from the value added by a language teacher’s L1-specific instruction, while bilingual resources in software used in a second language environment may pose problems.

In a FL learning environment (learning Chinese in Boston), where your students all speak the same L1 (as NS or NNS), you can address issues in learning the target language specific to particular L1 speakers. In a SL learning environment (English in Boston), your students might come from many different first language backgrounds, making bilingual instruction or support unavailable or at least impractical (in addition to being undesirable in a direct-approach inspired intensive language immersion environment).

Consider a pronunciation program, such as Pronunciation Power, which presents activities to discriminate and produce 52 English phonemes (at right). In a FL environment, students could be knowledgeably instructed which sounds to focus on because they do not exist in their L1, and which to recognize as common with their L1. This is highly unlikely to occur, to much effect, in a heterogeneous SL environment.

American school children, compared to, say, the French, do not, by and large, get much overt grammar instruction, in the form of grammar and syntax. They might be introduced to some vocabulary expansion techniques via morphology, but otherwise not a lot of meta language instruction.

Which might explain why I didn't know much about English grammar until I started studying French. I found a slim volume titled English Grammar for Students of French, part of a series that included other undergraduate staples, like Spanish and Italian. Structural approaches to FL instruction, as French was traditionally taught (back in the day), used grammar terminology as the common language, when, as I suggest, American students don't know their present progressive from their indirect object. In a way, then, this little primer proffered information to me when it was relevant and immediately applicable (digesting French grammar instruction) rather than abstract (accompanying native language instruction). Similarly, a FL teacher can add value to pedantic material, such as tutorial CALL software, by making the same kind of contrasts and comparisons with the students’ first language and perhaps taking a focused and selective approach to using the software, as in the case with studying pronunciation.

So the same program may be more appropriate for or implemented very differently in a FL environment than a SL one. Moreover, multilingual resources in software, which not all users in a SL environment may benefit from--depending on their first language and whether it's represented in the software--may be problematic to use. In addition to some students feeling left out, software with bilingual features also presents less of a linguistic challenge to other students, whose L1 is represented, because they can lean on their L1 for instruction.

From EuroTalk's Talk More American English, multilingual help.

This all gets back to a previous discussion of evaluating software, where we saw that one must be familiar with the local environment—where the software will be used, how it will be implemented, and who will use it—in order to make the most relevant determination of its utility.

Why we use computers in education

2007-11-21T08:09:00.000-05:00

We can take a thoughtful, knowledgeable approach to evaluating educational software, diligently following our checklist, guide, or principled framework of choice (see previous post) and still overlook some fundamental issues in computer-aided instruction, issues that strike at the core of why we use computers in education in the first place.

Some titles, let's say a grammar or writing reference, might strike some educators as being not much different from a textbook. It might even offer the same exercises from the structuralist's or behaviorist's playbook. Even so, the computer provides the better medium:

Stimulation. No text can match the graphical potential of the computer display. Most texts are printed in black only, or perhaps with one ("spot") color in a range of hues. Printers charge per color: the fewer, the cheaper the job. Not so the computer monitor, which deals in millions of colors (16.7 million, reportedly). And while most programs use "interactive" as a marketing buzzword more than a design criterion, they do stimulate and engage students in ways that textbooks can't.

Multimedia. Software can incorporate audio, video, and animated graphics to supplement textual information where the textbook must make text and graphics alone suffice.

Internet extension. Software can provide links within to external resources, such as publishers' companion sites, which offer updated and expanded resources as well as learner communities.

Feedback. Textbooks offer feedback in the form of an answer key at the back of the book, which usually doesn't go beyond confirmation of correct or incorrect. Software can offer it immediately, embedded in response to each incorrect or correct answer and include links to internal or external reference material. Moreover, the type of feedback is far more useful to the learner as it may be explanatory, diagnostic, or elaborative, providing far more information than simply confirmation of right or wrong.

Yet, some critical positions on educational software seem to compare what a particular software title offers not to a comparable book but to what a teacher could offer a student. Tutorial CALL software occupies a place somewhere between human instruction and a textbook. It needs to be assessed in its place on this continuum and not with the poles.

Guidance. A learner studying on his own with a tutorial CALL program, outside of an educational setting, lacks some critical support that a teacher with CALL experience could provide:

Software selection. An experienced CALL teacher is in a much better position to assess the pedagogical value of a piece of software than an independent learner, who may be, for the most part, at the mercy of wildly unrealistic marketing hype as his only source of information about the program. This is especially treacherous when navigating the waters of retail programs marketed to individuals, as opposed to more elaborate systems marketed to institutions. With the former, the company need only concern itself with making each individual sale, not with satisfying the more rigorous quality criteria of an informed institutional client with wide purchase decision-making responsibility.

Orientation. A teacher can introduce a student to the functionality and resources of a program, explain its goals, demonstrate its procedures, and perhaps warn against some pedagogically unsound or confusing parts, all with the objective of maximizing the program's language teaching potential.

Direction. An experienced language teacher can assess each student's specific needs and recommend a custom study plan for the use of software, rather than leaving a learner to stumble through the program in haphazard fashion or even work through it in a linear fashion, which may not be an effective use of his time.

Context in Language Software Evaluation

2007-11-08T20:10:00.000-05:00

Language teachers are frequently called on by their school or department to evaluate tutorial CALL software for students to use in class or in a self-access lab on their own. Such evaluations are seldom carried out by someone with all the relevant skills and experience to conduct a pedagogically sound evaluation or, if so, may not apply a principled approach to evaluation. Typically, an evaluation takes the form of a superficial sight-seeing trip through randomly selected parts of a program until one tires of the experience and a feeling from the gut urges "use it" or "forget about it."

What is tutorial CALL?
We're talking here about pedantic, tutorial software that offers explicit language instruction, not generic computing or CMC tools that might be used in a language learning environment. Evaluations may be for

software
websites
courseware

Some common criticisms of tutorial CALL

"Students don't learn a language with a computer program."

True, but only the likes of Rosetta Stone and others traffic in the kind of absurd marketing claims that software alone can teach language. Moreover, with only a textbook students won't learn to communicate in a language either, but we still use it to provide structure.
Teachers experienced in using technology in the classroom know that tutorial CALL programs supplement classroom activities; they don't replace them. And they only accomplish that if well chosen.

"The problem with the software is that you don't know if students are using it."

Again, this is only true if we chose the wrong software. If we looked broadly at "educational software," we would see that the vast majority of titles are designed for the retail market (individual not institutional use). The design criteria there seems limited in scope to flashy graphic features that reproduce well in printed ads, rather than the more bothersome design assumptions involving language teaching methodology, what Hubbard calls "teacher fit" (approach), or the theory that describes ideal conditions for instructed SLA and the construction of tasks that provide those conditions (Chapelle).

Retail programs usually do not concern themselves with providing mechanisms for accountability. Institutional settings may require the software to report student time on task as well as scoring, either through e-mail notification, logs accessible by the teacher, or some kind of built-in drop box. More elaborate programs used in schools integrate these kind of LMS features.

Purposes of evaluations

to make a purchase or implementation decision on software, a process that effectively ends when this practical outcome is reached (a decision-driven evaluation);
to give design feedback in the developmental stages of software (a formative evaluation);
for research motivated by a hypothesis or open-ended question;
for a published software review (a summative evaluation).

General problems with evaluations

Evaluators use different criteria.
Evaluators are informed by different interests, knowledge, and experience.
Evaluations lack consistency across reviews.
Evaluations lack of inter-rater reliability.

The dilemma facing most evaluation situations is that while only a local decision can take the specific learning environment and population into account, only one based on a principled approach (see below) by someone with a grounding in language teaching methodology, instructional design, and of course content expertise can render an evaluation that's valid and consistent. Teachers selecting their own software tend to evaluate subjectively based on their own teaching and learning experience, computer literacy, and personal preferences.

Types of evaluation
If we look specifically at summative evaluations of language learning software, we find the following common approaches:

Checklists
Guides
Surveys
Principled frameworks

Checklists, the most common approach, offer a set set of questions, usually binary options or fill-in. They are simple to follow and may raise awareness among teachers inexperienced in CALL of the wide range of factors to be considered. They are more meaningful if questions elicit commentary.

Criticisms of checklists abound:

Terms used are not defined, used inconsistently, or open to varied interpretation.
Elements are not weighted; some influence may seem disproportionate.
Their simplicity belies the need for background knowledge and experience to accurately, appropriately respond.
Questions are little more than lists of features to look for.
They are focused on technology more than teaching and learning (language learning potential).
They lack reliability, validity.

For a checklist example, see the Software Evaluation Guidelines by the National Center for ESL Literacy Education (2003). This checklist addresses technical and pedagogical issues but not methodology specifically, which is a common omission. Among the questions:

"Do the individual program lessons fit within the time constraints of class or lab sessions so that a learner can finish a lesson in one sitting?"

This question seems to assume some validity in the use of tutor-type software in a class—at the expense of human instruction and more meaningful and authentic student-student or student-teacher interaction. For the most part, aside from introducing the functionality of tutorial CALL software to students, class is not the place to work on these programs. The tutor mode of computer use implies the absence of a teacher and the presence of a virtual teacher, if you will.

Guides are what I would describe as a hybrid between a checklist and more discursive prompts for thinking through, if not formally evaluating, the pedagogical value and instructional design efficiency of software. I created such a guide ten years ago, A Guide for Evaluating Language Learning Software.

Surveys assess student or teacher response to software or courseware after a considerable period of use, such as a semester.

Principled frameworks* represent an organizing scheme to characterize relationships between elements of language teaching and learning and computer use. Among the best known and most often referred to in the filed include

Philip Hubbard's framework:

methodology driven
emphasizes need for evaluator to understand LL approach taken in design and fit to instructional approach
Non-hierarchical model**

Teacher fit (approach): assumptions about the nature of LL in light of what’s possible with computer-aided instruction.
Learner fit (design): realization of approach: syllabus, tasks, activities, language difficulty, skill focus, roles of teacher and learner materials.
Operational description (procedure): the form the approach and design take in the program: layout, activity type, feedback.

Carol Chapelle's framework

theory-based, task-oriented
driven by interactionist position***
focused on design and structure of LL task
evaluation can be judgmental at initial selection based on how well suited it appears to be and it can be done empirically based on data from actual student use

CALICO Journal Software Review (Jack Burston)

descriptive not prescriptive
discursive not intuitive
software requirements (must meet first two and some combination of last three)
1. Pedagogical validity
2. Curriculum adaptability
3. Efficiency
4. Effectiveness
5. Pedagogical innovation
Four categories, based on Hubbard’s framework, form the template for evaluation for the journal:
1. Technical features
2. Activities (procedure)
3. Teacher fit (approach)
4. Learner fit (design)
The CALICO software evaluation template thus presents a consistent qualitative measuring device.

*As described by Levy and Stockwell in CALL Dimensions: Options and issues in computer-assisted language learning, pp 59–64.
**Based on Richards and Rogers model (1982, 1986) of Approach, Design, and Procedure.
***Language is a rule-governed cultural activity learned in interaction with others; environmental factors are more dominant in language acquisition, as opposed to innate abilities of the nativist position.

The Thinking Person's Wiki

2007-11-03T19:38:00.000-04:00

Wikis are all the rage in education. And why not? They represent the phenomenal improvement in ease-of-use and separation of content from form. In Web 1.0 days, putting content on a Web page was a feat that required considerable technical skills and time, much of which was spent on form--that is, formatting text, manipulating placement, coding relative and absolute links, etc.--much of it to force a Web page layout to simulate the printed page.

With wikis, students can create pages, add links, gadgets, RSS feeds, embedded media--images, audio, and video--with hardly a thought about how this happens. It is genuinely point-and-click, and students can finally concentrate on content. Formatting of content in a wiki doesn't move beyond the complexity of RTF or what you might do in a word processor.

What else? They offer a virtual community for students to work collaboratively, encourage peer review, demonstrate the importance of responsible network citizenship, and even make for an alternative tool for group presentations. Mistaken deletions or edits can be rolled back to a previous working version (a fix for sabotage or inadvertent changes), and teachers can monitor student participation. Collaboration can serve as "crowd sourcing," where a teacher charges students with creating some specific content (e.g., guidelines, evaluation criteria, tests) instead of simply providing it himself, thereby harnessing the collective intelligence of many. (Watch this Campus Technology webcast about wiki use in business schools.) This doesn't work for all crowds. That prospect must be evaluated beforehand.

There are free versions for education (see pbwiki), and there's no software to maintain locally for all this functionality beyond a free Web browser and an uninterrupted broadband Internet connection (which might not be free).

So wikis are wonderful, but there's a catch: they require thoughtful, responsible users who understand the public nature of the space. What are some of the issues that these enlightened users must grapple with?

Pedagogical value
While sound pedagogy should always drive the curriculum, sometimes technology pulls it. Teachers and students may be awed by technology and lured into using it before they have a compelling use for it. Your students can learn to embed a YouTube video on a wiki page in minutes, but what's the pedagogical value of doing so? In their zeal to either keep up with their students or push the envelope and impress them with their command of Web 2.0 mash-ups, some teachers evidently believe that any task pursued in a constructivist vein that maintains students' interest is justified. It might score points on teacher evaluations, but what is the effect on language learning?

The trick is to think it through. Don't merely have students add media to a page but do so with a purpose. For example, find a video that might model grammar, vocabulary or usage of particular interest and follow up with an activity that elicits specific language, such as with a description.

Structure
Wikis can seem somewhat amorphous for new users. With no apparent structure, no linear site map of old, users must be shown how it grows somewhat organically with new pages added, sidebar links fleshed out, and meta pages created to organize distinct groups of content. But unlike static HTML pages--Web 1.0 authoring--this form takes shape during content creation and not in anticipation of it.

Authority
Who controls a wiki? Everyone and no one. Certainly the person that creates it also administers it, but for the most part that's limited to deciding how long it lives and how users access it for editing. Otherwise, the idea of authority must be considered diffuse, shared equally among users, and self-policing in order for this communal space to work effectively as a whole.

Controversial topics
The Wikipedia entry for "Armenian Genocide," one of considerable depth, engenders far more words in the accompanying comment and discussion pages than in the entry itself. That a wiki is a collaborative environment suggests that some didactic can and should emerge, not a reasonable prospect for some topics--abortion, gay marriage, war, George W. Bush come to mind. Such topics would more fruitfully be discussed in an environment where individuals don't feel momentarily empowered to speak for the group or proffer a final statement.

Comments
Should a language learner's writing be subject to comment by strangers? Would the ego-destructive effects of negative criticism of content or form outweigh the possible benefits of having such an outlet for expression to begin with? Well, it seems like a reasonable concern, but consider that language learners, in a second language learning environment anyway, are subject to such negative feedback (misunderstanding, ridicule) to their production as they engage native speakers in authentic contexts outside the classroom. They are not protected from such feedback so must instead be empowered with the wherewithal to take it in stride.

Consider also that most students' wiki or blog entries are not likely to attract outside attention to begin with, so the danger is likely statistically insignificant. As a safeguard, wiki and blog comments can be moderated by authors (aka cherry-picked), so undesirable comments will not remain. Guest access or comments can also be disabled altogether.

Participation
Because they represent a team effort but through individual contributions, wikis enable freeloaders, users who do not contribute as much as others but reap the benefits of the collaborative product. Some of this can be tracked through access logs, but not with practical effectiveness. In this regard wikis suffer just as public radio does: a listener can benefit from the commercial-free programming whether or not he contributes.

Public nature of content
If you're concerned about your wikis, blogs, Google Docs, comments, or Picasa or Flickr photos being subpoenaed, you should probably rethink what you're doing in those spaces. It's hard to imagine content created and shared by reasonably thoughtful teachers or students for language learning purposes as posing a threat of criminal prosecution or civil litigation. But if that does concern you, keep in mind that your e-mail and hard drive can be subpoenaed as well.

Permanence
Your class might spend an entire semester contributing to and refining content on a class wiki. Will all that work vanish at the end of the semester? It might. Or it could easily be archived by creating a meta page with links to pages created by a particular class. Unlike Wikipedia, student work on a language class wiki is more about process than product; the value thus derives from the experience of contributing. If not, archive the work or have students move what they want to their own personal wiki, blog, or social networking page.

Conclusion
We must conclude that such Web 2.0 applications, or hosted services, as wikis entail greater user responsibility than a visitor to a Web 1.0 site. We are no longer merely information consumers in a one-way lecture with simple assumptions about authority and credibility, but participants in a much more complex web of ambiguous identities and an organic dialectic. With this democratization of control comes a distributed burden on contributors to be enlightened and responsible. It's not supposed to be easy, but it may just lead to a more intelligent information and communication system.