Home » 2010

Yearly Archives: 2010

Developing Programs

Though colleges and universities are accredited by large institutions both regionally and nationally, they often have very different methods for developing programs, approving courses (new and change of delivery format), and for assessment.  Sadly, there is a very large portion of them that have yet to get a handle on building assessment into program design so that it makes sense, so that it is more than just something they do when they come up for accreditation, so that it is something that helps determine what courses will fit into a specific program, and how best to change course content when there is a change in delivery format (e.g. online).

Programs should have at their core a stated list of objectives:  the knowledge, skills, and attitudes students will demonstrate they possess upon graduation.  These objectives need to built into the program design.  The design needs to articulate where the objective will be assessed and on what level of achievement: introductory, intermediate, and advanced.  If this is in place, then it is easy to determine and justify what courses will fit best into the program, as well as what things need to be looked at when there is a change of format.

I’d like to look at this last item first, because it is very often overlooked.  If a program includes an objective that a student will have skill in a certain application program, and that objective is assessed (or taught) in a particular course, then when that course has a change in instructional format, it needs someone to insure that particular objective will still be met.  In the case of a course that is put partially or wholly online, that would mean insuring the student will still have access to that particular application program even if they are not on campus.

To facilitate this, I suggest that programs first begin with their list of objectives, and then these objectives be cross referenced in a table that outlines the program and where they will be assessed (and taught).  (Please see an example below):

Knowledge Objectives:

1.o Auctor tempus elit.
1.1 Dolor urna.
1.2 Arcu vestibulum gravida.
1.3 Amet in, massa donec lacus, enim assumenda.
1.3.a Nibh viverra massa.
1.3.b Tincidunt sit.
1.4 Lacinia dui.
1.5 Enim voluptatum.

Skills
2.0 Tincidunt nam nunc, tempus ultrices.
2.1 Auctor ipsum enim, eget feugiat eu, iusto nec cursus.
2.2 Arcu in sem.
2.3 Sed amet, ut ullam.
2.4 Nunc posuere, eros posuere.
2.5 Lobortis auctor, suspendisse sagittis.

Attitudes
3.0 At mauris duis, donec viverra.
3.1 Nunc leo elit, nec lacus, dolor tempor lectus.
3.2 Gravida pellentesque, turpis tincidunt quis, lacus eros in.
3.3 Nullam urna.

Freshman Sophomore Junior Senior
Course X101

obj: 1.0, 2.0

Course X201

obj 1.0 (adv)

Course X301

obj 3.3 (int)

Course X401

obj 2.4 (adv)

Course Y101

obj: 1.2, 2.3

Course Y201

obj: 1.5 (int)

Course Y301

obj 2.1

Course Y401

obj: 3.1

Course En101 Course En201

obj 2.5, 1.3

Course TTd301

obj 2.2, 3.0

Course Spec401
Course R101

obj 2.0, 3.0

Course Pt201 Course Ex301 Course Ist401

That’s you all over

Bits and pieces

There’s a scene in The Wizard of Oz where the Wicked Witch sends her monkeys to tear the Scarecrow apart, leaving pieces of him strewn in the field.  The Tin Man and the Lion arrive after the fact.  When the Scarecrow explains what happened, the Tin Man humorously remarks “Well, that’s you all over…”  This is what I think of when I hear people talk about having students use Web 2.0 tools to create ePortfolios for their classes. I can understand the desire and even the pedagogy behind instructing students on the building of an ePortfolio, but what happens is that after 4 years, there are pieces of the student all over the web, and there is actually less hope of putting those pieces back into one body, than there was of putting the Scarecrow back together.

While I understand the value of having students use a program like LinkedIn to create an online resume and begin networking,  such sites show only one side of the student and can only be used for very targeted activities.  To remedy this, I think it is vitally important that institutions of learning begin offering all students access to an institutionally sponsored, comprehensive, student centered ePortfolio program. Yes, a lot of terms there, but not impossible.  A program that is student centered, and that does include several other features: blogs, social networking components, public and private views, and the ability to export the contents should it be necessary, are available.

Eportfolios have become an important educational tool, not just for program assessment and not for accreditation purposes, but because they showcase a “whole” student, because they allow for authentic assessment, and because they allow instructors to work with students in forming the students own brand.  If we don’t offer a comprehensive ePortfolio solution that students and faculty want to use, they will go elsewhere and the various pieces will be scattered all over the place.

What are we asking?

test takingAt a recent presentation, I did a survey of the audience asking how many of them had ever taken a survey in which there was at least one question they felt they couldn’t answer because they weren’t sure what was being asked.  As you probably have guessed every single person in the audience raised their hand.  So I asked them the same question I’m asking you:  if the individuals we are surveying have to guess what we are asking in order to answer, then is the data we gather reliable?  And if it doesn’t matter, then why are we even asking the question?

One of the ways that this commonly occurs is when a question is worded in such a way that it actually contains more than one question, asks more than one thing, and therefore requires more than one answer.  For example:  “Course material was up-to-date, well-organized, and presented in sufficient depth”  What if the course material was up-to-date, but not well-organized?  How can an individual, then, respond accurately to this question?  If they choose “neutral” or “agree” or “disagree” what does it tell us?  Of what benefit are the ratings to the instructor?  If this item was rated poorly would the instructor have any idea what change needs to be made?

Another type of question that can lead to misinterpretation, is the question that is ambiguous: How well did the course meet your expectations? Unless this question is accompanied by a previous question that asks “What were your expectations?”, we do not know what the answer tells us, and yet this appears on many course evaluation surveys.  However, just as in the first example, even if we did ask what the expectations were,  the answer would only provide us with an average of how well they were met, rather than information regarding how well each of those expectations was met.

All assessment, even when it comes down to individual questions on a survey tool, need to begin with a clear outline of objectives.  The questions need to be asked in such a way that they will accurately provide the information desired.

Changes in NCATE and others

A meeting with our Education faculty a few days ago, led to some new and interesting information for me as I continue the role out of Mahara.  Apparently NCATE, and perhaps other accrediting bodies, don’t want to see artifacts any longer; they simply want the data.  So, it would seem the impetus for implementing e-portfolios would be going away, however I see this as something that will actually allow e-portfolios to become what they should’ve been in the first place: student-centered.  They now can develop more freely to be a collection of a students learning and his/her reflection on that learning.

Most of the e-portfolios that schools adopted to meet accrediting requirements were difficult to use and pushed out like text books are.  There was little student buy-in, and few students who wanted to use them for anything other than meeting program requirements.  An e-portfolio designed to give students a place to showcase who they are, how they think, and what they can do, is an e-portfolio system that will be adopted and used.

Our role out of Mahara has been successful beyond our wildest dreams.  Starting as a pilot in January, we’ve already had several departments completely adopt it, and non-academic departments are included in this: co-curricular activities and career services to name a couple.  Students use has taken off too–on their own.  That to me is true success and that’s what happens when you have a product that meets people’s needs.  That’s how facebook, youtube, and flickr took off–they provided a service that people wanted.  It’s wonderful to have an e-portfolio system like that: one that our students, faculty and staff want to use!

The blind leading the blind

I realize this is going to be quite a controversial post, but I’m going to say this nonetheless.  Too often, Deans or Chairs will hire an individual to teach an online course merely because they have taught online elsewhere–as if that experience guarantees they know how to teach online.  Often these individuals show their online courses to other instructors as examples of how to teach online.  Too many of them, however, are not good courses.  Too many online courses are simply this: read documents, post to the discussion board, do a project.  I’ve taken a number of online courses and they almost always follow this format.  Even in this format, though I’ve grown quite tired of it, there are some effective discussion boards and some that are not.  Those that are worthwhile are those in which the topic of discussion is clear.  That is that the instructor hasn’t asked the students to discuss several topics within one posting.  That is that the instructor hasn’t asked several questions, but one.  Effective discussion boards also have some input from the instructor other than simply a grade.

If an instructor says “10 points for your initial posting, 5 points each for responding to two other postings” and tells you at the time she scores points that you got 20 points for that discussion board (and says nothing else) you tend to wonder if the instructor really reads the postings, or simply looks at the number of postings and words.  I know, because that’s how I feel about a course I’m taking this semester.  This instructor teaches a large number of courses online for that University, but in my experience as an instructor, a course designer, a course evaluator and a student, this is a poorly structured course.  If this instructor went to some other institution with the credentials:  taught 2 courses each semester online for the past 5 years, it is my experience that the new institution will be impressed merely by the fact the individual has taught online.

This all leads me to a quote from The Little Prince: “Grown-ups like numbers.  When you tell them about a new friend, they never ask questions about what really matters.”