This is a long research paper about Course Management usability. It compares Blackboard CourseInfo with Intralearn and WebCT. Here are some of the more interesting comments:
from Evaluation of Web-based Course Management Software from Faculty and Student User -Centered Perspectives by Peg Halloran.
This is a long research paper about Course Management usability. It compares Blackboard CourseInfo with Intralearn and WebCT. Here are some of the more interesting comments:
From section 2.4 Discussion:
When inexperienced participants were asked to determine what features were important ... both the faculty and cadets focused on their own use of the software and did not appear to consider the requirements of other user groups.
Cadets ... rated the exam question types they favored and the ability to find materials as more important than other features.
Since neither group seems to have considered other user’s requirements, this illustrates the need to have all user groups involved in the selection process and development of requirements for these types of tools.
Also worthy of note was the comment:
Both faculty and cadets rated the ability to support foreign languages as one of the least important features of these tools, yet the ability to support languages other than English would be essential for developing web-enabled curriculum for a foreign language course. The 2 foreign language instructors and the 3 cadets enrolled in foreign language courses rated this feature as a “must-have”.
... most cited ease of use as the single most important factor that helped them make their decision. They indicated that the navigational interface was intuitive, it required fewer mouse-clicks, and the steps necessary to load assignments and course documents were obvious ...
If a task was easy to perform, the faculty tended to rate it as more useful to teaching and learning, and if the task was more difficult to perform, then the faculty rated it as less useful to teaching and learning.
The CourseInfo menu allows the user to view the different subheadings available and choose the desired function
faculty did not seek out help or use the online manuals very often when using any of the software packages.
yet a few expressed a dislike of CMS tools overall and may have been biased because they did not like the first package that they tried. As one participant stated “the first software package was so distasteful, you could not convince me to try a second.”
CourseInfo also included six out of the seven features that faculty rated as more important before they began evaluating the software packages. ...... Finding features that they had predetermined as useful may have validated their beliefs that this product was beneficial to teaching.
Ease of use. Ease of use. Ease of use.
The issue is how do we evaluate / measure ease of use of CHEF? We can count the number of steps taken to do common tasks but at the end of the day, useability needs to be measured in a Usability Lab setting.
A third of the cadets used the online manuals or asked for help.
Although this is low, it's a lot higher than the proportion of faculty that asked for help.
Many websites have unintuitive navigational layouts, and “hunt and click” navigation is frequently used by people who browse the web (Nielsen, 1998).
the course documents were not where the cadets expected to find them, which increased the amount of time the cadets had to search to find the appropriate handout.
cadets would have to search under different headings to find all the handouts and assignments that might be due in a particular lesson.
When cadets chose a a different CMS over the leading one, navigational layout was the defining reason.
Cadets also rated the features in the courses such as asynchronous and synchronous discussion groups, integrated calendar, using external links, taking quizzes and finding their grades as useful.
Not useful features:
The only feature that cadets did not rate as useful was the drop box for submitting assignments to their instructors. Many of the cadets commented that this was redundant to submitting assignments as email attachments, which is the current system.
Confusing
Several cadets also commented that using a separate calendar for courses was confusing, and that they would prefer to have all of their assignments and announcements integrated into one calendaring system.
This part of the study comprised 13 faculty who used the Blackboard CMS for a real course for one semester.
Importance of features: faculty and cadets focused on their own requirements.
Cadets focused on the types of quiz questions that would appear on exams and the ability to check their scores online, and the faculty continued to focus on administrative procedures and quiz construction. However, there was more agreement between these two groups than from the previous experiments. Both groups felt that student access to progress data, automatically graded quizzes, using a mixture of question types on a single quiz, table of contents and the ability to create pages without the knowledge of HTML were more important than other features required of a CMS product.
The faculty members and the cadets continued to rate the discussion groups, shared whiteboards, virtual field trips ... as features that are not necessary, or less important than most of the other features. In addition, faculty continued to rate the support of foreign languages as less important, although there were no foreign language instructors represented in this population.
Most of the faculty used CourseInfo to post documents and assignments, to send their cadets announcements, add external links, to report scores and to administer online quizzes.
What they liked most about using CourseInfo:
most of the faculty reported the use of automatically graded quizzes, cadet access to grade book, and a central place to post announcements and assignments. Although faculty used the online grade book, many had problems using it as designed and found it more cumbersome to enter grades into the program than into a spreadsheet such as MSExcel.
Collaborative Features:
Very few of the faculty used the collaborative features of CourseInfo including the discussion boards, chat feature, file sharing, drop box or group workspace.... that may be influenced by our use of the product to supplement traditional classroom education, rather than to deliver distance or online courses.
Features lacking:
there was no way to divide grade books, announcements or documents into multiple sections of the same course without duplicating the entire course. Similarly, it was not possible to share calendars, quiz questions, handouts, or content among instructors teaching the same course.
many cadets expressed disappointment that the faculty members were not using the software to its fullest extent. Some indicated that they would like to see more than just class notes and announcements posted. Cadets expressed interest in online quizzes and answers, and others expressed interest in the chat features.
This indicates that the software was evaluated by some of the “better” teaching faculty, and that using the software did not prevent them from providing quality instruction to the cadets.
Feature lists not useful for evaluating CMSes.
Blackboard CourseInfo received higher scores than the other products when ratings were based on user analysis of ease of use and usefulness.
Many of the features found in these products such as discussion groups, student web pages and collaborative work areas that contributed to their high number of features were not widely used, nor deemed important by both faculty and cadets.
Although some faculty may eventually use these features as they become more familiar with the software and pedagogy, at this time a CMS with an easy to use interface that contains a grade book, automated quizzes and a place to put announcements and course documents should be preferred to one that contains many collaboration features, yet has a difficult navigational interface or hard to use development tools.
faculty and cadets had different requirements of the CMS product for developing and accessing resources ... if only the faculty’s requirements are considered, or if the product is not evaluated by all of the user groups, the probability of purchasing a sub-optimal product for a majority of the users increases.
To fully take advantage of the benefits of a CMS, there must be buy in from as much of the institution as possible, and course information should be integrated with other institutional databases into one system using an academic portal or other similar interface.
Presents conclusions and to do list from our CMS meeting.
Friday Feb 6th 2004 11:00am
Present :
What we need to do and when:
We agreed that our position would be as follows;
We have decided that the CHEF / SAKAI project is the way to go for our open source CMS : the document that we present will seek to justify this position. (Do we need to put this into context? - Maybe a separate paper)
What is CHEF / SAKAI?
A brief overview of CHEF and where it fits with SAKAI.
Meets the perceived immediate needs of faculty / student users (accomplished
by the current CHEF system).
These are :
Here we'd need to quote evaluation figures from the recent survey.
Here we'd need some CHEF survey data from faculty.
Other factors:
| CHEF | ||||
|---|---|---|---|---|
| Function | Ease of Use (faculty) | Ease of use (student) | Pedagogical Value | |
| Assignment document upload | Scale of 1 - 5 | Scale of 1 - 5 Or could use a 5 point scale similar to Consumer Reports |
Subjective opinion from faculty | |