cms @ ec evaluations >> Moodle >> student survey fall 04 (results)

Introduction

I asked faculty in the following classes if I could use the FAST online assessment tool to survey the students in their classes for their opinions about Moodle:

Division: Natural Sciences
FacultyCourse IdCourse Namenumber students
Brent SmithBIO 455Population & community Ecology16
Brent / Jon BranstratorENPR 111Environmental Science54
Jennifer ZiebarthMATH 420Abstract Algebra11
Jennifer ZiebarthMATH 180Calculus A37

Division: Social Sciences
Kathy MilarPSYCH 342Experimental Psychology47
Michael JacksonPSYCH 351Qualitative research Methods21
Mark PearsonMGMT 110Info Tech & Soc22

Division: Fine Arts
Julia MayART 382Art of the Americas39
Julia MayART 282Survey of Western Art40

Division: Humanities
Micelle PattersonHIST 121US to 186541
Sandrine SanosES History modern anti-semetism 18
total346

A total of 346 students were canvassed by email from 8th to 10th December after I had obtained permission from the class teachers. 93 students responded, a 27% success rate, and there was at least one respondent in each of the courses canvassed [comments:Q15].

go to top of page

Survey Results: questions asked

Questions one and two seeked to discover whether the student was new to Moodle, and if not, how he/she compared it to what was used beforehand. Questions three and four followed this further asking whether students found Moodle easy to use ('intuitive') and whether they considered it an asset to the course (which also depends upon how the teacher used Moodle). 'Ease of use' is not neccessarily equivalent to 'intuitive' — a system may be easy to use but not accomplish what the teacher desired — but I wanted the students to consider internal consistency in how they did things with the Moodle system. There was no formal instruction offered to students and question five asked whether students would have liked such a session. While question four looked at how the course was presented with Moodle, question six seeked to discover whether students considered Moodle to be a help or hindrance to their learning efforts. Questions 7 - 10 looked at how students made use of the Moodle system. Question seven looked at frequency of use outside class time, question eight at whether students found the 'home page' arrangement useful (course contents presented in a Weekly or Topic outline), and question nine asked how students used the uploaded files in Resources. Finally, question ten sought to discover whether student would (hypothetically) like to post assignments online. Questions eleven and twelve asked for written feedback about Moodle likes and dislikes (Q11) and whether students felt that Moodle could have been used in other ways (Q12). Given some previous comments about the performance of the Moodle system questions 13 and 14 concerned the responsiveness of the system in general (Q13) and from the student dorm rooms (Q14). Finally, students were asked to give their course code so that we could check that all the courses surveyed had respondents.

go to top of page

Survey Results: data

Questions Strongly Disagree Somewhat Disagree Not Applicable Somewhat Agree Strongly Agree
Q1. Which Course Management system did you use before using Moodle in this class ? (If more than one pick the one you're most familiar with) 0 20 14 12 47
  Course Compass CHEF Moodle Other (such as Blackboard) None
Q2. If you used a different CMS beforehand how would you compare it to Moodle. 0 2 65 17 9
  Strongly prefer the other system Mildy prefer the other system I am using Moodle for the first time / no opinion Mildly prefer Moodle Strongly prefer Moodle
Q3. Overall, would you agree that you found Moodle intuitive to use? 1 5 1 72 14
Q4. Overall, would you agree that using Moodle was an asset to your course? 1 6 1 55 30
Q5. What sort of formal instruction on using Moodle would you have liked? 3 1 8 21 60
  I needed an in-depth session I needed a solid overview N/A (I did have a session to learn Moodle) Some introduction would have helped I did not need any since Moodle was so easy to use
Q6. How did Moodle help or hinder your learning progress with the class? 1 4 19 47 22
  Having to use Moodle made the class much more difficult for me I made progress in spite of having to use Moodle Neutral. I could have done the same without it Moodle was useful but not essential Using Moodle greatly enhanced my learning
Q7. How frequently did you use Moodle outside of class time? 7 6 46 29 5
  1 Only when I was sent an announcement via email 2 Only when told during class to check it 3 1 & 2 & when I needed to do an assignment 4 1 & 2 & 3 & occasionally to see latest activity 5 I checked it every day regardless
Q8. Do you agree that it was helpful to have the course contents viewable in the weekly or topic outline? 2 6 13 48 24
Q9. How much use did you make of the files posted into Resources ? 9 24 18 39 3
  1. No use 2 Only when I was explicitly told to 3 2 & to do assignments 4 2 & 3 & for revision for Quiz 5 2 & 3 & 4 & after each class session
Q10. If you did not submit your assignments online would you like to do this? Yes 48 No 45  
Q11. Which aspect / feature of Moodle did you like the best? The least? Written response
Q12. Are there other ways you can think of that Moodle could have been used in the course? Written response
Q13. Generally when you used the Moodle system how did you rate its responsiveness? 3 15 30 37 8
  Far too slow Slowish Good enough Reasonably responsive Very responsive
Q14. Do you agree that Moodle was just as responsive from your dorm room as in the Lab? 8 15 32 31 7

go to top of page

Survey results : charts


go to top of page

Discussion

Prior to the start of the fall semester 2004, 50% of the students surveyed were completely new to online Course Management Systems [Q1:chart], 22% had used Chef, 15% had used Moodle and 13% other systems such as Course Compass based on Blackboard, in other courses before the Fall 04 semester. Thus there was a preponderance of novice students.

Question 2 wished to elicit from students who had used other systems beforehand how they rated Moodle in comparison. If we leave aside those who had not used a CMS before (50% of the total) we find that a mere 4% mildly preferred the other system they used (2 students), 39% didn't care either way (18), but 37% mildly preferred Moodle (17) and 20% strongly preferred Moodle (9). Thus Moodle was preferred by 57% of students who had used other course management systems as well. This is consonant with the survey of students using Moodle and Chef at the same time in different courses [link]. There, a total of 63% of students thought that Moodle was easier to use than Chef. What we can conclude from this is that on the whole students who have used course management systems prefer Moodle to others. We will explore some of the nuances of student's response to the Moodle system in questions 11 & 12.

The vast majority of students found Moodle intuitive to use [Q3:chart] but much depended upon how the teacher organized the course within Moodle. Thus, tellingly, one student wrote that it was natural and simple to use As long as the teacher keeps it updated and site listings organized. and another student complained that it was not clear where to put what. so unclear what was due, when and where to put it. [Q3:Comments].This latter is a fault with the instructions the teacher gave (or didn't give) for how to respond to the given assignment. [I did address this issue in the winter Moodle workshop where we practised writing instructions which were directive and not vague]. However, most students agreed with the sentiment expressed thus, I found it very easy to use without a lot of instructor guidance, and I'm not very good with computers. [Q3:Comments]. Even though Moodle itself was deemed to be user friendly, some students expressed the need for some sort of formal instruction, so, I wish that I was told how to use moodle more clearly at the outset of the course. [Q3:Comments]; maybe this would be another topic to cover in the Technology Literacy sessions for freshmen. Moreover, the scope of a Course Management System (or, better, Online learning Environment) is by no means obvious and one student commented that It did not seem intuitive what its powers were, and how much the professor was actually using.. Here again, this could be addressed within the context of Technology Literacy. Finally, some students commented on the Moodle interface itself pointing out some interface design flaws based on Fitt's Law and requesting that the resources button be more prominent (no), and one student suggested borrowing a feature of Blackboard when I used blackboard I knew that there was a way to post notes to the prof. and other students and a way for them to respond. [Q3:Comments]. This may be a feature which we could work on with Earlham programmers.

A massive 92% of students agreed that using Moodle was an asset to their course [Q4:chart] with 60% in somewhat agreement and 32% in strong agreement. Thus, most students had positive comments such as it kept things organized and sorted, It was SUCH an asset to have access to all of her lectures, images, and additional resources., and it certainly made handing in assignments easier. [Q4:Comments]. The pedagogic value of online resources was apparent from comments such as these, If it weren't for powerpoints on Moodle, I would have failed numerous tests!, We would have had to take at least twice as much time in class without the powerpoint presentations on moodle., and When it came time to study for a test all of the lecture notes that were on powerpoint were available and it was easy to recall class discussion based on the slides but nevertheless one or two disliked the extra work which was entailed in reading websites and online articles [Q4:Comments]. Once again, the way in which Moodle was used conditioned the response from students; thus The more the teacher is comfortable with it and uses it, the more helpful it is to the students., and I wish the professor had continued to put our grades online and there was what one student termed the one page problem, where (ie. the professor put everything on one page) [Q4:Comments]. Students also alluded to problems inherent in posting large Acrobat pdf files where paper was wasted when students printed everything out; however, PDF files are awkward to handle, and a lot of paper and ink is wasted printing off images. ... When there were 50 images for a test, and i did not watn to just study off the screen, then i would have to print all 50 images.. There is an inherent tendency to print pdf rather than view or read on screen but it's easy to select individual pages to print and if the Labs had duplex printing enabled paper would be saved. Thus we have a training as well as an equipment issue.

When asked what sort of formal instruction (if any) students would have liked (question 5) approximately two thirds said that they didn't need any instruction [Q5:chart] but 23% said that some (formal) instruction would have helped their usage of Moodle. One student (in the Q4 section) pointed out that he had figured things out through trial and error from a previous course that used Moodle and that his rating of 5 on this question ("not needing any instruction on using Moodle") should not be taken as a complete endorsement of the ease of use and that There is still plenty of streamlining that it needs. Thus, there is a fairly strong case for covering the basics of Moodle (enrolling in a course, accessing resources, uploading an assignment, contributing to a discussion forum) in the Technology Literacy sessions for incoming freshmen.

Student responses to question six (does Moodle help or hinder your learning) were split into roughly three groups [Q6:chart]. Half the respondents (51%) considered Moodle to be useful but not essential, but a quarter (24%) thought that it greatly enhanced their learning (an encouraging statistic), while 20% were neutral and reckoned they could have done the same without it. Thus even in this early trial with most faculty using Moodle for the first time the student's perceptions of the pedagogical value of Moodle were on the whole very positive. This suggests that with perhaps more guidance for 'best practices' with using Moodle there is potential for a consistent positive influence on course pedagogy.

For a faculty using Moodle for the first time there is not just the question of how to use the system, but also of what to use and when to employ it. In the Information Technology and Society course I discovered that students need to be directed explicitly to access their Moodle course; they would not check Moodle regularly as they would their email. This was borne out by responses to question 7. 8% of respondents only accessed the Moodle course when they received an announcement in their email (postings made by the teacher to the 'News Forum' get emailed to all students enrolled in the course) and 6% used it only when explicitly directed so to do during class time [Q7:chart]. However, 50% accessed their Moodle course in order to do an assignment and 30% would even check occasionally to see whether there was anything new. A mere 5% checked it every day as they would their email. This pattern of usage was confirmed by the answers to question nine. Fully a quarter of students used Moodle only when their teacher explicitly told them so to do [Q9:chart]; this implies that if these students were not told to use Moodle to revise for a quiz or to research for an assignment they would not have done so! However, an extra 19% used their initiative to look on Moodle for resources relating to an assignment, and 42% would do this together with using Moodle resources to revise for a quiz or exam [Q9:chart]. Here again, a mere 3% checked for new resources independently of proximate need.

Unlike some other systems (eg Chef) Moodle allows the teacher to input and display the contents of the course and to customise the course layout. 50% of students agreed that having the course work scaffolded in this way was helpful [Q8:chart] and another quarter of the students strongly agreed that this was helpful. This confirms the pedagogic value of the Weekly or Topic outline and also suggests that it's important to make this outline clear and comprehensible.

The response to question 10 was interesting [Q10:chart]; students were equally split in wanting to submit assignments online and not wanting to do this. Written responses [Q10:comments] mirrored this result. About a third of the comments pointed out that they already submitted assignments online and that it was fun, it worked great, and I really liked it. Saving paper was given as a reson for submitting assignments online, It is absolutely imperative that we reduce the paper usage on campuses across the world and Moodle is an integral asset in accomplishing these goals. Some students expressed a strong preference for turning in hard copy assignments for similar reasons to this: I feel like turning in a hard copy is a way to make sure that there is no fault except the student's for not getting in an assignment on time. One student commented that i want a hardcopy back, with comments for all my work and this is certainly one area where an improvement is needed in the Moodle system (the ability of a teacher to return a graded assignment). Technical issues were also given for why online submissions were not a good idea; for example, my math homework is usually hand-written, which wouldn't work well online. (but Moodle can accomodate Tex math scripting as Jennifer Ziebarth is discovering in her Calculus B class), similarly worksheet assignments were not suitable for online submission, and formatting and display issues were cited as concerns as was the use of Microsoft software, I do not use Micro$oft word. {As an aside, there are a lot of advantages to not using Microsoft software such as Internet Explorer, Word, Powerpoint, Excel, with Moodle or any other online system [link]}.

Earlham students are nothing if not opinionated and so it was no surprise that 70% of the total number of respondents gave their likes and dislikes of the Moodle system (question 11, Q11:comments). Taking the positive comments first we can discern some themes. Students really liked having resources available online so that they could cover things missed in class or study the material for exams (this was especially the case with Powerpoint presentations - many students commented on the value of having them available outside of class time). However, one student did whine about the extra information in Powerpoint notes being included in the exam (evidently he did not review the presentations online before the exam). The availablity, ease of use, immediacy and accessibility of the course on Moodle were positive factors that were mentioned; students appreciated having all the course material (especially homework assignments) in one place (centrally located, localized) together with the straightforward user interface. In particular, the Course Outline display was liked, The approachable quality of class content online was what i liked.. But the user interface did come in for some criticism; design flaws were mentioned and one student commented about the colours used; why that heinous pink. it makes me anxious. it looks like errant pepto-bismaly epidermis. {Yes, we need to pay some attention to the styling of our installation - colour scheme and text fonts & sizes}. Ability to easily upload assignments was mentioned several times as a plus and availability of assignment grades online was a common theme, eg assignments - submitting with this system was easy and useful, grades - nice to be able to check and keep up with status. Students appreciated the savings that accrued from being able to access resources online — good way to cut overhead on course costs — and they also appreciated the fast response to questions, the fact that I could ask questions and ppl would send me answers back faster than if I had tried to call. Another pedagogically powerful feature which was liked was the online feedback to assignments. Finally, for one student the best part of using Moodle had little directly to do with class; My favorite part of moodle was definitely the profile option. I found it so liberating to post a photograph to represent my online persona! I also enjoyed viewing the "avatars" of others, not to mention being able to see when the last time everyone logged on was. — {I think that this 'social' use of Moodle is the way to get students 'hooked' into the system. The question would then become - what features should we look for to encourage the formation of a dynamic online community?}.

Some negative aspects ascribed to Moodle [Q11:comments] were a consequence of how teachers used it; thus the complaints that grades were not kept up or posted at all. One student appreciated this; would have liked all class notes to be there before during and after the pertinent class but that was up to the teacher's disgression (sic), but another complained about The fact that my late assingments wouldn't be graded.!! Other students expressed a willingness to read resources online to conserve paper but were concerned about the wastage of paper when they had to print out because of eye strain. {Printing is really an issue that needs to be addressed}. technical difficulties proved to be bothersome; some students did not have the Microsoft Office suite on their machines and so could not access Word documents, others were annoyed by the security check my computer has to do through before loading each page — Internet Explorer doesn't handle the secure site properly — and the fact that word, excel & powerpoint files were displayed within Internet Explorer was also aggravating to some— Word documents should not display when I click them. They should download, and not be displayed in-line . {Both of these issues are a 'feature' of Internet Explorer and the solution is to use Mozilla Firefox}. Students pointed out some design flaws; the poor design of the gradebook in terms of display, there needs to be like an excel-looking matrix with just the dates, and the way it works, inability to know whether or not an assignment has been graded, or the teacher simply hasnt gotten to it {a 'hold/release' mechanism would be useful where faculty could 'hold' the graded assignments and then 'release' them when they have all been completed. Also, a feedback note on the assignment page with assignment seen, currently grading, etc}. A major flaw in Moodle is the absence of group working features, thus one student hated not being able to post things or communicate with group members easily and another felt it would be nice if moodle had drop boxes like chef. A frequent theme was a perceived slowness in loading pages or general sluggishness of the system some of which was a consequence of the teacher uploading large Acrobat or Powerpoint files which then took a long time to display, but the Moodle system also did not scale well with displaying large amounts of information. For example on the Weekly outline for Information Technology and Society course all users had to switch to 'show individual week' view because with each week taking between 1 & 2 printed pages the total size was too large to display on a single page — 'show all weeks' option displayed only up to half way through week 7. Similarly students were unable to see all their journal entries because the display showed the whole of the instructor's prompt as well as their response for each Journal entry— when we view the journal page it shows the entire journal and feedback instead of just a link to the journal. It takes a long time to load. If it was a link it would be better. Finally, students wanted better integration with Earlham's systems— no link to moodle from earlham website — and one student expressed a desire for a universal login feature which would be supplied by a 'portal' system My least favorite part of moodle was that I had to log in every single time. After the first few times, typing in my password got so tiring that if it weren't for the awesome services moodle provided, I wouldn't have had the heart to do it.

Question 12 asked students to think of other ways that Moodle could have been used for their course. A persistent theme was posting grades and keeping them updated (5 responses) — Maybe posting our test grades, both letter and percent and having an available reference to be able to refer back to of what the range of points was for say an A or B or C etc — and students were also keen to have more ways to communicate with each other; to discuss with other students out of your view. like a student only message board. but really, viewable to nonstudents. interactive discussions with T.As!!!! and I think it would have been helpful to have postings from those in the class (questions or concerns) and then responses from other classmates and/or TAs and the prof when they had time. Discussion groups, open chat forums and 'drop boxes' for group work were among the ideas that students came up with.

Despite prior comments about problems with sluggish performance, responses to question 13 were upbeat [Q13:chart]. A third of students considered Moodle to have 'good enough' response time, while 40% thought it 'reasonably responsive' and 9% very responsive. Finally, only 19% of students rated overall response as 'slow'. This pattern was mirrored in responses to question 14 41% agreed that Moodle was as responsive from dorm as lab and 25% disagreed [Q14:chart]. However, the written comments showed widespread dissatisfaction with internet response from the dorms and college houses; wireless connections were perceived as being inferior (i have wireless in a campus house, which never is as responsive as the landlines the computer labs have. The campus houses need more reliable internet. I want to use ethernet again.) and one student commented on the chronic problem of students' machines infected with viruses, worms, backdoor apps etc — Traffic can be seen from infected Windows machines all the time, and NetSquid seems to not catch them ... Lines are permanently saturated with activity on all student lines ... Methinks it is time for ECS to drastically reduce the number of infected machines. Clearly if we expect Moodle to be more widely used outside of class some improvement needs to be made in bandwidth available to students accessing the Moodle server.

go to top of page

Conclusions : Moodle in Context

Clearly, the use of Moodle in this pilot period has been a success from the student's point of view. Despite some negative vibes only one student expressed opposition to any form of online course system; i did not like using a system like moodle. not necessaryily moodle spacificaly but more as a general dislike for such a system. Moodle has the potential to become as useful, universal and indispensible as email. Moodle as an online learning environment has the potential radically to change teaching at Earlham. However, to make it a success of this venture we need to pay attention to the context of which the system is a part.

with Training

  • Workshops for Faculty.
  • other faculty training opportunities
  • Technology Literacy for incoming freshmen
  • out of class sessions for students (how)

with Teaching classes

  • Moodle 'best practices'
  • Moving coursework online
    • clear and directive instructions
    • online assignments and grading
    • workflow issues
    • online grade

with student software

  • creating resource files that are accessible to all students - including those who don't have MS Office, are Mac users, dial in via a modem.
    • Use Acrobat format instead of Word, Excel or Powerpoint or use HTML (best)
    • For short answer assignments use Journal (inline) instead of Assignment (upload file)
    • 'Light' file size - keep file sizes to 10k or less.
  • awareness of technophobes

with network / printing facilities

  • printing - duplex printing / pritning kiosks (currently being implemented by ECS)
  • address network bandwidth problems in dorms and college houses
    • address bandwidth pollution from infected machines
  • tweaks to the Moodle server to improve performance and scalability with large Moodle pages

Moodle Improvements

  • improved gradebook with hold/release system for assignment grades
  • ability for student groups to share files
  • ability for faculty to return corrected assignments
  • ability for student to send notes to prof or other students (student initiated)
  • restyle user interface

 

go to top of page