[Note: this is an expanded version of the paper delivered at EdMedia 2000.  The additions are highlighted in yellow.]

Facilitating Learning via Computer Conferencing: Aspirations, Requirements and Hard Facts.

Dr Stuart D Stein.

University of the West of England

United Kingdom

Stuart.Stein@uwe.ac.uk 

Abstract: This paper explores the validity of contentions made in the literature concerning the advantages of computer conferencing in facilitating learning. It asserts that much of the supporting research is lacking in conceptual clarity, methodological rigor and systematic analysis.  The findings have frequently emerged from studies conducted under conditions removed from everyday instructional contexts.  Consequently, these can only be generalised with difficulty to extra-research settings, if at all. The paper includes a report of case studies on the use of computer conferencing in settings that probably approximates more closely the typical instructional context in many institutions of higher education.  Non-moderated computer conferencing was employed on a number of courses and the results compared to others that employed traditional, non-electronically mediated, pedagogic approaches.  The results indicate that in the absence of any moderator input, students can achieve results comparable to those attained by more traditional methods. 

Overview 

The literature on computer conferencing is already very extensive and has periodically been subjected to extended reviews (e.g. Kaye 91; Mason 91; Paulsen 95; Berge)  Its presumed benefits derive, of course, not from the medium itself, but from it being an exemplar of collaborative work.  Currently it is widely accepted that collaborative work results in learning outcomes that are superior to those of individual scholarship.  In the list of benefits Kaye includes: (a) the achievement of deep-level understanding arising from conversation; (b) the development of general problem-solving skills and strategies; (c) the sharing of different perspectives; and, (d) the enhancement of communication skills through the externalisation of cognitive processing (op.cit., pp.3-4).  Although a significant volume of work has been undertaken to establish whether or not the deployment and use of electronic mediating technologies enhances learning outcomes, many findings have been called into question on conceptual, methodological and empirical grounds. (e.g. Schute; Russell, 99; Neal; Cooper and Robinson; Springer, et al. )  Moreover, the findings tend to be somewhat inconsistent, as Kaye (91) has noted. Although there appears to be a broad consensus that computer conferencing can enhance the learning experiences of students, especially as far as distance learning is concerned, not much evidence has been adduced that the learning/pedagogic outcomes are in some sense superior to those attainable via more traditional face-to-face interactive forums.     

Much of the literature is comprised of case studies of the ongoing experiences of educators.  Generally, this has a tendency to be methodologically unstructured.   It is often non-comparative in orientation, does not include adequate control groups, and is rarely enlightened by appropriate statistical analysis.  Even when some of these studies meet the minimal requirements for experimental design, the conclusions drawn by the authors could be construed in some cases as being somewhat problematic¹. Consequently, many of the findings are essentially anecdotal, being, for the most part, reports from the electronic frontier.  Although it may very well turn out in the medium to long term that many of these “common-sense” assertions are borne out by subsequent research, at this juncture many of the claims that have been made appear to be unproven, not a few of which are, in Popperian terms, untestable.   

To illustrate with one example, the literature abounds with claims concerning the benefits of computer mediated group work without specifying clearly what the dependent variables are, and what would constitute reasonably adequate confirmation. Thus, Henri 91, (see also Kaye 91; Harasim 89) contends that group-work “consistently yields results of a higher calibre than those attained by the average group member.  This is due to the greater amount of information available within the group, the greater diversity of interpretations of fact and the opportunity to test individual ideas.  …the combination of social osmosis, the circulation of ideas, and the links established among the participants all contribute to the efficiency of CMC group exchanges.” Perhaps, but a very elaborate and problematic research design would be required to provide evidence attesting to this (see Berthold M.R. et al., 97; Rafaeli et.al., 94, on mailing list interactivity). At present the most frequently employed objective evaluative criterion is grade performance, whilst the subjective counterpart is student satisfaction. Detailed qualitative analyses of exchanges are the only means of grappling with such wide reaching claims. The required research designs are intensively resource demanding, and the findings are unlikely to meet requirements of replication and objectivity, however loosely defined. (see Mason, 91; Berthold M.R. et al., 97; Rafaeli et.al., 94; Henri 91; Graham, et al., 99, on the need for qualitative assessments of content) 

In these reports there is all too often-insufficient consideration given to the large number of interacting variables plausibly associated with perceived outcomes, learning and other. Some of the most significant relate to characteristics associated with participants in conferencing exchanges.  There is no question that until very recently most case studies were based on experiences with distance education students. (e.g., Kaye 91; Mason 91; McConnell 91; McConnell, 94; Soby, 91) A disproportionate number involve students taking information technology related courses, students who can be expected to be more favourably disposed toward pedagogic programs employing electronic platforms (e.g. Hiltz 95).  The student subjects in many studies tend to be older than the typical campus-based student, and differ from them in other important respects which probably have a direct bearing on learning related outcomes.  Their motivation and commitments, their social relations, the nature of their social networks, their economic situation, and their pedagogic backgrounds and experiences, all potentially relevant to learning outcomes, are probably significantly different from those of the vast majority of students undertaking campus-based higher education courses.  Also, many of the case studies include relatively small numbers of students. (e.g. Warren and Rada, 98 [n1=15, n2=11)  

The list of potentially relevant variables is too lengthy to include a detailed exposition on here.  They include, but are by no means limited to: (1) The category of student [mature/school graduate; graduate/post-graduate; experienced electronic communication media users/inexperienced electronic communication media users; continuing education/life-long learning]. (2) The conferencing system employed and its organisation, [which relate to ease of use, difficulty of hierarchical inclusion and integration, feedback on activities engaged in by participants, availability of private and public fora, integration with other electronic applications and their transparency/ease of use];  (3) The extent to which the conferencing system is integrated with other electronic applications [e.g. use of the Web, electronic revision/testing applications]; (4) The role of the conferencing component in the overall structure of the delivery of pedagogic materials. (5) The overall pedagogic context and its structure. (6) The nature of the task undertaken. (7) The mode of assessment linked to the task undertaken.   

Two other clusters of variables that I wish to draw attention to here are the instructional context and instructor moderation. In the research literature on computer conferencing there is often a failure to emphasise the somewhat ideal conditions under which a significant number of these projects have been undertaken.  Many studies, as noted earlier, have been carried out on mature and highly motivated students, frequently undertaking courses at a distance. The sample sizes, or tutor/student ratios, are frequently relatively small (e.g., McConnell 91; Soby 91; Warren and Rada 88; Mason, 91; Heath 98).  Many studies have been funded with additional financial resources that have enabled instructors to carry out such projects unburdened by the same degree of teaching and administrative constraints that the majority of instructors in higher education institutions have to contend with.  Invariably such research is conducted in an educational context in which there is a relatively good tutor/student ratio.  For instance, in Mason’s (91, pp.161-2) study, the ratio was 18.1, whereas on the modules that I am responsible for the ratio is as high as 1:120. Also, a disproportionate number of studies reported tend to involve students who are taking courses in open and distance learning or are related to computing and information technology.  Rather less appear to involve, for instance, students taking basic science courses, philosophy, or languages.   

The instructional context in which the case studies described below were carried out approximate more closely the conditions described by Hiltz as obtaining at NJIT, namely, “first generation college students”, who must work while attending school and who commute rather than live on campus, who are faced with overcrowded classes and lecture halls, and who are tutored by staff who are themselves under severe time constraints.  

My use of the terminology instructional context subsumes the whole complex of pedagogical related experiences and programmes that the student encounters within the same time frame that includes the research project.   The performance of students in a collaborative group environment, whether mediated by electronic platforms or not, is affected by the way in which that experience melds with the pedagogical approaches employed in other courses that they are pursuing.  Many of the presumed benefits currently attributed to collaborative conferencing and other work might, in considerable measure, be a consequence of a high novelty factor, something that I would expect to be substantially dissipated if all courses were undertaken employing similar approaches.   

As far as computer mediated communications based on conferencing applications are concerned, the role of the moderator has frequently been taken to be of critical importance. (Brochet 89, quoted in Paulsen 95; Paulsen 95; Feenberg 89; Mason 91; Collins and Berge 99; Berge; Kaye 91 [“access to computer conferencing….is not a sufficient condition for successful electronic collaboration.    …the role and active presence of the team leader in the electronic environment is crucial to a successful outcome.”, p.12 He concludes that “the role of conference organiser in the CMC environment is the major critical factor in determining the quality of online collaboration in work groups.” P.12] ; McConnell 91) Numerous roles are allocated to the moderator in the literature, most of which are deemed to be important to the success of conferencing outcomes.  The more effectively implemented, the greater the benefits.  Paulsen, who reviews the literature, lists 15.  There is no reason to assume that if these roles are effectively adopted and implemented that students will not benefit.  However, and related to the point that I made earlier concerning the relationship between the research and typical instructional context, this literature also stresses the amount of time that is required to carry out such roles effectively.  Mason (91), in reviewing one such study, carried out under the auspices of the UK Open University, noted that the 300 students had the benefit of the attention of 16 tutors, in addition to a “super-tutor” who moderated and organized the conference, “under whom the other tutors made their own contributions.” This ratio of 1 to 18 is somewhat removed from that of 1 to 95-120 that applies with respect to the case studies that I discuss below. 

 Case Studies  

Context: 

The empirical study focuses on the analysis of computer conferencing by undergraduate students at the University of the West of England.  The students were drawn from two programmes.  Those enrolled in the Faculty of Economics and Social Science and undertaking one of the programmes of study in Social Science that it offers, and students who were taking an Inter-Faculty mix of programmes, one of the two strands being a programme offered by the Faculty of Economics and Social Science.    

The University of the West of England is one of the newer UK universities.  Located in Bristol, in south west England, it currently has an enrolment of some 28,000 students undertaking a wide variety of degrees, diplomas and certificates, ranging form sub-degree vocational courses to PhD programmes.  It has a mix of full and part-time students, weighted toward the former.  

The students who are the subjects of this study were all undertaking undergraduate programmes, the overwhelming majority of whom were 18 to 19 year-olds who entered university directly after the completion of their secondary education.  Hardly any of them had had any prior experience of using a conferencing system. All students undertaking programmes offered by the Faculty of Economics and Social Science are provided with the opportunity to attend IT induction sessions, which include familiarisation with the use of FirstClass conferencing, the majority of whom take this up.   

FirstClass conferencing is widely used by staff in the Faculty of Economics and Social Science, principally as a broadcast medium, providing information to students on administrative matters relevant to courses, and as a repository of articles and lecture notes relating to specific modules.  Other than in the modules discussed below, FirstClass conferencing in the Faculty of Economics and Social Science is not employed in interactive mode, nor are conferencing contributions used as part of course assessments. 

In the Faculty of Economics and Social Science the academic year is divided into three trimesters.  In practice only the first two are teaching terms, the summer term being used principally for examinations. 

Case Studies Course Modules: 

The students whose conferencing work forms the basis of the discussion and analysis below were taking the following course modules.  The term assigned work below refers to coursework other than tests or examinations. 

(i) Methods of Analysis.  A 1^st year undergraduate module that all students studying Social Science programmes are required to take.  Its aim is to provide students with critical appraisal of qualitative and quantitative research methods.  The analysis and discussion below is based on work undertaken by students during the first trimester of the 1999/2000 academic year. There were 119 conferences. The total number of students taking this course in the first term of 1999/00 =420.   Comparison was made with assigned written, non-conferencing, work carried out by students taking this course in 1998/99, the number of students =290 

(ii) Introduction to Social Psychology. This is a compulsory module for those majoring in Sociology but optional for all those taking other Social Science programmes.  It can be taken by 2nd and 3rd year students.  Analysis is based on assigned work undertaken during the first trimester of 1998/1999 N=61, divided into 19 conferences, membership ranging from 3 to 6, 11 of which had 4 members, one with 6. Comparison was made with performance on assigned and examination work on other modules taken by the students, excluding (iii) below. 

(iii) The Holocaust and Other Massacres. This is a compulsory module for those majoring in Sociology but optional for all those taking other Social Science programmes.  It can be taken by 2nd and 3rd year students.  Analysis based on those taking this course in 1998/99 divided into 22 conferences of between 2 and 6 members in the first trimester with 66 students, and 22 of between 1 and 5 in the second, also with 66 students.  There was only one conference with 1 member.  Most conferences had a membership of between 3 and 5, relatively evenly distributed in both trimesters. Comparison was made with performance on assigned work and overall results on other modules taken by students, excluding (ii) above. 

Students were posed essay type questions relating to the subject matter of the courses that they were undertaking and were required to discuss this through the medium of the FirstClass conferencing system.  Because of the large numbers of students involved, and student commitments and time constraints, very few of these students met face to face in connection with the fulfilment of conferencing course requirements.  Students were allocated to conferences alphabetically, or on the basis of surname after they had chosen an assignment topic.  They were assessed on the basis of the quality of the contributions that they made to the conferences, and written work that summarised their own conclusions concerning the essay questions.  Students were given detailed information concerning what was expected from them as far as the volume of contributions they should make, when these should be made, the range of issues that should be covered (e.g., methodological issues, relevant theoretical frameworks, status of empirical data), and, in the case of course (i) above, requirements concerning alternating turn-taking in assuming a lead and reactive/interactive contributing role.  All students were given the opportunity of having 30 minutes training using the conferencing system. 

The overall course grades for (ii) and (iii) above are made on the basis of the assigned work and a final examination, each contributing 50 percent to the total.  The assigned work was a mix of conferencing contributions and written work related to the question posed for conferencing contributions.  All other modules are also assessed on a combination of written coursework and final examinations, each contributing, with few exceptions, fifty percent of the total.  None of these other courses use conferencing for assessment purposes, although instructors on many courses employ the conferencing system as a broadcast medium. Students taking these courses take a total of six courses per year, with the exception of the very few part-time students.  The performance of the students on the conferencing courses was compared with their performance on the other courses that they took in the same academic year.  The comparison was with the average performance on coursework grades, examination grades, and the overall average of coursework and examination grades on these other courses.  In most cases these were averages of performance on five courses. However, in some instances the comparison was confined to a comparison of performance on the conferencing related module with performance on four other courses, as some students opted to study both courses in the same academic year.

As far as the methods course was concerned, (i) above, comparison was made between the assessed work on the conferencing system of students in the first term of the 1999/00 academic year, with that of written coursework undertaken by students taking the same course in the first term of the academic year 1998/99. These latter students did not use the conferencing system for assessed work in any courses during that year.

Although I designed the essay questions, wrote the guidelines relating to conferencing work requirements, and administered the conferencing system, the assessment of conferencing contributions, written, and examination work, was undertaken entirely independently by the course seminar instructors. The methods course had 10 seminar instructors, whereas (ii) and (iii) had one each. None of the conferences were moderated.  None of the instructors made any contributions to the conferencing discussions, with a few exceptions when it was necessary to intervene for administrative purposes, for instance, informing students that they were sending the message to the parent conference rather than to the project group conference.  Consequently, students undertook all conferencing work independently. 

Findings

With respect to modules, (ii) and (iii) student t-Tests, paired two samples for means, were carried out on a number of data series, the Null Hypothesis being that there was no difference between the means attained on conferencing work modules and the averaged performance of students on all other modules. For module (i) the t-Test two sample assuming unequal variance was employed. This was undertaken separately for the assigned coursework and examination grades.  No significant difference was found at the .01 level in respect of (iii) for the second term. There was, however, a significant difference for (iii) in the first term, the mean for coursework on the conferencing module being 3% higher than for the average for all other modules.  There was a 6.7% difference between the final mark on the conferencing module and the overall final mark, for (iii) in the first term.  Significant differences persisted when the tests were carried out separately for males and females.  However, for (iii), term two, no significant differences were found respecting any of the data series, although the means for performance on all of the data series relating to the conferencing modules were higher than those with which they were compared. The differences between terms (i) and (ii) could conceivably be interpreted as being a product of the wearing off of the novelty effect, but many other explanations are equally plausible. 

For module (ii), the only significant difference was between the means for the overall grade for the module and the average overall grade on other modules, the mean for the former being nearly 6% higher. The means in all other comparison were higher for the conferencing module.

For the methods course, (i) above, the t-Test between the written assignment work for the 1998/99 and the 1999/00 conferencing work was significantly different at the .01 level, the means differing by 4.2%, being higher for the conferencing work.  The t-Test between the 1998/99 and the 1999/00 written work was not significantly different at the 0.05 level, the mean for the former being higher. 

Case Studies Conclusions

The analysis, along with mixed effects regression and correlation analysis, not reported upon here, indicate that conferencing undertaken without the benefit of moderation is not reflected in poorer performance, as measured by grades in written and examination work, compared with courses using traditional higher education instructional modes.  In fact, students performed better on average on two of the conferencing modules (ii, iii), than they did on other coursers. There are various reasons that can be advanced for this.  As noted earlier, as the pedagogic context is one in which conferencing work is a novelty on the case study courses, this may impact on student interest, perseverance and motivation.  Another possible explanation is the impact of what social psychologists refer to as social facilitation. It is also possible that the subject matter of the assigned conferencing work is perceived by students to be more interesting than that than which they are required to undertake on other courses.  It is also plausible to argue that many of the benefits singled out by other researchers as being associated with collaborative work, are also active in the minimalist intervention computer conferencing contexts that applied in the present case studies. 

Summary and Conclusion

Although there is now a substantial body of literature dealing with varied matters relating to the deployment of interactive communication technologies in educational settings, we are still a long way from knowing or understanding how to deploy them effectively in order to achieve pedagogic outcomes.  This is hardly surprising given that disputes concerning the effectiveness of more traditional pedagogic mediums, such as lectures and seminars, are far from being settled. Some of the reasons are not difficult to track.  Much research has been anecdotal, case study oriented, findings being applicable to very specific contexts, and therefore non-replicable. Much research has been carried out under conditions that do not approximate those of typical educational settings. Many of the assertions deployed with missionary zeal by educational technology enthusiasts have not been systematically tested.  At the same time, it is possible to demonstrate that such technologies might be effectively deployed under less than ideal conditions to achieve modest improvements. 

The findings from the case studies reported on here, which found “no significant difference” with minimalist intervention in terms of moderation/facilitation, are in line with results from other studies on collaborative learning work in asynchronous learning networks (Alavi et al., 97; Benbunan-Fich and Hiltz, 99; Hiltz, 94; Russell, 1999).  Although the case studies therein reported on cover so many different mixtures of variables that it is very doubtful that the meta-conclusions are based on like being compared with like, at present it does seem clear that there is no significantly perceptible disadvantage associated with ALN based instruction at the college/higher education level.  This seems to apply whatever the degree of moderation/facilitation exercised.

It is not my position that “minimalist intervention,” that is, the absence of moderation/facilitation in collaborative ALN work, is an objective that educators should actually aspire to.  However, given that we cannot with any degree of certainty specify what the cognitive pedagogic advantages of moderation/facilitation are, or measure them, that ALN collaborative work may hold some appeal to students independent of the presumed benefits of moderation/facilitation (eg., novelty, social facilitation, transferable skill acquisition), and that for many educators time-management considerations would prevent anything other than cursory moderation, it appears that there are few reasons not to utilise ALN collaborative forums for teaching purposes.  The one caveat, of course, is that the structure of such non-moderated forums, and the tasks allocated to participants, need to be carefully considered. 

Footnotes

1. The recent study by Benbunan-Fich and Hiltz (99) can be used to illustrate this.  They employed an experimental design that was “a 2x2 factorial crossing teamwork (individual vs. group work) with communication support (manual-offline vs. asynchronous computer conference),” exploring differences between students undertaking a case study on “computer ethics”. The students were divided into four groups: (1) those who worked offline on their own; (2) those who worked offline in a group; (3) those who worked online on their own; (4) those who worked online in a group.  The online students employed a conferencing system application. The dependent variable was the reports produced by the students in the different conditions, which were evaluated by independent experts. The grade point averages achieved in the different conditions are reproduced in the table: 

References:

Bart P. B. (1999). Keeping Online Asynchronous Discussions on Topic. JALN Volume 3, Issue 2

Brochet, M. G. (1989). Effective moderation of computer conferences: Notes and suggestions. In M. G. Brochet (Ed.), Moderating Conferences. Guelph, Ontario: University of Guelph.

Collins, M. P, Berge, Z.L. (1999) Moderating Online Electronic Discussion Groups. Online at http://www.gl.umbc.edu/~berge/sur_aera97.html  

Cooper, J. &, Robinson, P. Small-group Instruction in Science, Mathematics, Engineering and Technology (SMET) Disciplines: A Status Report and an Agenda for the Future. http://www.wcer.wisc.edu/cl1/CL/resource/smallgrp.htm  Accessed 29/02/00

Cueso, J. (1992) Cooperative Learning Vs. Small-Group Discussions and Group Projects: The Critical Differences. Cooperative Learning and College Teaching, 2.3

Feenberg, A. (1989). The written world:On the theory and practice of Moderating Educational Computer Conferences. In R. Mason, A. Kaye (Eds.). Mindweave: Communications, Computers, and Distance Education (pp. 22-39). Oxford: Pergamon Press.

Graham, M &, Scarborough, H., & Goodwin, C.(1999). Implementing Computer Mediated Communication in an Undergraduate Course-A Practical Experience. JALN Volume 3, Issue 1

Harasim, L. (1989) Online education as a new domain.  In R D Mason, & A. R. Kaye (Eds.) Mindweave: Communication, Computers and Distance Education. Oxford: Pergamon Press. 

Heath, E.F. (1998) Two Cheers and a Pint of Worry: An On-Line Course in Political and Social Philosophy JALN, Vol. 2, Issue 1-March

Henri, F (1991). Computer Conferencing and Content Analysis. In  A.R.Kaye (Ed.) Collaborative Learning Through Computer Conferencing. Springer-Verlag, Berlin, 1991

Hiltz, S.R. (1994). The Virtual Classroom. Norwood, New Jersey: Ablex Publishing Corporation

Hiltz, S.R. (1995). Impacts of college-level courses via Asynchronous Learning Networks: Focus on Students. Sloan Conference on Asynchronous Learning Networks, Philadelphia. Draft.  

Kaye, A. R. (1991).  Learning Together Apart.  In  A.R.Kaye (Ed.) Collaborative Learning Through Computer Conferencing. Springer-Verlag, Berlin, 1991 

Mason, R. (1991) Analysing Computer Conferencing Interactions.  Computers in Adult Education and Training, Vol. 2, No. 3

 McConnell, D. (1991). Computer Mediated Communication for Management Learning. In  A.R.Kaye (Ed.) Collaborative Learning Through Computer Conferencing. Springer-Verlag, Berlin, 1991