EFFECTIVENESS OF WEB BASED PBL USING COURSE
MANAGEMENT TECHNOLOGIES: A CASE STUDY
Havva H. Basak
Department of Computer Engineering
Yasar University, Turkey
Serdar Ayan
School of Maritime Business and Management
Dokuz Eylül University, Turkey
Keywords: Web-based learning environment, problem based learning, maritime business education.
Abstract: Maritime education and training has typically focused on delivering practical courses for a practical
vocation. In the modern environment, maritime personnel now need to be more professional, more open to
change and more business-like in their thinking. This has led to changes in the education system that
supports the maritime industries. Teaching thinking skills has become a major agenda for education.
Problem Based Learning is a part of this thinking. Problem-Based Learning (PBL) within a web-based
environment in the delivery of an undergraduate courses has been investigated. The effects was evaluated
by comparing the performances of the students using the web-based PBL and comparing the outcomes with
those of the traditional PBL. The outcomes of the experiments was positive. By having real life problems as
focal points and students as active problem-solvers, the learning paradigm would shift towards the
attainment of higher thinking skills.
1 INTRODUCTION
The Problem-Based Learning (PBL) approach is not
new to training. Since its implementation in medical
training in the early sixties (Thomas and Chan,
2002), the parameters of education have taken on
new dimensions with the world becoming more
compact, and information more fluid and immediate.
The rapidity of knowledge change means that
learners have little time to reflect on the knowledge
presented. They most often surface learn the area of
knowledge and then discard it as soon as the need is
over. They prefer to be directed in the learning
agenda and choose to be told rather than search to
find out information. In line with the new work
environment “where workers are given increased
responsibilities, including individual and team based
goal-setting responsibility and greater latitude to
achieve these goals”, it is felt that learners need to be
given more opportunities for self direction
(Gijselaers and Segers, 2002).
The advances of Computer Mediated
Communication (CMC) present an enormous shift in
the manner in which PBL is conducted. The
provision of synchronous and asynchronous
collaboration and the availability of enormous and
ever expanding course related web-pages and
materials provide a new dimension to the PBL
approach. The process of PBL that requires the
individual to seek information and knowledge to
construct new understanding, meanings and
concepts and the collaboration between peers
towards the solution of authentic and real-world
problems can be readily supported by current
communication media based on the computer and
ICT (Corderoy and Copper, 2000).
Apart from the role of CMC in supporting the
process of conventional face-to-face PBL, attempts
have also been carried out to incorporate the entirety
of the PBL learning processes in the web based
learning environment suitable for the delivery of
courses in open and distance learning (Dick and
Carey, 1996),(Harper-Marinick, 2001),(Koschmann,
2002),(MacAlpine and Clements, 2001). In open and
distance learning, there exists a spatial and time gap
between students and teachers and the ability of
CMC to surmount the physical and temporal
constraints makes a web-based PBL approach for
170
H. Basak H. and Ayan S. (2006).
EFFECTIVENESS OF WEB BASED PBL USING COURSE MANAGEMENT TECHNOLOGIES: A CASE STUDY.
In Proceedings of the First International Conference on Software and Data Technologies, pages 170-176
DOI: 10.5220/0001321501700176
Copyright
c
SciTePress
course delivery particularly useful and
advantageous. The web-based PBL involves the
creation of instructional materials that facilitate
problem presentation; the required self-
investigations and analysis can be conducted
through online resources and the social interaction
for peer-peer collaboration and student-teacher
facilitation can likewise be easily performed through
asynchronous forum boards or synchronous chats.
The final presentation of the answers to the problem
can also be easily conducted through the array of
educational media technology tools made available
by CMC.
There have been several studies that attempt to
evaluate the effectiveness of web-based PBL
(Norman and Schmidt, 1992) Reported that students
perceive the web-based PBL to be motivating,
providing access to greater richness of resources and
developing collaborative networks; the skills that
they subsequently acquire are appropriate for their
future professional activities. (Orill, 2002) Revealed
that with a higher degree of student control related to
aspects such as content, the instructional path, pace
and feedback, the PBL approach contributes
significantly to improved cognitive gains. (Ronteltap
and Eurelings, 2002) Highlighted that the
combination of synchronous and asynchronous tools
in the web-based PBL leads to deeper levels of
information processing when students compose
documents that represent their personal knowledge
based on their research from online resources.
The importance of the web-based PBL especially for
course delivery in distance and open learning, a
study that attempts to evaluate the effectiveness of
this approach relative to the widely available online
Content-Based Learning (CBL) both in terms of
students’ performances and perceptions (Gijselaers
and Segers, 2002).
According to (Chickering and Gamson, 1991) and
Chickering and Ehrmann, 1996), positive online-
learning environments incorporate seven principles
of good teaching practice. These should include:
• encouraging students-faculty contact,
• encouraging cooperation among students,
• encouraging active learning,
• giving prompt feedback,
• emphasizing time on task,
• communicating high expectations, and
• respecting diverse talents and ways of
learning.
One important outcome of any educational effort is
to prepare the candidate to face the challenges in the
working environment. How can that be measured?
Donald Kirkpatrick’s model can be used to address
the knowledge, skill & abilities needed to
successfully apply the online learning material in the
“real world” (Kirkpatrick, 1996). Kirkpatrick’s four
evaluation measures are reaction, learning, behavior,
and results.
For the evaluation of the relative effectiveness web-
based PBL and traditional PBL we will find the
answers of the following questions:
1. Effectiveness of the web-based PBL
environment compared with traditional
PBL in terms of students academic
achievements
Table 1: Kirkpatrick’s four levels (Alliger, et al. 1998, Tannebaum 1998).
Level Definition Questions Addressed Guidelines for Measuring Each Level
Level 1:
Reaction
Assesses participants’ initial reactions to a course.
Offers insights into participants’ satisfaction, or the
effectiveness (value) of the training as perceived by the
trainee. Usually assessed through a survey aka a
"smiley sheet."
Does not measure learning.
Were the participants pleased?
What do they plan to do with what they learned?
Determine what you want to find out.
Design a form that allows questionnaire results to be easily tabulated.
Encourage honest written comments and suggestions. Attain an immediate
response rate of 100 percent.
Develop standards.
Measure reactions against the standards and take appropriate action.
Communicate participants’ reactions.
Use focus groups to acquire qualitative feedback (i.e. more specific comments)
Level 2:
Learning
Assesses the amount of information (principles, facts
and techniques) understood and absorbed by trainees.
May use a criterion-referenced test
What skills, knowledge, or attitudes have
changed? By how much?
Use a control group, if feasible.
Evaluate knowledge, skills, or attitudes both before and after the training.
Attain a response rate of 100 percent.
Use the results of the evaluation to take appropriate action.
Level 3:
Behavior,
or Transfer
Assesses the amount of material used on-the-job after
taking the course, e.g. a week to 6 months (or longer)
after taking the course. Assesses on-the-job behavior
based on the objectives of the course and assessed
through tests, observations, surveys and interviews.
Did the participants change their behavior based
on what was learned in the program?
Use a control group, if feasible.
Allow enough time for a change in behavior to take place.
Survey or interview one or more of the following groups: trainees, their bosses,
their subordinates, and others who often observe trainees' behavior on the job.
Choose a statistically significant sample, or 100 employees.
Repeat the evaluation.
Consider the cost of evaluation versus the potential benefits.
Level 4:
Business
results
Measures results, e.g. reduced costs, higher quality,
increased production, and lower rates of employee
turnover. Measure 6 mos. to 2 yrs. after completing the
course.
Did the change in on-the-job behavior positively
affect the organization?
Use a control group, if feasible.
Allow time for results to be achieved. Amount of time depends on course
context.
Measure both before and after training. Repeat the measurement.
Consider the cost of evaluation versus the potential benefits.
EFFECTIVENESS OF WEB BASED PBL USING COURSE MANAGEMENT TECHNOLOGIES: A CASE STUDY
171
2. Students perceptions
3. Teacher perceptions
The results of the research questions could provide
institutions with alternative delivery methods
through open and distance learning. Also, this will
lead the students to deeper levels of information
processing when students compose documents that
represent their personal knowledge based on their
research from online resources.
2 THE DESIGN OF THE SYSTEM
In order to examine the effectiveness of web-based
PBL in terms of the students’ academic enhancement,
specially designed web based systems were developed
and installed online. The system was designed
according to the PBL approach which acted as an
experimental system that would include the tools to
support traditional Content-based sitting in classroom.
For the conference room we provided some tool to the
users that were included within the synchronous
distance learning environment. These tools include,
Discussion Forum, File Exchange, E-mail, Real-time
Chat, Video Services, and Whiteboards. All mentioned
tools ware developed by using flash scripting and PHP
programming language for all the user interfaces.
(Cakir and Basak, 2005)
2.1 Student Evaluation of the
Course and the Instructor
Student learning was evaluated using the Teaching
Effective Questionnaire (TEQ). The TEQ instrument
consists of 10 items that ask students to rate the
instructor and the course on a Likert scale ranging from
1 to 5 with 1 being strongly disagree and 5 strongly
agree. The items and the mean scores on each item are
shown in the Tables below. The mean scores are
reasonably high and comparable to those obtained by
the author for a similar course taught in a traditional
face-to-face PBL classroom. The other tables show the
mean scores for a similar class taught by the same
author in a WEB based PBL Clasroom environment.
2.2 Evaluating Students Learning
Use of multiple assessment techniques is necessary
to derive reliable results when evaluating students
learning outcomes. In this course, students’ learning
outcomes were assessed by a variety of means that
include individual assignments, group projects,
participation in weekly chat sessions, participation in
bulletin board discussions and participation in group
discussion sessions. In each of the activities,
students were graded based on a rubric which is
specific to that particular activity. The rubric for
each activity is available to students at the beginning
of each activity so that they know exactly what areas
they will be assessed on. Table 1 shows a rubric that
was used to grade the first simulation project. The
assessment techniques are quite comprehensive and
thorough, and because of the fact that comments are
written, which usually requires more effort from
both the students and instructor, it means there is a
lot of detail involved. One of the advantages of this
is that all communication between instructor and
students is automatically archived. Faculty members
and students can access transcripts of past chats to
determine levels of participation and accuracy or to
review guidance and explanations. Providing
students with immediate feedback is another practice
that helps students derive maximum benefit from the
online learning experience. Actually, in this course
students consistently pointed out that the one thing
they appreciated most was the immediate feedback
they received on their assignments and projects.
Table 2: The mean scores for the items Traditional PBL
Room scores.
Instructor Questions
Score
From
Traditional
Score
From
Online
Explained Course
objectives clearly
4.12 4.32
was well prepared
for class sessions
3.24 4.97
made effective use
of class time
4.51 4.46
Explained concepts
and ideas clearly
3.22 4.11
Answered questions
in a helpful way
3.47 3.20
was willing to meet
with students
outside of class
time
3.41 4.32
assigned grades
fairly
4.24 4.43
made the course
content interesting
3.24 4.11
significantly
increased my
understanding of
the subject matter
4.32 4.14
instructor is an
excellent teacher
4.22 4.71
Mean
A
—Mean
B
=-0.478 t=-2.42 df=9 P=0.038611
ICSOFT 2006 - INTERNATIONAL CONFERENCE ON SOFTWARE AND DATA TECHNOLOGIES
172
Table 3: The mean scores for the items Scores on both
Sites about Scenarios.
Scenarios Questions Score
Story 3.31
Fluency 4.11
Learning goal 4.53
Real life 3.89
Table 4: The mean scores for the items Problem Based
Learning Group.
Group Questions
Score
From
Traditional
Score
From
Online
Performance in
group
3.44 4.01
Share knowledge 4.10 4.37
Attending activities 3.73 4.24
Tried enough 3.82 4.14
Mean
A
—Mean
B
= -0.4175 t= -5.76 df= 3 P= 0.010399
Table 5: The mean scores for the items – Myself.
Myself Questions
Score
From
Traditional
Score
From
Online
Use of knowledge 3.21 4.60
cross-examine 4.49 4.01
Communication 4.22 4.72
Contribution to
group
3.51 4.12
Evaluation ability 3.84 4.51
Mean
A
—Mean
B
= -0.538 t= -1.8 df= 4 P= 0.146238
Table 6: The mean scores for the items - Scientific
Consultant.
Scientific
Consultant
Questions
Score
From
Traditional
Score
From
Online
Behavior of
Consultant to
you
3.61 4.64
Knowledge of
Consultant
4.56 4.06
Time spend to
solve your
problem
4.29 4.73
Mean
A
—Mean
B
=-0.3233 t= -0.73 df= 2 P= 0.541316
In general, traditional PBL room teacher uses a
whiteboard, handouts and pre-prepared documents
for students. Books and library resources are also
provided online to the learners. Students read
research and prepare themselves for the class before
the lecture hour. In the lecture, students discuss the
solutions of the problems given, writing findings on
a white board in a face to face collaboration and
discussions environment by tool with the teacher.
Students can use their online statistical analysis tools
for finding solutions to the problems. On the other
hand, online PBL room, we provide all necessary
tools like calculator, statistical web based
mathematical tools to support the online
collaboration (macromedia, http).
The system has been tested by eight students. The
system consists of three sittings that consist of at
least 4 sections. Each section has an unstructured
problem to be solved and at least 5 to 8 questions
about the problem. Students discuss the possible
solutions of those problems. On each answer
students discuss with collaborative environment
provided by the system and the possible outcomes
are noted on the whiteboard provided to them. In the
traditional PBL room students are evaluated by the
teacher whose performances are evaluated during
the PBL session. For each session the teacher fills
these forms manually and these files are collected by
the department for the end of semester grading
performance. Students also evaluate the teacher
which questions are explained in the above tables.
These results are collected by the department also
for the evaluation grade of the teacher at the end of
the semester.
Processes explained above, are moved online with all
the facilities and tools provided to both parties.
Moreover, we provide more tools for the students that
are not available in a traditional PBL room like, a
statistical mathematical package for WEB. Library
links are also provided for easy access to the materials
that supports the online PBL classroom. For online
collaboration, we provide synchronous voice, video
and chat tools which are effectively supported by
Macromedia Inc. (Poon et al., 2004). To support real
time transport we have used the Real Time Transport
Protocol which is highly supported nowadays by IP
networks anywhere in the world. However, for rural
area
low speed internet connections we provide
lower rate video frames like 10 frames per second.
The system has been implemented at the department
of Maritime Business and Management of the Dokuz
Eylül University.
PBL System
The experimental web-based PBL system as
explained in the above paragraphs was used in an
undergraduate course at Dokuz Eylül University.
The design was adapted on the model suggested by
Harper-Marinick (2001) and consisted of the
following sequences of learning,
1. Introductory information – introducing
the process of PBL and the role it should
play to accomplish the learning tasks.
EFFECTIVENESS OF WEB BASED PBL USING COURSE MANAGEMENT TECHNOLOGIES: A CASE STUDY
173
2. Presentation of an ill-structured and
real-world problem – serves as the
organizing centre and context of learning.
3. Online collaboration – discussion
among peers to propose the hypotheses and
identification of learning issues. The group
subsequently delegated responsibilities to
each individual to find out more
information about learning issues.
4. Online resources – each individual was
engaged in individual online research on
the learning issue assigned to him.
5. Follow-up online collaboration –
students reported on the research done,
identifying overlapping issues, and
discussing the new hypotheses and learning
issues.
6. Solution to the problem - students
collectively agreed upon the solution of the
problem and the plan of the presentation.
3 RESEARCH METHODS
The sample in the present study consisted of first-
year undergraduate Maritime Business and
Management students, a group tested by the
department of Maritime Business and Management
at Dokuz Eylül University in the 2005-2006 Fall
Academic semester. Total of 8 students enrolled in
this test. All students were the third year students
who have a three year traditional PBL experiment.
The schematic representation of the experimental
design is given in Figure 1. The measured dependent
variables in this study were the knowledge
enhancement both in terms of academic performance
and students’ perceptions. The independent variables
were the instructional design of the web page.
The post-test consisted of subjective questions listed
above, assessing and evaluating the students’ actual
academic attainments and understanding pertaining
to the respective lesson’s learning objectives.
A test involving a different sample was also carried
out to ensure the internal consistency of the
formative questionnaire as well to improve the
design of the Web based system for the experiment.
The data collected were analyzed using a standard
statistical package.
4 DISCUSSIONS
Tables 2-6 show the comparative analysis between
the means of the traditional PBL room and online
PBL room approaches. As can been seen, there is no
significant difference between the mean marks of the
two approaches, implying that both groups were
homogeneous in terms of the background knowledge
before the treatments were carried out. Any
discrepancy that might have existed in the
background knowledge between the samples prior to
the treatment would not have any influence on the
results of the comparative academic performances
and perceptual analysis.
It is evident that both approaches produced a consi-
Figure 1: Online Problem Based Learning Support System Design.
Virtual Learning Environment
Curriculum
Delivery
Assessment Teacher Support
Communication Logging
Online
Learning
Offline
Learning
Session Tracking
Export System
xml + txt +html
MYSQL
Database
User Interfaces
ICSOFT 2006 - INTERNATIONAL CONFERENCE ON SOFTWARE AND DATA TECHNOLOGIES
174
-derable enhancement in terms of the knowledge
constructed by the students. However,when the two
approaches are compared, it is evident that the PBL
approach yielded a superior learning enhancement.
The students’ perceptual analysis in terms of
knowledge enhancement was carried out by
comparing the questionnaire between the two
approaches. The results of the analysis are shown in
tables 2-6 with no significant difference between the
two approaches. Nevertheless, it is interesting to
note that all the items recorded higher mean
values
for the online PBL approaches. It is, therefore, quite
evident that these results highlighted the
advantageous features of the online PBL approach as
perceived by the students. The higher knowledge
enhancement through the PBL approach perceived
by them complemented the above findings of
superior academic performances following the
application of the online PBL approach compared to
the traditional approach.
5 CONCLUSIONS
The effeciveness in teaching business and
management online can be as effective as classroom
teaching especially when instructors use content to
help students acquire learning skills, use content to
promote self-awareness of learning by students, and
let students use content so that they experience it
firsthand. The effectiveness of the online web based
PBL approach both in terms of the students’
academic performances and perceptions compared to
the commonly available traditional PBL and web
based online PBL approach. The results show that
the web-based PBL approach has the ability to
perform the web based approach.
Other advantageous features of the WEB based PBL
approach have also been indicated, such as learning
through social interaction, acquisition of kills in
metacognitive reasoning and proficiency in problem
solving in the workplace context. As such, efforts
should be taken by the institutions of open and
distance learning to consider the web-based PBL
approach as one of the mechanisms for the delivery
of courses in their educational program.
The most important issue is not whether business
and management should be taught online or in a
classroom, the issue is whether the course promotes
a learner-centered approach. A course is likely to
achieve its objectives if students become the center
of the instructional universe, and when the content
functions as a means as well as an end of instruction.
REFERENCES
Mary Thomas and Lai Pheng Chan , (2002), Achieving
Learner Independence Using the Problem-Based
Learning (PBL) Approach Journal of Language and
Linguistics Volume 1 Number 3 2002 ISSN 1475 -
8989
Arts J.A.R., Gijselaers W.H. and Segers, M.S.R. (2002).
Cognitive effects of an authentic computer- supported,
problem-based learning environment. Instructional
Science, 30, 465-495.
Boud, D. and Feletti, G. (1991). The Challenge of PBL.
London: Kogan. Atan, Sulaiman and Idrus 437
McAlpine, I. and Clements, R. (2001). Problem based
learning in the design of a multimedia project.
Australian Journal of Educational Technology, 17(2),
115-130. [Online]. Available:
http://www.ascilite.org.au/ajet/ajet17/mcalpine.html.
[26th June 2005].
Corderoy, R.M. and Copper, P. (2000). The development
of an online Problem Based Learning Environment to
support the development of engineering professional
practice skills: the Virtual Engineering Consultancy
Company (VECC). Indian Journal of Open Learning,
9(3), 339-350.
Dick, W. and Carey, L. (1996). The Systematic Design of
Instruction. (4th Edition). New York: Harper Collins
Publishing.
Harper-Marinick, M. (2001). Engaging students in
problem-based learning. Maricopa Centre for Learning
and Instruction. [Online].
http://www.mcli.dist.maricopa.edu/forum/spr01/t11.ht
ml [Accessed 23 June, 2005].
Koschmann, T. (2002). Introduction to special issues on
studying collaboration on distributed PBL
environments. Distance Education, 23(1), 5-9.
MacAlpine, I and Clements, R. (2001). Problem-based
learning in the design of a multimedia project.
Australian Journal of Educational Technology, 17(2),
115-130.
Norman, G.R. and Schmidt, H.G. (1992). The
psychological basis of problem based learning: A
review of the evidence. Academic Medicine 67, 557-
565.
Orill, C.H. (2002). Supporting on-line PBL: Design
considerations for supporting distributed problem
solving. Distance Education, 23 (1), 41-57.
Ronteltap, F., and Eurelings, A. (2002). Activity and
Interaction of Students in an Electronic Learning
Environments for Problem-Based Learning. Distance
Education, 23(1), 11-22.
Chickering, A. W., & Gamson, Z. F. (Eds). (1991).
Applying the seven principles for good practice in
undergraduate education (New directions for teaching
and learning, No. 47). San Francisco: Jossey.
EFFECTIVENESS OF WEB BASED PBL USING COURSE MANAGEMENT TECHNOLOGIES: A CASE STUDY
175
Chickering, A. W, & Ehrmann, S.C. (1996). Implementing
the seven principles: Technology as lever. AAHE
Bulletin, 49(2), 3-6.
Kirkpatrick, D. (1996). Revisiting Kirkpatrick’s four-
level-model. Training & Development, 1, 54-57.
16. Schramm, R. M., Wagner, R. J., & Werner, J. M.
(2001). Student perceptions of the effectiveness of
web-based courses. NABTE Review, 27, 57-62.
Arbaugh, J. B. (2000b) Virtual classroom versus physical
classroom: An exploratory comparison of class
discussion patterns and student learning in an
asynchronous Internet-based MBA course. Journal of
Management Education, 24(2), 207-227.
Cakir, S., Basak, H.H., “Virtual Classroom
Implementation on the Web”. 2005 IEEE
International Professional Communication
Conference(IPCC), 10-13 July, 2005. vol: 1, p:318 –
322. Limerick, Ireland.
http://www.macromedia.com
Poon, S.K., Reed, S. and Tang, C. (1997). Problem-based
learning in distance education. Proceedings of the 5th
International Conference on Modern Industrial
Training, Jinan, China. 593-600. [Online].
http://www.ic.pdyu.edu.hk/ocss/papers/PBL.pdf
[Accessed June 13, 2004].
Rontelap F. and Eurelings A. (2002). Activity and
interactions in an electronic learning environment for
problem-based learning. Distance Education, 23(1),
11-22.
Taplin, M., Leung, A., Chan M.S., Wan, C.W., Kuen,
K.S., Lam, M., Mok, V., Lee, A.N.O., Siaw, I.R.C.,
Swearse, B. and Tsui, C. (1999). The role of problem-
based learning in distance education. In D. Xingfu
(Ed.) Proceedings of the 13th Annual Conference of
Asian Association of Open Universities, pp 175-189,
Beijing, 14-17 October, Vol 2.
Varanelli, A., Baugher, D. and Hall, J. (2001). A problem-
based, collaboration learning approach to distance
education at the MBA level:e.MBA@PACE. Business,
Education and Technology Journal, Spring, 36-44.
ICSOFT 2006 - INTERNATIONAL CONFERENCE ON SOFTWARE AND DATA TECHNOLOGIES
176
