The Australian Institute of Management (AIM) is a not for profit  membership organisation, dedicated to enhancing the capacity of managers, leaders and organizations.  We do this through a range of services in business and management training, events and seminars, coaching, research and publications, supported by a specialist management bookshop and library services . For more information about our organisation please visit our website http://www.aimnsw.com.au.

In 2009 we were successful in acquiring funding through the Australian Flexible Learning Framework (Framework) to further embed e-learning into our training practices.  This project provided AIM with an opportunity to extend our current assessment support process by developing a collaborative online assessment model, designed to support learners who are undertaking work-based assessment with AIM.

Project implementation and sustainability

The project lifecycle ran from April 2009 to November 2009.

The following five phases were implemented to keep the project team on track and to ensure that we met our project outcomes:

1. A meeting of key stakeholder and the formalisation of the team
2. Investigate and select appropriate e-tools
3. Induct AIM staff, develop e-learning resources and undertake preliminary trials.
4. Participants undertake Frontline Management training and assessment activities and provide feedback using various evaluation methods
5. Conduct interim review and undertake end of project activities, including project reporting and attendance at elearning 09.

The problem

Competing demands upon learner’s time often prevents them from successfully completing their assessment tasks. Our model assisted our learners to integrate assessment within their existing workload. The support methodologies included the provision of coaching (for example, evidence collection or developing and maintaining an assessment plan) as well as opportunities to collaborate with peers. We trialled the supported assessment process with students who had completed face to face training in the Certificate IV. The intention was to develop a framework using e-learning tools and strategies, that would lead participants to a successful outcome. This process is transferable to work-based assessment in any management subject.

The solution

The project trialled a supported assessment process using e-learning tools and strategies for employees of the NSW Ambulance Service who have completed face to face training in the Certificate IV in Frontline Management. The intention was to develop a framework that would lead participants to a successful outcome and introduce e-learning tools. This experience was documented in a blog which will include an assessment model and a case study.

The process

In the beginning….

At the beginning our team imagined a collaborative process which may involve asynchronous discussion between participants and assessors and a virtual conference room to provide one-on-one or group assistance and to gather feedback from participants throughout the project.  So we began with an with an exploration into the tools and strategies we might adopt for the project.  We conducted some of our meetings face to face and others in the Discovere virtual conference room and considered a Ning social network three wikis – Wikispaces, Wetpaint, and a PB Wiki we also considered the Moodle learning management system.  We used a Wikispace to document our progress.

To evaluate the various tools we considered the following questions:

• What are the features of the technology?
• Can the assessor provide instruction and materials?
• Can the learners/assessor conduct discussions?
• Can the learners easily access the materials and upload their work?
• Does the technology provide opportunities for collaboration?
• On a scale of 1-10 how easy or difficult is this technology likely to be for our learners/assessors to grasp?
• What sort of support might learners need to use the technology?
• Are there likely to be barriers such as firewalls, download restrictions etc.?
• Do the features of the technology support the assessment process? If yes, how? If no, why no

Most of the software we evaluated fulfilled most of the criteria we had set.  However it was felt that the Ning social network was just that – a social network, and was a little informal for assessment.  Participant’s evidence could be uploaded with a discussion post but we wanted to provide a single page for each participant so that all uploaded materials were together.  This did not appear to be possible in the Ning. The Moodle learning management system was felt to be a complex environment and may have led to a sharper learning curve for our team and for participants.  As well, AIM did not require a second learning management system. The three wiki’s fulfilled all our criteria.  Wetpaint was considered to be the most user-friendly and the most visually appealing and Wetpaint became our chosen environment.  At this stage we still intended to use Discovere to work with individual participants, small groups and/or for feedback.

The next step was to present out ideas to an appropriate client and the NSW Ambulance Service agreed to participate in the pilot.

Developing the wiki

We developed a timeline for the project and our idea was that the trainer/assessor would be intimately involved in the development of the environment.  This was considered to be a critical part of the process as ownership is a key motivator to engagement.   Marcelle was highly motivated and enthusiastic from the outset.

The development of the site was undertaken as a series of face to face meetings with some development in between and took approximately18 days.

The content for our project existed in paper-based form and the AIM compliance department was concerned that our pilot group did not receive any advantages that were not available to other groups of students.  Consequently they requested that the assessment questions and evidence requirements remained the same as the paper-based assessment which was usually undertaken.  On the whole we were able to comply with this request although some questions were adapted so that knowledge sharing and collaboration was built into the provision of evidence through the discussions.

Hear what Alison has to say about the design and development of the site.

We began with an introductory video and some general information about the site and divided the site into six topics to be completed over six weeks.

Each topic was created on a separate page and contained two or three discussion questions and a work-based task.  Participants were asked to upload evidence of their task, for example a roster they had prepared, as well as third party reports to substantiate that the work was their own.  Each participant had their own evidence page.  We also created pages for instructions which were provided as downloadable text documents and screen capture videos created with Jing, and we provided information about assessment.

Our assessment model guided participants through conceptual discussions, to practical tasks and concluded with analytical discussions.

The pilot

AIM deliver the Certificate IV in Frontline Management as a face to face training program during which our pilot group were introduced to the project by their trainer, Marcelle.  We felt that for the pilot were to be successful, it was important to provide continuity for our learners as they moved from the physical classroom into the online environment. We achieved this by having the classroom trainer facilitate in the online environment. However, AIM policy is that the trainer cannot also be the assessor.  We believe this adds authenticity to the process.  The solution was a second assessor who worked closely with Marcelle and who ultimately had the final say as to whether participants were found competent or not yet competent in each task.  Sue Nunn joined our team and tells us about her experience as an assessor.

The unexpected hiccups

A number of unexpected hiccups occurred during the trial:

The invitations

We were unable to demonstrate the site in the training session as internet access was not available. We had prepared handouts and powerpoint slides beforehand  but they were not as good as the real thing and we could not ask participants to log on at that time.

Immediately after the training we sent out the invitations and Karen will tell us what happened!

Any hiccup at this early stage was bound to affect the participant’s (and the client’s) confidence in the system.   A delay to the start date meant that we had to organise an extension to the finish date, which is a complex procedure in our internal computer system. Also, for the sake of consistency across groups, we did not want to give this group more time than the others had had.  We did, however, offer participants an additional two weeks to complete their tasks.

Next time we will ask participants to access the site and set up their profiles before the training session.  Any issues which arise can be solved before the assessment process begins and the facilitator’s explanation about the site is likely to make more sense.

Participant profiles

Another issue was that participants set up their profiles using nicknames, and we were unable to identify who participants were!  Again, we felt that asking participants to register with the site before the training and providing clear instructions about what information does or doesn’t go into each profile would eliminate this problem.  In the future we will ask that profiles include:

• full name
• location
• job role
• contact details
• photo (optional)

We would, however encourage the use of nicknames as well, as we felt that it created an environment of camaraderie.

Accessing the site

Towards the end of the pilot project, we discovered that at least one participant was clicking on the wrong tab when he accessed the site. He was able to see and access discussion threads, but was not seeing all the peripheral support information.  We had provided instructions, but like many people, this participant had thought of them as a last resort, to be used if all else failed. He was able to get into the site, and didn’t realise he was not seeing everything.

Our solution will be to create an activity to be undertaken at the beginning of the assessment process that requires participants to access the peripheral information, and we will follow up if it is not completed correctly.

Giving feedback to participants

A trainer perspective

A management perspective

At the end of the project ninety percent of our participants had successfully attained all four units of competence – a significant improvement over our paper-based assessment model.

The results and the benefits

Evaluation of the project was embedded within our process. The project team met monthly to discuss progress and issues as they arose, and we stayed in regular contact by email throughout the life of the project.

The results and benefits from our project were captured using comments from our learners on the wiki and through telephone conversations at the end of the project. AIM also met with the NSW Ambulance Service management at the end of the project and recorded some video feedback.

Feedback from learners, the client, the assessor and the RTO have been extremely positive.

The learners

Some direct quotations from our learners follow:

“I think this has been an excellent way to do assessment. I believe that with more people it would promote even more interaction. I personally think I will take away more from this online module than a paper based assessment with no interaction or reading about other problems and solutions (both good and bad)”

“I might actually learn something about the net amongst other things”

“Love the idea, a lot easier for me to do it this way. It is also good to read everyone elses thoughts on the questions, the idea of learning from my peers is great”

“What a shame our other assessments could not be accessed online”

The client

The client reported that the project was a “great return on investment for the Ambulance Service”. Completion rates have improved substantially and the client stated that the convenience of being able to see how participants were progressing was a significant time-saver. They were also impressed by the collaboration between station managers (that is, the learners) who continued to learn practical day to day management practices from each other throughout the process.

Hear what Amanda Crammon, Learning and Development Manager from the NSW Ambulance Service has to say.

Another interesting observation was that

“the relationship between assessors and participants led to assessors being more motivated to get participants over the line [that is to achieve competence]. But it also worked the other way. The participants were more motivated to finish and make assessors proud”.

Amanda summed up the project for us.

The assessor

The assessors enjoyed the challenge and the interaction with students. Specific challenges were becoming familiar with the technology and learning how to adopt an appropriate tone in the online environment. They commented that a substantial amount of time was involved encouraging and motivating participants to begin and then to sustain their efforts.

Hear what Marcelle has to say.

The RTO

Our project alerted our management team to the fact that small groups throughout the organization are investigating various e-learning tools and strategies and has resulted in an organization-wide strategic approach to integrating e-learning within our training and assessment model. The model will include the online assessment process developed in this project and is likely to follow the steps outlined in the short slide show below.  Please click through the fives slides to view the model.

In summary, the supported assessment resulted in a less stressful assessment experience overall, through:

• the provision of online coaching
• setting achievable goals
• maintaining contact with the students throughout the assessment process
• understanding what constitutes evidence and guiding participants in a process to gather and produce their evidence which could then be uploaded electronically
• constructing knowledge through collaboration
• providing a flexible environment, where participants can log in at their convenience
• continually offering feedback to students on the development of their portfolio

By undertaking workplace assessment, learners were able to transfer their learning into their workplace leading to an immediate application of newly learned skills thereby embedding those skills within the learners practice.

Although e-learning is a growing trend, current perceptions within the business community remain focussed on classroom-based learning and assessment. This project has promoted and strengthened our position that e-learning strategies are accessible, social, and offer valuable career development opportunities for learners. We believe this project has begun to change the corporate perception about how learning and assessment can be undertaken.

Framework resources

A diverse range of Framework products and resources are available from the Framework website www.flexiblelearning.net.au to support teams with their e-learning initiatives. Here are some that we found useful.

• E-learning for Industry
• E-tools n Tips
• Innovate and Integrate
• Your guide to social e-learning

Framework connection

The national training system’s e-learning strategy, the Australian Flexible Learning Framework (Framework) funds and supports E-learning Innovations projects which aim to embed e-learning into the national training system by supporting and enabling innovation in training design and delivery, at the state and territory level.

Acknowledgement

This is a NSW E-learning Innovations project output, developed by the Australian Institute of Management, with seed funding from the Framework.

For more information
Australian Institute of Management
Karen Lee
Curriculum Development Manager
Byte Size Assistance
Ph: (02) 9956 3066
Email: karenle-at-aimnsw.com.au

For more information on the Australian Flexible Learning Framework:
Phone: (07) 3307 4700
Email: enquiries@flexiblelearning.net.au
Website: flexiblelearning.net.au

http://flexiblelearning.net.au