Marietta van Rooyen
In the November 2006 Cutting Edge I published an article about the lessons learned from animal welfare in the USA (View previous article here) regarding critical control points when doing quality assurance.
The author of the book Animals in Translation is Temple Grandin. Her approach to the animal welfare audits she does for the US Department of Agriculture is to analyse the critical control points in a farm animal?s well-being.
She defines a critical control point as a single measurable element that covers a multitude of sins. I suggested that this approach to quality assurance could be adopted in South Africa regarding the simplification and practicality of quality audits of ETQAs and providers.
Quality assurance in education and training
Quality assurance in ETD practices is a given and has not been questioned in our industry. The Association of Private Providers of ETD (APPETD) has often assisted quality assurance bodies to draw up the criteria for accreditation. They also actively lobby the Department of Education to point out the crippling effect of the registration regulations on ETD businesses and NGOs.
Most private providers are proud of the quality of their delivery and do not want to exploit learners, as is often insinuated. It is clear enough from the media that public providers are not totally innocent of exploitation of learners either, and at times their quality of delivery is dismal. All providers need to take their quality of delivery much more seriously and ask themselves how they can better transfer skills and knowledge to learners.
Some examples of the irrational application of quality criteria
At provider forums and in discussions with colleagues I often come across providers that are on the brink of giving up on accreditation and are totally frustrated with the service they get from ETQAs.
Let me give you some recent examples of reasons for refusing accreditation of providers or just making life so difficult for providers that they end up throwing in the towel, due to bungling from various ETQAs.
1. Small business provider of distance education for one course aimed at on-sight training of employees (Level 5):
o Library "not extensive enough' with no library staff or electronic system.
o No qualified specialist in distance education on the staff.
o Not enough computer access at the premises for learners.
o "Too many' part-time academic staff members.
o "Not enough' admin staff (this despite all admin being up to date and correct).
2. Accredited SMME provider for full qualifications needs to re-accredit for the individual skills programmes within these same qualifications.
3. Accredited provider with registered assessors may not do RPL assessment unless they are registered as an "RPL provider'.
4. Large reputable provider being moderated on assessments:
o Learners did not spend enough time in the classroom. No matter that they are now declared competent and left to go back to work, no certificate will be issued unless they are called back to come and sit more hours in the classroom.
o Integrated assessment not accepted by moderator in spite of clear evidence of competence being shown.
5. Large reputable teacher trainer cannot be accredited due to the fact that they will not appoint the required staff before they are accredited. How can they? The process has already taken more than a year. Are they supposed to pay the staff for doing nothing?
6. NGO cannot educate learners (at no cost to the learners) due to the fact that their first registration was rejected and now they have to wait two years before they may apply again for registration. No matter that all the issues found wanting had long since been addressed.
7. Large provider of thousands of learners nationally not registered for months due to DoE bungling of the paper-work. Matter has since been settled in court in favour of the provider and at the expense of the Department of Education.
8. The lack of a culture of service in some of the ETQA administrative processes makes one wonder that any providers ever get accreditation. Some battle for years. I know of cases where the registration of assessors is months in arrears and many applications are lost as a matter of course.
9. Finally, to make things even more difficult, providers are subject to long delays in payments due to slack administration and lack of management skills in client organisations, especially public organisations.
A case of not seeing the wood for all the trees
Providers end up being confused and demoralised by the hundreds of requirements and the constant shifting of goalposts. I think it used to be much easier to be accredited in the early days, as the criteria were fairly straight forward and uncomplicated.
As the article of November 2006 points out, some officials are interpreting criteria so literally that hardly any provider has a chance to make the grade. Many auditors think that a simple system cannot work. According to Temple (the author mentioned above) they make the following mistakes:
? They write auditing standards that are too subjective and vague, with requirements like "sufficient library resources' and "enough admin staff members'. Individual officials and evaluators then have to figure out for themselves what exactly these requirements are. A good audit checklist has objective and measurable standards and criteria.
? For some reason, ETD quality systems and criteria mostly focus on measuring inputs, such as hours in the classroom, research done, and library facilities, instead of outputs, such as learner throughput and uptake into employment.
? Officials simply make the audit too complicated. A 100-item checklist does not work nearly as well as a 10-item checklist, and this can be proved.
? Evaluators drift into paper audits, in which they audit a provider?s records instead of its results.
? People tend to lose sight of what is important in education and training, and end up treating small problems the same way they treat big problems (such as arrows pointing to doors).
These mistakes actually harm the learners and the providers.
Basic principles in ETD quality assurance
My organisation is an ISO-listed private provider and I chair the ETQA Sub-committee at SAQA, I thus get to see the results of many audits. In addition, our College carries out many audits of other providers on behalf of several ETQAs.
I find that these audits are very bulky and could well be erring in exactly the way Temple warns about. The lists of items that both ETQAs and providers need to comply with to be accredited are simply too long and confusing! We need to find the critical control points that will make for good provisioning. That means that we need to find some measurable elements that "cover a multitude of sins' in terms of learning systems.
Make it simple and incremental
If you give an auditor a 100-item checklist, he will tend to treat 50 items as important, while perhaps only 10 items are critical enough to warrant an adverse audit. When a provider fails one of ten critical items, it is clear that the provider should get an adverse audit.
But when the same item is one of a hundred, it does not seem to warrant the de-accreditation of a provider, yet it may be a critical shortcoming. By complicating the list with hundreds of detailed items, the auditor stands a good chance of not seeing the wood for all the trees.
Temple points out that the danger about paper audits with 100-item checklists is that they can set you up for a situation where things slowly deteriorate without anyone realising it.
When you drift away from the reason for education and training and start auditing the paperwork, the bad can become normal pretty quickly. Paper audits end up masking small, incremental declines in standards that result in very large drops in education and training standards.
Evidence of compliance with criteria
In some cases auditors and evaluators base findings on the wrong evidence. An example of such wrong evidence based on inputs ifs the matter of having a library. One should not require a provider to have a library and librarian, but rather ensure that HET learners are actually encouraged to do research and critical reading during their studies.
These skills can be assessed through appropriate assignments and assessment instruments, which are more objective. It seems to me that the moderation of assessment instruments and activities used will supply much more reliable evidence of actual research taking place than libraries that may be outdated in any case.
At Assessment College we analysed the work done with several ETQAs and hundreds of providers to come up with the critical control points that could set the standard for good practice in ETD delivery.
These are the points we came up with:
1. A Quality Management System that includes proper policies and procedures governing the business of education and training. There must also be evidence that this QMS is used routinely and that the aim is continuous improvement.
2. The QMS system must focus on improving learner throughput per programme.
3. Learning materials need to be aligned to unit standards.
4. Properly moderated assessment instruments at the right level and focussed on outcomes need to be available.
5. Qualified facilitators must be appointed, either part-time or permanent.
6. Registered assessors and moderators need to be available internally or outsourced.
7. A practical assessment and moderation system suited to the subject matter must be in use.
All the other requirements usually listed should fall somewhere within these seven critical control points. We believe each organisation will need to start at its own baseline and set objectives for improving their services and delivery as they grow.
External and internal auditing will assist the management to understand where and how they can improve and where they need to focus for better delivery and throughput.
Animals in Translation; Temple Grandin and Catherine Johnson, Scribner, New York: 2005 ISBN-13: 978-0-15-603144-2
Marietta van Rooyen is Executive Chair of Assessment College