HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS British Journal of Psychiatry Advances in Psychiatric Treatment All RCPsych Journals		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) Submit an eLetter Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via CrossRef Citing Articles via Google Scholar Google Scholar Articles by Currie, C. Articles by Scott, J. Search for Related Content PubMed Articles by Currie, C. Articles by Scott, J.

Evaluating the quality of educational materials about schizophrenia

Department of Psychological Medicine, Academic Centre, Gartnavel Royal Hospital, Greater Glasgow Primary Care NHS Trust, Glasgow, G12 0XH

	Abstract

Top Abstract Introduction Method Results Discussion References

 
AIMS AND METHOD

To identify and evaluate the reliability and quality of educational materials provided to individuals with schizophrenia and their carers. Materials used by mental health professionals working in community and in-patient settings were collated. Two independent raters used the ‘Discern’ questionnaire to assess the publications.

RESULTS

Fifteen documents were identified, but only 11 were suitable for evaluation. Interrater reliability of ratings using the Discern tool was highly significant. No educational package scored maximum marks, but four scored in the good quality range. About a fifth of the materials in widespread use were assessed as of poor quality.

CLINICAL IMPLICATIONS

The use of a rating instrument to assess the quality of educational publications appears to be a reliable and acceptable way to identify the strengths and weaknesses of widely available materials. Staff can use these data to assess the quality of their preferred materials against other publications and make an informed selection.

	Introduction

Top Abstract Introduction Method Results Discussion References

 
Leaflets and other information packages are regarded as integral to educational strategies designed to promote physical health (Coulter, 1998). These packages provide patients with information to support their participation in treatment decisions and in the management of their health problems (Coulter, 1997). Much work goes into the production of educational materials. However, to be effective they must contain comprehensible, relevant and accurate information. Unfortunately, few patient information materials currently meet these standards and many leaflets are inaccurate or misleading (Coulter, 1998).

This study evaluates the quality and usefulness of educational materials about schizophrenia that were regularly used by mental health professionals working at a large primary care trust in Scotland. This evaluation was important because psychoeducation is clearly recommended as part of the basic psychosocial management of schizophrenia (Scottish Intercollegiate Guidelines Network, 1998). Also, a baseline audit of trust practice suggested a need for greater rigour in the provision of information and education to patients, families and carers.

Various tools have been advocated to enhance the quality of health information (Entwistle et al, 1996; Jadad & Gagliardi, 1998; Silberg et al, 1997). The NHS Direct website identifies the Discern questionnaire as the first standardised index of the quality of consumer health information (Charnock, 1998). Discern has undergone detailed development and review and particularly assesses the quality of written information about treatment choices. However, there are no publications regarding its use in the assessment of educational materials about mental health topics. This study had the following objectives:

1. To evaluate the quality and usefulness of educational materials about schizophrenia used regularly by staff at the Greater Glasgow Primary Care Trust.
   
2. To document the strengths and weaknesses of all the packages reviewed.
   
3. To comment on the acceptability and reliability of the Discern tool in assessing mental health information.
   

	Method

Top Abstract Introduction Method Results Discussion References

 
The study began with a stocktake exercise to collate materials that were currently used by mental health professionals working in community and in-patient settings. Locality teams were contacted and asked for information on and copies of educational materials being used within their service. A list of the educational materials provided was compiled and sent back to the same staff seeking confirmation that the data-set was complete.

Each piece of educational material was initially assessed to determine if it included comments on treatment choices and whether it was widely available to staff in other trusts. Two independent raters then assessed each publication using the Discern questionnaire. This comprises 16 questions rated on a 1-5 scale. The questionnaire is divided into three sections. Section one (eight questions) concerned reliability, section two (seven questions) measured the quality of information on treatment choices and section three (one question) was an overall rating based on the preceding 15 questions. The scores for each section and the overall rating ranged from 1-5. Ratings using Discern are classified as good, fair or poor according to the following criteria:

Good — score = 4 or 5. The publication rated highly on the majority of questions. A high overall score indicates the publication is a useful and appropriate source of information about treatment choices.

Fair — score = 3. The publication rated high and low on a similar number of questions, or most scores fell in the mid-range. A moderate overall rating suggests the material is a useful source of information on treatment choices but has some limitations. Additional supporting information would be required.

Poor — score = 1 or 2. The publication rated low on most questions. A low overall quality rating indicates the publication is poor quality — it has serious shortcomings and is not a useful or appropriate source of information about treatment choices. It is unlikely to be of benefit and should not be used.

Interrater reliability was recorded and a score calculated. All scored publications were summarised for strengths and weaknesses. The overall findings and ratings were compiled into a table and fed back to the clinical teams.

	Results

Top Abstract Introduction Method Results Discussion References

 
Fifteen publications were identified as being in regular use in the trust for patients with schizophrenia, their families and carers. Four were excluded from further assessment. Two publications were linked specifically to pharmacological products and were not considered suitable for generic use. Two other packages were excluded as they were used as part of a local psychoeducation project on schizophrenia, were undergoing review and were not widely available outside of the trust. Eleven publications were independently scrutinised using the Discern tool. Interrater reliability for overall rating was very good, with a of 0.87 (P<0.0001). A review of ratings of each Discern question showed the largest difference in ratings (only two marks on the item rating scale) occurred on only one occasion.

As shown in Table 1 none of the publications achieved either a maximum score or a minimum score, indicating that every publication had at least some good aspects but also some limitations. Only two documents showed disparities in the ratings of section one (reliability of information) and section two (quality of information on treatment choices) of the Discern questionnaire. The first (Wilkinson & Kendrick, 1996) scored 3.3 for reliability and 5.0 for quality of information. The second (PatientWise, 1997) scored 1.9 and 3.2 on sections one and two, respectively. Based on the Discern ratings, four publications currently in use could be recommended with confidence and two recommended for withdrawal. The remaining five publications got a moderate or fair score, indicating some benefits if used in conjunction with other educational materials.

Interestingly, there were three weaknesses common to all documents: most failed to provide details of any external appraisal of the information presented, sources of information were not adequately described or dated and areas of uncertainty were not clearly acknowledged. Other recurring weaknesses were failure to provide references, inadequate descriptions of the risks of treatment and failure to consider the ‘no treatment’ option. Last, the date of publication was only printed on four of the documents.

	Discussion

Top Abstract Introduction Method Results Discussion References

 
This exercise to evaluate the quality and usefulness of the main schizophrenia educational material in the trust showed that most of the material was at least of fair quality. Although the overall score of each publication does not provide any great insight into the strengths and weaknesses of a publication, the Discern questionnaire can provide a greater level of detail if item scores are reviewed.

Having used Discern, the documents that scored most highly shared the following characteristics:

• the documents started with clear aims on what the publication was about, what it was to cover and who it was for
  
• information was provided that was specific and relevant to its target audience
  
• sources of information and dates of publications were clear
  
• details of other sources of appropriate information on the topic were provided
  
• the mechanism of how treatment actually works, and its risks and benefits were detailed
  
• information was objective and uncertainties were acknowledged
  
• the consequences of not having treatment were detailed
  
• choices in treatment were highlighted
  
• the idea of shared decision-making was promoted.
  

The Discern questionnaire was relatively easy to use and interrater reliability for assessments was good. The learning time was also acceptable, taking about 1-2 hours initially to become familiar with the rationale and layout of the document. However, rating each document took 1-3 hours as the reader has to become very familiar with the publication being rated to ensure each Discern question was rated as directed. There are also one or two limitations of the Discern document. First, the developers clearly state that it cannot be used to assess the scientific quality or accuracy of the evidence on which a publication is based — this means information has to be checked against other sources. Second, Discern is described as a stand-alone tool, allowing judgement of a publication without specialist knowledge of the topic under review. Having used the tool, this is something that the project group would not entirely agree with. For example, knowledge on the possible mechanisms of action of certain treatments is required in order to assess if a document described this fully. The same applies to judgements of the range, benefits and risks of treatments. However, these limitations aside, this is a robust tool that can be used to evaluate medical and psychiatric educational materials.

Finally, it is important to note that a major advantage of using the Discern questionnaire and then providing detailed feedback on ratings of reliability and quality of information, is that staff and service users were able to make informed choices about which educational materials to access. This approach avoids the problems that can arise if staff perceive practice guidelines as too prescriptive. To ensure that assessment of the quality of publications used is maintained in the future, one member of staff from each locality team has been trained in the use of Discern so that they can evaluate the quality of new educational materials and provide feedback to their team members.

Qualitative details on the strengths and weaknesses of each document reviewed are available from the first author.

	References

Top Abstract Introduction Method Results Discussion References

 
CHARNOCK, D. (1998) The DISCERN Handbook: Quality Criteria for Consumer Health information: User Guide and Training Resource. Oxon: Radcliffe.

COULTER, A. (1997) Partnerships with patients: the pros and cons of shared clinical decision-making. Journal of Health Services Research and Policy, 2, 112-121.

COULTER, A. (1998) Evidence based patient information is important, so there needs to be a national strategy to ensure it. BMJ, 317, 225-226.[Free Full Text]

ENTWISTLE, V., WATT, I., DAVIS, H.J., et al (1996) Developing information materials to present the findings of technology assessments to consumers. International Journal of Technology Assessment in Health Care, 8, 425-437.

HEALTH EDUCATION BOARD FOR SCOTLAND (1994) Talking About Schizophrenia. Edinburgh: HEBS.

JADAD, A. & GAGLIARDI, A. (1998) Rating health information on the internet: navigating to knowledge or to Babel? JAMA, 279, 611-614.[Abstract/Free Full Text]

LEFF, J., BERKOWITZ, R., EBERLEIN-FRIES, R. J., et al (1991) Schizophrenia: Notes for Relatives and Friends. Kingston-upon-Thames: NSF.

LILLY PSYCHIATRY (1997) Understanding Schizophrenia: A Guide for Sufferers and Carers. London: Lilly/NSF.

NATIONAL SCHIZOPHRENIA FELLOWSHIP (1996) Caring and Coping. Kingston-upon-Thames: NSF/G.P.L.

NATIONAL SCHIZOPHRENIA FELLOWSHIP (1998) Schizophrenia. Kingstonupon-Thames: NSF.

NATIONAL SCHIZOPHRENIA FELLOWSHIP & ROYAL COLLEGE OF PSYCHIATRISTS (1998) Schizophrenia. (Help is at hand leaflet) London: RCPsych/NSF. (http://www.rcpsych.ac.uk/info.help.schiz )

PATIENTWISE (1997) Medical and Health Information for Patients: Schizophrenia. Chichester: John Wiley & Sons Ltd.

PRELAPSE (1995) Coping with Schizophrenia. London: Prelapse.

ROYAL COLLEGE OF PSYCHIATRISTS (2000) Schizophrenia (Changing Minds). London: Royal College of Psychiatrists. (http://www.rcpsych.ac.uk/campaigns/cminds/leaflets/sch/sch.htm )

SANE (1993) Schrizophrenia: The Forgotten Illness. London: Sane Publications.

SCOTTISH INTERCOLLEGIATE GUIDELINES NETWORK (1998) Psychosocial Interventions in the Management of Schizophrenia. Edinburgh: Scottish Executive.

SILBERG, W., LUNDBERG, G. & MUSACCHIO, R. (1997) Assessing, controlling and assuring the quality of medical information on the Internet. JAMA, 277, 1244-1245.[CrossRef][Medline]

WILKINSON, G. & KENDRICK, T. (1996) A Carer's Guide to Schizophrenia. London: Royal Society of Medicine Press.

This article has been cited by other articles:

				P. Timms, D. Hart, A. Cohen, M. Briscoe, and M. McClure 'Help is at Hand'on the web - what do our readers think? Psychiatr. Bull., January 1, 2005; 29(1): 24 - 27. [Abstract] [Full Text] [PDF]

This Article Abstract Full Text (PDF) Submit an eLetter Alert me when this article is cited Alert me when eLetters are posted Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Download to citation manager Citing Articles Citing Articles via HighWire Citing Articles via CrossRef Citing Articles via Google Scholar Google Scholar Articles by Currie, C. Articles by Scott, J. Search for Related Content PubMed Articles by Currie, C. Articles by Scott, J.