HL Deb 06 November 1986 vol 481 cc1222-41

6.3 p.m.

Baroness Ewart-Biggs rose to ask Her Majesty's Government what steps they intend to take to increase the accessibility of information of concern to members of the public and, in particular, access to their personal files.

The noble Baroness said: My Lords, I should like to say how grateful I am to noble Lords who have stayed behind, though I quite understand if some noble Lords are now disappearing. However, I should like to say how grateful I am to noble Lords who are remaining to this last business at the end of a long and arduous parliamentary session. I think it shows a great fidelity to the subject that I am raising.

There are many views on what we should do about official secrecy—what information should be open and what should be protected. I have no doubt that most people believe that they ought to see the information public authorities and other institutions hold about themselves or regarding their own personal affairs: information which schools keep about their children; which doctors and hospitals keep about their health; which public authorities keep about their entitlement to housing and various benefits; the personnel records which employers keep or the records which social workers and other professionals hold.

I should like, as briefly as I can, to cover the following points. The first is my reasons for believing that a freer disclosure of such information would not only be right in principle but would also be advisable from a practical point of view. Secondly, I wish to describe the level of support which such a proposed measure is attracting. I should also like to outline the safeguards as contained in the draft Access to Personal Files Bill, safeguards which would counter any adverse effects that this great freedom would bring about. Lastly, I should like to say that I feel that the time is right to make progress of this kind.

From a moral point of view, I believe most sincerely that people should have a legal right to see records about themselves or their children. I think this is right in principle. I also think the practical arguments for freer access are overwhelming. I should like to describe some of these arguments. Such records are the basis often of crucial decisions about our lives and it is essential that we should be able to check them to make sure that any mistakes or false impressions are corrected.

One has recognised the fact that many people do not even realise that the authorities and institutions with whom they have dealt briefly at some time in their lives, hold files on them. Still less do they realise that information from such files may be passing from one agency to another. For example, when a person applies for a job or for life insurance it is usual now to be asked to authorise the taking up of a medical reference. The doctor is asked to give the employer or insurance company a report on his patient's health. In some cases the doctor may barely know him. He may just have arrived in the practice. However, the doctor will report on his physical health and perhaps also his mental health, using the records passed to him by a former doctor. The record may describe a condition which existed some time ago, but fail to report that that condition has since been completely cleared up. There may be mistakes. The person's employment or insurance prospects may thus be damaged. At the moment he is not entitled to see the report the doctor writes or the medical records the report is based on. There is no chance of any errors being corrected.

Another danger is that mistakes can affect the medical treatment that people are given. This is a very important point. If past symptons, treatments or tests are wrongly described an incorrect diagnosis may be reached and the real cause of illness may be overlooked. On a further point, a small survey of records at one medical practice reported in 1978 that doctors were ignorant of much of the information considered to be important for the clinical care of patients. The survey found that 50 per cent. of women who have suffered complications of pregnancy, miscarriage, stillbirth, terminations, have no note of this on their medical records. Ten patients in this practice had known allergies to drugs. However this was not recorded on their medical files.

There are many examples of these mistakes that I could give your Lordships. However, it is sufficient to say that one general practice in London which does encourage patients to look at their own records reported recently that as many as 12 per cent. of their patients discovered errors on their medical notes. Many of the errors were minor, but some were not. Some of these files had details relating to other patients which had been misfiled, and were included in their own notes. Mistakes are found not only in medical records; they can also occur in social services files. I know of one instance of an elderly lady whose file contained wildly different accounts of her supposed age. Her entitlement to various benefits depended entirely upon her age. The taking up of these benefits could well have been jeopardised by the false information. There can be little doubt that the most effective way of maintaining accurate records is by allowing the relevant person to correct the mistake himself or herself.

There is another reason why people may want to see their records; that is, to learn about themselves. To a person who has spent his or her childhood in care, the record kept by the social services department may be a very real and vital source of personal history. Denying such people access to their own records when they grow up seems a particular injustice. Others may wish to see their records because they are suspicious about their contents. They may think that decisions are being taken behind their backs, or that they have been misrepresented or ignored altogether.

Indeed, it has been shown that authorities which do allow access to their records report that it often works wonders in removing misplaced suspicion. People are able to see precisely how their affairs are being looked after and dealt with. Sometimes they have not realised what considerable efforts are being made to help them, and therefore they are greatly impressed by what they learn from their records.

It is sometimes suggested that a right of access to medical records would cause special problems. I am quite sure that most of us would prefer to learn about our health by asking our doctor directly. No one would suggest that we should stop doing so and turn to our records whenever we want information. There are all kinds of reasons why most of us would not wish to do so, not least of all perhaps the difficulty of reading the doctor's handwriting, though I must ask my noble friend Lord Rea to forgive me for saying that. However, there are times when people simply do not get the information they need from their doctors. The British Medical Journal reported this June on the results of a study into the way in which young doctors handled consultations with patients. It found that very few provided information about the cause or likely prognosis of an illness or about any tests that were being done.

To put the problem from the patient's point of view, I know of a woman who, when in hospital—and it has happened to many of us—described how she obtained information about her blood pressure. She looked at her chart when the nurse was not looking, and when the doctor came in, instead of finding out from him, she listened to the information that he was giving the students. I know that I have done that, especially when I have taken my children to hospital. I have obtained my information through listening to what the students were being told by the doctor.

Moreover, there are still too many cases of patients not being told their diagnosis when they are seriously ill, and too many cases of people learning not from their doctor but from some other source that they have a serious condition. Indeed, examples have been given in the Lancet of how people have learnt of their serious illness from a receptionist, or indeed in one case from a home help. Many of these patients had questioned their doctor about their illness, but had been put off, perhaps because the doctor feared that an emotional scene would result. Some patients may not want to be given such bad news, and of course it should never be forced on them. However, equally, patients should not be put in the position of having to force it out of the doctor. Surely the best solution must be that the person should be given the option of either informing himself through his records or remaining in ignorance. However, it is his decision rather than the doctor's.

I should like to say a few words about the attitudes of the medical profession, though of course my noble friend Lord Rea will be doing this. I regret the fact that the British Medical Association's annual meeting this year voted, by the narrowest of margins, against allowing patients any right to see their records. However, on the other hand, it is encouraging that the BMA's GPs' committee and also its ethical committee both recommended that patients should have a right of access, though they wanted the doctor to retain the opportunity to withhold information where it was felt that disclosure would harm the patient's health.

I should like to comment briefly about the support there is for the measure that I am suggesting. It is particularly encouraging to see that there is a high level of support from the professions for this principle of opening up the records. This has come from such bodies as the Institute of Management, the Institute of Housing, the British Association of Social Workers, the Royal College of Midwives, the Health Visitors Association, and many others. This morning, I received a letter from the National Consumers Council, which of course was very supportive of this particular measure.

Still more significant are the steps that some authorities have already taken, on their own initiative, to allow access to the records that they hold. This has been done by a number of education and housing authorities and by some doctors, whose patients are given their records to see while they wait. Also, a number of employers, including the IBM, British Coal, Swan Hunter Shipbuilders and British Gas allow employees to see their own personal records. Such records are on occasions also open to officers and soldiers in the army.

Therefore I think that this shows that the process of allowing greater openness has been started, and I am sure that it will come as no surprise to your Lordships to know that the measure is enormously popular with the public. A MORI poll commissioned in July this year found that of some 1,900 people questioned, as many as 73 per cent. wanted a right of access to their medical records. In the same survey, 67 per cent. Of those asked favoured a right of access by parents to their children's school records.

As your Lordships know, from November 1987 the Data Protection Act will give us the right to see and correct personal information held on computer, and this is a very important step. However, our concern today is that the Act does not extend to records kept on paper, and I have no doubt that as people begin to exercise their new right, this anomaly will strike them forcefully. Perhaps it will even seem ridiculous. Why should their right to ensure that decisions about them are taken on the basis of accurate information depend on the form in which the information is recorded? Indeed, it seems ironic that this right should be determined according to whether or not a particular authority uses a computer. I cannot help feeling that it will be a great struggle to have two systems—one of open files held on the computer and one of closed manual files.

There are disturbing signs that some computer users are considering deliberately transferring what they regard as sensitive information from computer disk onto paper solely in order to avoid having to disclose it. Therefore it seems inevitable that soon there will be a very strong demand for the right of access to be extended to paper records. It is for this reason—in order to complete the process started by the Data Protection Act—that the Access to Personal Files Bill has been prepared.

I shall not go through the provisions of this Bill, but I should like to conclude on the protections that are being offered in the Bill by way of safeguards. The first safeguard protects personal information on a file which may relate to some third party—perhaps another member of the family. The second would protect the identity of members of the public who had provided information to an authority about the subject of the record. The third would allow information to be withheld where disclosure might expose someone to the risk of physical attack. The fourth would allow information to be withheld from the subject of the record where a doctor believed that he or she could be seriously harmed by the disclosure. This recognises that there may be occasions—for example, if a person is already in the midst of a severe emotional crisis—when it would be better to defer access to information about a particular traumatic event until the crisis had passed or counselling to help the person cope with the revelation could be provided.

Therefore I think that everything is moving in the same direction. We have a Data Protection Act; we have a Department of the Environment which has provided for applicants for council houses to check that the information they have supplied has been accurately recorded. The Education Act 1981 provided the right of access to some information to parents of children with special education needs. In 1983, the DHSS launched a consultation process on access to social work records and in a circular encouraged local authorities to adopt a more open approach.

As I say, I think that everything is now coming together to show that we want to go this last step and make files open for people to see, whether they are on computer or on paper. I believe that that would really show that the public are becoming a great deal more interested in health and medical information. They are showing a great deal more sophistication in this matter. Therefore the right of access would be to the benefit of everybody and it would support those individuals who, after all, wish to be more responsible for themselves. I very much hope that the Government will do everything possible to further this aim.

6.20 p.m.

Lord McNair

My Lords, this is not the first occasion, and I am sure it will not be the last, on which the House finds itself very much indebted to the noble Baroness, Lady Ewart-Biggs, for bringing before it an interesting subject and one which is of deep personal concern to a considerable number of people. She has covered the ground so comprehensively that she has not left a lot for the rest of us to say, but on past form I doubt whether we shall find that too inhibiting. There will be many specific areas in the much wider area, that she has covered that we can amplify or underline.

The first thing I want to do is to assure her that my noble friends on the Alliance Benches fully support this freedom of information campaign of which she has spoken. Indeed my Liberal Party has chosen as its president for this year Mr. Des Wilson, who is the chairman of the campaign committee. However, our support for this campaign should not, and does not, blind us to the fact that some progress has already been made in desirable directions.

I want to talk today about children, and it would be most ungenerous if I failed to remind your Lordships that the Children Act 1975 considerably increased the right of adopted children to discover their origins. I suggest that it is difficult to exaggerate the importance of measures of this sort. Those unforgettable words "know thyself", which were written for all to read in Apollo's Temple in Delphi, must surely be the best advice ever given to anybody by anybody else. For children who, through no fault of their own, have had a chequered upbringing self-knowledge may very likely be the essential prerequisite, without which they have little or no hope of achieving a sound, rational and non-neurotic attitude towards other people and towards the world into which they have, often so unthinkingly, been brought.

Under the Children Act an adopted child on reaching the age of 18—and I wonder whether perhaps it ought to have been 16—has a legal right to receive a copy of his or her original birth certificate, but only—and this is important—after careful counselling. He or she will at least learn from that the name of the natural mother and is then free to pursue what further inquiries he or she may wish. I am told, incidentally, that only a very small proportion of such children go so far as to seek out and confront their natural mother, but the Act is still young, it is only about 10 years old and it would probably be premature to draw statistical conclusions.

The adopted child, particularly in cases of what one might call traditional adoption, where the baby joins its adoptive parents at a very tender age, is in many ways in an enviable position compared with some others. If you have had a happy childhood and if your adoptive parents have correctly given you as much information as they can about your origins as and when you ask for it, you may well in your teens be interested to find out more about your genetic inheritance but you probably will not be unduly bothered about it. The much more worrying case is what one might call the young human shuttlecock; the child who has been batted to and fro, in and out of care, fostered here and fostered there, with periods in between in various children's homes.

Imagine the questions which may ferment, perhaps possibly even fester, in your growing mind: "I wonder why they chucked me out?" "Where on earth was I between that family and that home?" "What went wrong? I quite liked it there. I can't remember". You are almost in the position of one of these fictional characters who comes round after some physical or mental shock and cannot remember who he is. You are a young person with a half-remembered, patchy past; and the answers to all these questions are locked up in the files of perhaps several social service departments.

These will almost certainly not be computerised, and so the Data Protection Act will not help you. You are one of the many kinds of person who would be helped by the sort of draft Bill to which the noble Baroness referred. The schedule to that draft Bill mentions among the records to which access, apart from essential safeguards, would be open to all, the following: records relating to any individual who has been or is in the care of a local authority; and, in the private sector, records relating to any individual in residential accommodation for orphans or other children no longer being cared for by their parents or adoptive parents.

There are many other categories of person who would benefit from such a Bill. For example, people who have either been rejected or struck off the list of possible foster parents. Should they not have a right to know why, to see the record and, if it is wrong, to correct it? In conclusion, the sooner we get some such Bill the better. I hope that the noble Lord, Lord Belstead, will be able to tell us that the Government are contemplating some steps in this humane and, I would hope, politically uncontroversial direction.

6.28 p.m.

Lord Denning

My Lords, the law has often had to consider the confidentiality of reports. It is important in nearly all cases that the person who makes the report should be able to do it fully and frankly without fear of a libel action being brought against him. Therefore in many cases the public interest requires the report to be kept confidential. But there are cases when it should be made open. It depends so much on the individual cases.

Let me take the first matter, employer and employee. It is everyday practice for reports to be made to the manager or superior officer on the conduct and efficiency of an employee. Often they are confidential and are not disclosed. In one case we had, a man after he had been dismissed managed to get sight of a confidential report which said that he was unstable, highly sensitive and the like. He brought a libel action on account of it, and the jury awarded £15,000. We had to reverse it afterwards. But that is employer and employee.

We had other cases concerning employer and employee where a lady complained that she had not been promoted, whereas two men had been promoted and she was much better than they were. She said "I want to see my reports and the reports on those two men, because I am sure that they were not as good as mine". She was allowed to see her report but the men said that theirs were confidential and that she must not see them. We said it was to be left to the discretion of the chairman to decide. But in another case we were told (I hope it is right still) that the Civil Service and many other big employers allow the individual person to see all his own personal reports. I believe that is the practice in the Civil Service and in many of the big companies. I should have thought that was a good advance, but I do not think that I would make it a legal obligation.

Then we have social workers reporting on whether a couple are fit to adopt a child. Social workers say whether a child has been maltreated or not, reporting to the council what they find. I am afraid that some parents say "We want to see all those reports. They must be untrue. We want to sue for libel". In most Cases the reports must be full, frank and honest. It would be quite a mistake to allow the parents or other people to have complete access to them. If it comes to a court or inquiry, whoever it is will use his discretion to say whether that should be allowed or not. It may be the worst thing possible to let the proposed parents, or whoever it may be, see the reports.

What about medical reports? How often have we come across those? They often come up in actions against doctors or hospitals for negligence. Continually it is the case that they disclose where agreeable the hospital records and reports, subject to the medical report being full and frank. We had one case of suspected cancer. Was it right for the patient to see that and be worried accordingly? Quite often doctors for the sake of their patients do not tell them the whole truth. It is a question of medical knowledge and expertise as to how much the patient should be told. But when the doctor is making the report so that those advising or following on can see it, he ought to make it full and frank, not having to worry whether the patient will see it afterwards.

In a way it is a delicate matter of judgment. It would be a pity to give every patient a right to see every medical report made upon him. Doctors would not be fair and frank or give a proper report if they thought that a patient would be worried out of her senses if she knew the truth. In other words, it is a delicate balance to take in all these cases, whether one is to protect these confidences, which are vital in many cases, or whether one can have open information.

I should like to have whatever is in the best interests of those concerned. I suggest that it is not a matter in which the courts or even the law or Parliament can intervene. I should like those immediately concerned to lay down rules of guidance about what should be done and when there should be disclosures, not in the law but as a matter of guidance in all the difficult and infinitely varied problems that may arise in life.

This is a most important question. I should be against any lawmaking upon it. We should leave it to the good discretion of the professions and of others concerned.

6.36 p.m.

Lord Rea

My Lords, the Question which my noble friend has raised has been discussed at great length by professionals from a number of disciplines, including legal experts. We have heard a notable contribution from one of the most eminent of these. Also lay people representing the general public have thought deeply about these matters. As has been mentioned, there is a campaign for freedom of information which has produced a draft Bill on this subject. I have not been at the forefront of these discussions, but I feel qualified to speak in support of my noble friend's Question because I am a member of a profession that probably makes and holds records about more people than any other. Almost everybody has a general practitioner and some sort of note is usually made at each consultation. About 200 million consultations occur with GPs every year. The records of some patients contain long and detailed letters and reports to and from specialists giving in-depth assessments of a particularly personal nature. The notes of hospital patients are sometimes even more complex and often impossibly bulky. Sensitive information of a similar kind is also held in the files of social service and education departments, housing authorities and other bodies and professions which deal with people and their problems.

Because nearly everybody has a medical record held by his GP it is not surprising that in the recent MORI poll mentioned by my noble friend it was general practice records that most people wanted to see. It was 73 per cent. as she mentioned. It is traditional to regard medical files as sacrosanct. However, they are obviously of enormous interest. I have often gone out of my consulting room to fetch something and on my return found the patient guiltily flipping through the notes—like a naughty schoolchild caught smoking behind the bike shed. I think we should remove that guilt. The tradition that "doctor (or nurse) knows best" was always accepted rather reluctantly and was imposed by a profession which relied on a mystique for its authority because until 50 years ago it had very few scientific remedies to offer. Notes were more scanty—and often in Latin—and the traditional bad handwriting of doctors, which I own to, was an aid in perpetuating professional mythology. But, as diagnosis and treatment have become more rational and the importance of social factors in medicine has become increasingly recognised, more complete notes are needed, especially as medical practice becomes more of a team effort. This applies to other professions as well, especially to social work. I personally believe that the patient, as far as possible, should be a member of that team.

If that became the case, should they not have the right to know what information is held about them? I should like to be able to give an unqualified "yes" because of the apparent justice of the principle and because there are many advantages to be gained by adding the patients' scrutiny of their notes to that of the professional. There has necessarily to be some qualification of the principle, and I shall go into this a little bit later, but, first, the advantages of patients having access to their records include, for instance, from a purely practical point of view the fact that records are likely to become more factually correct even on something so simple as date of birth.

There are sometimes quite glaring inaccuracies in people's notes which may affect how other doctors and professionals approach them. One unusual example that has been brought to my attention was that of a woman who was recorded in the medical notes as being a registered drug addict, which she was not. In fact, she had been impersonated by a drug addict who had used her name while the patient had been out of the country for three years.

More open records lead to greater trust between the client or patient and the professional worker and this particularly applies in the social-work field. Another problem is that professional records today all too often contain judgmental and opinionated statements and I think that that goes beyond what is "fair and frank", in the words of the noble and learned Lord, Lord Denning. I personally have to confess that I have recorded derogatory remarks in a patient's notes on occasions when I have felt that someone is going on too long when there is limited time at my disposal. Of course, it might be possible that if a patient should read the derogatory remark a difficult area might be opened up, beneficially.

I consider that it would be very salutary for doctors to consider carefully the effect of what they write on the future doctor/patient relationship should the patient read it. Sometimes patients read things even now, and the effect is not always good. Also, such remarks when read by the next professional may well bias him or her against patients before even meeting them. That is something very much to be deplored and which more open access to records would avoid.

People sometimes need to read their records to find out important things about themselves, as the noble Lord, Lord McNair, has pointed out. It so happens that only yesterday a patient of mine, a woman of 26, came to see me and said that she was trying to complete a record of her origins and childhood so that her son, who is now a baby, could read it when he was older, as she knew very little about her early life. I was able to go back into her records and show her a report which had been made when she was two years old which was a summary of her admission to the Whittington Hospital. The report read: Diagnosis: generalised bruising and cephalhaematoma. History: child of problem family, mother known to knock the children, particularly Ann (a fictitious name), about. Father had found the child crying, bleeding from the right ear with a large bruise in the occipital region. This woman, apart from always having had a bad relationship with her mother, knew nothing about this episode. I was able to advise her on how to get further information from the Social Services Department about her early history and how to get details about her birth.

People can make important decisions more easily if they know the full truth about their medical condition—the fact that they are suffering from multiple sclerosis, for instance. I take the point of the noble and learned Lord, Lord Denning, that sometimes doctors are not certain of the diagnosis and that it is perhaps right not to divulge their degree of uncertainty if it is going to worry the patient greatly. But I shall go on to that in a moment because there is another side to it.

Medical opinion is coming round to agreeing to patient access to records, but with safeguards. The most important group backing the principle of freer access is the inter-professional working group established by the BMA and chaired by Sir Douglas Black. Their report says: Access encourages openness and can improve the quality of the record by correcting factual errors and reducing misunderstanding. They go on to discuss the safeguards which they feel are needed to prevent disclosures causing distress, harm the breaching of other people's privacy and to ensure that information is properly explained.

These safeguards have been spelt out in the draft Bill mentioned by my noble friend and seem to me to provide reasonable protection. The most difficult area—and I do not think that the proposed Bill as it stands has fully cleared the difficulty—is in deciding when disclosure might cause distress or harm to the patient or client and, I might add, alter the professional relationship in an unhelpful way. This might occur, for example, in a psychological or pschyiatric framework when the patient is still in a vulnerable, disruptive or disturbed state. Here I think there is a case for a degree of paternalism, or maternalism if you like, if the doctor or therapist is consciously using it appropriately as a therapeutic tool.

I am not sure that, if the right to access were declined and the patient used his suggested "right to seek access through an independent health professional of his choice", this situation would be adequately covered. Faced with this situation, the notes that the professional concerned makes during a consultation could be modified so that remarks could be made which mean one thing to patients and another to a fellow professional. And to some extent this is already done, although considerable difficulty could arise with regard to information that is already recorded in notes were clients to have access to their entire records. My noble friend has mentioned that the proposed Bill has thought about this and that it is only from the date of the legislation becoming law that patients will have full access, or even modified access, to their records.

I am inclined to agree that the professionals concerned should be able to prevent past records which were thought to be damaging from being divulged and that only information which was recorded from the date of the Act should be covered. It certainly would be a formidable operation to "vet", if one may use that word, all the records held relating to past events, and I think that some such rule would be necessary. I hope that the further discussion which will doubtless take place if legislation along these lines is introduced will result in a clearer solution to the kind of problems I have been mentioning.

As I have said, the medical profession is now on the brink of accepting the principle, with safeguards, and the BMA's annual representative meeting only just failed to agree to this by a vote of 183 to 169 at the Scarborough meeting this June.

The General Medical Services Council of the BMA, which represents GPs, is in favour of modified access and has accepted the principle, as has the Royal College of General Practitioners. Hospital doctors are not so much in favour, and the Royal College of Psychiatrists, while in principle in favour of free access, felt that greater safeguards would be required in the case of psychiatric problems, not least because so many people other than the patient are involved in giving information on the understanding that it is confidential. I very much hope that the Government will look favourably on any legislation based on this draft Access to Personal Files Bill which may be proposed in either House in the next parliamentary Session. Of course it would be better still if they were to introduce such a Bill themselves, but I shall be very interested indeed to hear the views of the noble Lord, Lord Belstead, when he replies tonight.

6.50 p.m.

Lord Jenkins of Putney

My Lords, I hope your Lordships will permit me to make a brief intervention. My name should have been on the list but, because of inadvertence and an error, it was missed off. Nevertheless I will endeavour not to detain your Lordships for too long. I should like to join those who have congratulated my noble friend on introducing this subject. It is an important one and she has performed a service in bringing it before us this evening, even if we are getting towards the end of our present Session.

As my noble friend has said, next year personal records are to become available where they are kept on computer; but there seems to be no reason why they should not continue to be concealed by keeping difficult or sensitive cases off the computer and in typescript, which is still the normal means of recording the bulk of official information. I therefore think that this proposed Bill is necessary for that purpose alone, if for no other, because it would be an absurd anomaly that the whole issue should depend on what particular form of record was adopted. One can easily envisage a situation where it would become the norm to have one form of computerised record of anything which could be released without any problems at all, and to have a second, non-computerised record of things which might give rise to difficulties or problems.

However, our discussion tonight has concentrated very largely upon the medical aspects of matters, and there are other things as well. In particular, so far as the Data Protection Act is concerned and also so far as the draft Bill is concerned, there are some areas of exclusion. Oddly enough, there is one particular area of exclusion—a normal area of exclusion—which I particularly think should not be excluded. It is a difficult area, certainly: it is the problem of national security. Because national security is, as it were, not to be looked at, not to be investigated and to be put on one side, there are good grounds for believing that much more evil and much more error exist in this area—for the simple reason that none of the ordinary means which exist in all other areas, medical or otherwise, for keeping an eye on the scene is permitted to occur in this area.

There is reason to believe that error in this area is not unusual. Indeed, there are those who suspect that it is the norm. The difficulty arises in distinguishing between a desire to improve the world and a determination to destroy it. At first glance this may seem a simple distinction to make, but in practice the line between seeking to change society and overthrowing the establishment is not always an easy one to draw.

So far as the secret services are concerned, socialism and subversion are virtually synonymous. Indeed, it is said that there are a number of files in the secret service on a substantial number of Labour Members in another House, and indeed even on one or two Members of your Lordships' House, I am sorry to say that the time when I was myself regarded as a danger to society is probably long past; but unless they have been shredded, I suppose the files are still available.

The secret service was a nuisance to me for a number of years with their tapping, bugging and so on—and even burglary on one occasion. They affected the course of my post-war career quite substantially, and indeed my presence here this evening is probably due to the secret service more than to anyone else. That may be regarded as a benefit or otherwise, according to how your Lordships may look at it, but the secret service brought this about by advising the BBC in the immediate post-war period that, as a security risk, I could not be employed by them. I therefore turned to trade unionism and politics, and thus ended up in your Lordship's House. So you have the secret services to thank—if that is the right word—for what has happened: otherwise I might now be a long-retired disc jockey!

Seriously, the problem is the total lack of supervision and the inability of the suspect to know that he is suspected, let alone of what he is suspected. This is the problem. If a freedom of information Bill were to come before your Lordships' House, I should take the view that every voter in this country is entitled to know whether or not he is regarded as a security risk and whether the security services are storing information about him or her. In this case it may be difficult to give every suspect access to full information. As I say, it is a difficult area and the right to see files is one which I think would have to be looked at with some considerable care. But in view of the belief—we cannot say whether it is a fact or not because we simply do not know—that a lot of the alleged information held in these files is inaccurate, something must be done to bring the security services under parliamentary supervision, as is done in many other countries.

Before I sit down—and I promised that my intervention would be short—I should like to draw to your Lordships' attention the report by Richelson and Ball on the intelligence co-operation between the United Kingdom, the United States of America, Canada, Australia and New Zealand and to give your Lordships a short quote from it. They are talking about the degree to which throughout the world there exists a framework of communication of security services which, for the most part (and particularly in this country) nobody knows anything about. More is known in the United States and in Australia and the other countries I have mentioned. In those countries examinations have taken place. The report says: In each case, also, the investigating bodies have discovered a wide range of security service activity which would generally be considered either improper or clearly illegal". I know this to be the case from my own experience. Improper and illegal activity, I suspect, while not necessarily the absolute norm, is a pretty frequent occurrence in the secret service area; and I think it is time that Parliament took a look at this problem. Parliament should decide that we really cannot any longer go on in this way whereby something is happening all the time which we know nothing whatever about because there is no effective control over it.

This Bill, which we hope will become before us, may or may not be the right way to approach this problem, but if we are going to move into a more fully informed society then we ought not to put on one side the one area of which we know least about of any. We ought to make some movement into this area and try to bring it under some more rational control, because the most peculiar and odd things take place at the present time which have no rhyme or reason in them whatsoever.

I again congratulate my noble friend on introducing the subject and I hope we are going to hear much more about it. I look forward to the Government's reply and, in particular, as to whether the noble Lord will say anything at all about the aspect of the problem which I have briefly introduced this evening.

7 p.m.

Lord Silkin of Dulwich

My Lords, my noble friend is to be thanked for bringing to the attention of this House the question of accessibility of information, which is of concern to members of the public, and in particular the question of access to personal files. It is a wide subject and both my noble friend's speech, which was extremely, thoughtful and delivered with great feeling, and the other contributions which the House has heard concentrated for the most part on the second part of the Question, with regard to access to personal files. The notable exception was my noble friend Lord Jenkins of Putney who, as I think he will agree, somewhat widened the area of discussion.

In the discussion before the present one the House was concerned with the provisions of the European Convention on Human Rights and it is perhaps of interest to observe that the right to information is not one of the fundamental rights and freedoms enshrined in the convention and that such a right could therefore not become part of our law if the convention were simply enacted in United Kingdom law. If anything, the convention rather leans the other way and tends to favour the protection of citizens' correspondence and indeed subjects even the right of freedom of speech to restrictions to prevent the disclosure of information received in confidence. However, I think it has become more and more widely recognised that the fundamental freedoms—freedom of speech, the right in a democratic society to exercise one's vote and the right to impart or receive information—must, in order to be fully effective, be informed freedoms. The citizen cannot make an informed choice without the necessary knowledge of the issues before him, and similarly he cannot effectively exercise his right to the measures to protect his reputation without having the right to become aware of what is said about him and the record about him which may be to the detriment of that reputation. Hence the two pillars upon which my noble friend's Question is founded are bound up with the preservation of our freedoms. They are necessarily ancilliary to the freedoms which the convention safeguards.

I want to say a word or two on each of those pillars, although I have recognised that the debate has been mainly about the personal files aspect of information of concern to the public. Dealing with that aspect first, I am bound to say that, like my noble friend and others who have spoken in this debate, I have never understood the logic of giving access to computer data but not to manual records which relate to precisely the same information. The ability to defend oneself against attack, and indeed even to know in what respect an attack is being made, has no reality if one cannot find out the nature of the attack. Unless files are open to inspection there can be a great temptation to make careless and unjustified comments. I am sure that that is not a temptation confined by any means to doctors, but is one which may affect very many people.

There are indeed a number of examples given in the well-argued report issued by the Campaign for Freedom of Information and the noble and learned Lord, Lord Denning, referred to the possibility of opinionated statements which may not have any apparent basis of fact but on which others may rely. I hope the House will allow me briefly to give an example of that which affected my own family—with what consequence I think it is for each person to judge for himself.

It is part of the lore of my family that in the first decade of this century, when my father was coming to the end of his schooldays, he sat for and won a mathematical scholarship to Worcester College, Oxford, but his means were such that without a grant from the local authority he would not be able to take it up. To get that grant he had to have a recommendation from his head teacher. His head teacher gave a recommendation that stated that in the head teacher's opinion "this boy would not benefit from a university education". That was a matter of opinion and not of fact. My father always used to say, in a somewhat sceptical way, that had that not been done and resulted in his not going to the university, he would have probably lived his life doing the very important task of a mathematics teacher rather than as Minister of Town and Country Planning. Being an opinion, that could not be demonstrated to be false—at least not until later on.

It is quite clear that the right to examine records cannot be wholly unlimited and the report of the campaign very fairly recognises this. Indeed it considers with very great care the exemptions that would be necessary. It would be wrong in this debate for me to go into those in detail, because we may at some time in the future have either a Government Bill or a Private Member's Bill and that will be the time to work out the precise boundary lines. I think it is sufficient to say that the balancing of public interest has always been a problem which has delayed legislation in fields analagous to this one, particularly on the subject of freedom of information. The question is always where the line is to be drawn. While I agree with the noble and learned Lord, Lord Denning, that it does raise extremely difficult questions, I do not myself think that one should therefore fight shy of legislating at all. I think that this is the difficulty which Parliament has faced in other fields and which it ought to be prepared to face in this one.

Perhaps I may say a word or two about the more general question of freedom of information. The Labour Government issued a White Paper (Cmnd. 2285) in 1978 which traced the long history of this problem and made the pertinent comment that: Recent Governments have, in fact, done more and more to provide information to the public". One ought not to assume that things have stood still through the report of the Franks Committee and the events of the years that followed it. Whether in the form of White Papers, consultation papers, briefings, evidence to Select Committees or otherwise, far more information has been made available to Parliament and to the public than was made available 30 or more years ago. I hope we would all agree that Parliament, above all, is entitled to information, and that we need to be very wary of any Government seeking to circumscribe the information that may be revealed to Parliament through Select Committees.

However, the Command Paper went further. It emphasised the very close connection between that which, whether for security or other essential reasons—my noble friend Lord Jenkins will forgive me for saying "essential"—must be protected by the criminal law, and the residue of information which cannot be released to the public under the most liberal system that one could enact. Indeed one has to recall that even in the United States and Sweden, which are usually held up to us as being countries where liberality in this field goes furthest, there are exemptions for security and other reasons. The White Paper went on to advocate a system of the kind indicated, though possibly, but not necessarily, that was for future discussion to be embodied in statute law.

To give it credit, the Conservative Government that followed tried, though unsuccessfully, to embody the form of Section 2 of the Official Secrets Act in statute form. But in doing so they demonstrated that today reform of Section 2 must go hand in hand with positive legal rights for the citizen to obtain information. Since then the Government seem, naturally, to have been a little shy of this subject.

My noble friend has done a service by bringing it back to your memories, particularly perhaps with a general election in the offing, even if the Question was concentrated more particularly on the matter of personal files. When we come to the general election, I for one hope that all the parties will subscribe to the proposals made by my noble friend, and will indeed go further and subscribe to the idea of a general increase of information available to the public, so that when the election manifestos are prepared this subject will not be a party political one. I have no doubt that my own party will present a manifesto in those terms.

7.12 p.m.

The Minister of State, Ministry of Agriculture, Fisheries and Food (Lord Belstead)

My Lords, I join in expressing my gratitude to the noble Baroness, Lady Ewart-Biggs, for raising for debate this evening a subject which is of great interest and covers a vast range, from access to government information to access personal data, whether held manually or electronically, whether held by government or some other official body, or by private firms or individuals.

The noble and learned Lord, Lord Silkin, in making the final speech from the Opposition Front Bench, referred briefly to the aspect of access to non-personal government information and I think it would he your Lordships' wish this evening that I follow the noble and learned Lord in only a sentence or two, simply saying that the Government's policy on access to general information held within government is to make as much information as possible available, while preserving the confidentiality essential to the effective working of government. As well as stating this publicly, my right honourable friend the Prime Minister has made sure that the Government's policy is known to Ministers in charge of departments and through them to their officials.

But the Government are opposed to a statutory right of access to official information and the two reasons which we have given again and again, and which I shall not embellish this evening, are, first, that we feel, as the noble and learned Lord, Lord Silkin, said, that there is now an enormous amount of official information being made available, the best example being the work of the Select Committees which, after all, the previous Conservative Government in 1979 helped to set up, and which make available interviews, memoranda and other documentation from the Executive. We see good examples of that from the Select Committee reports in your Lordships' House and from the other place. The other reason that the present Government have always given is that we think that a statutory right of access to official information would be inappropriate because it would cut across our system of parliamentary democracy. I do not think all noble Lords will agree with me on those two legs of the reasons I have given for that, but, as this evening's debate has centred upon personal files, perhaps your Lordships would prefer that, having said what I have just said, I should try to answer the debate by sticking to that subject.

Let me say a few brief words about the Data Protection Act 1984, which comes into force, as some of your Lordships have said, in November next year. As your Lordships will well know, the Act works through a system of registration by data users with the Data Protection Registrar. Individuals are already entitled to compensation if they suffer damage as a result of inaccurate data or because data have not been properly protected. There is also a right to the correction or removal of inaccurate data or data held in breach of any of the principles of data protection set out in the Act. As I think the noble Baroness said, the rights under the Act are of considerable significance and importance.

The Act again, as your Lordships will remember, extends to data held by government departments, local authorities and other official organisations. There are inevitably exemptions from the rights of access; for example, data held for law enforcement and revenue purposes, if access to them would be likely to prejudice law enforcement or revenue collecting. But overall I am confident that the rights being introduced under the 1984 Act are a significant addition to the freedom of information in this country.

I think it right that I should just say this evening in this short debate that, whereas some critics have said that the Act does not go far enough in protecting the interests of data subjects, most of the complaints which have been received by the Home Office and the registrar have come from people and organisations that feel that the registration requirements are an unnecessary burden on the business community. It is argued that the form-filling required and the fee demanded for registration run counter to the Government's general intention of lifting burdens of this kind from industry in general and in particular from small businesses.

As I said and as some of your Lordships have said this evening, a considerable number of exemptions from the requirements of the Act have been devised, but what has proved impossible is to provide a general exemption for the small-scale user. In speaking about small businesses in this connection, I simply say that the Government do not think that in practice the registration requirement will prove to be such an onerous burden. After all, the Act sets out very real rights that data subjects will acquire. They will have great advantages and we look forward therefore to the coming into force of the Act.

The same system of registration would plainly, I suggest, not be appropriate for manually held records. It requires only a moment's thought to imagine what a bureaucratic nightmare would be created if everyone had to identify and register the information they held on normal files. It has to be borne in mind that the impetus behind the European convention and our legislation springs from people's concern about the use and potential for abuse of computers. It is their ability to hold vast amounts of data, to process it rapidly in so many different ways and then to transmit it elsewhere with great ease and rapidity which gives rise to concern. These considerations do not apply to manually held data.

There are nevertheless, and understandably, pressures to allow rights of access to manually held records. The noble Baroness drew attention to the Bill of the Campaign for Freedom of Information with that object in view. I find it interesting that the Bill does not, as I understand it, propose a registration system but simply provides a right to receive copies of certain defined types of records. The Government's view is that even a right of this kind would impose considerable burdens on the organisations holding the records, especially in government departments themselves. It could also prove very costly. I have to say to your Lordships that some departments have said that the scheme would require yet more staff and the Government would not be prepared to agree to extra public expenditure for this purpose.

The noble Lord, Lord Jenkins of Putney, touched on the very difficult subject of security services. I must say that I cannot share his views that those individuals who are regarded as a security risk to their nation should be given warning of that fact. I suggest to the noble Lord that on reflection he recognises the serious risks that that would entail.

In trying to make progress so far as manually held files are concerned we are very much bearing in mind the warning which I understood the noble and learned Lord, Lord Denning, gave, that some advice of this nature needs to remain confidential while other advice needs to be made available. It is in pursuit of that that as long ago as 1983 the Department of Health and Social Security issued a circular setting out the general principles for access by clients to local authority social services manual records; and authorities generally are already on the way to implementing arrangements for access.

The Government have already announced their intention to make regulations during the first half of next year to allow access to the educational records of pupils. Consultations have recently taken place on the practical means to implement access to housing records. Following the report of the Royal Commission on Environmental Pollution, steps are now beng taken to improve public access to information obtained by pollution control authorities. Indeed, the importance the Government attach to this last field is shown by the resolution tabled only last week for consideration by the European Communities' Environment Council urging all member states to adopt the same approach.

The noble Baroness and the noble Lord, Lord Rea, spoke about medical records. Health records are, of course, maintained by health professionals to help them in the care and treatment of their patients. It is a matter for professional judgment whether in particular cases it is in the patients' best interests to have access to their own records. Although there is no automatic right of access to medical records it is recognised as good practice for health professionals to be as open and informative to their patients as possible. The Government support that, but do not think that legislation or blanket coercion are the best routes.

The process of greater openness is accelerating in, for example, the field of maternity care; and on computerised records the DHSS has circulated a consultation document to a wide range of professional bodies and consumer groups. The overwhelming majority favour access to computerised health records in one form or another, as do DHSS Ministers. They have therefore decided that patients should have access to their own personal health records held on computer, subject to the condition that doctors may withhold information which is likely to cause harm to the wellbeing of the patient or of others. The DHSS is discussing with the medical profession how best the twin concerns of access and protection for those in need can be accommodated.

In this general context, let me add to what the noble Lord, Lord McNair, said about the careful arrangements that are provided for counselling young adopted people before they are shown their original birth certificates. I think that underlines very clearly how careful one has to be to balance the interests of the individual in being given information about himself or herself and the emotional harm that can be caused if that information is disclosed in an unprepared or thoughtless way.

Finally, let me thank the noble Baroness and noble Lords who have spoken in this debate. There is, rightly, considerable concern that information should not be withheld unnecessarily from the public. As a result of my reply I trust that your Lordships will feel that, although I have by no means completely gone along with the noble Baroness, the Government nevertheless share the concern that has been expressed and are taking steps to ensure that more information is made available both by government and others in ways that are appropriate to the information in question.