Called into question

'Evidence-based regulation' sounds like a good thing. It implies that regulators will only take action where there is evidence to support any decision that they might make. I'm all in favour of that approach. This week, however, the Legal Services Board, our oversight regulator, has claimed to have produced evidence that firms are failing clients in complaints handling. But to suggest that the survey conducted by YouGov is 'evidence' is something of a stretch.

YouGov has done a good job. Its survey is clear and has been conducted on well-established and transparent principles. The respondents were selected from a pool of those whohad stated in a previous survey that they had been dissatisfied with the service provided by a legal services provider in the last three years. In all, 1,275 people responded to the survey. Although my inner scientist would like more information about the predecessor survey from which the pool of respondents was selected and the geographic spread of the survey, the problem lies not in the survey methodology but in the underlying assumptions.

For example, the LSB highlights that 57 per cent of respondents believe they were never told that firms had a complaints procedure (I assume that answers were given from memory so this may not be reliable in itself).Yet the requirement to inform clients that the firm had a complaints procedure only came into force in October 2010.

Until that date, firms had to inform clients of their right to complain and the identity of the complaints partner. Complaints procedures only had to be provided on request or where the firm believed that a client was attempting to complain.

It is difficult to say with any certainty when one might have expected these people to have complained, as two thirds of them were simply dissatisfied with the service and never made a complaint at all. It is suggestive that two thirds of respondents didn't complain; the vast majority (78 per cent) stated either that they didn't think it was worth it or that they were just too fed up to bother. At this point, one begins to wonder just how dissatisfied the respondents were with the service they received.

The figures suggest that around 83 per cent of respondents instructed their lawyer (not necessarily a solicitor) before October 2010 and that, of those who complained (425), around 70 per cent (around 23 per cent overall) did so before October 2010. By my calculations, very few of the respondents ever got to the stage where they needed to be told about the complaints procedure, so, if the 57 per cent figure is accurate (which is a fairly big 'if'), it is hardly indicative of systemic breaches of the rules.

The survey acknowledges the introduction of the Legal Ombudsman in October 2010 and provides appropriate disclaimers as to the reliability of the information given the change in the regime. No acknowledgement at all is given to the changes occasioned earlier in 2009 and 2010 as a result of the abolition of remuneration certificates, nor is any acknowledgement given to the rules of other branches of the legal profession. It seems that all legal services providers have been lumped under the same umbrella for the purposes of the survey.

It is also interesting that YouGov's notes state that any sample size of less than 50 should not be expressed as a percentage because the sample is too small to be relied on for extrapolation. The survey struggles to provide any reliable results on progression to second-tier complaints handling because only 56 people took their complaint that far. If people start dazzling you with suggestions that 69 per cent of complainants to the Legal Ombudsman had their complaint fully or partially upheld, remember that equates to 29 people.

I do not pretend that solicitors are necessarily good at complaints handling. I am sure there are solicitors who immediately take a defensive position and respond to complaints in an overly aggressive manner. There are those who bury their heads in the sand and ignore complaints. We are not going to address this issue by commissioning surveys relying on out-of-date information and an incomplete or inaccurate understanding of the rules.

Evidence-based regulation should be a good thing. Unfortunately, it's only as good as the evidence on which it's based.