Less box ticking, more research – the survey and statistics crisis

Questionable survey marketing and lazy journalism threaten the media, public debate and the legitimate art and antiques trade


Survey image


COMMENT: Type in the word ‘survey’ under the Google News search and see what comes up.

I have just done this and found that there are 79.9m results. A significant number of these refer to media reports of new surveys conducted into all aspects of life and business.

Top of page one among the Google results was a Guardian news story on an Amnesty International survey concerning attitudes to the criminal status of abortion in Ireland. It tells us that the findings come from 1000 phone interviews made during May.

So here we have a survey by a named organisation with the reporter giving us at least some idea of its methodology. Not bad, although as Amnesty International is not an independent body but a campaigning organisation, the question arises as to how independent the survey actually was. If Amnesty brought in a professional polling company to conduct the survey on its behalf, the report does not say.

Another story tells of a survey conducted by Car Finance 2 Go, which assesses what annoys motorists the most about each other. It made great copy for The Bath Chronicle and The Gloucester Citizen, but the report gives no details of how independent the poll was nor how it was carried out beyond the Citizen saying over 1000 people took part.


Product placement

Global software security group Kaspersky Lab conducted a survey among 1000 people into the relationship between smartphone usage and amnesia.

How did this amnesia manifest itself? According to The Express Tribune, stablemate of The International New York Times, “Many adults could not remember important phone numbers of family members and friends. The survey also showed that people are not doing much to protect their information online.”

What’s more, the survey additionally reveals “Less than a third of the people surveyed put security precautions on their devices”.

By happy coincidence pollsters Kaspersky Lab are on hand with the solution. As the home page of their website tells us, they are “the world’s largest privately held cybersecurity company offering endpoint protection solutions to consumers and businesses around the globe”.

Work your way down the Google list and in many cases it’s the same story: product placement posing behind the fig leaf of a ‘survey’ to burnish the findings with some sort of scientific credentials that make it newsworthy.

The increasing frequency of this marketing technique is no accident; it is a symptom of the erosion of proper investigative journalism in the face of plummeting revenues in the traditional publishing sphere, largely brought about by the rise of the internet and free information online.

The good news is that there are still decent journalists out there who will not fall for this junk. The bad news is that there are others who will. The worst news of all is that many of those who do see questionable surveys for what they are still file stories on them because of deadline pressures – they need to fill space and an ever-shrinking set of resources leaves them with little alternative.

A lot of the time none of this matters. And if you run a marketing department, who can blame you for doing a bit of lateral thinking and turning what would otherwise be an advert into rather more powerful editorial?

The problem is that sometimes it really does matter.


TV series promotion

In September 2013, as the then Editor of Antiques Trade Gazette, I conducted an investigation into another so-called survey splashed across a number of UK national newspapers and the Mail Online, now the world’s most visited website.

The UK Fakes and Forgeries Report, as it was referred to in these articles, claimed that 40% of all antiques on the UK market were fakes or forgeries.

On closer view, it appeared that the report’s ‘40% of antiques are fake’ figure had been extrapolated from the finding that 43% of people who buy antiques don’t get them authenticated and that 68% of people who buy antiques are ‘worried’ that they may be fake.

So no evidence at all to support the 40% fakes claim in the survey.

The report coincided with the launch of UKTV Yesterday Channel’s new series of Treasure Detectives.

Needless to say, when I contacted the programme makers they were rather coy about who had actually conducted the survey. Pressed, they admitted that it was actually called The Yesterday UK Fakes and Forgeries Report. So, did they produce it themselves?

When I asked to see a copy of the survey they emailed me the press release that all the other news outlets had swallowed wholesale and regurgitated in screaming headlines.

Even then, the ‘findings’ in the release did not support its own headline – The nation’s love of antiques has led to rise in the number of fakes and forgeries – in any way.

On pressing the TV company again for a copy of the original survey, I was told: “I’m afraid we don’t release the survey data”, before they went on to disclose that the survey was completed by 2000 adults, using a reputable survey company (never named) and then the clincher, “The rest of the report was comment and expertise of Curtis Dowling”, the programme’s presenter, who gained substantial publicity for himself and his new show from the ensuing media storm.

As my later comment piece argued, “Count on it, in a year’s time when the issue of fakes and forgeries is raised again, that statistic will have morphed into hard, authoritative, indisputable fact for the news media – and possibly even Whitehall – to regurgitate, reinforce and beat our industry over the head with once more.”

And I explained that my comment piece would also be posted online in the hope that it would go some way towards diluting that effect of the TV report.


The $3 billion question over the antiquities trade

As my comment piece also mentioned, this method of validating unsubstantiated statistics has form, not least in providing ammunition to attack the legitimate trade in antiquities and at Government level in devising legislation to tackle stolen art and antiques.

This dubious practice persists today and news outlets continue to fall for it, feeding the propaganda in the process. In fact, it has been compounded by the emergence in the past couple of years by news aggregating websites that repurpose other outlets’ content to drive traffic to their own sites.

Lazy journalism is often as much to blame as the deliberate dissemination of questionable material dressed up as scientific findings.

In recent weeks, for instance, I have noticed a number of mentions of the figure of $3 billion as the estimated value of the global antiquities market.

As far as I can see, not one report among those published gives a source for this figure. The Toronto Star publishes it in the sub heading to its report Islamic State cashes in by peddling art loot on eBay, Facebook, and lists it again in the body of the report without attribution. Many of the other articles appear to have been inspired by, or even acknowledge, Bloomberg’s report titled Islamic State is selling looted art online for needed cash.

Bloomberg also quotes the $3 billion figure but again gives no source for it. Artnet news, which aggregates other news sources, put the figure in its headline while quoting Bloomberg.

In fact Bloomberg seems to have sourced its article from The Economist’s June 13 online report Save our stones, which makes no mention of the value of the global art market at all, but does mention the figure of $3 billion. However, it refers to this figure as being the estimated value of Egyptian antiquities lost since the fall of President Hosni Mubarak in 2011, according to Deborah Lehr of American association the Antiquities Coalition.

Have later reports, starting with Bloomberg, confused these figures? I don’t know, but I do know that no one I have consulted has any idea of where the $3 billion figure came from in referring to the value of the global antiquities trade. Antiquities dealers I have spoken to believe the trade is worth far less.

Just to confuse matters further, another online article titled Plundering the past, by David Johnson, states that “A recently released report by the McDonald Institute for Archaeological Research in the University of Cambridge, England, states that up to $3 billion in art and artifacts are stolen each year around the world.”

The Institute’s report is titled Stealing History: The Illicit Trade in Cultural Material, says Johnson.

His article is undated, but ranks at number 3 of the articles on the first page for a Google search under the term ‘$3 billion antiquities’.

A search through the McDonald Institute’s online archive under the report’s title as given by Johnson reveals that it was published in June 2000 and is now out print. Further research reveals one of the authors to be Peter Watson, one of the most outspoken campaigners against the antiquities trade.


Background to illicit trade claims

Eventually, I managed to secure a pdf of the report and in paragraph 1.9 it gives a value for the illicit trade in antiquities as $2 billion, attributing the figure as an estimate by Geraldine Norman and qualifying the claim further by adding: “other estimates have ranged down to $150 million. As already pointed out, because the trade is clandestine, reliable data is hard to find.”

So recently turns out to be 2000 and when checking the report’s own reference to the Geraldine Norman estimate, that came from an article she wrote for The Independent titled Great sale of the century, dated 24 November 1990.

Just to recap then, Johnson has an article ranked 3 on the home page of the relevant section of Google quoting the $3 billion figure as applying to the illicit trade in antiquities and giving the institute’s report as the authoritative source. The institute’s report turns out to be 15 years old and quotes the different figure of $2 billion, which it attributes to a newspaper report by then already ten years old, noting that it was one person’s estimate. It also admits that other estimates put the illicit trade at less than a tenth of that figure and effectively further admits that none of this can be relied on because “reliable data is hard to find”.

We are now left with two claims: that the global trade in antiquities is worth $3 billion and that the global trade in illicit antiquities is worth $3 billion (or $2 billion if, as it seems, Johnson has misquoted the McDonald Institute report.

So what? Why does this matter?

Because, as with the bogus 40% of all antiques are fakes or forgeries claim, in a very short time indeed this figure has become established ‘fact’ and is being used widely in arguments against the antiquities trade. How much time will it take before academics who have long been trying to shut the trade down start to say that if the market has such a value, then the risk of associated crime must be very high and the trade should be bashed once more?

Believe me, they will jump on this one as soon as they can.

Whether it was Disraeli or Twain who coined the phrase lies, damned lies and statistics, my belief is that there has never been more need of reliable and truly independent statistics in the arena of public debate.

Those with an axe to grind can do serious and unjustified damage to legitimate interests by manipulating marketing techniques and taking advantage of poorly resourced media outlets that feed voracious 24/7 newswires.

For their part, reporters and feature writers need to be far more circumspect about the information put before them, especially in these embattled times for journalism, or they risk accelerating the rate at which the media becomes mistrusted and irrelevant.