A Third Supermarket: Possible Flaws in Methodology of the Survey

http://www.thisisjersey.com/2008/12/08/huge-vote-for-third-store/

Huge vote for third store By Dolores Cowburn

ISLANDERS have voted overwhelmingly for a third supermarket operator to come to Jersey, according to a consumer survey released today. Commissioned by Economic Development and created by the Statistics Unit, the public survey found that 84% of people wanted more choice and seven out of ten Islanders were not happy with the current range of supermarkets in Jersey. More than 1,000 people completed the survey titled 'A Third Supermarket in Jersey?', with three-quarters of those who want a third operator favouring a British chain. It is one of the biggest responses to a States survey. Only a third of people expressed concern that smaller shops may close as a result - which was one of the fears expressed by the Chamber of Commerce if another large supermarket operator was brought over.

http://www.gov.je/ChiefMinister/Statistics/News/SupermarketSurveyPage.htm

Given the size of the dataset, and using weighting to ensure all subgroups of the population are suitably represented in the analysis, we can be confident that the inferences drawn in this report robustly represent the views of Island residents.  Over two thousand households were sampled at random. These randomly selected households received a survey form through the post and were asked that the person in the household who had the next birthday (and who was aged 16 years or over), fill it in and post it back to the Statistics Unit. This method of sampling ensured that the survey randomly sampled the adult population of Jersey. The survey achieved an extremely high response rate, with 60% of sampled households filling in and returning the survey form. Such a high response rate, together with the method of sampling, ensures the sample results are both accurate and representative of the full adult Island population.

To be fair to the Statistics Unit, they then compared various Census statistics, such as age banding with the sampling done, to see how closely they matched, and found a degree - but insignificant - of younger people unrepresented in the survey. So this was a good sample.

But one with all surveys, the problems usually lies with what is not checked. One very immediate and apparent flaw - and I checked with the main document - stands out in the report. It is this - the questionnaire sent out was in English, and no mention is made of any checks to see if the minority but not insignificant Portuguese population (often the poorer members of Society) were adequately represented. Any weighting is missing here. Quite a number of this population have very poor English language skills, and answering a complex form in English, would probably be tempted to ignore it. They could have formed part of the missing 40%, the "dark figure" in the sample, and this might have produced significant differences in response to questions.

The other matter is to do with the presentation in the JEP, with its catchy "Islanders have voted..." headline. This makes the argument that if Islanders want this, they should get it, and there will be no associated problems which should be considered. Of course, it is easy to see the flaw in this - just conduct a random survey asking the question "Do you think taxes should be reduced?".

Part of this is the problem with this kind of survey itself; it only looks for immediate short-term responses, and does not see how people are actually thinking. Obviously, memory is short, because the last time a Third Supermarket was in Jersey - the original Safeway - prices did not drop significantly, as Safeway factored in not just freight costs and rental overheads, but also what the market could take. Prices for many goods in Safeway were more or less the same as in any of the other Supermarkets. Why would a Third Supermarket make a difference now? That would be a good question to ask Islanders in favour of another supermarket.

The construction of the questions also does not focus on sustainable alternatives, such as an expansion of the Farm Shop network, which has been steadily growing in the last few years, and which could be seriously damaged by another Supermarket, and which can provide produce at reasonable prices. Do you ever use a farm shop? Have you ever considered it? Do you think a third Supermarket might force farm shops out of business? Alternatives that were not really well addressed in this survey. It is well known that how questions are asked can get people to think, and deliver different outcomes, and a little section looking at this might have also been helpful.

Links:
http://www.gov.je/NR/rdonlyres/2440BD6D-B2BD-49F6-9D8F-945D14246DBF/0/SupermarketSurveyFinalreport.pdf