http://www.thisisjersey.com/2008/09/27/what-you-think-of-the-states/
The JEP have gone to bona-fide pollsters, and the result is a survey which is "what you think of the States?"
There are two good things about this survey. It is:
a) random
Not self-selecting like the usual JEP or online poll. The problem with polls which ask people to phone in is that they are not representative of the Island as a whole, only of those people who want to reply, and may even count people twice or more. Remember the JEP phone poll on yes / no to tall buildings on the Waterfront where an automated phone system managed to clock up thousands of "yes" votes! Blog polls are the same, as they depend on checking IP addresses. Most businesses have fixed IP addresses, but the average home user is allocated an IP address (via their provider) from a pool when they connect to the internet, and can cheerfully vote each time they go on-line. I once added 30 extra votes to an on-line poll to test this.
A random poll has the advantage over this in that it takes a sample of people regardless of whether they would vote on an online poll or not. If you think "I have not been asked, and I didn't know where to find the poll", that is because the pollsters are doing their job correctly.
b) stratified by age
"Stratified" is a technical term for the JEP's own description as "posed to a sample of Islanders which was weighted according to the age profile of the electorate". That means that the pollsters get information on people's age and adjust the results in a number of possible ways to match the electorate. More accurately this poll uses a "stratified cluster" because it needs to group ages in bands.
General Comments
One way - the simplest - of doing this kind of stratified random poll is to ask the age, then discard those once you have too many for an age group. This is called "proportionate sampling" where the strata sample sizes are made proportional to the strata population sizes. For example if the first strata is made up of males, then as there are around 50% of males in the UK population, the male strata will need to represent around 50% of the total sample.
Another - and probably that used -is to get the results but adjust them in each group so that they match the age profile. This is the cheaper option. This is called a "disproportionate method", where the strata are not sampled according to the population sizes, but higher proportions are selected from some groups and not others. The results are then weighted to bring them to the proportions required.
As a simple example, suppose in an office complex, there are 1000 staff, 40% male, 60 % female.
We could poll 100 people in our sample, making sure there are 40 males, 60 females.
Or we could poll 100 people, and if we get (say) 45 males, 55 females, then we adjust their votes accordingly, by multiplying each part of the results by 40/45 and 60/55 to give the sample the same weighting as the original population.
Problems with the JEP survey
This works quite well, but several deficiencies are apparent.
Firstly, this is not the only way in which sampling can take place. While there may be little or no difference between male and female votes, by taking age alone, this excludes that. More importantly, it excludes different economic strata within Jersey, which is at least as likely to be representative of satisfaction with the States as age groups. I would say this is a pretty major flaw.
Secondly, this method can lead to overcompensation if there is large disproportion between the profile of the sample and the profile of the population. In our example, if they asked 10 males and 90 females, the male vote would be weighted at 40/10, the female by 90/60. This means that our 10 males are representing all the 400 males in the organisation, and even allowing for weighting it is clear this means their opinions can be disproportionate. The large the sample, the less likelihood of error - but that leads to the next problem..
Thirdly, we are not told how large the sample size is. Doubling sample size reduces sampling error by half. Along with this there is a lack of figures for sampling error, which can be expressed as a percentage range of how accurate responses will be in representing the population as a whole.
Fourthly, we are not told how the survey was conducted. Sampling is notoriously difficult to do properly because of what is termed "coverage error" which occurs because a sampling method excludes members of the population. For instance, a phone survey will not pick up on ex-directory people. The time it is taken may exclude workers, or shift-workers. Stopping people in the street only gets the people who live in that street. Mail surveys may get non-responses.
All told, the JEP survey leaves a lot out!
The JEP have gone to bona-fide pollsters, and the result is a survey which is "what you think of the States?"
There are two good things about this survey. It is:
a) random
Not self-selecting like the usual JEP or online poll. The problem with polls which ask people to phone in is that they are not representative of the Island as a whole, only of those people who want to reply, and may even count people twice or more. Remember the JEP phone poll on yes / no to tall buildings on the Waterfront where an automated phone system managed to clock up thousands of "yes" votes! Blog polls are the same, as they depend on checking IP addresses. Most businesses have fixed IP addresses, but the average home user is allocated an IP address (via their provider) from a pool when they connect to the internet, and can cheerfully vote each time they go on-line. I once added 30 extra votes to an on-line poll to test this.
A random poll has the advantage over this in that it takes a sample of people regardless of whether they would vote on an online poll or not. If you think "I have not been asked, and I didn't know where to find the poll", that is because the pollsters are doing their job correctly.
b) stratified by age
"Stratified" is a technical term for the JEP's own description as "posed to a sample of Islanders which was weighted according to the age profile of the electorate". That means that the pollsters get information on people's age and adjust the results in a number of possible ways to match the electorate. More accurately this poll uses a "stratified cluster" because it needs to group ages in bands.
General Comments
One way - the simplest - of doing this kind of stratified random poll is to ask the age, then discard those once you have too many for an age group. This is called "proportionate sampling" where the strata sample sizes are made proportional to the strata population sizes. For example if the first strata is made up of males, then as there are around 50% of males in the UK population, the male strata will need to represent around 50% of the total sample.
Another - and probably that used -is to get the results but adjust them in each group so that they match the age profile. This is the cheaper option. This is called a "disproportionate method", where the strata are not sampled according to the population sizes, but higher proportions are selected from some groups and not others. The results are then weighted to bring them to the proportions required.
As a simple example, suppose in an office complex, there are 1000 staff, 40% male, 60 % female.
We could poll 100 people in our sample, making sure there are 40 males, 60 females.
Or we could poll 100 people, and if we get (say) 45 males, 55 females, then we adjust their votes accordingly, by multiplying each part of the results by 40/45 and 60/55 to give the sample the same weighting as the original population.
Problems with the JEP survey
This works quite well, but several deficiencies are apparent.
Firstly, this is not the only way in which sampling can take place. While there may be little or no difference between male and female votes, by taking age alone, this excludes that. More importantly, it excludes different economic strata within Jersey, which is at least as likely to be representative of satisfaction with the States as age groups. I would say this is a pretty major flaw.
Secondly, this method can lead to overcompensation if there is large disproportion between the profile of the sample and the profile of the population. In our example, if they asked 10 males and 90 females, the male vote would be weighted at 40/10, the female by 90/60. This means that our 10 males are representing all the 400 males in the organisation, and even allowing for weighting it is clear this means their opinions can be disproportionate. The large the sample, the less likelihood of error - but that leads to the next problem..
Thirdly, we are not told how large the sample size is. Doubling sample size reduces sampling error by half. Along with this there is a lack of figures for sampling error, which can be expressed as a percentage range of how accurate responses will be in representing the population as a whole.
Fourthly, we are not told how the survey was conducted. Sampling is notoriously difficult to do properly because of what is termed "coverage error" which occurs because a sampling method excludes members of the population. For instance, a phone survey will not pick up on ex-directory people. The time it is taken may exclude workers, or shift-workers. Stopping people in the street only gets the people who live in that street. Mail surveys may get non-responses.
All told, the JEP survey leaves a lot out!
Posts
No comments:
Post a Comment