Here are four research factors to keep in mind when scrutinising AMPS figures.
The first thing to look at when using any data is the stability of the sample. For the filters you've created, were enough respondents interviewed to ensure the data is stable from survey to survey?
The various software packages which can be used for the SAARF products highlight data based on unstable sample sizes: fewer than 100 respondents (unstable) and fewer than 50 (highly unstable). Top-line data is always robust, but if you over-target, and create filters which are too niched, you will often find the bases are unstable, meaning the data may fluctuate from survey to survey.
For instance, if you create a filter for English-speaking people, aged 25 - 34, living in the Northern Cape, who drink tea, the results would be based on five respondents. It would be a mistake to drill down into the demographics of this result. While this year, those five might be female, last year they may have all been male. There are too few respondents to reflect a constant profile for this set of filters. Be wary of making decisions on data from over-segmented samples.
The second thing to remember when tracking trends is margin of error.
In any research, there is a margin of error: the level of correspondence between the sample and the population it represents. The only way to eliminate it would be to interview the entire population. Since this is impossible, the best we can do is calculate the margin of error, which is the range in which the data falls with 95% confidence.
It's essential to consider this margin when comparing figures, or you could be hiring and firing and changing campaigns based on changes which aren't statistically significant.
The easiest way to calculate margin of error is to download the Quick Equation Excel file on the SAARF website (www.saarf.co.za, under “Calculate Significant Difference”). Put in the sample size and the penetration, and the margin of error will be calculated for you. The same link offers a detailed users guide.
If the margins of error for the two figures being compared overlap, the change is not significant. If there is no overlap, the change is significant.
Here is an example of the incidence of sugar usage by two different sets of consumers, over a seven day period:
| Filter | AMPS 2005/2006 | AMPS 2008BA |
|---|---|---|
| Sugar Ever | Sugar Ever | |
| KZN black women LSM 2-6 | 8.0% | 7.1% |
| Limpopo black women LSM 2-6 | 4.7% | 4.3% |
On paper, both figures have declined. But were they statistically significant declines? The margin of error will answer that.
The range in which the data falls with 95% confidence for each period is as follows:
| Filter | AMPS 2005/2006 | AMPS 2008BA | Significance |
|---|---|---|---|
| KZN black women LSM 2-6 | 7.55%-8.45% | 6.68% to 7.52% | No overlap, so a significant decrease |
| Limpopo black women LSM 2-6 | 4.35%-5.05% | 3.97% to 4.63% | Overlap, so not a significant change |
A further consideration that should be taken into account is the changing size of the South African population. When you calculate the percentage change from one year to another, it is highly recommended that you use the percentage figure and not the thousands. Population size changes, making the figures in thousands not directly comparable with one another.
The example below shows how misleading this can be:
| Filter | % change for Sugar Ever from 2005/2006 to 2008BA (using %) | % change for Sugar Ever from 2005/2006 to 2008BA (using ‘000) |
|---|---|---|
| KZN black women LSM 2-6 | -11.3 | 0.3 |
If you use the thousands figure instead of the percentage figure, you will think there has been a 0.3% growth instead of an 11.3% decline.
In the above example, the difference in the change could be due to population size changes. It could also be due to differences in the LSM profiles, which evolve from year to year.
South African society is constantly developing, with people moving up through the LSM ranks. Every year, LSM 1 and 2 gets smaller, with LSM 5 to 7 swelling. When looking at data trends year on year, bear in mind how the changes in the LSM groups will affect the figures.
By keeping these four realities of research in mind, you will get the right picture when working with data.
