Perception Panel
I’ve had a lot of comments, for and against, about our – a new format we use around Prime Minister’s Questions on The Daily Politics every Wednesday. Jane wrote to us and said, “I no longer shout at my TV as I can now tell the politicians exactly what I think".
But then Manjit said (in response to my last blog entry)...
- “What I would love for Jamie Donald to do is to come onto this blog and justify why the Ö÷²¥´óÐã continue to spend money on the Perception Panel and how much it exactly costs?â€
The Perception Panel is the world’s largest focus group. It can tell you straight away how people in Britain may be feeling about an issue or a politician. We run it every Wednesday, and to take part you must tune in just before Prime Minister’s Questions at noon, and ring the freephone number on the screen. You are asked some questions and then by pressing buttons on your phone you can register your positive and negative feelings about what you’re seeing and hearing on your TV. All those touches up and down the country are transformed – by a clever computer - into a continuous wave of approval and disapproval. After PMQs we show some of the highlights, with the moving graphic of the viewers’ reaction on top. You can see the results of last week on our website.
It costs about £1,000 each time we play. At the moment we pick up the full cost of the calls, and we’ve set the limit on the number of people who can get through at 600 (and we get literally thousands trying to ring in, so it’s a case of first come first served). It’s not perfect. For example it’s sometimes difficult to distinguish between man and ball, sometimes the graphs are a little unclear (we’re working on it) and there’s a hint of the blunt Roman 'thumbs up thumbs down' about it.
But it is robust. That’s because we weight it. If you are a young Lib Dem voter in Liverpool (of whom we have not that many on the Daily Politics), our computer ensures your touch may be worth a little more than an older Tory man from the Ö÷²¥´óÐã Counties, depending on the numbers of types of people who get through. It’s what the pollsters do all the time with their samples of public opinion, especially those who operate online rather than face to face. So while you may think it’s just a random sample of viewers to the programme (and so what value does it have beyond instant gratification and the pursuit of everything interactive) we see it as a fairly good snapshot of how Britain is reacting.
From it we’ve already picked up the strength of feeling about the NHS, the shift in women’s votes towards David Cameron, and how Labour in the North has turned on Tony Blair. It also spotted David Cameron as the next Tory leader a year ago, but Frank Luntz got there on Newsnight a day before us.
I’d like to know what you think, whether you’ve played the Perception Panel, seen it, or are reading about it for the first time.
Comments
This technique has been used in the US for a number of years and appears to be a good way for politicians to guage exactly which policies are favoured and which are not by the public as they are heard, without the editorisation of their words by the media before any poll data comes through.
Whether the results are actually read and influential to the policy makers is another matter altogether though.
Hmmm, lets see to use this service you have to be able to watch a show that goes out on Wednesday at noon. As large sections of the voting public will be working at that time of day I am not sure how balanced such a panel can be.
How do you weight it so that people at work are fairly represented or capture the views of those who think all politicians are charlatans!?!
I haven't 'played' The Perception Panel.
I can be rightly accused of repeating myself, when I suggest that 600 responses admitted to a poll - however cunningly weighted - will not give a window on the feeling on the national ground.
Only wannabes, mightavbeens and needalifes (that's me!...)need reply to such poll-itical calls. 600 respondents admitted? That's only a ball-park 0.001% sample, isn't it? And that's assuming that the aforementioned subsets are unique in identity!
A poll can be used to gain unbiased opinion - or carefully constructed to 'prove' a viewpoint that may be in the minority of the designer. A straw poll is not a truly democratic device, is it? It proves only one thing: You can fool some of the people, some of the time....etc.
#2: Indeed. What kind of person is able to play along at all? Surely you have a vastly, vastly skewed sample set.
So if I claim to be, for example, a young lib dem voter in liverpool, my vote will count more...? I expect there to be loads of people claiming to be that next time around.
Jamie, in your article you wrote:
If people in this country are forging their political stances on Prime Minister’s Question Time, it is hardly any wonder the country is going to 'Hell in a basket'.
Whenever I watch it, all I see is the people elected to govern us, trying to score points by seeing who can be the most insulting. And having a good laugh about it.
No, PMQT is not the way to judge what the electorate think of political issues. It is a good way to judge who the electorate think came back with the wittiest retort though.
Bernard
The big weakness in this idea is the fact that people are voting based on PMQs that have been edited by the Ö÷²¥´óÐã. The vote will only be a reflection on the bits the Ö÷²¥´óÐã has chosen to show. I’d rather note for MP in a proper election than on the Ö÷²¥´óÐã’s favourite bits.
As a voter I find this trend worrying as it can easily be manipulated and you may attempt to claim that the results are without bias which is obviously not the case. With the current criticism outside and inside the Ö÷²¥´óÐã about liberal bias, you can surely see that this is not a healthy idea.
As co-creator and operator of said clever computer, I can assure Ian there is no purchase to be made in pretending to be someone else on the Perception Panel to improve your weighting. You'd have to know who was under-represented and that changes every week. And if you did guess right, there's still have to be several dozen of you to make even a small difference. And if there were several dozen of you pretending to be northern libdems then that group might then cease to be under-represented anyway. Plus we monitor unusual patterns very closely so we'd know if it was happening.
On Mark's point, there's one thing not appreciated about Perception Panel which is 'ahem' ...IT'S NOT AN OPINION POLL!!!
It measures perception, not opinion, hence the name. Which means the correct pool of voters is not the country at large, but the audience actually watching the show. It doesn't matter in this context what people who don't watch PMQs think of PMQs, because, as they don't watch it, they therefore have no perception of it. We're measuring how well the politicians are going down on TV. Perception, not opinion. It's not better or worse than an opinion poll - it's a completely different beast more akin to a focus group except with statistically valid numbers taking part.
A couple of other points therefore arise:
1) Why weight it then? Well it has to be regularised otherwise we can't make comparisons. If one week we're 55% men and the next week 55% women, then we can't reasonably compare the two unless we've made the numbers comparable. In fact we can't even properly compare men and women within the same panel if we don't know the composition of the panel. Weighting means we always know the composition of the panel. In a 10 minute turnaround from vote to results presentation, that's pretty important.
And it's right to weight it to the actual national demographics since that's intuitively what people expect to be looking at.
2) Does it matter that the panel is self-selecting? Well opinion polls are also self-selecting. Pollsters ring people up at random and then the people say yes or no to taking part. That's self-selection. We just ask them all at the same time and we do it on the television rather than on the phone. So self-selection is always a matter of degree.
Having said which - yes it does make it less good that there is a stronger element of self selection than if we recruited 600 or 1,000 individuals in advance. But that approach would be prohibitive (believe me - I once tried it.)
But that doesn't alter the fact that it's still a very strong piece of research with its own methodology that has no precedent anywhere else in the world. I'm amused that people think the technique has been 'used in the US for a number of years.' As far as I'm aware, no-one has ever done this before. Indeed the know-how and computer power to do this didn't exist commercially until a couple of years ago. For example our computers re-weight the entire panel every 30 seconds as people hang up and join the vote. I'm not aware that anyone but us knows how to do that.
Listening carefully to the views put forward by politicians, especially members of parliament, and making them accountable to what they have expressedly stated and keeping them on their toes will be a boon to the whole political process. Developing the critical sense and commenting intelligently are vital requisites in a democracy. The perception panel is an excellent idea. May British politics improve by leaps and bounds as a result. Politicians need to pull up their socks and feel the pulse of the electorate. Only then will there be good governance.
Ian - unfortunately if you (and everyone else) says they're a young lib dem from liverpool, that section of society will be over-represented, meaning that weighting will make your votes will count for less not more...
You will only get the views of the retired, unemployed and the stay-at-home mums with this; students aren't awake by noon and everybody else is at work or at school. Why not put it on in the evenings? I mean, how many people get to watch TV at work (and have a phone handy to ring in)?
Ken #8 wrote:
The big weakness in this idea is the fact that people are voting based on PMQs that have been edited by the Ö÷²¥´óÐã.
Ken the vote is conducted on the first 20 minutes of PMQs (we always shut the lines at 12:20, once all the leaders have spoken.)But the Daily Politics broadcasts the whole of PMQs, so there's no question of highlights.
Of course we do select which bits of the panel result you get to see. But that's about to change rather dramatically. Watch this space....
adamliv:
While this is no doubt entertaining and adds to the program, I can't help but think that calling it 'a strong piece of research' is pushing things a bit.
Yes, there is an element of sampling error in any survey- but a truly self selecting survey (which this is, since people have to phone up) based on an unrepresentative population (which this is, since viewers of PMQs at mid-day on a Weds are not typical) can in no way claim accuracy on a national level. Weighting it up to such a level is misleading, even if it does make good TV.
Still, if you do any weighting at all then that's better than most TV/radio/newspaper polls...
Gareth - that would be fair comment if we were measuring the opinion of the national population. But we're not. We're measuring the perceptions of the TV audience. But through the weighting we have nevertheless found the results have foreshadowed national trends surprisingly well.
And I repeat, unless opinion pollsters force people at gunpoint to take part, then they also contain an element of self-selection, since people have the option of not taking part. It's all a matter of degree.
Still I agree this the last word in disciplined polling methodology. But nor is it worthless information. It's actually pretty good, but we're not going to be putting MORI out of business any time soon.
This is the first I've heard of it.
I'm a student (who does get up before 12!) and irregular watcher of TDP - haven't seen it for ages I'm afraid.
As an idea it sounds intriguing - could it be (has it been) used during a live debate between party leaders at election time?
Unfortunately I have to agree with Bernard (no. 7) about the retorts/punch and judy politics of PMQ - even for someone with an interest in politics, it's a turn-off.
Shelly #12 wrote:
You will only get the views of the retired, unemployed and the stay-at-home mums with this; students aren't awake by noon and everybody else is at work or at school. Why not put it on in the evenings?
So if this is shown at lunchtime, by your definition, I am either retired, unemployed or a stay-at-home mum. I watch PMQ's and I am a) 23, b) in full time employment (working a late shift), and c) male.
If this is shown in the evenings, what percentage of people are going to watch it? I could probably count the number of viewers safely on my hands. Everyone would rather be watching the mindless drivel of soaps and reality television (which thankfully I miss because I am at work).
Thanks for explaining the costs and how the system works Mr Donald.
I have to say I’m still baffled as to why the Ö÷²¥´óÐã is investing money in this 'perception panel' business. Also it seems to me that it wastes valuable time on the show that could a) be spent analysing PMQ's with the Ö÷²¥´óÐã's political editor. Or b) you could actually spend some time explaining Parliament to the public i.e. some of it's functions and some of the debates that are currently going on during the course of the week that are not normally given much attention.
You claim to have picked up trends from the 'perception panel' but were these trends not available in other existing polls from commercial organisations? i.e. the strength of feeling over the NHS, the shift of women voters to Cameron (thou that is very debatable in some polling data). You also wax lyrical about Frank Luntz and his so-called discovering of Cameron, but don’t we all know how flawed that process was? Luntz showed speeches of Cameron that showed him in a far more positive light where as his opponents were cast in a negative light.
However I'm glad you realise that there are serious flaws in the 'perception panel' in the sense of how blunt it is. One just wishes that Andrew Neil would not keep saying 'fascinating result' each week when clearly they are not.
Message 12# Shelly:
A little bit of a sweeping generalisation, as a student I think it's little unfair to say that we all stay in bed to noon and have no interest in politics etc. I and many of my friends are regular viewers of the Daily Politics even thou we despair at the way they treat there audience as children i.e. the cartoon's and silly sound clips.
Not everybody who works works 9-5. However, I don't watch PMQs as the real news is the policies that are put forward -- not the weekly sparring match.
Adamliv:
"Gareth - that would be fair comment if we were measuring the opinion of the national population. But we're not. We're measuring the perceptions of the TV audience. But through the weighting we have nevertheless found the results have foreshadowed national trends surprisingly well."
You can't have it both ways! Either you're claiming only to be 'measuring the perceptions of the TV audience' or you are trying to show/predict 'national trends'. The fact that you are weighting implies that you are trying to do the latter (since that is what weighting is for), but when I point out that your methodology is weak you claim not to be. There is no point in weighting unless you're trying to make your sample represent the population, but despite your claim that:
"...unless opinion pollsters force people at gunpoint to take part, then they also contain an element of self-selection, since people have the option of not taking part. It's all a matter of degree."
the views of people able to be at home at lunch-time in the week, who are interested enough to watch PMQs and further interested enough to phone up and take part in the pole CANNOT claim to be a representative sample of the population as a whole!
It might only be 'a matter of degree' but it's a huge degree! It's like claiming that the top speed of a train is 'a matter of degree' better than that of a bike.
Unlike Manjit I think that this poll is probably pretty good value for money in terms of entertainment/popular opinion polls. I have no problem with the poll being done and it fits well into the Ö÷²¥´óÐãs move towards interactive entertainment- but claiming that the data is 'pretty good', or that it provides any more some very rough and ready results requires a lot more backing up than I've seen.
Elections are 'self-selecting' too. And I hazard a guess that the people who use the 'perception panel' are most probably all voters.
The 'perception panel' therefore probably accumulates much more useful data for being 'self-selected' as it tells us the opinions of those people who ultimately matter - the one's who make their voices heard come election time.
Gareth wrote:
You can't have it both ways! Either you're claiming only to be 'measuring the perceptions of the TV audience' or you are trying to show/predict 'national trends'. The fact that you are weighting implies that you are trying to do the latter (since that is what weighting is for), but when I point out that your methodology is weak you claim not to be. There is no point in weighting unless you're trying to make your sample represent the population.
I take your point and I'm not being clear enough in my explanation I fear. We are only claiming to measure the perceptions of the TV audience. But we do have to weight it in order to create a level playing field between panels and also so we know the internal composition of the panels. So what do we weight it to? The argument for weighting it to the national demographic is (1) It's what people expect to see and therefore requires no explanation. (2) What else might we weight it to? We don't know the demographics of the actual audience so we can't weight it to that and if we did, would it mean anything to the viewers to do so? (3) If we weight it to the national demographic, then in as much as our sample does behave like the population at large, we can detect national trends better if we weight it that way. So far anecdotally we have seen a few trends that have been picked up by 'proper' polling as well.
This discussion is very reminiscent of the debate that surrounded Yougov when they first started Internet polling. Everyone said their samples were skewed because they could only poll people with Internet access. You don't hear much of that any more though. In fact some pollsters (so I understand) have begun to imitate them.
I'm glad you find it entertaining. I hope I can persuade you it is a valid piece of journalism too. I wouldn't be doing it otherwise.
By the way, Gareth, my post #15 should have read:
"Still I agree this isn't the last word in disciplined polling methodology. But nor is it worthless information. It's actually pretty good."
Somehow the word "isn't" got lost between post and publication.