In my doctoral research I look at the communication between constituents and their Members of Parliament. I was looking for a simple way to judge the relevance of the mail that MPs receive from their constituents. As I found, MPs tend to refer to their “postbag” in order to emphasize the importance of an issue as e.g. the Simon Hughes did in a recent debate on climate change (see video below):

So in order to judge how often that happens, I needed to have a look at Hansard, the written record of proceedings in parliament. Now the main online Hansard record at the UK Parliament website is rather difficult and does not provide a search functionality so I turned to TheyWorkForYou.com, one of mySociety’s projects that does not only provide detailed information on MPs but also offers a nicely formatted, searchable version of Hansard, now dating back to 1935 (!).

In this way it was a matter of seconds to find out how often MPs and Lords have mentioned their postbags in parliamentary proceedings since 1935 (it was 1,621 times). A multitude of options allow to filter your search accordingly so that now I know that the majority of these mentions were made during House of Commons debates (989) and that in the current parliament Conservative MP Mark Field leads the table (with 9 references to his postbags). Finally I could quickly produce a figure showing the development of references to their constituent mail:

btw the search is intelligent enough to look for the word “postbag” as well as similar words such as “postbags” or “post bag”

I cannot start to imagine how long it would have taken to produce this figure with the limited capabilities of the official Hansard and it would not have been possible at all at the time when all this data was really only a written record in the literal sense. Not saying that this particular piece of information is a world-changing discovery but it is a good example of how the availability of data in a structured and searchable format (!) can contribute to scholarship nevertheless. In this respect the various Open Data initiatives by governments offer a huge potential for social scientists with the appropriate statistical and computational skills to offer fresh insights. See for example the Guardian World Government Data Initiative that offers the datasets opened up by various different governments in a uniform format.

Ok, here is the task: Automatically query Google Blogsearch to find the number of “all” English-language blog posts containing a number of words that have been published during a certain period. Should be easy enough? Well, you have no idea…

Before I start, why to use Google Blogsearch anyway? This is why: Technorati does not allow to limit the time for searches, Blogpulse does not offer to filter languages and is very slow anyway and last but not least Bloglines, while doing a good job in general, reports less results than Google and also gives very erroneous estimates for total post counts.

First problem: Index update intervalls

At first sight Google Blogsearch is great. For any kind of search Google will usually show you some posts that have just been published. This is great but clearly Google simply cannot index all blogs at the same time. The crawler will visit less frequently updated blogs less frequently and even then we don’t know when Google updates the Blogsearch index. If you are interested in “all” posts that have been published in the last week (or at least all posts that have been published on blogs Google tends to index), you should better wait to query Google Blogsearch until the index has been updated to include all those posts that have been written in the last week but have yet to make it into the index.

But how long should you wait? I’ve used a random sample of 500 queries that were first submitted to Google exactly two days after the period of interest and repeated those searches again. I divided the newly obtained result by the initial ones (so this should be 1 if no change in total post count or larger than one if more posts were found in the second query) and plotted them by days that have passed since the date of interest.

What you can see is that first of all that there is a huge variation in the counts, so usually you will even receive a lower total count if you query Google Blogsearch after some days. However, this is a problem in Google’s estimation techniques and I will deal with that later. What can be seen is that there is a clear increase in posts starting about 10-12 days after the period of interest. Ergo, it is probably safe to assume that Google has completely updated the Blogsearch index after a two week interval – assuming that the index is updated continuously. Clearly this experiment should be repeated to be safe that we not just hit the middle of for example a 4-week cycle.

In contrast, below is a picture of Yahoo News. As one would expect, there is less variation because this search has a lesser number of outlets to index and crucially relies on being up to date.

First lesson learned: If you really want all posts published at day X from all blogs Google indexes, wait about two weeks from day X onwards before you do your query.

Second problem: total result estimates

As you probably know, basically the number Google reports for the total number of results is an estimate – and a VERY bad one. The reason is that Google cannot be bothered to really search for ALL posts containing a word if 99.9% of people are only interested in the top 10 ie. most relevant ones. Therefore it just checks the most relevant ones and by some magic produces an estimate of what the total number is likely to be. Once you start browsing through the list of results you will likely find out there are much less results than originally indicated. Need a proof? See the screenshot below:

The interesting bit is that Google will give you 649 results (as you can see there is no way to request more) but still it claims there would be a total of 2,088 results which is clear rubbish (Note that this does already include the duplicates!)

Second lesson learned: Do not, never and by no means use the total result estimate. Instead, use the number of the last result Google returns. This has the problem that Google will never return more than a 1,000 results.

Third problem: The same query will give different results (just keep trying)

Taking the lessons learned from problem one and two you could just construct a query that asks Google for the last results page straight away (ie. you append &start=990 to your query). However, if you do the same query again you often obtain a different, usually higher, number of results. Give this one a try. Initially it brought something in the region of 600 results, then it were 900 and if you play long enough (by browsing a bit through the list) you may get to a 1000. Again it seems like Google is not putting much energy into the first query but once you do the second it has already a head start and will give you more.

Third lesson learned: In order to get a more accurate number from Google, do a query for the top ten results first and after a few seconds, do another one for the last results page.

Fourth problem: different results from feed and interface

This really, really, really is just annoying. You will obtain different results if you do exactly the same query and access the results via the user interface or via the XML feed.

Fourth lesson learned: Decide for one way of accessing the results and stay with it to allow to compare the obtained counts. The feeds seems to return the higher number of results.

Fifth problem: Google is sorry…

Well, just when you think you have solved all your problems (at least related to this search thing…), Google blocks your access. I have yet to figure out why exactly as the policy is somehow inconsistent (sometimes it happens very quickly, sometimes it takes half a day) but apparently Google does not like queries for the last results page – which is no wonder as it requires more computing power. When only checking for the top 10 results I never got the following page:

This seems to be unrelated to whether one is using the normal web interface or the XML feed.

No lesson to be learned here…

On SPAM

One problem that keeps cropping up is SPAM. You can never totally eliminate bogus blogs that were just set up to generate some ad revenue and lure unsuspecting people to dubious business offers. But why for some queries posts like the one below do get included in droves is totally beyond me.

This seems to be a particular problem with blogs on (Google’s) blogspot platform that has loads of automatically generated blogs. I try to treat SPAM posts as random error, hoping that it will affect all queries equally (and this really is only a hope…).

Last words

I hope this is going to help some people who also need some reliable counts on the number of posts dealing with a certain topic. In my research area, this certainly is an issue. What is more, let me know if you have more or different experiences so that we can learn from each other. btw most of these findings should also be true for Google search more generally.

Finally, of course one has to be somehow grateful to Google to at least provide a service like that for free (yeah, I know they do ads). But what really sucks is that Google sits on the data and could give you all the numbers you need straight away with 99% accuracy. So to end with my usual plea: open up your index and make it accessible (at least for research purposes)!!!!

I have been thinking of a way to test whether the blogosphere really does constitute a counter public. I have developed a tool that compares the media agenda – that is a ranking of stories reported within 24 hours – to the blogging agenda and measures the overlap between the two. The main objective is to find out whether bloggers are applying different criteria to rank the importance (salience) of a news story than traditional journalists.

My paper describes it in more detail but basically I construct the agenda from the stories on Google News, extract the key words for each story with the Yahoo Term Extractor and search with the help of Google Blog Search how many posts cover this story. You can have a look at the data on this website but you will realise that data collection stopped some time ago.

In due course there will be an update of the tool along with the paper that will improve the data reliability but for now I very much welcome your feedback on it!