The Verdict of the Herd: Leverage the wisdom of the crowds to combat malware and document Internet filtering
4 Comments Published by tobias.escher May 18th, 2007 in *OIINEWS, filtering, Security, web 2.0The “first public conference to discuss the current state of play of Internet filtering worldwide” is in full swing in St. Anne’s college in Oxford and there are already some online accounts of the main findings of the study (see their main website if indeed you are able to access it).
(update:for a near live coverage see Lex Ferenda)
UPDATE: if you’ve missed the conference, the Berkman Centre has now made videos of the conference available)
I will try to cover what’s on the horizon.
Jonathan Zittrain (OII and Berkman) and John Palfrey (Berkman) introduced a new distributed application that they are devloping. It’s called Herdict – meaning “a verdict from the herd” – and is basically a software application people can install on their machines for the following purposes:
- Detect Badware: collect vital life signs of the computer
- Document Internet filtering: document Internet sites the user cannot access
- Measure Network Neutrality: measure network latency
The data collected by Herdict is sent back to a Berkman server, aggregated and distributed again so that each user can compare the performance of his/her machine/network with the performance of other machines in his neighbourhood or across the world. Eventually the software could help users
- to decide whether or not to install a certain piece of software (based on other user’s experiences with it)
- finding out that there is a problem with your machine as yours is performing much worse than everybody else’s
- produce a near real-time map of Internet filtering around the world that allows for an analysis of where filtering is actually taking place: on state, ISP or institutional (eg. company) level
The setup is likely to remain a centralized client-server architecture (P2P is possible but creates a whole host of security and trust problems) but the ultimate aim is to allow also interfacing with the data via an open API so that 3rd parties can create widgets using the collected information.
While the audience was widely in favour of the capabilities of such a tool, many concerns were raised about whether the privacy and security of Herdict users would be at risk. An Ethan Zuckerman in full constructive-criticism mode raised a whole host of issues but also suggested some potential solutions. Others joined in the discussion which I try to summarize below together with some responses from Jonathan and Steven J. Murdoch who is responsible for some of the technical ideas behind Herdict:
- Does the collected information about a machine’s (mis)configuration not help malware programmers?
JZ: These people do already know enough about the weaknesses of other people’s machines (e.g. via their botnets) so this information won’t help them but will make a difference for the good guys - Does one maybe draw attention to an insecure configuration on his/her machine so that it can be targeted specifically?
JZ: Definitely important to strike a balance between openness and the ability to misuse this information. Possible solutions could be to synthesize the collected data. However, the applications installed on your computer give out so much of your information already (e.g. Skype, IM) that one should maybe worry less about Herdict but just start to use the information for a good purpose (if you cannot stop it from being distributed anyway) - Doesn’t the centralized client-server architecture make it easier to manipulate and interfere with the data collection?
JZ: Exactly because the data is centralized it is easier to guarantee its freedom if it is operated by a trusted party with adequate licensing (see Wikipedia example)
SM: client-server could be more secure than P2P because you only need to make sure that the server is trustworthy, not the whole array of hosts in the network. Also there might be a centralised aggregation of the collected data but a distributed communication to transport the information. The data will be sent encrypted and stripped of personal information. - Is a centralized architecture not easily blocked?
JZ: This would be a sign of success (as governments would apparently take Herdict seriously) but if that would really happen one could think of new ways of working around that. - Even with Herdict it would be difficult to detect tempering with pages (e.g. the BBC page is served but with slightly different content, e.g. less critical of your countries government)?
JZ: One could incorporate a review or user comparison of web pages, something like the ESP Game - Do people not draw attention from the authorities to themselves if they start using Herdict and in this way accessing forbidden sites to document filtering?
Ethan: One idea could be just to report failure of accessing a site back to the Herdict server or a model similar to SETI@home where you download a bunch of URLs for testing. Also, no need to disclose full IP address as class C should be enough to estimate your location.
If you are interested, download the alpha version. which so far is only measuring and comparing the machine’s life signs (e.g. amount of free memory, number of processes running etc.)
4 Responses to “The Verdict of the Herd: Leverage the wisdom of the crowds to combat malware and document Internet filtering”
- 1 Pingback on May 18th, 2007 at 5:12 pm
- 2 Pingback on May 18th, 2007 at 5:21 pm
- 3 Pingback on May 19th, 2007 at 11:10 am
Leave a Reply
Search
About
Since October 2006 I am both a DPhil student as well as a research assistant at the Oxford Internet Institute and here I share with the accidental reader my musings on different aspects of the Internet and society. Feel free to comment or simply ignore :-)
-----------------------------------
Tobias Escher
Oxford Internet Institute
1 St. Giles
Oxford OX1 3JS
firstname.lastname@oii.ox.ac.uk
+44 (0)1865 287210

Thanks for the link. Unfortunately live blogging was interrupted by a real bad day of the server (ironically, I ended up with comments disabled – ironic, ain’t it?). Should be fixed now, but a distraction (I had to do a full backup/reinstall on WordPress).