Bernie Hogan » OII

Craigslist Discrimination paper accepted for City & Community

BERNiE — Sun, 12 Jun 2011 18:18:37 +0000

What’s in a name? Is Bernie more likely to get a call back for an apartment than Jamal or Abdul? The answer is most likely, at least on Craigslist (and we have just as much reason to believe elsewhere as well) . We explore this issue in a new paper that just yesterday was accepted for publication in City & Community. This journal is the official journal of the Community & Urban Sociology section of the American Sociological Association, and we think a fine home for this work.

We explore this issue using a novel form of an online audit study. Past audit studies have looked at whether people get call backs for resumes, turned down on the telephone and call backs for apartments in person. Our work is somewhat different in the sense that we define a methodology for scaling up this sort of work by using an automated emailing program that I designed and implemented. Brent Berry, my co-author from the University of Toronto came up with the idea and we worked rather hard in putting the pieces together over the last several years.

It will appear in issue 10(4) sometime later this year. A pre-print copy of the article is available here for download [pdf link]. Please note that there will be some slight typographical changes between this version and the final version (so be careful when quoting), but the substantive writing should be the same.

NodeXL / Network Workshop at UKSNA

BERNiE — Thu, 09 Jun 2011 21:39:08 +0000

I’m giving a one day workshop at the upcoming UK Social Networks Association conference in Greenwich. The workshop is on Wednesday, July 6th. Registration is still open.

Here’s the write up from the page on short course :

From data to discovery: NodeXL for social network capture, analysis and visualisation

This workshop will familiarise attendees with the tools to begin social network capture, analysis and visualisation, especially the analysis of data from the web, such as tweets, Facebook friendships and hyperlinks. While the workshop is focused on social media, it is applicable to a host of social networks, such as 2-mode, ego networks and whole networks.

The workshop will quickly cover some of the basic topics in data collection and analysis and move on to visualisation through the NodeXL package. This software, designed by consortium of HCI specialists, social scientists and technologists from Microsoft Research is designed to make the discovery of patterns with social networks straightforward and intuitive.

By working with data already in Excel, NodeXL makes the preparation of network data sets simple and enables the researcher to focus most directly on analysis and intuition building. By having data capture tools embedded directly, it also serves as an extensive and highly customisable way to capture data, not just analyse it.

NodeXL is a free download but requires Microsoft 2007 or 2010 for Windows (Mac users need to use Bootcamp or Parallels). It is designed by the nascent Social Media Research Foundation, and already is actively used by both industry and academic researchers. A learner’s guide and academic text, ‘Analysing Social Media with NodeXL’ was published last year by Morgan Kaufman.

Topics covered include:

Network types and network metrics
Intuition building through visualisation
Coupling metrics and visualisation
NodeXL as a data capture tool (Twitter, WWW, Email, Facebook)
Importing and exporting data and images from NodeXL
Advanced NodeXL Features: Groups, Automation, Macros.

Its a tall order, but I’ve already done a few of these workshops so far, and I can guarantee it will be fast paced, engaging and hopefully insightful. If you are interested in have any questions, feel free to email me. The course costs £100 (and £50 for students).

OII Network course 2011: A Showcase

BERNiE — Fri, 03 Jun 2011 17:19:42 +0000

For the second year in a row, I co-taught the OII’s Online Social Networks course with Dr. Sandra Gonzalez-Bailon. Again this year we had some excellent studies and some excellent students. But I thought the graphs were so visually interesting, and often creative that I’ve asked for permission to compile them in a gallery. This isn’t a complete list of sociograms (and of course, not every network analysis needs a sociogram) but this work ought to give you a sense of what we have been doing.

The students have provided the text themselves and often additional material if you follow links in the descriptions and comments to their own personal blogs and websites. Enjoy!

What you can and can’t get from Facebook

BERNiE — Thu, 25 Nov 2010 15:42:49 +0000

I have had several interesting requests for my NameGen program, including the much maligned desktop application that is not really active development (pending future grant money). What is interesting is the sorts of requests that come in based on misunderstandings of what Facebook provides to an application developer.

So, what can and can’t you get from the Facebook API about an individual and their friends?

I. You can get a list of a user’s friends, their names, and their profile pic. Here is the FQL:

SELECT user,name FROM user WHERE uid IN (SELECT uid2 FROM friend WHERE uid1 = $user_id)

If the user has made other information public, that is often accessible as well, such as gender, date of birth and hometown. Some protected information such as relationship status is never available to the API for anyone other than the user. You can customize what information is available to others through Privacy Settings -> Applications, Games and Websites -> Information Accessible through your friends. This figure shows what I have set. I believe it is the default.

The customization screen for Facebook's information sharing between friends via the API

II. You can get a 1.5 degree personal network. (Please note – this is a simplified and suboptimal query that will not return a complete network in most cases where the network is larger than 200 nodes).

SELECT uid1, uid2 FROM friend WHERE uid1 IN (SELECT uid2 FROM friend WHERE uid1 = $userid ) AND uid2 IN (SELECT uid1 FROM friend WHERE uid2 = $userid)

This will return a list of friend links. There is a more inefficient way of querying this information using the get_friends() query. It requires you to send all possible friend combinations, and it will return, for each one, whether the two users are friends. So the list length will be n(n-1)/2 where n is the number of friends. That’s approximately 125k elements for a 500 person network. Those elements have to be sent in batches of 5k max (as far as I recall).

III. You can get the links between friends in a group. I don’t have this query on hand, partially because Facebook will not return a complete list of group members (at least through FQL), but if you are in a group, you can ask if other group members are friends. The same cannot be said of a fan page (i.e. The ones where you ‘like’ rather than the ones where you join). One attempt to explore this network is in http://apps.facebook.com/netvizz

An example showing the 1.0, 1.5 and 2.0 ego nets (taken from my chapter in Analyzing Social Media with NodeXL - click pic for book

What can you view through Facebook (and theoretically spider) but not get through the API:

The 2-degree network: If you can view a friend, you can view all of their friends. That said, the API will not return these people. It only will return the friends of a single user.
All members of a group.
All members of a page (if you are a page administrator).

So…what does this mean? It means that basically, Facebook allows people to learn about the network immediately surrounding them and that’s it. It similarly means Facebook will not allow you easily to learn the personal network for someone else. So consider the following requests that won’t work:

The personal network of a person’s friend. Being able to access this would be a problem as a clear form of surveillance. So if you want to view a friend’s network, you should ask them to download it themselves.
This also holds for parents - no I will not help you download your teenager’s network just because she put you on limited profile. Moreover, this is not even possible unless you and your daughter and all of her friends are in a group and you can get the complete list of people in the group. And recall, while it is possible to add other people to a group, those people must already be your friend, so you can already access those ties anyway.
The personal network of someone you are investigating. If you do not have a warrant to download this network (or to compel a suspect to download their own network), and you want this information, this again, is surveillance. Also, I’m pretty sure that taking the network of someone else would be a breach of the terms of use. And as for the warrant – I would tread carefully, as this will probably involve all sorts of legalistic red tape (especially in privacy friendly countries such as Germany).
This also holds for people curious about the personal network of anyone they are not friends with. A Facebook friendship is a form of information access and redistribution. If you are not friends with someone, they have not granted you this access. This is the sort of notion that got Pete Warden into legal hot water with his 200 million-strong global Facebook network that he was planning on releasing.

Personally, I think that data portability is a good thing, and I am currently following the debacle between Google and Facebook rather closely, but I also think this needs to be a carefully metered out. And I don’t think that Facebook is a sinkhole of data – in fact, their API is very liberal. In some cases, perhaps too liberal. This is especially considering the perennially evolving terms of use. But they have made some careful decisions that I tend to agree with (such as limiting the 24-hour caching requirement), and while I wouldn’t give them carte blanche support, I think they are making baby steps in the right direction. Thankfully, those steps do not involve downloading the networks of others without their permission and collecting friends of friends of friends.

Facebook’s “Download your Information” reviewed

BERNiE — Wed, 10 Nov 2010 20:56:16 +0000

As part of Facebook’s march to capture as much of your life as possible while making you feel okay about it, last month they announced a “Download your Data” feature. This feature has been rolled out in waves, and happily, that wave lapped up on my shores yesterday. So here’s a review of the feature

The short story is that this is a pretty neat feature that is clearly still a first step. In particular, there are a handful of absences, such as not knowing who ‘liked’ your posted. Also, it won’t download any of your friend’s data, so if you want to view your personal network, you’re still left with applications like my own namegenweb or netvizz.

The data that comes down is basically a series of queries on the big table that holds all of this data, spit out as a series of sparse html files. The source code of the files is nice and clean, with legible div tags, but its no easier to parse this data than it is to do so through a series of fql queries. Obviously, most users aren’t going to either look at the source code or use fql queries, but its nice to know that their API is so thorough and open that you were able to get all of this data even before the “download your data” feature.

To get started, you have to go to account -> account settings. There should be a feature there called download your information, with a link called “learn more”.

Click on account settings

If you are part of the roll out, then you can click on this link. If not, then it tells you to sit tight, and that it will be available sometime in the future. When available, it gives you a big preamble and a simple “download” button.

Once clicked, that button should turn into a greyed out “pending” button. The first time I did this, it emailed me and said there was something wrong with my file, and that I could try again. I tried again, and the button never turned to “pending”. So I logged out of Facebook, closed my browser and tried again. This time it worked.

If everything works, you should see this screen before waiting.

It took about 2 hours for the email to come to me, and it was very simple to click on that link and get a zip file.

Inside the zip file was a series of folders and a pretty straightforward “index.html” file. I’ve posted a stripped down version of mine to NameGen so you can get a sense of what this whole thing looks like.

Example Profile Screen

Comments to photos appears as usual. People tagged in photos do not. Both messages on one’s wall, and the replies show up, but interestingly, only the number of likes shows up. You cannot see who actually liked it. I can’t comment on “places”, since I haven’t used that feature and do not know how it would show up.

Curiously, Event IDs are the only thing that is clickable. The remainder are not – that is, you cannot click on a message and go to the live version on Facebook, or click on a friend and go to their profile.

It is nice that this is all in a clean html format. Its a step towards data transparency, but not towards data portability. There is a great deal of work that would be required to actually slice up this data. But as I mentioned, most of this is already covered in the Facebook API, so it doesn’t seem like having another point of entry to machine readable structured data was necessary.

(Face)Book week for Bernie

BERNiE — Thu, 16 Sep 2010 20:52:42 +0000

Wow, what a week for Facebook and me! Nevermind the fact that results from the “So you want to be a scientist” project with Nina Jones went massive on BBC’s website (as the top story two days running with almost a million views). But much hard work and perseverance from colleagues of mine have turned into two fine new books work checking out. And, as you might have suspected, I’m featured in both of them. And so I’m happy to give two utterly biased reviews to two books on social media released this week. “Analyzing Social Media with NodeXL” and “The Facebook Era, 2nd ed.”. Details below.

The first is the new offering from the NodeXL team: “Analyzing Social Media Networks with NodeXL: Insights from a Connected World”. This book is an introduction to social network analysis, particularly through the use of NodeXL, a rapidly evolving free set of social network analysis tools for Microsoft Excel 2007/2010. Sadly, it doesn’t work on a mac, but on windows (and parallels) it works wonders. NodeXL has always considered itself ‘social network analysis for the masses’. That is not to say it is watered down, so much as designed with an exceptional attention to user interface details. You will not find a lot of data manipulation tools or sophisticated regression models in NodeXL, but you will find some of the most straightforward tools for visualizing networks, clustering them, performing interactive filtering, and importing and exporting. But perhaps NodeXL’s marquee feature is its ability to tap into many of today’s prevailing APIs, such as Flickr, Twitter and YouTube. Using these APIs, it is possible to very rapidly download and visualize user and tag networks from these sources.
The NodeXL book is in many ways meant for the network newbie and the business practicioner. As such, you would think the tone would be simple and practical. For better or worse, however, the tone is actually thoughtful and deep in many places. Fortunately, the authors have a strong enough grasp of the material to keep things clear. I am personally impressed by their ability to walk this line. People may not be using NodeXL in its current state five or ten years down the road (or maybe the will), but in either case, this thoughtfulness will give this book a notable longevity.
I’m the author of the chapter on Facebook networks. NodeXL does not natively download Facebook networks, but it is possible to do this through my application, and through another application called NetVizz. I think my application could certainly look a little prettier, but it (usually) gets the job done. (Unless you have over 4000 nodes, then things tend to get a little touch and go). Look for improvements over the coming month.

The second book I received in the mail this week was Clara Shih’s “The Facebook Era”, Second Edition. Clara is a social media networking maven, and is building an impressive resume as the go-to person for building brands and businesses on Facebook. I thought the first edition of the book was a bit too straightforward in its focus on business brand building. It made the book handy for the complete neophyte to Facebook, but not for most others. By contrast, this version of the book is far more sophisticated in my opinion, while still preserving its focus on the target audience and its clear writing. In addition to several expanded sections on social media generally and some quasi-academic musings on social networks, this book includes many sidebars from the who’s who of social media research and practice. Clara has done a fantastic job lining up people to give their two cents on the topic, and the book is better for it. I’m among them giving a little report on the visualization of a Facebook network.

Its not often that I get to say I learned something new from trade business books on social media. I’m happy to report that this book was not simply clear and timely, but thoughtful and interesting.