Data Scraping

Josh Greenstein

In order to find an appropriate social media platform to find data, it was important to consider the ‘target audience’ for this assignment. I found that Twitter was the social media platform with the most dense, diverse and age-appropriate data for my target audience. According to a study by the American Press Institute, (Rosenstiel et al. 2015) 40% of Twitter users are active ‘to be alerted to or find out more about breaking news’ and 31% ‘to tell others what I am doing and thinking about’. It’s from this data that we can see how Twitter has become a modern day ‘soapbox meeting’ where anyone and everyone can have their say, and it will often be heard. Twitter has become such a meeting place for ideas and opinions that as I’m writing this, 7,331 tweets per second.

Since I started typing this very sentence at around 5ish on a Monday afternoon, there have been over 87,000 tweets.

It is thanks to this unique aspect of Twitter that I’ve chosen it as my social media platform to scrape data from.

By using the Twitter Archiver add-on shown to us in class, I was able to craft a set of parameters to attain some data I could then visualise. I wanted to keep these parameters quite simple so as to not narrow the data set too much.

I searched for anyone tweeting anything with the phrase ’affordable housing’ within a 15 mile (24km) radius of Sydney, so as to get a more local perspective. I figured that the phrasing of the search parameters would ensure the results would feature a lot of parliamentary decisions and discussion surrounding housing affordability.

The results, although not overwhelming in number, did shed a lot of light on the matter. I found 44 tweets in the last week about ‘affordable housing’.


A lot of the results I found from the search were negative, and blaming someone else for the deepening problem of housing affordability. Some people blame their state government:

Screen Shot 2016-09-05 at 5.42.14 pm.png

While some people blame the entire preceding generation:

Screen Shot 2016-09-05 at 5.41.26 pm

One person in particular raised an interesting point showing where many Australians priorities lie:

Screen Shot 2016-09-05 at 5.44.21 pm.png

What I found most interesting is that there is no unified front on the matter. People can sit around tweeting with one hand and pointing the finger with the other all they want but there is no one suggesting anything to make it go away. In terms of generating a visual design response, perhaps there may be something in this idea – creating a ‘who’s being blamed vs. who is actually to blame’ for the issue of housing affordability within Sydney. Something a little more developed than the graph below could be an interesting way to visualise this issue:


One thing I found quite limiting about this subject matter is that its not quite a ‘red-button issue’. Other topics in this course deal with issues that create vastly differing opinions, sometimes creating offensive sentiments. No one talking about housing affordability is going to cause a smidgen of the ‘social media buzz’ of a Sonia Kruger or some other racist. Nevertheless, this issue is multilayered and will affect every Australian in some form, so it is one I am invested in.

%d bloggers like this: