POST 6 Scraping the web for data

Scraping The Bottom Of The Bowl That Is The Web

Being someone who is incredibly technologically NOT inclined, the whole idea of web scraping and tools shown to us in lectures and tutes, plus those that I came across on google were something I found to be a little too detailed, crazy and overwhelming. Having attempted the simplest version of a data scraping tool with the twitter archiver add on, and having little success (it didn’t want to archive for me – 48 hrs later and still no data on my spreadsheet), I settled on a simple yet effective from of web scraping – looking up hashtags on Instagram and twitter.

I’m not sure if I was just doing the web scraping wrong, but after using twitter’s advanced search, with the hashtags #HousingAffordability and #AffordableHousing and #Sydney – it came up almost exclusively with tweets from building/housing and architecturally affiliated companies or just retweeted news articles on the issue. Plus the timeline of tweets available to me only went back to Aug 2016 – why? Because technology clearly hates me. There were the couple tweets that reinforced that popular opinion that housing affordability is in fact an issue in Sydney. And that the current climate needed to be solved more holistically. Eg. Government policies are at fault, the structure of affordable housing solutions need to be reconsidered. Twitter pretty much regurgitated/retweeted or rebutted all the opinions and views covered in the media news outlets. Those of which we explored in the first news article blog posts.


From the very beginning of this research, it has been made apparent that there is so much “housing jargon” on media articles and so many economic and political opinions surrounding the issue that it has been hard to find a “man on the street” opinion on the matter. Something that isn’t clouded by all these fancy terms and words to describe the housing crisis – which is all I found on twitter.

Instagram is a more of a casual, less opinion heavy and loaded forum, so I turned to that to see if I could find a “layman’s” (for lack of better word) perspective and vibe on the issue. I used a hashtag search of #HousingAffordability and #AffordableHousing


Instagram is great because it captures a candid moment or thought, through a visual format. Aware as I am that Instagram posts can be staged and pre-meditated as much as any other social media, a picture is much easier to filter through and understand in a matter of seconds. And it was interesting to see how people represented the issue of housing affordability in their images, some humorously like with this woman’s wry statement on her cat becoming a home owner before herself.


Or a more serious note, of social commentary – with this poignant candid picture of a homeless person sleeping blanket-clad with the hashtag #cityofsydney and #housingaffordability. I feel like images like this with a satirical/ironic tone, really capture and reinforce the sentiments of city goers on the housing climate. Especially that of our generation – that resort to dry humour and juxtaposition of images to make a statement – a sort of resignation to the issue.


Overall data/web scraping is something I feel like I am going to take a while to master (lucky i’ve got the rest of semester), but an effective way to find a position on a topic of interest or a way to look at the overarching trends and confirm/debunk your initial assumptions. Also will never understand why there were so many social media hits for #Vancouver when I was #Sydney…maybe a hint for a future topic/housing context space to explore?

%d bloggers like this: