🏠 P6

Scraping the web for data

By Clyde Overton


To further my understanding and retrieve a stronger insight in housing affordability from a perspective of an individual or organisation, I chose to gather information and data through the use of Twitter web scraping. I’m an avid Twitter user, and I probably use it the most out of the most popular social media platforms. Twitter brings a real, live, public view on things from current events, trends, to what someone has had for breakfast. Using Twitter to scrape data, I was able to find information on a platform I am comfortable in. Using Twitter meant that I was able to find a more personal opinion on the subject matter of housing affordability. This is why I find Twitter the most fitting platform for the housing affordability sector.

The first process I performed was using the Twitter Archiver, on Chrome. I really liked how this method organised the information and Tweets. The Twitter Archiver additionally organised and collates over information such as location, Retweets, and the platform they Tweeted on. This gives an additionally insight in how and what position the Tweeter is Tweeting in. For this process, I simply used a number of key words of housing affordability, all within the Sydney area. I was overwhelmed with the hundreds of thousands of Tweets that the Twitter Archiver gathered. This was a broad web scrape, but a basis for further refining my desired information. This format gave a strong insight in what people are talking about, as well as their unfiltered thoughts on the subject matter of housing affordability.


For my second process, I chose to use the inbuilt advanced search feature in Twitter itself. This was a more comfortable format to work with, however I did miss the over complications and irrelevant meta data that the Twitter Archiver offered. However, I was able to refine my searches for a more specific Tweets in mind. The following Tweets that I have embedded below show an insight in what people are talking about in terms of housing affordability in Sydney. I have chosen a few of the Tweets that I found for simplicity. These include two news related Tweets, which show how the news broadcasters are introducing new market information to a more casual context. The other Tweets are much more personal and opinionated, with them either responding to news articles or giving their own take on the current affordable housing market in Sydney. Overall, I found the web scraping insightful in how more personal information can be found and gathered, and how it can be compared to more scholarly texts or articles.