POST 6: Scraping the Web for data
BY CHRISTY HUI
By web scraping the Internet for specific data, it helps with researching and insights others have put up on to a social media platform, for this case, Twitter. By downloading a Google Chrome Extension, Twitter Archiver, when a set of rule is created for this extension it helps search and collate content from Twitter.
I wanted to collect results based on housing affordability in Australia. By creating a set of rules on Twitter Archiver, the results I received was poor, with only 3 tweets. So then I decided to focus more on a topic I was more interested in which was affordable housing. I created a new Twitter search rule, where I included words that can potentially help me with my research and also hashtags that may help refine the search.
Twitter Search Rule: affordable housing, #affordablehousing, #housing and lang: English.
Twitter Archiver accumulated 1038 tweets in the past week using the rule I created.
It has collected a wide rage of interesting results and observations since Twitter is used by a community of people, who shared thoughts, information, data and ideas. Twitter allows users to post on Twitter about their personal stories or comments about ongoing social issues, celebrities or simply using it for networking as another form of social media.
I went through a few results by skimming through “Tweet Text”, from this I selected a few interesting articles that others have tweeted about the keywords I have input in to the search system.
From my observation, I realised that most of the tweets are on a based on a global scale. Most of the tweet are looking at the US, UK and Asia, this is because there is a larger density in population in these areas. Most of these tweets raises question about the issue with housing affordability, to bring awareness to those who are interested in knowing more about the housing crisis.
I found most of these tweet really interesting as they have different approaches to the issues, some are more informative and explores about the causes and effects. I also observed that most of these tweets are tweeted by different government organisation, business companies and design institutions. When using the Twitter Archiver I found a non-profit organisation called Next City, with a mission to inspire social, economic and environmental change in cities through journalism and events around the world.
Web scraping was useful, but it was very difficult to filter all the result to a desirable result. I wanted to look into affordable housing in Sydney, but when searched in the Twitter Archiver some of the result were unappealing. I decided to take a step further and gone into Twitter and searched “affordable housing Sydney“, instantly it presented me with more promising results without having to scroll through the Twitter Archiver.
Although I was able to look at this issue on a global scale, I wanted to focus more on affordable housing within Australia. I explored the idea of innovative living and affordable housing that can be implemented into our busy city. With the search from Twitter, it made it a lot more convenient to browse through content since I was able to understand each article based on the headline, most of these tweets seemed to be very biased and positioned in such negative connotation. It seems like people are just tweeting and blaming others rather than confronting the problem and finding solutions to solve these issues.