Post 6: Scraping the web for Data

The internet has become a growing resource for information. With so much data available on the internet, to reach what we need and to find credible information is by searching, and filtering through to find the most utilisable data.

As the internet is being filled with data, new technologies are being created. Lucky for the most of us there are search engines and data scrapers to refine our searches. From that point we are able to collate the data into documents that can be utilised in design.

At this point in time I wanted to collect data to back up my idea that there is desensitisation evident on social media. So I looked up different searches along the lines of:

  • “Today is…”
  • “I’m so depressed..”

My google sheets didn’t work the twitter archiver very well. I then attempted to use another recommended application called “Data Pipeline”.



I did go through those first searches to begin with but I then preferred to have results showing intellectual discussion. I chose to search up “mental health cope” and “mental health joke.”



I preferred how it was presented in a preview. This showcased the most popular tweets. When I received the data results on excel, the results showed all the retweets which were repetitive despite receiving 500 tweets. I learnt the importance of filtering within a search, to exclude, and the importance of knowing which words would retrieve the best results. From what I had gathered I had learnt:

  • Twitter is self-aware of joking about mental health can be a coping mechanism
  • There is an invisible and undefined line when a joke about mental health is inappropriate
  • New Zealand has a terrible mental health system
  • There is a belief/trend that only those with mental health issues may joke about it
  • intellectual discussion can happen, it is not all angry ranting


North Concepts Inc, 2016, Export your Twitter Searches to Excel, viewed 4 September 2016 <>


Author: Danyen Rosa Nguyen

Visual Communication Student at University of Technology, Sydney

%d bloggers like this: