Mental Health Post 6: Scraping the web for data
By Rachel Mah
The value of the information extracted during the process of Web Scraping is often more easily realised once the data is processed into a ‘clean’ and usable format.
Using the Twitter Archiver Google add-on, I conducted a web scraping with the objective of understanding the personal perspectives, experiences, thoughts and effects between mental health and the Australian community.
To narrow down the search, I limited the search rule to areas close to Sydney, New South Wales as well as the keyword ‘depressed’.
Image 1: My Twitter Search Rule
The search results returned 168 results onto the spreadsheet. As I read through the tweets, it became apparent that there were four main types of tweets that made up a majority of the list, with the remaining being irrelevant (things like song lyrics, TV shows or artwork).
To make the results clearer, I colour-coded the four main Tweet breakdown sections which included tweets that expressed feeling depressed (blue), stigmatising/negative tweets (red), personal methods of dealing with depression (pink) as well as tweets of support and advocacy towards mental health (green).
Image 2: Tweet Breakdown Legend
I scoured through the results, reading every single tweet carefully and classifying them into their respective colours which produced the following results (as seen in Image 3):
Image 3: Results Categorised
Image 4 & 5: Breakdown by timeline (left), Breakdown arranged by Tweet type (right)
To allow myself a better understanding of the collated results, I arranged them, first by timeline and then by category. It was clear that expression of being depressed by Twitter users (blue) made up the largest portion of tweets (approximately 55%), followed by Tweets advocating or supporting mental health (green), negative/stigmatising tweets (red), and lastly, tweets about methods of dealing with depression (pink).
Unsurprisingly, apart from the general results, there were many statements about depression being associated with ‘first world problems’ such as being depressed over material things and other wants (being very careful as to point out those that were dramaticised over those that were perhaps actual expressions of being upset or depressed).
Image 6: Honorable (and dishonorable) mentions
In conclusion, this web-scraping activity has brought to my attention that:
- Majority of the tweets that expressed depression focused on the lack of material things, or were a result of petty incidences like being upset over Survivor Australia, football, or celebrities. This may show the diminished emphasis of depression as a serious matter in today’s society as more people take the matter lightly.
- Those who tweeted about being depressed because of the reasons stated in the first point often had a large number of followers, ranging from several thousand to 30,000+ followers. I wondered if this could be an indication that our community has assimilated ‘depression’ as an everyday word to replace sadness based on what we learn from social media leaders as well as pop culture.
- There were more tweets of support or advocacy towards mental health than insensitive tweets. This shows a more sensitive side of our community towards the hardship of others in dealing with mental health issues.
- Twitter users can be extremely demeaning in their words, however these may not reflect upon their actual beliefs as internet trolls often say things they do not mean to trigger a response from others. However, internet bullying is a different issue altogether and has serious mental health consequences on those who are vulnerable.
- Tweets that talked about methods of dealing with depression was the smallest category. These users often pointed towards retail therapy, finance and music as the few things that helped to manage their depression.