POST 6 Scraping the web for data

In this weeks tutorial, we learnt that social media can play a very important role when collecting data ethnographically. Twitter is a social media platform where individuals can share their opinions on topics publicly and essentially agree or disagree with others. Hashtags are an integral part of Twitter and they are used to highlight a topic of interest for ease of search and to group similar topics together.

For this web scraping activity, I chose to search pay gap, that can include the words gender equality and potentially the hash tags #gender and #feminism. I thought these words would relate well to the issue of gender equality in the work force.

Screen Shot 2016-09-04 at 12.41.59 pm

Using the Twitter archiver add on tool, it fetched 1261 tweets with the search rule I had created over the last week.

Screen Shot 2016-09-04 at 12.47.03 pm

I chose the top five tweets from the list (that weren’t spam) and analysed those.

Screen Shot 2016-09-04 at 5.00.38 pm.png

Image sourced from: Snapp, K. 2016, Katie K Snapp on Twitter: “What #Gender Pay-Gap Statistics Aren’t Capturing #skirtstrategies”, viewed 4 September 2016, <;.

The first tweet on my spreadsheet was probably the most interesting. The tweets caption immediately gained my attention as I feel as though I have seen so many different statistics showing how women are still earning less than men, the thought of their being different information intrigued me. The link that was on the tweet took me to a Huffington Post page where it discussed the fact that the pay gay statistics were in fact incorrect as women choose different jobs to men that have different salaries. This idea was summed up nicely in their sentence ‘And so, the argument goes, men and women aren’t paid the same amount of money because they are choosing to go into different professions, and the labor market rewards their choices differently. In other words: unequal work, hence unequal pay’ (Huffington Post, 2016). This definitely opened my eyes to a whole different perspective for this issue, I had never considered the fact that perhaps women were choosing different professions. This is definitely something I want to look into, and I wouldn’t have found it if it weren’t for this archiver tool.

Screen Shot 2016-09-04 at 5.15.52 pm

Image from: Sees, S. 2016, Twitter, viewed 4 September 2016, <;.

The second tweet was from the UK and lead to a link for the Guardian which provided different statistics on the fact that in the UK women were still earning 18% less than men (the Guardian, 2016). Having this tweet directly after the one I previously looked at confused me as I wasn’t sure if these statistics were correct. Were these different genders in the same profession or were they in completely different professions? I am yet to form an opinion on this side of the issue, I feel very conflicted.

Screen Shot 2016-09-04 at 5.16.00 pm

Image from: leswolves, 2016, leswolves on Twitter: “Looking for ways to tackle the gender pay gap | Letters: I was interested to read the news on t… #gender #news”, viewed 4 September 2016, <;.

This tweet was also interesting and allowed me to gain insight into another different perspective. The link on this tweet was to another Guardian article, interestingly written by a man. He discussed how this gender pay gap is made up of many assumptions. The biggest one being that somehow for women ‘jobless means workless’ (the Guardian, 2016). This statement is referring to the fact that women are being ‘penalised’ in the economy due to the fact they are dropping out of the workforce to care for their newborn babies, young children and just generally wanting to be a part of their children’s lives. The pay gap doesn’t take into account the ‘unwaged labour’ (the Guardian, 2016). I think it is unfair that the pay gap isn’t taking into consideration the effort and commitment mothers go to look after their children.

Screen Shot 2016-09-04 at 5.16.33 pm.png

Image from: MiVR, 2016, Twitter, viewed 4 September 2016, <;.

I found the caption of this tweet immediately interesting! I wasn’t quite sure how these two concepts linked together but I thought it was a very interesting way of gaining audience views. The link on this tweet took us to a website that looked at a new form of software that allowed women to use virtual reality to essentially practice negotiating their pay with their bosses. The statistic on this website that ‘1 in 8 women are willing to negotiate their compensation compared to 1 in 2 men’ (Winkler, 2016) really intrigued me. I therefore think that this virtual reality software is a good idea for women to practice using a narrative format (pick your own story). I know I like to practice saying certain things in my head if I am going into a serious conversation, so having something close to reality would be quite helpful. This would definitely be a small step in the right direction for gender equality. However another astounding statistic on this website is that ‘women won’t reach pay parity until 2059, or another 44 years’ (Winkler, 2016). That’s a very long time, but I do agree that it will take a while, but small inventions like these will definitely help the issue and boost confidence among women.
Screen Shot 2016-09-04 at 5.17.12 pm

Image from: World Economic Forum, 2016, Twitter, viewed 4 September 2016, <;.

This tweet was interesting as it was from Australia, tweeted by The Jobs Agenda. It took me to a link that had many different charts and graphs representing different statistics of gender equality on a global scale. Similar to the second tweet, I feel conflicted reading these as I am unsure whether they have measured the results correctly and fairly.

What I have learnt from this exercise is that:

  1. There are so many more perspectives out there about this issue, and I have definitely had to re-evaluate my own opinion
  2. The Twitter archiver is a great tool for research as it has provided me with more websites to refer back to
  3. The Twitter archiver does pick up spam tweets that will not work on the computer
  4. Perhaps using other words next time would be useful as all of these tweets were links to a website not just a tweet on its own. It would have been interesting to see more regular tweets on gender equality, I only came across a handful in my research.
  5. I could perhaps use these tweets in a data visualisation design, to raise awareness about the gender pay gap in relation to maybe who posted these tweets, or even the differing opinions of the gender pay gap.


Allen, K. 2016, UK women still far adrift on salary and promotion as gender pay gap remains a gulf, the Guardian. viewed 4 September 2016, <;.

Gray, A. 2016, The gender pay gap gets even wider once women have children, World Economic Forum. viewed 4 September 2016, <;.

Huffington Post, 2016, What Gender Pay-Gap Statistics Aren’t Capturing, The Huffington Post. viewed 4 September 2016, <;.

leswolves, 2016, leswolves on Twitter: “Looking for ways to tackle the gender pay gap | Letters: I was interested to read the news on t… #gender #news”, viewed 4 September 2016, <;.

MiVR, 2016, Twitter, viewed 4 September 2016, <;.

Schuller, T. 2016, Looking for ways to tackle the gender pay gap | Letters, the Guardian. viewed 4 September 2016, <;.

Sees, S. 2016, Twitter, viewed 4 September 2016, <;.

Snapp, K. 2016, Katie K Snapp on Twitter: “What #Gender Pay-Gap Statistics Aren’t Capturing #skirtstrategies”, viewed 4 September 2016, <;.

Winkler, E. 2016, Can virtual reality help close the gender pay gap?, Quartz. viewed 4 September 2016, <;.

World Economic Forum, 2016, Twitter, viewed 4 September 2016, <;.

%d bloggers like this: