Machine Learning & Deep Text Combat Cyberbullying

Tianyi Zhao and Adey Zegeye

Machine Learning –  using “algorithms to get computer systems to go through content (images, text) and identify various trends and patterns across all of those data, based on what we have told them to look for (e.g., training it on labeled data – content that a human has already manually classified in some way – toxic or not toxic, abusive or not abusive). This can actually be done on unlabeled data as well, via what is called unsupervised learning – where the algorithm(s) tries to cluster the data into groups on its own).”

Deep Learning –” subset of machine learning– after the system identifies trends and patterns across data by analyzing content, we ask it to constantly improve its probability of accurately classifying that content by continually training itself on new data that it receives.”

How Machine Learning Can Classify Online Abuse

The different layers might:

1. Extract Curse words (that the programmer has listed as abusive)

2. The second later would count all the curse words up and divide them by the number of words in the text message it appears (to signal severity_

  1. Third layer might look at words in all CAPS
  2. Foruth layer might look at how many hateful words have second-person pronouns meaning they were directed at someone else
  3. Fifth layer might check if this poster has been previously flagged for abusive content 
  4. Sixth layer might look at punctuation (could imply tone)

Additional layers might check for attached images/video and see if that has been classified as abusive before 

DeepText Utilized in Instagram:

DeepText was firstly successful in spam filtering, and then moved to develop the mean comments elimination. Each person in the developing team looks at a comment and determines whether it is appropriate. If it’s not, he sorts it into a category of verboten behavior, like bullying, racism, or sexual harassment. Until launching in 2017, the raters, all of whom are at least bilingual, had analyzed roughly two million comments, and each comment had been rated at least twice. Simultaneously, this system had been testing internally, and the company adjusted the algorithms: selecting and modifying ones that seem to work and discarding ones that do not. The machines give each comment a core between 0 and 1, measuring the comment is offensive or inappropriate. If it is above a certain threshold, the comment is deleted.

The comments are rated based on several factors, semantic analysis of the text, the relationship between the commenter and the poster, and the commenter’s history. The system analyzes the semantics of each sentence, and also took the source into account. A comment from someone that the user does not follow is more likely to be deleted than one from someone the user does. Also, the comment that repeated endless on Martin Garrix’s feed is probably being made by a human.The technology is automatically incorporated into users’ feeds, but it can be turned off as well.


Figure 1. Turn-on/off the Comment Filter in Settings



Pros & Cons


  1. Automating the process of deleting hate speech and offensive comments helps filter out unwanted content on Instagram
  2. DeepText becomes more effective by allowing users to manually enter words or phrases they want blocked


  1. Characters in hateful words are replaced with symbols to avoid detection;
  2. Some comments may not contain any problematic words but still might be incredibly offensive;
  3. Acronyms and Internet slang are changing constantly;
  4. The system may delete innocuous or helpful comments by mistake.

Works Cited:

Systrom, Kevin. “Keeping Instagram a Safe Place for Self-Expression.” Instagram Press, Jun. 29, 2017.

Systrom, Kevin. “Protecting Our Community from Bullying Comments.” Instagram Press, May 1, 2018.

Marr, Bernerd. “The Amazing Ways Instagram Uses BIg Data And Artificial Intelligence.” Forbes, Mar. 16, 2018.

Hinduja, Sameer. “How Machine Learning Can Help Us Combat Online Abuse: A Primer.” Cyberbullying Research Center, Jun. 26, 2017.

Thompson, Nicholas. “Instagram Unleashes an AI System to Blast Away Nasty Comments.” Wired, Jun. 29, 2017.

Bayern, Macy. “How AI Became Instagram’s Weapon of Choice in the War.” Tech Republic, Aug. 14, 2017.