Filings made by Elon Musk’s legal team in his battle with Twitter have been questioned by leading bot researchers, the BBC reports.
Botometer – an online tool that tracks spam and fake accounts – was used by Musk in a countersuit against Twitter.
Using the tool, Musk’s team estimated that 33% of “visible accounts” on the social media platform were “false or spam accounts.”
However, Botometer creator Kaicheng Yang said the figure “doesn’t mean anything.”
Yang questioned the methodology used by Musk’s team, and told the BBC they had not approached him before using the tool.
Musk is in dispute with Twitter, after trying to pull out of a deal to purchase the company for $44 billion.
A court case is due in October in Delaware, at which a judge will rule on whether Musk will have to buy Twitter.
In July, Musk said he no longer wished to purchase the company, as he could not verify how many humans were on the platform.
Since then, the world’s richest person has claimed repeatedly that fake and spam accounts could be many times higher than stated by Twitter.
In his countersuit, made public on Aug. 5, he claimed a third of visible Twitter accounts, assessed by his team, were fake. Using that figure the team estimated that a minimum of 10% of daily active users are bots.
Twitter says it estimates that fewer than 5% of its daily active users are bot accounts.
Botometer is a tool that uses several indicators, like when and how often an account tweets and the content of the posts, to create a bot “score” out of five.
A score of zero indicates a Twitter account is unlikely to be a bot, and a five suggests that it is unlikely to be a human.
However, researchers say the tool does not give a definitive answer as to whether or not an account is a bot.
“In order to estimate the prevalence [of bots] you need to choose a threshold to cut the score,” says Yang. “If you change the threshold from a three to a two then you will get more bots and less human. So how to choose this threshold is key to the answer of how many bots there are on the platform.”
Yang says Musk’s countersuit does not explain what threshold it used to reach its 33% number.
“[The countersuit] doesn’t make the details clear, so [Musk] has the freedom to do whatever he wants. So the number to me, it doesn’t mean anything,” he said.
In Musk’s countersuit, his team says: “The Musk Parties’ analysis has been constrained due to the limited data that Twitter has provided and limited time in which to analyse that incomplete data.”
Botometer was set up by the University of Indiana’s Observatory on Social Media.
Clayton Davis, a data scientist who worked on the project, says the system uses machine learning, and factors such as tweet regularity and linguistic variability, as well as other telltale signs of robotic behaviour.
“Humans behave in a particular way. If an account exhibits enough behaviour that is not like how humans do things, then maybe it’s not human,” he says.
The researchers behind Botometer have tried to calculate how many spam and fake accounts are on Twitter in the past.
In 2017, the group of academics behind the tool published a paper that estimated that between 9% and 15% of active Twitter accounts were bots.
However, Davis says the report was heavily caveated and reliant on limited data.
“The only person who has a God’s eye view is Twitter,” Davis says.
Twitter says it calculates the number of fake accounts through mainly human review. It says it picks out thousands of accounts at random each quarter and looks for bot activity.
Unlike other public bot analysis tools, Twitter says it also uses private data – such as IP addresses, phone numbers, and geolocation – to analyse whether an account is real or fake.
When it comes to Botometer, Twitter argues its approach is “extremely limited.”
It gives the example of a Twitter account with no photo or location given – red flags to a public bot detector. However, the owner of the account may be someone with strong feelings about privacy.
Unsurprisingly, Twitter says its way is the best system to evaluate how many fake accounts exist.
Michael Kearney, creator of Tweet Bot or Not, another public tool for assessing bots, told the BBC the number of spam and fake accounts on Twitter is partly down to definition.
“Depending on how you define a bot, you could have anywhere from less than 1% to 20%,” he says. “I think a strict definition would be a fairly low number. You have to factor in things like bot accounts that do exist, tweet at much higher volumes,” he said.
There is no universally agreed upon definition of a bot. For example, is a Twitter account that tweets out automated tweets – but is operated by a human – a bot?
Fake accounts are often run by humans, while accounts such as weather bots are actively encouraged on Twitter.
Despite this definitional problem, Twitter says it detects and deletes more than a million bot accounts every day using automated tools.
But its systems do not catch them all, and Twitter accepts that millions of accounts slip through. However, it says they make up a relatively small proportion of its 217 million daily active users.
