Multimodal Detection of Bots on X (Twitter) using Transformers (2024)

II-A Traditional Machine Learning Algorithms

In [8], the authors proposed two approaches for detecting bots in Twitter both at account and at tweet-level. In terms of the account level classification, the authors extracted a large set of features per user, applied feature selection algorithms, sampling techniques for dealing with the imbalanced datasets, and trained shallow machine learning algorithms.

Similarly, the authors in [10] adopted methods for recognizing bots both at account and tweet level. In terms of the account level, the authors extracted a set of features, including number of statuses (number of tweets and retweets), Followers Count, Friends Count, Favorites Count, Listed Count, Default Profile, etc. Next, the authors combined SMOTE with data enhancement via edited nearest neighbors (SMOTENN) and Tomek Links (SMOTOMEK), and trained traditional machine learning algorithms. In terms of the tweet level classification, the authors introduced a deep neural network consisting of LSTM and dense layers. The authors used as input GloVe embeddings of tweets and tweet metadata. Auxiliary output was also used, whose target was also the classification label.

The authors in [9] extracted a set of style-based features, including the number of punctuation marks, number of hashtags, number of retweets, number of user mentions, number of url links, and many more. They trained the following classification algorithms: Random Forest, Naive Bayes, Logistic Regression, Support Vector Machine with rbf and linear kernel, and Convolution Neural Network.

A wavelet-based approach was introduced by [17]. Specifically, the authors exploited the discrete wavelet transform and extracted a set of features, namely wavelet magnitude, wavelet phase, wavelet domain score, and so on. After this, a feature selection technique, namely a Correlation-based Feature Subset Selection, was proposed for reducing the dimensionality of the feature vector. Finally, a Random Forest Classifier was trained.

The authors in [18] proposed Botometer, which is a publicly available bot detection tool via the website, Python, or REST APIs. The authors use network, user, friends, and temporal features and train a Random Forest Classifier. The only input to this publicly available tool is the screen name.

II-B Deep Learning and Transformer-based Approaches

In [19], the authors exploited three types of LSTMs and four types of features, namely tweet content, tweet metadata, account metadata, and user description. In terms of the tweet content and the user description, the authors exploited the GloVe embeddings. Regarding account metadata, the authors extracted 30 features, while 15 tweet metadata features were also extracted. One limitation of this study is the feature extraction procedure and the usage of GloVe embeddings in conjunction with LSTM models instead of pretrained language models based on transformers, which capture better the context and require significantly less time for fine-tuning.

A similar approach was introduced by [11], where the authors utilized GloVe embeddings for representing the user description field as embeddings. Also, the authors extracted several types of features, namely number of followers and friends, number of tweets and retweets posted by the user, length of the name, entropy of the screen name, entropy of the description, etc. They passed the GloVe embeddings of the user description field into LSTM models and concatenated the output of the LSTMs with the rest of the features.

In [8], the authors proposed two approaches for detecting bots in Twitter both at account and at tweet-level. With regards to the tweet-level classification, the authors utilized a deep neural network. Specifically, the authors exploited GloVe embeddings and passed them through BiLSTM layers coupled with an attention mechanism.

Similarly, the authors in [10] adopted methods for recognizing bots both at account and tweet level. In terms of the tweet level classification, the authors introduced a deep neural network consisting of LSTM and dense layers. The authors used as input GloVe embeddings of tweets and tweet metadata. Auxiliary output was also used, whose target was also the classification label.

A different approach was proposed by [20], where the authors exploited Generative Adversarial Neural networks (GANs) for recognizing bots in Twitter. Specifically, the authors addressed the limitation of original SeqGAN [21] by introducing the GANBOT setting, where a shared LSTM was used between generator and classifer. Similar to [10], the authors used GloVe embeddings as input to LSTMs. GANs were also exploited by [22]. Specifically, the authors extracted first a set of features consisting of user meta-data, sentiment, friends, content, network, and timing. Finally, they proposed a new method based on Conditional GANs by introducing the Wasserstein distance with a gradient penalty and a condition-generation method through the modified density peak clustering algorithm. The authors compared their approaches with Random Oversampling, SMOTE, and ADASYN.

In [23], the authors use language models based on transformers, namely BERT and RoBERTa, for recognizing bots in Twitter. The authors concatenated the text embeddings along with additional metadata and trained a neural network consisting of dense layers.

In [15], the authors proposed a new method for recognizing bots in Twitter based on image recognition. Specifically, they introduced a new approach for converting the digital DNA sequence into an image consisting of three channels. Finally, the authors applied ResNet and stated that this method achieved competitive results to state-of-the-art approaches. However, the authors created a digital DNA sequence based only on the type of the tweets, while they fine-tuned only one pretrained model.

The authors in [24] utilized the concept of digital DNA sequences along with pretrained BERT models. First, the authors trained BERT on sentiment analysis classification tasks. Next, they passed each tweet through a BERT model and predicted the sentiment of each tweet, i.e., positive, negative, and neutral. In this way, they create a digital DNA sequence consisting of characters denoting the sentiment of each tweet.

A deep learning architecture consisting of two branches was proposed by [25]. Specifically, the authors used word2vec and passed the respective embeddings through CNNs. Next, they extracted features, including number of followers, number of tweets, account age/reputation, etc., and passed them through dense layers. After this, they concatenated the two branches and passed the resulting vector through a dense layer for getting the final prediction.

An active learning approach was adopted by [26]. Specifically, the authors extracted a set of features, including metadata-based, interaction-based, content-based, and timing-based. Next, active learning was employed for efficiently expanding the labeled data. Finally, a deep neural network consisting of ResNet, BiGRU, and attention layer was trained.

The study in [27] introduced a CNN and BiLSTM-based deep neural network model coupled with an attention mechanism. Specifically, the authors passed the profile, temporal, and activity information to a two-layers stacked BiLSTM, whereas they passed the contentinformation to a deep CNN. An attention mechanism was exploited at the top of the proposed architecture. Findings showed that the proposed approach outperformed the state-of-the-art approaches.

A CNN-LSTM network was also proposed by [28]. Specifically, the authors model the social behaviour per user, i.e., posting and retweeting, and exploit an LSTM network. Next, the authors consider users’ history tweets as temporal text data and use a CNN-LSTM network. Finally, they fuse the representation vectors obtained by the two networks and get the final prediction.

In [29], the authors extract word embeddings, character embeddings, part-of-speech embeddings, and named-entity embeddings. They pass these embeddings through BiLGRU [30] and get the final prediction. Findings stated that the proposed model performed better or comparably to state-of-the-art Twitter bot detection models.

In [31], the authors used GloVe embeddings as input to a 3-layer BiLSTM neural network for distinguishing Twitter bots from human accounts.

II-C Graph-based Approaches

Recently, graph convolutional networks have been also used for identifying bots in Twitter. Specifically, the study in [12] created an heterogeneous graph and employed Relational Graph Convolutional Networks. The authors represented each user as a vector consisting of user description, tweets, numerical and categorical properties. To be more precise, the authors employed RoBERTa to get a representation vector of the user description. Similarly, they used RoBERTa and averaged the representations of all the tweets posted by the user into one single vector. In terms of the numerical properties, the authors created a vector consisting of the number of friends, followers, favourites, etc. Regarding the categorical properties, the resulting vector was composed of binary variables, i.e., if the user has a profile image, if the user is verified, etc. However, this method cannot incorporate the intrinsic heterogeneity of relation. Therefore, the authors in [32] introduced propose a novel Twitter bot detection framework thatis graph-based and heterogeneity-aware. Specifically, the authors encoded each user adopting the procedure followed in [12]. Finally, the authors exploited relational graph transformers and semantic attention networks.

The authors in [33] proposed a text-graph interaction module along with a semantic consistency module. Specifically, the authors exploited the description and tweets and passed them through RoBERTa for getting the textual representation (embeddings). For graph-based approach, they adopted the method proposed by [12]. Next, they introduced a method for capturing the interactions between the text and graph modality. After that, a semantic consistency detection module was proposed, which exploits the attention weights obtained by the RoBERTa model. Finally, the respective vectors were concatenated and were passed through a dense layer for the final classification.

Multi-view graph attention networks were exploited in [34]. The authors used different datasets for training and evaluating their proposed approaches under a transfer learning scenario. Profile features were also utilized, including number of followers/friends, age in days, and many more.

Motivated by the fact that existing methods almost ignore the differences in bot behaviors in multiple domains, the study in [35] introduces a domain-aware approach for recognizing bots. Specifically, multi-relational graphs were exploited coupled with a user representation learning module, consisting of a series of graph embedding layers and semantic attention layers. Finally, domain-aware classifiers were exploited for detecting bots. Additionally, according to the authors this is the first study employing a federated learning framework.

The authors in [36] introduced a Graph Convolutional Neural Network for exploiting the characteristics of the accounts’ neighbours and identifying bots. Each user is represented by a node consisting of the following feature set: age, favourites_count, statuses_count, account length name, followers_count, and friends_count. Results showed that the proposed approach outperformed the state-of-the-art ones.

II-D Unsupervised Learning

Cresci et al. [14] introduced digital DNA for recognizing bots in Twitter. Specifically, the authors set the tweets of each user into chronological user and created a digital DNA sequence according to the type and content of each tweet. In terms of the type of the tweets, the resulting DNA sequence was composed of A, C, and T, where a "C" indicates a reply, a "T" denotes a retweet, and an "A" denotes a simple tweet. With regards to the content of the tweet, the authors examined whether the tweet contained hashtags, URLs, mentions, media, etc. Finally, the authors calculated similarities between these sequences using the longest common subsequence (LCS) similarity and clustered users based on their similarity scores.

In [37], the authors introduce MulBot, which constitutes an unsupervised bot detector based on multivariate time series. Specifically, the proposed approach consists of the following steps: multidimensionaltemporal features extracted from user timelines, dimensionality reduction using an autoencoder, extraction of statistical global features (optional), concatenation of global features with vectorial features in output from the encoder (optional), and clustering algoritm. Results showed that the proposed approach yielded satisfactory results in both the binary task and the multiclass classification one.

The bot detection task was considered as an anomaly detection task in [38]. The authors utilized 95 one-gram features from tweet text along with user features and modified two stream clustering algorithms, namely StreamKM++ and DenStream.

II-E Reinforcement Learning

A reinforcement learning approach was proposed by [39] for searching the GNN architecture. In this way, the most suitable multi-hop neighborhood and the number oflayers in the GNN architecture are found. An Heterogeneous Information Network was also exploited for modelling the entities and relationships in the social networks. Finally, the authors exploited self-supervised learning approaches. Also, the authors in [40] introduced a deep Q-network architecture by incorporating a Deep Q-Learning (DQL) model. The authors extracted tweet-based, user profile-based, and social graph-based features. In terms of the tweet-based, the authors utilized syntax, semantic, and temporal behaviour features. Regarding the user profile features, the authors employed features pertaining to user behaviour (Posting tweets in several languages, URL/hashtag/mention ratio) and user interactions (Number of active days, Number of retweeted tweets). Finally, they defined social graph-based attributes, such as clustering coefficient, closeness centrality, betweenness centrality and pagerank centrality for each user.

The authors in [41] extracted URL-based features, including URL redirection, frequency of shared URLs, and spam content in URL, and designed a learning automata based model. Specifically, the authors designed a trust computation model, which contains two parameters, namely direct trust and indirect trust. Findings suggested that the proposed approach improved the existing ones.

II-F Related Work Review Findings

In spite of the rise of deep learning algorithms, existing research initiatives extract still a large number of features and train traditional machine learning algorithms. However, feature extraction constitutes a tedious procedure demanding domain expertise. Thus, the optimal set of features may not be found. Additionally, existing research initiatives still exploit GloVe embeddings and train LSTMs instead of employing language models based on transformers, which achieve state-of-the-art results across a large number of domains. In addition, methods employing BERT cannot be employed for languages other than the english one. Recently, there have been introduced graph-based approaches. However, these methods still extract a large number of features per user for creating a feature vector per node-user. At the same time, large graphs are created, which require access to computational resources and need a lot of time for training.

Therefore, our proposed work differs significantly from the aforementioned research works, since we (1) exploit only the user description field and create images according to the type and content of the tweets posted by each user, (2) create images consisting of three channels and train several pretrained models comparing their performances, (3) employ TwHIN-BERT which is a new multi-lingual Tweet language model that is trained on 7 billion Tweets from over 100 distinct languages, (4) introduce multimodal models, which take as input the user description field and the 3d images.

III-A Cresci’17 Dataset

We use the dataset introduced in [42] for conducting our experiments. Specifically, this dataset consists of genuine accounts and some sets of social and traditional spambots. In the present study, we exploit the set of genuine accounts and the set of social spambots #1. For collecting genuine users, the authors adopted the methodology proposed in [43]. Specifically, the authors contacted with some accounts and asked them a question. Then, the authors examined the users, who replied to the question and verified 3,474 accounts. In terms of the social spambots users, the authors used a group of social bots that was discovered on Twitter during the last Mayoral election in Rome, in 2014. Specifically, this set of social spambots corresponds to retweeters of an Italian political candidate. As mentioned in SectionII-D, the authors presented a figure illustrating the LCS curves of humans and bots. Findings showed that the LCS of bots were long, even when the number of account increases. A sudden dropin LCS length of bots is observed when the number of accounts gets close to the group size. On the contrary, the LCS curve of humans indicates little to no similarity. Also, in [42], the authors illustrate the cumulative distribution function (CDF) of join date and number of followers. Results indicate that social spambots have anomalous distributions. In our experiments, we keep users having available the user description field in their profile. To address the issue of the imbalanced dataset, we create a dataset consisting of 943 real users and 943 social spambots. This technique, which addresses the issue of imbalanced dataset by downsampling the set of real users, has been adopted by previous studies [15, 37].

III-B TwiBot-20 Dataset

We use the TwiBot-20 dataset to conduct our experiments [44]. Contrary to other datasets which include a specific type of users, this dataset consists of diversified bots that co-exist on the real-world Twittersphere. Users of this dataset have an interest in four domains, including politics, business, entertainment, and sports. Thus, this dataset constitutes a challenging task, since it includes multiple types of bots. For proving the high quality of annotations, the authors illustrate the CDF of account reputation (number of friends and followers), user tweet counts, and screen name likelihood. CDF plots show that genuine accounts present higher reputation scores than bots in the TwiBot-20 dataset. In terms of user tweet counts, CDF plots indicate that bots generate fewer tweets than humans, in order to avoid the traditional detection methods. Finally, findings of CDF plot regarding the screen name likelihood show that bot users in TwiBot-20 do have slightly lower screen name likelihood. Also, the authors in [44] conduct a user diversity analysis and study the distribution of profile locations and user interests. Findings of a geographic analysis study demonstrate that India, United States, Europe, and Africa are represented in the TwiBot-20 dataset. Finally, the authors examine the most frequent hashtags, present their analysis through a bar plot, and show that twitter users in TwiBot-20 are proved to be diversified in interest domains.TwiBot-20 dataset includes 229,573 users. However, the authors have labelled 11,826 users. This dataset includes the 200 recent tweets per user.The authors in [44] have divided TwiBot-20 dataset into a train, validation, and test set. We remove users who have not posted any tweets and have not added a description field in their profile. TableI presents the distribution of genuine accounts and bots into each set.

IV-A Experiments

V-A Concatenation

In terms of the textual modality, we pass the user description field through a TwHIN-BERT model [16] and extract the [CLS] token. Let $f^{t}\in\mathbb{R}^{d_{t}}$ denote the representation vector of the textual modality. $d_{t}$ denotes the dimensionality and is equal to 768. In terms of the visual modality, we create a 3d image as described in SectionIV. Then, we exploit our best performing model, namely VGG16 for Cresci’17 dataset and AlexNet for TwiBot-20 dataset, as mentioned in SectionIV-A3. Specifically, we remove the last layer of VGG16 (and AlexNet) and replace the next-to-last dense layer consisting of 4096 units with one dense layer consisting of 768 units. Let $f^{v}\in\mathbb{R}^{d_{v}}$ represent the visual representation vector. $d_{v}$ is equal to 768.

After having calculated $f^{t}$ and $f^{v}$, we concatenate these two representation vectors into a single vector as following:

After this, we pass $z$ through a dense layer consisting of 128 units with a ReLU activation function. Finally, we use a dense layer consisting of two units, which gives the final prediction.

Our introduced model is illustrated in Fig.1(a).

V-B Gated Multimodal Unit

In terms of the textual modality, we pass the user description field through a TwHIN-BERT model and extract the [CLS] token. Let $f^{t}\in\mathbb{R}^{d_{t}}$ denote the representation vector of the textual modality. $d_{t}$ denotes the dimensionality and is equal to 768. In terms of the visual modality, we create a 3d image as described in SectionIV. Then, we exploit our best performing model, namely VGG16 for Cresci’17 dataset and AlexNet for TwiBot-20 dataset, as mentioned in SectionIV-A3. Specifically, we remove the last layer of VGG16 (and AlexNet) and replace the next-to-last dense layer consisting of 4096 units with one dense layer consisting of 768 units. Let $f^{v}\in\mathbb{R}^{d_{v}}$ represent the visual representation vector. $d_{v}$ is equal to 768.

Next, we use a gated multimodal unit introduced in [56] for controlling the information flow of the textual and visual modalities. The equations governing the gated multimodal unit are presented below:

Finally, we pass $h$ through a dense layer consisting of two units for getting the final prediction.

Our introduced model is illustrated in Fig.1(b).

V-C Crossmodal Attention

In terms of the textual modality, we pass the user description field through a TwHIN-BERT model. Let $f^{t}\in\mathbb{R}^{N\times d_{t}}$ denote the textual representation. $N$ indicates the sequence length, while $d_{t}$ denotes the dimensionality and is equal to 768. In terms of the visual modality, we create a 3d image as described in SectionIV. Next:

Motivated by [57, 58, 59], we exploit two crossmodal attention layers, i.e., one from textual $f^{t}$ to visual features $f^{v}$ and one from visual to textual features.

Specifically, we calculate the scaled dot attention [60] as follows:

Similarly, we design another scaled dot attention layer, where the visual modality corresponds to the query (Q), while the textual modality corresponds to the key (K) and value (V). Let $y$ be the output of the scaled dot attention layer.

Next, we concatenate $z$ and $y$ and pass the resulting matrix into a global average pooling layer.

Finally, we exploit a dense layer consisting of two units for getting the final output.

Our introduced model is illustrated in Fig.1(c).

VI-A Baselines

VI-B Experimental Setup

All the details are provided in SectionIV-A1. We use the TwHIN-BERT-base version from the Transformers library in Python [63].

VI-C Evaluation Metrics

We use the evaluation metrics, which are described in SectionIV-A2.

VII-A Cresci’17 Dataset

The results of our proposed approaches described in SectionV are reported in TableVI.

Regarding our proposed approaches (transformer-based models based on the type of tweets), one can observe that TwHIN-BERT + VGG16 (Cross-Modal Attention) constitutes the best performing model achieving an Accuracy of 99.89%, a Precision of 100.00%, a Recall of 99.79%, a F1-score of 99.90%, and a Specificity of 100.00%. It outperforms the other introduced models in Accuracy by 0.10-0.31%, in Precision by 0.21-0.22%, in F1-score by 0.11-0.34%, and in Specificity by 0.20-0.22%. Also, it outperforms both TwHIN-BERT and VGG16 (type of tweet). Specifically, it surpasses TwHIN-BERT in Precision by 0.41%, in Recall by 0.61%, in F1-score by 0.52%, in Accuracy by 0.52%, and in Specificity by 0.44%. It outperforms also VGG16 (type of tweet) in Precision by 0.22%, in Recall by 0.24%, in F1-score by 0.23%, in Accuracy by 0.21%%, and in Specificity by 0.20%. Additionally, TwHIN-BERT + VGG16 (GMU) outperforms TwHIN-BERT + VGG16 (Concatenation) in Precision by 0.01%, in Recall by 0.45%, in F1-score by 0.23%, and in Accuracy by 0.22%. Also, it outperforms both TwHIN-BERT and VGG16 (type of tweets) in Accuracy by 0.42% and 0.11% respectively. In terms of TwHIN-BERT + VGG16 (Concatenation), one can observe that this model obtains worse performance than VGG16 (type of tweets). Specifically, VGG16 (type of tweets) outperforms TwHIN-BERT (Concatenation) in Recall by 0.21%, in F1-score by 0.11%, and in Accuracy by 0.10%. We speculate that this decrease in performance is attributable to the concatenation operation, which assigns equal importance to each modality ignoring the inherent correlations between the two modalities. Overall, we believe that TwHIN-BERT + VGG16 (Cross-Modal Attention) obtains better performance than the other two multimodal models, since it captures the crossmodal interactions. On the other hand, the Gated Multimodal Unit controls the information flow from each modality, while the concatenation operation neglects the inherent correlations.

In terms of our proposed transformer-based models (images based on the content of tweets), one can observe that TwHIN-BERT + VGG16 (Cross-Modal Attention) constitutes the best performing model outperforming both TwHIN-BERT + VGG16 (GMU) and TwHIN-BERT + VGG16 (Concatenation). Specifically, TwHIN-BERT + VGG16 (Cross-Modal Attention) outperforms TwHIN-BERT + VGG16 (GMU) in Recall by 0.42%, in F1-score by 0.19%, and in Accuracy by 0.19%. Although equal Precision scores are achieved, TwHIN-BERT + VGG16 (Cross-Modal Attention) yields a better F1-score, which constitutes the weighted average of Precision and Recall. Additionally, TwHIN-BERT + VGG16 (Cross-Modal Attention) outperforms TwHIN-BERT + VGG16 (Concatenation) in Precision by 0.23%, in Recall by 0.19%, in F1-score by 0.21%, in Accuracy by 0.19%, and in Specificity by 0.20%. In comparison with unimodal approaches employing either text or images, we observe that TwHIN-BERT + VGG16 (Crossmodal Attention) outperforms both TwHIN-BERT and VGG16 (content of tweets). Finally, TwHIN-BERT + VGG16 (Cross-Modal Attention) with images based on the content of tweets outperforms all the introduced models exploiting images based on the type of tweets. Therefore, TwHIN-BERT + VGG16 (Cross-Modal Attention) with images based on the content of tweets constitutes our best performing model.

In comparison with the state-of-the-art approaches, one can observe that our best performing model, namely TwHIN-BERT + VGG16 (Cross-Modal Attention) with images based on the content of the tweets, outperforms these approaches in Precision by 0.50-22.34% (except for DeepSBD and DBDM), in Recall by 0.46-6.96%, in F1-score by 0.17-24.35%, in Accuracy by 0.15-21.78%, and in Specificity by 1.90-5.50%. Similarly, TwHIN-BERT + VGG16 (Cross-Modal Attention) with images based on the type of tweets, surpasses the existing research initiatives in Precision by 0.50-22.34% (except for DeepSBD and DBDM), in Recall by 0.29-6.79%, in F1-score by 0.09-24.27%, in Accuracy by 0.06-21.69%, and in Specificity by 1.90-5.50%. Although our best performing model outperforms DeepSBD by a small margin of 0.17% and 0.15% in F1-score and Accuracy respectively, our best performing model has multiple advantages over DeepSBD. Firstly, DeepSBD processes the tweets of each user, extracts GloVe embeddings, and creates a 3d matrix which is given as input to a 6-layer CNN increasing the computational demands. Additionally, this method extracts a set of features per user. On the contrary, our method seems to be simpler and more effective, since it does not rely on a feature extraction strategy; it relies only on the user description and the sequence of actions performed by the user. Additionally, the authors use GloVe embeddings instead of using a language model based on transformers. Thus, this approach inherits the limitations of the GloVe embeddings. In terms of the training time per epoch, 120 to 180 seconds are required for training by our model, while the training time of DeepSBD ranges from 248 to 634 seconds.

VII-B TwiBot-20 Dataset

The results of our proposed approaches described in SectionV are reported in TableVII.

In terms of the proposed transformer-based models using images based on the type of tweets posted by the users, one can observe that TwHIN - BERT + AlexNet (Cross-Modal Attention) constitutes the best performing model in terms of Recall, F1-score, and Accuracy. Specifically, it outperforms the other introduced multimodal models in Recall by 2.55-2.85%, in F1-score by 0.83-1.26%, and in Accuracy by 0.28-0.76%. Additionally, TwHIN-BERT + AlexNet (Cross-Modal Attention) with images based on the type of tweets, surpasses the performance achieved by unimodal models, i.e., TwHIN-BERT (using user description) and AlexNet (using images). Similar differences in models’ performance are observed, when the images based on the content of tweets are used as input to proposed approaches. To be more precise, the usage of the crossmodal attention as a fusion method yields the best results surpassing the performance achieved by concatenation and gated multimodal unit as fusion approaches. TwHIN-BERT + AlexNet (Cross-modal Attention) outperforms the other multimodal models in Recall, F1-score, and Accuracy by 3.16-4.43%, 1.26-2.03%, and 0.70-1.36% respectively. TwHIN-BERT + AlexNet (Cross-Modal Attention) outperforms also the unimodal models employing either text or images. Specifically, it surpasses TwHIN-BERT in F1-score and Accuracy by 3.18% and 3.25% respectively, while it also outperforms AlexNet in F1-score and Accuracy by 9.50% and 7.47% respectively. Overall, we observe that the usage of multiple modalities improves bots’ detection performance. Cross-modal Attention boosts performance, since it models the cross-modal interactions. On the other hand, the gated multimodal unit refers to a weighting strategy, which controls the contribution of each modality to the bots’ prediction task. The concatenation operation is not capable of capturing the interactions of the two modalities or controlling the contribution of each modality, since equal importance to text and image information is assigned.

In comparison with state-of-the-art approaches, our best performing model outperforms these approaches in Accuracy and F1-score by 3.40-26.73% and 0.97-65.58% respectively. We observe that although Botometer was capable of identifying the social spambots in Cresci’17 dataset, the performance drops significantly in TwiBot-20 dataset, which verifies the fact that bots in Twitter have changed their behavior and have managed to evade previous detection methods. Similarly, we observe that the other approaches employing feature extraction techniques fail to detect bots. On the contrary, our approach relying only on the user description and the sequence of actions performed by the user based on the content of tweets seems to be an effective method.

IX-A Limitations

This study has some limitations. Firstly, we did not apply explainability approaches to explain the predictions made by our proposed approaches. Additionally, this study requires labelled data. On the contrary, self-supervised learning approaches address the issue of labels’ scarcity.

IX-B Scalability and Real-time detection

An often neglected concern among researchers introducing methods for identifying bots in Twitter is the reliance on the platform APIs [64]. Specifically, data must be retrieved from social media platforms, so as to ensure that the researchers perform the necessary experiments.

However, in 2023, Twitter decided to end free access to their APIs. As a result of this modification, several research projects³³3https://www.reuters.com/technology/elon-musks-x-restructuring-curtails-disinformation-research-spurs-legal-fears-2023-11-06/ have been cancelled or suspended for a period of time. After that, legal obligations in the bloc’s Digital Services Act (DSA)⁴⁴4https://techcrunch.com/2023/11/17/change-in-xs-terms-indicate-eu-researchers-will-get-api-access/ require larger platforms, including Meta, Twitter, and Google, to provide data access to external researchers doing public interest research on systemic risks. However, the access to Twitter APIs still remains uncertain.

Modifications also to the API affect the bot detection algorithms. For example, the field geo_enabled was removed in 2019 due to privacy reasons. Therefore, algorithms trained with this features, should be modified.

Data access is inextricably linked with scalability. Scalability refers to the analysis of streaming data with limited computing resources [65]. Specifically, the speed at which a method processes a group of accounts depends highly on the possibility for access to the Twitter API. For instance, although the construction of large graphs leads to detection of bots with increasing evaluation performance, fetching such information is not feasible. The study in [66] claims that the following methods are required for improving the scalability of bot detection algorithms:

IX-C Generalization of our approach on other social media

The introduced approaches in this paper can be generalized on other social media, including Facebook and Reddit. Similar to Twitter, users on the Facebook platform make posts, reposts, and reply to other users. At the same time, their posts may include mentions to other users, hashtags, and URLs. Additionally, each user may include a description field in his/her profile.

However, platforms’ policy about data access is critical. For instance, investigating bot detection approaches on the Facebook platform is a very difficult task due to the unwillingness of Facebook to share individual account data [64].