A Capsule Neural Network (CNN) based Hybrid Approach

1. TECHNOLOGYSUBJECT A Multidisciplinary Open Access Journal Machine Learning I Data Science I Information SecurityTOPIC(S) ISSN: 2995-8067 Article Information Mini Review Submitted: December 08, 2023 Approved: January 11, 2024 Published: January 12, 2024 A Capsule Neural Network (CNN) based Hybrid Approach for Identifying Sarcasm in Reddit Dataset How to cite this article: Faseeh M, Jamil H. A Capsule Neural Network (CNN) based Hybrid Approach for Identifying Sarcasm in Reddit Dataset. IgMin Res. Jan 12, 2024; 2(1): 013- 017. IgMin ID: igmin137; DOI: 10.61927/igmin137; Available at: www.igminresearch.com/articles/pdf/igmin137.pdf Copyright license: © 2024 Faseeh M, et al. Th is is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. Muhammad Faseeh and Harun Jamil* Department of Electronic Engineering, Jeju National University, Jeju-si, Jeju-do, Republic of Korea, 63243 Keywords: Sarcsm; CNN; LSTM; Sentiment; Word2vec; Tf-IDF *Correspondence: Harun Jamil, Department of Electronic Engineering, Jeju National University, Jeju-si, Jeju-do 63243, Republic of Korea, Email: harunjamil@hotmail.com Abstract Sarcasm, a standard social media message, delivers the opposite meaning through irony or teasing. Unfortunately, identifying sarcasm in written text is diffi cult in natural language processing. Th e work aims to create an eff ective sarcasm detection model for social media text data, with possible applications in sentiment analysis, social media analytics, and online reputation management. A hybrid Deep learning strategy is used to construct an eff ective sarcasm detection model for written content on social media networks. Th e design emphasizes feature extraction, selection, and neural network application. Limited research exists on detecting sarcasm in human speech compared to emotion recognition. Th e study recommends using Word2Vec or TF-IDF for feature extraction to address memory and temporal constraints. Use feature selection techniques like PCA or LDA to enhance model performance by selecting relevant features. A Capsule Neural Network (CNN) and Long Short-Term Memory (LSTM) collect contextual information and sequential dependencies in textual material. We evaluate Reddit datasets with labelled sarcasm data using metrics like Accuracy. Our hybrid method gets 95.60% accuracy on Reddit. Introduction convey contradiction, which can lead to unclear communication if misunderstood. Sarcasm can be transmitted through tone of speech, facial expressions, and nonverbal cues, as well as in written forms like emails, texts, and social media posts. In recent times, sarcasm has gained popularity in digital communication due to its ef?iciency in delivering humour or criticism. However, written sarcasm is challenging to detect and can result in miscommunication and misunderstandings. Therefore, sarcasm is a distinct communication method that requires careful interpretation. Applications of sarcasm detection in various domains: Social Media Monitoring, Customer Service, Mental Health, Political Discourse, etc. Sarcasm involves saying something contrary to one’s true meaning, often with a mocking tone and context-dependent cues. It can be challenging to detect in written English. For instance, a statement like “We have nice weather” on a rainy day may appear positive but can be sarcastic due to underlying frustration. Social media, like Twitter, has become famous for expressing thoughts on various topics. Many companies are keen on gauging public sentiment across diverse topics such as movies, songs, politics, and product reviews. In text analysis, it can be challenging to discern the true meaning of a sentence that may appear innocuous but is satirical or biting, like a review of a cof?in being “very comfortable for sleeping.” Automated methods, such as sarcasm identi?ication, have emerged to tackle this issue, with studies like Shubham Rendhalkar, et al. [1] focusing on sentiment analysis. Sentiment analysis helps organizations understand people’s opinions, including politics, services, and individuals. Sarcasm, a standard tool for societal commentary or expressing dissatisfaction, can also be employed for humour, wit, and cleverness. Most sarcasm detection systems today leverage NLP techniques and machine learning models. These techniques usually use sentiment analysis, language patterns, and contextual information to identify sarcastic sentences. However, there are issues with these methods that keep them from being precise. Many current methods effectively identify overt sarcasm but struggle with more complex or context-dependent cases. The complexity and context-dependence of sarcasm make it challenging for existing approaches to produce consistently correct ?indings. Sarcasm is a nuanced and context-dependent form of communication. It relies on correctly interpreting cues such as tone, context, and shared cultural knowledge. Sarcasm’s purpose is to The suggested method uses various machine-learning approaches to set itself apart from current methods. The suggested solution uses 013 www.igminresearch.com

2. DOI: 10.61927/igmin137 2995-8067 ISSN cutting-edge deep learning models to embrace the dynamic nature of sarcasm, in contrast to many existing approaches that rely on prede?ined linguistic features. This methodology uses the power of Word2vec and TF-IDF with the Capsule CNN method to achieve the desired results. Literature review three modalities. They also constructed a multi-modal sarcasm detection dataset based on Twitter. They discovered their approach, and the three modalities were valuable and bene?icial, as seen by dataset evaluation ?indings. AAFAB is an Adversarial and Auxiliary Features-Aware Model (AAFAB) proposed by A. Kumar and Narapareddy [13]. This model captures semantics by capturing sentence meaning with contextual word embeddings from BERT and integrating it with high-quality manually generated auxiliary features for sarcasm detection. MHABiLSTM (Multi-head Attention Bidirectional LSTM) is their proposed model for identifying sarcasm in the SARC Reddit Dataset. Experiment results indicate that combining AAFAB with supplemental features and adversarial training improves performance. Most sarcasm detection algorithms worldwide concentrate on separating utterances or classifying textual data. A limited amount of research is dedicated to altering textual data components to enhance classi?ication accuracy signi?icantly. Various Depp learning techniques are present to detect sarcasm detection in text [2-5]. To improve the accuracy of sarcasm categorization, Mehndiratta, et al. [6] Deep learning models such as CNN and LSTM were utilized, in addition to hybrid approaches such as CNN-LSTM and LSTM- CNN. They used the GloVe and FastText word embedding methods to analyze the SARC Reddit dataset, focusing on the deployment of balanced data. Qiao Nie L, et al. [7] use a Mutual-enhanced incongruity learning network for multi-modal sarcasm detection, one of the best novel techniques for multi-model sarcasm detection. Methodology The dataset used for this task is the Self-Annotated Reddit Corpus (SARC), which contains 1.3 million sarcastic comments from Reddit. The dataset was created by harvesting Reddit comments with the sarcasm tag. Redditors frequently use this tag to indicate that their comment is intended to be regarded in jest and should not be taken seriously, and it is generally a reliable indicator of sarcastic comment content. Data is available in balanced and imbalanced (i.e. accurate distribution) forms. (True ratio is roughly 1:100). We used an 80/20 ratio for training and testing. Usman Naseem, et al. [8] conducted research that studied sentiments expressed on social media platforms and identi?ied irony and sarcasm. Their T-DICE intelligent contextual embedding strategy used transformer-based deep algorithms to deal with noise relevant to the context. Additionally, they used attention-based Bidirectional Long Short-Term Memory to determine sentiment in postings displayed across several platforms. A different group of researchers, Ghosh, et al. [9], used two other models to identify instances of sarcasm. The methodology used in Figure 1 in this work entails several procedures to preprocess the text data and build an effective sarcasm detection model. Preprocessing the Reddit dataset is essential in improving data quality before applying more complex algorithms like Word2Vec and Tf-IDF. Tokenization is the ?irst stage in breaking down text into discrete pieces called tokens, which are often words or sentences. Tokenization is required in the case of the Reddit dataset to convert the raw text into a format that can be easily analyzed. This technique contributes to comprehending the text’s structure, allowing token-level stages such as stop word removal and stemming to be applied. Stop words are common words with no signi?icant meaning and are frequently deleted to focus on the more informative keywords. Examples include the words “the,” “and,” and “is.” During preprocessing, removing stop words reduces noise in the dataset, allowing the model to focus on the relevant material and enhance ef?iciency during subsequent analysis. Lowercasing is the process of turning all text to lowercase. This step guarantees that the text data is consistent, as machine learning models may treat uppercase and lowercase versions of the same word as separate entities. Lowercasing solves this problem by ensuring uniformity in word representation and increasing the model’s overall performance. Stemming is reducing words to their root or basic form by removing suf?ixes. For example, “running” is shortened to “run.” This technique is especially important when different spellings of the same term have the same meaning. By treating related words as the same item, stemming reduces the complexity of the dataset and improves the model’s capacity to generalize. Lemmatization is a more complicated strategy than stemming in that it entails reducing words to their base or root form while considering context and meaning. Instead This research aims to investigate a deep neural network architecture that combines a recurrent neural network (RNN) with a convolutional neural network (CNN) for word-based deep learning inside a particular task embedding. Three different deep learning models, namely RNN, CNN, and Attentive RNN, are investigated in this study. These ?indings prove that Attentive RNN performs extraordinarily well when applied to datasets derived from Twitter. This correlates with the research by Zhang, et al. [10], which involves evaluating several neural network algorithms to identify sarcasm in tweets. In addition, the study’s results investigated the effects of continuous automatic and discrete manual characteristics. More speci?ically, a bi-directional gated recurrent neural network is deployed to gather information about grammatical and semantic aspects. Researchers in this ?ield have used local tweet data and neural networks to extract contextual clues from Twitter history automatically. Their ?indings reveal that adding neural features enhances sarcasm detection accuracy, with different error distributions compared to discrete manual approaches. They’ve also used sophisticated false positive detection techniques [11]. Tay, et al. [12] investigated sarcasm study using LSTM-based intra-attention. They created a system for detecting multi-modal sarcasm in tweets by analyzing text and image data from Twitter. The work proposes a multi-modal hierarchical fusion model that combines information from text, images, and image attributes across January 12, 2024 - Volume 2 Issue 1 TECHNOLOGY 014

3. DOI: 10.61927/igmin137 2995-8067 ISSN Data Preprocessing Feature Extraction Tokenization Stop words Removal Word2Vec Raw Data Normalized Data Lower Casing Reddit Dataset Tf-IDF Stemming Lemmitization Model Building Feature Extraction LSTM Performance Analysis PCA CNN Accuracy LSTM + CNN LDA Capsule + CNN Figure 1: Proposed Capsule Neural Network (CNN) based Hybrid Approach. of stemming, Lemmatization assures that the generated words are legitimate and meaningful. This stage adds to better semantic understanding, essential in applications where the original words must be preserved. is a bag of words after the initial preparation procedures. LDA then identi?ies subjects in the documents likely to generate the observed words. This aids in comprehending the underlying themes present in the dataset and allows for document categorization based on these themes. LDA provides a probabilistic framework for assigning themes to documents and words to topics, making it a valuable tool for exploring the Reddit dataset’s latent structures. After completing these preparation processes, the Reddit dataset suits more complex approaches such as Word2Vec and Tf- IDF. Word2Vec collects semantic associations between words and generates vector representations that re?lect their context. Tf-IDF (Term Frequency-Inverse Document Frequency) applies weights to words based on their frequency in a document and across the entire dataset, allowing essential phrases to be identi?ied while reducing the in?luence of common words. These strategies, when combined, form a robust foundation for deriving valuable insights from the Reddit dataset. By including PCA and LDA after the ?irst preprocessing stages, we optimize the dataset for ef?icient modelling and obtain insights into the underlying structures and topics of the Reddit content. These strategies aid in extracting relevant patterns and allow for a more thorough understanding of the data included in the dataset. PCA’s decreased dimensionality and LDA’s identi?ied topics are valuable inputs for machine learning tasks. In sarcasm detection, my research employed various deep learning architectures, including LSTM, CNN, LSTM + CNN, and Capsule with CNN. Through rigorous experimentation and analysis, it became evident that the Capsule with CNN model outperformed the others in accuracy and effectiveness. Combining the strengths of capsule and convolutional neural networks, this hybrid approach demonstrated superior capabilities in capturing intricate patterns and nuanced contextual information, which are essential for discerning sarcastic nuances in text. The robust performance of the Capsule with CNN model underscores its potential as a powerful tool for sarcasm detection tasks, showcasing the signi?icance of innovative hybrid architectures in natural language processing applications. The comparative evaluation of LSTM, CNN, LSTM + CNN, and Capsule with CNN highlights the advancements in deep learning methodologies and emphasizes the importance of choosing the exemplary architecture for speci?ic tasks, ultimately contributing to the advancement of sentiment analysis and natural language understanding. Following the initial preprocessing phases, using Principal Component Analysis (PCA) and Latent Dirichlet Allocation (LDA) improves the feature representation and gives more profound insights into the Reddit dataset. PCA is a dimensionality reduction approach that converts high-dimensional data into a lower-dimensional space while maintaining the most critical information. After tokenization, top word removal, lowercasing, stemming, and lemmatization, the Reddit dataset may still contain many features (words). PCA aids in dimensionality reduction by identifying the primary components or directions in the data that capture the most variance. This not only expedites subsequent analysis but also helps to visualize the material in a more manageable space. Using PCA may remove redundant information, and the model’s computational ef?iciency can be enhanced without surrendering too much information. LDA is a probabilistic model used for document categorization and topic modelling. LDA can be used to detect hidden topics inside documents in the context of the Reddit dataset. Each document (post or comment) January 12, 2024 - Volume 2 Issue 1 TECHNOLOGY 015

4. DOI: 10.61927/igmin137 2995-8067 ISSN Experimental results We use accuracy as an evaluation matrix for sarcasm prediction using the Reddit dataset. Accuracy is a popular statistic for evaluating classi?ication jobs whose goal is to anticipate the class or category of instances correctly. Accuracy can be used to evaluate the performance of your selected machine learning models after feature extraction and dimensionality reduction in the context of your preprocessing and subsequent analysis of the Reddit dataset. Accuracy is achieved through equation 1. ( ) TP TN Accuracy TP FP TN FN    Various combinations of neural network models have been evaluated for accuracy until the maximum results are achieved, as shown in Table 1 and Figure 3. The ?irst model, a combination of Capsule Networks and Convolutional Neural Networks (CNN), demonstrates a high accuracy of 0.956. This suggests an effective synergy between the Capsule Network’s ability to capture hierarchical relationships and CNN’s prowess in feature extraction. Next, a model employing Long Short-Term Memory (LSTM) networks alone achieves an accuracy of 0.876. This indicates LSTM’s competence in handling sequential data, although it’s slightly less effective than the Capsule + CNN model. The standalone CNN model closely rivals the top performer with an accuracy of 0.955, underscoring   (1) ( ) Figure 3: Accuracy Comparison of Applied Techniques. Confusion Matrix c i t s a c r a S 10500 297 c i t s a c r a S - n o N Table 1: Accuracy comparison of Multiple techniques. Techniques LSTM LSTM+CNN CNN Capsule + CNN 407 4967 Accuracy 0.876 0.894 0.955 0.956 Sarcastic Non-Sarcastic Figure 4: Confusion matrix of Capsule + CNN technique with best results. Figure 2: Th e comparison of the Proposed Capsule Neural Network (CNN) based Hybrid Approach with other Approaches in Terms of Training and Testing Loss. January 12, 2024 - Volume 2 Issue 1 TECHNOLOGY 016

5. DOI: 10.61927/igmin137 2995-8067 ISSN CNN’s robustness in pattern recognition tasks. Lastly, the hybrid LSTM+CNN model, which combines the sequential data processing strength of LSTM with the spatial data processing ability of CNN, achieves an accuracy of 0.894. This blend, while effective, seems to be slightly less ef?icient than the standalone CNN or the Capsule + CNN models in this speci?ic dataset (Figure 2-4). sarcasm detection model to meet current needs and evolve to address emerging challenges in the dynamic realm of online communication. References 1. Shubham R, Chandankhede C. Sarcasm detection of online comments using emotion detection. 2018 International conference on inventive research in computing applications (ICIRCA). IEEE, 2018. Conclusion 2. Vitman O, Kostiuk Y, Sidorov G, Gelbukh A. Sarcasm detection framework using context, emotion, and sentiment features. Expert Systems with Applications. 2023; 234: 121068. The hybrid strategy of merging Capsule Neural Network (CNN) and LSTM models has provided numerous bene?its for text data processing in NLP. The Capsule Neural Network (CNN) supports hierarchical feature representation, collecting local and global text input aspects. The power of Word2vec and Tf-IDF, along with PCA and LDA, shows tremendous results with an accuracy of 0.956. Combining Features extracted from PCA and LDA, Capsule Neural networks can capture complex patterns and representations in text data, leading to increased performance in various NLP tasks such as text classi?ication, sentiment analysis, named entity recognition, etc. The experimental results and literature study indicate that the hybrid strategy can outperform individual models or other traditional approaches in terms of performance. Depending on the dataset and task, it has demonstrated promising outcomes regarding accuracy. The hybrid technique has also proven robustness to varying-length inputs, adaptability to diverse NLP tasks, and, in some situations, the possibility for state-of-the-art performance. 3. Šandor D, Babac MB. Sarcasm detection in online comments using machine learning. Information Discovery and Delivery, (ahead-of-print). 2023. 4. Pandey R, Singh JP. BERT-LSTM model for sarcasm detection in code-mixed social media posts. Journal of Intelligent Information Systems. 2023; 60(1): 235-254. 5. Tan YY, Chow CO, Kanesan J, Chuah JH, Lim Y. Sentiment Analysis and Sarcasm Detection using Deep Multi-Task Learning. Wirel Pers Commun. 2023;129(3):2213-2237. doi: 10.1007/s11277-023-10235-4. Epub 2023 Mar 4. PMID: 36987507; PMCID: PMC9985100. 6. Pulkit M, Soni D. Identifi cation of sarcasm using word embeddings and hyperparameters tuning. Journal of Discrete Mathematical Sciences and Cryptography. 2019; 22.4: 465-489. 7. Qiao Y, Jing L, Song X, Chen X, Zhu L, Nie L. Mutual-enhanced incongruity learning network for multi-modal sarcasm detection. In Proceedings of the AAAI Conference on Artifi cial Intelligence. 2023; 37: 9507-9515. 8. Usman N. Towards improved deep contextual embedding for the identifi cation of irony and sarcasm. 2020 International joint conference on neural networks (IJCNN). IEEE. 2020. In the future, our goal will go beyond the successful online deployment of the sarcasm detection algorithm, as we aim to develop and improve its performance constantly. The inclusion of real-time learning processes is one possible option for future research. We hope to secure the system’s ability to identify sarcasm in the ever- changing landscape of internet communication by integrating adaptive algorithms that can dynamically update the model based on growing language trends and nuances in online conversation. Furthermore, user feedback and participation will be critical in the model’s progress, as we intend to construct a feedback loop that allows for continual improvement based on user input. Furthermore, we intend to expand the model’s applicability to multiple social media sites, adjusting its architecture to each platform’s speci?ic characteristics to give a seamless and successful sarcasm detection experience. Finally, we intend to stay on the cutting edge of advances in natural language processing and machine learning, allowing our 9. Aniruddha G, Veale T. Fracking sarcasm using neural network. Proceedings of the 7th workshop on computational approaches to subjectivity, sentiment, and social media analysis. 2016. 10. Zhang M, Zhang Y, Fu G. Tweet sarcasm detection using deep neural network. In Matsumoto Y., Prasad R. (Eds.), Proceedings of COLING 2016, the 26th international conference on computational linguistics: Technical papers. 2016; 2449–2460. 11. Alexandros PR, Siolas G, Stafylopatis AG. A transformer-based approach to irony and sarcasm detection. Neural Computing and Applications. 2020; 32: 17309-17320. 12. Yi T. Reasoning with sarcasm by reading in-between. arXiv preprint arXiv:1805.02856 (2018). 13. Avinash K. Adversarial and auxiliary features-aware bert for sarcasm detection. Proceedings of the 3rd ACM India Joint International Conference on Data Science Management of Data (8th ACM IKDD CODS 26th COMAD). 2021. How to cite this article: Faseeh M, Jamil H. A Capsule Neural Network (CNN) based Hybrid Approach for Identifying Sarcasm in Reddit Dataset. IgMin Res. Jan 12, 2024; 2(1): 013-017. IgMin ID: igmin137; DOI: 10.61927/igmin137; Available at: www.igminresearch.com/articles/pdf/igmin137.pdf January 12, 2024 - Volume 2 Issue 1 TECHNOLOGY 017

6. Publisher note: Thank you for providing this insightful research study—it’s a valuable asset that will empower us in our future undertakings. INSTRUCTIONS FOR AUTHORS APC In addressing Article Processing Charges (APCs), IgMin Research: recognizes their signi?icance in facilitating open access and global collaboration. The APC structure is designed for affordability and transparency, re?lecting the commitment to breaking ?inancial barriers and making scienti?ic research accessible to all. IgMin Research - A BioMed & Engineering Open Access Journal is a prestigious multidisciplinary journal committed to the advancement of research and knowledge in the expansive domains of Biology, Medicine, and Engineering. With a strong emphasis on scholarly excellence, our journal serves as a platform for scientists, researchers, and scholars to disseminate their groundbreaking ?indings and contribute to the ever-evolving landscape of Biology, Medicine and Engineering disciplines. At IgMin Research - A BioMed & Engineering Open Access Journal, fosters cross-disciplinary communication and collaboration, aiming to address global challenges. Authors gain increased exposure and readership, connecting with researchers from various disciplines. The commitment to open access ensures global availability of published research. Join IgMin Research - A BioMed & Engineering Open Access Journal at the forefront of scienti?ic progress. For support@igminresearch.us. The Copyright Clearance Centre’s Rights link program manages article permission requests via the journal’s website (https://www.igminresearch.com). Inquiries about Rights link can be directed to info@igminresearch.us or by calling +1 (860) 967-3839. book and educational material reviews, send them to IgMin Research, at https://www.igminresearch.com/pages/publish-now/author-guidelines https://www.igminresearch.com/pages/publish-now/apc WHY WITH US IgMin Research |A BioMed & Engineering Open Access Journal employs a rigorous peer-review process, ensuring the publication of high-quality research spanning STEM disciplines. The journal offers a global platform for researchers to share groundbreaking ?indings, promoting scienti?ic advancement. JOURNAL INFORMATION Regularity: Monthly Review Type: Double Blind Publication Time: 14 Days GoogleScholar: https://www.igminresearch.com/gs Plagiarism software: iThenticate Language: English Collecting capability: Worldwide License: Open Access by IgMin Research is licensed under a Creative Commons Attribution 4.0 International License. Based on a work at IgMin Publications Inc. Online Manuscript Submission: https://www.igminresearch.com/submissionor can be mailed to submission@igminresearch.us Journal Full Title: IgMin Research-A BioMed & Engineering Open Access Journal Journal NLM Abbreviation: IgMin Res Journal Website Link: https://www.igminresearch.com Topics Summation: 150 Subject Areas: Biology, Engineering, Medicine and General Science Organized by: IgMin Publications Inc.