fix tags

Author: mdrafiqulrabin

_publications/rabin2019testing.markdown

@@ -7,6 +7,6 @@ year: 2019
 additional_links:
    - {name: "ArXiV", url: "https://arxiv.org/abs/1908.10711"}
    - {name: "code", url: "https://github.com/mdrafiqulrabin/tnpa-framework"}
-tags: ["testing", "robustness", "transformation"]
+tags: ["evaluation", "refactoring"]
 ---
 Deep neural networks have been increasingly used in software engineering and program analysis tasks. They usually take a program and make some predictions about it, e.g., bug prediction. We call these models neural program analyzers. The reliability of neural programs can impact the reliability of the encompassing analyses. In this paper, we describe our ongoing efforts to develop effective techniques for testing neural programs. We discuss the challenges involved in developing such tools and our future plans. In our preliminary experiment on a neural model recently proposed in the literature, we found that the model is very brittle, and simple perturbations in the input can cause the model to make mistakes in its prediction.

_publications/rabin2020demystifying.markdown

@@ -2,11 +2,11 @@
 layout: publication
 title: "Towards Demystifying Dimensions of Source Code Embeddings"
 authors: Md Rafiqul Islam Rabin, Arjun Mukherjee, Omprakash Gnawali, Mohammad Amin Alipour
-conference: RL+SE&PL (Co-located with ESEC/FSE)
+conference: "RL+SE&PL (Co-located with ESEC/FSE)"
 year: 2020
 additional_links:
    - {name: "ArXiV", url: "https://arxiv.org/abs/2008.13064"}
    - {name: "code", url: "https://github.com/mdrafiqulrabin/handcrafted-embeddings"}
-tags: ["understanding", "embeddings", "features"]
+tags: ["evaluation", "representation", "naming", "interpretability"]
 ---
 Source code representations are key in applying machine learning techniques for processing and analyzing programs. A popular approach in representing source code is neural source code embeddings that represents programs with high-dimensional vectors computed by training deep neural networks on a large volume of programs. Although successful, there is little known about the contents of these vectors and their characteristics. In this paper, we present our preliminary results towards better understanding the contents of code2vec neural source code embeddings. In particular, in a small case study, we use the code2vec embeddings to create binary SVM classifiers and compare their performance with the handcrafted features. Our results suggest that the handcrafted features can perform very close to the highly-dimensional code2vec embeddings, and the information gains are more evenly distributed in the code2vec embeddings compared to the handcrafted features. We also find that the code2vec embeddings are more resilient to the removal of dimensions with low information gains than the handcrafted features. We hope our results serve a stepping stone toward principled analysis and evaluation of these code representations.

_publications/rabin2021generalizability.markdown

@@ -7,6 +7,6 @@ year: 2021
 additional_links:
    - {name: "ArXiV", url: "https://arxiv.org/abs/2008.01566"}
    - {name: "code", url: "https://github.com/mdrafiqulrabin/tnpa-generalizability"}
-tags: ["adversarial", "generalizability", "robustness", "transformation", "AST", "GNN"]
+tags: ["evaluation", "adversarial", "generalizability", "refactoring", "summarization"]
 ---
 With the prevalence of publicly available source code repositories to train deep neural network models, neural program models can do well in source code analysis tasks such as predicting method names in given programs that cannot be easily done by traditional program analysis techniques. Although such neural program models have been tested on various existing datasets, the extent to which they generalize to unforeseen source code is largely unknown. Since it is very challenging to test neural program models on all unforeseen programs, in this paper, we propose to evaluate the generalizability of neural program models with respect to semantic-preserving transformations: a generalizable neural program model should perform equally well on programs that are of the same semantics but of different lexical appearances and syntactical structures. We compare the results of various neural program models for the method name prediction task on programs before and after automated semantic-preserving transformations. We use three Java datasets of different sizes and three state-of-the-art neural network models for code, namely code2vec, code2seq, and GGNN, to build nine such neural program models for evaluation. Our results show that even with small semantically preserving changes to the programs, these neural program models often fail to generalize their performance. Our results also suggest that neural program models based on data and control dependencies in programs generalize better than neural program models based only on abstract syntax trees. On the positive side, we observe that as the size of the training dataset grows and diversifies the generalizability of correct predictions produced by the neural program models can be improved too. Our results on the generalizability of neural program models provide insights to measure their limitations and provide a stepping stone for their improvement.

_publications/rabin2021understanding.markdown

@@ -7,6 +7,6 @@ year: 2021
 additional_links:
    - {name: "ArXiV", url: "https://arxiv.org/abs/2106.03353"}
    - {name: "code", url: "https://github.com/mdrafiqulrabin/SIVAND"}
-tags: ["understanding", "explainability", "interpretability", "transparency", "simplification", "reduction", "delta debugging", "attention", "features"]
+tags: ["interpretability", "refactoring", "information extraction"]
 ---
 A wide range of code intelligence (CI) tools, powered by deep neural networks, have been developed recently to improve programming productivity and perform program analysis. To reliably use such tools, developers often need to reason about the behavior of the underlying models and the factors that affect them. This is especially challenging for tools backed by deep neural networks. Various methods have tried to reduce this opacity in the vein of "transparent/interpretable-AI". However, these approaches are often specific to a particular set of network architectures, even requiring access to the network's parameters. This makes them difficult to use for the average programmer, which hinders the reliable adoption of neural CI systems. In this paper, we propose a simple, model-agnostic approach to identify critical input features for models in CI systems, by drawing on software debugging research, specifically delta debugging. Our approach, SIVAND, uses simplification techniques that reduce the size of input programs of a CI model while preserving the predictions of the model. We show that this approach yields remarkably small outputs and is broadly applicable across many model architectures and problem domains. We find that the models in our experiments often rely heavily on just a few syntactic features in input programs. We believe that SIVAND's extracted features may help understand neural CI systems' predictions and learned behavior.

_publications/rabin2022memorization.markdown

@@ -7,6 +7,6 @@ year: 2022
 additional_links:
    - {name: "ArXiV", url: "https://arxiv.org/abs/2106.08704"}
    - {name: "code", url: "https://github.com/mdrafiqulrabin/CI-Memorization"}
-tags: ["memorization", "generalization", "noise", "capacity", "language model"]
+tags: ["evaluation", "memorization", "generalizability", "refactoring", "language model"]
 ---
 Deep Neural Networks (DNNs) are increasingly being used in software engineering and code intelligence tasks. These are powerful tools that are capable of learning highly generalizable patterns from large datasets through millions of parameters. At the same time, their large capacity can render them prone to memorizing data points. Recent work suggests that the memorization risk manifests especially strongly when the training dataset is noisy, involving many ambiguous or questionable samples, and memorization is the only recourse. The goal of this paper is to evaluate and compare the extent of memorization and generalization in neural code intelligence models. It aims to provide insights on how memorization may impact the learning behavior of neural models in code intelligence systems. To observe the extent of memorization in models, we add random noise to the original training dataset and use various metrics to quantify the impact of noise on various aspects of training and testing. We evaluate several state-of-the-art neural code intelligence models and benchmarks based on Java, Python, and Ruby codebases. Our results highlight important risks: millions of trainable parameters allow the neural networks to memorize anything, including noisy data, and provide a false sense of generalization. We observed all models manifest some forms of memorization. This can be potentially troublesome in most code intelligence tasks where they rely on rather noise-prone and repetitive data sources, such as code from GitHub. To the best of our knowledge, we provide the first study to quantify memorization effects in the domain of software engineering and code intelligence systems. This work raises awareness and provides new insights into important issues of training neural models in code intelligence systems that are usually overlooked by software engineering researchers.

_publications/rabin2022understanding.markdown

@@ -7,6 +7,6 @@ year: 2022
 additional_links:
    - {name: "ArXiV", url: "https://arxiv.org/abs/2205.14374"}
    - {name: "code", url: "https://github.com/mdrafiqulrabin/ci-dd-perses"}
-tags: ["understanding", "explainability", "interpretability", "transparency", "simplification", "reduction", "delta debugging", "perses", "features", "adversarial"]
+tags: ["interpretability", "refactoring", "adversarial"]
 ---
 Neural code intelligence (CI) models are opaque black-boxes and offer little insight on the features they use in making predictions. This opacity may lead to distrust in their prediction and hamper their wider adoption in safety-critical applications. Recently, input program reduction techniques have been proposed to identify key features in the input programs to improve the transparency of CI models. However, this approach is syntax-unaware and does not consider the grammar of the programming language. In this paper, we apply a syntax-guided program reduction technique that considers the grammar of the input programs during reduction. Our experiments on multiple models across different types of input programs show that the syntax-guided program reduction technique is faster and provides smaller sets of key tokens in reduced programs. We also show that the key tokens could be used in generating adversarial examples for up to 65% of the input programs.