Merge branch 'main' of github.com:Awesome-guys-in-ECE1747/ECE1512_202…

…2W_ProjectRepo_J.Xu_and_W.Xu into main

ProjectA/Report/ProjectA.tex

@@ -830,27 +830,25 @@ \subsection{Question 2}
 %implement 2d capable functions.
 In this section, we describe the heatmap (explanation map) of attribution methods application in HMT. As mentioned before, HMT is a 2-Dimensional based dataset, and therefore we need to either develop a capable XAI methods taking both 3-Dimensional and 4-dimensional tensor corresponding to 1-D and 2-D dataset, or we set up separate methods dealing with 2-Dimensional dataset, HMT.
 
-There are three cases we'd like to discuss in the paper, all the cases are under tumor category, but they indicates the feature of the attribution methods and the dataset itself significantly.
+There are three cases we'd like to discuss in the paper, all the cases are under tumor category, but they indicates the feature of the attribution methods and the dataset itself significantly. As there's significant difference subjectively within a category, there's no wonder why all the XAI methods don't have as good performance as they do in MNIST-1D, which will be shown in the next section. 
 
-%Another modification needs to be mentioned is that there is a slightly modification practically in the implementation. It is modified due to troubleshooting of HMT explanation. The origin function gives an extremely unsatisfying result thus we change the way we calculate secondary gradient and third gradient. We use multiplication instead of slope to get the gradient value due to its low magnitude, the absolute value of both results are similar, however the multiplication is positive constantly.\par
-%There are some potential reasons we could come up with. First,the Grad-CAM++ are suitable for positive gradients however in both datasets, negative values take most places and therefore it it limited to perform well. Secondly, it is mentioned in the paper that Grad-CAM++ has some guided or modified variants but Our implementation works for the basic explanation. So it will be highly influence by the noise and unpredictable factors. Thirdly, among all the CAM methods, a number of the approaches choose Grad-CAM as a benchmark, so the advance Grad-CAM may also suffer from some problems which cannot adapt in multiple networks.\par
-%Subjectively, it could be clearly seen in Fig that ablation-CAM has higher contrast feature heat.
+The first case is case 8, which has confidence score of 0.93915. It has several dark spots stand for the tumor cell in the left-top side of the graph.
 
-The first case is case 8, which has confidence score of 0.93915.
 \begin{figure}[h] 
     \centering
     \includegraphics[width=0.5\textwidth]{./pics/HMT/Tumor-Tumor-0.93915-output-8.png}
     \caption{Explaination Map of case 8, confidence score=0.93915} 
 \end{figure}
-The second case is case 20, which has confidence score of 0.83804.
+
+The second case is case 20, which has confidence score of 0.83804. It is significantly influenced by the bright part all across the tissue graph.
 
 \begin{figure}[h] 
     \centering
     \includegraphics[width=0.5\textwidth]{./pics/HMT/Tumor-Tumor-0.83804-output-20.png}
     \caption{Explaination Map of case 20, confidence score=0.83804} 
 \end{figure}
 
-The third case is case 30, which has confidence score of 0.99755.
+The third case is case 30, which has confidence score of 0.99755. The sample expose the dark part of tumor cells most, and it received the best confidence score, as best performance.
 
 \begin{figure}[h] 
     \centering
@@ -937,12 +935,12 @@ \section{Task 4: Quantitative evaluation of the attribution methods}
 \end{table}
 
 The rates are actually metric called model truth-based metrics which measure the relationship between the explanation maps generated by attribution methods and the target model's outputs. So obviously, it could measure the performance of the attribution methods taking the explanation maps as input.\par
-Drop rate refer to that if we remove unimportant features from the input, then the model's confidence score have no significant drop, and increase rate indicates that if we remove misleading features from the input, the model's confidence score may increase. Based on the thesis, drop rate indicate the explanation map's extent to remove unimportant features while increase rate infers the extent to remove misleading features from the input. Thus, a low drop rate and high increase rate could demonstrate a good attribution model.\par
+Drop rate refer to that if we remove unimportant features from the input, then the model's confidence score have no significant drop, and increase rate indicates that if we remove misleading features from the input, the confidence score of prediction may increase. Based on the thesis, drop rate indicate the explanation map's extent to remove unimportant features while increase rate infers the extent to remove misleading features from the input. Thus, a low drop rate and high increase rate could demonstrate a good attribution model.\par
 From the table we could know that in HMT, Ablation-CAM works best, as it has lowest drop rate among the algorithms and highest increase rate. Besides in MNIST-1D dataset, ablation-CAM also shows greater performance than the other CAM-based attribution methods, which probably means it will be a good choice in some specific scenarios.\par
 We will further discuss the evaluation result in the next section.
 
 \subsection{Discussion}
-Q:\'Were your selected methods successful in interpreting the target model trained on the HMT and MNIST-1D dataset correctly?\'\par
+Q: \'Were your selected methods successful in interpreting the target model trained on the HMT and MNIST-1D dataset correctly?\'\par
 A: From what is shown in the qualitative and quantitative analysis, GradCAM and Ablation-CAM show a better performance in MNIST-1D and HMT, while LIME get a very bad grade on drop rate. This is decided by their inherent attributes. LIME is a local explanation method. Although it can be used to interprete the global model, it still uses the local neighbours to explain the target cases, this will definitely lead to bad performance in the global interpretation.
 Compared with MNIST1D, XAI methods did not work better on HMT dataset. This is due to the features in the datasets. 
 \begin{itemize}
@@ -952,15 +950,15 @@ \subsection{Discussion}
 	This is supported by the normalized confusion matrix. 
 	As is shown in Fig.10 and Fig.22, the matrix for HMT is worse than MNIST1D. In the label 4 of HMT dataset, there is 44\% possibility of mistaking it for label 1. This is much too high and most equivalent to the true label, so the features of cases in HMT is similar to each other. MNIST1D matrix does not have data like that, so it's obvious that it's easier to interperte the MNIST1D dataset.
 \end{itemize}
-Q:\'In what cases they fail to explain the target model’s predictions?\'\par
+Q: \'In what cases they fail to explain the target model’s predictions?\'\par
 A: Let's take the case 20 in HMT as an example, as is shown in Fig.25, LIME does not extract the correct parts to explain the image. This is because:
 \begin{itemize}
 	\item First, features of different types in dataset don't show much difference from each other, while they differs a lot within the same label. This is illustrated above and we can find it in the confusion matrix.
 	\item Second, the attributes of the methods themselves is important. LIME is a local explanation. This decides the bad performance in global interpretation, so this method performs extremely bad on these cases.
 \end{itemize}
 So we can conclude that the cases which vary a lot from other cases in the same label group are most likely to be failure cases to explain. They are likely to be misinterpreted as some other categories, and they can be identified in the normalized confusion matrix. If value in row $i$, col $i$ is small, it means that this category is easy to be misinterpreted.
 \par
-Q:\'If you select two attribution methods, in what cases each one of them works better than the other one?\'\par
+Q: \'If you select two attribution methods, in what cases each one of them works better than the other one?\'\par
 A: In our evaluation, LIME failed to have improved performance comparing to the other methods in all datasets because it's local interpretation. 
 Maybe there are some cases in which LIME performs better, but generally speaking, LIME is not a good choice for global interpretation. 
 If we want to get a local explanation, LIME may be a good choice.\par
@@ -973,9 +971,6 @@ \subsection{Discussion}
 However, there is also an important drawback of Ablation-CAM that, it has much longer running time than the others due to its complete visit to every unit in every maps. 
 In those small-scaled problems, especially one-dimensional based problem, Grad-CAM will take great advantages of it.
 \par
-%This approach choose three different kind of CAM methods to evaluate two different model with different dimensions. It is obvious that Ablation-CAM holds the best performance on both datatsets. According to the paper of the ablation method, this algorithm can fit into all kinds of CNN architecture, but could perfectly fit in those networks with not so many layers. Our experiment includes one-hot and VGG-7 networks, and neither of them could be classified as DNN.\par
-%Ablation method comes out to fix the problem of gradient-related principle in the previous network. In the origin paper, it uses Grad-CAM as benchmark and use VGG and Inception-v4 as networks, resulting in a high performance in VGG but no significant improvement in Inception-v4, which is similar to our conclusion from the result.\par
-
 
 \begin{thebibliography}{00}
 \bibitem{b1} Selvaraju, Ramprasaath R., Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. "Grad-cam: Visual explanations from deep networks via gradient-based localization." In Proceedings of the IEEE international conference on computer vision, pp. 618-626. 2017. https://arxiv.org/abs/1610.02391

ProjectB/Report/.texpadtmp/ProjectB-Report.aux

@@ -1,5 +1,36 @@
 \relax 
+\citation{b6}
+\citation{keras}
+\citation{blog}
 \bibcite{b1}{1}
+\@writefile{toc}{\contentsline {section}{\numberline {I}Task 1: Knowledge Distillation in MNIST dataset}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-A}}Question 1}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsubsection}{\numberline {\mbox  {I-A}1}SubQuestion a}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsubsection}{\numberline {\mbox  {I-A}2}SubQuestion b}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsubsection}{\numberline {\mbox  {I-A}3}SubQuestion c}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsubsection}{\numberline {\mbox  {I-A}4}SubQuestion d}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsubsection}{\numberline {\mbox  {I-A}5}SubQuestion e}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-B}}Question 2}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-C}}Question 3}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-D}}Question 4}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-E}}Question 5}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-F}}Question 6}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-G}}Question 7}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-H}}Question 8}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-I}}Question 9}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-J}}Question 10}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-K}}Question 11}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-L}}Question 12}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-M}}Question 13}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {section}{\numberline {II}Task 2: Knowledge Distillation in MHIST dataset}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {II-A}}Question 1}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsubsection}{\numberline {\mbox  {II-A}1}SubQuestion a}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsubsection}{\numberline {\mbox  {II-A}2}SubQuestion b}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsubsection}{\numberline {\mbox  {II-A}3}SubQuestion c}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsubsection}{\numberline {\mbox  {II-A}4}SubQuestion d}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsubsection}{\numberline {\mbox  {II-A}5}SubQuestion e}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsubsection}{\numberline {\mbox  {II-A}6}SubQuestion f}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {II-B}}Question 2}{1}{}\protected@file@percent }
 \bibcite{b2}{2}
 \bibcite{b3}{3}
 \bibcite{b4}{4}
@@ -16,7 +47,7 @@
 \bibcite{b15}{15}
 \bibcite{b16}{16}
 \bibcite{b17}{17}
-\@writefile{toc}{\contentsline {section}{\numberline {I}Task 1: Knowledge Distillation in MNIST dataset}{1}{}\protected@file@percent }
-\@writefile{toc}{\contentsline {section}{\numberline {II}Task 2: Knowledge Distillation in MHIST dataset}{1}{}\protected@file@percent }
-\@writefile{toc}{\contentsline {section}{References}{1}{}\protected@file@percent }
-\gdef \@abspage@last{1}
+\bibcite{keras}{18}
+\bibcite{blog}{19}
+\@writefile{toc}{\contentsline {section}{References}{2}{}\protected@file@percent }
+\gdef \@abspage@last{2}

ProjectB/Report/ProjectB-Report.aux

@@ -1,6 +1,43 @@
 \relax 
+\citation{\global }
+\citation{\egroup }
 \bibcite{b1}{1}
+\bibcite{b2}{2}
+\bibcite{b3}{3}
+\bibcite{b4}{4}
+\bibcite{b5}{5}
+\bibcite{b6}{6}
+\bibcite{b7}{7}
+\bibcite{b8}{8}
+\bibcite{b9}{9}
 \@writefile{toc}{\contentsline {section}{\numberline {I}Task 1: Knowledge Distillation in MNIST dataset}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-A}}Question 1}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsubsection}{\numberline {\mbox  {I-A}1}SubQuestion a}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsubsection}{\numberline {\mbox  {I-A}2}SubQuestion b}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsubsection}{\numberline {\mbox  {I-A}3}SubQuestion c}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsubsection}{\numberline {\mbox  {I-A}4}SubQuestion d}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsubsection}{\numberline {\mbox  {I-A}5}SubQuestion e}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-B}}Question 2}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-C}}Question 3}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-D}}Question 4}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-E}}Question 5}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-F}}Question 6}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-G}}Question 7}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-H}}Question 8}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-I}}Question 9}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-J}}Question 10}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-K}}Question 11}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-L}}Question 12}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {I-M}}Question 13}{1}{}\protected@file@percent }
 \@writefile{toc}{\contentsline {section}{\numberline {II}Task 2: Knowledge Distillation in MHIST dataset}{1}{}\protected@file@percent }
+\@writefile{toc}{\contentsline {subsection}{\numberline {\mbox  {II-A}}Question 1}{1}{}\protected@file@percent }
 \@writefile{toc}{\contentsline {section}{References}{1}{}\protected@file@percent }
-\gdef \@abspage@last{1}
+\bibcite{b10}{10}
+\bibcite{b11}{11}
+\bibcite{b12}{12}
+\bibcite{b13}{13}
+\bibcite{b14}{14}
+\bibcite{b15}{15}
+\bibcite{b16}{16}
+\bibcite{b17}{17}
+\gdef \@abspage@last{2}