\chapter{Experiments and Results}\label{ch: exp}  
\thispagestyle{myheadings}

There are two main aspects important to evaluate a detector's performance: true positive rate and false positive rate. These parameters are not independent, since a very permissive detector may achieve almost perfect results in terms of true positives, but with prohibitive amounts of false positives as a consequence. The conclusion is that these parameters must be assessed simultaneously in order to achieve a meaningful performance evaluation. This can be done through the use of a detection performance curve that draws the relationship between true positive and false positive rates. \newline
To test the detector's performance, experiments were carried out in two distinct datasets. The INRIA dataset provides a test set that allows for a straightforward quantification of results and makes them directly comparable to other detectors that were tested in the same manner, conferring objectiveness and significance to the obtained results. \newline
The detector was also tested in a dataset acquired aboard the \textit{Atlas Car} in Aveiro. Achieving a significant quantification of results on the \textit{Atlas Pedestrian Dataset} demands the establishment of a ground truth that reflects the necessities of the application, so, rather than aiming for pedestrian individualization, situations representing any potential risk to pedestrians were the considered targets. \newline
All the relevant aspects related to the attainment of results are discussed in this chapter.

\section{INRIA Dataset Experiments}\label{sec: inria}

The INRIA Dataset provides a test-bed that largely facilitates the process of generating detection performance curves. The adopted methodology, ground truth rules and results are discussed in this section.

\subsection{Ground Truth}\label{subsec: grdthr_inria}

To produce the data to draw the curves, the algorithm was tested on 1132 positive windows and around 10 million negative windows taken from 452 different images. In these circumstances the ground truth is established in a straightforward manner, since it is known before-hand that each one of the 1132 positive windows contain a pedestrian, and none of the 452 images from which the negative windows originate contain a single pedestrian.

\subsection{Methodology}\label{subsec: meth_inria}

For a given threshold, the positive windows are evaluated and every negative response from the detector means a missed pedestrian. Then the negative windows are tested in the same fashion, only now any positive response from the detector is a false positive.\newline
This process is repeated several times for different thresholds and the curves are drawn by connecting the various data points. To understand the effect of the feature pool size on the detector performance, two detectors are tested in the same way and compared, one trained with 15K features and the other with 10K.

\subsection{Results}\label{subsec: inriaresults}

Figure 4.1 shows the detection performance curves for both detectors. The x-axis measures in false positives per window and is log-scaled for a better visualization. The y-axis measures detection rate in percentage, which is the quotient between the true positives and the total number of pedestrian windows tested.

\begin{figure}[H]
  \centering
    \includegraphics[width=15cm]{Imagens/results.png}
    \caption[Detector performance on the INRIA dataset]{Detector performance on the INRIA dataset. \label{fig: results}}  
\end{figure}

\subsection{Discussion}\label{subsec: comments}

On the reference value of 10$^{-4}$ false positives per window, the algorithm trained with 15K features achieves 79.5$\%$ detection rate, and, although this is not a bad figure at all, the original implementation documents a detection rate of 91$\%$, over 10 points above the presented implementation. \newline 
It was hypothesised that the explanation for this difference in performance could lie in the fact that the original classifier was trained with 30K features, rather than 15K. To back this hypothesis, a similar classifier was trained but now with only 10K features; if the number of features could be causing this under-performance, an even lower detection rate was expected, and, in fact, that is what the results show, with the 10K detector performing $\sim$3.5$\%$ points below the 15K detector. That is not to say that if this detector was trained with 30K features the results would be exactly the same as in the original, they would probably still be lower for different reasons. The method may be the same, but the implementations are unique, and this algorithm has still much room for tweaks and improvements. \newline
Tests weren't made for 30K features because OpenCV's data structures wouldn't even allow for such big feature spaces. So, instead, it was decided that 15K and 10K feature detectors were to be trained and tested for performance in order to establish a coherent relation with the results shown in the original document.

\section{\textit{Atlas} Pedestrian Dataset Experiments}\label{sec: atlas}

One of the main goals for this project was the implementation of a pedestrian detection infrastructure on the \textit{Atlas Car}, making the evaluation of the detector on data recorded by the car's cameras a paramount procedure. This dataset was obtained in Aveiro on some areas known for being usually busy, such as the Avenida Lourenço Peixinho and the area around the university. Because it is important to test the detector on a wide variety of urban environments, some areas with less pedestrian activity were also recorded and tested. \newline
Quantifying results isn't nearly as straightforward on the \textit{Atlas} Pedestrian Dataset as it was on the INRIA Dataset, since labelling all the acquired data frames was an impossible to complete task in the available time to complete the project. It was then necessary to adopt a different methodology for obtaining significant results. \newline
In this section ground truth rules, methodology and results will be discussed.

\subsection{Ground Truth}\label{subsec: atlasgrdtrh}

A close analysis of the \textit{Atlas} Pedestrian Dataset leads to the conclusion that in order to achieve any meaningful result out of real road data, a careful ground truth rule that takes into account the intended application has to be established. One can easily think that the results of an application for counting pedestrians have to be perceived differently then the ones for driving assistance, where detecting all pedestrians in the environment is not only unnecessary, but undesirable, since many appear in situations that do not pose any risk. Having this in mind, only pedestrians that appear at a close to medium range on the image are taken into account, since these are the ones that may be at risk, and pedestrians that appear either in the far opposite side of the street, or very far into the horizon are discarded. Also, rather than individualizing pedestrians, it is important to detect situations that may represent any potential risk. It was then assumed that two or more pedestrians that stand relatively close to each other represent one target. Figures 4.2 through 4.7 illustrate some situations where pedestrians were considered valid targets, and others where they were not. For easier visualization, green rectangles were manually drawn around valid targets, and red rectangles around not valid ones.

\begin{figure}[H]
  \centering
    \includegraphics[width=13cm]{Imagens/grdtruth1.png}
    \caption[Ground truth example 1]{Ground truth example 1 - Close and medium range pedestrians that stand in the sidewalk are targets, whereas far range pedestrians in the opposite side of the road  or far in the sidewalk are not. \label{fig: grdtruth1}}  
\end{figure}
\newpage

\begin{figure}[H]
  \centering
    \includegraphics[width=13cm]{Imagens/grdtruth2.png}
    \caption[Ground truth example 2]{Ground truth example 2 - Occluded pedestrians that stand in close or medium range are also valid targets. \label{fig: grdtruth2}}  
\end{figure}

\begin{figure}[H]
  \centering
    \includegraphics[width=12cm]{Imagens/grdtruth3.png}
    \caption[Ground truth example 3]{Ground truth example 3 - Groups of pedestrians standing close to each other are assumed to be only one target. \label{fig: grdtruth3}}  
\end{figure}

\begin{figure}[H]
  \centering
    \includegraphics[width=13cm]{Imagens/grdtruth4.png}
    \caption[Ground truth example 4]{Cyclists are considered valid targets. \label{fig: grdtruth4}}  
\end{figure}

\begin{figure}[H]
  \centering
    \includegraphics[width=13cm]{Imagens/grdtruth5.png}
    \caption[Ground truth example 5]{Ground truth example 5 - Two pedestrians crossing the street together are considered to be one target. \label{fig: grdtruth5}}  
\end{figure}

\begin{figure}[H]
  \centering
    \includegraphics[width=13cm]{Imagens/grdtruth6.png}
    \caption[Ground truth example 6]{Ground truth example 6 - Difficult examples such as shadowed pedestrians are valid targets. \label{fig: grdtruth6}}  
\end{figure}


\subsection{Methodology}\label{subsec: atlasmethodology}

To assess the detector's performance on the \textit{Atlas} Pedestrian Dataset, the program is set to draw green rectangles around positive detections on the images, and save them to disk. Afterwards, the images are analysed one by one to identify the risk situations involving pedestrians, and whether they are detected or not. \newline
A preliminary assessment of the detector's behaviour makes it clear that most of the false detections occur on vertical-shaped, pedestrian-like objects. To objectively understand the behaviour of the detector on false detections, the concept of \textit{Pseudo-False Positives} (PFP) was used. These are false detections that instead of behaving randomly, follow a predictable pattern that can be easily understood. In this work, false detections on objects of vertical expression, such as trees, traffic signs or mail boxes are labelled as PFP. So, in an attempt to lower the PFP rate and determine whether the nature of the training data could influence the performance, a third round of bootstrapping was carried out on top of the previous INRIA data. Using just a few selected images with no pedestrians, $\sim$750 hard negative windows were added to the training data and a new detector was trained. The detector trained solely with INRIA data is referred as \textit{INRIA Detector} and the one that was modified to include some \textit{Atlas} examples is referred as \textit{Atlas Bootstrapped Detector}. They are compared in terms of detection rate over a given number of False Positives per Image (FPPI).
 

\subsection{Results}\label{subsec: atlasres}

The detectors described in the previous section were both tested in the same data logs using the \textit{rosbag} feature provided by ROS, which allows for data to be replayed as it was recorded in the field. \newline
It is important to mention that to perform these experiments, the re-player had to be set to run 16 times slower than the original frame-rate and, since experiments were not carried out at the same time for both detectors, the frames that were analysed were not exactly the same. Because of that, there is a slight difference in the number of frames tested for each detector and consequently a varying number of targets. This situation is also explained by the fact that in an experiment a pedestrian might happen to be visible for multiple frames and not on the other. However, rather than invalidating the evaluation approach, this circumstance further attests to the method's capacity for generalization. \newline
The evaluated parameters displayed on table 4.1 are the following:
\begin{itemize}
\item True Positives: correctly labelled targets.
\item False Negatives: not correctly labelled targets.
\item False Positives: wrong positive classifications of non-targets.
\item Pseudo-false Positives: wrong positive classifications of pedestrian-shaped non-targets.
\item Detection Rate: ratio between the correctly labelled targets and the total number of them.
\item False Positives per Image: ratio between total number of both types of false positives and the total number of frames tested.
\end{itemize}

\begin{table}[H]
\centering
\begin{tabular}{ |l|r|r| }
\hline
\multicolumn{3}{ |c| }{\large{\textbf{Atlas Pedestrian Dataset Results}}} \\ \hline
\textbf{Parameter} & \textbf{INRIA Detector} & \textbf{Atlas Bootstrapped Detector} \\ \hline
Total frames analysed & 529 & 540 \\ \hline
Total number of targets & 189 & 207 \\ \hline
True Positives & 81 & 101 \\ \hline
False Negatives & 108 & 106 \\ \hline
False Positives & 102 & 89 \\ \hline
Pseudo-false Positives & 142 & 88 \\ \hline
Detection Rate & 42.8$\%$ & 48.7$\%$ \\ \hline
False Positives per Image & 0.45 & 0.33 \\ \hline

\end{tabular} \label{tab_res}
\caption{Atlas Pedestrian Dataset Results}
\end{table}
To provide a better understanding of how the detection works, some example results are shown in figures 4.8 through 4.19, in which green rectangles were automatically drawn by the program. Some of the most common urban pedestrian detection situations are covered by the example images that follow.

\begin{figure}[H]
  \centering
    \includegraphics[width=12cm]{Imagens/Res8.jpg}
    \caption[Example result 1]{Example result 1 - Close range pedestrians crossing the street don't go undetected. \label{fig: res1}}  
\end{figure}

\begin{figure}[H]
  \centering
    \includegraphics[width=12cm]{Imagens/Res1.jpg}
    \caption[Example result 2]{Example result 2 - In bright environments, close range pedestrians walking on the sidewalk are usually detected.\label{fig: res2}}  
\end{figure}

\begin{figure}[H]
  \centering
    \includegraphics[width=12cm]{Imagens/Res10.jpg}
    \caption[Example result 3]{Example result 3 - Shadowed pedestrians highly contribute to the miss rate. \label{fig: res3}}  
\end{figure}

\begin{figure}[H]
  \centering
    \includegraphics[width=12cm]{Imagens/Res2.jpg}
    \caption[Example result 4]{Example result 4 - Groups of pedestrians are often detected as one target, rather then being individualized. Also, partially occluded pedestrians are often missed.  \label{fig: res4}}  
\end{figure}

\begin{figure}[H]
  \centering
    \includegraphics[width=12cm]{Imagens/Res9.jpg}
    \caption[Example result 5]{Example result 5 - Medium and far range pedestrians are sometimes missed even in bright environments. \label{fig: res5}}  
\end{figure}

\begin{figure}[H]
  \centering
    \includegraphics[width=12cm]{Imagens/Res3.jpg}
    \caption[Example result 6]{Example result 6 - Example of a common pseudo-false positive. \label{fig: res6}}  
\end{figure}

\begin{figure}[H]
  \centering
    \includegraphics[width=12cm]{Imagens/Res4.jpg}
    \caption[Example result 7]{Example result 7 - Trees are also commonly detected as pedestrians. \label{fig: res7}}  
\end{figure}

\begin{figure}[H]
  \centering
    \includegraphics[width=12cm]{Imagens/Res7.jpg}
    \caption[Example result 8]{Example result 8 - Traffic signs are also confused by the detector rather often. On the right, another example showing that occlusion and shadowed environments are major issues in pedestrian detection. \label{fig: res8}}  
\end{figure}

\begin{figure}[H]
  \centering
    \includegraphics[width=12cm]{Imagens/Res6.jpg}
    \caption[Example result 9]{Example result 9 - The detector occasionally makes random false detections. \label{fig: res9}}  
\end{figure}

\begin{figure}[H]
  \centering
    \includegraphics[width=12cm]{Imagens/Res12.jpg}
    \caption[Example result 10]{Example result 10 - Cyclists are often missed when they lack the vertical expression of a pedestrian. \label{fig: res11}}  
\end{figure}

\begin{figure}[H]
  \centering
    \includegraphics[width=12cm]{Imagens/Res13.jpg}
    \caption[Example result 11]{Example result 11 - Cyclists are detected if they appear on frontal or back perspectives. \label{fig: res12}}  
\end{figure}

\begin{figure}[H]
  \centering
    \includegraphics[width=12cm]{Imagens/Res11.jpg}
    \caption[Example result 12]{Example result 12 - A result obtained on the highway, where a motorcyclist was wrongly identified as a target. \label{fig: res10}}  
\end{figure}



\subsection{Discussion} \label{commentary}

Although a quantification of the results on this dataset was obtained, it is important to have in mind that these are preliminary results, and they might not assess with a great degree of confidence to the performance of the detector. Obtaining more confident results would require more experiments and a larger testing data. It is, however, a preliminary result that gives a fair idea of the potential of this approach when applied to driving assistance.\newline
At a first glance, the achieved detection rate seems to be a low one, but, to make a correct interpretation of the results, it is crucial to have in mind that the primary detector that was used in the \textit{Atlas} data was solely trained with images from the INRIA dataset, which are of completely different nature than the \textit{Atlas} ones, as most aren't even of urban environment, and the sensors used to acquire the data are also different. This provides a hint that the adopted method generalizes well for images of different nature. However, the results improved significantly just by adding a few hundred examples from the \textit{Atlas} dataset to the training data. The improvement of the results is best observed in the lowering of the false positive rate, which accounts for the fact that only negative examples were added to the training data. The lower occurrence of false positives made possible for a less demanding threshold to be used in the \textit{Atlas Bootstrapped Detector} and, because of that, the detection rate went from 42.8$\%$ to 48.7$\%$ with an even lower rate of False Positives per Image. One is inclined to think that if the detector was trained solely or mainly with \textit{Atlas} data, the results would improve even further.
\newpage\null\thispagestyle{myheadings}\newpage