%
% uaThesis example (for a thesis written in Portuguese)
%
% the complete list of options and commands can be found in uaThesis.sty 
%

\documentclass[11pt,twoside,a4paper]{report}
\usepackage[DEM,newLogo,final]{uaThesis}

\def\ThesisYear{2014}

% optional packages
\usepackage[english]{babel}
\usepackage[hidelinks]{hyperref}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{float}
%\usepackage[demo]{graphicx}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{enumitem}
\usepackage{acronym}
\usepackage{calc}
\usepackage{multirow}
\usepackage{array}
\usepackage{textcomp}
\usepackage{url}
\usepackage{fancyhdr}                             % Utilização de fancy headings
\usepackage{listings}
\usepackage{color}

\definecolor{dkgreen}{rgb}{0,0.6,0}
\definecolor{gray}{rgb}{0.5,0.5,0.5}
\definecolor{mauve}{rgb}{0.58,0,0.82}

\lstset{frame=tb,
  language=Java,
  aboveskip=3mm,
  belowskip=3mm,
  showstringspaces=false,
  columns=flexible,
  basicstyle={\small\ttfamily},
  numbers=none,
  numberstyle=\tiny\color{gray},
  keywordstyle=\color{blue},
  commentstyle=\color{dkgreen},
  stringstyle=\color{mauve},
  breaklines=true,
  breakatwhitespace=true,
  tabsize=3
}

\makeatletter
\def\namedlabel#1#2{\begingroup
    #2%
    \def\@currentlabel{#2}%
    \phantomsection\label{#1}\endgroup
}
\makeatother

\newcolumntype{C}[1]{>{\centering\let\newline\\\arraybackslash\hspace{0pt}}m{#1}}

% custom macros
\def\I{\mathtt{i}}
\def\Exp#1{e^{2\pi\I #1}} % argument inside braces, i.e., "{}"
\def\EXP#1.{e^{2\pi\I #1}} % argument finishes when a full stop is encountered, i.e., "."

% optional: visual delimiters for floats (figures and tables)
\def\topfigrule{\kern 7.8pt \hrule width\textwidth\kern -8.2pt\relax}
\def\dblfigrule{\kern 7.8pt \hrule width\textwidth\kern -8.2pt\relax}
\def\botfigrule{\kern -7.8pt \hrule width\textwidth\kern 8.2pt\relax}

\acrodef{LAR}[LAR]{Automation and Robotics Laboratory}

\acrodef{ADAS}[ADAS]{Advanced Driver Assistance Systems}

\acrodef{LIDAR}[LIDAR]{Light Detection And Ranging}

\acrodef{RADAR}[RADAR]{Radio Detection And Ranging}

\acrodef{MLE}[MLE]{Maximum-Likelihood Estimation}

\acrodef{ROI}[ROI]{Region Of Interest}

\acrodef{SVM}[SVM]{Support Vector Machine}

\acrodef{LDA}[LDA]{Linear discriminant analysis}

\acrodef{NBC}[NBC]{Naïve Bayes Classifier}

\acrodef{GMM}[GMM]{Gaussian Mixture Models}

\acrodef{ANN}[ANN]{Artificial Neural Network}
 
\acrodef{MTT}[MTT]{Multi Target Tracking}

\acrodef{CRF}[CRF]{Conditional Random Field}

\acrodef{HD}[HD]{High Definition}

\acrodef{FPS}[FPS]{Frames per second}

\acrodef{BB}[BB]{Bounding Box}

\begin{document}

%
% First thesis pages
%
% First alternative, with a figure
\TitlePage
  %\GRID  % for debugging ONLY
  \HEADER{\BAR\FIG{\includegraphics[height=60mm]{Figures/uaLogoNew.pdf}}} % the \FIG{} is optional
         {\ThesisYear}
  \TITLE{Rui Filipe Cabral de Azevedo}
        {Sensor Fusion of LASER and Vision in Active Pedestrian Detection}
  \vspace*{5mm}
  \TITLE{}
        {Deteção Ativa de Peões por Fusão Sensorial de LASER e Visão}
\EndTitlePage
\titlepage\ \endtitlepage % empty page

\TitlePage
  \HEADER{}{\ThesisYear}
  \TITLE{Rui Filipe Cabral de Azevedo}
          {Sensor Fusion of LASER and Vision in Active Pedestrian Detection}
    \vspace*{5mm}
    \TITLE{}
          {Deteção Ativa de Peões por Fusão Sensorial de LASER e Visão}
  \vspace*{15mm}
  \TEXT{}
       {Dissertação apresentada à Universidade de Aveiro para cumprimento dos requisitos necessários à obtenção do grau de Mestre em Engenharia Mecânica, realizada sob a orientação científica de Vítor Manuel Ferreira dos Santos, Professor do Departamento de Engenharia Mecânica da Universidade de Aveiro}
\EndTitlePage
\titlepage\ \endtitlepage % empty page

\TitlePage
  \vspace*{55mm}
  \TEXT{\textbf{O j\'uri~/~The jury\newline}}
       {}
  \TEXT{Presidente~/~President}
       {\textbf{Professor Doutor Jorge Augusto Fernandes Ferreira}\newline {\small
        Professor Auxiliar da Universidade de Aveiro}}
  \vspace*{5mm}
  \TEXT{Vogais~/~Committee}
       {\textbf{Professor Doutor Paulo Miguel de Jesus Dias}\newline {\small
        Professor Auxiliar do Universidade de Aveiro - DETI}}
  \vspace*{5mm}
  \TEXT{}
       {\textbf{Professor Doutor Vítor Manuel Ferreira dos Santos
		}\newline {\small
		Professor Associado da Universidade de Aveiro (orientador)}}
\EndTitlePage
\titlepage\ \endtitlepage % empty page

\TitlePage
  \vspace*{55mm}
  \TEXT{\textbf{agradecimentos~/\newline acknowledgements}}
       {\'E com muito gosto que aproveito esta oportunidade para agradecer a todos os que me
        ajudaram, direta ou indiretamente, durante este 5 anos e colaboraram para o desfecho desta formação académica.}
  \TEXT{}
       {Ao meu Orientador, Professor Doutor Vítor Manuel Ferreira dos Santos, pela confiança, criticas e apoio que sempre me deu ao longo deste projecto, estando assim apto para agarrar a vida pelos cornos.}
  \TEXT{}
       {Aos meus pais, Alcindo e Fatinha, a oportunidade que me deram para prosseguir os meus estudos e apoio incondicional em todos os anos da minha vida, cujo amor incondicional sempre foi e será um
       autêntico pilar na minha vida. MUITO OBRIGADO por tudo.}
  \TEXT{}
       {Aos meus padrinhos e prima, que sempre acreditaram que eu seria capaz e sempre me acolheram com um sorriso.}
  \TEXT{}
       {Às minhas grandes avós, que me ensinaram o que é trabalhar na terra e mostraram-me o quanto a vida pode ser difícil, mas não impossível.}
  \TEXT{}
       {À minha melhor amiga, Luísa, por toda a paciência, carinho, atenção e apoio que sempre me deu. Obrigado por me ajudares a ultrapassar todos os obstáculos.}
  \TEXT{}
       {Aos meus verdadeiros amigos, aos amigos de mecânica e aos meus colegas do LAR, o meu muito obrigado pelos bons momentos que passamos e de certeza que vamos continuar a passar :$)$. Queria também agradecer especialmente ao grande Jorge Almeida, que nunca deixou de parar o que quer que fosse para me ajudar.}
   \TEXT{}
       {A todos um Muito Obrigado!}
\EndTitlePage
\titlepage\ \endtitlepage % empty page

\TitlePage
  \vspace*{55mm}
  \TEXT{\textbf{Palavras-chave}}
       {Deteção de peões, Fusão sensorial, LIDAR, Classificação de possiveis peões, Multi-thread, Calibração}
  \vspace*{5mm}
  \TEXT{\textbf{Resumo}}
       {Este trabalho explora uma técnica de fusão sensorial que visa dotar veículos de mecanismos rápidos de detecção de peões em ambiente exterior. O método restringe as zonas de procura numa imagem com base em indicadores obtidos por outro sensor (LIDAR). Esta técnica tem como base a idéia de que havendo um registo entre os sensores envolvidos, um sensor "rápido" mas pouco preciso, pode indicar as regiões onde potencialmente há alvos, e outro sensor, "lento" mas mais robusto, é utilizado para fazer a confirmação da deteção. Com vista a explorar essas propriedades, foi criado um algoritmo que utiliza a informação de dois sensores, para primeiro selecionar, de entre muitos objectos, possíveis peões(fase LIDAR) e dada a informação da localização do possível pedestre, uma imagem já à escala e precisa da localização, é recortada da imagem inicial, sendo a mesma enviada a ser processada por um detetor de peões (sensor mais robusto), permitindo a sua rigorosa classificação. O método é testado em dois conjuntos de dados diferentes e os resultados confirmam a sua validade.}
\EndTitlePage
\titlepage\ \endtitlepage % empty page

\TitlePage
  \vspace*{55mm}
  \TEXT{\textbf{Key words}}
         {Pedestrian Detection, Sensor Fusion, LIDAR, Possible Pedestrian Classification, Multi-thread, Calibration}
  \vspace*{5mm}
  \TEXT{\textbf{Abstract}}
       {This work explores a technique of sensor fusion that aims to equip vehicles with pedestrian fast detection mechanisms in exterior environments. This method restricts image areas of search based on indicators obtained by another sensor (LIDAR). This technique is based on the idea that when having a registration among the involved sensors, one "fast" sensor, but inaccurate, that can indicate regions where potential pedestrian are located on the image, and another sensor, "slower" but more robust that is used to confirm detection more accurately. So, an algorithm was created to merge two algorithms, a LIDAR-based tracking and a vision-based detection algorithm; The LIDAR indicates the precise location and scale of the potential pedestrian on the image, and crop the image relative to the potential pedestrian, being processed afterwards by one pedestrian detection algorithm to validate the classification. The method is tested in two different cases and the results confirm their validity.}
\EndTitlePage
\titlepage\ \endtitlepage % empty page


%
% Tables of contents, of figures, ...
%

\pagenumbering{roman}
\tableofcontents

\cleardoublepage
\listoffigures

\cleardoublepage
\listoftables

% Generate the glossary
\cleardoublepage
\vspace*{12mm}

\section*{\Huge{List of Acronyms}}	% =========== LISTA DE ACRONIMOS =================

\addcontentsline{toc}{chapter}{List of Acronyms}

\vspace*{12mm}

\begin{acronym}

\acro{LAR}{Laboratório de Automação e Robótica (Automation and Robotics Laboratory)}

\acro{ADAS}{Advanced Driver Assistance Systems}

\acro{LIDAR}{Light Detection And Ranging}

\acro{RADAR}{Radio Detection And Ranging}

\acro{MLE}{Maximum-Likelihood Estimation}

\acro{ROI}{Region Of Interest}

\acro{SVM}{Support Vector Machine}

\acro{LDA}{Linear discriminant analysis}

\acro{NBC}{Naïve Bayes Classifier}

\acro{GMM}{Gaussian Mixture Models}

\acro{ANN}{Artificial Neural Network}
 
\acro{MTT}{Multi Target Tracking}

\acro{CRF}{Conditional Random Field}

\acro{HD}{High Definition}

\acro{FPS}{Frames per second}

\acro{BB}{Bounding Box}
 

\end{acronym}




\cleardoublepage
\pagenumbering{arabic}

\newcommand{\thesisdegree}{Mestrado} 
\newcommand{\authornamelong}{Rui Filipe Cabral de Azevedo}
\pagestyle{fancy}
\renewcommand{\chaptermark}[1]{\markboth{\thechapter.#1}{}} % Capítulos em minúsculas
\fancyhf{}                                                  % Reset aos headers e footers
  \fancyhead[LE,RO]{\thepage}                               % Header Left-Even (LE), Right-Odd (RO)
  \fancyhead[LO]{\leftmark}                                 % Header Left-Odd (LO)
  \fancyhead[RE]{\leftmark}                                 % Header Right-Even (RE)
  \fancyfoot[LE]{\authornamelong}                               % Footer Left-Even (LE)
  \fancyfoot[LO]{\authornamelong}                               % Footer Left-Odd (LO)
  \fancyfoot[RE]{\textit{Dissertação de \thesisdegree~/~Master Thesis}}     % Footer Right-Even (RE)
  \fancyfoot[RO]{\textit{Dissertação de \thesisdegree~/~Master Thesis}}     % Footer Right-Odd (RO)
  \renewcommand{\headrulewidth}{0.25pt}                     % Espessura da linha de header
  \renewcommand{\footrulewidth}{0.25pt}                     % Espessura da linha de footer
  \addtolength{\headheight}{0.5pt}                          % Espaçamento para a linha
%\chapter{Introdução}
\chapter{Introduction}

\section{ATLAS Project}

ATLAS is a project created and developed at the \ac{LAR} from the Department of Mechanical Engineering of the University of Aveiro, Portugal. \\

The project took its first steps in 2003 with the purpose of competing in autonomous driving competitions taking place at the Portuguese National Robotics Festival.
Since then, ATLAS has been growing, with the development of several series of robots, having had a continuous success in those competitions and won many prizes; the project is divided into 2 types, two small robots (figure~\ref{fig:ATLAS2010_ATLASMV}) and one car size robot (figure~\ref{fig:ATLASCar}).
% colocar imagem dos ATLAS

\begin{figure}[H]
\centering
\begin{minipage}{.5\textwidth}
	\centering
	\captionsetup{justification=centering}
	\includegraphics[scale=0.3]{Figures/atlas2010_MV3_real1.png}
	\caption{Atlas 2010 (left) and Atlas MV (right)}
	\label{fig:ATLAS2010_ATLASMV}
\end{minipage}%
\begin{minipage}{.5\textwidth}
	\centering
	\includegraphics[scale=0.3]{Figures/AtlasCar.jpeg}
	\caption{AtlasCar}
	\label{fig:ATLASCar}
\end{minipage}
\end{figure}

%\section{Enquadramento}
\section{Motivation}

In modern \ac{ADAS} there is a rising concern about attaching environment detection mechanisms into vehicles, particularly on the traffic agents detection, like pedestrians and other vehicles.
%Nos sistemas modernos de apoio à condução (ADAS) há uma preocupação crescente em dotar os veículos de mecanismos de deteção do ambiente exterior, em particular na deteção de outros agentes em trânsito, como sejam outros veículos ou peões. \\

Pedestrians are particularly important because of security issues and the complexity of the solution of the main problem. A lot of research has been done in this area, using different approaches, like visual and thermal image, \ac{LIDAR}, \ac{RADAR}, etc. However, there is no definitive solution and the problem is still being studied by many researchers, from universities to automotive industries. 

The use of visual image, for example, is a low cost solution for this matter, but it still has a very low efficiency, especially in image processing time, which limits real-time application.

%A questão dos peões reveste-se de particular importância pelas questões de segurança que envolve e pela complexidade da solução do problema no caso geral. Têm sido desenvolvidos muitos trabalhos nesta área, e vários tipos de sensores e abordagens têm sido experimentadas usando imagem visual, imagem termográfica, LIDAR, RADAR, etc.  Porém, não há ainda soluções definitivas e o problema é ainda estudado por muitos interessados no assunto, desde académicos à indústria automóvel.O uso de imagem visual, por exemplo, assenta em soluções de baixo custo, mas tem uma eficácia ainda muito baixa, em especial no tempo de processamento da imagem que limita a aplicação em tempo real.\\
 
On the other hand, there are other sensors like \ac{LIDAR} that have a fast performance, but with very uncertain results on the assessment of the detected targets. Therefore, combining the information of these two different sensors, might be a way to obtain a more efficient and robust system. 

%arise relevance on studying and develop an application that can combine different sensors information, like visual image and laser, in order to obtain a more efficient and more robust system.

The approach can be to explore one technique that narrows searching areas in an image, using indicators obtained by another sensor (\ac{LIDAR}). In other words, combine the sensors involved, one "fast" sensor not very accurate, but which can indicate areas with potential targets, and another sensor, this time a "slow" one, more robust, which can be used to confirm detection.

%Sabe-se por outro lado que há outros sensores, como o caso do LIDAR que têm um desempenho muito rápido, mas com resultados muito incertos na apreciação dos alvos detetados. Assim, surge como pertinente estudar o desenvolvimento de uma aplicação que combine informação de sensores diferentes (imagem visual e laser) no sentido de se obter um sistema mais robusto e eficaz. A abordagem pode ser a de explorar um técnica que restringe as zonas de procura numa imagem com base em indicadores obtidos por outro sensor (LIDAR), ou seja, havendo um registo entre os sensores envolvidos, um sensor “rápido” mas pouco preciso pode indicar as regiões onde potencialmente há alvos, e o outro sensor, “lento” mas mais robusto pode ser usado para fazer a confirmação da deteção.

%\section{Objetivos Principais}
\section{Objectives}

The main objectives of this work are:

\begin{itemize}

%\item Adaptação de um algoritmo de deteção visual de peões já existente para operar com diferentes parametrizações. % Feito

\item Adaptation of an existing visual pedestrian detection algorithm, so it can operate with different parameters. % done

%\item Desenvolver uma ferramenta para calibração manual ou  semi-automática entre uma câmara e sistemas de medição LIDAR. 
% Calibração Manual,    dará para fazer a automática

\item Development of a manual or semi-automatic calibration tool, between \ac{LIDAR} and a single camera. %Manual calibration made

%\item Desenvolvimento de uma aplicação para a deteção ativa de peões por procura localizada em imagem usando formas integradas de combinar os dados LIDAR e visuais. % Feito

\item Development of an application for pedestrian active detection, with image localized search, combining \ac{LIDAR} and vision data. % done

\end{itemize}

%\section{Estado da Arte}
\section{State of the Art}

%Esta secção incidirá primeiro na calibração, seguindo-se de a obtenção da ROI e finalmente passando para os critérios usados para selecionar possíveis peões.
This section will focus on four main points: the use of \ac{LIDAR} data to easily separate objects in the environment; pedestrian classification resorting to vision image; sensor fusion classification methods and lastly, the most common automatic calibration methods.

\subsection{LIDAR-based systems}

Pedestrian detection systems based on data provided by a 2D \ac{LIDAR} have different types of applications such as detection, tracking, classification and high-level decision systems. Some related work is described below.

\subsubsection{Detection and Classification}

Premebida says the \ac{LIDAR}-based classification stage consists of a decision-making function which decides the class(PED or nPED) a given detected object belongs to ~\cite{Premebida2012}. 

%This section focuses on detection and classification methods, that only with laser data, classify the moving object into either a person, a group of people, a bicycle or a car.

Concerning detection using only laser data, \cite{Zhao2009} propose a ego-vehicle localization and object detection system in urban environments. In the case of cars, the data appearance varies dramatically because the relative position and direction to sensor change. They call each edge an ”axis”. In different cases where only the rear or front side of a car is measured, a motion vector vertical to the extracted axis is detected, making up an additional axis, thus a car is characterized as a two axis object. 


For bicycles, no obvious axis could be detected when the laser is measuring its rear or head, it only could fit a line when seen from the side, characterizing a bicycle as a one axis object.


In the pedestrians case, no axis can be extracted, because the appearance (resembles to semi-circle) does not change dramatically; therefore, it was characterized as a zero-axis object.
The object models are shown in figure~\ref{fig:Zhao_object_model}.

\begin{figure}[H]
	\centering
		\includegraphics[scale=0.4]{Figures/Zhao_object_model.png}
	\caption{The object models for pedestrians, bicycle and car\cite{Zhao2009}}
	\label{fig:Zhao_object_model}
\end{figure}

For all laser-segments, the objects' speeds are estimated and, for pedestrians classification, a set of features is used to model the pedestrian-likelihood.
%When given a feature estimation, if the value is valid the probability of being the according object is calculated using Parzen window.

Other authors use a 4-layer \ac{LIDAR} (Ibeo Alasca XT) to perform, in each laser-layer a segmentation, then each layer is fused and pedestrians are detected based on \ac{MLE} which is modeled using Parzen method~\cite{Parzen-Window}. Figure~\ref{fig:Gidel_multilayer} shows a scheme used in \cite{Gidel2009} of how the multilayer works.

\begin{figure}[H]
	\centering
		\includegraphics[scale=0.11]{Figures/Gidel_multilayer_laser_sensor_scheme.png}
	\caption{Pedestrian detection algorithm using a multilayer laser sensor\cite{Gidel2009}}
	\label{fig:Gidel_multilayer}
\end{figure}

%The most commonly used classification strategies are: \ac{SVM}, \ac{LDA}, \ac{NBC}, \ac{GMM}, and \ac{ANN}.

\clearpage
\subsubsection{Multi Target Tracking}

Much work has been developed on the subject of multi target tracking over the past few decades \cite{BlackmanS.S.andPopoli1999}. This is a very important matter, because normally \ac{LIDAR} data are much faster than vision, therefore this technique is suited for detection and tracking of multiple moving targets in situations of strong occlusion. The process starts with the application of temporal filters to the raw data in order to remove noise, followed by a multi phase segmentation with the goal of overcoming occlusions. The resulting segments represent objects in the environment. For each segment a representative point is defined. This point is calculated to better represent the object while keeping some invariance to rotation and shape changes. In order to perform the tracking, a list of objects to follow is maintained and all visible objects are associated with objects from this list, using search techniques based on the predicted motion of objects. A search zone shaped as an ellipse is defined for each object and it is in this zone that the association is preformed. The motion prediction is based on two motion models, one with constant velocity and the other with constant acceleration and in the application of Kalman filters. The algorithm was tested in diverse real conditions and shown to be robust and effective in the tracking of people even in situations of long occlusions \cite{Almeida2010}. 

%The Laser used on the experience was the Hokuyo utm, Fig. ~\ref{fig:hokuyo_utm}.
The tests were made with the help of the laser ranger finder shown in figure~\ref{fig:hokuyo_utm}; the moving targets are being tracked since they were firstly detected by the laser without losing the respective ID of the target (figure~\ref{fig:Lidar_Tracking_system_jorge}).

\begin{figure}[H]
	\centering
		\includegraphics[scale=0.42]{Figures/atlascar_hokuyo_utm.png}
	\caption{Laser ranger finder} 
	\label{fig:hokuyo_utm}
\end{figure}

\begin{figure}[H]
	\centering
		\includegraphics[scale=0.283]{Figures/Lidar_Tracking_system_jorge_flip_2.pdf}
	\caption{LASER localization on the environment (left), moving target tracking (right)} 
	\label{fig:Lidar_Tracking_system_jorge}
\end{figure}

\clearpage
%\subsection{Obtenção da ROI}
\subsection{Vision-based pedestrian detection}

Visual detection of humans is a field with an extensive range of applications such as robotics, entertainment, surveillance, care for the elderly and disabled, road safety and others. The benefits of this become obvious when thinking about a car being driven in an urban scenario. If the circumstances are such that the driver is always aware of the surrounding pedestrians, the danger of accidents involving them would most likely decrease dramatically.

\subsubsection{Feature extraction}

Sliding window detection is a technique performed by applying a Detection Window (DW) over the image, evaluating a set of features, and then sliding it to an adjacent place to repeat the process. The DW has constant dimensions (64x128 in most sliding window detection methods), so, in order to find pedestrians with different sizes, the image has to be rescaled and reanalyzed multiple times. An image channel is a representation of the original, where the output pixels are obtained by using linear or non-linear transformations on the input ones, thus preserving the overall image layout. To make this clearer, an illustration of 20 possible random features is shown in figure~\ref{fig:feature_example}. 


\begin{figure}[H]
	\centering
	\captionsetup{justification=centering}
		\includegraphics[scale=0.7]{Figures/random_feature_example_pedro.png}
	\caption{In this illustration, the black rectangles represent the DW over the 10 channels, and the red rectangles represent examples of random rectangles over which local sums are calculated.} 
	\label{fig:feature_example}
\end{figure}

To extract any number of random features, 5 parameters are necessary to define a rectangle: Channel index, Width, Height, X coordinate of the upper-left corner of the rectangle and Y coordinate of the upper-left corner of the rectangle;

\subsubsection{Multi-scale Image Analysis}

Once the feature extraction from DW's is set, the next step is to build an architecture for full image analysis. There are different types of approach to detect pedestrians at multiple scales, (figure~\ref{fig:ped_detect_multi_scale}).

\begin{figure}[H]
\centering
\captionsetup{justification=centering}
\subcaptionbox{N models, 1 image scale\label{fig:immediate}}{\includegraphics[width=7.2cm]{Figures/different_pedestrian_detection_aproach_a.png}}\quad
\subcaptionbox{1 model, N image scales\label{fig:dense_image_pyramid_PDollar}}{\includegraphics[width=7.2cm]{Figures/different_pedestrian_detection_aproach_b.png}}\\
\subcaptionbox{1 model, $\dfrac{N}{K}$ image scales\label{fig:distal}}{\includegraphics[width=7.2cm]{Figures/different_pedestrian_detection_aproach_c.png}}\quad
\subcaptionbox{$\dfrac{N}{K}$ models, 1 image scale\label{fig:combined}}{\includegraphics[width=7.2cm]{Figures/different_pedestrian_detection_aproach_d.png}}
\caption{Different approaches to detecting pedestrians at multiple scales (Number of scales - N, Reducing factor - K).~\cite{benenson2012pedestrian}}
\label{fig:ped_detect_multi_scale}
\end{figure}

Explaining the images (top to bottom, left to right) each paragraph explains each image.

Assuming that object appearance is invariant to translations in the image, to implement the "N models, 1 image scale" approach, one should train as many models as there are scales, see figure~\ref{fig:immediate}. The number of scales N is usually in the order of $\sim$50 scales.

The analysis is made by rescaling the image multiple times, to find pedestrians with different sizes. This process is called dense image pyramid, (fig~\ref{fig:dense_image_pyramid_PDollar}). A few parameters are needed to build this pyramid. The image is rescaled depending on the number of scales per octave (nPerOct). An octave is the necessary scaling to rescale the image by a factor of 2. So, each downsampled image is rescaled according to equation~\ref{eq:DownScale}. The logic is the same for upsampling (equation~\ref{eq:UpScale}) ~\cite{Silva2013}.

\begin{equation} \label{eq:DownScale}
	DownScale = 2^\frac{-1}{nPerOct}
\end{equation}

\begin{equation} \label{eq:UpScale}
	UpScale = 2^\frac{1}{nPerOct}
\end{equation}

Dollar et al. proposed a new approach for fast pedestrian detections, named "\textit{FPDW}" ~\cite{Dollar2010}. Instead of rescaling the input image N times, they propose to rescale it only $\dfrac{N}{K}$ times, see figure~\ref{fig:distal}. By reducing the number of image resizing and feature computations by a factor of K, the total detection time is significantly reduced.

Recently, Benenson et al. reversed the "\textit{FPDW}" idea. They approximated the feature responses across scales and decided how to adjust a given stump classifier to classify correctly, as if the feature response had been computed at a different scale.


\subsubsection{Classification}

There are multiple Machine Learning (ML) methods, each with its specifications and applications, so, one must choose a method that correctly fits the problem. The adopted method needs to classify between two classes, Pedestrian and Not Pedestrian, needs to handle thousands of features per sample and should also be fairly resistant to over-fitting. The method that best fits these requirements is AdaBoost, and a compact description will be carried out in the following section. The basic idea behind AdaBoost, short for Adaptive Boosting, is that it is possible to generate a very accurate prediction rule, or strong classifier, through the aggregation of rough and moderately inaccurate linear rules, weak classifiers, provided that they perform just slightly better than a random classifier would.A graphic illustration of this is shown in figure~\ref{fig:Lidar_Tracking_system_jorge} \cite{Silva2013}.

\begin{figure}[H]
	\centering
		\includegraphics[scale=0.5]{Figures/aggregation_weak_classifiers_pedro.png}
	\caption{Weak classifiers aggregation} 
	\label{fig:Aggregation of weak classifiers}
\end{figure}

\subsection{Sensor Fusion Information Systems}

The \ac{LIDAR} is used as a primary detection sensor generating ROIs which are projected onto the image.
Multistage schemes can be arranged to combine information from \ac{LIDAR} and camera in the same structure, in \cite{Ludwig2011}, a \ac{LIDAR}-dependent processing stage is used to decrease the complexity and the processing time.

To deal with the problem of object scale variations in images, \cite{Douillard2007}, the classification method was evaluated and compared using several features: geometrical, visual and the combination of both. The authors used \ac{CRF} and a LogitBoost classifier.

\ac{LIDAR} and camera fusion can be performed using centralized or decentralized configuration \cite{Premebida2009}.
 
With the purpose to overcome the limitations of each sensor and to enhance the performance of the overall system, the \ac{LIDAR} and image information is conceptually represented by two strategies:

\begin{itemize}
	\setlength\itemsep{0.5em}
	\item Centralized architecture: \ac{LIDAR} and camera information are combined at the feature level, thus a common/central classification framework is used to process the laser and the image-based features as a single, and jointly, feature vector.
	\item Decentralized architecture: information from each sensor is processed separately, hence the output of each sensor-based module, in the form of a likelihood	or a confidence-score, is combined in the decision level.\cite{Premebida2012}"
\end{itemize}


Broggi et al, approach a system designed to search designated areas (critical areas), in specific situation in urban scenarios (fig.~\ref{fig:Broggi_approach_to_urban_pedestrian})\cite{Broggi2009}. By using both sensors, the two can scan the area if one cannot reach it. The main system characteristics are:

\begin{itemize}
	\setlength\itemsep{0.5em}
	\item quickly detect pedestrians, given the short working range	and the particularly high danger of collision with a pedestrian suddenly appearing behind an obstacle;
	\item detect pedestrians as soon as they appear, even when they	are still partly occluded;
	\item limit the search to specific areas, which are determined by a quick preprocessing;
\end{itemize}


\begin{figure}[H]
	\centering
		\includegraphics[scale=0.2]{Figures/Broggi_approach_to_urban_pedestrian.png}
	\caption{Shadow area locations\cite{Broggi2009}.} 
	\label{fig:Broggi_approach_to_urban_pedestrian}
\end{figure}

Fusion can provide a quick and robust detection in case of suddenly appearing pedestrians. A list of areas, in which a pedestrian may appear, are provided by the laser scanner, whereas the camera is able to detect the pedestrians, even when they are not yet visible to the laser scanner. 

Pedestrians may not be detected by a laser scanner positioned in the front bumper but can be detected using vision, even if partially occluded, see figure~\ref{fig:Broggi_approach_to_urban_pedestrian_1}\cite{Broggi2009}.


\begin{figure}[H]
	\centering
		\includegraphics[scale=0.2]{Figures/Broggi_approach_to_urban_pedestrian_1.png}
	\caption{Pedestrian partially hidden by parked cars \cite{Broggi2009}.} 
	\label{fig:Broggi_approach_to_urban_pedestrian_1}
\end{figure}

%\subsection{Métodos de Calibração}
\subsection{Calibration Methods}

The calibration is an essential way to determine the geometric transformation relationship between two coordinates, combining laser scanner and the image coordinates with the vehicle coordinates, so that in the end, the laser points world coordinates could be associated with image pixel points.

Li et al. designed a rectangular frame with one diagonal connected~\cite{Li}, shown in figure~\ref{fig:calibration_method_1_a}. A bracket on the frame bottom corner was used to hold the rectangular frame perpendicular to the ground surface. This method allows calibration even when laser is not parallel to the ground, derived from its shape.

\begin{figure}[H]
\centering
\begin{subfigure}{.5\textwidth}
	\centering
	\includegraphics[scale=0.7]{Figures/MetodoCalibracao.jpeg}
	\caption{}
	\label{fig:calibration_method_1_a}
\end{subfigure}%
\begin{subfigure}{.5\textwidth}
	\centering
	\includegraphics[scale=0.25]{Figures/MetodoCalibracao_explanation.jpeg}
	\caption{}
	\label{fig:calibration_method_1_b}
\end{subfigure}
\caption{Calibration Tool Example: Rectangular frame with one diagonal connection       ~\cite{Li}}
\end{figure}


Guan et al. developed a special right angled isosceles triangle board that allows automatic calibration ~\cite{Guan2009}. Experimental tests on running vehicle showed that the system could avoid interference, ambiguity and display the driving environment including obstacles, vehicles, pedestrians, figure~\ref{fig:CHAO_calibration_method}.

\begin{figure}[H]
	\centering
		\includegraphics[scale=0.47]{Figures/CHAO_calibration_method.pdf}
	\caption{Feature points extraction and calibration processing \cite{Guan2009}.} 
	\label{fig:CHAO_calibration_method}
\end{figure}

%Supposing the scanning plane of the laser scanner intersects the rectangular frame at points $P_L$, $P_M$ and $P_R$; the line $D_3P_MD_1$ is perpendicular to $C_1C_2$; the line $P_MD_2$ is perpendicular to $C_1C_4$; as shown in Fig~\ref{fig:calibration_method_1_b}. Although the intersected point PM is invisible, its place can be revealed indirectly by geometric knowledge:
%
%\begin{equation} \label{eq:PMD3}
%	\cos(\theta) = \frac{[C_1D_1]}{[P_LP_M]} = \frac{[P_LP_M] + [P_MP_R]}{[C_1C_2]}
%\end{equation}
%\begin{equation} \label{eq:PMD2}
%	[P_MD_2] = [C_1D_1] = \frac{[P_LP_M]}{[P_LP_M] + [P_MP_R]}[C_1C_2]
%\end{equation}
%\begin{equation} \label{eq:PMD1}
%	[P_MD_1] = [C_1D_2] = \frac{[P_MP_R]}{[P_LP_M] + [P_MP_R]}[C_1C_4]
%\end{equation}
%
%Where the lengths of $C_1C_2$ and $C_1C_4$ are measured directly and the lengths of $P_LP_M$ and $P_MP_R$ are computed from the coordinate information of $P_L$, $P_M$, $P_R$ in the laser scanner coordinate." \cite{Hao_LI_calibration_tool}\\

%Another way to perform calibration, is to use Zhang and Pless method \cite{Zhang_and_Pless}. "The calibration is based on observing a planar checkerboard pattern and solving for constraints between the “views” of a planar checkerboard calibration pattern from a camera and laser range finder. We give a direct solution that minimizes an algebraic error from this constraint, and subsequent nonlinear refinement minimizes a re-projection error."\cite{Zhang_and_Pless} Figure ~\ref{fig:MetodoCalibracao_Zhang} shows a scheme of the calibration problems considered.
%
%\begin{figure}[H]
%	\centering
%		\includegraphics[scale=0.5]{Zhang_calibration_method.png}
%	\caption{A planar calibration pattern is posed in the both views of the camera and the laser range finder.} 
%	\label{fig:MetodoCalibracao_Zhang}
%\end{figure}


%%\subsection{Critérios seleção de possíveis peões}
%\subsection{LIDAR Pedestrian Segmentation}
%

\subsection{Discussion}

Despite existing some work in both subjects (vision pedestrian detection and pedestrian LIDAR-based tracking) outside \ac{LAR}, it was decided to take advantage of the "Visual Pedestrian Detection using Integral Channels for ADAS", by Pedro Silva~\cite{Silva2013} and "Target tracking using laser range finder with occlusion", by Jorge Almeida~\cite{Almeida2010} algorithms.

It was used Pedro Silva algorithm, because full algorithm was provided.

Despite Jorge Almeida algorithm being recent (2010), it has already been cited in some master thesis. This demonstrates it's value. For this reason, this algorithm was used in this project.


\cleardoublepage
\chapter{Experimental Setup}
%	Descrição do sistema onde se aplica ou desenvolve o trabalho,
%    do ponto de vista hardware e/ou software, o ROS, etc.

%\section{System description}

The programming language used was C++ under Linux platform and, in addition to this, a set of indispensable development tools were utilized; first section will focus on existing software tools and AtlasCar hardware, necessary for this project; second section will focus on used calibration methods, enabling the extraction of extrinsic camera parameters.

In this project two different types of algorithms were fused, an image pedestrian detection and a \ac{LIDAR} target tracking, in order to detect pedestrians with a better performance than image pedestrian detection alone. Because the laser is invisible, laser points cannot be determined directly and they can only be reached indirectly, calibration is needed to know the location of sensors frameworks.

\section{Software and Hardware Tools}

\subsection{Software}

\subsection*{Robotic Operation System}

The Robot Operating System (ROS) provides a flexible framework that is designed to create and maintain robot software. It has a collection of tools, libraries, and conventions that aim to simplify the task of creating complex and robust robot behavior across a wide variety of robotic platforms, which helps to handle the output of different types of sensors, such as cameras, lasers, actuators, contacts and other common elements in robotics environments.

ROS allows the elaboration of an infrastructure that can communicate with any running process, establishing a communication between different software modules (nodes). This communication works in three steps: first, a node advertises a topic, then, once that topic is advertised, the same node is able to publish messages on that topic, and finally, those messages can be listened by any node that subscribes to that same topic.
Such messages can be of any kind, from simple strings of characters, visual or laser data to special messages with several parameters. Figure~\ref{fig:ROS_Architecture} illustrates a very simple ROS communication architecture.

\begin{figure}[H]
	\centering
		\includegraphics[scale=1.0]{Figures/ROS_basic_concepts_comunication_1.png}
	\caption{Simple ROS Scheme communication architecture} 
	\label{fig:ROS_Architecture}
\end{figure}

It is easy to understand that this structure for information exchanging has a great potential when applied to robotics since a uniform and standardized communication facilitates the development of complex multi-sensor applications.
Another important feature in ROS is that it provides the possibility to record several messages (rosbags) that can be replayed later. This allows data to be collected in real scenarios, without worries, and be post-processed in a laboratory environment with the same conditions as those on the field. It also contains a visualizer, known as RViz, which is a very useful program because and in this case, it allowed the visualization of laser data, facilitating 3D points extraction for calibration process.\\

For these reasons, the work presented here was developed in ROS.


\subsection*{OpenCV}

OpenCV is a popular open source computer vision library written in C, C++ and Phyton, that was designed for high computational efficiency and with a strong focus on real-time image processing applications. In this case OpenCV was used to extract extrinsic camera parameters, make image manipulation (threshold, draw rectangles, lines, labels), resize images, elaborate calculation, project laser points on image and write data on hard drive.

%\subsection*{Matlab}
%
%MATLAB® is a high-level language and interactive environment for numerical computation, visualization, and programming. 
%
%It was used to calculate algorithm performance metrics, see "Results" chapter, and the cameraCalibrator app was used to estimate camera intrinsics and lens distortion parameters, so that can be obtained a rectified image, see "Calibration Methods" section.

\subsection{Hardware}


\subsection*{Camera}

First, for in-lab testing, a logitech c310 webcam was used, (figure~\ref{fig:logitech_c310_webcam}), with \ac{HD} video capture at 720p (1280 x 720) with 30 \ac{FPS}, perfect for steady footage but inappropriate when making a non-steady video, for example, with all the shaking in a moving vehicle, the result is a blurred video instead of a focused and clear image. This happens because the camera was developed for steady web video capture.

The car is equipped with a Bumblebee® XB3, (figure~\ref{fig:Bumblebee_XB3}), a 3-sensor multi-baseline IEEE-1394b stereo camera designed for improved flexibility and accuracy. The extended baseline and the high resolution (up to 1280 x 960) provide higher precision at longer ranges, having a maximum \ac{FPS} of 16. It also allows the moving video to be smooth, making it appropriate to be the top camera in AtlasCar.

\begin{figure}[H]
	\centering
	\begin{subfigure}{.5\textwidth}
		\centering
			\includegraphics[scale=0.24]{Figures/logitech-hd-webcam-c310-gallery-2.png}
		\caption{Logitech HD c310 webcam \cite{Logitech}} 
		\label{fig:logitech_c310_webcam}
	\end{subfigure}%
	\begin{subfigure}{.5\textwidth}
		\centering
			\includegraphics[scale=0.13]{Figures/BBXB3_white_background_large.jpg}
		\caption{Point Grey Bumblebee® XB3 \cite{XB3}} 
		\label{fig:Bumblebee_XB3}
	\end{subfigure}
\caption{Hardware Cameras used}
\end{figure}


\subsection*{LIDAR}

The Laser used was a Sick LMS 151, (figure~\ref{fig:Sick_LMS_151}) which is small and light-weight and has a maximum scanning range of 50 m, supply voltage range from 10.8 to 30 V DC, allowing the use of 12 V (car battery voltage) to power up the laser, a scanning frequency of 50Hz, and IP-67 protection class, enabling it to outdoor use.

\begin{figure}[H]
	\centering
		\includegraphics[scale=0.1]{Figures/atlascar_LMS151.jpg}
	\caption{Sick LMS 151 \cite{Sick}} 
	\label{fig:Sick_LMS_151}
\end{figure}

%\section{Calibration Object}
%
%A triangle shaped object, colocar figura, was used to enable automatic calibration between the camera and the lasers, this particularly shape allows the software to know where the laser is focus on the white board. The width varies along the height, making all the values different 



\section{Calibration Methods}

The calibration process allows the user to know the precise location of all the object frames from a reference frame perspective.

\subsection{Manual calibration tool}

\subsection*{Sensors on a tripod}

In this phase, a manual calibration was tried to see how complex the process was and hopefully later on, try to calibrate the XB3 camera with the Sick lasers, with a similar procedure. Figure ~\ref{fig:LIDAR_cam_assembly} illustrates the assembly made between \ac{LIDAR} and the logitech webcam on a tripod, making it light, steady on every ground and easy to attach and detach. This allows gathering both data with fix translation and rotation between the two.

\begin{figure}[H]
	\centering
		\includegraphics[scale=0.30]{Figures/LMS151_and_logitech_cutted.pdf}
	\caption{Sick LMS 151 and Logitech HD c310 webcam tripod assembly} 
	\label{fig:LIDAR_cam_assembly}
\end{figure}

The manual calibration for this set consists of 3 traffic cones displaced on the environment, (fig.~\ref{fig:ManualCalibration}) so that only the top tip of the cone is intersected by the laser, making it easy to extract the correspondent image points (px, py), but also the positions of the cones (x, y, z); because \ac{LIDAR} data is a 2D laser-ranger finder, hence, z=0.

Once the images are being acquired by a camera, this sensor also needs to be calibrated.

Calibration is the estimation of a camera’s intrinsic and lens-distortion parameters. Having the lens-distortion parameters, it was able to correct image distortion by transforming the image into a standard coordinate system. Matlab cameraCalibrator app was used to estimate camera intrinsic and lens distortion parameters, so that a rectified image could be obtained. The final output image is a rectified image.


\begin{figure}[H]
	\centering
		\includegraphics[scale=0.18]{Figures/Calibracao_manual_3,8m.png}
	\centering
		\includegraphics[scale=0.18]{Figures/Calibracao_manual_6m.png}
	\centering
		\includegraphics[scale=0.18]{Figures/Calibracao_manual_10m.png}
	\caption{Manual Calibration Example: 3,8m / 6m / 10m from the LIDAR} 
	\label{fig:ManualCalibration}
\end{figure}

After obtaining several image points (extracted from a rectified image) and object world points, they were saved in separate files.

For a rectified image, the distortion coefficient vector is considered null.

By using the OpenCV function, solvePnP, the object pose can be found from 3D (object point) and 2D (image point) point correspondences. Figure~\ref{fig:SolvePnP} illustrates how the function input and output works.


\begin{figure}[H]
	\centering
		\includegraphics[scale=0.3]{Figures/Calibrate_camera_extrinsic_parameters.pdf}
	\caption{SolvePnP, OpenCV function (Inputs above / outputs below)} 
	\label{fig:SolvePnP}
\end{figure}

A rotation vector is the most convenient and compact representation of a rotation matrix (since any rotation has only 3 degrees of freedom); by using Euler–Rodrigues formula the vector can be converted into a matrix, for example using OpenCV Rodrigues function. 

Having the rotation matrix and the translation vector, a [R|t] matrix can be created; this is a 4x4 matrix composed by a 3x3 rotation matrix and a translation vector; matrix~\ref{eq:RT_matrix} illustrates how the matrix is composed, being the r's the rotation matrix and t's the translation vector.
To extract the position of the camera, from the \ac{LIDAR} point of view, the $^{C}[R|t]_{_{FB}}$ (Referential transformation of C (Camera) to FB(Front Bumper)) matrix needs to be inverted.
%
\begin{equation}\label{eq:RT_matrix}
^{C}[R|t]_{_{FB}} = \begin{bmatrix}
\textit{$r_{11}$} & \textit{$r_{12}$} & \textit{$r_{13}$} & \textit{$t_1$} \\ 
\textit{$r_{21}$} & \textit{$r_{22}$} & \textit{$r_{23}$} & \textit{$t_2$} \\
\textit{$r_{31}$} & \textit{$r_{32}$} & \textit{$r_{33}$} & \textit{$t_3$} \\
0          & 0          & 0          & 1          \\ 
\end{bmatrix}
\end{equation}
%
%Having the rotation and translation matrix, the [R|t] matrix is saved in a extern file. The position of the camera, from the \ac{LIDAR} point of view can be obtained by inverting the [R|t] matrix.

%\subsection{Between LIDAR and a Single Camera}
\subsection*{Sensors onboard the car}

After knowing what was necessary to calibrate a laser and a camera, the next phase was to calibrate the XB3 camera and the front left bumper Sick laser, Fig.~\ref{fig:XB3_and_Front_Bumper_localization}.

The intrinsic parameters of the camera were calculated using \cite{Toolbox}, whose intrinsic values obtained are summarized in Table~\ref{table:AtlasCar_xb3_Intrinsic}.

\begin{table}[h]
\centering
\caption{Intrinsic camera parameters: K} 
	\begin{tabular}{l l}
	\hline
	Focal Length & \multicolumn{1}{c}{$fc : [1027.941 ; 1027.751]$} \\ 
	Principal Point & \multicolumn{1}{c}{$cc : [658.264 ; 486.631]$} \\ 
	Distortion Coefficients & \multicolumn{1}{c}{$kc : [-0.366; 0.206; 0.000060; -0.00016; -0.0784]$} \\ \hline
	\end{tabular}
\label{table:AtlasCar_xb3_Intrinsic}
\end{table}

The calibration was done by applying the same principles as described earlier, with traffic cones, modifying their height, so that only the tip could be intercept by the laser.


Figure~\ref{fig:AtlasCarManualCalibration} shows an example of the experimental calibration which is, how to extract the laser xyz coordinates and the correspondent image taken, to manually calibrate the \ac{LIDAR} and XB3 camera. 

\begin{figure}[H]
	\centering
		\includegraphics[scale=0.75]{Figures/AtlasCar_xb3_sick_positions.pdf}
	\caption{XB3 and Front Bumper laser localization in the car} 
	\label{fig:XB3_and_Front_Bumper_localization}
\end{figure}

\begin{figure}[H]
	\centering
		\includegraphics[scale=0.36]{Figures/rviz_manual_calib_atlascar_2.pdf}

	\caption{AtlasCar Manual Calibration Example} 
	\label{fig:AtlasCarManualCalibration}
\end{figure}

In some cases, multiple points correspond to one cone (figure~\ref{fig:AtlasCarManualCalibration}), in these cases, a mean was made to sort out the correspondent middle point of the tip of the cone.

Using the object points, image points and the camera intrinsic matrix, the $^{C}[R|t]_{_{FB}}$ matrix could be obtained with the function described earlier (figure~\ref{fig:SolvePnP}). The following matrix (~\ref{eq:result_RT_matrix}), necessary to make a rigid correspondence between the laser scanner and the camera reference system, was obtained, being the translation vector components in meters.

\begin{equation}\label{eq:result_RT_matrix}
^{C}[R|t]_{_{FB}} = \begin{bmatrix}
-0.0184  & -0.9998  &  \multicolumn{1}{r}{-0.0027}  & 0.2337\\
-0.1384  & \multicolumn{1}{r}{0.0052}  & -0.9904  &  \multicolumn{1}{r}{0.8455}\\
\multicolumn{1}{r}{0.9902}  & -0.0179  & -0.1384  &  \multicolumn{1}{r}{1.9781}\\
\multicolumn{1}{r}{0} &  \multicolumn{1}{r}{0} & \multicolumn{1}{r}{0} &  \multicolumn{1}{r}{1}\\
\end{bmatrix}
\end{equation}

Knowing these parameters, the location of the camera can be virtually shown in Rviz (figure~\ref{fig:AtlasCar_xb3_location_world}).

\begin{figure}[H]
	\centering
		\includegraphics[scale=0.28]{Figures/Disposicao_xb3_mundo.pdf}

	\caption{Camera view, laser view and XB3 location frame in world frame} 
	\label{fig:AtlasCar_xb3_location_world}
\end{figure}
%\section{Semi-Automatic calibration tool development}
%
%\subsection{AtlasCar}

\subsection{Automatic calibration tool}

Despite not being able to make a successfully automatic calibration tool, an algorithm was developed in order to provide it. The object shape used was a trapezoid (figure~\ref{fig:automatic_calibration_method_suport}); it has the following parameters: 1m max width, 0.2m min width, 0.8 height. Figure~\ref{fig:automatic_calibration_method_explication} shows the laser point extraction process, where the red dotted line illustrates a possible laser read and the "A" area, the consequent used values to extract the percentage height to withdraw points on the image. 

The region was painted white, with the edges in brown tape, to easily distinguish the object from the environment.

The idea was to set the laser parallel to the floor, reducing inclination problems, thus enabling extraction of parallel points on the real image.

\begin{figure}[H]
	\begin{subfigure}{.58\textwidth}
		\centering
		\captionsetup{justification=centering}
			\includegraphics[scale=0.05]{Figures/automatic_calibration_method_suport_front.jpg}
			\includegraphics[scale=0.05]{Figures/automatic_calibration_method_suport_back.jpg}
		\caption{(front on the left / back support on the right)} 
		\label{fig:automatic_calibration_method_suport}
	\end{subfigure}
	\begin{subfigure}{.4\textwidth}
		\centering
			\includegraphics[scale=0.425]{Figures/automatic_calibration_method_explication.pdf}
		\caption{Point extraction illustration}
		\label{fig:automatic_calibration_method_explication}
	\end{subfigure}
	\caption{Object used to automatize the calibration process}
	\label{fig:automatic_calibration_method}
\end{figure}

The initial object angle ($\alpha$) is 66.37. To withdraw the object width through laser data, "$laserwidth$", "$y$" can be calculated through equations~\ref{eq:x_laser_extraction} and~\ref{eq:y_laser_calculation}. To transfer laser position ("$y$") to image coordinates, the value was converted to percentage. This allowed that the laser location could be extrapolated, despite the size of the object, equation~\ref{eq:laserheighttotop_calculation}.

\begin{equation}\label{eq:x_laser_extraction}
x = \frac{laserwidth-0.3}{2}
\end{equation}

\begin{equation}\label{eq:y_laser_calculation}
tan(\alpha)= \frac{y}{x}
\end{equation}

\begin{equation}\label{eq:laserheighttotop_calculation}
\% laser height to top= \frac{y}{0.8}
\end{equation}

After calculating the laser position, the algorithm grabs an image and scans it, looking for the white board position and extracting its height in pixel, then the $\%laserheighttotop$ variable is used to choose the pixel line where the laser intercepts with the white board. Where "$B$" stands for Board, the maths made is:

\begin{equation}\label{eq:height_image_extraction}
ImageLaserLocation = (BBottomPixel - BTopPixel) \times \%laserheighttotop
\end{equation}

Knowing the vertical pixel where the laser intercepts the board, and the location of the board to make it stand out, the edge pixel can be obtained. These points, image (2D) and laser (3D), are used with the same principle as previous manual calibrations, to extract extrinsic camera parameters.

Figure~\ref{fig:automatic_bad_calibration_example} shows that this calibration method was not reliable.
In this process, the calibration board needs to be completely parallel to the car, so that a perfect point extraction can be obtained. Not having a perfect board displacement, the extrinsic camera matrix was obtained with errors. 

\begin{figure}[H]
	\centering
		\includegraphics[scale=0.239]{Figures/automatic_calibration_example.pdf}

	\caption{Automatic Calibration Result Image} 
	\label{fig:automatic_bad_calibration_example}
\end{figure}
   
\subsection{Discussion}

Comparing the previous results of both (manual and automatic) $AtlasCar$ calibration methods, the "Manual" was the chosen one to continue this work. The transformation matrix used on the results above, was $^{FB}[R|t]_{_{C}}$. 

In table~\ref{table:Calibration_result_comparision}, the rotation matrix (3x3) was converted to RPY angles for a better comparison; "Automatic" calibration method has a huge difference in translation when compared to the "Manual" method. The $AtlasCar$ system already had a calibration, but to prevent possible twisting in camera and lasers, the calibration was made again. "Manual" and "Existing" values are very similar, leading us to believe that the tests were correct.

\begin{table}[h]
\centering
\caption{Calibration results (Manual vs Automatic)} 
	\begin{tabular}{lC{1.5cm}C{1.5cm}C{1.5cm}|C{1.5cm}C{1.5cm}C{1.5cm}}
	\cline{2-7}
		& \multicolumn{3}{c|}{RPY angles} & \multicolumn{3}{c}{Translation}\\\cline{2-7}
					   & \textbf{Roll} & \textbf{Pitch} & \textbf{Yaw} & \textbf{X} & \textbf{Y} & \textbf{Z}  \\\cline{1-7}
	\multicolumn{1}{l}{Existing} & \multicolumn{1}{c}{-1.7104} & \multicolumn{1}{c}{-0.0000} & \multicolumn{1}{c|}{-1.5708} & \multicolumn{1}{c}{-1.8405} & \multicolumn{1}{c}{0.2400} & \multicolumn{1}{c}{1.1170}\\\hline
	\multicolumn{1}{l}{Manual} & \multicolumn{1}{c}{-1.7097} & \multicolumn{1}{c}{0.0027} & \multicolumn{1}{c|}{-1.5892} & \multicolumn{1}{c}{-1.8375} & \multicolumn{1}{c}{0.2646} & \multicolumn{1}{c}{1.1118}\\\hline
	\multicolumn{1}{l}{Automatic} & \multicolumn{1}{c}{-1.7709} & \multicolumn{1}{c}{-0.0059} & \multicolumn{1}{c|}{-1.5769} & \multicolumn{1}{c}{-2.5894} & \multicolumn{1}{c}{0.2380} & \multicolumn{1}{c}{1.8401} \\\hline
	\end{tabular}
\label{table:Calibration_result_comparision}
\end{table}


\cleardoublepage
\chapter{Algorithm for Active Pedestrian Detection}

This chapter describes the different stages needed to achieve Active Pedestrian Detection (APD); figure~\ref{fig:APD_resume} summarizes the process. Pedestrian protection systems can be divided, in general words, in two fields of research: passive and active safety systems.

Active safety systems, of a high interest here, are based on pedestrian detection using sensors onboard the vehicle, and/or on the infrastructure, with the role of predicting and anticipating possible risks of collision~\cite{gandhi2007pedestrian}.


\begin{figure}[H]
	\centering
		\includegraphics[scale=0.37]{Figures/APD_resume.pdf}
	\caption{APD resume} 
	\label{fig:APD_resume}
\end{figure}

\section{Re-Parametrization of Existing Pedestrian Detection Algorithm}

The original pedestrian detection algorithm, by Pedro Silva, subscribes an image published on ROS environment and resizes it to 640x480 for optimal processing~\cite{Silva2013}.

%An adaptation of the existing algorithm had to be made so that even when small images are subscribed, the program could run fast and precise, instead of taking too much time to scan a large image.

The program had to be adapted to work with images of any sizes.

The previous detection process of scanning pedestrians on an image is shown on the left side of figure~\ref{fig:PedestrianAlgorithmScheme}. 

When interpreting the former process, one stage of the diagram can be removed, because when combining \ac{LIDAR} and Camera information, the output is a smaller one-person image, with known size. 

The algorithm uses the dense image pyramid process (figure~\ref{fig:dense_image_pyramid_PDollar}) for pedestrian detection. Because all the algorithm architecture was built around it, the process cannot be switched, but some topics can be modified to work with one-person images.

The dense image pyramid process rescales the image multiple times, while the model keeps intact. Knowing this, the given image needs to be rescaled, for correct pedestrian detection (in this type of process). Given a one-person image to classify, the algorithm does not need to spend too much time scaling the entire area of the initial image (640x480). Instead of that, a smaller image runs multiple times to make sure a pedestrian is well classified. Changing input image sizes, the algorithm running time decreases, allowing a quicker classification.

Understanding that the given image already has a potential pedestrian, the step search was decreased, allowing the image to be swept with more detail. The step search consists of the number of pixels the DW skips, when making an image column scanning- The process continues the same along the lines.

The output result was also modified. After ending the image scan, the original algorithm draws rectangles on different image locations. Now, after ending the potential pedestrian scan, the only output is the result of a single classification, (Ped or nPed). 

The modifications made can be visualized on the right side of figure~\ref{fig:PedestrianAlgorithmScheme}.

\begin{figure}
	\centering
		\includegraphics[scale=0.26]{Figures/Pedestrian_Algorithm_Detection_Scheme_2.pdf}
	\caption{Pedestrian Detection Algorithm Scheme (before on left/ after on the right)} 
	\label{fig:PedestrianAlgorithmScheme}
\end{figure}


\section{Sensor fusion algorithm}

This section describes the process to separate potential pedestrian from the rest of the objects and the development of an application for pedestrian active detection and its explanation.

\subsection{LIDAR Classification of Potential Pedestrian}

After knowing where the objects are, in a \ac{LIDAR} scan, the potential pedestrians need to be organized to be further classified by a more robust algorithm. By applying the \ac{MTT} algorithm\cite{Almeida2010}, IDs can be attributed to different groups of laser points, allowing the objects to be tracked continuously, until they vanish from the laser radar.

Initially, only the object width was used to sort out the potential pedestrians from the \ac{LIDAR} data. Figure~\ref{fig:Potential_pedestrian_search} shows that only with this condition the number of potential pedestrians was narrowed to a few, compared to the amount of the objects found by the \ac{LIDAR}. The blue rectangle shows the origin of the $AtlasCar$ coordinate frame, the black labels correspond to object IDs.

\begin{figure}
	\centering
	\captionsetup{justification=centering}
		\includegraphics[scale=0.325]{Figures/Screenshot_from_2014-04-10_07_01_23.png}
	\caption{Potential Pedestrian Search Example shown in RViz, with the correspondent image on the lower left} 
	\label{fig:Potential_pedestrian_search}
\end{figure}

%\clearpage



\subsection{Active Pedestrian Detection}

The Active Pedestrian Detection algorithm consists of 4 stages: candidate list creation, candidate classification, candidate visualization and candidate elimination. These stages are explained below in the same order. 

The application starts subscribing the \ac{MTT} data and the camera color rectified image at the same time. The image is only used after the \ac{MTT} data is received and processed. Until that moment the image is only shown, as if the application is a mere camera info and a rectified image reader. 

\subsubsection{Candidate List Creation} %Criação de candidatos

After making a fast-low resolution classification with the \ac{LIDAR} data, the candidates are then stored with their properties: ID, distance to \ac{LIDAR} in a straight line, left and right world point of the candidate, correspondent image in a vector and classification of the candidate. The ID is the only fixed value, until the candidate is removed. 

The candidates vector is cleaned before starting the program.

Figure~\ref{fig:Sensor_fusion_sequeme_LIDAR} illustrates the \ac{LIDAR} candidate list creation process, when new \ac{LIDAR} data is subscribed.

\begin{figure}[h]
	\centering
		\includegraphics[scale=0.26]{Figures/APD_Candidate_Creation.pdf}
	\caption{Sensor Fusion LIDAR Process Scheme} 
	\label{fig:Sensor_fusion_sequeme_LIDAR}
\end{figure}

After the \ac{MTT} data subscribed, the search for the ID in the stored vector and the Target data begins. Initially the candidates vector is null, therefore, there is a need to create a new candidate.

For the \ac{MTT} target to be considered a potential pedestrian, the object needs to have a width considered in this range [0.15;0.75] (m).

After scanning all the incoming Target data, the candidate vector is sorted by distance to \ac{LIDAR} in a straight line, so that the candidate closer to the \ac{LIDAR} can be classified first, for safety reasons.

After receiving multiple times MTT data, IDs start to match the ones in the candidates vector. After being found, the candidate vector is refreshed. Candidate location is updated. 

If the candidate classification is Non-Pedestrian and its velocity is non zero, the classification changes to "Unknown". Normally, an object does not have motion. This fact helps the candidate classification process to reclassify and make sure the object is not a "Pedestrian".

If an object has a width larger than 1 meter, the candidate is marked for deletion. This parameter was tested and no pedestrian was ruled out. For example, if one candidate walks along a wall, the wall can be added as a new candidate. However, the candidate is walking off, the width of the wall starts to grow. If there had not been conditions to control this factor, the wall could be marked continuously. This problem is illustrated in figure~\ref{fig:Wall_problem.png}. Initially the pedestrian number 19767 walk left to right on the sidewalk, making object number 19662 appear. This was corrected using the "> 1 meter" parameter.

\begin{figure}[H]
	\centering
		\includegraphics[scale=0.45]{Figures/Wall_problem.png}
	\caption{Sensor Fusion LIDAR-Camera Scheme} 
	\label{fig:Wall_problem.png}
\end{figure}

\subsubsection{Candidate Classification} %Classificação de candidatos

When finished sorting the candidates vector, the classification process checks if there are any candidates available (figure~\ref{fig:Sensor_fusion_sequeme_IMAGE}); if so, by knowing the location of the candidate on the image, one portion is cropped and attached to the candidate image topic, in the candidate vector; if the candidate already has a classification different from "Unknown", it means that it already has an image of itself, so the extract and classification process is skipped, the same is done until candidates run out.

Classification process runs every time an image is received.

\begin{figure}[H]
	\centering
		\includegraphics[scale=0.26]{Figures/Sensor_fusion_squeme_IMAGE_only_3.pdf}
	\caption{APD candidate classification process} 
	\label{fig:Sensor_fusion_sequeme_IMAGE}
\end{figure}

Afterwards, a thread creation enables the scan to continue, instead of spending time waiting for the verdict. The multi-thread process helps the program to classify different candidates at the same time, running the classifications alongside the application. 

The thread calls the Single Pedestrian Detection algorithm, giving it the candidate scaled image for classification. When finished, the candidate classification status is changed to Pedestrian or Non-Pedestrian depending on the verdict of the algorithm.

Candidate classification is changed to "Processing" until classification thread terminates, see figure~\ref{fig:different_types_classification} to acknowledge the different types of visualized boxes. 

%\begin{figure}[h]
%	\centering
%	\captionsetup{justification=centering}
%		\includegraphics[scale=0.675]{Figures/different_types_of_classification.pdf}
%	\caption{Different types of Candidate classifications, Unknown, Processing, Pedestrian, Object (Non-Pedestrian), left to right respectively.} 
%	\label{fig:different_types_classification}
%\end{figure}


 
%For the second run and so on, \ac{MTT} data is subscribed once again, but now, in the search for the same id, some id's have been found, then the location of the candidate is updated to current status. Some condition had to be added, just like, some object that was classified has Non-Pedestrian and continuous to move, the classification is changed to "Unknown", so that the object can be classified once again by the Pedestrian Detection Algorithm, because the classification could be mistaken, for example, a person that hasn't been well classified. 



\subsubsection{Candidate Elimination} %Eliminação de candidatos

While the candidate remains in the LIDAR sight, the candidate vector continues to be updated.
When the candidate walks out of the LIDAR sight zone, its vector is eliminated. This process needs to occur, or the candidates vector would grow continuously, until stopping the program due to memory leak.

Another reason for elimination to occur, is that the classification can now focus on classifying the objects that can be a real threat.

The most complicated case for elimination, is when a candidate is still being processed by the classification thread and has already left the LIDAR sight zone. In this case, the thread is forced to shut down, to enable candidate elimination without damaging the candidates vector.

Elimination process is needed, because it allows a better candidates vector management and increases the overall algorithm performance. Figure~\ref{fig:APD_Candidate_Elimination} summarizes all the elimination process.
%Enquanto o candidato for visualizado pelo LIDAR, a sua lista (posição) continua em update constante. 
%Quando o LIDAR deixar de visualizar o candidato, este é eliminado. O caso mais complicado de eliminição é quando esse mesmo candidato se encontra, ainda, a ser processado pelo algoritmo de detecção.
%Nesse caso, a thread criada anteriormente é forçada a ser eliminada, para não haver uso excessivo de threads que já não são precisas. 

\begin{figure}[H]
	\centering
	\captionsetup{justification=centering}
		\includegraphics[scale=0.26]{Figures/APD_Candidate_Elimination.pdf}
	\caption{APD candidate visualization process} 
	\label{fig:APD_Candidate_Elimination}
\end{figure}

\subsubsection{Candidate Visualization} %Visualização de candidatos

The candidate visualization only is needed for: debug process, driver assistance and a qualitative visualization of the classification results.

For autonomous driving, this part is not important, because no driver is onboard to visualize the classification, persisting the essential topics (3 previous ones).

%A parte visual serve para questões como: debug; alerta do condutor do veiculo; visualização qualitativa de resultados. Para condução autonoma a parte visual não é muito importante, pois não existe nenhum condutor a bordo da viatura, ficando só a predurar os tópicos essenciais (os 3 anteriores).

The candidate is classified while remaining in the image region. For different classifications, different visuals are needed, so it was proposed distinct color rectangles (\ac{BB}), to highlight classification. Whether the candidate has the Pedestrian or Non-Pedestrian status, this is highlighted on the seen image with a green or a red \ac{BB}, respectively. The "Unknown", is also highlighted with a red \ac{BB}, see figure~\ref{fig:different_types_classification}.

%Desde que o candidato se encontre dentro da area da imagem, este é classificado. Para a classificação visual de objectos recorre-se ao uso de rectangulos. Cada classificação tem a sua caracteristica visual, ver figura \ref{fig:different_types_classification}.


\begin{figure}[h]
	\centering
	\captionsetup{justification=centering}
		\includegraphics[scale=0.65]{Figures/different_types_of_classification.pdf}
	\caption{Different types of Candidate classifications, Not Processed (Unknown), Processing, Pedestrian, Object (Non-Pedestrian), left to right respectively.} 
	\label{fig:different_types_classification}
\end{figure}



\cleardoublepage
\chapter{Experiments and Results}

To test the application, experiments were made in three different places: the first one inside the \ac{LAR}, then the tripod assembly facing the Mechanical Engineer building and finally with the $AtlasCar$ facing a sidewalk. The test was moved to the outside in order to check if it had a good performance while some people were walking side by side and crossing each other.

To have quantitative results one needs to create a ground truth. It defines the ideal output of a perfect detection. Therefore, it was obtained by manual selection. Actually, every single algorithm output image needs to have a correspondent ground truth image, to enable correct result extraction.

\section*{Classification performance metrics} 

The classification performance considered on this test and the principles used to classify TP, TN, FP, FN, are explained in Table~\ref{table:Results_performance_metrics} ~\cite{Confusion_Matrix}.

\begin{table}[h]
\centering
\caption{Terminology and definitions for performance metrics} 
	\begin{tabular}{l |m{2cm}|m{2cm}| l}
	 &  Ground truth & Algorithm Output &  \\\hline
	True Positive (TP) &  Present & Found  & equiv. to hit \\ 
	True Negative (TN) &  Not Present & Not Found  & equiv. to correct rejection \\ 
	False Positive (FP) &  Not Present & Found & equiv. to false alarm \\
	False Negative (FN) &  Present & Not Found  & equiv. to miss \\\hline
%	& & & \\
	True Positive Rate (TPR) & & & TPR = TP/(TP + FN) \\
	True Negative Rate (TNR) & & & TNR = TN/(TP + FN) \\
	False Positive Rate (FPR) & & & FPR = FP/(FP + TN) \\
	False Negative Rate (FNR) & & & FNR = FN/(FN + TP) \\\hline
	\end{tabular}
\label{table:Results_performance_metrics}
\end{table}

Figure~\ref{fig:TP_FP_TN_FN_example_outcome} displays the different types of classification, as described in Table~\ref{table:Results_performance_metrics}, while figure~\ref{fig:TP_FP_TN_FN_example_gt} shows the ground truth where the known real pedestrians are highlighted for comparison. 

The list below describes each \ac{BB}, drawn in figure~\ref{fig:TP_FP_TN_FN_example_outcome}, left to right. Each topic explains the \ac{BB} classification, comparing it to the ground truth image, figure~\ref{fig:TP_FP_TN_FN_example_gt}:

\begin{description}[style=unboxed, labelwidth=1.0cm]
	\setlength\itemsep{0.5em}
	\item[\namedlabel{itm:rect1}{BB 1.}] A column highlighted by a green \ac{BB}: it means that a pedestrian has been detected in that location, but when compared to the ground truth image, no pedestrian is present in that location, making it a FP classification;
	\item[\namedlabel{itm:rect2}{BB 2.}] The first red \ac{BB} means that no pedestrian was detected; comparing it to the ground truth image, no pedestrians had been detected on both, so the output classification is TN;
	\item[\namedlabel{itm:rect3}{BB 3.}] The pedestrian is correctly highlighted in both images, classification TP;
	\item[\namedlabel{itm:rect4}{BB 4-5.}] Red \ac{BB} resembles the same condition as \ref{itm:rect2}, output classification TN;
	\item[\namedlabel{itm:rect6}{BB 6.}] Last \ac{BB} is red, so outcome is "Not Found". On the other hand, the ground truth has a green \ac{BB}, meaning "Pedestrian Present", making the output classification FN;
\end{description}

\begin{figure}[H]
	\centering
		\includegraphics[scale=0.415]{Figures/8384_outcome.pdf}
	\caption{Example of output image and results of a Pedestrian classification}
	\label{fig:TP_FP_TN_FN_example_outcome} 
		\includegraphics[scale=0.415]{Figures/8384_gt.pdf}
	\caption{Ground truth for the same image as in figure~\ref{fig:TP_FP_TN_FN_example_outcome}} 
	\label{fig:TP_FP_TN_FN_example_gt}
\end{figure}

\clearpage

\section{Experiment 1 - Tripod mounting indoors} 

Actually, in spite of not being big, the \ac{LAR} was very helpful at testing if the candidates classification and tracking were fine. By using the assembly in figure~\ref{fig:LIDAR_cam_assembly} some data was extracted.

The first time the application was processed, non candidates were being selected, this resulted in an image filled with red \ac{BB}s. Each \ac{BB} corresponds to a LIDAR candidate (not selected), see figure~\ref{fig:LAR_inside_initial_testing}. 

The red dots seen in figures ~\ref{fig:LAR_inside_initial_testing}, ~\ref{fig:LAR_inside_testing} and~\ref{fig:LAR_inside_testing_3}, approximately in the center of the image, are the projected laser points.

\begin{figure}[H]
	\centering
		\includegraphics[scale=0.4]{Figures/LAR_inside_sensor_fusion_testing_2.png}
	\caption{LIDAR objects shown, without any previous selection} 
	\label{fig:LAR_inside_initial_testing}
\end{figure}

After running all 4 stages of the program, the massive amount of \ac{BB} disappeared, remaining only Potential pedestrians for classification, see figure~\ref{fig:LAR_inside_testing}. Some of the candidates are still being processed (status: PROCESSING). At the same time, the algorithm has already processed some objects and one pedestrian.

Observing that the program was running as expected (not excluding any pedestrians from the LIDAR search), some modifications were made concerning the visualization tool. 

The font color was changed to black as well as the font type of the status on top of the rectangles, when the candidates are not pedestrians. Candidate ID was added to the status, making it easy to extract the time of classification in the terminal, for the initial debug. See changes in figure~\ref{fig:LAR_inside_testing_3}.

\begin{figure}[H]
	\centering
		\includegraphics[scale=0.545]{Figures/LAR_inside_sensor_fusion_testing_1.png}
	\caption{Differences between candidates, Object/Pedestrian/Processing} 
	\label{fig:LAR_inside_testing}
\end{figure}

The candidate time of classification is directly related to LIDAR distance. The farther they were from the \ac{LIDAR}, the faster they were to classify, because the candidate image gets smaller. 

One particular case happens when a person, instead of walking towards the \ac{LIDAR}, walks in the opposite direction. This lead to a much longer classification processing time than it was used to. 

Bearing in mind these results, a modification had to be made. Actually, before sending a candidate image to the Single Pedestrian Detection Algorithm, depending on the actual size of the image, a rescale had to be made. For example, if the image has a pixel height superior to 250, the image is rescaled to 250 pixels. The image rate is preserved.

The rescaling process allows candidates which are close to the \ac{LIDAR}, to be classified faster than they used to be. 

Figure~\ref{fig:LAR_inside_testing_3}, shows the last modifications made, the different font type, color and added IDs. On average, a candidate image this size (Pedestrian: 189) would take 10 seconds to classify; with the rescale modification the classification only took 3.27 seconds.

\begin{figure}[H]
	\centering
		\includegraphics[scale=0.545]{Figures/LAR_inside_sensor_fusion_testing_3.png}
	\caption{Figure shows last modification made to the algorithm} 
	\label{fig:LAR_inside_testing_3}
\end{figure}

%\clearpage
%\vspace{4cm}

\section{Experiment 2 - Tripod mounting outdoors} 

The next step was to take the assembly outdoors. The exterior of the Mechanical Department sidewalk seemed to be the most appropriate spot to carry on this experiment, as here there are many pedestrians at any time of the day. In fact, it became suitable for gathering pedestrians data. (see tripod mounting location in figure~\ref{fig:crosswalk_outside_department}.) 

%\begin{table}[h]
%\centering
%\caption{Weather conditions} 
%	\begin{tabular}{lccc}
%	\hline
%	\multicolumn{2}{c}{Testing set: } & \multicolumn{1}{l}{Sunny day} & \multicolumn{1}{l}{Summer season}\\\hline
%	\end{tabular}
%\label{table:Weather_conditions}
%\end{table}

%\clearpage

%\begin{figure}[h]
%	\centering
%	\captionsetup{justification=centering}
%		\includegraphics[scale=0.42]{Figures/Mapa_Campus_Web_DEM_cutted_1.pdf}
%	\caption{Tripod Assembly Spot Location on the UA Campus, 22 - Mechanical Engineering Department} 
%	\label{fig:crosswalk_outside_department}
%\end{figure}

%Since some results were available after storing the extracted data in a rosbag, it also urged to extract some frames from the rosbag to be processed, alone, by the initial Pedestrian Detection Algorithm. 

The gathered data allowed the comparison between the two algorithms (Sensor fusion and Pedestrian Detection).

Furthermore, a comparison requires the existence of a ground truth data. Therefore, the comparison was made by using a manual labeling of the pedestrians positions in the image.\\ 

The data gathered for this experiment had 2275 image files. Because the logitech camera, figure~\ref{fig:logitech_c310_webcam}, records at a rate of 30 fps (frames per second) and to avoid catching the pedestrian always on the same spot, the 2275 images were reduced to 220, containing 127 ground truth positive bounding boxes.

The results shown in figure~\ref{fig:DEM_tripod_matlab_results} were obtained (compared with the correspondent ground truth) considering the Pascal criteria, where two \ac{BB}s are considered a match if the overlap area is superior to $0.5$, \cite{Everingham2009}.

Red lines represent the results only with the Pedestrian Detection Algorithm, whilst blue lines correspond to Sensor Fusion Algorithm results.

\begin{figure}[H]
	\centering
	\captionsetup{justification=centering}
		\includegraphics[width=10.9cm]{Figures/Mapa_Campus_Web_DEM_cutted_1.pdf}
	\caption{Tripod Assembly Spot Location on the UA Campus, 22 - Mechanical Engineering Department~\cite{mapUA}} 
	\label{fig:crosswalk_outside_department}
\end{figure}

\begin{figure}[H]
	\centering
	\captionsetup{justification=centering}
		\includegraphics[width=12cm]{Figures/DEM_tripod_matlab_results_final.pdf}
	\caption{Experiment 2, TPR (True Positive Rate) and FPR (False Positive Rate) Results} 
	\label{fig:DEM_tripod_matlab_results}
\end{figure}

\begin{table}[H]
\centering
\caption{Experiment 2 detailed results} 
	\begin{tabular}{C{1cm}| ccc|C{2.5cm}C{2.5cm}}
	\cline{1-6}
	Life	& \multicolumn{3}{c|}{Sensor Fusion Algorithm} & \multicolumn{2}{c}{Pedestrian Detection Algorithm}\\\cline{2-6}
			time		   & & \textbf{TPR} & \textbf{FPR} & \textbf{TPR} & \textbf{FPR}  \\\cline{1-6}
	\multicolumn{1}{c}{2} & & \multicolumn{1}{c}{0.5116} & \multicolumn{1}{c}{0.0185} & \multirow{7}{*}{0.5982} & \multirow{7}{*}{0.1653} \\\cline{1-4}
	\multicolumn{1}{c}{4} & & \multicolumn{1}{c}{0.5938} & \multicolumn{1}{c}{0.0187} & & \\\cline{1-4}
	\multicolumn{1}{c}{6} & & \multicolumn{1}{c}{0.5852} & \multicolumn{1}{c}{0.0189} & & \\\cline{1-4}
	\multicolumn{1}{c}{8} & & \multicolumn{1}{c}{0.6179} & \multicolumn{1}{c}{0.0450} & & \\\cline{1-4}
	\multicolumn{1}{c}{10} & & \multicolumn{1}{c}{0.5625} & \multicolumn{1}{c}{0.0541} & & \\\cline{1-4}
	\multicolumn{1}{c}{20} & & \multicolumn{1}{c}{0.6589} & \multicolumn{1}{c}{0.0885} & & \\\cline{1-4}
	\multicolumn{1}{c}{$\infty$} & & \multicolumn{1}{c}{0.9504} & \multicolumn{1}{c}{0.8846} & & \\\hline
	\end{tabular}
\label{table:DEM_tripod_matlab_results}
\end{table}

The Sensor Fusion Algorithm has some parameters that can be changed. The one that makes the biggest difference, is the time of life of the candidates vector. This way, further research was made through different tests, changing the time of life of the candidate.

On these grounds, the time of life of the candidate was counted through the number of images passed with the same classification. The candidates vector is reset every time the image counter equals the number of the given life time. For example, if the life time is "2", candidates classification is reset every two images received.

Based on this evidence, one can say that resetting the candidates vector means that every classification is changed to "Unknown". Further tests were carried out with 2, 4, 6, 8, 10, 20 and $\infty$ ($\infty$ \textrightarrow $ $ there wasn't a candidate time limit). The Pedestrian Detection Algorithm was run with the same parameters as the Sensor Fusion Algorithm, to allow comparison (Step search = 4).  

To provide a better understanding of how the fusion detection works for the "$\infty$" case and why the FPR is high compared to other experiments, some example results are shown in figures~\ref{fig:DEM_tripod_matlab_results_image1} through ~\ref{fig:DEM_tripod_matlab_results_image4}, in which the program itself drew green and red \ac{BB} automatically. Figure~\ref{fig:DEM_tripod_Pedro_Algorithm_matlab_results_image} shows an example of FPR for the Pedestrian Detection Algorithm, an object that was far away and was still highlighted. This can be easily avoided with the help of the \ac{LIDAR}.

The data also allowed a process time comparison, see table~\ref{table:DEM_tripod_matlab_Time_results}. The process time tends to decrease, when candidate time limit increases.

\begin{table}[h]
\centering
\caption{Mean time (in seconds) to process an image} 
	\begin{tabular}{>{\centering\arraybackslash}c|>{\centering\arraybackslash}c|>{\centering\arraybackslash}c}
	\cline{1-3}
	\multirow{2}{*}{Life time}	& \multirow{2}{*}{Sensor Fusion Algorithm} & \multirow{2}{*}{Pedestrian Detection Algorithm}\\
	 & & \\\cline{1-3}
	\multicolumn{1}{c}{2} & \multicolumn{1}{c|}{0.4021} & \multirow{7}*{61.5298} \\\cline{1-2}
	\multicolumn{1}{c}{4} & \multicolumn{1}{c|}{0.3766} & \\\cline{1-2}
	\multicolumn{1}{c}{6} & \multicolumn{1}{c|}{0.3648} & \\\cline{1-2}
	\multicolumn{1}{c}{8} & \multicolumn{1}{c|}{0.3539} & \\\cline{1-2}
	\multicolumn{1}{c}{10} & \multicolumn{1}{c|}{0.3973} & \\\cline{1-2}
	\multicolumn{1}{c}{20} & \multicolumn{1}{c|}{0.2905} & \\\cline{1-2}
	\multicolumn{1}{c}{$\infty$} & \multicolumn{1}{c|}{0.1478} \\\hline
	\end{tabular}
\label{table:DEM_tripod_matlab_Time_results}
\end{table}

\begin{figure}[H]
	\centering
	\captionsetup{justification=centering}
		\includegraphics[scale=0.4]{Figures/7517.png}
	\caption{Example Result 1 - TP and TN classifications.} 
	\label{fig:DEM_tripod_matlab_results_image1}
\end{figure}

\begin{figure}[H]
	\centering
	\captionsetup{justification=centering}
		\includegraphics[scale=0.4]{Figures/7560.png}
	\caption{Example Result 2 - Target tracking allows Pedestrians to maintain the same ID, even when occluded by a column.} 
	\label{fig:DEM_tripod_matlab_results_image2}
\end{figure}

\begin{figure}[H]
	\centering
	\captionsetup{justification=centering}
		\includegraphics[scale=0.4]{Figures/7595.png}
	\caption{Example Result 3 - Evidence shows that the tracking gets confused with some people crossing (3 pedestrians and one column -4 objects in total-). Moreover, it appears to be very complex for the further objects to be correctly tracked, since the surrounding BB of the column has now a pedestrian ahead, which induces into a classification error, switching it from "Object" to "Pedestrian", figure~\ref{fig:DEM_tripod_matlab_results_image4}. The nearest pedestrians continued to be well tracked.} 
	\label{fig:DEM_tripod_matlab_results_image3}
\end{figure}

\begin{figure}[H]
	\centering
	\captionsetup{justification=centering}
		\includegraphics[scale=0.4]{Figures/7649.png}
	\caption{Example Result 4 - Outcome of fig.~\ref{fig:DEM_tripod_matlab_results_image3}, previous Pedestrian 18 became Pedestrian 46, reclassified after receiving a new ID.} 
	\label{fig:DEM_tripod_matlab_results_image4}
\end{figure}

\begin{figure}[H]
	\centering
	\captionsetup{justification=centering}
		\includegraphics[scale=0.5]{Figures/007293.pdf}
	\caption{FP example using Pedestrian Detection Algorithm} 
	\label{fig:DEM_tripod_Pedro_Algorithm_matlab_results_image}
\end{figure}

%\clearpage

\section{Experiment 3 - AtlasCar} 

As experiment 2 provided some satisfactory results, some data was extracted aboard the $AtlasCar$ in order to make evidence more realistic and reliable. This way, the choice of a location near the $AtlasCar$ garage, figure~\ref{fig:AtlasCar_garage_location}, seemed adequate for this purpose (test the algorithm), since it is the university students ordinary footpath to attend classes and is, in fact, crowded all day long.

Previous parameters were maintained while conducting this experiment.

The non-existence of columns here allows extracting the full data without having any tracking confusion problem. These experiments were performed on a cloudy day.

The performance metrics used were the same as in Experiment 2. 

Figure~\ref{fig:AtlasCar_matlab_results} shows that the FPR were reduced, in some cases becoming 0, and in cases with prolonged time of life, the rate continued to be tiny compared with the original Pedestrian Detection Algorithm value.

The data initially had 5950 image files, because the XB3 camera (figure~\ref{fig:Bumblebee_XB3}) records at a rate of 16 fps (frames per second) and to avoid catching the pedestrian always on the same spot, the 5950 images were reduced to 1190, containing 901 ground truth positive bounding boxes.

\begin{figure}[H]
	\centering
	\captionsetup{justification=centering}
		\includegraphics[width=10.9cm]{Figures/Mapa_Campus_Web_AtlasCar_Garage.pdf}
	\caption{AtlasCar recording location, on the UA Campus, 3 - CESAM/CICECO/TEMA Department~\cite{mapUA}} 
	\label{fig:AtlasCar_garage_location}
\end{figure}

\begin{figure}[H]
	\centering
	\captionsetup{justification=centering}
		\includegraphics[width=12cm]{Figures/AtlasCar_matlab_results_final.pdf}
	\caption{Experiment 3, TPR (True Positive Rate) and FPR (False Positive Rate) Results} 
	\label{fig:AtlasCar_matlab_results}
\end{figure}

\begin{table}[h]
\centering
\caption{Experiment 3 detailed results} 
	\begin{tabular}{l| ccc|C{2.5cm}C{2.5cm}}
	\cline{1-6}
	Life	& \multicolumn{3}{c|}{Sensor Fusion Algorithm} & \multicolumn{2}{c}{Pedestrian Detection Algorithm}\\\cline{2-6}
			time		   & & \textbf{TPR} & \textbf{FPR} & \textbf{TPR} & \textbf{FPR}  \\\cline{1-6}
	\multicolumn{1}{l}{2} & & \multicolumn{1}{c}{0.0843} & \multicolumn{1}{c}{0} & \multirow{8}{*}{0.3242} & \multirow{8}{*}{0.1802} \\\cline{1-4}
	\multicolumn{1}{l}{4} & & \multicolumn{1}{c}{0.0997} & \multicolumn{1}{c}{0} & & \\\cline{1-4}
	\multicolumn{1}{l}{6} & & \multicolumn{1}{c}{0.1204} & \multicolumn{1}{c}{0} & & \\\cline{1-4}
	\multicolumn{1}{l}{8} & & \multicolumn{1}{c}{0.1465} & \multicolumn{1}{c}{0} & & \\\cline{1-4}
	\multicolumn{1}{l}{10} & & \multicolumn{1}{c}{0.1447} & \multicolumn{1}{c}{0.0022} & & \\\cline{1-4}
	\multicolumn{1}{l}{20} & & \multicolumn{1}{c}{0.2224} & \multicolumn{1}{c}{0.0022} & & \\\cline{1-4}
	\multicolumn{1}{l}{40} & & \multicolumn{1}{c}{0.2884} & \multicolumn{1}{c}{0.0022} & & \\\cline{1-4}
	\multicolumn{1}{l}{$\infty$} & & \multicolumn{1}{c}{0.5502} & \multicolumn{1}{c}{0.0260} & & \\\hline
	\end{tabular}
\label{table:AtlasCar_matlab_results}
\end{table}

Even though the research results have emerged as satisfactory, the tested time of life of "10" instead of growing as expected, decreased a little.

The pedestrian recorded data was similar to the one on a road, in spite of the fact that this data had been recorded in one steady place, where people passed not only in front but also on both sides of the car (see figure~\ref{fig:AtlasCar_Location}). The $Atlascar$ is located at the (0,0) point. Each color tracked, corresponds to a potential pedestrian.

\begin{figure}[H]
	\centering
	\captionsetup{justification=centering}
		\includegraphics[width=12cm]{Figures/Atlascar_Location_color_2.pdf}
	\caption{Pedestrians Route in $AtlasCar$ dataset} 
	\label{fig:AtlasCar_Location}
\end{figure}

\begin{figure}[H]
	\centering
	\captionsetup{justification=centering}
		\includegraphics[width=12cm]{Figures/732.png}
	\caption{TP Fusion Algorithm Example 1} 
	\label{fig:TP_AtlasCar_example_Fusion}
\end{figure}

\begin{figure}[H]
	\centering
	\captionsetup{justification=centering}
		\includegraphics[width=12cm]{Figures/5097.png}
	\caption{TP Fusion Algorithm Example 2} 
	\label{fig:TP_AtlasCar_example_Fusion1}
\end{figure}

\begin{figure}[H]
	\centering
	\captionsetup{justification=centering}
		\includegraphics[width=12cm]{Figures/1803.png}
	\caption{TP Fusion Algorithm Example 3} 
	\label{fig:TP_AtlasCar_example_Fusion2}
\end{figure}

\begin{figure}[H]
	\centering
	\captionsetup{justification=centering}
		\includegraphics[width=12cm]{Figures/3107.png}
	\caption{FP Fusion Algorithm Example - After a pedestrian passed the bushes, the classification continued to be shown for a short period of time} 
	\label{fig:FP_AtlasCar_example_Fusion}
\end{figure}

\begin{figure}[H]
	\centering
	\captionsetup{justification=centering}
		\includegraphics[width=12cm]{Figures/FP_AtlasCar_example_Vision_only.pdf}
	\caption{FP Pedestrian Detection Algorithm Example} 
	\label{fig:FP_AtlasCar_example_Vision_only}
\end{figure}

The data also allowed a process time comparison, see table~\ref{table:Atlas_matlab_Time_results}. The process time tends to decrease, when candidate time limit increases.

\begin{table}[H]
\centering
\caption{Mean time (in seconds) to process an image} 
	\begin{tabular}{>{\centering\arraybackslash}c|>{\centering\arraybackslash}c|>{\centering\arraybackslash}c}
	\cline{1-3}
	\multirow{2}{*}{Life time}	& \multirow{2}{*}{Sensor Fusion Algorithm} & \multirow{2}{*}{Pedestrian Detection Algorithm}\\
	 & & \\\cline{1-3}
	\multicolumn{1}{c}{2} & \multicolumn{1}{c|}{1.1554} & \multirow{8}*{158.1968} \\\cline{1-2}
	\multicolumn{1}{c}{4} & \multicolumn{1}{c|}{0.8164} & \\\cline{1-2}
	\multicolumn{1}{c}{6} & \multicolumn{1}{c|}{0.8170} & \\\cline{1-2}
	\multicolumn{1}{c}{8} & \multicolumn{1}{c|}{0.7835} & \\\cline{1-2}
	\multicolumn{1}{c}{10} & \multicolumn{1}{c|}{0.5739} & \\\cline{1-2}
	\multicolumn{1}{c}{20} & \multicolumn{1}{c|}{0.5930} & \\\cline{1-2}
	\multicolumn{1}{c}{40} & \multicolumn{1}{c|}{0.5293} & \\\cline{1-2}
	\multicolumn{1}{c}{$\infty$} & \multicolumn{1}{c|}{0.1466} \\\hline
	\end{tabular}
\label{table:Atlas_matlab_Time_results}
\end{table}

\section{Discussion}

The discussion on Experiment 2 lies on the quantification of the results. Although the research has allowed getting an average satisfactory dataset achievement in terms of quantification of the results, it proved to be a very small one (only with 220 images). Due to this, errors were induced (when changing the time of life) into the results by a slight percentage, causing a rapid oscillation (a rapid rise and descend).

Facts prove that the location was not very helpful because of the columns; as a matter of fact, the sensor fusion algorithm kept tracking some of the objects, even when crossing each other. Due to this, some objects were classified wrongly. For example, pedestrians who walked in front of it, passed their classification on to columns.

The available evidence seems to suggest that even if this dataset was small, the results were very satisfactory, it means, both the FPR was reduced significantly compared to the initial Pedestrian Detection Algorithm and some of the experiments exceeded the Pedestrian Detection TPR. It seems fair to emphasize that it was unexpected; it proved to be very good, however.

The best performance accomplished with this dataset, increased the TPR by 6\% and decreased the FPR by 8\%.

The image processing time was decreased by a means of $\simeq$281\%, comparing to the initial Pedestrian Detection algorithm.

FP detections decreased due to the LIDAR object selection.

In experiment 3, the algorithm was tested in a different environment and in different weather conditions. 

Actually, this dataset was larger than the previous one (1190 images), which implies that the values vary in a more consistent way, instead of changing abruptly.

The best performance accomplished with this dataset, increased the TPR by 23\% and decreased the FPR by 16\%.

The image processing time was decreased by a means of $\simeq$633\%, comparing to the initial Pedestrian Detection algorithm.

The results were very satisfactory, having a growing trend TPR and at the same time the FPR maintained low values. 

\cleardoublepage
\chapter{Conclusions and Future Work}

Pedestrians are a very important issue in security driving. Due to the complexity of pedestrian detection, the usage of visual images is probably the best way to acquire its detection. However, it is still not very effective specially regarding CPU processing costs. With the support of a sensor (LIDAR), which helps to select potential pedestrians, visual search can be improved and faster at detecting pedestrians.

This project, where this work is inserted, combines a pedestrian tracking algorithm (LIDAR Processing) and a pedestrian detection algorithm (Image Processing) enhancing the best of the two sensors on pedestrian detection field. 

Three different experiments were carried out to verify its validation. These experiments were performed in three different places with two different types of situations. First, on LAR (Automation and Robotics Laboratory) and then on the Department of Mechanical Engineering at the University of Aveiro facing the main entrance and finally onboard of $AtlasCar$ at one of the main University pedestrian exits. On the first two situations a camera was vertically placed on top of the LIDAR.
 
This work features multi-thread process, which enables multi parallel potential pedestrian detection, while the program continues to run freely. It also stores each potential pedestrian properties and enables their constant moves (location update). Another feature of this work is the creation and elimination of candidates, potential pedestrians management which increases the overall algorithm performance because the eliminated candidates frees memory. Last but not the least, it helps the user to visualize potential candidates (Red bounding box) and actual classified pedestrians (Green bounding box) to an output image.

The first experiments relies on testing and improving some visual aspects as well as the tuning of some parameters to decrease the CPU processing cost. Here, a candidate "life time" parameter was also created that resets the candidate vector every time the image counter reaches the given life time limit. For example, if the life time is "10", candidate classification is reset in every ten received images. This parameter was also implemented on the second and third experiment. During the second experiment, the best performance was achieved while having a candidate time of life of "20" which increased the detection TPR (True Positive Rate) by $6\%$ and decreasing the detection FPR (False Positive Rate) by $8\%$. On the third experiment, the best performance was obtained having no candidate time of life resulting in an increase of the detection TPR by $23\%$ and a decrease of the detection FPR by $16\%$. 

The second and third experiments were carried out outdoors in order to make it more realistic and reliable.

It was observed that pedestrian detections are influenced by environment light, where the second experiment had a better performance compared to the third one.
Shadow areas seem to be responsible for most missed pedestrians. 

Sensor fusion method showed that the CPU memory (RAM) cost can be decreased (2GB down from 12GB, in a 1280x960 image) as a result of images that do not need to be full sized, but instead, potential pedestrian size image. As result, the CPU processing cost was substentially decreased allowing faster detections and a decreasing regarding the false positive rate because the searching areas can be narrowed to the important ones with the \ac{LIDAR} help.
       
The main goals of this work were achieved by: a re-parametrization of a Pedestrian Detection Algorithm becoming a Single Pedestrian Detection Algorithm allowing pedestrian detection on images of any size; A development of a manual calibration tool between the LIDAR and the camera; A semi-automatic calibration tool was built and tested, however, unsuccessful; Creation of an application that combines LIDAR and vision data in order to activelly detect pedestrians by means of image localized search.

Achieving all main goals means that the sensor fusion of LASER and Image processing application for active pedestrian detection was successfully implemented.
      
The greatest contribution of this project was that detection processing time decreased substantially. The incorporation of the system in the $AtlasCar$ is a feature that had not yet been developed since the begining of this work. This project can, and most certainly will, be improved in the future helping not only pedestrians but also car detections in real time situations aboard $AtlasCar$.  

Experiments were conducted in a Asus N53SV notebook, for more informations on the Hardware and Operating System see appendix~\ref{apendix:PCU_notes}.

The present document was written in English to extend language knowledge.

\subsection*{Future Work}

In the future, the fusion situation may be able to: 

Predict spaces where the laser cannot reach and entrust the camera to test if anyone near exists, for example, when two cars are parked parallel to the sidewalk. The gap between the two cars (critical point) is often a path chosen by the pedestrian to cross the road and surprise the driver, as illustrated in Figure 1.11.

Take advantage of pedestrian classifications to enlarge detection datasets for further usage; 

Classify different types of object, modifying objects parameters and changing detection datasets; 

Use the expected time of collision to dynamically studying the pedestrians and choose whether or not to classify. This matter can be use to predict if a pedestrian is walking towards or moving away from the car. For example, if a pedestrian is in an eminent danger of colliding with the car, that pedestrian needs to be the top priority for classification and so on.

%Pedestrians are very important because of driving security issues. Due to the complexity of pedestrian detection, the use of visual images is probably the best way to confirm detection, but it is still not very effective, specially in processing cost. With the support of a sensor (LIDAR), that selects potential pedestrians, visual search can be precise and faster at detecting pedestrians.
%
%This project combines a pedestrian tracking algorithm (LIDAR) and a pedestrian detection algorithm (Vision), enhancing the best of the two sensors on pedestrian detection matters. Being this a fusion project, no vision training detection was made, achieving results with previous $Atlas$ Pedestrian Dataset. 
%
%Three different experiments were done to demonstrate its validation. Tests were performed in 3 different places with 2 types of situations, first in LAR (Automation and Robotics Laboratory), then facing the University of Aveiro Mechanical Engineering Department, in both of these situations it was used a camera vertically placed on top of the \ac{LIDAR} and the final location was facing a sidewalk aboard $AtlasCar$.  
% 
%This project features multi-thread process, this enables multi parallel potential pedestrian detection while the program continues to run freely. It also stores each potential pedestrian properties, enabling their constant update (location). Another feature is the creation and elimination of candidates, potential pedestrians management, increasing the overall algorithm performance. Last but not least, it helps the user to visualize candidates on output image.
% 
%"Experiment 1" was used to improve some visual aspects and decrease processing cost. It was also created a candidate "life time" parameter, that resets the candidate vector every time the image counter reaches the given life time limit. For example, if the life time is "10", candidate classification is reset every ten images received. This parameter was implemented in experiments 2 and 3. In the second experiment, the best performance was obtained having a candidate time of life of "20", increasing the detection TPR (True Positive Rate) by $6\%$ and decreasing the detection FPR (False Positive Rate) by $8\%$. In the third experiment, the best performance was obtained having no candidate time of life, increasing the detection TPR by $23\%$ and decreasing the detection FPR by $16\%$. 
%
%Experiments 2 and 3 were performed outdoors, in order to make evidence more realistic and reliable.
%
%It was observed that pedestrian detections were influenced by environment light, making "experiment 2" performance better than "experiment 3". Shadow areas seem to be responsible for most missed pedestrians. 
%
%Sensor fusion method showed that not only the CPU memory (RAM) cost can be decreased, as a result of images do not need to be full size but instead a potential pedestrian size image, but also CPU processing cost can also be decreased, allowing faster detections and finally, the false positive rate can also be decreased, because searching areas can be narrowed to the important ones with the \ac{LIDAR} help.
%       
%The main goals were achieved with: a re-parametrization of a Pedestrian Detection Algorithm, becoming a Single Pedestrian Detection Algorithm, allowing pedestrian detection on images of any size; a development of a manual calibration tool, between the LIDAR and the camera; a development of a semi-automatic calibration tool was also tried, but unsuccessful; a development of an application that combines LIDAR and vision data, in order to active detect pedestrians by means of image localized search.
%
%Achieving all main objects, we can say that the sensor fusion of LASER and Vision application for active pedestrian detection was successfully implemented.
%      
%The greatest contribution of this project was combining two different frameworks, allowing the detection of all objects that \ac{LIDAR} can detect, and incorporate it in the $AtlasCar$, a feature that had not yet been developed. This project can and most certainly will be improved in the future, helping not only pedestrians but also car detection in real time situation aboard $AtlasCar$.  
%
%In the future, the fusion situation can: 
%
%Predict spaces where laser cannot reach and entrust the camera to test if anyone near exists, for example, when two cars are parked parallel to the sidewalk exists a gap between them  (critical point), in this gap a pedestrian can suddenly appear and surprise the driver. Detection can be entrusted to the camera, that can try to detect an half body pedestrian;
%
%Take advantage of pedestrian classifications to enlarge detection datasets; 
%
%Classify different types of object, modifying objects parameters and changing detection datasets; 
%
%Classify any type of object using the expected time of collision, dynamically studying the pedestrians and choose whether or not to classify. This matter can be use to predict if a pedestrian is walking towards or moving away from the car  If a pedestrian is in an eminent danger of colliding with the car, that pedestrian needs to be the top priority to classification and so on;

%
% The bibliography
%
\cleardoublepage
\renewcommand{\bibname}{References}
	% Use this is the final version
%	\bibliographystyle{apalike}% use any style you like (I recommend the harvard package)
\bibliographystyle{reference/apalike-url-1}% use any style you like (I recommend the harvard package)
%	\raggedright
\bibliography{my-bib-file}% replace by the name of name of your .bib file
%\nocite{*}
%   
\cleardoublepage

\appendix
\setcounter{chapter}{0}

\chapter{Usage Instructions}

Launch files were created to launch the necessary ROS nodes for the program to run. 

\section{Sensor Fusion Program}
\label{section:Sensor_Fusion}
To enable the full usage of the Sensor Fusion Program, two roslaunch programs need to be started. They are:

\begin{itemize}

\item generate\_single\_laser\_planar\_cloud\_with\_tf.launch

\end{itemize}

This launch file transforms the SICK laser frame in a $AtlasCar$ front bumper frames, because the two front lasers are combined in the $AtlasCar$ front bumper frame. Subscribes "/snr/las/2/scan" and publishes the "/scan\_cloud" message

It also starts the MTT program that subscribes the transformed laser points into MTT targets ("/scan\_cloud"), publishing them with the "/new\_targets" name. The next code shows its parameters.\\

\lstset{language=XML}
\begin{lstlisting}
<launch>
<!-- Inside contains a comment -->
<!--     Remap AtlasCar transformations (/tf to /trf/frames) needs input: "/trf/frames" -->
    <remap from="/tf" to="/trf/frames"/>
        
    <group ns="/pcp/fus/planar_pc">
<!--    Left (/snr/las/2/) or Right (/snr/las/3/) Laser; input needed: "/snr/las/2/scan" -->
        <remap from="/laserscan0" to="/snr/las/2/scan"/>
        
<!--         Output tracking frame -->
        <remap from="/tracking_frame" to="/atc/vehicle/center_bumper"/> 
        <remap from="/pc_out" to="/scan_cloud"/>
        
<!--     Starts the laser transformation to output tracking frame (AtlasCar) -->
        <node name="planar_pc" pkg="mtt" type="simple_planar_pc_generator_atlasmv" output="screen"> 
            <param name="output_frequency" value="200.0"/>
            <param name="perpendicular_treshold" value="0.15"/>
            <!-- simple_planar node output: "/scan_cloud"-->
        </node>
        
        <!-- This starts multi target tracking for stage laser measurements-->
        <node name="node" pkg="mtt" type="mtt_pedect">
            <remap from="/points" to="/scan_cloud"/>
            <remap from="/markers" to="/ids"/>
            <!-- MTT node output: "/new_targets"-->

        </node>
    </group>
</launch>
\end{lstlisting}

\begin{itemize}
\item sensor\_fusion.launch

\end{itemize}

This launch file launches an image republisher, which rectifies the camera images. It subscribes the "/xb3/right/image\_raw" and the "/xb3/right/camera\_info", publishing the already rectified "/xb3/right/new\_info/image\_rect\_color" and the \\"/xb3/right/new\_info/camera\_info".

It also starts the sensor fusion node where the classifier location is used for the Single Pedestrian Detection Algorithm.

The sensor fusion node subscribes the "/xb3/right/new\_info/image\_rect\_color", the "/xb3/right/new\_info/camera\_info" and the "/new\_targets" (provided by the MTT node). The next code shows its parameters. \\

\begin{lstlisting}
<launch>

<!--   XB3 proc image republisher   -->
  <!--   XB3 proc image republisher input: "/xb3/right/image_raw/compressed "  -->
    <node ns="/xb3/right" name="republish_right" type="republish" pkg="image_transport" args="compressed in:=image_raw out:=image_decompressed"/>
        
    <!--  Camera info republisher (replace camera info with calibrated camera info)  -->
    <node ns="/xb3/right" name="info_changer_right" type="camera_info_republisher" pkg="multisensor_aquisition" args="image_raw:=image_decompressed">
        <param name="camera_calibration_file" value="package://xb3/calibrations/right_full_resolution.yaml"/>
        <param name="camera_name" value="right"/>
    </node>

    <node ns="/xb3/right/new_info" name="right_camera_rectification" type="image_proc" pkg="image_proc"/>
  <!--   XB3 proc image republisher output: "/xb3/right/new_info/image_rect_color" and "/xb3/right/new_info/camera_info"  -->
   
<!-- Sensor Fusion Program -->
  <!--   Sensor Fusion Program input: "/xb3/right/new_info/image_rect_color", "/xb3/right/new_info/camera_info" and "/new_targets". If one wants to observe the laser projected points in the image, the "/scan_cloud" subscriber topic needs to be uncomment on sensor_fusion.cpp  -->
    <node  name="sensor_fusion_prog" pkg="multimodal_pedestrian_detection" type="sensor_fusion" output="screen">
        
<!--    Classifier Location    -->
        <param name="classifier" value="/.../trained_boost_15Kf_2000w_19Ks_m8_M64_boot3.xml"/>
        
    </node>
</launch>
\end{lstlisting}

\subsubsection{How to run the sensor fusion program}

The sensor fusion program can be executed by playing a recorded rosbag, when it contains every message topic mentioned above (commented code), or simply run it aboard the $AtlasCar$, applying the same conditions.

Provided access to the LAR tool kit, follow the next steps to run the program:

\begin{enumerate}

\item Open a terminal on Ubuntu.

\item Go to LAR tool kit primary folder (ex. "lar4", where is only seen the "src" folder) and type "catkin\_make" to compile every single package, creating a "build" and "devel" folder. To compile each package separately one can do: 

\begin{itemize}
\item If one wants to compile only the "MTT" package, inside the "build" folder enter "cd /perception/planarobstacles/mtt/" and type "make". 

\item To compile only the "sensor fusion" package, in the "build" folder enter "cd /perception/pedestrians/multimodal\_pedestrian\_detect" and type "make".
\end{itemize}

\item Run the program by entering "roslaunch multimodal\_pedestrian\_detection \\sensor\_fusion.launch".

\item On another terminal enter "roslaunch multimodal\_pedestrian\_detection \\generate\_single\_laser\_planar\_cloud\_with\_tf.launch".

\end{enumerate}


\section{Calibration Process}

For calibration purposes, a program was created to calculate and output the transformation matrix ($^{C}[R|t]_{_{FB}}$), when given two .txt files with image and object points coordinates (input).

If one wants to execute the calibration launch file, the "multimodal\_pedestrian\_detect" package needs to be compiled (process mention in Section~\ref{section:Sensor_Fusion}). The next code shows the "calibrate\_extrinsic\_parameters.launch" file and its parameters.\\

\begin{lstlisting}
<launch>
    
    <node name="Client_extrinsic_calibration" pkg="multimodal_pedestrian_detection" type="calibrate_camera_extrinsic" output="screen">
        
<!--   Image Points Location, file = .txt (Input) -->
       <param name="imagePoints1FileName" value="/.../imagePointsAuto_1280_960_xb3.txt"/>
        
<!--   Object Points Location, file = .txt (Input) -->
       <param name="objectPoints1FileName" value="/.../objectPointsAuto_1280_960_xb3.txt"/>
       
<!--   File storage Location, file = .yaml (Output) -->
       <param name="fileStorageYAML" value="/.../bumblebeeXB3_1280_960_transform_matrix_to_2Dlaser.yaml"/>
        
    </node>
</launch>
\end{lstlisting}

To execute this launch open a terminal in Ubuntu and type "roslaunch multimodal\_pedes-trian\_detection calibrate\_extrinsic\_parameters.launch".

\cleardoublepage
\chapter{Hardware and System Notes}
\label{apendix:PCU_notes}

This work was conducted on a Asus N53SV notebook with the following specifications:

\begin{itemize}

\item Hardware

	\begin{itemize}
		\item Intel® Core™ i7-2630QM CPU @ 2.00GHz × 8 
		\item 6GB (3+3) DDR3 1333 MHz SDRAM
		\item NVIDIA® GeForce® GT 540M with 1GB DDR3 VRAM 
		\item SSD Samsung 840 EVO (250GB) hard drive
	\end{itemize}
 	
\item System

	\begin{itemize}
		\item Ubuntu Linux version 12.04 LTS (64 bits)
		\item Linux kernel version 3.11.0-26-generic
		\item ROS Hydro version
	\end{itemize}

\end{itemize}

\end{document}
