Back in June of 2019, I was given an opportunity to present a paper called Assessing Modeling Variability in Autonomous Vehicle Accelerated Evaluation at AISC, a Toronto-based group that discusses interesting machine learning papers or concepts. The slides can be found here. The paper focused on leveraging importance sampling to reduce the required number of samples to achieve meaningful precision in safety evaluation by a few orders of magnitude.
Even though I work in the data science field, in my role as a manager, I don't often get a chance to dig deep into the weeds of research papers. I will typically read the abstract only or get a summary from a team member. This was a good opportunity for me to learn more about the autonomous vehicle space as well as practice my public speaking skills. The hard deadline also gave no room for me to procrastinate. This was happening no matter how over or under-prepared I was.
Given that the YouTube video available, I will not go into detail regarding the actual paper. Instead, I want to share my experiences that's not captured in the video - how I prepared for the presentation and my takeways from this experience.
Challenges
The first challenge was in understanding all of the notation. The notion of an environment ξ and the corresponding parameters θ was not clear to me. I initially thought that the environment spanned the literal environment that the AV would sit in such as the wheel angular velocity, distance between objects, and throttle/brake pedal application. The corresponding parameters might be the tire/surface friction coefficient or the engine torque. The second major confusion I had was around one of the key equations - (14) in the original version of the paper. I couldn't make sense of it without a normalizing 1/n term. Serena (who was coordinating) had already sent out an email to the authors so I asked some clarifying questions with the presentation a couple of days away.
Luckily, one of the authors respond promptly explaining that the environment ξ was much simpler than I had imagined and confirming that there was a typo in equation (14). The environment referred to the essential data points to establish the likelihood of crashing. For instance, if you are driving toward a wall, you might limit the environment to the velocity of the car and the distance between the car and the wall. Aha - things were starting to make sense.
Final Thoughts
The phrase "you get back what you put into it" feels very relevant to this experience. As I spent hours and hours poring over every detail of the paper, I started to really appreciate it at a level not possible previously. This sense of enjoyment from mastery is hard to overstate and something you need to experience personally. That being said, it was humbling to see that my explanations were not as clear as they could be - meaning that my understanding of the topics had room for improvement. Whether you're a seasoned veteran or getting started in data science, I would highly recommend that you try to present a paper at a local meetup.