My first Kaggle competition ended with a team gold medal, 8th place

The Kaggle competition ended today, June 21, 2022, at 9:00 in the morning. Results appear immediately, so I watched them with my team members. We finished 8th out of 1,975 teams, the same as our public LB rank, and received a gold medal. It was my first Kaggle competition, and I was fortunate to be on an excellent team.

The team's solution is published in English, and I expect a Japanese version will be published separately. So here I will write mainly about my impressions as a first-time participant, rather than the solution itself. About 40 days earlier, Aota invited me to try U.S. Patent Phrase to Phrase Matching, a patent and natural language processing competition. My Kaggle knowledge was only at the level of having read a beginner book and tried the Titanic practice competition, so he taught me more concrete ways to proceed and how to think when working as a team.

For tools, we used Notion for information sharing, mainly a kanban board, and shared implementations on GitHub. Scores were recorded in a spreadsheet together with implementation serial-number files. Implementations were branched by plain copying. At first I wondered whether a notebook-style workflow, like Colab plus diff management on save, would be better. In practice, the copy-based approach made each file easier to compare, and because many variants were created, more than 100 training implementations in the end, copy-paste implementation was convenient. File-level diffs were also easy to take.

Asynchronous flow-style information sharing happened in chat, while synchronous discussions happened in a weekly regular meeting. When the team reached consensus on a method, we updated the baseline implementation. Aota mostly grew the baseline into something strong. Deciding the best CV split method early also paid off more and more as data accumulated. Running the baseline saved not only scores but also metadata needed to derive those scores, such as preprocessed data, which made it easier to investigate why a score appeared.

That is how the competition work began. From then on, I spent almost all the time I had previously spent on games and other things on Kaggle. Once I started, Kaggle had a game-like addictiveness. To get a higher score, I wanted to try one idea after another. Because it was my first competition, I also gained a lot of knowledge along the way, which made the process intellectually satisfying and fun.

That said, Kaggle is not easy enough that a beginner can get high scores just by imitating what they read in Code or Discussion. Around the middle of the competition, there were periods where improving local CV produced the opposite movement on LB. My motivation dropped a little. Because I was on a team, though, I still felt that we were moving forward together, which gave me enough motivation not to quit midway. At least for me, if I had been doing it alone, I probably would have stopped working on it partway through. The chart below is from the period when CV and LB were completely negatively correlated.

Ten days before the deadline, we merged with Raiii and Masuda, who had been participating as another team. We shared methods and ideas, built a reliable model that was the best at that point, and started creating ensembles for final submission. Then Raiii implemented a brilliant training idea three days before the deadline, thinking it might work. A single model with that method suddenly entered the 30s, and when we ensembled it, we climbed to 4th on the LB at once. The idea initially considered a graph-network solution, but the problem could be treated as a simpler 1:N problem, so perhaps it would train well if fed as a sequence. In hindsight that sounds plausible, but I would not have come up with the idea, and I was impressed by the skill needed to implement it in such a short time.

From this point the competition became too interesting. On the weekend right before the deadline, I spent almost all my time on Kaggle: creating various ensembles from new training methods, incorporating a way to automatically compute optimal blending weights for a weighted ensemble, because it is a linear combination with a unique optimum, and analyzing score data repeatedly. My small contribution was probably around there. It felt like an intense three-day festival. In the end, CV and LB scores became almost correlated, so we trusted the best CV and chose the final submission. On the morning of the deadline, we watched the result together while doing KPT and chatting. That gave the team a real sense of unity and achievement.

As written at the beginning, the final result was 8th place and a gold medal. It was simply very interesting. I was lucky to have strong team members, gain a lot of knowledge, and experience an exciting situation. This was an NLP competition, and because I went through the tutorials and beginner course for Transformers, I also gained practical knowledge that feels useful for work. Kaggle has competitions in many other domains too, and in unfamiliar domains everything seems fresh and full of things to learn.

My first Kaggle competition went very well because I was blessed with the team. I am grateful to the team members. But if you take a competition seriously, I think you should expect to spend at least about 50 hours on one competition. If you cannot make that time, joining may be difficult. This time, over about 40 days, I think I spent around 80 to 100 hours. Conversely, if you have the time, I felt that you can gain a lot from it, as described above. It is also simply fun if you think of it as a game. I would like to join another competition on a topic that interests me, but personally I may give up midway if I do it solo, so I would like to participate with at least one teammate for continuity and motivation.