Vikathon — Hacking run, having fun
Code.Code.Code.Code.Code.FUN.Code.Code.Code.Code.FUN
Why do we do hackathons? Frankly speaking, I believe that there is no one “right” answer. It can be for serious reasons, creating POCs that tackle global problems, to silly reasons — like creating meme generators that aim to entertain the masses.
But it is not just about the results. The journey to attaining our goals in a short timespan, building camaraderie with teammates, and gaining new knowledge, are just as important.
Viki had previously organised a hackathon in celebration of this culture. This time, Viki embraced this tradition for the newer generation of Vikians, stylizing the internal hackathon as “Vikathon”. Any employee could participate in this two day event, up to a maximum of three for each team.
Hitting close to home, this Vikathon’s theme was focused on improving or reinventing our current services. This was an opportunity for developers such as myself, to pitch for a direction or area of interest that the company should advance towards. At the end of the two days, teams will be judged on three different areas - impact, technical depth and creativity.
I’m Weiyuan and this is my story for this Vikathon 😝
Day 0: The Day before “Vikathon”
There are hushed tones of excitement permeating throughout our workstations. Different teams were gathering in their individual spaces, discussing how to execute their visions over the next couple days.
For myself, it was another ordinary day. I did not sign up for the event simply because I did not had any interesting ideas to work on.
“Chocolate abs? Is that a new kind of candy?”
— Some guy. I cannot confirm or deny if I am that person.
That afternoon, I was approached by Kristie, my friend and colleague + community experience specialist. Now, Kristie is a person of many “interesting” ideas, but what she suggested now redefined that word entirely.
“Weiyuan, I need you to build something that that can recognize my Oppa’s chocolate abs!”
“Wait, what?!!”
After further discussion, I finally understood Kristie’s ideals. She wanted to examine different K-Dramas, and “collect” scenes where the male body was *ahem* “glorified”. These scenes will be saved within the celebrity collections, for their fans to view at their leisure.
Since having a full team was beneficial for getting more things done during the hackathon, we decided to scout Julius as our last member, who was a mobile engineer at Viki.
And so the team was formed… 😄
Later that night, we explored deeper and realized the underlying potentials of Kristie’s ideas. Abs recognized from a video via object recognition, lends credence to the untapped prospects of a VOD service, such as automatically generated highlights related to users’ interests. It could also benefit our sister service, Rakuten Sports, in automating highlight discovery for significant events, such as the act of scoring goals.
Another aspect was that facial recognition of celebrities could contribute to a new measurement relating to their involvement, complementing our current systems of rating scores and genre categories. For example, we could recommend fans of a certain celebrity other dramas, which will be sorted by the same celebrity’s screen time or involvement.
The problem was that none of us had any background in ML, or object + facial recognition. But hey, that’s what the hackathon is about!
Vikathon — Day 1, starting the hackathon
The hackathon started formally after a company all-hands. Participants gathered at the event hall for this Vikathon, to work on their ideas.
For my team, we divided the work according to our strengths. Julius would work on a mobile prototype that will showcase the presence of a celebrity and the “abs” highlights within a video feed. I would work on understanding how to harness object and facial recognition on a video input, and then outputting the various highlights in JSON format to interface with Julius’s planned implementation. Kristie would work on automation to gather images in training our models (and facial classifier) — and as our leader, keeping us on the same page at all times.
Since I had little experience with object recognition, I started off with a series of video guides (Note: I watched this till part 5), to gain better understanding on how I could utilize the open-sourced Tensorflow libraries and examples to perform object recognition.
The tutorials were encouraging, producing what seems to be highly accurate results as seen above. From the tutorial, which was written in python, I modified the logic to run iteratively on frames from a video feed instead of a single image. Then the following happened:
Since our intentions was to build a POC in recognizing “abs”, we could discard the inaccurate model used above, and refocus our efforts in training for an intended model.
While waiting for the images to be collected for training our “abs” model, I started work on understanding facial recognition. Object recognition and facial recognition stem from the same ML concepts. However, we could not utilize the same code from above, as facial recognition differ from object recognition in some ways. One way to describe the difference is that a face can be an object, but the distinctive features of a face (distance between features, shape, color and expressions) decides the actual identity of the same face.
Fortunately, there were facial recognition libraries out there that already solved the facial recognition problem. Following the same process with learning how to perform facial recognition as linked from the repository above, I experimented and endeavored to train our own classifier from the base model to recognize our fans’ “Oppas” and “Noonas” better.
At this point, the sun was setting and time for the first day was running out. Our team caught up on each of our individual progress — Julius showed us a prototype augmented from the current Viki’s android application with clickable highlights of a target celebrity.
Kristie also managed to automate the process for downloading our training images from Google. While the results were beautiful, for the next two days, my “Downloads” folder basically became the following:
With the gathered images, we started the training process which progressed throughout the night.
Vikathon — Day 2, finishing up and presentations
Day 2 started with some mishaps — the abs model was not sufficiently trained to provide for a high enough accuracy in recognizing abs. I decided to invest more time to train the model for more iterations. Things turned around and we managed to get more accurate content by noon.
(On the side) We started messing with the trained model, which produced some amusing results:
While waiting for the abs recognition model to finish training on my machine, I worked on getting the JSON output from the trained facial recognition classifier. Based on the facial recognition results, I added tolerance weights to each frame. This ensured that celebrities only needed to appear for a proportion of the frames in the assigned second, in order for the presence criteria to kick in. Further computation is performed to transformed the presence of each second to time ranges (representing a highlight clip).
To boost the accuracy of the gathered highlights further, I added in stitching logic to ensure that highlights that were considered close (below a defined time threshold) are combined. This helped to reduce the fragmentation of highlights caused by inaccurate data points and short transitions within the video.
To top the icing on the cake, scoring criteria was also added as discussed previously. This was measured as the total time presence, as well as close and far shots, based on the size of the face recognized in the video.
Combining the above logic with the trained abs recognition, we managed to achieve a POC (abs to be worked on!) of our intended objectives:
Wrapping up our development, Julius also shared his updates on a fully fledged application with the playable highlights. This application was also equipped with a home page POC with suggested content that fans could be interested in perusing, which we planned to show during the judging processions.
Presentations
For the judging process, two judges were selected from within Viki, and another two judges from Rakuten. Each team was given some time to share their end product with the judges and the audience, and the motivation behind realizing their visions through this Vikathon.
A total of 14 teams submitted their ideas to be presented. The ideas ranged from fun ideas, such as allowing users to interact with the video feed (throwing hearts when a certain character shows up), to ideas relating our day to day pain points such as a Spinnaker themed Continuous Delivery and/or Deployment setup (dependent on environment).
For my team, we settled on the theme as “Viki Moments”, showcasing Julius POC mobile application combined with the derived results from parsing videos worked on by Kristie and me.
Some of the presentations are currently being uploaded (and will be linked here soon!)
After the judging process, the following winners were announced:
Most Impactful — Thirst Quenchers (Julius, Weiyuan, Kristie) (Us!!!)
Best Technical Innovation — Dishdash Master, Chrome extension to remind us to order our daily lunch as well as to provide suggestions (Candra, Longfei)
Most Creative — OMO, a new interactive video experience (Stephanie, Laura, Annabel)
Special Rakuten Prize — RIT new translations and real time (Liling)
Congratulations to all who won!
As the festivities drew to a close, there was much to ponder. Looking back, there was a lot that was accomplished over the two days by each of the participating teams. Some of the ideas even started to come to fruition just slightly under a month after the event, by the initiative of the related engineers. There was also talk on discussing rolling out some of the projects as formal features of our services.
For myself, while I would not say that I am now a ML expert, there was much that I learn and grew to appreciate in this field over this short time span. There was also much solidarity and friendship exchanged between my team, with mutual support between teams in supporting each other during the presentation.
I think this chalks up as a success for this Vikathon! Looking forward to the next iteration and new ideas in the coming year!
