Staying up late, watching sprinting boxes

This is my journey to create the hardware rig, video processing pipeline, machine learning audit app, and footage annotation layers to create ultimate frisbee footage that my kids are excited to share with their friends. 

sprinting-boxes on the screen

What are you doing?

  1. Capture. Take two GoPros and aim them at different halves of the field. Together they capture the whole match. Nobody is watching the camera, everybody gets to watch the game.
  2. Stitch. Synchronize the footage from the two streams. Find an overlapping seam to help stitch the pair into a huge panorama.
  3. Detect. Find all of the people, track where they go, and apply ultimate frisbee heuristics to detect important events.
  4. Audit. Can’t let the perfect be the enemy of the good. Real footage is messy and sometimes people crowding the sideline confuses things. I might improve that eventually, but for now we give all of our hypotheses a quick human in the loop audit. They confirm things like:
    • When a pull happens, which side did it, which team it was
    • When a score happens
    • Halftime
    • Person tracking blips
  5. Reframe. Since we know where all the people are, we can automate the story of the game. We can build a path of zooms and pans to keep the action in the frame.
  6. Publish. Make it pretty! Cut out dead space, add a scoreboard and point intros, bookmark timestamps for each pull, and upload to youtube!

A tinkerer’s dream: the journey is the destination

I was recently asked, “what would the title of your story be?” After some introspection, I realized that I am a tinkerer at heart. I have gone through life just wondering about random stuff. I coined a blog back in the day called inner geek moments. I’d get distracted thinking about people trying to clap in unison at a concert, with sound waves hitting your ears at irregular random intervals. On my pre-honeymoon, I found myself explaining different levels of infinity while walking around a rainforest in the Philippines.

When thinking about Sprinting Boxes, my mind went in a hundred different directions. I needed to keep a huge record of ideas to chase down, experiments to run, or just cool technologies to try out along the way. I relished having a problem that was adjacent to several passions in my life: software and ultimate frisbee. It included a bunch of new areas for exploration: cameras and computer vision, ai programming (yes, the robots did most of the work), automating video production. Over the course of my writing, you’re going to see just how deep some of these rabbit holes go.

I’ll be breaking down the journey into posts and eventually cross-linked wiki pages to possibly satisfy anyone else’s curiosity. We’ll showcase what I tried, what I measured, what I got wrong, and probably have a bunch of random sci-fi and fantasy references sprinkled along the way.