Ask LukeW: Integrated Video Experiences

by May 24, 2023

What happens when AI eats your Web site? The information you've spent years publishing gets reassembled in a new way for people to interact with your content. So far, we've integrated text-based articles and audio files into this experience on Ask LukeW. Today we're adding over 6,000 minutes of video as well.

The Ask LukeW feature is powered by AI models that break down content into related concepts and reassemble it into new answers. When that content is sourced from an article, audio file, and now video file, we cite it and give you the ability to go deeper into the original source file.

Video sources citations in  the Ask LukeW interface

In the example above, the information about loading indicators is sourced from two different videos which are linked in the source card to the right. Selecting one of these videos opens an integrated (within the broader conversational UI) video experience that looks like this:

Video specific features in  the Ask LukeW interface

The inline video player starting point is set to where the answer is mostly sourced from. I say "mostly" because a LLM generated answer may pull from multiple points in a video file to create a response, so we start you in the right vicinity of the topic being discussed. Below the video player is an indication of who is speaking that also serves as a scrubber to let you move through the file with context by displaying the current transcript directly underneath.

You can access the full speaker separated transcript and search within it as well. The transcript is generated with an automatic speech recognition AI model and diarized to indicate who is talking at any given time.

Video transcripts in  the Ask LukeW interface

For each video file integrated within Ask LukeW, we make use of LLM language operations to:

  • summarize the video file
  • extract speakers, locations, key topics, and even generate a title from the video file (if we lack the data)
  • create follow-on questions to ask based on the content of the video file
  • enable people to ask questions using just the content within the video file
  • generate answers in response to what people ask

It's like there's AI models everywhere. We even did a bit of machine learning to automate thumbnail selection for the videos. Just generate several thumbnails from each video, train on a few examples, and away you go.

Lastly but not least, we added the transcript search feature to all the audio files in Ask LukeW as well.

Audio transcripts in  the Ask LukeW interface

For more on how AI continues to eat this Web site, check out these earlier articles or go Ask LukeW about it.

Further Reading

Much thanks to Yangguang Li (front end), Thanh Tran (design), and Sam Pullara (back end) in helping these explorations move forward.