Personal Computation Mediums & AI

by April 21, 2023

In his AI Speaker Series presentation at Sutter Hill Ventures, Geoffrey Litt discussed his work on personal computation mediums and the impact of generative AI on them. Here's my notes from his talk:

  • Alan Kay realized everyone would have their own computers and said: "We now want to edit our tools as we have previously edited our documents." But how can we actually make this vision real?
  • When it comes to computers and software, we were promised a bicycle for the mind. We got aircraft carriers instead. Bike are personal, modifiable, light. Aircraft carriers are industrial, heavy, and manufactured.
  • So how do we make software that is more personal and dynamic? how can our software be more like a bicycle? and how can AI help influence this?

Personal Dynamic Mediums

  • When someone sends you a spreadsheet, you don't just get the numbers, you also get little programs you can modify. There's editable bits of computation within.
  • Some additional examples of this idea: Webstrates allows you to edit a document in different editors and even edit the editors. Inkbase turns sketches into computational elements. Wildcard images every website was built with a spreadsheet that you can edit.
  • Dynamic software is software that is modifiable by the user at run time. Not just what the developer intended.
  • Potluck is an experiment in applying this idea to a note taking app for things like recipes: How do people save recipes? Usually in a notes app or dedicated recipe apps.
  • Text-based notes are flexible, but static. They're more like a bicycle.
  • Specialized apps are dynamic but rigid. Usually they are fixed to a domain, have a fixed feature set, and use rigid schemas to get data into right the right formats. These aircraft carrier apps are too much for what most people need.
  • Potluck uses a concept called gradual enrichment to try and give you best of both worlds. It uses extensible searches that allow you to decide what data you want to pull out that has meaning out of that text so that the computer can understand it.
  • Potluck then basically pulls that data into a spreadsheet so you can write little spreadsheet formulas and compute with it.
  • The last step is dynamic annotations where the app takes whatever you've computed in that spreadsheet and it's put it back in the text file.
  • Potluck highlights what elements can be turned into computational elements. Numbers become sliders to edit quantities, times become timers for cooking, etc.
  • This illustrates how to think about designing tools that are end user modifiable.
  • These kinds of tools can be reused across different domains, support multiple tools per medium, and allow people to make use of personal micro-syntaxes (when people decide on formats that make sense to them) because text is both an input and output in the UI.

AI Speaker Series April 20 2023 Geoffrey Litt

AI Opportunities

  • People who are not programmers however, really struggle to get going with tools like Potluck. Programmers tend to do pretty well because they are familiar with this kind of thinking. They do it every day.
  • Can we use large language models to plug that gap a bit?
  • This doesn't mean adding chatbots to the tool. Chatbots are often not the best UI. For example, to clip a video, a set of sliders with visual feedback is much better than using text to do a similar action through text. GUIs and direct manipulation have their advantages.
  • A bicycle is a tool that amplifies what you can do and provides you fine control. A chauffeur is more like an assistant that automates a task for you but to do so, you have to give up some control.
  • When you want tool feel, it's very important not to get a assistant feel.
  • But with fixed applications, you can get things done faster/better but are limited by what the developer decided you can do when building the app. So if you need a new feature, you have to go to the developer. That's a long and costly loop.
  • What if you could ship the feature you need to modify your app as you need it?
  • In all computational media, you ultimately need to have code that is written to perform functions. Someone needs to write the capabilities. An LLM can help you write these computational elements.
  • You could have written the formula but the UI is helping you to get things done. After working with it for a while, maybe you don't need it anymore cause you learned how to do it.
  • How do you know what the AI did? You can get written explanation from AI on how it implemented the feature.
  • This way of thinking about collaborating with AI really empowers the user, puts them in the driver's seat. They can add capabilities and also learn how.
  • Intent specification is iterative. Assume it won't be right on the first try. Iteration enables a higher ceiling, which is basic agile development.
  • Of course, there's more to figure out like: how do you design the iteration loop? what is the back and forth process to improve results driven by feedback?
  • But the possibilities of combining personal computation mediums with AI are exciting.