Scribbling Interfaces is an experimental, AI-powered design tool that translates scrappy wireframe sketches into components from an existing Design System.
It does so by inferring sketched element types using machine learning and translating them into a previously linked design components.
This allows for quick iterations in low fidelity while evaluating surfaces in high fidelity.
The plugin uses a TensorFlow object recognition model to recognize drawn elements and assign them to a component.
The model was custom-trained using a training set of 500 hand-drawn wireframes which contains a variation of image, button and text frames. 1
Putting things together
After training the model for several generations2 and running a few predictions on a local machine, I've wrapped the model in a small web app so friends and colleagues could interact with the model. This helped me to see if the predictions are accurate (enough) and if the whole idea sparks some excitement.
When inferring a YOLO object-recognition model, the model is 1.) fed with a source image, 2.) the model then does some inference magic and 3.) eventually returns the image along with respective bounding boxes of every predicted element.
These bounding boxes provide us with the width, height and x/y position of every detected elements — or in other words, a blueprint we can use to re-draw the image. All we have to do is substitute each bounding box with its respective detected element et voilà, we magically increased the fidelity of our wireframe.
In the initial napkin version, the translation of scribbles to wireframes was very naïve because it translates the bounding boxes 1:1, without paying attention to ex. alignment or layout. While this can be improved by aligning items in near proximity (as seen in the Figma plugin), there are certainly more sophisticated solutions which ex. could utilize Figma's built-in layout features. Certainly something to work on for a next version.
For the sake of keeping this page concise, I'm not diving into technical details or rationales. YOLOv4 was chosen due to its relative ease of use and because similar work used it and vouched for it ↩
A generation (or epoch) refers to one training cycle of a machine learning algorithm. Put simply, each generation increases the precision of the algorithm because the algorithm has more time and data to 'learn'. ↩