It comes as no surprise that when it comes to image-rich web applications, every single image matters in defining the quality of experience (QoE) for the end user. So how does one offer individually-tuned settings for optimal image delivery while being able to scale up to millions of images across the entire web delivery pipeline? There’s some fun math that goes into answering this question; at Instart Logic, we call it SmartVision technology. Today, aligned with the public release of our first formal academic publication describing SmartVision technology, let me use this blog post to give you the basic ideas behind the technical core of this technology and how it enables optimized delivery of image-rich web applications as a whole, while selecting individually-tuned settings for each image within a given web application.
Intuitively speaking, the key to optimal delivery of an image is to have a content-dependent signature (or hash code) for computing the impact of web delivery on the given image, and using said signature to prioritize various constituent parts of the image file. In our work, we developed a simple computational signature that captures the impact of web delivery pipeline on image quality; we call it VoQS (variation of quality signature). In our experiments, we also discovered that large corpuses of images can be effectively split into coherent clusters based on the VoQS similarity. Taken together, these two simple insights combine to result in an efficient algorithmic approach – SmartVision – for finding adaptive settings for each individual image, delivered via a web delivery service.
For technical details on the algorithm and experimental results on empirical datasets, please see the academic publication that we are presenting today at the ACM (Association for Computing Machinery) Multimedia Conference. While there is a large body of research out there on the topics of image categorization and computer vision-based image content analysis, our paper is one of the first publications (to our knowledge) where quality-dependent image categorization in the context of web delivery is directly addressed.
The following flowchart shows how the SmartVision algorithm works:
As you can see in the flowchart, the categorization part can be done offline (with intermittent updates) to adapt to a changing image corpus pooled across the web delivery service. The real-time aspect simply depends on efficient computation of VoQS and a nearest-neighbor lookup against the pre-stored exemplars, that were estimated during the offline categorization step.
While message-passing algorithms such as Affinity Propagation [Frey & Dueck, 2007] offer the advantage that one doesn’t need to pre-specify the number of expected clusters as well as get the cluster-specific exemplars as a side product, the algorithmic complexity of Affinity Propagation makes it impractical for really large image datasets (such as the ones we encounter with our Software-Defined Application Delivery service). In scenarios where the image corpus is very large, one can use faster algorithms such as K-means (with appropriate care and safety checks) for clustering, and choose the image exemplars by minimizing aggregate distance in the VoQS metric space. It is worth noting that the entire algorithmic flow (and the categorization aspect) happens in an unsupervised fashion – so it is highly amenable to automation in the context of an always-on web delivery service. In our experiments, we found that we could find optimal delivery thresholds for a large corpus of images quickly, while minimizing the loss of visual quality (see Figure-3 in our ACM-Multimedia paper). In addition, our approach is not really dependent on any particular image format; thus, we can apply it for most of the popular image formats used by the web community.
So what can you do when the client isn’t installed on the target device, such as is the case with a native mobile application?
For further technical details and empirical experimental results, click on this link to access our ACM Multimedia publication.