Advanced Smart Crop – intelligent, automatic image cropping with ImageKit

We have made significant improvements to our smart crop algorithm which will make automatic image crops even better. With improved face detection, feature detection and different crop modes, the new algorithm provides higher cropping accuracy for both, facial and non-facial images. We have updated the text and examples to demonstrate this improvement.

Each image is different. Each device is different. Therefore, each image needs to be treated differently to build a perfect responsive experience across devices. Smart Crop is a feature that intelligently looks at the content of an image and automatically focusses on the most important part of an image for cropping.

What is Smart Crop?

When we resize images for different devices or placeholders and the aspect ratio of the requested size does not match that of the original image, some cropping is bound to take place on the sides of the image. That is when we do not want to stretch or skew the image forcefully. This can be seen from the graphic below where we request a 1:1 aspect ratio output from a 4:3 aspect ratio input image –

demonstrate cropping while resizing

Now, since cropping would take place unavoidably in these circumstances, if the subject of the image is not in the centre then it would get cropped out of the image. For example, if I try to resize the image below to get a squared thumbnail, the sides of the image (the part with the buildings) will get cropped.

Original Image
A 200×200 image, buildings get cropped on the sides with regular cropping

Smart Crop is a way of intelligently determining the most important part of the image and keeping it in focus while cropping the image. By using this feature you would be able to generate perfect thumbnails that are centred on the image’s subject or the most important feature with a high probability

For example, for the above image, Smart Crop generates the following thumbnail

Original Image
A 200×200 image, buildings on the side are in focus with smart cropping

As you can see, ImageKit’s Smart Crop could intelligently determine that the buildings in the image are important and preserved them in the final thumbnail.

How do we determine the most important part of the image?

We take into account multiple factors while calculating the most important region of the image. The most obvious bit is to detect a face (or multiple faces) in an image. If we find a face in the image, we try to centre the cropping on that face as closely as possible. If, however, we are not able to find a face in the image, we try to do a feature detection on the image. The centre of such non-facial features is then used to calculate the cropping dimensions on the original image.

Can we control how the cropping is done?

ImageKit offers two crop modes to use with smart cropping. One is the auto crop mode and the other is the face crop mode.

In auto crop mode, we try to preserve as much of the image as possible, while centring on the most important part of the image. This is useful when you are trying to create an image gallery. Each thumbnail in that gallery will contain the most important part of the image. You need to use the “fo-auto” transform for this crop mode.

Original image
Face Crop mode
Auto Crop mode

In the face crop mode, we try to extract out the largest face from the image and the rest of the image is discarded. This is useful when you are trying to create profile pictures from a larger image. You need to use the “fo-face” transform for this crop mode.

How can I start using Smart Crop for images?

If you do not have an ImageKit account, you can register for a free trial here.

Like all other image transformations and optimizations in ImageKit, this feature too works in real-time by simply adding a parameter to the URL. In this case, we need to set the value of “focus” in the transformation string to “auto” (for auto crop mode) or “face” (for face crop mode). This means, you need to add fo-auto or fo-face to the transformation string of your thumbnail URLs.

For smart crop to work, you need to specify both height and width of the output image. If you specify only the height or only the width then ImageKit maintains the aspect ratio without cropping. You can read more about different focus modes in our detailed documentation here.

If your normal thumbnail URL is

https://ik.imagekit.io/demo/img/tr:h-200,w-200/default-image.jpg

To enable smart crop with auto crop mode, this URL will change to
https://ik.imagekit.io/demo/img/tr:h-200,w-200,fo-auto/default-image.jpg

To enable smart crop with face crop mode, this URL will change to
https://ik.imagekit.io/demo/img/tr:h-200,w-200,fo-face/default-image.jpg

Examples of Smart Crop in action (Auto-Mode)

Original Image
Normal Crop, 200x200px thumbnail
Smart Cropped 200x200px thumbnail

 

Original Image
Normal Crop, 200x200px thumbnail
Smart Cropped 200x200px thumbnail

 

Original Image
Normal Crop, 200x200px thumbnail
Smart Cropped 200x200px thumbnail

 

Image example – Person
Normal Crop, 200x200px thumbnail
Smart Cropped 200x200px thumbnail

 

Image example – Object
Normal Crop, 200x200px thumbnail
Smart Cropped 200x200px thumbnail

 

Original Image
Normal Crop, 300x100px thumbnail
Smart Cropped 300x100px thumbnail

 

Examples of Smart Crop in action (Face-Mode)

Original Image
Auto Crop Mode, 200x200px thumbnail
Face Crop Mode 200x200px thumbnail

 

Original Image
Auto Crop Mode, 200x200px thumbnail
Face Crop Mode 200x200px thumbnail

 

Original Image
Auto Crop Mode, 200x200px thumbnail
Face Crop Mode 200x200px thumbnail

 

Original Image
Auto Crop Mode, 200x200px thumbnail
Face Crop Mode 200x200px thumbnail

 

Best applications for this feature

Smart Crop will work well for any case where you need to create smaller images or thumbnails from a larger image. A few examples could be creating user profile pic thumbnails from a larger profile pic using the face crop mode. Or for creating product image galleries where each image should have the product in the centre regardless of where it was in the original image.

Can I use this smart crop feature with other transforms

Of course, you can. You can combine smart cropping with any other transformation offered by ImageKit. For example, you can combine smart cropping with responsive image tags to create a perfect responsive image experience across all devices for your users. You can also club it with a radius transform in a chain, to create rounded profile pictures of your users like this


https://ik.imagekit.io/demo/img/tr:w-200,h-200,fo-face:r-max/https://images.pexels.com/photos/874158/pexels-photo-874158.jpeg

Feedback related to smart crop

While we have tested the performance and results for a wide variety of images, smart cropping may return inaccurate results for a small percentage of images. Smart cropping is a difficult problem to solve while keeping things near real-time. We are open to your feedback that will help us further improve the accuracy of the cropping algorithm. Please give it a try and do share your feedback in the comments below. Or you can write to us on developer@imagekit.io.