Advanced Smart Crop — Intelligent, Automatic Image Cropping
We have made significant improvements to our Smart Crop algorithm making automatic image crops even better. With improved face detection, feature detection, a new object-aware crop mode with 80+ named objects, and other crop modes, the new algorithms provide higher cropping accuracy for facial, non-facial, and multi-object images. We have updated the text and examples to demonstrate this improvement.
Each image is different. Each device is different. Therefore, each image needs to be treated differently to build a perfect responsive experience across devices. Smart Crop is a feature that intelligently looks at the content of an image and automatically focuses on the most important part of an image for cropping.
What is Smart Crop?
When we resize images for different devices or placeholders, and the aspect ratio of the requested size does not match that of the original image, some cropping is bound to take place on the sides of the image. That is when we do not want to stretch or skew the image forcefully.
This can be seen from the graphic below, where we request a 1:1 aspect ratio output from a 4:3 aspect ratio input image -
Now, since cropping would take place unavoidably in these circumstances, if the subject of the image is not in the centre then it would get cropped out of the image.
For example, if I try to resize the image below to get a squared thumbnail, the sides of the image (the buildings) will get cropped away.
Smart Crop is a way of intelligently determining the most important part of the image and keeping it in focus while cropping the image. By using this feature you would be able to generate perfect thumbnails that are centred on the image's subject, or the object you specify, or the most important feature with a high probability.
Let's take the original image below as an example and see how Smart Crop can automatically center around the bicycle, when the default center crop leaves it out.
Default Crop Mode: The bike gets cropped out here
Auto Smart Crop Mode: The bike is now in the center of the thumbnail automatically.
As demonstrated, ImageKit's Smart Crop could intelligently identify the key object in the image and preserved it in the final thumbnail.
How do we determine the most important part of the image?
There are a few different ways in which Smart Crop works in ImageKit to identify the area to crop around. It depends on the mode of smart crop you use in the URL.
Smart cropping algorithm in auto crop mode: In this mode, ImageKit tries to detect faces in the image using machine learning. If a face (or multiple faces) are found, we center around them as closely as possible. If ImageKit cannot find a face in the image, we try to do a feature detection on the image. The center of such non-facial features is then used to calculate the cropping dimensions on the original image.
Smart cropping algorithm in face crop mode: In this mode, we try to detect a face in the image and crop closely around that. If multiple faces are found, we crop around the most prominent face (determined by the number of pixels occupied) found in the image.
Smart cropping algorithm in object-aware crop mode: In this mode, we use machine learning to find the object mentioned in the URL inside the image. If the object is found, the crop is created around it.
Different modes of Smart Crop in ImageKit
ImageKit offers three crop modes to use with smart cropping - the auto crop mode, the face crop mode, and the object-aware crop mode.
Auto and Face-crop mode in Smart Crop
In auto crop mode, we try to preserve as much of the image as possible, while centering on the most important part of the image. This is useful when you are trying to create an image gallery. Each thumbnail in that gallery will contain the most important part of the image. You need to use the "fo-auto" transform for this crop mode.
In the face crop mode, we try to extract out the largest face from the image, and the rest of the image is discarded. This is useful when you are trying to create profile pictures from a larger image. You need to use the "fo-face" transform for this crop mode.
For example, let's use the image below and run both auto and face crops on the same image to understand how the two work.
Original image
Smart crop with "auto" crop mode: The thumbnail centers around the boy instead of the center of the image, but it also includes the parts of the image around the boy.
Smart crop with "face" crop mode: The thumbnail is cropped tightly around the boy's face
You can also control the zoom in face crop mode using the zoom (z) parameter in ImageKit. Specifying a value less than 1, zooms out of the face, whereas a value greater than 1, zooms into the face. For example, to zoom out from the face in the above example, we can add z-0.75 to the URL.
https://ik.imagekit.io/ikmedia/Graphics/smart_crop/Smart_crop_example.jpg?tr=w-300,h-300,fo-face,z-0.75The output is a less tight crop around the face.
Object-aware cropping mode
As an extension of smart crop and face crop, object-aware crop can be used when you want to single out a specific object from a multi-object image.
Use the parameter fo-<object name> in your URL to carry out the transformation and you can choose from a variety of 80+ objects ranging from food, drink, animals, everyday essentials and more. The full list of objects that can be identified in an image can be found here.
For example, if you have an image of a living room, you can crop around the couch or the chair by specifying the focus parameter as couch or chair in the URL transformation, respectively.
Original Image of the living room
Cropped around the couch
Cropped around the chair
How can I start using Smart Crop for images?
Like all other image transformations and optimizations in ImageKit, this feature too works in real-time by simply adding a parameter to the URL. In this case, we need to set the value of "focus" in the transformation string to "auto" (for auto crop mode) or "face" (for face crop mode) or <object-name> (for object-aware crop mode).
For auto and face crop modes to work, you need to specify both, height and width of the output image. If you specify only the height or the width, then ImageKit maintains the aspect ratio without cropping. You can read more about different focus modes in our detailed documentation here.
If your normal thumbnail URL is
https://ik.imagekit.io/demo/img/tr:h-200,w-200/default-image.jpgTo enable Smart Crop with auto crop mode, this URL will change to
https://ik.imagekit.io/demo/img/tr:h-200,w-200,fo-auto/default-image.jpgTo enable Smart Crop with face crop mode, this URL will change to
https://ik.imagekit.io/demo/img/tr:h-200,w-200,fo-face/default-image.jpgTo enable Smart Crop with object-aware crop around a "couch", this URL will change to
https://ik.imagekit.io/demo/img/tr:h-200,w-200,fo-couch/default-image.jpgExamples of Smart Crop in action (Auto-Mode)
Example 1
Example 2
Example 3
Examples of Smart Crop in action (Face-Mode)
Example 1
Example 2
Example 3
Examples of object-aware crop mode in action
Example 1
fo-person)fo-boat)Example 2
fo-refrigerator)fo-oven)Example 3
fo-pottedPlant)fo-chair)Best applications for this feature
Smart Crop will work well for any case where you need to create smaller images or thumbnails from a larger image, like creating user profile picture thumbnails from a larger profile picture using the face crop mode, or creating product image galleries where each image should have the product in the centre regardless of where it was in the original image.
Can I use the Smart Crop feature with other transforms
Of course, you can. You can combine smart cropping with any other transformation offered by ImageKit.
For example, you can combine smart cropping with responsive image tags to create a perfect responsive image experience across all devices for your users. You can also club it with a radius transform in a chain, to create rounded profile pictures of your users like this
Or, as shown below, you can combine object-aware cropping with the Layers transformations in ImageKit to create rich banners for your products in real time.
Feedback related to Smart Crop
While we have tested the performance and results for a wide array of images, smart cropping may return inaccurate results for a small percentage of images. Smart cropping is a difficult problem to solve while keeping things near real-time. We are open to your feedback that will help us further improve the accuracy of the cropping algorithm. Please give it a try and do share your feedback in the comments below. Or write to us on developer@imagekit.io.
Like the idea? Want to try it out for yourself? You can try it for free!