AI Art Generator With Conditioning Using ControlNet

You want to inspire your kid to learn yoga? One option is to connect yoga with one of the kid's favorite character such as Kung-fu Panda. It is now so easy to generate AI Art with conditions like that using state-of-the-art AI model ControlNet.

ControlNet was introduced in Adding Conditional Control to Text-to-Image Diffusion Models by Lvmin Zhang and Maneesh Agrawala. It introduces a framework that allows for supporting various spatial contexts that can serve as additional conditionings to Diffusion models such as Stable Diffusion.

Training ControlNet is as following:

1) Cloning the pre-trained parameters of a Diffusion model, such as Stable Diffusion's latent UNet (trainable copy), while also maintaining the pre-trained parameters separately (locked copy). It is done so that the locked parameter copy can preserve the vast knowledge learned from a large dataset, whereas the trainable copy is employed to learn task-specific aspects.

2) The trainable and locked copies of the parameters are connected via “zero convolution” layers which are optimized as a part of the ControlNet framework. This is a training trick to preserve the semantics already learned by frozen model as the new conditions are trained.

Example is below:

The Yoga poses to be used as conditions for ControlNet model

The sample prompt:

"a cute adorable kungfu panda character in japanese garden,physically based rendering,3d blender render, beautiful bright colors, 8k render,best quality, extremely detailed"

The conditions to the ControlNet can be simply added as parameter (in the example is "yogaposes"):

output = pipe(
    [prompt] * 4,
    yogaposes,    
    generator=generator,
    num_inference_steps=50,
)

And the generated images are created under that yoga poses condition that we specify for the generation process.

The generated Panda under the yoga poses conditions

The sample code can be download from Github: https://github.com/tsnguyenducphuong/ai-art-public/blob/main/Control_AI_Art_Generated_Image_With_ControlNet.ipynb.

About AI Illustwitter, an amazing AI-powered Creative Design App: It so simple and quick to generate high quality, super resolution (8K, 8192x8192 pixels) images within seconds from text description on mobile devices. It can be used to generate AI artworks, nice images for design prototypes, illustrated images for social media posts, news.
There is no limit what kind of images can be generated from text using AI Illustwitter. The artwork can then be shared via AR-vatar and viewed in Augmented Reality.
In addition, you will enjoy the advanced AI-powered Image Editing features to named a few: Generative Fill, Generative Expand, Magic Move, AI Upscaler (upscale to 4X, 16X, and 64X to 8K 8192x8192 pixels images all in one go!), background remover, change background easily using text description and last but not least the magic AI Layering and automatic Repositioning in which intricate image is break down into separate layers and automatically reposition objects, transforming the way we approach image editing. AI Illustwitter is available to download from Apple Store and Google Play. Check it out!

Author: AI Illustwitter

Next article to read:

PNG 2 SVG

In the ever-evolving world of digital design, efficiency and versatility are keys. One of the most common challenges designers face is converting raster images, like PNGs, into scalable vector graphics (SVGs). Fortunately, AI-powered tools are stepping in to simplify this process, enabling designers to unlock new creative possibiliti1es without sacrificing quality. Read more about AI Illustwitter at: https://ai-illustwitter.art

2nd November 2024

5 min read

Magic 3D

Whether you're creating content for social media, preparing e-commerce product's catalog, or creating marketing materials, a beautiful 3D animation video can significantly impact the perception of your product. The right animation video can catch user's attention for your product. With AI Illustwitter Magic 3D tool, now you can create beautiful 3D animation of your product from a single image...All with a simple click. Read more about AI Illustwitter at: https://ai-illustwitter.art

22nd October 2024

5 min read

Magic Background

Creative expression. Whether you're creating content for social media, preparing e-commerce product's catalog, or creating marketing materials, the image's background can significantly impact the perception of your image. The right background can set the tone for your image. With AI Illustwitter Magic Background tool, now you can remove the background, create realistic shadow, generate blur background, change background image, or convert to negative background color...All with a simple click. Read more about AI Illustwitter at: https://ai-illustwitter.art

30th September 2024

5 min read

Magic Overlay

Remember the time using tool like Photoshop to combine multiple images into one frame? That could be a time-consuming task for creating layer masks, multiple layers and arrange them over your original base image. Now with AI Illustwitter Magic Fill and Magic Overlay, adding new objects or place it in the middle layer is just a simple click. Read more about Magic Overlay at: https://ai-illustwitter.art

24th September 2024

5 min read