AI Art Generator With Conditioning Using ControlNet

You want to inspire your kid to learn yoga? One option is to connect yoga with one of the kid's favorite character such as Kung-fu Panda. It is now so easy to generate AI Art with conditions like that using state-of-the-art AI model ControlNet.

ControlNet was introduced in Adding Conditional Control to Text-to-Image Diffusion Models by Lvmin Zhang and Maneesh Agrawala. It introduces a framework that allows for supporting various spatial contexts that can serve as additional conditionings to Diffusion models such as Stable Diffusion.

Training ControlNet is as following:

1) Cloning the pre-trained parameters of a Diffusion model, such as Stable Diffusion's latent UNet (trainable copy), while also maintaining the pre-trained parameters separately (locked copy). It is done so that the locked parameter copy can preserve the vast knowledge learned from a large dataset, whereas the trainable copy is employed to learn task-specific aspects.

2) The trainable and locked copies of the parameters are connected via “zero convolution” layers which are optimized as a part of the ControlNet framework. This is a training trick to preserve the semantics already learned by frozen model as the new conditions are trained.

Example is below:

The Yoga poses to be used as conditions for ControlNet model

The sample prompt:

"a cute adorable kungfu panda character in japanese garden,physically based rendering,3d blender render, beautiful bright colors, 8k render,best quality, extremely detailed"

The conditions to the ControlNet can be simply added as parameter (in the example is "yogaposes"):

output = pipe(
    [prompt] * 4,
    yogaposes,    
    generator=generator,
    num_inference_steps=50,
)

And the generated images are created under that yoga poses condition that we specify for the generation process.

The generated Panda under the yoga poses conditions

The sample code can be download from Github: https://github.com/tsnguyenducphuong/ai-art-public/blob/main/Control_AI_Art_Generated_Image_With_ControlNet.ipynb.

About AI Illustwitter, an amazing AI-powered Creative Design App: It so simple and quick to generate high quality, super resolution (8K, 8192x8192 pixels) images within seconds from text description on mobile devices. It can be used to generate AI artworks, nice images for design prototypes, illustrated images for social media posts, news.
There is no limit what kind of images can be generated from text using AI Illustwitter. The artwork can then be shared via AR-vatar and viewed in Augmented Reality.
In addition, you will enjoy the advanced AI-powered Image Editing features to named a few: Generative Fill, Generative Expand, Magic Move, AI Upscaler (upscale to 4X, 16X, and 64X to 8K 8192x8192 pixels images all in one go!), background remover, change background easily using text description and last but not least the magic AI Layering and automatic Repositioning in which intricate image is break down into separate layers and automatically reposition objects, transforming the way we approach image editing. AI Illustwitter is available to download from Apple Store and Google Play. Check it out!

Author: AI Illustwitter

Next article to read:

Artography

A stunning combination of photography and painting offers an incredible opportunity to explore creativity in ways that were previously unimaginable. As Generative AI continues to evolve, the possibilities for creative experimentation expand. AI Illustwitter Artography tool uses advanced algorithm to analyze the photo you upload and magically apply painting-like effects to one part of the image. The result is a beautiful transformation where reality and creativity exist side by side: half of the image is a photograph, and the other half transforms into a beautiful painting.

13th January 2025

5 min read

Magic Renovate

Renovating your home is an exciting yet often daunting process. Deciding on the perfect design, materials, and layout requires a lot of time, effort, and creativity. However, with the advancement of Generative AI, homeowners can now make renovation decisions more confidently and easily. One such innovative tool that is transforming the renovation process is the AI-powered photo editor that generates beautiful renovation ideas by simply uploading an image. From a DIY project to a major renovation makeover, remodeling anything in your house is as easy as clicking a button with AI Illustwitter Magic Renovate tool: https://ai-illustwitter.art

13th January 2025

5 min read

Paintify

In the digital age, photo editing has evolved far beyond the traditional tools and techniques of yesteryears. Thanks to advancements in artificial intelligence (AI), you can now take your photos and seamlessly transform them into stunning works of art. Read more at AI Illustwitter website: https://ai-illustwitter.art

4th January 2025

5 min read

Light Transfer

AI Illustwitter Light Transfer tool seamlessly transfers the lighting from one photo to another, giving your photo the beautiful lighting as you want. No more endless manual tweaking, let AI Illustwitter handle the time-consuming task for you. Read more about AI Illustwitter at: https://ai-illustwitter.art

14th December 2024

5 min read