Source
r/StableDiffusion•
Lerc

We need more ControlNet experimentation

Discussion

I have always hated the phrase "What's the use case?" As a reason not to do things. Especially with relation to implementing functionality that is designed to interact with other functionality. The possibilities are endless and often not obvious until discovered.

The recent explosion of images using QR Monster demonstrates the possibilities of using a tool for what it fundamentally does as opposed to what it was designed to do.

There are lot of people out there prepared to dedicate a great deal of GPU time to training Checkpoints and LoRas.

I'd like to call upon people to think up some new and strange Control Nets that might be used for things even if you don't don't yourself know what they would be good for.

Post below any ideas you might have.

Here's some starter ideas.

Perspective lines ControlNet. This would let you have a degree of control over image perspective without explicit lines appearing in the generated image as you would get with using canny (like on building edges).

Pixel variance ControlNet. A little had to describe this, but I imagined a preprocessor that firstly generates a difference map (let's call it QImage) of an image where each value is set to abs(max(thispixel - neighbour)) then a map of the radius around each pixel where the sum of all values in QImage crosses some threshold. In areas of high frequency information the radius would be small. In areas of low frequency information the radius would be large. A ControlNet trained on images using this preprocessor would be able to provide information of where image detail should be, but would not impose any particular orientation on that detail.

Posted Sep 23, 2023, 9:28 PM UTC

18
fewjative2•

The main problem is that most of the datasets for ControlNets are really big.

A) It's daunting to put together a CN if it's not something I readily need and hoping at least someone finds some use

B) Getting the data is a challenge in itself. The QR Monster CNet you see is on the 2nd or 3rd generation where the current training set is like 100,000 or 1,000,000 assets ( I know adding a 0 is a huge difference I just can't recall the exact # from the tweet )

Another issue is that current training sets and process are mostly private. There IS documentation on training but it would be really nice to see 'this is the set of data, this is how it was created, these are the parameters we used for training, etc' for all Cnets being made. Feels very gate kept. When I wanted to learn more about the QR Monster CNet, I reached out to the crew who made it and was ignored ( and this was before the CNet went big ).

1
[deleted]•

[removed]

1
rkfg_me•

I like the perspective idea but you need a preprocessor first. While for landscapes and cityscapes it's probably doable, what about portraits? I'm not sure how you'd infer the horizon position and the camera angle from that. Because every CN starts with a preprocessor, then you get a dataset with prompts and produce the corresponding controlling images from it, then train the CN on all that. So without a way to automatically create the perspective lines from any image, even those that don't have any reference points, it'd be a very hard task (unless you do it manually or use microtasking websites like Mechanical Turk). At best you'd only be able to make buildings and mountains.

1
LercOP•

I gave this some thought and did some googling. There are some tools for generating perspective lines from pictures but all examples I have seen rely heavily on regular artificial structures (such as buildings). Lines and boxes which are not so common in nature.

I wonder if you could build a training set based on pictures which do contain areas of straight lines and boxy shaped structures, but once perspective is determined, Clip (or inpaint if small central areas) a lot of the boxy bits so that the frequency of such structures is more representative of the standard.

Maybe something like.

Start with very large data set. Filter down to a set where perspective can automatically be determined with a high degree of confidence. Clip or inpaint images to produce a set of images more representative of the original data set.

The obvious solution would be to do a ML approach to generating perspective lines much like they use Midas for Depth, I am not aware of any such models for perspective lines though. I guess it's possible to train such a thing from particularly realistic computer graphics.

1
rkfg_me•

Interesting idea about inpainting! But it yet again boils down to manual work, unless you can automate finding such areas somehow. And I'm also not sure if training a model on generated data is a good idea. It's not proven AFAIK but it might harm the result quality, at least that was the motivation for watermarking the AI images.

1
LercOP•

I was thinking perhaps something like segment anything for auto detection and then only do inpaint if it were below a certain threshold of image size.

The sweet spot for this technique would be images with small regular structures that are enough to auto detect perspective but easy enough to remove.

Some training mechanisms have the ability to use an alpha map, where they don't train in the transparent area of images, This would be the best solution here, then you can arbitrarily cut holes in your data set and that just counts as "no-data" rather than a fixed color.

1
[deleted]•

[removed]

1
GeroldMeisinger•

ControlNet trainer here. I basically agree with what you say. What would help us even more would be:

* Do the first steps! see discussion here . u/aalmada researched edge detector alternatives and came up with the "edge drawing" algorithm, u/djbielejeski implemented a pre-processor in A1111 (without having model yet!), which prepared everything well enough for me to actually train the model . Having said that, you could research algorithms and compare and discuss them, or research and prepare image datasets, develop little helper scripts if you can... and let a trainer do the rest. Let's work together.

* Do structured evaluations! I'm bad at Stable Diffusion. Heck, I don't know the correct values for my own control nets, how would I? Hence I don't know if the training failed, the dataset was bad, my control net idea doesn't work or I'm just using SD wrong. It's so frustrating. Artists can do so much! u/BoostPixels this is so valuable Demystifying 'Spiral Effect': A Deep Dive into Parameters Please do more of this, make huge XYZ plots, annotate and discuss what you see, provide feedback!

* Do compiled research! Most of the things mentioned on reddit are already out there . I think it would benefit everyone if the information gets compiled and evaluated in a single place.

Thank you!

1
hung_process•

This is something I keep thinking about as well; in a similar vein to perspective I'd love to see a cnet trained on sequential art (probably not achievable yet but I can dream), as well as a model trained specifically on more dynamic human poses (acrobatics, martial arts) and foreshortening. But I'm completely ignorant about how cnet datasets are structured. Is it image/hint image pairs? Caption/image/hint image tuples? Something else entirely? To say nothing of the actual training process.

1