Back in the game after a long time

Last time I experimented with some models locally was when SD (XL I believe?) came out, I was still using Automatic1111, but I only played around with it very briefly, I lost interest and my 2070S wasn’t enough to test larger models that came out later.

I’ve heard about Z-Image recently, and it looks like many other models came out in the meantime, ComfyUI seems far more polished than when I first tried it, it’s been a while.

I’d really like to get a more general overview of the current landscape of self hosted generative AI, what in your opinion have been some breakthroughs in the field, and what kind of expectations I should have with my current hardware (which is still my old 2070S).

I’m considering investing some time again now that the ecosystem and tooling seems more mature, but at the moment I just don’t have the budget for an expensive Nvidia GPU.

What’s the state of running local models on, say, Intel Arc cards? I believe I should be able to get my hands on some 16 GB cards that are far cheaper than any current gen Nvidia of equivalent VRAM. Of course I don’t expect any similar performance without CUDA, but more VRAM and even a slight improvement on generation speed over a 2070S would be a good start for the time being. Same reasoning applies to AMD of course, I’m really not up to date with the market right now.

What would really be useful to me is image editing, e. g. being able to selectively modify photos of real humans and change them through prompts, primarily just changing environment, lighting, poses, facial expressions, clothing, race, physical build, etc, while preserving the exact same facial traits.

The result should be a real photo of the same person, just taken in a different context.

This is something that proprietary models like the ones used in ChatGPT and Gemini can’t do as far as I understand, they always seem to reconstruct a subject that kinda resembles the original, but is obviously different.

On the other hand, I’ve had really good success with this using an iOS app called Hypic, before they locked its AI Editor behind a paywall. I have no idea what kind of model it runs, but results were good with the right prompts.

I’m a developer, I have no issues self hosting anything, but I’d just like to set my expectations straight on what I could potentially achieve with a small initial investment, and eventually spend more money for a higher end GPU, should generative models provide value to me in the long run.

A bonus would be 3D models generations for my other hobbies, but not essential. Things like image to video models would be cool too, but again, not super important and I imagine those models might need much more vram.

I haven’t really played around with Loras and upscaling, or anything other than basic image generation through a checkpoint model, I’d be happy to hear what you use all of these fancy workflows for and experiment with them myself