glGetTextureImage crashes?
I'm at my wits end here. I'm trying to bake a texture by rendering to a framebuffer object then fetching the pixels to do some (CPU-side) checks.
A very heavy shader bakes the texture; it needs to go over all triangles of a high-poly mesh to bake the vertex colors of it down to a low-poly target.
When the texture is <=512x512 and I run the full shader, it always works flawlessly.
When the texture is bigger than that, glGetTextureImage generates a STACK_BUFFER_OVERRUN (I'm using Rust) only if I run the full shader before it. If I comment out the for loop in the baking shader and just output something simple (like the vertex normals of the low-poly mesh itself) the texture resolution can be as high as I want.
I really don't understand this. Everything CPU side is 100% solid. Everything shader side is 100% solid. I can render high resolution textures and fetch the pixels just fine, but not when actually baking the data. I can bake the data just fine, but not when the texture resolution is too high.
The two seem completely unconnected, but still that combination is the only way to crash the app (and I think it crashes the OpenGL driver as well, since my IDE also uses OpenGL and sometimes closes when the crash happnes).
Does anyone have any idea why this might be happening?!
Posted Jul 26, 2023, 1:38 AM UTC
What happens if you call glFinish() before calling glGetTextureImage() after a full shader execute?
EDIT: If your GPU driver is resetting it could just be that your draw call is taking too long. I had this issue with some complex frag shaders and on Windows you can increase the GPU timeout via registry, but because I'm distributing software people use I had to implement a system that divides up the work for the whole input image into Y-slices (i.e. each draw call only works on a horizontal strip of the whole thing). Ideally I would switch to compute shaders but I have a lot of non-computer-centric older folk using my software on older hardware who've paid good money to use it and I'm not about to turn around and say "sorry, I don't like how my code is so I decided to force you to buy new hardware just to keep using my software that you paid for!"
EDIT 2: The default GPU timeout is 2 seconds, so if what you're doing is taking longer than that then that's your problem.
I agree that it sounds like a timeout or other similar issue. I'm guessing if you (OP) were to remove the
glGetTextureImagecall, you'd just get the crash on the next GL call.Compute shaders might not fix it either, I have the same issue with those for extreme workloads.
What I usually do is also just divide the work into several dispatches (or draws in this case), calling
glFinishinbetween groups of dispatches. There might be a more efficient "divider" thanglFinish, but I deem it not to make that much different since it's not that long if it's a 10-second workload requiring a couple ofglFinishcalls.I found while developing my software that because of the huge speed discrepancy between GPUs it was important that I roughly gauged the GPU on startup to determine how many separate draws (with glFinish called between them) to split a processing operation up into, without being excessive, because there certainly is some overhead cost incurred by issuing a draw call, so being able to generate as few calls as possible is ideal - but of course it really all depends on the application itself.
OP might find that some models will process just fine with a fixed number of draws to process it, but models with more vertices will still timeout the driver on some slower hardware. Even my 'gauging' that I implemented to try to scale processing to the capabilities of the hardware available still felt a bit jank and I had to leave some headroom in there. I opted to settle for trying to get draw calls to be a certain number of milliseconds. Originally I tried to do this dynamically - basically start with a draw call that's 1 row of pixels and then estimate by how quickly that executed to scale up the next strip to get toward a target number of milliseconds. That ended up not working so great (I don't remember why, just that it didn't) for some GPUs, and it wasn't ideal for performance.
Anyway... :P
Is there not a way for you to create some sort of optimization algorithm during installation or startup (executing)? That way, it can run an initial benchmark of sorts so your code will allow for appropriate draw calls without exceeding timeouts occurring?
That's what I meant by "gauged the GPU on startup". It generates progressively larger draw calls and times them until it gets a timing sample that takes at least 66ms, to get an idea how long it takes to process a megapixel of data. If the benchmark just did a fixed draw call size the timing would be obscured by bus speed and whatnot - it was important to find a decently sized draw call for the GPU to deal with to get a more accurate idea of how fast it was at processing, while simultaneously avoiding inadvertently stalling out the GPU driver just while benchmarking it.
Then when an actual operation is being performed on the GPU it uses the time it calculated during init that it takes to process a megapixel of data to scale the individual draws it splits it up into to where they're each ~20ms.
That's exactly what I tried before coming back to Reddit and that was indeed what happened. I had no clue there was a timeout for GL calls.
I already tried glFlush and glFinish.
I think the shader takes too long and the watchdog times out. I have to render it in tiles anyway if I want a progress bar, that's what I'm doing today. Probably will fix it.
I don't think you need to bother with tiles - I've had no issues just dividing up the draw along the Y axis into horizontal strips to be processed. Unless you think a single row of pixels might end up taking 2+ seconds there's not a need for dividing up the draw on both the X and Y, at least I would think, but you do you.
I also have some CPU processing/generation algorithms that I implemented multithreaded versions of and found that dividing an operation into a column of horizontal sections was faster just because it was more cache coherent - each thread would have a contiguous block of pixels to operate on, whereas with tiles each thread must read/write to spans of pixels that are farther apart for their individual tile. GPUs though tend to organize image data differently though (at least that's what we've been told) to improve data locality and splitting something into tiles probably won't hurt performance the way it does on CPU when compared to horizontal sections.
That's the thing, I'm importing ZBrush models and baking its vertex colors and getting a heightmap (normalized to [0, 1] but crucially with info to recalculate the actual offset, both negative and positive distance), I just caveman style go over all triangles of the highpoly mesh for all fragments of the lowpoly mesh's texture output. This means that a single row could indeed go up to more than 2 seconds.
The test object (a limestone rock I sculpted years ago) has 100K polys, as stated before rendering to >512x512 for the whole thing ends up taking longer than 2 seconds (on my trusty GTX1070). If I in the future want to bake 5M+ polys to a 4K texture (which I might do) I want to be sure I can get it to work.
My current thought is bucketing the rendering process (instead of the tiling stated before) in stages that take a predefined amount of triangles from the highpoly mesh, discarding any hits that are closer or further (depending on a variable). The shader already has a variable to specify the range inside the VBO/EBO (bound as SSBO) for the highpoly mesh to sample (since my implementation stores all meshes with the same vertex format into 1 VBO/EBO combo so I can adhere to AZDO principles).
That way I only have to tune to make sure a device can handle X triangles in under 2 seconds and I'm golden. I can tune for performance by widening/narrowing the amount of tris to render per "bucket".
/rant (sorry)