Horde3D
http://www.horde3d.org/forums/

Severe bug in Renderer::drawModels - wrong vertex layout
http://www.horde3d.org/forums/viewtopic.php?f=3&t=1171
Page 1 of 1

Author:  AlexL [ 02.06.2010, 11:30 ]
Post subject:  Severe bug in Renderer::drawModels - wrong vertex layout

Hi guys.

Over the last weeks we encountered some strange black triangles with some of our skinned models under some conditions.
We first thought the graphics card might be the problem or a buggy driver, and the problem persisted and finally I found a repro case (100%) on 3 different machines (NVIDIA Quadroplex, MacBook pro - GL 2.1 under Parallels and GTX 285, all running openSUSE 11.2).
It turned out that there is a problem in Renderer::drawModels (besides the one mentioned here: http://www.horde3d.org/forums/viewtopic.php?f=3&t=1155.
The actual problem is that the vertex layout is not applied correctly after a shader/material and geometry resource has changed.
This results in calling glDrawRangeElements with the wrong buffers bound and hence wrong vertices / and sometimes complete lock-ups.

Here is the scenario:
Model A: GeoResource GeoA, 1 mesh: Mesh A with Material MatA (Shader S1)
Model B: GeoResource GeoB, 1 mesh: Mesh B with Material MatB (Shader S1)
Model C: GeoResource GeoB, 1 mesh: Mesh B with Material MatB (Shader S1)

Code:
DrawModel call:
Model A:
    GeoResource GeoA bound
    Mesh A:
        - MatA and Shader S1 set
        - Vertex layout for GeoA and S1 set
        - Draw

Model B:
    GeoResource GeoB bound
    Mesh B: culled

Model C:
    GeoResource GeoB IS NOT re-bound, because already bound
    Mesh B:
         - MatB and Shader S1 set
!        - Vertex layout for GeoB and S1 IS NOT SET, because GeoResource and Shader didn't change since last Draw
!        - Draw: now with wrong Vertex Layout / Vertex Buffer combination


So, the simple fix is to add another flag "bool bForceApplyVertexLayout = false;" in line 1545.
Then add this:
Code:
            if( curMatRes != meshNode->getMaterialRes() )
            {
               if( !Modules::renderer().setMaterial( meshNode->getMaterialRes(), shaderContext ) )
               {
                  curMatRes = 0; // because setMaterial has a side-effect: renderer().getCurShader is 0 after it
                  continue;
               }
               curMatRes = meshNode->getMaterialRes();
               bForceApplyVertexLayout = true; // force re-apply of vertex layout, because shader changed
            }


And then
Code:
         // Apply vertex layout
         if( bForceApplyVertexLayout || curShader != prevShader || curGeoRes != prevGeoRes )
         {
            if( !Modules::renderer().applyVertexLayout( Modules::renderer()._vlModel ) )
            {
               Modules::renderer().setShaderComb( 0x0 );
               continue;
            }
            bForceApplyVertexLayout = false;
         }

This change will force a re-application of the vertex layout in case a shader did change, which has to be done anyway in that case.
The curShader != prevShader and curGeoRes != prevGeoRes check cannot catch the above mentioned scenario.
I guess one can further simplify the check, but currently no time.
If you need a SVN diff, let me know, although above changes should be ok for the current SVN 243.

Alex

P.S. sorry for lengthy post

Author:  marciano [ 03.06.2010, 18:17 ]
Post subject:  Re: Severe bug in Renderer::drawModels - wrong vertex layout

Hi Alex, as usual, thank you very much for the very detailed problem analysis. We have checked in a fix. It's slightly different from your proposed solution. Please let us know if it works for you. You really help a lot to make horde perfectly stable :)

Author:  AlexL [ 03.06.2010, 22:00 ]
Post subject:  Re: Severe bug in Renderer::drawModels - wrong vertex layout

Hi Marciano.
Yes, svn 245 works fine. Thank you and Volker for submitting! Very much appreciated.
Regarding the issue I prefer having explicit flags like the "applyVertexLayout" instead of the previous checks, as (to me) it makes the code more readable and more obvious why a certain code path is taken.
As we are heavily working with horde in a "real-word scenario" we're facing some of the remaining horde issues a bit more likely, but it's always great to have such a compact code-base (and also be able to fix the things ourselves), sth which isn't possible with torque, unity3d or other closed source stuff.
So I think it's a win-win :-)
And while working with horde we're coming up with a lot of feature requests...

Page 1 of 1 All times are UTC + 1 hour
Powered by phpBB® Forum Software © phpBB Group
https://www.phpbb.com/