Horde3D

I'm currently doing chroma keying in glsl on the playing video. Everything works fine, but the edges of the actor are rather rough, so I decided to blur the texture. It seems that the most popular blur method is gaussian blur, but it is done in two passes. Is it possible to blur only the texture of the object after keying and not the whole screen? If the blur is done in two passes then what steps should be made? Draw the geometry (actor plane) with the keying shader and then redraw it with another shader?
I'm using the standard forward pipeline, modified to support antialiasing (draw to texture, then draw quad to framebuffer).

I am not sure if I am following you 100%, but you can add another buffer and processing step into your pipeline to create that extra pass to create a Gaussian blur. You can use the depth buffer to filter out objects from the background (check the wiki).

However, another approach would be to not use Gauss blurring and instead use a cheap single pass blur filter, if that makes it easier for you. For that just grab the 4 surrounding pixels and average them in the shader. Here is a more complicated example:
http://www.pasteall.org/10779

Thanks, blurring with 4 surrounding pixel is sufficient. Keying is rather good now .