Horde3D

Next-Generation Graphics Engine
It is currently 28.03.2024, 09:07

All times are UTC + 1 hour




Post new topic Reply to topic  [ 69 posts ]  Go to page Previous  1, 2, 3, 4, 5  Next
Author Message
 Post subject: Re: Texture Compression
PostPosted: 27.08.2008, 19:43 
Offline

Joined: 26.03.2008, 02:58
Posts: 160
swiftcoder wrote:
Most OpenGL ES1.1 hardware doesn't have ST3C, although that may change as ES 2.0+ becomes common.

I have no particular need for it, since all my targets feature S3TC, so a software fallback is indifferent for my particular needs.

Quote:
As a specific example, Apple and PowerVR both provide a wrapper format for PVRTC, and a toolchain to produce them. Although they use the same compression, the wrapper format is non-compatible, and supports slightly different features. The Apple tool runs only on Macs, while the PowerVR tool runs only on Linux and Windows.


I have never used a Mac platform, nor developed for the iphone. I also was not aware that that they provided a PVRTC toolchain/wrapper, I hope it is possible to abstract the differences and use a unified set of functions to work with PVRTC textures.

Quote:
Speed/quality isn't really an issue. However, I do wonder how people would feel about rolling this out of the engine entirely?

I am talking about building a Horde3D-specific (though the format would be open, of course) wrapper format, which can hold all types of compressed/uncompressed image data (uncompressed, RLE, JPEG, S3TC, PVRTC, ETC, etc.). There would be a tool that leveraged FreeImage (or similar) to convert/batch convert other image formats into the Horde format, which would move this whole issue out of the engine and into the content pipeline.

I have a feeling it is too big of a break from tradition to be well received - but it is the approach taken by the WildMagic engine.

I don't think i understand what you mean. Can you provide more details on how this woul work? what are the benefits and issues?

Quote:
FreeImage doesn't offer anything apart from .PSD support, over what we have already - and it doesn't have mipmap generation or DDS support, last I checked.

DDS is supported. I am not advocating for freeimage, just putting a question in the open regarding the need for a more advanced image manipulation libraries, DevIL or any other that does the job right is ok by me. personally i have no need for them.


Last edited by DDd on 28.08.2008, 01:37, edited 1 time in total.

Top
 Profile  
Reply with quote  
 Post subject: Re: Texture Compression
PostPosted: 27.08.2008, 19:49 
Offline

Joined: 22.11.2007, 17:05
Posts: 707
Location: Boston, MA
DDd wrote:
swiftcoder wrote:
Most OpenGL ES1.1 hardware doesn't have ST3C, although that may change as ES 2.0+ becomes common.
I have no particular need for it, since all my targets feature S3TC, so a software fallback is indifferent for my particular needs.
We have had a couple of requests for both ETC and PVRTC, so someone must need them.

Quote:
I have never used a Mac platform, nor developed for the iphone. I also was not aware that that they provided a PVRTC toolchain/wrapper, I hope it is possible to abstract the differences and use a unified set of functions to work with PVRTC textures.
it is possible to abstract, but it multiplies the code we have to maintain.

Quote:
Quote:
I am talking about building a Horde3D-specific (though the format would be open, of course) wrapper format, which can hold all types of compressed/uncompressed image data (uncompressed, RLE, JPEG, S3TC, PVRTC, ETC, etc.). There would be a tool that leveraged FreeImage (or similar) to convert/batch convert other image formats into the Horde format, which would move this whole issue out of the engine and into the content pipeline.

I don't think i understand what you mean. Can you provide more details on how this woul work? what are the benefits and issues?
We strip all texture loading code out of the engine, and replace it with a custom format, specifically designed to contain any type of compressed/uncompressed image data.

Then we provide a tool that will use FreeImage (or whatever) to convert standard image formats into our custom format.

Pros: simplifies texture handling in engine, allows support of *any* format with minimal work and no complexity on the engine side, simplifies art pipeline for different device targets
Cons: requires separate tool - can no longer drop an image in and it works, you now have to convert the image first.

_________________
Tristam MacDonald - [swiftcoding]


Top
 Profile  
Reply with quote  
 Post subject: Re: Texture Compression
PostPosted: 27.08.2008, 20:14 
Offline

Joined: 26.03.2008, 02:58
Posts: 160
swiftcoder wrote:
We have had a couple of requests for both ETC and PVRTC, so someone must need them.

Personally I need support for the formats i don't need the software fallback.

Quote:
Quote:
I have never used a Mac platform, nor developed for the iphone. I also was not aware that that they provided a PVRTC toolchain/wrapper, I hope it is possible to abstract the differences and use a unified set of functions to work with PVRTC textures.
it is possible to abstract, but it multiplies the code we have to maintain.

Does Apple change the wrapper allot? I know that PVR does not make many changes.

Quote:
We strip all texture loading code out of the engine, and replace it with a custom format, specifically designed to contain any type of compressed/uncompressed image data.

Then we provide a tool that will use FreeImage (or whatever) to convert standard image formats into our custom format.

Pros: simplifies texture handling in engine, allows support of *any* format with minimal work and no complexity on the engine side, simplifies art pipeline for different device targets
Cons: requires separate tool - can no longer drop an image in and it works, you now have to convert the image first.


Hum... i don't get how it would simplify the art pipeline, i would still need to have a file for each platform, one compressed with pvrtc and one with s3tc... Maybe i am not getting the concept. It looks like allot of work, and is a fundamental change to how the engine handles images, i kind of like to have this kind of stuff at a low level, and not abstracted... :|


Top
 Profile  
Reply with quote  
 Post subject: Re: Texture Compression
PostPosted: 28.08.2008, 15:02 
Offline

Joined: 22.11.2007, 17:05
Posts: 707
Location: Boston, MA
DDd wrote:
Hum... i don't get how it would simplify the art pipeline, i would still need to have a file for each platform, one compressed with pvrtc and one with s3tc...
The idea is to make the pipeline the same on each platform - right now you need completely different tools on each platform.

Mainly, it simplifies the enigne code itself. Just the code to handle DDS is a lot of added cruft, and by the time we add ETC, and 2 variants of the PVR wrapper, it is getting ridiculous.

The current scheme is also fundamentaly at odds to Horde's philosophy - we don't try to support every 3D model file under the sun (.obj, m.3ds, .x, ...), instead we provide a converter and a single format that exactly matches our needs.

_________________
Tristam MacDonald - [swiftcoding]


Top
 Profile  
Reply with quote  
 Post subject: Re: Texture Compression
PostPosted: 28.08.2008, 21:26 
Offline

Joined: 26.08.2008, 18:48
Posts: 120
I don't know the solution. But I think a simple cross-platform asset pipeline looks something like this:
(I know that many engines has plugins(material/exporter/...) for DCC apps but that is a lot of work)

Every data (model, texture) is stored in 'raw' format.
3d Model/animation are in collada :wink: /fbx/... textures in psd/png/hdr/...

For every resource/platform there is a description how the data should be converted. (eg. model vertex compression/ animation compression, texture - format, mipmap count/generation params). It can be implemented like a 'make' or something similar build program and can be made user friendly with some GUI.
There could be even more output version for the same textures (eg on PC for cards with/without 3dc texture compression).

So I don't think that the same asset would be good for every platform(not supported compressed format is a pain to handle). Especially if the loading times are important and for 'not so powerful' plaforms like ipod.

I like the idea of having a horde3d texture format, but without a viewer (plugin?) it would be a pain to use.
For a new format I think dds would be a good starting point. I think with new FOURCC's it could be extended to support the mentioned formats, but maybe this would't be the best option. It would be great if there were some other well established graphics card friendly image format...


Top
 Profile  
Reply with quote  
 Post subject: Re: Texture Compression
PostPosted: 16.09.2008, 19:18 
Offline

Joined: 26.08.2008, 18:48
Posts: 120
Offtopic :wink: : I have implemented a different resource loading mechanism (async loading is work in progress). And I'm again impressed by the design of the interface. In Ogre I have to create multiple listeners to modify the resource referencing and it seemed so hacky. Here I've added a preprocess step for the xmls and I think it is much cleaner (maybe a little slower but the cost will be only some ms for a whole map).

Then I was thinking about that I could easily add dds support. Simply convert the dds to ?tga? (or something) in the loader. But of course I couldn't upload precompressed textures and mipmaps. But if the engine had a binary texture format(this could be dds) that would support compressed formats/mipmaps/cubemap then a loader would convert data to that format. This way the current image loading lib could be moved from the engine to the utils. And it would be much easier to add new formats. This would also make the horde3d simpler.
In our project probably I would only include the dds support and drop the stbi library.

I've read the forum and saw that Marciano wrote about something similar.
What do you think?


Top
 Profile  
Reply with quote  
 Post subject: Re: Texture Compression
PostPosted: 16.09.2008, 20:15 
Offline

Joined: 18.05.2008, 17:47
Posts: 96
http://icculus.org/physfs/ I am using this for file loading


Top
 Profile  
Reply with quote  
 Post subject: Re: Texture Compression
PostPosted: 17.09.2008, 16:25 
Offline

Joined: 19.11.2007, 19:35
Posts: 218
Even if you support a custom format, it's still in best interests to still support JPEG, and TGA (or some other alpha supporting format) for the reasons of hastened development. Yes plugins can be written for graphics software, but converter dependent lines are crude, and automating can be a butt.


Top
 Profile  
Reply with quote  
 Post subject: Re: Texture Compression
PostPosted: 17.09.2008, 18:10 
Offline

Joined: 26.08.2008, 18:48
Posts: 120
I was just talking about an internal custom texture format. But If someone has a pipeline that has a converter for textures (like for meshes from dae) he/she could just use this format for storing data.

The Horde3d could support other formats(jpg,tga,dds) but image support could be in Horde3DUtils.dll instead of Horde3D.dll. If someone uses the Horde3DUtils's resource loading they wouldn't have to change anything.

In this way a new image format(pgf,jpeg200,...) could be added without modifying the core engine, which is a pain in an open source engine.

but it's just my 2 cents


Top
 Profile  
Reply with quote  
 Post subject: Re: Texture Compression
PostPosted: 17.09.2008, 18:20 
Offline

Joined: 19.11.2007, 19:35
Posts: 218
That would create a dependency on Horde3DUtils. A utility library should not be a requirement.


Top
 Profile  
Reply with quote  
 Post subject: Re: Texture Compression
PostPosted: 17.09.2008, 18:34 
Offline

Joined: 18.05.2008, 17:47
Posts: 96
I agree with attila. IMHO core engine doesn't need to know about different formats (also limiting i/o operations to the memory would be good)
But well that would make Core dependent on image upload functions. Smaller core & thicker shell VS bigger core & thinner shell. Probably just a choice of design


Top
 Profile  
Reply with quote  
 Post subject: Re: Texture Compression
PostPosted: 17.09.2008, 21:30 
Offline

Joined: 22.11.2007, 17:05
Posts: 707
Location: Boston, MA
kal wrote:
But well that would make Core dependent on image upload functions. Smaller core & thicker shell VS bigger core & thinner shell.
Not all applications need those functions, so not a dependency in general. For instance, I use procedural synthesis for all my textures, so all I need is an upload function.

_________________
Tristam MacDonald - [swiftcoding]


Top
 Profile  
Reply with quote  
 Post subject: Re: Texture Compression
PostPosted: 17.09.2008, 22:16 
Offline

Joined: 19.11.2007, 19:35
Posts: 218
Quote:
IMHO core engine doesn't need to know about different formats


The core engine has to submit that data therefore it must know a good deal about it, unless you'd propose to write a single proprietary image format, which would be a bad idea, topping or even competing with most existing formats is an unrealistic goal (they each have their pros and cons).

At the same time, a solution for wider format support should also consider the fact that solely using XML for other resources is silly, at some point optimal binary archives for release builds would be ideal, especially in circumstances where streaming is a necessity, the same approach used for handling image after image should at its core be suitable for dealing with similar issues elsewhere.


Top
 Profile  
Reply with quote  
 Post subject: Re: Texture Compression
PostPosted: 18.09.2008, 14:56 
Offline

Joined: 22.11.2007, 17:05
Posts: 707
Location: Boston, MA
AcidFaucet wrote:
Quote:
IMHO core engine doesn't need to know about different formats
The core engine has to submit that data therefore it must know a good deal about it, unless you'd propose to write a single proprietary image format, which would be a bad idea, topping or even competing with most existing formats is an unrealistic goal (they each have their pros and cons).
We only need a very restricted set of formats - compressed/uncompressed, 1-4 channels, with/without mipmaps. That would be very easy to package into a custom container format, perhaps easier than abusing DDS to support other forms of compression.

Quote:
At the same time, a solution for wider format support should also consider the fact that solely using XML for other resources is silly, at some point optimal binary archives for release builds would be ideal, especially in circumstances where streaming is a necessity, the same approach used for handling image after image should at its core be suitable for dealing with similar issues elsewhere.
Agreed. To my mind, the core engine should only know how to load binary data from memory, and the current XML resource manager should be factored out into a separate (optional) layer.

_________________
Tristam MacDonald - [swiftcoding]


Top
 Profile  
Reply with quote  
 Post subject: Re: Texture Compression
PostPosted: 18.09.2008, 19:23 
Offline

Joined: 26.08.2008, 18:48
Posts: 120
Quote:
We only need a very restricted set of formats - compressed/uncompressed, 1-4 channels, with/without mipmaps. That would be very easy to package into a custom container format, perhaps easier than abusing DDS to support other forms of compression.


I totally agree with you. It should support cube texture( and maybe 3d texture) too. DDS uses format (Eg. D3DFMT_A8R8B8B8, D3DFMT_R16F, D3DFMT_L8, D3DFMT_DXT1) for describing the compressed/uncompressed and 1-4 channels part, something similar could be used in horde3d's format with additional support for PowerVR and other formats.

Quote:
Agreed. To my mind, the core engine should only know how to load binary data from memory, and the current XML resource manager should be factored out into a separate (optional) layer.


This could be great also, but for me it is not high priority. The data heavy resources(geometry, animation, texture) are in binary format. I know commercial engines that uses 'text' for similar data, because of flexibility and ease of development/debugging. But for loading large maps this could be a bottleneck...


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 69 posts ]  Go to page Previous  1, 2, 3, 4, 5  Next

All times are UTC + 1 hour


Who is online

Users browsing this forum: No registered users and 17 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group