Horde3D

Would it be an idea to move to JSON for the data files? It's more lightweight, human readable, widely supported and easy to convert from XML. The shaders have already moved away from XML. An example:



or even more tight:

JSON looks interesting but I see two problems: it does not seem to support comments and it would be even more difficult to reuse it as simple scripting language (required for pipelines). Although Horde has a lot of XML, I am not a huge fan of XML per se. It is just used because it is very practical and everyone knows it. But I would be open to replace it by another (also custom) syntax if that would bring a real advantage.

AFAIK json comments are just /* comment */

XML isn't very appropriate for a scripting language either, because strictly speaking, the order that elements are stored in a file is supposed to be irrelevant (XML stores sets of data, not sequences of data).

What about in the long run if all of Horde's XML functionality was moved into a "file to class translation" module (responsible for converting data from disc into internal classes/objects). Then perhaps the file formats (XML/JSON/binary) could be changed with Horde extensions?

This is a little offtopic : Is there any way to compile XML files to hide the Shaders, Scenes and ... from intruders?

Yes, Horde's resource loading code doesn't actually implement file-loading (it only loads resources from memory, not from files).

Horde3D utils is the part that actually loads files - so you need to replace this utility file loading code to use a "virtual file system" (which could be encrypted to stop people from looking at the files).
If you're interested in this, I would suggest using PhysFS as a starting point (it implements a .ZIP virtual file system).

Optimally I would like s-expressions, but that might scare people...

If you are going to move away from XML, I would suggest skipping the intermediate stage of JSON, and moving all the way to YAML.

_________________
Tristam MacDonald - [swiftcoding]

I also though that at first but it does not seem to be in the spec, at least that's what I found by some googling.


Yes I know, but it sort of works


I think we should stick to some well-engineered but fixed file formats in Horde. Although I like flexibility, too much of it can hurt. It would become nearly impossible to write tools if everyone uses his own formats (e.g. different material syntaxes using XML or JSON). If someone really requires a different format, it is easy to change the loading code but we shouldn't encourage that. Much of an open source engine like Horde is the community projects and tools around it, so I think we should not do anything that complicates this development.


I would say it is ok to stick with XML for now. Our XML files are not that big and I think they are readable enough. The first example that jimbo brough is slightly unfair since the XML did not contain any indentation. The big advantage is that everyone knows XML and has a parser for it (when creating small tools).
Concerning performance: I started writing my own extremely lightweight XML parser some time ago. It is non-extractive (just stores references instead of copying data) and uses node pools to minimize the number of memory allocations. It should be really fast so I don't expect any performance problems with XML files.

There are quite a number of these already out there - you might want to have a look at rapidxml and fastxml, before committing to maintaining a custom solution

_________________
Tristam MacDonald - [swiftcoding]

We are using http://code.google.com/p/pugixml/ in our title. In our test this was the fastest and required the less memory, we haven't checked fastxml though.

I think XML is the best solution, the worst I can imagine is a custom format. There is no need to reinvent the wheel... (Simplicity rulez. )

Thanks for the links! Especially rapidxml looks very interesting. BTW, my own parser is already working (was just a longer evening of work) but it only takes a quite limited subset of XML so I would not be against using something more proven (like rapidxml). But I guess the XML parsing is not a bottleneck for anyone at the moment so I would consider it as one of the later optimization steps.


Pugixml looks also interesting, although according to this (biased) benchmark, rapidxml seems to be a bit faster. But the difference is really marginal.


The only real motivation for a custom format would be that we could create one that can embed binary data. So model and animation files could also become more readable. But anyway, that's not essential and XML has the other advantages of popularity and broad support so we will probably keep it.

I am not disagreeing with your overall point, but XML is not really designed for simplicity - if that were the only goal, JSON or YAML would be better, being specifically designed for simplicity, and also widely used (FaceBook, Google, etc.).

As is, we don't really derive any benefit from using XML, since a) nobody wants to hand-craft these files, b) our format isn't widely supported by 3rd party tools, and c) we aren't using Schemas, DTDs, etc. (the one area where XML truly excels).

_________________
Tristam MacDonald - [swiftcoding]

That's a good point..

It's pretty obvious though that everyone has a different preference for file format though!

Would you be interested in non-human readable formats (e.g. binary material files, etc)?

If there were two standard file formats (human/XML and binary), which were interchangeable and could be converted from one to the other, then people could use YAML etc in their own work-flow, as long as they write their own YAML<-> binary converter.

Then if someone needs to view a YAML file in XML form then a binary interchange format makes it possible: YAML -> binary -> XML.

Binary files would be nice on the face of it, but since by production time you tend to have all these files in a compressed, packed-archive format, it doesn't really matter. Also a binary format means we need to deal with endian issues, etc.

_________________
Tristam MacDonald - [swiftcoding]

I don't think endianness is the biggest issue with binary files. It requires some care during the design phase (either indicating the endianness used upon storage in the file or using a fixed endian and converting upon read/write on machines that need it) but is relatively easy to deal with.

In my experience storage of recursive data structures and backward/forward compat are much bigger concerns than endianness (none likes to maintain several loaders/writers just to keep support for old file formats).