Comments on: The trouble with storing binary data structures http://blog.linuxgamepublishing.com/2009/01/29/the-trouble-with-storing-binary-data-structures/ Commercial gaming for Linux Thu, 26 Aug 2010 09:00:14 +0100 http://wordpress.org/?v=2.8.4 hourly 1 By: Mike Phillips http://blog.linuxgamepublishing.com/2009/01/29/the-trouble-with-storing-binary-data-structures/comment-page-1/#comment-21 Mike Phillips Thu, 29 Jan 2009 19:13:01 +0000 http://blog.linuxgamepublishing.com/?p=20#comment-21 Actually, that problem was the very first major problem in the Majesty port "Way Back When" -- a packing/alignment issue was causing a crash because the data was being used as an index. (...and I could tell you stories about some similar fun at Loki :-) ) Glad to see the torch was picked up and the binary will be updated :-) Actually, that problem was the very first major problem in the Majesty port “Way Back When” — a packing/alignment issue was causing a crash because the data was being used as an index. (…and I could tell you stories about some similar fun at Loki :-) )

Glad to see the torch was picked up and the binary will be updated :-)

]]> By: Harald Nikolisin http://blog.linuxgamepublishing.com/2009/01/29/the-trouble-with-storing-binary-data-structures/comment-page-1/#comment-20 Harald Nikolisin Thu, 29 Jan 2009 18:44:07 +0000 http://blog.linuxgamepublishing.com/?p=20#comment-20 I do not begrudge the LGP folks for porting the quick&dirty stuff, which rules the windows world. @doomwarriorx: I don't think that there much space to change things without breaking compatibility with existing windows version (a saved game on linux should be playable on windows).. @michael: and a mac saved game on linux ;-) besides this is very unlikely that a commercial game will have a more exotic plattform than Linux/x86 I do not begrudge the LGP folks for porting the quick&dirty stuff, which rules the windows world.

@doomwarriorx: I don’t think that there much space to change things without breaking compatibility with existing windows version (a saved game on linux should be playable on windows)..
@michael: and a mac saved game on linux ;-)

besides this is very unlikely that a commercial game will have a more exotic plattform than Linux/x86

]]> By: michael http://blog.linuxgamepublishing.com/2009/01/29/the-trouble-with-storing-binary-data-structures/comment-page-1/#comment-19 michael Thu, 29 Jan 2009 17:34:56 +0000 http://blog.linuxgamepublishing.com/?p=20#comment-19 We do know this issue, as we said at the start, this is something we solved ages ago, but we thought it would make an interesting read for people. we bumped into it again yesterday when we had some bad data reads, and that turned out to be the issue. I could't agree more about the idea of saving data as a raw dump of a struct, it is a terrible idea if you ever want portability, but when something has been written without portability originally, and you have to then port it, this is the kind of thing you have to live with. The solution offered is the only one that makes sense, in our case, as if packed data has been saved into a datafile which cannot be changed, well, the only other option would be to memcpy into each element of the array one at a time. This is fine, but slower, and the example was for use in porting where you really do not want to change the original code too much. And if, for example, you are reading in a data structure with 300 elements (they exist) you really do not want to do sit there for 3 hours writing 300 memcpy instructions! If you are copying to an architecture that does not handle packed data, then you are going to have to do that, but the idea of the article was to illustrate the problem, and how we solved it for the architecture we are porting to, rather than to be all things to all people. We do know this issue, as we said at the start, this is something we solved ages ago, but we thought it would make an interesting read for people. we bumped into it again yesterday when we had some bad data reads, and that turned out to be the issue.

I could’t agree more about the idea of saving data as a raw dump of a struct, it is a terrible idea if you ever want portability, but when something has been written without portability originally, and you have to then port it, this is the kind of thing you have to live with.

The solution offered is the only one that makes sense, in our case, as if packed data has been saved into a datafile which cannot be changed, well, the only other option would be to memcpy into each element of the array one at a time. This is fine, but slower, and the example was for use in porting where you really do not want to change the original code too much. And if, for example, you are reading in a data structure with 300 elements (they exist) you really do not want to do sit there for 3 hours writing 300 memcpy instructions! If you are copying to an architecture that does not handle packed data, then you are going to have to do that, but the idea of the article was to illustrate the problem, and how we solved it for the architecture we are porting to, rather than to be all things to all people.

]]> By: doomwarriorx http://blog.linuxgamepublishing.com/2009/01/29/the-trouble-with-storing-binary-data-structures/comment-page-1/#comment-18 doomwarriorx Thu, 29 Jan 2009 17:24:41 +0000 http://blog.linuxgamepublishing.com/?p=20#comment-18 hopefully this sounds not to harsh. How could you port several games without not knowing this issue? It is a common problem for e.g. opensource games which were closed source before. Your solution isn't good at all, because there are several cpu architectures which doesn't support "packing" of data. It's more a works for me, hopefully nobody other have to port my stuff to some exotic platform. So for platform independent communication it is a bad practice to memset into a structure and vice versa. hopefully this sounds not to harsh. How could you port several games without not knowing this issue? It is a common problem for e.g. opensource games which were closed source before.
Your solution isn’t good at all, because there are several cpu architectures which doesn’t support “packing” of data. It’s more a works for me, hopefully nobody other have to port my stuff to some exotic platform. So for platform independent communication it is a bad practice to memset into a structure and vice versa.

]]> By: Harald Nikolisin http://blog.linuxgamepublishing.com/2009/01/29/the-trouble-with-storing-binary-data-structures/comment-page-1/#comment-12 Harald Nikolisin Thu, 29 Jan 2009 13:42:09 +0000 http://blog.linuxgamepublishing.com/?p=20#comment-12 good article, I'm going to appreciate this blog, and now another use-case I have X2-The Threat on Linux (your product) and on Mac OS X (PPC - ported by VP). If I save a game on Linux, I can load it into the Mac OS X game and proceed. If I save the game on Mac OS X and load it into the Linux Version, X2 crashes. Can you provide a bug fix? to improve the alignmen ;-)) I can send you a saved gamed from Mac OS X...... good article, I’m going to appreciate this blog, and now another use-case

I have X2-The Threat on Linux (your product) and on Mac OS X (PPC – ported by VP).
If I save a game on Linux, I can load it into the Mac OS X game and proceed.
If I save the game on Mac OS X and load it into the Linux Version, X2 crashes. Can you provide a bug fix? to improve the alignmen ;-))
I can send you a saved gamed from Mac OS X……

]]> By: michael http://blog.linuxgamepublishing.com/2009/01/29/the-trouble-with-storing-binary-data-structures/comment-page-1/#comment-11 michael Thu, 29 Jan 2009 13:40:06 +0000 http://blog.linuxgamepublishing.com/?p=20#comment-11 Yep, it means we have a new majesty patch on the way, its been on the way for a while, but its closer now. Yep, it means we have a new majesty patch on the way, its been on the way for a while, but its closer now.

]]> By: basement cat http://blog.linuxgamepublishing.com/2009/01/29/the-trouble-with-storing-binary-data-structures/comment-page-1/#comment-9 basement cat Thu, 29 Jan 2009 11:42:47 +0000 http://blog.linuxgamepublishing.com/?p=20#comment-9 I believe the best way to handle this is by writing an abstraction layer that takes care of reading and writing file data, and does so in a way which is consistent across different platforms. This library seems to be a pretty good example of how the issue can be solved: http://www.leonerd.org.uk/code/libpack/ I believe the best way to handle this is by writing an abstraction layer that takes care of reading and writing file data, and does so in a way which is consistent across different platforms.

This library seems to be a pretty good example of how the issue can be solved: http://www.leonerd.org.uk/code/libpack/

]]> By: Eric http://blog.linuxgamepublishing.com/2009/01/29/the-trouble-with-storing-binary-data-structures/comment-page-1/#comment-8 Eric Thu, 29 Jan 2009 11:18:22 +0000 http://blog.linuxgamepublishing.com/?p=20#comment-8 Hi Michael, That's very interesting. It means also we will get a new patch for Majesty ? I love this game, I continue to play it from time to time. I hope to read more programming articles in your blog. Regards, Eric Hi Michael,

That’s very interesting. It means also we will get a new patch for Majesty ? I love this game, I continue to play it from time to time.

I hope to read more programming articles in your blog.

Regards,
Eric

]]>