Some gotchyas in porting from Visual C++ to g++

December 3rd, 2009 by Michael Simms (CEO and head of Development)

Todays technical article is going to go into a couple of issues that we have run across porting from Visual C++ to g++.

I will say right now that I am an application developer, I don’t know the fine details of how the inner workings of the compilers work, so some of this is merely an educated guess as to WHY the problem happens, but the effect and solution is correct.

The first issue for today is a fairly rare situation where g++, upon hitting a certain piece of code that builds fine under Visual C++, gives the helpful error message:

some/source/file.cpp:172: sorry, unimplemented: called from here

Now, I don’t know about you, but I find that error message singularly unhelpful. It took some time to run this one down the first time. It seems  that g++ and VC++ perform some symbol resolution in different passes to each other. Because what this message is saying is that ‘at line 172, you have tried to use __forceinline, or something you reference has a member function that has __forceinline in it, but I do not know what the thing is, so I cannot inline it’.

EDIT: g++ does not have the __forceinline keyword. There is an equivalent which does the same job however

#define __forceinline inline __attribute__((always_inline))

As VC++ has no problem with the same calls, the only real answer must be a compiler difference that allows VC++ to get away with something that g++ doesn’t. The simple solution is to just change __forceinline to inline and let the compiler inline as it sees fit. Otherwise, you will need to hunt down the exact problem and resolve the order of events for g++

The second issue I thought I would raise today is one that actually happens quite commonly on porting from Windows to Linux. This is the case of bad memory management.

On Windows, the way memory is allocated allows for a certain sloppiness in code. While HIGHLY unrecommended, this is the sort of thing that happens in Windows code all the time. The following is a highly simplified example of the problem.

char *str;
str=(char *)malloc(6)
strcpy(str,"hello");
free(str)
switch (str[0])
{
  ...
}

Now, as you can see here, the application uses the variable str right after it has been free()’d. On Windows it seems that this kind of thing can be gotten away with. The memory manager on Windows is highly unlikely to either assign this memory address somewhere else, or to mark it as unavailable. On Linux however, you will often find that this kind of code leads immediately to a segmentation fault.

The example above is highly simplified, but illustrates what some of the more complex segmentation faults we have seen boil down to.  If you see an error like this, which works on Windows and not on Linux, check your order of accessing. I know it sounds obvious, but it happens so often in commercial code that I feel it is worth stressing – free memory only when you have REALLY finished with it.

  • Share/Bookmark

Playing well with distros

November 24th, 2009 by Eskild Hustvedt (Community Manager and Junior Developer)

We often get a question similar to “why don’t you create native packages?”. I’m going to make an attempt at answerring that.

Current Linux distros primarily use either RPM or DEB (and a load of other less common ones that are only used by a distro or two). Most deb distros are somewhat compatible, as most of those are in one way or another based upon Debian. However, on the RPM side we’ve even got two completely different development trees of rpm itself, and a load of distros that are not compatible with each other. Last I checked (feel free to correct me here), most RPM distros let you install a 32bit package on a 64bit system, but last I tried I couldn’t do the same on a deb system. So now we’re up to three packages. One 32bit RPM, one 32bit DEB and one 64bit DEB. But now we’re assuming that all people have one of those two, but the fact is that they don’t (yes I know RPM is part of LSB, that doesn’t really guarantee that it is always present, nor properly set up). So we’re going to need another one anyway. We could go with a tarball, which at least gentoo and slackware will be used to, and possibly others, but for the others, well, we’ll either have to provide a lengthy technical README, or an installer. So, that’s five.

Now, consider that many of our games are several gigabytes, it is completely impossible for us to package all of them on the DVD. As far as I know, neither RPM nor DEB can have their payload as a separate and compatible file. Things could be copied in post-install hooks, but then we’re just about back to square one, as we’re pretty much bypassing the package manager anyway. As the installer could be made to use the tarball, we’ll need four full-size packages, and all of this is assuming that the package formats will stay compatible.

So to sum it up, not only would it be a lot of work to test and document it all, we’d still have to provide the packages we’re providing now to keep it accessible to everyone, but it would also take roughly four times the space, and I for one would not pay extra for a game to have four install DVDs containing the same game, just in several different installation formats, when one would suffice (yes yes, I know it would provide you with backups, but with the new copy protection system we have added you get free downloads of your game anyway, so that’s not a valid argument:).

If you have any input, suggestions or questions for me, feel free to ask them here in the comments, on IRC (Zero_Dogg in #lgp on irc.freenode.net), via identi.ca/twitter or via e-mail (to eskild at the domain linuxgamepublishing dot com).

  • Share/Bookmark

argv and argc, and just how to get them

October 12th, 2009 by Michael Simms (CEO and head of Development)

argv and argc. They are vital parts of many programs. For the uninitiated, in many programming languages, they are the variables that hold the values typed into the commandline by the user. This article is about a particular problem we ran into trying to use them in a way that wasn’t really forseen. It really only applies to C and C++

Now most C/C++ programmers are probably thinking ‘what is so hard, you get them when you start main, right’…

int main(int argc,char **argv,char **envp)

That’s true, and in virtually all cases, you can store these values, either in a class, a global variable, a function, whatever your programming style uses. And from then on, you can simply access them from anywhere in the program. Its really simple, and something that most programmers know how to do in their sleep.

But what do you do before main happens? Or what about if you don’t have a main()

I know, you always have a main and nothing runs before it.

But that is very not true. Lets take two examples

Firstly, in C++

class gamedata
{
public:
  gamedata()
  {
    commandline=get_argv();
  }
  char **commandline;
}

static gamedata datastore;

int main(int argc,char **argv)
{
   set_argv(argv);
   ...
}

To start, this looks OK. Sure, Ive missed out a whole bunch of things, assumed what the user functions set_argv and get_argv do. But it looks fine. Until you look again. That static class instantiation will run its constructor before main runs. How on earth can you POSSIBLY get argv at this stage??

Lets look at an example in C now

If you are creating a shared object, you will very often need to initialise it to ensure that its state is known for the first call into the objects functions.

static char **commandline;

void __attribute__ ((constructor)) localinit()
{
  commandline=get_argv();
}

Again, you try this, and you get a big nothing. This function is called before main runs, before you ever have access to the values.

So, what is the answer?

The answer is, unfortunately, ugly. There is, in Linux, no good way of doing this. There are two perfectly good symbols inside libc which do the job perfectly, __libc_argv and __libc_argc, which are defined way before the program gets to any user-created code. Unfortunately they are declared private and you as a user are not permitted to see them. So, another way is needed.

We came up with two ways to make it work. Neither are portable, and one of them, while it works just fine, does make me go ewww. I’ll leave it to your imagination which one makes me go ewww the most.

char **get_argv()
{
  static char **preset_argv=NULL;

  if (!preset_argv)
  {
    FILE *fp;
    fp=fopen("/proc/self/cmdline","r");
    if (fp)
    {
      //Your implimentation to take the commandline as typed from this
      //file and turn it into argv. Its fairly basic
    }
  }
  return preset_argv;
}
char **get_argv()
{
  static char **preset_argv=NULL;

  if (!preset_argv)
  {
    extern char **environ;
    char ***argvp;
    void *p;
    asm ("mov %%esp, %0" : "=r" (p));
    argvp = p;
    while (*argvp != environ)
      argvp++;
    argvp--;
    preset_argv = *argvp;
  }
  return preset_argv;
}

So, a little explaination

The first example relies on /proc, the part of the filesystem that you can get all sorts of interesting information from. /proc/self/cmdline is always an exact duplicate of the command typed to execute the application, or the exact value passed in from the menu option you clicked to get it working. I haven’t bothered to create the bit of code to separate out the commandline parts onto their components. Partly because that code is fairly straightforwards, and partly because it is quite long and dull (remember it isn’t just a case of separating by spaces, you have to take into account things grouped in quotes, and other fun stuff). This is not portable beyond Linux, and people keep telling me that not all Linux distros have /proc either.

The second example requires a tiny bit of assembler knowledge. It is portable across most unix flavours, but I expect trying it outside of unix will cause you much pain. It relies on the fact that a unix standard is to push the argc and argv values right onto the top of the stack when a program starts. The example reads from the top of the stack until it finds the environ value (which is globally available at all times), and then reads back one value to get the pointer to argv. This way has the advantage that the values are correctly parsed for quotes and the like already. It makes the assumption that environ is the next thing on the stack after argv. I have seen many reports claiming this is always true, but I cannot find the location of an authoritative piece of documentation saying that it is specified true and will not change in future.

It isn’t often that you will need to do this. Most of the time, argv and argc are perfectly usable in the way you will probably have been using them for years. Even the examples above can be ‘worked around’ using initialisers called immediately after main() starts. But if one day, you come up against a problem where you need your argv where you usually don’t have access, I hope you find this post useful.

  • Share/Bookmark

Give a name to the new addition to LGP

October 5th, 2009 by Michael Simms (CEO and head of Development)

Just under two weeks ago, the LGP office had a visitor.

kitten

It seems that she has adopted us, as she spends pretty much all of her time in the office, and is a constant disruption to work. At every opportunity she loves to sit on keyboards – especially when typing is going on, and she is very insistent when she wants some attention, as you can see from the picture above when she decided to sit on my keyboard and demand my time.

I decided we weren’t going to name her until she had stuck around for a couple of weeks, afterall, she could have just wandered off again. But it does seem she is here to stay.

So, we have a few ideas for names, but we can’t quite decide which, if any, of them are right. So, I thought, lets get a bit of community involvement, and set up a name poll.

So, add your votes below, and if you have suggestions for a better name that isn’t on the list, add a comment, an if we like it, we’ll add it to the poll! You can choose multiple answers if you like.

EDIT Oct 8th: Well, the poll has been going for just under 4 days now, and you can’t leave a cat without a name for too long! We will now be taking the top three, Qwerty, Switch, and CatMonster, and over the next week we’ll see which one seems to fit her best. The one that works best, we’ll name her, and update this post with the final result! Thanks everyone for your contribution, and I’m sorry I never got around to adding some of the new suggestions to the poll!

EDIT Oct 16th: After a weeks trialling, one name is clearly a winner based on her personality and attitude and what comes naturally when talking to her. CatMonster! It is great as an all-purpose name that can have its start changed, so when she meows a lot she is a NoiseMonster, when she’s asleep she can sometimes be the SnoozeMonster. And when she sneezed a while ago she was the SnotMonster. All in all, CatMonster wins!

So, what should we call the LGP cat?

View Results

Loading ... Loading ...

A bit of explanation as to some of these names…

  • CatMonster: A bit like KateMonster from Avenue Q
  • Ceefer: Ceefer Cat. Say it aloud, you’ll get it.
  • Natasha: The whole Boris and Natasha thing, and she likes eating spiders (Boris the spider). Its a bit convoluted, but made the list
  • Piper: She’s a mouser. So from the Pied Piper. Also has a nice link in to the ‘Charmed’ TV show
  • Quark: She loves having her ears scratched, so a Ferengi name was needed!
  • Qwerty: She is a keyboard cat. Obvious really
  • Splodge: I mentioned the new addition to my mother on the phone a week or so ago, and she insisted this be the new name.
  • Spot: Another Star Trek reference. Data’s cat.
  • Switch: Her favourite place to sit is on the office gigabit switch. Its warm and by the window. Also a Matrix tie-in
  • UsrBin: /usr/bin/cat. OK so cat is usually in /bin but you can’t call a cat just Bin
cat11 cat2
  • Share/Bookmark

Microblogging

October 2nd, 2009 by Eskild Hustvedt (Community Manager and Junior Developer)

After some internal discussion, we have decided to try something new, namely microblogging. Initially we are just testing it to see how much it will be used, but we will make it permanent if it proves successful. It is meant as a simple means to send quick messages to LGP and get quick information about releases, announcements and patches.

We will also be running some competitions where you will get a chance to win a download of an LGP game, as well as some small ones now and then where we will hand out free game rentals.

As the free and open source software loving company we are, we use identi.ca which runs on the AGPLd status.net microblogging platform. But for those that prefer twitter, our account is mirrored there as well.

So follow us, @LGP on identi.ca or @L_G_P on twitter. See you there.

If you have any input, suggestions or questions for me, feel free to ask them here in the comments, on IRC (Zero_Dogg in #lgp on irc.freenode.net), via identi.ca/twitter or via e-mail (to eskild at the domain linuxgamepublishing dot com).

  • Share/Bookmark

Intermittant downtime over the last two days

September 26th, 2009 by Michael Simms (CEO and head of Development)

Once more LGP has been hit by outages affecting companies we work with. This time, the hosting company, RapidSwitch, who own the datacenter where we host our webservers.

We would like to apologise to all customers who had their downloads affected, or who had their ability to play rental games affected. We are immediately extending the rental period of all rental games that are active now, or would have been active during the downtime, by four days to make up for the intermittant downtime.This affects rentals purchased from any LGP reseller.

We are pleased to report that players with non-rental versions of the game, both download and boxed copy, experienced no lockout. As designed, the LGP protection system coped perfectly with the outage.

The outage also affected PenguinPlay, meaning that ingame multiplayer functionality via PenguinPlay would have been unavailable.

This will also have affected our resellers who sell downloadable copies of games, and request download keys dynamically. If any of your customers received blank keys, please contact us at support@linuxgamepublishing.com, and we will fix the problem.

Finally, on a lighter note, we would like to briefly laugh at RapidSwitch. Who as compensation for the downtime, have offered to upgrade our servers for free. Beside the fact that our servers are fully loaded and not upgradeable any further, I am still waiting for their reply to my question of why on earth did they think that appropriate compensation for extended downtime is – effectively – more downtime. I’ll summarise their response in a comment as and when it is received.

  • Share/Bookmark

Shadowgrounds: A surprise release

September 16th, 2009 by Michael Simms (CEO and head of Development)

Well, a lot of you have been waiting for Shadowgrounds Survivor. For all of you waiting, we have a nice surprise for you.

Late in the porting of Shadowgrounds Survivor, it became fairly clear that it would be little work to also bring the original Shadowgrounds game to Linux. We discussed it with Igios, who were doing all the hard work, and we decided that we would also release Shashadowgroundsdowgrounds.

But, being the sneaky Linux Game Publishing company that we are, we decided to keep it secret. I guess, I just wanted to give everyone a nice surprise, and get a new game out that none of you were expecting. As it has the same game core as Survivor, it didn’t need the extensive beta test, making it one of the few times we would be able to release a game without a beta to let you all know it was on the way.

Later this week we will be releasing Shadowgrounds Survivor, and so we felt that the best thing to do for Shadowgrounds is to make it a budget title on release. We have decided to set the retail price at just £10, partly because it was easier to port and so the price being lower is fair. Also, and I’ll be honest here, I want lots of people to buy it, get hooked on the gameplay, and just feel that urge to buy the higher priced Survivor. I’m not going to claim to be completely altruistic, we ARE a business afterall {:-)

Now, one thing that will disappoint some players is that neither Shadowgrounds, nor Survivor, come with the co-op multiplayer option that Windows has. This is because of a conflict in the game design, and the design of SDL. It is actually technically impossible to provide the multiplayer (which on Windows requires two keyboards and two mice) with the architecture as it stands, and by the time we realised this, it would have required a complete rewrite of the whole game core to get this working.

However, good news, is that the next version of SDL will contain all we need to get co-op into the game, and Igios has committed to releasing a patch to provide this functionality as soon as SDL can do it.

Shadowgrounds has taken a lot of dedication from the guys at Igios, so I’m counting on the community to do their bit, and buy the game, give them some thanks for the long hours of work they have put in to bring this game to you. You can buy it boxed, or it will be available to download, either for purchase or rental.

  • Share/Bookmark

LGP History pt 3: The long haul

August 25th, 2009 by Michael Simms (CEO and head of Development)

With Majesy out of the door, and releases complete for Mindrover and Candy Cruncher, we began what turned out to be the long slog.

At that point, fresh from our completion of Majesty, we were convinced world domination was just around the corner and we would all have our ferraris quite soon. We had a number of copies of Majesty printed (no, I’m not saying how many), and expected to have to get a reprint within 2 months, judging by the number of applications for the beta test, and the enthusiasm shown by people.

Over the next few months, reality, and a certain level of depression set in. We didn’t need to reprint Majesty. In fact in the first 3 months we didn’t even sell a quarter of the copies we had produced, and after the first few months, sales begin to slow down, so it didn’t look likely that we would suddenly see a huge rush of orders. The optimism pretty much evaporated.

So, realising it wouldn’t all be parties and glamour and free money by the bucketload, we settled down to some real work. We had more projects going on, with NingPo, Soul Ride and HDB, and we had new ideas. One of the new ideas that we went with was to open a physical shop.

Not a lot of people know we did that. We opened a shop in Nottingham, that was selling games. I admit we didn’t just sell Linux games, we sold all kinds of games, but the idea was to get a bit of local publicity for Linux games, while raising money using sales of other games, to fund the development of Linux games. It also didn’t feel horrible to see a shop where Linux games were on the same shelves as Windows games.

To be honest it didn’t do that great. The shop was small, and the people we rented the premises from made vastly inflated claims of how busy the mall we rented in was. Oh they didn’t lie, but they neglected to mention that the figures for mall visitors also included people walking in one side and straight out of the other side, as it was the only direct route from the city centre to the train station… However we sold a few Linux games, and increased company turnover (which always looks good on the books). In the end though, it was fairly obvious it wasn’t going to be the cash cow we had hoped for, in fact it was more a cash sink. So we closed it down after a few months.

Shortly after, I had been invited to talk at LinuxWorld Expo in San Francisco, and on my return to the UK, found we had had a disaster. Overnight on the day I returned, our stock room (also our server room) flooded. Our premesis is on a hill, and a severe rainstorm caused a building uphill from ours to flood, and the flooding cascaded downhill until reaching our stockroom, which seemed to be waterproof on the downhill side and not on the uphill side. End result was that the water pooled there, causing thousands of pounds of damage, and days of downtime. 6 inches of water took several days and a quite elaborate pumping system to remove, and obviously while half of the electrical system was underwater, we couldn’t fire up the servers!

Luckily, the damage wasn’t catastrophic. while hundreds of games had been destroyed, and computers had been submerged, no game was completely wiped out of stock, so we had no mad rush to reprint, and no orders were delayed. Backups ensured no data had been lost, but it was a bit of a scare! We spent a good few days raising everything in the room up by 6 inches, so that if it ever happened again we wouldn’t have a big problem, and we even installed a pumping system just in case!

Following one disaster, it is only appropriate that we mention another company disaster at this point. Disciples, a great game, but unfortunately LGP’s DNF. It was around this point that Disciples caused the first of many resignations from LGP. Mike Phillips left the company after one too many late nights trying to beat the game into submission. It isn’t that there is anything wrong with Disciples, it is simply that you need a developer with just the right development style to be able to port it, and they have proven hard to find over the years. And so over those years, Disciples has been part-ported several times, leading to belief it will never be released. I can say for sure it will be released, I just cannot, still to this day, say for sure when!

Despite the downsides, the flooding, the game that refused to be ported, and the staff that left, we had successes. Postal 2 became the fastest selling game in its first month, and when we looked back at the accounts we found that we had, in many ways done it right. The company had proven itself sustainable. Where Loki had come in in a blaze of glory and burned out just as quickly, we had been around for as long as Loki, and we were still here. We were not as high profile, the games we ported were not the ones you see advertised on TV, but they were all undoubtedly as good as the games produced by Loki. Just because a game is high profile doesn’t make it good, and just because a game is less well known doesn’t make it bad.

So, we were stable, we were ready for the future, and we now had to make some decisions. How could we grow. What could we do to drag Linux gaming into the mainstream. And how could we do it without the blaze of glory ending…

  • Share/Bookmark

Handling misbehaving libraries in binary products

August 18th, 2009 by Michael Simms (CEO and head of Development)

It is a bit of an arcane artform, making a game that runs on all Linux versions. We do quite well at LGP but there is always room for improvement.

Recently, we came across an issue with PulseAudio (I will rant about Pulse in some other post). It became almost impossible, due to the architecture of alsa, and the combination of our build environment, and our higher level sound libraries, to make sound work – pretty much at all, in Linux games. Those of you with older games will know that for a while they all stopped having sound. Some of them we still haven’t released patches for yet (they will be forthcoming).

Now, one of the big problems was that sound libraries (such as OpenAL or SDL) go through their own motions of opening sound. We have no way of directly controlling what they do or how they do it. This is a common feature with many libraries, they will load their own dependencies in a way we cannot control.

The biggest problem is that OpenAL and SDL try to dlopen libasound, and on some machines, libasound doesn’t work with our binaries. On others, it can actually crash the whole game due to incompatibilities. This is a common issue when dealing with unknown system configurations when sending out a binary-only product into the world. You never know just what peoples systems will have installed, and just how they may confuse the situation. But you cannot just say ‘well, bad luck, it crashes on your machine’, people will be understandably upset.

We came up with a rather horrible system to make this work, and one that I hope may be of use to those of you who come here for our technical insights. This system is deliberately designed to work with libraries that have the potential to actually crash a program. As a simple example that many here will follow, I am using SDL’s audio system as the target, and using it to test whether ALSA works or crashes the program.

  static int system_asound_ok=-1;

  if (system_asound_ok==-1)
    {
      forkval=fork();
      switch (forkval)
        {
        case -1:
          //This is an error, we couldnt fork() so we just say no to
          //system openal working
          system_asound_ok=0;
        case 0:
          //This is the child, try and run the open, and then close and exit
          {
            //Your code here to remove whichever signal handlers you have set up
          }
          fail=SDL_InitSubSystem(SDL_INIT_AUDIO);
          if (!fail)
            SDL_QuitSubSystem(SDL_INIT_AUDIO);
          //MUST use _exit here not exit
          _exit(0);
          break;
        default:
          //This is the parent, wait for the child to exit somehow
          waitpid(forkval,&fail,0);

          //The child has now exited somehow, check how
          if (WIFEXITED(fail))
            {
              //The child exited ok, we can try and use the system library
              system_asound_ok=1;
            }
          else
            {
              //The child probably crashed, dont try and use the system library
              system_asound_ok=0;
            }
        }
    }

At this point you know whether the system asound (ALSA) library is safe to use.

This is a truly truly horrible system, when you actually have to allow an application to crash, you know something is wrong, but sometimes there is no option.

As a technical note, note the use of _exit() instead of exit(). This ensures that when the program ends, no atexit() functions are called. Using exit() could mean things like SDL’s cleanup being called, or who knows what else!

Signal handlers should be removed at the point in the code indicated. They interfere with the proper signaled exit of the program and will prevent WIFEXITED from detecting a crash properly. Bear in mind the signal removing only happens in the child process and will not affect signals in the parent.

So, what can you do knowing that the system works or doesn’t work?

Well, knowing that it works, you can simply run the initialisation as it stands and enjoy the working library.

If you know it fails, you can either skip this system altogether (in this case run without sound) or you can switch to another library. If you setenv an LD_LIBRARY_PATH this will influence the path that dlopen looks for libraries. LD_LIBRARY_PATH is searched before any other directories, and so will override the default location of the library. This means, in this instance, that SDL_InitSubSystem will dlopen the libasound.so in the location you have set the LD_LIBRARY_PATH to.

I hope this will relieve some headaches for some developers out there. It is always difficult making any application when you do not know what the target system will have installed on it. Of course, we can always say ’sorry, you need a specific version of this or that library’ but when you have just bought a game, that is the last thing you want to hear, and so – we have to jump through some hoops to stop that happening.

  • Share/Bookmark

The toll of the recession on Linux gaming

August 13th, 2009 by Michael Simms (CEO and head of Development)

No, don’t worry, we aren’t going anywhere.

It has just been announced that yet another Linux-friendly company, Grin, has gone under. Add this to the big name of Ascaron a couple of months ago, and it is a sad time for those that had some faith in Linux gaming.

I just wanted to take a moment to send a thought out to those companies, and thank them for the time and effort they went to, working with us on bringing their great games (Ballistics, Bandits, Sacred) to Linux, and to wish the employees of these companies good luck in finding new places either within or outside the gaming industry.

  • Share/Bookmark