I fucked up...
Character encodings are a real PITA.
Earlier today, I improved the MobMap parser backend that parses the uploaded data to gain some more speed in order to be better able to cope with the spike of data that's currently being uploaded. Those were pretty simple and straightforward improvements, I did a quick test if they worked and they worked perfectly. So I pushed the software onto the server and kicked it off.
Some hours later I noticed a lot of garbage in the german MobMap database. It turned out that the parser program was messing around with the german Umlauts, inserting questionmarks instead of the Umlauts all the time - and it was already doing that for hours! Half of the german MobMap database was pretty much fucked up by this.
Of course I didn't intend this behavior - it was the result of me developing my optimizations on a new Linux-based notebook, where I had to install Eclipse (the premier Java development environment) and check out the source code from my repository first because I hadn't developed anything on this machine yet. While doing that, I missed one important detail: I should have set the encoding setting for the parser source file (for some reason the Linux and Windows versions of Eclipse behave differently in that matter, so you sometimes have to manually set the encoding you want if you do work on both platforms). I forgot that, and Eclipse messed up the Umlauts in the file without me noticing it - the file did of course compile fine, and it ran without any problems. Until I noticed this growing pile of garbage in the database...
OH SHIT! I pulled the emergency brake, shut down the whole thing (including the web frontend - that's why the MobMap site was inaccessible for a few hours) and inspected the mess. It was pretty clear that cleaning this mess up would take way longer than recovering with a database backup and parsing the whole stuff again. So I took the backup route.
The backup is in place now, and everything does at least seem to work again. However, this backup didn't include any of the new already-parsed patch 2.4 content :-( so please don't wonder that the MobMap database that's online for download now does miss all that stuff. I'm working to get the parser up and running again (and this time without messing up the whole database!) and hope to be able to create the first MobMap database files with the patch 2.4 stuff soon.
Earlier today, I improved the MobMap parser backend that parses the uploaded data to gain some more speed in order to be better able to cope with the spike of data that's currently being uploaded. Those were pretty simple and straightforward improvements, I did a quick test if they worked and they worked perfectly. So I pushed the software onto the server and kicked it off.
Some hours later I noticed a lot of garbage in the german MobMap database. It turned out that the parser program was messing around with the german Umlauts, inserting questionmarks instead of the Umlauts all the time - and it was already doing that for hours! Half of the german MobMap database was pretty much fucked up by this.
Of course I didn't intend this behavior - it was the result of me developing my optimizations on a new Linux-based notebook, where I had to install Eclipse (the premier Java development environment) and check out the source code from my repository first because I hadn't developed anything on this machine yet. While doing that, I missed one important detail: I should have set the encoding setting for the parser source file (for some reason the Linux and Windows versions of Eclipse behave differently in that matter, so you sometimes have to manually set the encoding you want if you do work on both platforms). I forgot that, and Eclipse messed up the Umlauts in the file without me noticing it - the file did of course compile fine, and it ran without any problems. Until I noticed this growing pile of garbage in the database...
OH SHIT! I pulled the emergency brake, shut down the whole thing (including the web frontend - that's why the MobMap site was inaccessible for a few hours) and inspected the mess. It was pretty clear that cleaning this mess up would take way longer than recovering with a database backup and parsing the whole stuff again. So I took the backup route.
The backup is in place now, and everything does at least seem to work again. However, this backup didn't include any of the new already-parsed patch 2.4 content :-( so please don't wonder that the MobMap database that's online for download now does miss all that stuff. I'm working to get the parser up and running again (and this time without messing up the whole database!) and hope to be able to create the first MobMap database files with the patch 2.4 stuff soon.
