MobMap development blog

Okay, this ain't really MobMap-related, but generally related to WoW. However, it has just turned out way too good to keep it to myself, so I somehow abuse this blog now to show you this:



I got myself an iPhone 3G right at release day cause my old cell phone broke down and I wanted to replace it with a phone with good internet support, to browse the web and write e-mails and ssh onto my servers from anywhere I want. So even though I generally dislike Apple for its "we know better what you like than you do, so eat what we serve you or die" philosophy the iPhone 3G was a perfect fit, and the soon-to-be-released jailbreak would solve the problem that Apple thinks they'd be in charge of dictating what software I'd like to run on my device.

During the last weeks, I have found some time to perfect my own iPhone WoW Theme, which was inspired by a screen shot I once saw on wowinsider.com. The Theme consists of three independent style modifications:

- WoW Icons replacing the standard icons of all the default apps on the phone (and some additional apps I have placed on the first screen, like Cydia and biteSMS).

- a WotLK background wallpaper (taken from the current WotLK beta client). This modification also adds a slight shadow behind the icon labels and changes their font size, thereby making the labels more readable on the background image.

- a WotLK background video wallpaper, which is based on the WotLK beta client's login screen, too. This one is really cool, as it shows that dragon on the login screen flying by, landing, roaring and all the time there's snow falling down - well, you know the sight if you've already started the beta client at least once (however, there usually are UI components in front of that epic background - it was quite tricky to hide those for video recording, I basically had to modify every single resource file that creates those components, making textures fully transparent and replacing the fonts with a font that only contains "empty" characters, as just modifying the .xml or .lua files that form the login screen and thus hiding the whole screen at once was not possible due to these files being protected by a digital signature). The video wallpaper however uses like 10% of the iPhone's CPU power at all times, which is why you might want to choose the static wallpaper instead.

The whole package is designed to be used with WinterBoard, which can be installed via Cydia on any jailbroken iPhone. The contents of the theme package have to be copied to /private/var/stash/Themes on the iPhone in order to be available in WinterBoard (via SSH/SCP or any other means). The theme package can be downloaded here.

The package file contains a .psd file I used to create the WoW-like icon .png files from the raw WoW icons. You might find it useful if you want to create your own icons. I might also create some more icons for some more applications installed via AppStore or Cydia some time later, as I'd really like to have all the icons on the phone being replaced by WoW icons, not just those on the first screen. However, as any user has different applications installed, you'll probably have to create at least some icons on your own if you adhere to the same goal. If you need some more in-depth knowledge about iPhone theme customization, you might find this tutorial and this site from the creator of WinterBoard very useful - I learned everything about iPhone theming from those two resources.

A few hours ago, I released MobMapUpdater v1.80, which included a few bugfixes, mainly concerning stability issues during uploads on certain systems and that bug which prevented uploads from happening via HTTP proxy servers.

Unfortunately that new version introduced a new minor bug, popping up a balloon tip in the systray in automatic mode whenever WoW was being closed without MobMap having collected any new data while playing. This has been fixed in v1.81, which I have just put online for auto-update.

And a side-note concerning WotLK: I do now have a fairly functioning WotLK-ready version of the MobMap addon. It ain't perfect yet, there are still some things I need to add or fix, but it's perfectly in time and a final version will be available whenever WotLK hits the live servers (with probably a beta version some time earlier).

Good news - I finally got hold of a WotLK beta key and have started on updating MobMap for the next expansion. I expect to be able to release a WotLK-ready beta version of MobMap (presumably numbered "v2.20") some time soon, though I won't start to process data collected by the addon until WotLK has hit the shelves and the live servers, as data from beta servers tends to be faulty and error-prone way more often than data from the live servers.

I have just pushed a little backend bugfix onto the MobMap server that fixes a problem which has been there for many months. You might have noticed that the links in quest objective texts that MobMap presents you sometimes lead to the wrong NPCs - usually to some which have a very similar name as the ones you're searching, but nevertheless the link was crap. I know about some quests in the German database which had this problem, but there probably are similar problematic quests in the English database.

The reason for this was some kind of bad design in the component which actually does create those links for you to click - the so-called Quest Text Enhancer. That is essentially a fuzzy string matcher to which a list of all NPCs and a text is given. The QTE then searches the quest text for occurrences of the given NPC names, and because those occurrences quite often dont't match the NPC name exactly (because they are written in plural form for example, while the NPC name is a singular, or because they are "bent", which is often the case in the German language) this search is done "fuzzy" - that means, "Tamed Hyena" for example does not only match "Tamed Hyena", but also similar names like "Tamed Hyenas" (notice the "s" at the end!). There's a rating for how much difference is allowed for a match which again depends on the length of the name, but to get a high matching percentage (especially in texts written in the German language which tends to modify the names more often and more radically than the English language) those ratings can't be set too high. That was the reason why there were some false positives in the mix, which resulted in crappy links in-game, linking to creatures with similar, but unfortunately wrong names.

The part with the "bad design" was this: if the fuzzy search algorithm found a potential match, it immediately created the link, without searching the rest of the NPC list if there were any more matches to be found. So if there were more than one match, it was just a game of luck if the correct match would be checked first or not. This has now been changed: now the whole list is always scanned, collecting all matches on the way and calculating a "similarity coefficient" which mathematically describes the similarity of the NPC name to the occurrence in the text. After the whole list has been searched, the match list is being sorted by this number, and the match with the highest similarity is picked for the link.

Besides that, there was another problem, concerning an NPC with the name "Hemet Nesingwary Jr." which exists in both the English and German versions of the game. That damn NPC had a dot in its name - and dots are considered to mark the end of a sentence by the QTE, not to be a part of a name. Therefore this NPC, if named in a quest text, was never being recognized as "Hemet Nesingwary Jr." - instead, it was recognized as "Hemet Nesingwary", another NPC that unfortunately exists, too. Well, it's logical from a machine's point of view, as "Hemet Nesingwary" seems to be a perfect match, and "Hemet Nesingwary Jr." does not match that good if the dot at the end is not considered to be a part of the NPC name. But of course this leads to confusion if you click that link in-game, expecting to be shown the location of the guy with the "Jr." at the end. This has been fixed, too.


As all of these changes are done in the backend, you won't really have to do anything to profit - you'll get the "less-faulty" links with your next database update (okay, of course you'll have to download that, either comfortably with the MobMapUpdater or manually). All quest texts and comments are being reprocessed right now, so the next database built in a few hours should contain them.

So there had to be another bugfix release. This one fixes one important bug in the quest event data collection mechanism that was collecting erroneous data with quests which have multiple quest events to fulfill (you might have noticed it; those multiple targets quite often showed the same dots on the map, that is, they couldn't be differentiated correctly).

So if you are collecting data for MobMap (thank you!!!) please fetch this update :o)

Other than that, there still seems to be a strange bug out there with quest events not being found correctly when clicking the quicksearch button next to them. I haven't been able to recreate this bug here on my development machine, so I'd be glad for any hints. Nevertheless, the erroneous data collection has been important enough to justify an immediate release, so if I can recreate and fix this strange search bug eventually (or find any other bugs) there'll be a v2.13.

-> go to the MobMap v2.10 patch notes

v2.12:
- fixed: A bug in the quest event data collection mechanism that resulted in erroneous data especially with quests which have more than one quest event objective.

This is just a tiny maintenance release, fixing a bug that caused MobMap to disrupt LightHeaded's operation in specific circumstances. As that was the only reproducable bug I've seen in MobMap v2.10, I don't expect I have to release a 2.12 any time soon. Though...you never know...


v2.11:
- fixed: A slight incompatibility with MobMap v2.10 and LightHeaded has been fixed. This problem could break LightHeaded's Next/Previous-Buttons if some specific circumstances were met.

-> go to the MobMap v2.10 patch notes

Tonight, just a few minutes after midnight, my server monitoring went havoc and sounded the "red alert" - which means as much as "the server is dead and doesn't respond to anything". And it truly was dead...well, almost, after waiting for like 50 seconds I could SSH into it and do the usual checks - first: take a look at "top" (which is sort of the linux version of the Windows Task manager). And top said that there were like 600 active processes - roughly half of them were Java virtual machines, the other half were apache2 threads - all waiting to get their slice of processing time, which caused the load on the server to shoot up to 163 (actually that's the highest load I've ever seen on a single machine that was not completely dead).

First I thought someone was actively DoSing me, throwing crappy data uploads at the server faster than they could be checked and thrown away. But a quick check revealed that those masses of uploads were actually valid, and they were coming from completely different IP addresses. So it was either a botnet, or...Blizzard! I suddenly remembered that there was some strange text on the login screen yesterday, speaking of "extended maintenance downtime" starting at - you guessed it - midnight. Usually maintenance downtime starts at 3 AM, so there were probably still quite some people playing tonight when the servers went down. The sudden disconnect of thousands of people with the automatic update feature of the MobMapUpdater activated has then caused a massive surge of uploads exactly at midnight. So it was some kind of DDoS attack in the end ;-)



Maybe it wasn't such a good idea to spawn a new JVM for every single upload, which only exists for a split second to run some structural verification checks on the uploaded XML data. Those JVM startups probably produce quite some overhead - resulting in hundreds of processes waiting for execution time when many uploads are being done simultaneously. I might have to see if I can devise a solution for this problem that scales better, maybe by creating a permanently-running verification service that gets called by the PHP upload script - but then this usually does not pose a problem when uploads are evenly distributed over the evening.

So here it is!

This version does finally add quest events to the database, so MobMap should now be able to pull up an answer for practically every quest target in WoW :o)

Besides this bigger feature addition there have been several smaller fixes and changes, as you can read in the changelog:

- changed: There has been a major DATABASE FORMAT CHANGE! You'll need to download a new database when installing v2.10!
- added: New quest event database, which can be queried through a new interface panel or by using the quicksearch buttons on quest event targets
- changed: Options panel was moved to the Blizzard Interface->Addons settings screen and reorganized slightly
- fixed: The new settings added with v2.02 (flashing dots, battlefield minimap, quest comment autoshow) are now being saved permanently
- changed: The functionality of the quicksearch buttons next to quest titles has been further improved. These buttons now use the Blizzard QuestIDs to find the correct quest whenever possible.

And you know the procedure: if you encounter any new bugs, please report them here in the comments!

In preparation of the soon-to-be release of MobMap v2.10 (scheduled to arrive in the next 24 hours), I have implemented a simple script that takes the current MobMap addon .zip file, adds the newest database files to it and puts the result up for download.

The result can be seen on the AddOn download page: there are now 3 options, either the MobMap addon without a database (that was the only option until now) or the addon together with either the german or english database. This should make it a little easier to manually install MobMap as you just have to fetch and extract one file now instead of two :o)

[UPDATE] After solving some problems with Wine (a tool which makes it possible to run many applications compiled for Windows on Linux) I was able to create a similar automated process to build the setup executable every time a new database is being built! This will ensure that the MobMap database delivered with the setup executable is always as up-to-date as possible and relieves me from having to manually rebuild this file from time to time (a chore I admittedly hated and "forgot about" quite often ;-)