[REV-2] A Hot-Off-The-Press Hack Update!

Blog Info: Looking at my blog’s statistics, I get the most hits for Hyrule Magic articles. RPG Maker & Eternal Lands are next, and nobody seems to read my REV series on interesting hacks. Which leads us to the back-story…

Back-story: I was hard at work on my next Hyrule Magic blog entry, which was a lot easier thanks to all your excellent & intelligent comments. (Thanks readers!) But then, I stumbled across this post in byuu’s forum. I’ll explain it as we go along, since the linked post only contains half the story.

Super Mario Odyssey 2 — Now with MSU1 !

Summary: Years ago, BMF54123 suddenly stopped working on Super Mario Odyssey. This made me sad. Then, just last week, I found a post stating that he’d re-started it, and had utilized byuu’s amazing new MSU1 chip to put near-CD-quality audio into the hack.

Wait, that audio’s real!?!?

Technology: Yes, the sound on that video is exactly what you would hear if you plugged the Super Mario Odyssey cart into a SNES and turned it on. There are some caveats; for example, the MSU1 is only emulated by bsnes at the moment. Also, actually making a “Super Mario Odyssey cart” is not possible  –although byuu is making a hardware implementation of the MSU1, so it will be possible in the future. That said, most gamers only play hacks on emulators anyway, so the fact remains: if you play SMW Odyssey on bsnes, expect awesome music!

Other notes: I really like the short “intro” story, showing how Bowser might have ended up with all the crazy time-travel technology that caused the Odyssey in the first place. Also, I like how the game starts with a simple “intro” level (like Yoshi’s island), and I like the “File Select” menu. BMF54123 certainly fine-tuned his ASM skills in the intervening years. (According to the SMW Central forums, he decided to rewrite from scratch because his original code was an undocumented mess.)

What this means: I’d like to think that this hack will usher in a new era of good hacks with rich environments backed up by nice ambient music. However, it’s equally likely to generate hacks of poor quality that are now multi-megabyte patches due to useless MP3 from popular bands.

Why I’m Excited: I haven’t been this excited for a hack in a while –it rivals my Quest for Calatia obsession! In fact, the MSU1 has nothing to do with this  –I’m really just excited that Super Mario Odyssey’s excellent level design and intriguing world design will carry over into a new hack. I’ve always felt that BMF54123 understands what makes a good hack more than most people, and his inclusion of high-quality audio is just icing on the cake.

Your Homework: Watch the video again, and try to pay attention to the level design of the intro level. Even though this is likely just a demo level, it’s very balanced, and far above some hacks’ release-quality levels.

My Homework: Work on a blog post!

[RM-4] TCP Sockets in RPG Maker VX

Blog Info: I was in a bit of a slump lately for new post ideas –it seemed that all my good ideas were far too big, and I didn’t really want to get side-tracked like I did for the EL client notifier. So, I decided to look into the possibility of doing networked games using RPG Maker. Along the way, I finally found a good way to highlight code samples on this blog –by using Scite’s “Export to HTML” feature!

Back-story: When RPG Maker VX came out, it radically changed the organization of Ruby code internal to the engine. What it did not change was the fundamentals of the scripting system itself. Thus, any code in RPG Maker XP could feasibly run on VX –assuming that it made no reference to windows, sprites, etc. Unfortunately, porting this code was not always easy. One module that never made the transition was the RPG Maker XP Online System, programmed by username “Blizzard”. This system was a beautifully designed, fully-functional API for RPG Maker network play. Fortunately, Blizzard released the source code for this engine under the Creative Commons (NC-SA) license. This first entry will deal with the simplest –but most important– problem: using sockets. After that, we will slowly build up.

Goal: Create a simple proof-of-concept that TCP sockets can work on RPG Maker VX.

Table of Contents

Step 1: Use Case

Step 2: Borrowing Library Code

Step 3: Initializing & Testing the Connection with NPCs

Step 4: Taking a Step Back

Step 5: A Slightly More Impressive Proof-of-concept

Game Prototype 1: Autosave and an IP NPC

Game Prototype 2: An Online Community

Game Prototype 3 – Chatting with Other Players

Further Work

Step 1: Use Case

Let’s start with a brand-new RPG Maker project. Create a starting map, and put down some “land” tiles. Then, add an NPC with the following commands:

• Message Box: “Opening Socket”
• Script:

s = TCPSocket.open('127.0.0.1', 7689)
s.close()

• Message Box: “Success!”

Now, start your game and talk to your NPC. You should see the first message box, and then the game will crash.

No "sockets" library means no TCPSocket class, which in turn means that our two-line script will crash horribly.

What just happened? Well, first we tried to create a socket. This socket assumed that we have a server running on 127.0.0.1 (which means “this computer”) and that it is bound on port 7689. Since no server has been started, we expect a crash –however, we get the wrong kind of crash. If you look at the message box, you’ll see that RMVX had trouble finding the “TCPSocket” item. In other words, RPG Maker VX does not ship the default Ruby networking libraries! Oh no!

The thing is, RPG Maker XP also didn’t ship with networking support. So how does RMX-OS use it? If you’ve studied programming languages, you know that features such as networking are not considered part of the language itself; they’re usually linked up directly to the operating system after the language has been created. Doing something like that manually is far too difficult for the average programmer. Fortunately, we might be able to “borrow” the code that Ruby itself uses to connect to sockets.

If you want to follow along, you can download Ruby’s source (it must be version 1.8) from some online source –I recommend the Debian repository, with links below. However, the source really isn’t necessary, since I’ll be quoting samples from it when you need them. Moreover, I can’t seem to find the relevant socket code (although I did find useful Mutex code), so you probably don’t need to download the source unless you’re curious. Here are the links:

http://ftp.de.debian.org/debian/pool/main/r/ruby1.8/ruby1.8_1.8.7.249.orig.tar.gz

…or, more generally, from:

http://packages.debian.org/en/squeeze/ruby1.8

After downloading it, unzip it and remember where you saved the source. The only file we’ll need is thread.rb.

Step 2: Borrowing Library Code

One thing you may have noticed if you’re familiar with ruby is that RGSS (RPG Maker’s Ruby scripting system) doesn’t support the “require” primitive. This makes sense –since “require” searches the standard library path,  you might accidentally load Ruby library code which your playtesters likely won’t have. That said, RPG Maker is still perfectly capable of running native code from DLLs. For example, the following works just fine:

Win32API.new(DLL, 'connect', 'ppl', 'l').call(1, 2, 3)

Thus, we might consider chaining together DLL calls to create a kind of “fake” networking library. Eventually, we’ll probably try to migrate to a so-called “native” library, so that we can gain the benefits of low-level primitives. An example is provided in the win32utils:

http://rubyforge.org/projects/win32utils/

If you download and scan, say, the win32-nio library source, you’ll see lots of “require” tags at the top of the main library files. Be aware that you’ll have to essentially replace these with compatible functionality if you want to use the library.

Therefore, we won’t use win32-nio just yet. Instead, we’ll copy the Win32 API that “blizzard” put inside RMX-OS. He claims that it’s from Ruby’s 1.8.1 library source, but for the life of me I can’t find it online. That’s not really a problem, though, as the Ruby License allows verbatim copies of code and distribution in some format.

RMX-OS’s networking code is, by the way, built on top of DLL calls. Here’s the code you need to copy:

Code Segment 1 — Apologies for the ODT format, but PDF removed newlines and tabs.

In the Script Editor (F11), right-click on “Main” and choose “Insert”. Type “Ruby Library Code” in the “Name” box, and then paste the source code from segment 1 into the edit panel. Choose “ok”, then save your game.

If you run your code now, you’ll get some vague, disheartening runtime error. So, before you get depressed, delete your test NPC and add the following code directly into the Main script, below Graphics.freeze:

s = TCPSocket.open('127.0.0.1', 7689)
s.close()

Now, save and run the game. You’ll get the following error on startup:

This error means the library code loaded, then couldn't find the server. We are making progress!

Now that’s the kind of error we’re looking for! Congratulations, the basic structure of the TCP code you added was correct!

Step 3: Initializing & Testing the Connection with NPCs

We need to test an actual TCP connection –in addition, we should probably fix that bug where NPCs can’t seem to trigger our code. These are our next tasks.

If you try to track down the error from the NPC-enabled network code, you’ll reach the following line:

  def self.check
    errno = Winsock.WSAGetLastError
    raise Errno.const_get(Errno.constants.detect { |c| Errno.const_get(c).new.errno== errno })
  end

The error is located inside that particularly nasy “raise” statement. If you don’t understand Ruby’s “block” mechanism, you’ll probably feel like quitting right here. Instead, try this simple abstraction:

  def self.check
    errno = Winsock.WSAGetLastError
    constName = Errno.constants.detect { |c| Errno.const_get(c).new.errno== errno }
    raise Errno.const_get(constName)
  end

Run this code again, and the error’s still on the “raise” line… but now it’s superficially easy to understand. The original programmer was trying to look up Win32 error messages by ID, and he didn’t account for the possibility that a null socket might be returned with NO error code being generated in the process. Well, this is easy enough to fix:

  def self.check
      errno = Winsock.WSAGetLastError
      constName = Errno.constants.detect {|c| Errno.const_get(c).new.errno == errno }
      if constName
        raise Errno.const_get(constName)
      else
        raise "Unknown network error code: #{errno}"
      end
  end

Now, if you try to trigger a TCP connection from the NPC, you’ll see your “Unknown network error code” message.

At this point, I should probably mention that using TCPSocket.open() is not technically correct, since it expects to be passed a block of code to execute upon completion. Although that might be ok for simple socket-related tasks (like requesting the current time or weather conditions from a central server), we’ll need a persistent connection for our game engine. As you might expect, our code will overflow the limits of an NPC’s script command. So, change the NPC’s script code to read:

tcptest()

…and add the following to the “Main” module, directly above the first “begin” statement:

  def tcptest
    #Createa a socket
    s = TCPSocket.new('127.0.0.1', 7689)

    #Send a test message
    s.send("Testing...\n")
    #Receive a result from the server
    msg = ''
    while
      buffer = s.recv(1024)    #Read UP TO 1024 bytes
      buffer.gsub!(0.chr, '') #Remove null bytes
      msg += buffer            #Append received data
      break if buffer.count("\n")>0   #Stop if we've reached the newline
    end
    #Done; close the socket, print our message
    s.close()
    print "Received: #{msg}"
  end

Our code is fairly simple; we use “new” instead of “open”, and then rely on “send” and “recv” to send and receive our data. One problem with “recv” is that it won’t wait until the server has sent the entire message, so we have to chain together all of the bits and pieces of the message as they arrive. On the upside, one of the guarantees of TCP is that we’ll never receive messages out-of-order, so that’s one less potential headache.

Now all we need is the server. I find Ruby a bit distasteful for client/server code, so let’s try writing one in Java. One issue with Java is that compiling and running it can be a pain. Rather than give you a full tutorial on Eclipse or javac, I’m going to ask you to download and install TextPad.

Edit: Some users reported needing to download the JDK, too. Open a command prompt and type javac -version. If you get an error message, you’ll need to download the JDK. If all else fails, see the “source code” section for compiled versions of the server code.

Now, create a new file, and save it as SimpleServer.java. The name must be spelled exactly this way; such a naming convention is a feature of Java. Inside the file, add the following code:

import java.io.*;
import java.net.*;

class SimpleServer {
  public static void main(String argv[]) throws Exception {
    String clientSentence;
    String capitalizedSentence;
    ServerSocket welcomeSocket = new ServerSocket(7689);

    for ( ; ; ) {
      System.out.println("Waiting for connection...");
      Socket connectionSocket = welcomeSocket.accept();
      System.out.println("  >Socket connected");

      BufferedReader inFromClient = new BufferedReader(new InputStreamReader(connectionSocket.getInputStream()));
      DataOutputStream outToClient = new DataOutputStream(connectionSocket.getOutputStream());
      clientSentence = inFromClient.readLine();

      System.out.println("  >Received: " + clientSentence);
      capitalizedSentence = clientSentence.toUpperCase() + '\n';
      outToClient.writeBytes(capitalizedSentence);
      System.out.println("  >Replied: " + capitalizedSentence);
    }
  }
}

Pretty simple? We’ve ignored error checking in this code, so it’ll be fairly easy to crash the server. But for a quick demo, it’ll do. The server reads a sentence from the client, and capitalizes the alphabetic characters. It then returns this uppercase version of the sentence. Since we send “Testing…” to the server, we expect the result to be “TESTING…”.

To compile the server code, choose “Tools->External Tools->Compile Java” (or Ctrl+1).

Compiling Java code in TextPad

If your code contained no errors (and it shouldn’t if you’re copying and pasting) then you’ll see a message in the output window that reads “Tool completed successfully”. Now, choose “Tools->External Tools->Run Java Application”. At this point, a command window will appear bearing the message “Waiting for connection…”

If you see this text, then your server has been successfully bound to port 7689.

This means that our server is ready for the game client to connect to it. Run the game, then talk to the NPC. You’ll see a popup dialog with the response from the server.

Our RPG has successfully communicated with the server, sending it a message and receiving one back in return.

The server window will also have a few more messages:

Our server log proves that the communication was real, and also that Java and ruby can "speak the same language" using sockets.

Note that, since we closed the connection, the server is now waiting for another connection to be opened. We could talk to the NPC again, or simply close and re-open the game, then talk to the NPC again. Eventually, we will require each client to hold on to a connection from the server, but for now this proves the concept: networked RPGs in RPG Maker VX should work just fine. Press Ctrl+C to close the window; you might then have to type “Y” and press Enter to confirm that you want the server to halt.

Step 4: Taking a Step Back

At this point, it’s worth pausing for a second to consider what we’ve actually accomplished. There’s a lot of “glue code” (like reading lines of text from ruby) that hides what’s actually going on:

• We’ve created a server that can “receive” connections on port 7689, and a client that can request a “connection” on this port.
• Once connected, a client and a server can send and receive a sequence of characters.

That’s all there is to it. Adding multiple clients, some kind of synchronization, and partitioning our data into “messages” or “game events” are just additional details on top of what the tcp client actually is. For example, instead of sending strings of messages, we could send pairs of NPC x/y co-ordinates. In that way, we could treat “heroes” in separate RPGs as “npcs” in your RPG world. We could send “tiles”, and allow users to build their own houses (as maps) which they can allow their friends to visit. We can send custom avatar pictures for each player, or even co-ordinate some kind of battle system. All of these things can be accomplished simply by sending letters and numbers over a TCP connection.

Step 5: A Slightly More Impressive Proof-of-concept

Our “borrowed” Ruby source code is about 500 lines in total. This is only about 8% of the total RMX-OS code, which includes things like Scenes for connecting to the network and a password system to help protect users’ passwords and privacy. My point is, making anything interesting is going to be a huge undertaking. On the other hand, a simple message box usually isn’t enough to prove to your boss that you’ve accomplished a huge feat like networking a game engine that doesn’t normally expect to be multiplayer. What, then, might a nice, simple prototype look like?

• We don’t want to play around with Scene overloading. So let’s make an NPC that basically allows us to connect to a network, and manage our network configurations.
• When the game begins, the player can choose one of four starting heroes. This information is transmitted over the network.
• At this point, we should create NPCs for each of the other users online. These will just wander randomly for now; they won’t reflect the actual position of the other users.
• The hero can talk to these NPCs to receive “messages” that they have sent (like a chat). Only the most recent message is shown.
• When an NPC gets a new message to display, we can show a “balloon” animation above him –an exclamation mark or something similar.
• To send a message, the hero can talk to another NPC (like the connection NPC). We might use the hero name dialog, or we might capture key presses. In the worst case, we’ll just let the user select some pre-defined messages from a list.

There’s a lot of bullets (e.g., a lot of things to show off) but individually, each is not terribly difficult to accomplish. Let’s get started!

VERY IMPORTANT NOTE: Whenever you change some of your game’s code (scripts), you should re-test using a new game. If not, you risk loading old code from an old save file (it happened once to me, while testing this blog entry) and it is frustrating to track down an error like this. In my case, the initialize method for my Game_Map class was not re-called upon loading an existing save file, and I couldn’t figure out why some class variables didn’t exist anymore.

Game Prototype 1: Autosave and an IP NPC

Our first prototype will have no networking capability at all. We’ll need two maps; one for the player to select an Avatar upon starting a “New Game”, and the other for normal play. Create these two maps and call them “Player House” and “Character Select”. Make them both small (20 x 15 or so) and go to the Character Select map. Give it a fancy layout and set the hero’s starting position to be somewhere on the map.

Our spartan starting map.

Now, add a few NPCs (five would be good), each with the graphics of the first five heroes in the party. If you don’t like the default hero pictures, you should go into the database editor (F9) and change them. Just make sure they match the pictures of the NPCs you just added. While you’re in the database editor, you might want to change the Initial Party (in the System tab) to include only hero 6, and change hero 6‘s walkabout to be something more mysterious. Name him SELECT to avoid confusion.

Five possible Avatars on our starting map.

If that was all too much for you, you might consider reading through a few RPG Maker tutorials –this blog post would likely double in size if I tailored it to complete beginners. (If someone’s willing to do a voiceover for me, I’ll put up a video walkthrough on Youtube.)

Open up the left-most NPC in the Event Editor (double-click) and add the following on page 1:

Talking to an NPC activates this event, which asks the user if she wants this avatar.

To summarize that code briefly:

• Show a text box asking if the player wants this hero.
• Have a “choice” command (Yes/No). We only care if the user picks “Yes”
• If so, set variable 1 (which we call Avatar ID) to be 1.
• We then add the first hero (“Ralph”) to the party.
• We remove the SELECT hero.
• We transport to a random point on the Player House map.
• We then “auto-save” using the following script:

Scene_File.auto_save($game_variables[2])

Now, before we talk about auto-saving, copy and paste the event code from NPC 1 to each of the other 4 NPCs. When you paste the events, make sure to change the text in red to reflect avatar 2, 3, etc. Each player should introduce him or herself, ask the player to choose him/her, and then save his/her details into variable 1 and create a party with only him/her in it. Isn’t gender neutral language fun?

Let’s talk about the auto-save. You can probably guess that “auto-save” is (or will be) a static method in the Scene_File class, and that it will take a save slot ID (which we will store in variable 2. Open the script editor (F11) and browse to Scene_File. Add the following directly underneath Scene File < Scene Base:

  #Save to a slot directly.
  #Indexed from 1, not 0.
  def self.auto_save(file_index)
    fileName = "Save#{file_index}.rvdata"
    file = File.open(fileName, "wb")
    Scene_File.write_save_data(file)
    file.close
  end

We copied most of this code from the do_save method, making sure to construct the right file name. By naming our code as self.auto_save, we make the method static –the user code doesn’t need a (new) object to use the function. Unfortunately, this requires that  write_save_data also be static. Find the following line in Scene_File:

 def write_save_data(file)

…and change it to

 def self.write_save_data(file)

However, this will break normal saving, which we probably don’t want. Find the following function in Scene_File:

 def do_save
   file = File.open(@savefile_windows[@index].filename, "wb")
   write_save_data(file)
   file.close
   return_scene
 end

…and change it to:

 def do_save
   file = File.open(@savefile_windows[@index].filename, "wb")
   Scene_File.write_save_data(file)
   file.close
   return_scene
 end

This way, do_save will still work properly (by calling the now-static function). There are other ways to accomplish this, but this is the easiest to describe.

One tiny problem remains: how do we save the preferred autosave slot into variable 2? I see three possibilities:

• Just pick a slot (e.g., “the first save file”) and use that. This is too hot –it forces an arbitrary choice on the player.
• Give the player a choice box with options “1,2,3,4”. This is too cold –it’s a bit impersonal for players used to the nice save dialog of RPG Maker.
• Force the player to save the game once, and use that value. This is just right –it’s natural to the player, and shouldn’t be too hard to figure out from our end.

Needless to say, we’re doing option 3. Create a new event with no graphic in the top-left corner of the map and give it two pages. The second page should have the “Switch” box checked under “Conditions”, and have “Switch 1” selected. (We call this switch Chose Save Slot.) The first page should have its trigger set to “Autorun”, and have the following event code:

The code for our auto-save enforcing NPC.

In other words, we first tell the user to select an auto-save slot. Then, we show them the save menu. Now we need to check if the user canceled the menu. We do this by setting variable 3 (called TMP Save Count) to be equal to “Save Count” (listed under “Other” in the variables dialog). We then do a simple conditional branch: if this variable is greater than zero, we continue. The first thing we now do is execute a nifty script:

  #Set var[2] to the save slot
  slot = $game_temp.last_file_index + 1
  $game_variables[2] = slot

The variable last_file_index is already part of the game engine. It stores the id of the last slot saved to, starting from zero. (Since save slots are actually numbered from 1, we increment this value.) The default value for last_file_index is zero, by the way, so we can’t just check this to determine if the user canceled or saved.

Moving on, we then set switch 1 to ON. Next, we set the names of heroes 7 and 8 to 127.0.0.1:7689. This is not very important right now; just add the lines and we’ll get back to them later. Finally, we perform our “classic auto-save”

  #Save the game to slow (ID-1)
  Scene_File.auto_save($game_variables[2])

You should memorize this code fragment; we’ll be using it a lot.

At this point, you can run the game, and you will be prompted to save. Following that, you can choose a hero and then walk around the Player House map. Not too bad for a quick-and-dirty auto-save, eh? In fact, if you play around with quitting and reloading games, you’ll find that our code is actually pretty robust; cancel or close the program whenever you want and everything should still work fine when you reload it.

Our auto-save dialog performs impressively.

Go to the Player House map and create a nice, homely setting. Add some ambient NPCs if you want; just make sure there’s a scholarly-looking NPC somewhere near the middle of the map. We’ll use him for connecting to the server. Here’s just one possible setup:

Only the primary NPC matters, but feel free to make this map as picturesque as you like.

This NPC will have a whole lot of functionality wrapped up inside his persona. To make the code more manageable, we’ll put different functions into different common events. The core code for the map NPC looks like this:

The main code of our "Connect" NPC.

To summarize:

• This NPC can either introduce himself or call the “Connect to a server” Common Event, which has the ID of 1. The rest is just fluff.

Open the database editor (F9) and browse to the Actors tab. Name actor 7 SAVED_SERVER_1 and actor 8 SAVED_SERVER_2. Then browse to the Common Events tab. Name common event 1 “Connect to a server” menu. Name common event 2 Prompt for server address. Name common event 3 Prompt to Save Server. Now, for common event 1, add the following code:

CE 1 handles connecting to a server. It is currently incomplete.

This event is quite simple:

• First, it asks the user to choose a server. The first two options are the names of the SERVER characters that we just created. At present, nothing happens when these are selected. The third option allows us to add a new server. It calls the following two common events in sucession.

The second common event is also quite simple, if a bit repetitive:

CE 2 allows us to construct new IP addresses. It's just a little cumbersome.

It performs the following routine five times:

• Tell the user the name of hero SELECTION, then ask for a number. Store this number in variable 4 (TMP server part)
• Append this number to the name of the SELECTION hero using the following script:

    #Set actor's name
    actor = $game_actors[6]
    actor.name = "#{$game_variables[4]}."

  • …note, of course, that the “.” changes to a “:” for entry 4, and “” (nothing) for entry 5.

The third common event simply checks if this is what the user wants:

CE 3 just confirms the user's entry, saves it, and resets the SELECTION npc's name.

It saves, e.g., actor 7 by using the following script:

 #Finalize actor's name
 $game_actors[7].name = $game_actors[6].name

Of course, we reset the SELECT hero’s name to avoid confusion later. We then wrap it all up with an auto-save. And that’s it! We now have a simple way to enter server addresses without requiring any fancy interface or custom save files. Here’s a screenshot of our system in action:

The orange text makes it look at least ten times more professional.

Once the selection has been saved, it shows up on our options list:

The beauty of using an Actor's name to save the IP address is that you can use it in a Message Box without any fancy tricks.

It’s nearly time to move on to our second prototype (which will introduce network elements). Before we get too far ahead, I’d like to highlight some of the merits of our approach:

• We relied for the most part on local scripts, requiring only a single change to the game’s source code. In our defense, the function write_save_data should have been static anyway, and could have been fudged if we were so inclined to create a dummy object.
• By using several Common Events, we avoided the RPG Maker-exclusive temptation to create one messy NPC with all our code. In fact, nearly all events had single-screen pages of Event Commands.
• We used “special heroes” to store our IP addresses. This allowed us to stick with event commands like “Show Choice” instead of requiring custom Scripts for everything.
• By and large, we used mostly RPG Maker constructs, relying on scripts only for tiny globs of “glue code” where the interpreter fell short. Most intermediate RMVX users could probably figure out what our code does, and how to improve it.

Enough resting on our laurels; it’s time to notch up the interest level of our audience.

Game Prototype 2: An Online Community

We strive for incremental improvements, so let’s try adding only the most basic of network functionality:

• When a user Connects, we’ll add an (arbitrary) NPC to the player’s home map.
• When a user Disconnects or closes the game, we’ll remove this NPC.

This will require a lot more foresight than you might expect. First, we need to decide how the client should handle messages. After all, we don’t want our game engine to hang just because the network goes down. Following that, we’ll need a simple message format that is simple to expand with new data types. Finally, we’ll need a messaging protocol: how often should we send updates to users? For now, we’ll just assume “always send every update”, since we’re not tracking anything expensive like a user’s X/Y co-ordinates. For bigger games, the messaging protocol is the most important part of the network design. If your game’s lagging, try reading a few papers on interest management (PDF link) in online games.

First, let’s talk about threading. Like most modern high-level languages, Ruby runs inside a virtual machine, which itself is a single process in the Windows operating system. All ruby-related code in your game must therefore run in sequence. This includes all parallel or autostart events, common events, NPC interactions, and scripts. Unfortunately, this makes it very difficult to add networking. Let’s say that you check for messages from the server before you service all parallel process events. If your game has a lot of these events running at once, the network buffer might overflow, and you’ll lose messages. (Very Bad.) What we really want is a way of telling Ruby to check for network messages “every so often”. That’s exactly what threads were designed to do.

Consider the following Ruby code:

  #Code our thread will execute
  def network_thread
    while true
      read_server_messages()
      send_waiting_messages_to_server()
      sleep(1) #Avoid spinning too fast
    end
  end

  #Actually make a thread
  t1 = Thread.new{network_thread()}

Conceptually, the line with Thread.new tells Ruby’s virtual machine to make the network_thread() function run once inside a new thread. This thread will alternate with the main (RPG Maker) thread in the virtual machine. According to Ruby’s time-slicing policy, each thread will run for 10ms before the other thread gets a turn to run. Inside network_thread, we call a function which reads all messages that the server has sent to us. Note that we do not process these messages yet; we’ll do that in the main game thread. Instead, we simply get them out of the network buffer and into the game engine. Following that, we send any waiting messages to the server. Again, these messages are generated by the game engine; we don’t really care how they’re made. Then, we perform a call to “sleep”, which forces the current thread to suspend. That way, if we only used 2ms of our time slice, the main game engine doesn’t have to wait 8ms while we do nothing useful.

This is only one way of doing things. We might also have put read_server_messages and send_waiting_messages_to_server into their own separate threads.

Now, let’s talk synchronization. Where does read_server_messages save the messages it reads? Presumably, we’ll have an array to store them:

 read_messages = []

The network thread can then put messages into the back of this array, and the game engine can read messages from the front. The problem is, what happens if both try to act on the array at the same time? The answer: something Very Bad. To avoid this, we should synchronize access to this array. For example, the game engine might contain code like the following:

  GameMessage nextMessage = nil
  @lock.synchronize { 
    nextMessage = read_messages.shift unless read_messages.empty?
  }
  if nextMessage
    #....process nextMessage
  end

The @lock variable might be created with a call to:

  @lock = Mutex.new

It ensures that all the code inside the locked block executes before any other locked code. To avoid unnecessary slowdown, it is a good idea to keep the code inside a lock block as simple as possible –note that we store the next message in nextMessage, rather than processing it inside the locked section.

There’s just one problem: Mutex.new requires the ‘thread’ library. And I mean requires; that is, we don’t have access to it in RPG Maker code. (Thankfully, we do have access to thread). We can’t use the equivalent “monitor” class; it’s also required. Fortunately, there’s an easier solution. Remember the ruby 1.8 source code we downloaded earlier? Go to that directory, and then browse to the lib directory. You’ll see the thread.rb file which contains mutexes. Copy and paste the source code into your Ruby Library Code module (F11) at the very top. Segment 2 contains this code, minus most comments:

Code Segment 2 – Ruby code.

And that’s all we need to do to enable mutex creation:

The Mutex object was created properly. (Sample source code not included.)

We nearly hit a dead-end, but synchronization works. Bullet dodged; that could have killed the entire project. (Although not entirely: RMX-OS doesn’t use threads and achieves reasonable performance.)

All drama aside, you should now be able to envision how we can combine synchronization with threading to read messages from the server. But what is a game message? We want something easy to extend, and something that (preferably) doesn’t rely on newlines to specify message boundaries. Typically, we would use an array of bytes. However, the Ruby 1.8 library code we borrowed doesn’t support this, forcing us to read and write strings. There are some very good reasons why we don’t want to force byte arrays into strings:

• For some reason, I can’t get the Socket.read command to work. This would have allowed us to read X bytes from the socket, instead of recv‘s “up to X” bytes. (I had actually written half this guide based on read until I realized it didn’t work. Silly me.)
• If we use recv to read bytes, the default for “un-read” bytes is 0. But what if we actually wanted the number 0? We can get around this (add 1 to each byte sent, or initialize our buffer array with -1 instead of 0) but these “solutions” feel more like hacks –and although I derive great enjoyment from puzzling out another programmer’s creative solutions, I dislike writing obfuscated code myself.

So, we’ll make this simple: a message is just a string, of the format:

• TYPE:xxx:yyy:zzz:\n

In other words, a message is composed of a series of strings, separated by colons, with a newline designating the end of the string. The type of the message determines how many remaining segments are expected, and what they mean. Here are our message types, and their corresponding data.

The first type is CONNECT_REQUEST, sent when a client wants to log in to the server.

CONNECT_REQUEST	Description
Type	The string CREQ
Avatar ID	The ID of the avatar our player has chosen.

The second type is CONNECT_RESPONSE, sent back from the server whenever a client tries to log in.

CONNECT_RESPONSE	Description
Type	The string CRES
Response	If 0, the server refused to allow you to connect. Any other number represents your ID on the server. Clients should save this value.

The third type is USER_CONNECT, sent whenever another user has connected to the server to all other players currently online. A series of these messages is also sent from the server to a client that has just logged on, lest he think he logged in to an empty world.

USER_CONNECT	Description
Type	The string UCON
Player ID	The ID of the player who has connected.
Avatar ID	The ID of the avatar this player has chosen.

The fourth message type is USER_DISCONNECT, sent when a user has disconnected.

USER_DISCONNECT	Description
Type	The string UDIS
Player ID	The ID of the player who has disconnected.

The server manages all user connections, so things like timeouts, etc. are not the client’s responsibility. In some cases, the server may shut down; if so, it might send USER_DISCONNECT messages to all clients. Thus, if a client receives a USER_DISCONNECT message with its own user ID, it can assume that the server will send no further messages. Moreover, if the client sends a USER_DISCONNECT message to the server, it is probably trying to request a disconnection.

Let’s actually get some of this code working. First, we’ll need a Game_Shared class and its respective $game_shared singleton for containing all our shared objects (which require mutually exclusive access). Add the Game_Shared module below Game_Interpreter:

#==============================================================================
# ** Game_Shared
#------------------------------------------------------------------------------
#  Contains all objects that may be shared among threads, and contains mutexes
#  for controllling these accesses.
#==============================================================================

class Game_Shared
  
  #--------------------------------------------------------------------------
  # * Object Initialization
  #--------------------------------------------------------------------------
  def initialize
    @read_messages = []     # Just read from server
    @read_lock = Mutex.new  # Lock for read_messages
    @messages_to_send = []  # To send to server
    @write_lock = Mutex.new # Lock for messages_to_send
  end
  
  #--------------------------------------------------------------------------
  # * Public Accessor Functions --From Server
  #--------------------------------------------------------------------------
  def getNextServerMessage
    ret = nil
    @read_lock.synchronize { 
      ret = @read_messages.shift unless @read_messages.empty?
    }
    ret
  end

  def queueServerMessage(msg)
    @read_lock.synchronize { 
      @read_messages.push(msg)
    }
  end

  #--------------------------------------------------------------------------
  # * Public Accessor Functions --From Client
  #--------------------------------------------------------------------------
  def queueClientMessage(msg)
    @write_lock.synchronize { 
      @messages_to_send.push(msg)
    }
  end

  def getNextClientMessage
    ret = nil
    @write_lock.synchronize { 
      ret = @messages_to_send.shift unless @messages_to_send.empty?
    }
    ret
  end

end

To create the singleton, go into Scene_Title and find the function create_game_objects. Add the following text, right at the end of the function:

  $game_shared        = Game_Shared.new

Now, we will have access to our singleton object for most of the game. The shared object singleton allows us to read and write messages to and from the server, using arrays to queue the messages properly. It handles all locking internally, so we don’t need to worry about Mutexes outside of our Game_Shared class.

We’ll also need a Game_NetMessage class; add it directly below Game_Shared.

#==============================================================================
# ** Game_NetMessage
#------------------------------------------------------------------------------
#  A light-weight class used to store messages sent to/from the server.
#  Contains helper methods to construct most message types.
#==============================================================================
class Game_NetMessage
  
  #Message types
  def self.CONNECT_REQUEST  
    "CREQ" 
  end
  def self.CONNECT_RESPONSE 
    "CRES" 
  end
  def self.USER_CONNECT     
    "UCON" 
  end
  def self.USER_DISCONNECT  
    "UDIS" 
  end
   
  #Helper methods
  def self.Make_ConnectRequest(avatarID)
    "#{Game_NetMessage.CONNECT_REQUEST}:#{avatarID}\n"
  end
  def self.Make_ConnectResponse(playerNewID)
    "#{Game_NetMessage.CONNECT_RESPONSE}:#{playerNewID}\n"
  end
  def self.Make_UserConnect(playerID, avatarID)
    "#{Game_NetMessage.USER_CONNECT}:#{playerNID}:#{avatarID}\n"
  end
  def self.Make_UserDisconnect(playerID)
    "#{Game_NetMessage.USER_DISCONNECT}:#{playerID}\n"
  end  
 end

This class functions entirely through static method calls; you’d never need to create a Game_NetMessage object. Instead, you’d use it like this:

  msgToSend = Game_NetMessage.Make_ConnectRequest(3)
  $game_shared.queueClientMessage(msgToSend)

…to make a connection request message and queue it for sending to the server.

All we need now is a $game_network class which performs the following tasks:

• Create a network thread when the user connects to the server. Destroy this thread (safely) when the user disconnects, or when the server kicks him off.
• Maintain some statistics relating to the network: Is it connected? What’s the latency?
• Make sure that the network thread calls $game_shared.queueServerMessage() when it receives a message from the server. Make sure it sends data to the server whenever $game_shared.getNextClientMessage() returns something non-nil.
• Catch most bad behavior (like trying to connect to a server multiple times, etc.)

Given this brief description, we might create Game_Network (placed directly under Game_NetMessage) as follows:

#==============================================================================
# ** Game_Network
#------------------------------------------------------------------------------
#  This singleton class controls network access
#==============================================================================
class Game_Network
  
  #--------------------------------------------------------------------------
  # * Object Initialization
  #--------------------------------------------------------------------------
  def initialize
    @s = nil                # Our socket
    @loc = ""               # Help with debugging a bit...
    @network_thread = nil   # Thread is non-nil when we have a server connection
    @kill_thread = false    # Set to "true" to gracefully terminate this thread
    @server_message_buffer = [] # Buffer of bytes received from the server
    @my_id = -1 #Managed here, not in the game manager
    @connect_mutex = Mutex.new
    @discon_mutex = Mutex.new
    
    #Always start OFF
    $game_switches[2] = false
  end
  
  #--------------------------------------------------------------------------
  # * Main Functionality
  #--------------------------------------------------------------------------
  def handle_messages
    #Write while we have messages to send
    @loc = "handle_messages():send"
    while true #Can always send
      #Anything to send?
      @loc = "handle_messages():check to send"
      clientMsg = $game_shared.getNextClientMessage
      break unless clientMsg
      
      #Send it
      @loc = "handle_messages():send messages"
      @s.send(clientMsg)
    end
    
    #Read while we have data available
    @loc = "handle_messages():receive"
    partial_message = nil
    while @s and @s.ready? #Can only recv if the socket is ready
      #Read as many remaining bytes as possible
      #NOTE: Using -1 means that there will always be a '' entry if 
      #      our string originally ended in \n
      readStr = @s.recv(0x1024)
      readStr.gsub(0.chr, '')
      
      #TEMP
      #print "Read: #{readStr}" unless readStr.empty?
      
      
      #Continue?
      next if readStr.empty?
      lines = readStr.split("\n", -1)
      
      #Any partial message to complete?
      if partial_message
        lines[0] = partial_message + lines[0]
        partial_message = nil
      end
      
      #Is the final message partial? If so, defer processing till later
      partial_message = lines.pop 
      partial_message = nil if partial_message.empty?
      
      #Ok, we've got a series of messages. Queue them for the client
      lines.each{|message| 
        @loc = "handle_messages():process single message"
        
        #Split this one too; remove newline
        segments = message.rstrip.split(/:/)
      
        #TEMP: DEBUG
        #print "Message: #{message}"
        
        #We don't always post it
        if segments[0]==Game_NetMessage.CONNECT_RESPONSE
          id = segments[1].to_i
          @my_id = id if id > 0
        elsif segments[0]==Game_NetMessage.USER_DISCONNECT and segments[1].to_i==@my_id
          destroy_thread()
        else
          @loc = "handle_messages():try to queue"
          $game_shared.queueServerMessage(message)
        end
      }
    end
  end
    
  
  #--------------------------------------------------------------------------
  # * Thread access
  #--------------------------------------------------------------------------
  def connect(serveraddr)
    @connect_mutex.synchronize { #Keep re-entrant calls legal.
      #Error?
      raise "Network error: already connected" if @network_thread
    
      #Parse out the port from the server address
      serverparts = serveraddr.split(':')
      if serverparts.length != 2
        raise "Network error: cannot connect to \"#{serveraddr}\", bad address"
        return
      end
      serverparts[1] = serverparts[1].to_i
      
      #Create a network thread to connect and parse messages
      # Note that threads must handle their own exceptions
      @kill_thread = false
      @network_thread = Thread.new {
        begin
          #Connect
          @loc = "connect():init connect"
          @s = TCPSocket.new(serverparts[0], serverparts[1])
          $game_switches[2] = true
          
          #Send first message: whoami?
          @loc = "connect():send first message"
          avatarID = $game_variables[1]
          msg = Game_NetMessage.Make_ConnectRequest(avatarID)
          $game_shared.queueClientMessage(msg)
          
          #Process
          while true
            #Done?
            @loc = "connect():check if done"
            @discon_mutex.synchronize {
              break if @kill_thread
            }
            
            #Process any messages
            @loc = "connect():call handle_messages()"
            handle_messages()
            @loc = "connect():returned from handle_messages()"
            sleep(1)
          end
        rescue StandardError => err
          #Print an error, destroy this thread
          print "Exception in network thread(#{@loc}): #{err}\n"
          @network_thread = nil
        end
      }
    }
  end
  
  def ask_to_disconnect
    if @my_id != -1
      #Queue a closing message
      msg = Game_NetMessage.Make_UserDisconnect(@my_id)
      $game_shared.queueClientMessage(msg)
    else
      #Just kill the thread
      destroy_thread()
    end
  end
  
  def destroy_thread
    #Signal the thread to stop
    @discon_mutex.synchronize {
      @kill_thread = true
    }
    if @network_thread and @network_thread != Thread.current
      @network_thread.join
    end
    @s.close() if @s
    
    #Rest to nil
    $game_switches[2] = false
    @network_thread = nil
    @s = nil
    @my_id = -1
  end
  
end

Ok, that was far longer than I’d hoped, but that’s probably because my understanding of Ruby is pretty lousy. Before we analyze it, add the following to the end of Scene_Title‘s create_game_objects function:

  $game_network       = Game_Network.new

This creates our singleton object; now, let’s talk about the main code. First, some disclaimers: even though I’ve added a few mutexes, this code definitely isn’t thread-safe. If you call connect() and destroy_thread() multiple times and quickly, the code will likely destabilize. However, it probably won’t mess up under normal usage conditions. There’s only three  main differences between this code and the snippet we used in Step 3 to test our network connection:

• We borrowed the message loop from RMX-OS’s listen method, although we changed a few things to suit our purposes.
• Although the game engine handles most messages, our Game_Network class automates connection/disconnection requests and the storage of our own ID. We only have to call the connect and ask_to_disconnect methods (and read all other relevant messages in the game thread, of course) for communication to work.
• We use Switch 2, which you should now name Network ON, to store the status of the network. This makes it easy for NPCs to react to “being online” without requiring them to use any scripting.

The client-server communication is only halfway complete; we also have to create client messages to send and respond to server messages. This will occur inside the game engine. Briefly, the following needs to happen:

• Call connect and ask_to_disconnect when appropriate.
• Handle USER_CONNECT and USER_DISCONNECT messages (and discard or err on unknown messages) at some central point in the client code.

That’s it. Our client sends no outgoing messages that aren’t handled in Game_Network (for now). The question remains: where should we check messages? It should be somewhere central, and somewhere accessed often (but not too often). I recommend Game_Map´s update method. Add the following line to Game_Map´s update method, after update_scroll and before update_events:

  update_network

Now, somewhere in that module, add the code for update_network:

  #--------------------------------------------------------------------------
  # * Update Network: Read messages  (write as-needed elsewhere)
  #--------------------------------------------------------------------------
  def update_network
    #"Process no more than X messages"
    nextMsg = nil
    for i in (1..10)
      #Read the next message
      nextMsg = $game_shared.getNextServerMessage
      break unless nextMsg
      
      #Split
      segments = nextMsg.rstrip.split(/:/)
      
      #Process
      if segments[0]==Game_NetMessage.CONNECT_REQUEST
        next
      elsif segments[0]==Game_NetMessage.CONNECT_RESPONSE
          #Always refresh
          $game_map.need_refresh = true
      elsif segments[0]==Game_NetMessage.USER_CONNECT
        #Add an NPC to the map with the right picture and behavior.
        #In range (3,9) , (11,14)
        avtID = segments[1].to_i - 1
        plrID = segments[2].to_i
        if avtID < @max_network_npcs
          #Get the event, place it
          #print "Moving NPC #{@net_npc_start + avtID}"
          newNPC = @events[@net_npc_start + avtID]
          moveNPCRandom(newNPC, 3, 9, 8, 5)
          
          #Set its variable flag (for picture)
          $game_variables[@net_npc_picvar_start + avtID] = plrID
          
          #Always refresh
          $game_map.need_refresh = true
        end
      elsif segments[0]==Game_NetMessage.USER_DISCONNECT
        #Remove the NPC
        avtID = segments[1].to_i - 1
        @events[@net_npc_start + avtID].moveto(1, 1)
        
        #Remove its picture
        $game_variables[@net_npc_picvar_start + avtID] = 0
        
        #Always refresh
        $game_map.need_refresh = true
      end  
    end
  end

This code is not complicated, but let’s review it anyway.

• First, we read through all available messages. Since a very busy server will probably never stop spawning messages, we limit our processing to 10 messages per game “tick” (which should equate to 600 messages processed per second). We might have to change this later.
• Next, we react to each message:
  • When a user connects, we take a pre-defined npc and move it to a random location between (3,9) and (11, 14). We will create placeholder events later. We then set a pre-defined variable to be equal to the avatar’s ID. Presumably, our pre-defined event will have a page for each avatarID with its respective walkabout.
  • When a user disconnects, we teleport that NPC to the top-left corner of the map, and set its avatar ID variable to 0, thus making it invisible.
• A note about $game_map.need_refresh; this function basically forces several computationally-intensive checks of the map to re-occur on the next map update. In particular, we use it to make sure that each event is updated to the proper page when our NPC-Avatar ID variable is set.

This code requires the the function moveNPCRandom; for now, just add this prototype somewhere in the Game_Map class:

  #For now, just non-random is ok
  def moveNPCRandom(npc, xMin, yMin, width, height)
    for y in (yMin..yMin+height)
      for x in (xMin..xMin+width) 
        if events_xy(x, y).empty?
          npc.moveto(x, y)
          return
        end
      end
    end

    #If all else fails, just place it in the middle (there will be a conflict)
    npc.moveto(xMin+width/2, yMin+height/2)
  end

Re-using existing NPCs is not the best way of doing things; unfortunately, the Event class is internal to RPG Maker and cannot (easily) be changed. For our proof-of-concept, however, it will work fine.

Before we talk about the variables @net_npc_start and so on, let’s add the placeholder NPCs to the map. Create five npcs and place them somewhere near the top of the map, like so:

Put our five dummy NPCs out of range. Make sure they have increasing IDs.

You must create these NPCs in order, so that they have increasing IDs. Confirm that they do, indeed, have increasing events IDs (they should be named Event 006, Event 007, etc.). The first NPC should be set up in this way:

• Page 1 has no graphic, and is below the hero. (You might add a graphic for debugging purposes, just so you can confirm when this NPC moves.)
• Page 2 checks if variable 10 (called NPC1-Avatar ID) is 1 or above. It is a random-walk NPC at the same level as the hero, and has the first hero’s walkabout as its graphic. Its event code consists of two lines:
  • Set variable 9 (called CurrTalk NPC) to be equal to 6. (Or, whatever your NPC’s id is).
  • Call Common Event 4 (called Show NPC Message)
• Pages 3 through 6 are just like page 2, except all the red text is incremented by one. In other words, there is a page for each hero graphic.

The first NPC should look something like this, on the last page:

The last page of the first NPC in our series of dummy NPCs.

Now, alter the other NPCs, except increment the blue text by one. You’re basically creating a new NPC for each character in the virtual world, with a variable to change its appearance. The common event Show NPC Message will be used to centralize all communication.

Finally, go back to your Game_Map code, and put the following in the initialize method, directly before the call to create_vehicles:

  @max_network_npcs = 5 #How many max?
  @net_npc_start = 6 #Start at ID 6
  @net_npc_picvar_start = 10 #Start at ID 10

You might have to change @net_npc_start if your first NPC’s ID was something other than 6.

All that’s left is to hook up $game_network.connect() and the disconnect function. Open the database (F9) and go to the Common Events tab. Go to Common Event 1, “Connect to a server” menu, and add the following to the existing “Choice” command.

• For “When \N[7]”, add the script:

    npcName = $game_actors[7].name
    $game_network.connect npcName

• For “When \N[8]”, add the script:

    npcName = $game_actors[8].name
    $game_network.connect npcName

Now, go to the network NPC you created earlier, and add a new page with the same graphic. For this page’s “conditions”, select “Switch 2 (Network ON) is ON”. Then, add the following event commands:

Our "Connect" NPC will ask us to disconnect if already connected.

Described briefly:

• The NPC will ask if you want to disconnect. If you say “Yes”, he will call $game_network.ask_to_disconnect.

At this point, our prototype is complete. The only thing missing is the server. Our server will be written in Java, and it only performs three important functions:

• Track connection requests and responses, as well as disconnections and timeouts.
• Maintain a list of all online users and their avatars.
• Perform required bookkeeping; making sure that users receive all relevant messages.

Here’s the code for this in Java:

import java.io.*;
import java.net.*;
import java.util.*;
public class VXServer implements Runnable {
  //Data
  private int port;
  private Hashtable<Integer, GameConnection> players;

  public VXServer(String port) {
    this.port = Integer.parseInt(port);
    this.players = new Hashtable<Integer, GameConnection>();
  }

  //Get the next available ID
  private int getNextID() {
    for (int nextID=1; ;nextID++) {
      if (!players.containsKey(nextID))
        return nextID;
    }
  }
  //Repeatedly accept new connections
  public void run() {
    try {
      ServerSocket mainSocket = new ServerSocket(port);
      System.out.println("Server started on local port: " + port);
      for ( ; ; ) {
        Socket childSocket = mainSocket.accept();
        int id = getNextID();
        System.out.println("New player connected; given id: " + id);
        GameConnection gc = new GameConnection(childSocket, id);
        synchronized (players) {
          players.put(id, gc);
          gc.startThreads();
        }
      }
    } catch (IOException ex) {
      System.out.println("Main socket error: " + ex.toString());
      ex.printStackTrace();
    }
  }
  public void removeClient(String reason, int id) {
    String msg = "UDIS:" + id;
    GameConnection remPlayer = null;
    synchronized(players) {
      //Remove from hash
      System.out.println("Closing client[" + id + "] - " + reason);
      remPlayer = players.remove(id);
      //Send a message to all other players
      for (int x : players.keySet()) {
        players.get(x).sendMessage(msg);
      }
    }
    //Disconnect
    remPlayer.stopThreads();
  }

  //Main method; processing starts here
  public static void main(String[] args) {
    if (args.length<1)
      new Thread(new VXServer("7689")).start();
    else
      new Thread(new VXServer(args[0])).start();
  }

  //////////////////
  //Helper class
  //////////////////
  class GameConnection {
    //Data
    public int id;
    public int avatarID;
    private Socket s;
    private boolean active;
    private BufferedReader readStream;
    private BufferedWriter writeStream;
    private Thread readFromClient;
    private Thread writeToClient;
    private ArrayList<String> pendingToSend;
    public GameConnection(Socket s, int id) {
      this.s = s;
      this.id = id;
      this.avatarID = avatarID;
      try {
        this.readStream = new BufferedReader(new InputStreamReader(s.getInputStream()));
        this.writeStream = new BufferedWriter(new OutputStreamWriter(s.getOutputStream()));
      } catch (IOException ex) {
        throw new RuntimeException(ex);
      }
      this.pendingToSend = new ArrayList<String>();
      this.readFromClient = new Thread() {
        public void run() { while(active) { readLoop(); } }
      };
      this.writeToClient = new Thread() {
        public void run() { while(active) { writeLoop(); } }
      };
    }
    public void startThreads() {
      this.active = true;
      this.readFromClient.start();
      this.writeToClient.start();
    }
    public void stopThreads() {
      this.active = false;
      try {
        this.readFromClient.join();
        this.writeToClient.join();
      } catch (InterruptedException ex) {}
      try { this.s.close(); } catch (IOException ex) {}
    }
    public void sendMessage(String msg) {
      synchronized(pendingToSend) {
        pendingToSend.add(msg);
      }
    }
    private void writeLoop() {
      //Wait for a message (messages are NON_null terminated)
      String nextMsg = null;
      for ( ;nextMsg==null; ) {
        synchronized(pendingToSend) {
          if (!pendingToSend.isEmpty())
            nextMsg = pendingToSend.remove(0).trim() + "\n";
        }
        if (nextMsg==null) { try { Thread.sleep(10); } catch (InterruptedException ex) {} }
      }
      //TEMP
      System.out.println("Sent message: " + nextMsg.trim());

      //Send message
      try {
        writeStream.write(nextMsg, 0, nextMsg.length());
        writeStream.flush();
      } catch (IOException ex) {
        removeClient(ex.toString(), this.id);
        return;
      }
      //Quit?
      String[] segments = nextMsg.split(":");
      if (segments[0]=="UDIS" && Integer.parseInt(segments[3])==this.id)
        removeClient("Client requested disconnect.", this.id);
    }

    private void readLoop() {
      for ( ; ; ) {
        //Read a message
        String message = "";
        try {
          message = readStream.readLine();
        } catch (IOException ex) {
          //Close the server
          removeClient(ex.toString(), this.id);
          break;
        }
        //TEMP:Debug:
        System.out.println("Read message: " + message.trim());

        //Split
        String[] segments = message.split(":");

        //React to the message
        if (segments[0].equals("CREQ")) { //Connect Request
          //Save Avatar
          this.avatarID = Integer.parseInt(segments[1]);

          //Send this player a connect_response with his ID
          String msg = "CRES:" + this.id;
          this.sendMessage(msg);
          //Send this client's avatar ID to everyone ELSE
          msg = "UCON:" + this.id + ":" + this.avatarID;
          synchronized(players) {
            for (int x : players.keySet()) {
              if (x!=this.id)
                players.get(x).sendMessage(msg);
            }
          }
          //Send to THIS player all previously-connected players.
          synchronized(players) {
            for (int x : players.keySet()) {
              if (x==this.id || players.get(x).avatarID==0) //Might not have been initialized yet.
                continue;
              msg = "UCON:" + players.get(x).id + ":" + players.get(x).avatarID;
              this.sendMessage(msg);
            }
          }
        } else if (segments[0].equals("UDIS")) { //(Request for) User Disconnect
          //Can only disconnect self
          int newid = Integer.parseInt(segments[1]);
          if (newid == this.id) {
            //Send a disconnect message to this player
            // We close the connection after sending our own message
            String msg = "UDIS:" + this.id;
            this.sendMessage(msg);
          }
        }
      }
    }
  }
}

At over 200 lines, the server code is somewhat long, but that’s usually what ends up happening with Java. Our server functions in a relatively simple manner. At first, there is only one thread, containing the ServerSocket. This is basically the “connection” class, which will constantly wait for connections from our clients. Each time a connection is detected, the main thread creates a small helper class with the socket it just opened. This class spawns two more threads, one for reading from the socket and the other for writing to it. Thus, the total number of threads on the server at any one time is equal to 2 X number_of_clients + 1. There are better ways of doing this (e.g., using the NIO library), but they would require us to modify our client (RPG Maker) code in ways that are somewhat difficult.

Anyway, once the server is running, each client thread simply reads and writes messages as one would expect. Because the code for our server is contained within one file, we can access most functions without requiring any Interfacing hacks. All in all, it’s a rather tiny server.

Compile your server (Ctrl+1 in TextPad), and then run it (Ctrl+2). It will use the default port (7689) on localhost (127.0.0.1). Open RPG Maker and run your game. Talk to the Network NPC and select the first connection (assuming you didn’t modify it). Switch back to the server window; hooray! you’re connected.

Our server window has messages from both players.

Run another game, and you’ll see your player there. Disconnect and connect as many clients as you like. However, for now I recommend disconnecting by closing the game client; for some reason, using the NPC to disconnect will occasionally crash the server. (This is undoubtedly a flaw in my server code, not a bug with networking in RPG Maker games.)

Our NPCs connect, and they walk around randomly.

Everything works just fine; when you close the second window, its corresponding NPC will disappear from the first’s window. Isn’t that amazing?

Game Prototype 3 – Chatting with Other Players

We could stop here; our point has been proven. However, Step 2 was particularly time-consuming, and I don’t want you to think that network games are so tough. In fact, with the simple infrastructure we’ve developed for part 2, we can easily add the remaining functionality in a simple, modular way. (Of course, we’ll eventually reach the point where the network becomes saturated with messages. At that point, it will become very difficult to proceed, because we’ll have to start talking about interest management techniques.)

Let’s add client messages as a kind of “poor man’s chat” interface. Surprisingly, the networking is now the easiest part:

CHAT_MESSAGE	Description
Type	The string CMSG
Player ID	The ID of the player sending the message.
Message	The message string.

We will use the same network message to send a message request to the server, and to receive a message from the server (on behalf of another player). Add the following to Game_NetMessage:

  def self.CHAT_MESSAGE
    "CMSG" 
  end

And:

  def self.Make_ChatMessage(playerNID, msg)
    "#{Game_NetMessage.CHAT_MESSAGE}:#{playerNID}:#{msg}\n"
  end

Add the following to Game_Network:

  def sendChatMessage(msg)
    #Note that this MIGHT occur if we haven't heard back from the server 
    # yet. We might have to manually police this case later.
    raise "Cannot send message; you are not connected!" if @my_id == -1
    toSend = Game_NetMessage.Make_ChatMessage(@my_id, msg)
    $game_shared.queueClientMessage(toSend)
  end

Add the relevant message handling code to Game_Map, in the update_network function, right after the code for handling USER_DISCONNECT.

      elsif segments[0]==Game_NetMessage.CHAT_MESSAGE
        #Retrieve the message
        cmsg = segments[2]

        #Save the player's message
        avtID = segments[1].to_i - 1
        @events[@net_npc_start + avtID].chat_message = cmsg

        #Show an emotion icon
        @events[@net_npc_start + avtID].balloon_id = 1
      end  

This, of course, requires a new property to be added to Game_Event. Add the following to the Game_Event class, below the other attr_readers

  attr_accessor :chat_message             # Current message from other user

…and in the initialize method:

  @chat_message = ""

Thus, each NPC now has a property, chat_message, which is updated whenever a network message of type CHAT_MESSAGE. From here, the code for the Show NPC Message Common Event should be obvious. First, hit F9 and go to the Actors tab. Press “Change Maximum…” and enter a number like 15. Then, name NPCs 10, 11, 12, and 13 to CurrNPCTxt Line 1, CurrNPCTxt Line 2, CurrNPCTxt Line 3, and CurrNPCTxt Line 4. Now, edit the Show NPC Message Common Event:

Despite its length; this common event is actually quite simplistic in nature.

This is one of those “exploding events” that PRG Maker famously espouses. Briefly described:

• We first run a simple script that takes the current NPCs chat_message and splits it whenever there’s a tab (\t). (We can’t split on newlines because newlines denote new messages in our network protocol.) The reason our script is so complex is so that we can handle cases such as the empty string (“no message”) or a string with less than 3 tabs (messages with less than 4 lines).

   #What NPC is talking? What's he saying?
   currMsg = $game_map.chat $game_variables[9]
  
   #Set NPCs 10..13 to be the lines of text
   lines = currMsg.split("\t", -1)
   rep = lines.empty? ? 4 : 4-lines.length
   lines += [""]*rep
   for id in (0..3)
    $game_actors[10+id].name = lines[id]
   end

• Next, we run a much smaller script with sets the value of variable 8 (which you can rename to Curr NPC Avatar) to be equal to value of the avatar variable for the given NPC. I thought this kind of functionality was offered by RPG Maker’s standard events, but I couldn’t find it anywhere.

   #Set variable 8 to the current NPC's avatar
   index = $game_variables[9] - 6 + 10
   $game_variables[8] = $game_variables[index]

• Next, we have a big mess of If statements. Each “if” statement wraps the following text box command:

     \N[10]
     \N[11]
     \N[12]
     \N[13]

• …this uses the values we just stored in the Actors’ names to spell our out message. The only difference between each text box in the if statements is their face picture; we change this based on the value of variable 8 (Curr NPC Avatar).

The first script in our event will require the following method to be added to Game_Map:

  #Helper
  def chat(npcID)
    msg = @events[npcID].chat_message
    msg ? msg : ""
  end

The name was kept short to prevent overflowing the tiny script box inside RPG Maker’s Script Event Command. The reason we check for nil values and replace them with the empty string is that some of the functions that call chat will expect strings to be returned.

So now, messages work –presuming that we can actually send one. The problem is, text entry in RPG Maker is notoriously obnoxious. We could use the “Hero Name” dialog to spell out the sentence word-by-word, and then display the partial sentence like we did with the IP dialog. This would be clumsy at best. We could provide a set of partial phrases, and allow the player to select lines 1 though 4. This would actually be a minor pain too. From the user’s point of view it would be simplest to make our own Scene_* class that traps key events and lets them just type out the sentence. But that could easily double the length of our simple prototype. Since the problem cannot be solved simplistically, I’ll just punt on it: we’ll give the player a set of four predefined messages. First, add the following event command to the end of Common Event 4, Show NPC Message:

   Call Common Event 5

Name Common Event 5 something like Respond to NPC and give it the following code:

Event that allows a player to communicate with everyone else.

This Common Event is easy to understand, although it is clearly a too cold type:

• Ask the player if he wants to respond to that NPC.
• If so, let them choose a simple (one-line) message, a multi-line message, or just a glitchy message (to test what happens). I know, boring, huh?
• Then, simply use the function we put in $game_network to send the message.

    msg = "Hi there, friend!"
    $game_network.sendChatMessage(msg)

Pretty simple, although I think it’s a bit of a bad abstraction (since we talk to a single NPC to change our message to all NPCs. But that’s a minor point; we could change this easily later if we wanted to. Anyway, all that’s left is our server code. In the function readLoop, add the following case:

        } else if (segments[0].equals("CMSG")) { //Send a chat message to all clients
          //Can only send messages from yourself
          int fromID = Integer.parseInt(segments[1]);
          if (fromID == this.id) {
            //Send the same message to all other users
            synchronized(players) {
              players.get(fromID).lastMsg = segments[2];
              for (int x : players.keySet()) {
                if (x!=this.id)
                  players.get(x).sendMessage(message);
              }
            }
          }
        }

You’ll have to change the definition of GameConnection to include this “last known message” parameter:

    public String lastMsg;

…and, in the constructor:

      this.lastMsg = "";

You will, of course, need to add a similar broadcast to the UCON case. We can modify the last use of synchronized(players) to include this:

          //Send to THIS player all previously-connected players and their messages
          synchronized(players) {
            for (int x : players.keySet()) {
              if (x==this.id || players.get(x).avatarID==0) //Might not have been initialized yet.
                continue;
              msg = "UCON:" + players.get(x).id + ":" + players.get(x).avatarID;
              this.sendMessage(msg);
              msg = "CMSG:" + players.get(x).id + ":" + players.get(x).lastMsg;
              this.sendMessage(msg);
            }
          }

I’ve named this server VXServer2, and included its full listing in Code Segment 3, below, in case you’re not familiar enough with Java to insert the code yourself.

Code Segment 3 — Make sure this is in a file called VXServer2.java, or it won’t compile.

Anyway, compile it (Ctrl+1) and run it (Ctrl+2) then try out our nifty message code.

It looks like your friend has something to say to you.

The exclamation also appears at potentially inappropriate times, like when the NPC first logs on, but I don’t think that’s such a bad thing –it helps by alerting the player that a new NPC has connected.

A message which, if not user-defined, is at least user-chosen.

The server scales nicely, although if you’re running these examples locally, you’ll be limited by the fact that RPG Maker “pauses” all game windows except the currently-active one (so messages won’t be received). Nonetheless, the effect is impressive:

Several NPCs; we are talking to one while another indicates that he has a new message for us. Pretty cool.

Project

For the first time, I’ve decided to release the final project’s code as a sample. Here’s a zip containing everything you’ll need; make sure you read the article to understand how it works. As always, all code is in the public domain, unless it is borrowed from another author (in which case I have noted it) and is thus released under the license chosen by the respective author. All such licenses are open-source in nature.

At the moment, I cannot find a good site to host the project file. However, I posted the sample code on the RPG Maker VX Forums; you can download it from here (but you’ll need an account). If I can find a good hosting site, I’ll upload a direct link. Apologies for the inconvenience, readers (but it’s a nice forum anyway).

Further Work

I’m personally quite pleased with what we’ve accomplished; however, it’s by no means polished. There’s still bugs in the system, and there’s a great many improvements that could be made. Here’s some “homework” for you; I might get around to writing a post about one of these eventually, but for now I’m tired of writing server code.

• BUGFIX: Send a “glitch” message (option 3) and then try to read it with another player. Surprisingly, the client will crash! This was actually unintentional on my part, but it presents a nice, easy exercise for you. Figure out where the system is glitching up: server side, client side, or somewhere in between. Then fix it!
• BUGFIX: For some reason, closing the network connection from the NPC might crash the server. Also, hitting “F12” to go back to the title screen might have a negative effect on the connection. Re-write parts of the server and client to be more robust at handling this type of thing.
• FEATURE: Once connected, we should measure the average lag to the server, and print this in the lower-right corner of the screen. Perhaps use the Debug Window to design some kind of “connected” box which always hovers in the lower-right corner. Bonus points if you animate it with a spinning Internet icon.
• FEATURE: Combine this article’s source with the NPC Talk Boxes one, and make NPC messages appear above their heads whenever they change. You can still show the exclamation emote when they first connect.
• FEATURE: Our NPC code uses “sir” a lot, even if the connected player is clearly a female. Extend the game engine to handle strings like \sir, which expand into “sir” if the player is male and “ma’am” if the player is female.
• FEATURE: Add user accounts and passwords, and do this in a way that keeps users’ passwords safe and un-crackable –a salted hash comes to mind.
• PROJECT: Allow players to fight each other. This isn’t as difficult as it seems, since RPG Maker’s battle system is turn-based. You can extend Scene_Battle to expect network input and wait until both teams have made their decisions. Then, you might add an icon to show if the other player is “ready”, “disconnected”, etc. Finally, you’ll have to take the opposing player’s “stats” and somehow turn them into monsters which you place on the opponent’s screen.
• PROJECT: Assuming the previous project has been completed, add the ability for players to form “teams” and fight against each other at the party level. You can update your heads-up-display to show the moves your teammates have chosen, and you’ll probably want a team-chat and battle-chat feature (with real-time typing, not RPG Maker’s name entry dialogue). I’m giving this project a gold star, since this is the project I would most like to do myself.
• PROJECT: Extend our code to handle events arbitrarily. In other words, make it so that we can do something like $game_map.add_npc(Event(3, 5)) to create a new NPC at location (3,5). This is harder than it sounds, and really only makes sense when preparing for the next item.
• PROJECT: Track NPC movement in real-time. You might have to move our update_network code to a more prominent location –say, the place where RPG Maker handles user input. Regardless, the key challenge with this project is network congestion; sending 60 messages per NPC per second is a sure way to bog down your server. In other words, only update what’s necessary. You might, for example, send a USER_LEFT message when the player wants to move left. The server can then determine if moving left is allowed, and send back OK_TOGO_LEFT or NOGO_LEFT to inform the player. However, this will introduce some noticeable lag on the client side, so you might want to allow the client to begin moving left anyway, and just teleport him back if NOGO_LEFT was received. A careful consideration of when to attempt a move before receiving a response from the server is difficult, and will require extensive testing. On that note, you’ll have to test this code from multiple computers, since (1) there’s no lag otherwise and (2) RPG Maker pauses all but the top-most window. In fact, this pausing is a problem, since it interrupts the flow of messages. You’ll need to account for that somehow (perhaps force the game to remain full-screen? Or install a Windows hook that tricks RPG Maker games into thinking they’re always a top-level window?). Either way, this project will easily take weeks of your time to get right.

Happy coding!

Update: Phoenix Demo

Speed used the code in this post (along with tons of other cool scripts, including a mouse capture script!) to make a really awesome demo called Phoenix. Check it out here.

Some screenshots:

My character, loaded into the main game world.

Loading of a second character, tracking down the first.

New features added by Speed include:

• Characters are tracked by x/y coordinates, and across various maps.
• Full-featured chatting, account creation, password storage, etc. (He saves data in Text files).

…plus a lot more. Pretty cool to see my code in action. And now, back to listening to OC Remixes.

[EL-2] Eternal Lands Tray Notifier – Part 1

Blog Info: I finally did it: I took on a task that was too big for me to finish. Fortunately, I managed to get its basic functionality all done by press time. If you can follow along, you’ll know more than enough to continue on your own from here.

Quick Note: I’m still trying to fix some of the formatting, but the content’s all here, so I decided just to post it. I’ll re-tab the untabbed code later.

Back-story: Our previous notifier was pretty hackneyed. It was also heavily coupled with the client program’s source, which might not be such a good idea. (What if the developer released the next version in Haskell or Forth? What if he closed the client source for the next release? What if —and this is the most likely— we don’t want to make our notifier any fancier using Win32 C++?) We need a new solution, and that means it’s time to pull out the big guns.

Goal: Write a new notifier plugin. Make it operate external to the Eternal Lands program. Don’t slow down the client or the server. Don’t break any rules. Allow multiple types of notifications. Make it flashy.

Licensing Notice

This post provides a lot of code. While I normally provide all code in the public domain, I am restricting this post’s code with a license. Scroll to the bottom of the post to read it. This license is not for my benefit, but for the benefit of the EL servers and admins. Basically, it prohibits you from using this code to automate, or to modify it in any way that causes undue strain on the servers. You cannot inject events, nor can you delay sending them between the client and server. Finally, you cannot modify the thread priority of the event handler. The EL team has volunteered their time and creativity for your benefit, and I won’t have my code used to cause trouble for them.

Step 1: General Approach

Most of the problems in our last notifier stemmed from hastiness. In light of that, this section will take it slow and explain why we chose to do things the way we did. We’ll discuss some alternative approaches. But first, a general overview of what we want, and why our last plugin is no longer sufficient:

• Event Notification — We want to see a message when something happens in Eternal Lands. Our initial program tagged harvest completion, but that was all. (Bonus question: How can you stop harvesting without displaying the “You stopped harvesting” message? It’s a tricky one… if you’re not sure, try harvesting some Seridium until you run out of Matter Essences.)
• Notification Timeliness — We want to show notifications only when necessary, and hide them when they’re not needed anymore. The previous plugin showed them if the Eternal Lands window was not the foreground window, and hid them when a new message came in. Later, we found this to be unsatisfactory. Better to hide all messages when the EL window gains focus.
• Variety of Notifications — Seeing a non-descript “You stopped harvesting” is different from seeing a shiny “You found an Enrichment Stone!” or a red “You have died and gone to the underworld”. We’ll add multiple types of notifications, which will necessitate allowing them to stack, fade in, and fade out. If you guessed that we won’t be doing all this graphical work in Win32 C++, you guessed right!
• Separation from Client Code — We won’t be using C++, nor will we be building directly on top of the Eternal Lands source. Rather, we’ll build a separate module that will load the EL client, and then filter all messages sent to it. We also want a system tray icon, and we might even consider removing the Eternal Lands window from the taskbar entirely.
• Fast and Extensible — We want the code to be fast (sorry, Perl). We want it to be easy to add new filters (sorry, C). We want it to operate directly on the network data (sorry, Forth) and have a great graphics library. If it’s native to Windows, all the better. Our choice is C#.

Why not just read the client log file? If you browse to “C:\Users\User Name\Documents\Eternal Lands\main\chat_log.txt” —possibly replacing “main” with the name of the server you plan on, and “Documents” with “My Documents” on Windows 7— you’ll find a nice log of (most of) the chat window. Fiddle a bit more, and you’ll find that it’s updated in real time. Why, then, do we not simply tail this file? In Windows, this is pretty easy to do in Java:

public void run() {
  File f = new File("C:\\ ... \\chat_long.txt");
  long fPointer = ;/* Seek to the end of the file */
  for (;;) {
    Thread.sleep(100);
    long len = f.length();
    if (len < fPointer)  //Log was reset
      fPointer = len;
    else if (len > fPointer) { //File was appended to
      RandomAccessFile raf = new RandomAccessFile(f, "r");
      raf.seek(fPointer);
      String line = null;
      while ((line = raf.readLine()) != null)
      processMessage(line);
      fPointer = raf.getFilePointer();
      raf.close();
    }
  }
}

Don’t use this code; it doesn’t check for exceptions, and it isn’t mine (it’s part of the java-tail project). More importantly, it doesn’t really work if the file is being constantly added to. You could re-read the file after exceptions, but then you’re entering into the messy possibility of reading the same line twice.

On a totally different level, this is sub-optimal because the client log doesn’t contain everything. What if you wanted to write a filter that tells you if you’ve been interrupted from harvesting by a PKer instead of a sub-lethal cave-in? Battles don’t send messages to the console, unless you die. Wouldn’t you rather find out while you still have a fighting chance?

Why not use Java? This question applies to any other language, but I feel the most reasonable alternative is Java. (I consider Visual Basic .NET to be identical to C# for all such discussions, by the way.) Java is a very good language, and there is no reason not to use it. Your favorite language is also probably a good choice, although I might not have heard of it. My reason for choosing C# was that, of all the candidates I considered, it was capable of (read: “I knew how to use it to”) solving all the sub-tasks I needed. Feel free to follow along in Python or Ruby or Tcl/Tk, converting the code as you like. Ooh, or Delphi; I like Delphi. Regardless, I’m doing this in C#, and any intermediate programmer should be able to follow along.

Why not patch the client code? My earlier reasons for not simply extending on our earlier project were true, but also somewhat motivation by the fact that I knew I was going to use a much higher-level (and thus incompatible) language from the start. What if you started this project using C++? In that case, I’d still encourage you to keep your code totally separate. Besides all the previously-listed reasons, here’s one more: bugs. If you’re recompiling the entire Eternal Lands code base with each new notification, then you’re bound to slip up from time to time. This is part of development; you can release bug-free software, but your internal builds will be fraught with bugs. Now, given the nature of bugs in C++, this means that your code could easily corrupt some of Entropy’s code. If this results in sending a million messages a second to the server, “ya might get banned”. That said, if your bug is passive, and other people use your plugin, you start to run a real risk of bringing down the server. And even if your bug is harmless, it will still confuse developers trying to nail it down. Trust me when I say that tracking a bug to third-party patches does not make developers happy.

How will we catch messages? We’ll start our program first, which should give it priority over any child processes and their system resources. Then, we’ll connect our program to the Eternal Lands server, and connect the EL Client to our program (by adding a new server to the config file). Finally, as we ferry messages between client and server, we’ll send them off to a separate, low-priority thread where we can match then against an arbitrary number of filters, spawning pop-up windows as we identify messages of interest.

Step 2: Connecting to the client

The first thing we need to do it connect to the server. From the Eternal Lands server config file (D:\Programs\Eternal Lands\servers.lst), we know that the “main” server connects to port 2000 on game.eternal-lands.com. Open up Visual Studio (the free Express Edition for C# is pretty nice) and start a new project. Make it a “Windows Forms Application”. You’ll be presented with the design view for “Form1.cs”. In case you aren’t familiar with Visual Studio, take a minute to note the toolbox (circled in red on the left) and the Solution explorer on the right. Right click on Form1.cs in the solution explorer and note the two options shown circled in blue. “View Designer” will bring up the form designer, like you see now. “View Code” will bring up the code that runs everything behind the scenes. Note that C# supports “partial” classes, so the “View Code” option will not show you any code generated by Visual Studio, unless you are running a very, very old version of Visual Studio.

Visual Studio's "Hiding Tool Bars"

Open the toolbox (by clicking on it) and drag a “Text Box” control onto your Form. (This is under the “Common Controls” group). Now, right-click on this box and choose “Properties”. The most important property is called “(Name)”, because this is what we’ll use to identify the control in our C# code. The following table details which controls to add, and what properties to set for them. All other properties can be left at their defaults, or you can change them to your liking. For example, you might change the “font” for your controls, or the “BorderStyle”. Go for it.

TextBox*
(Name)	txtPathToELFolder
Text	C:\Program Files\Eternal Lands

*Note: If you installed Eternal Lands to a different location, set the Text property accordingly. Escaped backslashes are not necessary when entering this string in the visual editor.

Button
(Name)	btnRead
Text	Read

ComboBox
(Name)	cmbServers
Enabled	False

Button
(Name)	btnConnect
Text	Connect
Enabled	False

Label
(Name)	lblZero
Text	0
TextAlign	MiddleRight
Font	Courier New, 14.25pt, style=Bold
ForeColor	DarkRed
AutoSize	False
Size	346, 399

TextBox
(Name)	txtConsole
ReadOnly	True
Multiline	True
Size	306, 188

Arrange these components like so:

Line Up Your Components Roughly Like This

A quick note on Hungarian Notation: in general, I do not support this hackneyed way of naming variables, especially in a strongly-typed language like C#. However, for GUI components, I often find myself saying “Ah, what was the name of that button I wanted?” By naming controls in an ordered fashion, I can type btn<Space> and get some help from Visual Studio’s auto-complete feature. Having large numbers of (essentially) global variables for your form controls is unavoidable in C#, and I find this approach scales really well, despite the frown that most people get when they see Hungarian Notation.

Compile your program (Build –> Build Solution) and then run it (Debug –> Start Without Debugging). None of the software logic is hooked up yet, but the general idea is that the user can change the client directory, then click “Read” to read the list of available servers. Next, the user can choose a server to connect to from the combo box, and click “Connect”. Log messages will be shown in our multi-line Text Box control, and the mysterious “Zero Count” will be used for something which will be explained later. Well, let’s start the magic.

Click on the “Read” button, and go to its properties. Notice that little lightning bolt, circled below in red? If you click that, you will switch to “Event” triggers. Click the button left of it to go back to properties view.

Bad Design Puts This Very Important Button in a Very Obscure Place.

Click the event button, find the event called Click, and type “ReadELFiles” in the box. Then, press “Enter”. Visual Studio will bring you to Form1.cs’s Code View, with a skeleton implementation of the ReadELFiles function used for your Click event delegate. Java users will be saying “What?”, as delegates and events make little sense from a minimalist point of view. We’ll have a good example of the power of delegates later.

Here’s the implementation of ReadELFiles:

private void ReadELFiles(object sender, EventArgs e)
{
  //Reload an entirely new list of servers.
  cmbServers.Items.Clear();

  //Don’t load anything if we specified an invalid directory.
  if (!File.Exists(txtPathToELFolder.Text + "\\" + "el.exe"))
  {
    txtConsole.Text = "Error: el.exe doesn't exist in EL folder";
    return;
  }
  else if (!File.Exists(txtPathToELFolder.Text + "\\" + "servers.lst"))
  {
    txtConsole.Text = "Error: servers.lst doesn't exist in EL folder";
    return;
  }

  //Open our servers file and read it.
  bool foundLoopback = false;
  StreamReader srvFile = new StreamReader(txtPathToELFolder.Text + "\\" + "servers.lst");
  while (!srvFile.EndOfStream)
  {
    String line = srvFile.ReadLine();
    if (line == null)
      break;

    //Coment? Empty?
    line = line.Trim();
    if (line.StartsWith("#") || line.Length == 0)
      continue;

    //Read:
    //ID, Config_Dir, Address, Port, Description
    String regex = "([^ \t]+)[ \t]+([^ \t]*)[ \t]+([^ \t]*)[ \t]+([^ \t]*)[ \t]+(.*)";
    Match m = Regex.Match(line, regex);
    ServerInfo sInf = new ServerInfo();
    if (m.Success && m.Groups.Count == 6)
    {
      sInf.id = m.Groups[1].Value;
      if (sInf.id.Equals("loopback"))
        foundLoopback = true;
      sInf.configDir = m.Groups[2].Value;
      sInf.address = m.Groups[3].Value;
      try
      {
        sInf.port = Int32.Parse(m.Groups[4].Value);
      }
      catch (FormatException)
      {
        //Don't add
        continue;
      }
      sInf.description = m.Groups[5].Value;
      cmbServers.Items.Add(sInf);

       //Loopback?
      if (sInf.id.Equals("loopback"))
        loopbackInfo = sInf;
    }
  }
  srvFile.Close();

  //Do we need to add our own local server?
  if (!foundLoopback)
  {
    //Copy existing server file
    StreamReader inFile = new StreamReader(txtPathToELFolder.Text + "\\" + "servers.lst");
    StreamWriter outFile = new StreamWriter(txtPathToELFolder.Text + "\\" + "servers.lst.new");
    for (String line = inFile.ReadLine(); line != null; line = inFile.ReadLine())
      outFile.WriteLine(line);
    inFile.Close();

    //Add our own category
    outFile.WriteLine(srvLine);

    //Close, swap, notify.
    outFile.Close();
    File.Delete(txtPathToELFolder.Text + "\\" + "servers.lst");
    File.Move(txtPathToELFolder.Text + "\\" + "servers.lst.new", txtPathToELFolder.Text + "\\" + "servers.lst");
    ShowConnectMsg("Created loopback entry.");

    //Add a new item to the list
    loopbackInfo = new ServerInfo();
    loopbackInfo.id = "loopback";
    loopbackInfo.configDir = "main";
    loopbackInfo.address = SERVER_ADDR;
    loopbackInfo.port = SERVER_PORT;
    cmbServers.Items.Add(loopbackInfo);
  }

  if (cmbServers.Items.Count > 0)
    cmbServers.SelectedIndex = 0;
}

Make sure you add the following using directives to the top of Form1.cs, otherwise you’ll get errors stating that “The name <something> does not exist in the current context”. You also won’t get auto-complete functionality until the proper directives are added. (Auto-imports is something I miss dreadfully from Eclipse.)

  using System.Text.RegularExpressions;
  using System.IO;

You will also need the following variables; add them as part of the class (e.g., one line after the opening brace following “public partial class Form1 : Form”)

public const string SERVER_ADDR = "127.0.0.1";
public const string SERVER_PORT = 5421;
private string srvLine = "127.0.0.1";"loopback        main            " + SERVER_ADDR
      + "               " + SERVER_PORT + "    Loopback adapter for use with the EL Notifier";
private ServerInfo loopbackInfo;

We also need this “ServerInfo” structure everyone’s been talking about. Add it as part of the namespace (not the class Form1).

public struct ServerInfo
{
  public String id;
  public String configDir;
  public String address;
  public int port;
  public String description;

  public override String ToString()
  {
    return description;
  }
}

The “ToString()” method is called by the Combo Box control to display the Object it contains; overriding it allows us to dictate that ServerInfo items are displayed by the description. There are better (but less concise) ways of controlling Combo Box output.

Note that structs in C# are very different than structs in C++. Indeed, you could have used a class here; the only difference would have been in its copy semantics, and possibly its memory footprint.

The only thing you’re missing is the ShowConnectMsg delegate, but for now just assume it’s a standard function call that does some non-essential logging. Let’s dissect the code we just wrote!

The SERVER_ADDR and SERVER_PORT variables help us to set up our program as a middleware between the EL Client and the EL Server. Our program has both client and server aspects; these two variables define access to the latter. We use srvLine to shim the servers list so that it can access our “Loopback” interface. Note that srvLine is effectively constant, even though it is not declared that way.

We are basically equivocating on the program’s socket expectations. From Wikipedia:

An Internet socket is characterized by a unique combination of the following:

• Protocol (TCP, UDP or raw IP).
• Local socket address (Local IP address and port number)
• Remote socket address (Only for established TCP sockets.)

The original connection from EL Client to EL Server has one socket: a TCP-enabled connection from 127.0.0.1:???? to game.eternal-lands.com:2000. This socket is bi-directional. We don’t know the local port, because local ports are almost always randomly assigned. We don’t technically know the server port either, but 2000 is what we connect to, so I’ll list that here.

Our shimmed middleware creates two bi-directional sockets:

TCP, 127.0.0.1:???? ? SERVER_ADDR:SERVER_PORT
TCP, SERVER_ADDR:???? ? game.eternal-lands.com:2000

Note that, as per the usual, we don’t know the local ports. Either way, we are guaranteed that they will be unique.

All that explanation for just three lines of code? Well, if you’re a non-network guy like me, figuring this out is one of the hardest parts of this project. Now, on to the rest of the code:

The ReadELFiles function is pretty easy to follow:

• Note that the function parameters “sender” and “e” are not used in our function. These are mandated by the .NET platform, but are not needed for our simple example.
• The foundLoopback variable is used to check each line read from servers.lst. If the id of that line is “loopback”, then we know a loopback entry exists. If, at the end of scanning, no such entry was found, we just add it to the end of the file. That way, when we start Eternal Lands, we can connect to the loopback server (instead of main or pk or testing) and fool the client into thinking it’s connecting to the real server, instead of our fake, localhost middleware.
• Note that we should check the loopback entry to ensure that its address and port parameters match ours. I’ll leave this as an exercise in regular expressions.
• Some regex pattern matching is used to filter each entry in our servers.lst file. We use regexes to check if a line matches a specific pattern, and we then use the Groups property to extract what was matched. If you don’t understand pattern matching, just skip the details. Learn it later, of course —it’s a fantastic way of dealing with text in almost any modern language.

We will now add the final missing piece of the puzzle. Add the following to the namespace (not the Form1 class):

public delegate void ShowConnectMsgDel(String msg);

Then, add the following function definition to the Form1 class:

private void ShowConnectMsg(String msg)
{
  Control ctl = txtConsole;
  if (ctl.InvokeRequired)
    ctl.Invoke(new ShowConnectMsgDel(ShowConnectMsg), msg);
  else
    ctl.Text = msg;
}

Our program will now compile. (I’ll explain delegates later, when it makes more sense to.) Run it, click the “Read” button, and you will see “Main game server” appear in the still-disabled Combo Box.

A Properly-Formatted Servers File Is Parsed.

If your program doesn’t compile, have a look at Listing 1 and try to figure out where you went wrong. (Apologies for the PDF; WordPress doesn’t like .txt files.) If it doesn’t load the server list, make sure you supplied the proper path to the EL Client folder.

Now, add the following code to the end of your ReadELFiles function:

//Enable the remainder of our form
cmbServers.Enabled = true;
btnConnect.Enabled = true;

That should enable everything you need. Now, in Design mode, click on the “Connect” button and give it a click Event called ConnectToServer. You’ll be given a function skeleton again; here’s what we’ll add to it:

private void ConnectToServer(object sender, EventArgs e)
{
  //Connect to our server
  UpdateConsole("Connecting.....");
  Thread t = new Thread(new ThreadStart(Check_Server));  
  t.IsBackground = true;
  t.Start();
}

This requires the CheckServers function, which will run in a separate thread:

private void Check_Server()
{
  //Our middleware acts as a client in this case.  
  TcpClient tcpclnt = new TcpClient();
  Stream srvStream;

  //Get our currently selected server and start it
  ServerInfo sInfo = (ServerInfo)cmbServers.SelectedItem;
  tcpclnt.Connect(sInfo.address, sInfo.port);

  //Display any messages we receive.
  srvStream = tcpclnt.GetStream();
  byte[] msg = new byte[1024];
  bool done = false;
  for (; !done; )
  {
    int amtRead;
    try {
      amtRead = srvStream.Read(msg, 0, msg.Length);
    } catch (IOException) {
      UpdateConsole("\r\nIOException; closing Server");
      amtRead = 0;
    }

    if (amtRead != 0)
    {
      //Echo to console
      String line = "MSG: [";
      for (int i=0; i<amtRead; i++)
      {
        byte b = msg[i];
        if (b >= 32 && b <= 126)
          line += (char)b;
        else
          line += '.';
      }

      line += "]";
      UpdateConsole("\r\n"+line);
    }
    else
    {
      UpdateConsole("\r\nRead zero; closing Server");
      done = true;
    }
  }

  //Must close BOTH
  srvStream.Close();
  tcpclnt.Close();

  //Ready to go again?
  UpdateConsole("\r\nServer Stopped");
}

Make sure you add the following using directives to the top of your file:

using System.Threading;
using System.Net.Sockets;

After that, you’ll only get errors for the UpdateConsole function, which we’ll implement now:

private void UpdateConsole(String msg)
{
  Control ctl = txtConsole;
  if (ctl.InvokeRequired)  
    ctl.Invoke(new AddToConsoleDel(UpdateConsole), msg);
  else
    ctl.Text += msg;
}

Add the following to our namespace, and we’ll be done:

public delegate void AddToConsoleDel(String msg);

Run the program, click “Read” to get a server list, then choose the “Main game server” from your dropdown list and hit “Connect”. You should see a nascent echo of the regular log-in message, and some unreadable garbage.

The "Garbage" Is Actually the Important Information, As We'll See Later.

If this just doesn’t work, have a look at Listing 2 to see where you went wrong.

If you scan the code, you’ll find everything in order. First, we create a TCPClient object to manage our TCP connection to the server. We then Connect it to the server, based on the item chosen in the drop-down list. Finally, we continue to read up to 1kb at a time from the server. To the best of my knowledge, the Eternal Lands server never sends more than a few hundred bytes of data at once, so we don’t have to merge messages.

Note that this all has to happen in a different thread, to avoid throttling the GUI dispatch thread. If you just ran a tight loop in the main Thread, your Windows Form would become sluggish and unresponsive. The C#-ism:

  Thread t = new Thread(new ThreadStart(Function_Name));

…capitalizes on delegates to encapsulate a function (Function_Name) in a new thread. This is a reasonably good example of a delegate, but we have a much better one: our UpdateConsole function.

The problem with ThreadStart is that it’s too easy to guess what it does, without actually understanding why we choose to do it this way. Instead, let’s have a look at UpdateConsole. I’ve highlighted it differently below:

private void UpdateConsole(String msg)
{
Control ctl = txtConsole;
if (ctl.InvokeRequired)
ctl.Invoke(new AddToConsoleDel(UpdateConsole), msg);
else
ctl.Text += msg;
}

First, our variables are declared in red. One is the value we wish to update the console with (as a String) and the other is a local variable used to reference the Text Box itself. Since all Text Boxes subclass Control, we can store this variable in a generic fashion. Note that the variable ctl was created to emphasize that this technique can be used on any part of our form. We could have put ctl as a second function variable, if we had multiple consoles.

Next, the blue text. The “InvokeRequired” property (think of it like a read-only variable) is a nifty shorthand which tells us if the given control was created by the thread we are currently running. As all you GUI programmers know, most actions that update GUI components should be done in the master dispatch loop of the thread that created the GUI. If, for example, this control was created in a random thread which is now finished running, then updating it from another thread will cause all sorts of exceptions at run-time. We could dispatch a new thread for each update operation, but the InvokeRequired property helps us to avoid spawning threads when we don’t need to.

The purple text is run if an Invoke is Required. It calls the Invoke method of ctl, which adds a new event to a thread guaranteed to be valid for modifying ctl. This thread is of type AddToConsoleDel, which we defined earlier as:

delegate void AddToConsoleDel(String msg)

…in other words, as a delegated function with one argument that returns nothing (void). This matches the function signature of our UpdateConsole function. Look back at the purple text, and you’ll see that we call new AddToConsoleDel(UpdateConsole) to  create a new delegate of type AddToConsoleDel that will run the UpdateConsole function when called. The second parameter to Invoke is the argument set that we intend to pass to our AddToConsoleDel delegate. Since UpdateConsole only takes one argument, we only pass in one (msg).

Which leads us to the green text: what happens if we are running in the right thread. If this is the case, we simply append the message string to the Text Box’s existing Text property.

So, the logic of our function is simple. If the thread is capable of modifying the control directly, it just appends the text. Otherwise, it creates a new delegate to the same function, passes along the same method arguments, and asks the given control to Invoke this delegate. That ensures that next time the function is called (by the delegate this time) it has the necessary permission to modify the control. This logic is a bit tricky to catch the first time you see it, but it’s one of the main design patterns of C#. And, it has the benefit of keeping the actual logic of the method in the same place as the invoke logic of the delegate. So we can just call UpdateConsole and not have to care about whether a thread is spun off to do the dirty work.

You can think of a delegate as a pointer to a function, with enhanced type-safety, if that’s any easier for you.
Take a break, grab a coffee, you deserve a break.

Step 3: Passing Messages Along Properly

Our middleware can locate the Eternal Lands server, and it establishes a TCP connection (proven by the fact that it receives two messages in sequence, not just two copies of the same message). We also jumped the gun and added a loopback entry to the client’s list of servers, to allow us to force the client to connect to our program instead of the real server. Now, let’s put that new entry to work, and tie the client and server together so that you can play Eternal Lands with our program as a local proxy.

Open your project in Visual Studio, and right-click on the “Solution” (top-most item) in the “Solution Explorer”. Choose “Add”, then “New Project”, then select “Class Library” and name it Eternal Lands Connector. Press “Ok”. Now, right-click on your main project (“Windows Forms Application 1”, in my case) and choose “Project Dependencies”. In the “Depends on” box, check the “Eternal Lands Connector” project. Click “Ok”.

The purpose of this secondary project is to isolate our code. Once we’ve written the C# necessary to tunnel messages from the client to the server, we’ll then be adding a whole bunch of filtering and display code. We don’t want to accidentally break our message passing code, so we’ll keep in a totally separate project. You could manage this project separately, and manually add the DLL to the main Windows Forms project each time you update it, but Visual Studio’s dependency tracker allows you to do this automatically.

Let’s get started. First, rename Class1.cs to ELMiddleware.cs. We’ll need another file, though (Right-click “Eternal Lands Connector”, choose “Add”, then “New Item…”, then select “Class”) called SleepQueue.cs. This queue manages a list of messages, and stops processing them when the queue is empty. It sets itself to sleep until a new message is received. This signaling approach is far more efficient than constantly checking if the queue has items (spinning). Here’s the code for this class:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Net;
using System.Net.Sockets;
using System.IO;

namespace Eternal_Lands_Connector
{
  public abstract class SleepQueue : IDisposable
  {
    //For threading: allow sleeping when no messages are received, by using an EventWaitHandle
    private Thread t;
    private EventWaitHandle wHandle = new AutoResetEvent(false);
    private Queue<PacketData> waitingMessages = new Queue<PacketData>();

    //Add a packet to the list of waiting messages. 
    // Wake the main thread.
    public virtual void AddPacket(PacketData p)
    {
      lock (waitingMessages)
        waitingMessages.Enqueue(p);
      wHandle.Set();
    }

    //What priority should the message queue thread be? 
    //  Sub-classes can override this to balance performance.
    public virtual ThreadPriority getInitialPriority()
    {
      return ThreadPriority.Normal;
    }

    //Dispose properly
    public void Dispose()
    {
      // Signal the consumer to exit.
      AddPacket(null);
     if (t != null)
        t.Join();
     if (wHandle != null)
        wHandle.Close();
    }

    //Start the queueing process.
    public void startQueueing()
    {
      t = new Thread(new ThreadStart(Handle_Queue));
      t.IsBackground = true;
      t.Priority = getInitialPriority();
      t.Start();
    }

    private void Handle_Queue()
    {
      //Do pre-inits
      Pre_Init_Activity();

      //Listen for queue updates
      PacketData msg = null;
      for (; this.isValid(); )
      {
        //Get the next task
        msg = null;
        lock(waitingMessages)
         {
          if (waitingMessages.Count > 0)
           {
             msg = waitingMessages.Dequeue();
           if (msg == null)
             {
             //Done.
             this.On_Message(msg);
             return;
             }
           }
         }

        //A bit confusing to have two null checks, but it makes sense.
        if (msg == null)
         {
          //No more tasks - wait for a signal
              wHandle.WaitOne();
         } else {
            this.On_Message(msg);
         }
       }

      this.On_Thread_Close();
    }

    //
    //Un-implemented functionality
    //

    //When does this queue exit?
    protected abstract bool isValid();

    //What should we do with each message?
    protected abstract void On_Message(PacketData msg);

    //What should happen when the thread closes? 
    protected abstract void On_Thread_Close();

    //What should happen before the message loop begins? (After the thread is started)
    protected abstract void Pre_Init_Activity();
  }
}

This depends on the PacketData.cs file, which is simply a wrapper on a byte array coupled with an id character:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace Eternal_Lands_Connector
{
  public class PacketData
  {
    public PacketData(byte[] data, char from)
    {
      this.data = data;
      this.from = from;
    }

    public byte[] data;
    public char from;
  }
}

The technical details of the Sleep Queue are not important, so long as you realize that:

1. Every message posted to the queue gets processed.
2. You can post null to stop processing and end the thread.
3. The thread takes up no CPU time when there are no messages waiting.
4. Sub-classes will have to tell the queue when to stop processing (naturally), what to do with messages that are received, and what to do directly before entering and after leaving the message loop.

We now have everything we need to implement the ELMiddleware class, whose purpose is to handle all messages between the client and the server.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Net;
using System.Net.Sockets;
using System.IO;

namespace Eternal_Lands_Connector
{
    //Global
    public delegate void AddPacketDel(PacketData msg);
    public delegate void ShowConnectMsgDel(String msg);
    public delegate void AddToConsoleDel(String msg);
    public delegate void IncrPacketCountDel(int amt);

    public struct ServerInfo
    {
        public String id;
        public String configDir;
        public String address;
        public int port;
        public String description;
        public override String ToString()
        {
            return description;
        }
    }

    public class ELMiddleware : SleepQueue
    {
        //Hooks for controlling the client and server.
        private ELClientHook client;
        private ELServerHook server;

        //Saved
        private ServerInfo currServer;
        private ServerInfo loopback;
        private String elClientDir;
        private ShowConnectMsgDel ShowConnect;
        private AddToConsoleDel UpdateConsole;
        private IncrPacketCountDel IncrPacketCount;
        private int hookThreadID;

        public override void AddPacket(PacketData p)
        {
            //Log
            IncrPacketCount(1);

            //Count
            base.AddPacket(p);
        }

        private bool started = false;
        public bool Started {
            get {
                return started;
            }
        }

        public ELMiddleware(ELClientHook client, ELServerHook server)
        {
            //Save
            this.client = client;
            this.server = server;

            //Hook
            this.client.PacketHook = new AddPacketDel(AddPacket);
            this.server.PacketHook = new AddPacketDel(AddPacket);
        }

        public void startAll(ServerInfo currServer, ServerInfo loopback, String elClientDir, ShowConnectMsgDel ShowConnect, AddToConsoleDel UpdateConsole, IncrPacketCountDel IncrPacketCount, int hookThreadID)
        {
            //Don't start twice
            if (started || client.Started || server.Started)
                return;
            started = true;

            //Save
            this.currServer = currServer;
            this.loopback = loopback;
            this.elClientDir = elClientDir;
            this.ShowConnect = ShowConnect;
            this.UpdateConsole = UpdateConsole;
            this.IncrPacketCount = IncrPacketCount;
            this.hookThreadID = hookThreadID;

            //Start the sleep queues
            base.startQueueing();
        }

        protected override void Pre_Init_Activity()
        {
            //Init Log
            ShowConnect("EL Notifier Init");

            //Start the server
            server.hookServer(currServer, UpdateConsole);

            //Start the client
            client.hookClient(loopback, elClientDir, UpdateConsole, hookThreadID);

            //Spin-wait for both
            while (!server.ReadyForData || !client.ReadyForData)
                Thread.Sleep(5);
            UpdateConsole("\r\nSpin-waiting done");
        }

       protected override void On_Message(PacketData msg)
        {
            //Count
            IncrPacketCount(-1);
            if (msg == null)
                return;

            //Forward this message to the appropriate third party
            if (msg.from == 'C')
            {
                //Send from the client to the server.
                server.sendData(msg.data);
            }
            else if (msg.from == 'S')
            {
                //Send from the server to the client
                client.sendData(msg.data);
            }
            else
                UpdateConsole("\r\nBad message code: " + msg.from);
        }

        protected override bool isValid()
        {
            return client.Started && server.Started;
        }

        protected override void On_Thread_Close()
        {
            //Make sure everything closed nicely
            if (!server.Done)
                server.Done = true;
            if (!client.Done)
                client.Done = true;

            //Done
            started = false;
        }
    }
}

Add two more classes to the project, called ELClientHook.cs and ELServerHook.cs, respectively. Mark each class public, but don’t add any implementation yet.

Let’s review the code for ELMiddleware. At the top, we have some global delegate definitions, presumably used for logging. We also have our ServerInfo struct —if you think this means we’ll be removing a lot of code from Form1.cs then you guessed right. Our ELMiddleware class extends the SleepQueue abstract class. It contains several delegate references, a mysterious hookThreadID, and a reference to our client and server hooks.

Most of the code after this is obvious, and should give you an idea what kind of functionality we’ll need in ELClientHook and ELServerHook. The onMessage() function simply checks the sender of the message, and then ferries the message across to the recipient. We will later add the ability to react to ‘S’ class messages.

Our client and server hooks will need to be able to “take” messages from the middleware, in addition to other miscellaneous features. First, let’s add ELServerHook. Its functionality will be very similar to the previous chapter’s exercises.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.IO;
using System.Net.Sockets;
using System.Threading;

namespace Eternal_Lands_Connector
{
    public class ELServerHook
    {
        private TcpClient tcpclnt;
        private Stream srvStream;
        private ServerInfo srv;

        //Callbacks
        private AddToConsoleDel UpdateConsole;

        private bool started = false;
        public bool Started {
            get {
                return started;
            }
        }

        private bool done;
        public bool Done {
            get {
                return done;
            }
            set {
                done = value;
            }
        }

        private bool readyForData = false;
        public bool ReadyForData {
            get {
                return readyForData;
            }
        }

        private AddPacketDel packetDel;
        public AddPacketDel PacketHook {
            get {
               return this.packetDel;
            }
            set {
                this.packetDel = value;
            }
        }

        public void hookServer(ServerInfo srv, AddToConsoleDel UpdateConsole)
        {
            //Don't start twice
            if (started)
                return;
            started = true;

            //Save
            this.srv = srv;
            this.UpdateConsole = UpdateConsole;

            //Start
            tcpclnt = new TcpClient();
            UpdateConsole("Connecting.....");

            Thread t = new Thread(new ThreadStart(Check_Server));
            t.IsBackground = true;
            t.Start();
        }

        public void sendData(byte[] data)
        {
            if (!readyForData)
                throw new Exception("Server is not ready for messsages");

            //Send the dat down this stream. Technically, this occurs in the middleware's thread,
            //   so it won't conflict with any data we're reading at the same time.
            try {
                srvStream.Write(data, 0, data.Length);
            } catch (Exception ex) {
                UpdateConsole("\r\nException writing to server: \r\n   " + ex.ToString());
            }
        }

        private void Check_Server()
        {
            //Connect to the server's TCP port.
            tcpclnt.Connect(srv.address, srv.port);
            readyForData = true;
            UpdateConsole("Connected");

            //Try reading a bit
            srvStream = tcpclnt.GetStream();
            byte[] msg = new byte[1024];
            done = false;
            for (; !done; )
            {
                int amtRead;
                try {
                    amtRead = srvStream.Read(msg, 0, msg.Length);
                } catch (IOException) {
                    UpdateConsole("\r\nIOException; closing Server");
                    amtRead = 0;
                }

                if (amtRead != 0)
                {
                    //Make a new packet, add it to the middleware's queue
                    byte[] pk = new byte[amtRead];
                    Array.Copy(msg, pk, amtRead);
                    PacketHook(new PacketData(pk, 'S'));
                }
                else
                {
                    UpdateConsole("\r\nRead zero; closing Server");
                    done = true;
                }
            }

           //Must close BOTH
            srvStream.Close();
            tcpclnt.Close();

            //Ready to go again?
            UpdateConsole("\r\nServer Stopped");
            Console.WriteLine("Server hook stopped");
            started = false;
        }
    }
}

The implementation is almost exactly what we had before. The call to srvStream.Read() is blocking, by the way —no Sleep Queue is necessary.

The ELClientHook implementation is much more elusive:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.IO;
using System.Net.Sockets;
using System.Text;
using System.Text.RegularExpressions;
using System.Diagnostics;
using System.Threading;
using System.Windows.Forms;
using System.Runtime.InteropServices;

namespace Eternal_Lands_Connector
{
    //Just a helper class; no threads.
    public class ELClientHook
    {
        //Const
        public const string SERVER_ADDR = "127.0.0.1";
        public const int SERVER_PORT = 5421;

        private ServerInfo loopback;
        private String elClientDir;
        private TcpListener listener;
        private Stream clientStr;

        private Form mainForm;
        private int hookThreadID;
        private bool elDone = false;

        private bool started = false;
        public bool Started {
            get {
                return started;
            }
        }

        private bool done;
        public bool Done {
            get {
                return done;
            }
            set {
                done = value;
            }
        }

        private bool readyForData = false;
        public bool ReadyForData {
            get {
                return readyForData;
            }
        }

        private AddPacketDel packetDel;
        public AddPacketDel PacketHook {
            get {
                return this.packetDel;
            }
            set {
                this.packetDel = value;
            }
        }

        //Callbacks
        private AddToConsoleDel UpdateConsole;

        public void hookClient(ServerInfo loopback, String elClientDir, AddToConsoleDel UpdateConsole, int hookThreadID)
        {
            //Don't start twice
            if (started)
                return;
            started = true;

            //Save
            this.loopback = loopback;
            this.UpdateConsole = UpdateConsole;
            this.elClientDir = elClientDir;
            this.hookThreadID = hookThreadID;

            //Start background client
            Thread t = new Thread(new ThreadStart(Start_Client));
            t.IsBackground = true;
             t.Start();
        }

        public void sendData(byte[] data)
        {
            if (!readyForData)
                throw new Exception("Server is not ready for messsages");

            //Send the dat down this stream. Technically, this occurs in the middleware's thread,
            //   so it won't conflict with any data we're reading at the time.
            try {
                clientStr.Write(data, 0, data.Length);

                //Logging code - uncomment for development
                /*StringBuilder sb = new StringBuilder();
                sb.Append("\r\nClient Write: [");
                foreach (byte b in data)
                    sb.Append(Convert.ToString(b, 16).PadLeft(2, '0') + ":");
                sb.Append("]");
               UpdateConsole(sb.ToString());*/
            } catch (Exception ex) {
                UpdateConsole("\r\nException writing to client: \r\n   " + ex.ToString());
            }
        }

        private void EL_Stopped(Object sender, EventArgs e)
        {
            elDone = true;
        }

        //Get our handle (no idea why this requires an invoke... maybe it creates it?)
        private delegate IntPtr GetFormHandleDel(Form f);
        private IntPtr GetFormHandle(Form f)
        {
            if (f.InvokeRequired)
                return (IntPtr)f.Invoke(new GetFormHandleDel(GetFormHandle), f);
            else
                return f.Handle;
        }

        private bool findMainForm(int[] allowedProcessIDs, String matcherRegex, int hookThreadID)
        {
            return true;
        }

        private void EL_Window_Activate()
        {
            Console.WriteLine("EL Client Window Activated");
        }

        private void EL_Window_Deactivate()
        {
            Console.WriteLine("EL Client Window Deactivated");
        }

        private void Start_Client()
        {
            //Start the socket
            listener = new TcpListener(IPAddress.Parse(SERVER_ADDR), SERVER_PORT);
            listener.Start();
           UpdateConsole("\r\nLoopback server running at: " + listener.LocalEndpoint);

            //Start the EL client program with the server specified as "loopback"
            UpdateConsole("\r\nStarting EL Client...");
            ProcessStartInfo ps = new ProcessStartInfo();
            ps.FileName = elClientDir + "\\el.exe";
            ps.Arguments = loopback.id;
            ps.WorkingDirectory = elClientDir;
            ps.ErrorDialog = true;
            Process p = Process.Start(ps);
            elDone = false;
            p.Exited += new EventHandler(EL_Stopped);

            //Get the full-screen window for this application
            mainForm = null;
            while (mainForm == null)
            {
                int[] validIDs = new int[p.Threads.Count];
                {
                    int id = 0;
                    foreach (ProcessThread t in p.Threads)
                        validIDs[id++] = t.Id;
                }

                if (findMainForm(validIDs, "Eternal Lands", hookThreadID))
                    break;
                Thread.Sleep(5);
            }
            UpdateConsole("\r\nClient spin-waiting done");

            //Wait for the client to try to connect to us
            TcpClient socket = listener.AcceptTcpClient();
            clientStr = socket.GetStream();
            readyForData = true;
            UpdateConsole("\r\nEL Client tried to connect to us: " + socket.Client.RemoteEndPoint);

            //Continually read from the client. This is the main client update thread.
            byte[] msg = new byte[1024];
            done = false;
            for (; !done; )
            {
                //Is the client still running?
                if (elDone)
                {
                    UpdateConsole("\r\nGame closed; closing Client");
                    done = true;
                }
                else
                {
                    int amtRead;
                    try
                    {
                       amtRead = clientStr.Read(msg, 0, msg.Length);
                    }
                    catch (IOException)
                    {
                        UpdateConsole("\r\nIOException; closing Client");
                        amtRead = 0;
                    }
                    if (amtRead != 0)
                    {
                       //Make a new packet, add it to the middleware's queue
                        byte[] pk = new byte[amtRead];
                        Array.Copy(msg, pk, amtRead);
                        PacketHook(new PacketData(pk, 'C'));
                    }
                    else
                    {
                        //We're done
                        UpdateConsole("\r\nRead zero; closing Client");
                        done = true;
                    }
                }
            }

            //Have to close all three
            clientStr.Close();
            socket.Close();
            listener.Stop();

            //Ready to go again?
            UpdateConsole("\r\nClient Stopped");
            Console.WriteLine("Client hook stopped");
            started = false;
        }
    }
}

You might get an error with the System.Windows.Forms using directive; simply right-click on the “References” section of your Connector’s Project file and choose “Add Reference”. Then, in the “.NET” tab, browse down to “System.Windows.Forms” and click “Ok”. By default, a library project does not have access to the Forms dll, for reasons that remain a mystery to me.

A quick browse through this code reveals that our fake middleware server will be situated at 127.0.0.1, on port 5421. The sendData() function is similar to the server’s copy; however, I’ve included some logging code for your convenience. If you uncomment this, it will print a properly-formatted hexadecimal representation of each message sent from the server to the client. This will allow you to manually debug unknown or tricky messages, and to determine how you should react to them.

The findMainForm() function is currently empty; it will later be used to determine which window is actually hosting the EL client. The only other function of interest is the StartClient() function. Note that we use a ProcessStartInfo object to handle starting a process:

1.We have to set the working directory to the Eternal Lands folder, otherwise the process won’t be able to find its resources (images, sounds, etc.).
2.We start the process el.exe. This, of course, is Windows-specific.
3.We start this process with the argument loopback. In other words, we call “el.exe loopback” from the command line, allowing us to choose which server to connect to.

The Thread-finding code which follows this is messy; just ignore it. Basically, our middleware program will have multiple running threads, and we need to figure out which one is hosting the EL Client’s window. That way, we can detect when it has focus, and avoid showing messages in that case.

Note that, like the server, the client continually reads from the middleware until no more data is sent. You’ll have to change Form1’s code to actually use this new middleware, of course.

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using Eternal_Lands_Connector;
using System.Text.RegularExpressions;
using System.IO;
using System.Threading;
using System.Net.Sockets;

namespace WindowsFormsApplication1
{
    public partial class Form1 : Form
    {
        //Count leftover packets. A good first-warning if something goes wrong.
        private int packetCount;

        //Our hooks
        private ELServerHook serverHook;
        private ELClientHook clientHook;
        private ELMiddleware middleWare;

        //Also...
        private ServerInfo loopbackInfo;
        private string srvLine = "loopback        main            " + ELClientHook.SERVER_ADDR + "               " + ELClientHook.SERVER_PORT + "    Loopback adapter for use with the EL Notifier";

        public Form1()
        {
            InitializeComponent();

            //Initialize connectors.
            serverHook = new ELServerHook();
            clientHook = new ELClientHook();
            middleWare = new ELMiddleware(clientHook, serverHook);
        }

        private void ReadELFiles(object sender, EventArgs e)
        {
            //Reload an entirely new list of servers.
            cmbServers.Items.Clear();

            //Don’t load anything if we specified an invalid directory.
            if (!File.Exists(txtPathToELFolder.Text + "\\" + "el.exe"))
            {
                txtConsole.Text = "Error: el.exe doesn't exist in EL folder";
                return;
            }
            else if (!File.Exists(txtPathToELFolder.Text + "\\" + "servers.lst"))
            {
                txtConsole.Text = "Error: servers.lst doesn't exist in EL folder";
                return;
            }

            //Open our servers file and read it.
            bool foundLoopback = false;
            StreamReader srvFile = new StreamReader(txtPathToELFolder.Text + "\\" + "servers.lst");
            while (!srvFile.EndOfStream)
            {
                String line = srvFile.ReadLine();
                if (line == null)
                    break;

                //Coment? Empty?
                line = line.Trim();
                if (line.StartsWith("#") || line.Length == 0)
                    continue;

                //Read:
                //ID, Config_Dir, Address, Port, Description
                String regex = "([^ \t]+)[ \t]+([^ \t]*)[ \t]+([^ \t]*)[ \t]+([^ \t]*)[ \t]+(.*)";
                Match m = Regex.Match(line, regex);
                ServerInfo sInf = new ServerInfo();
                if (m.Success && m.Groups.Count == 6)
                {
                    sInf.id = m.Groups[1].Value;
                    if (sInf.id.Equals("loopback"))
                        foundLoopback = true;
                    sInf.configDir = m.Groups[2].Value;
                    sInf.address = m.Groups[3].Value;
                    try
                    {
                        sInf.port = Int32.Parse(m.Groups[4].Value);
                    }
                    catch (FormatException)
                    {
                        //Don't add
                        continue;
                    }
                    sInf.description = m.Groups[5].Value;
                    cmbServers.Items.Add(sInf);

                    //Loopback?
                    if (sInf.id.Equals("loopback"))
                        loopbackInfo = sInf;
                }
            }
            srvFile.Close();

            //Do we need to add our own local server?
            if (!foundLoopback)
            {
                //Copy existing server file
                StreamReader inFile = new StreamReader(txtPathToELFolder.Text + "\\" + "servers.lst");
                StreamWriter outFile = new StreamWriter(txtPathToELFolder.Text + "\\" + "servers.lst.new");
                for (String line = inFile.ReadLine(); line != null; line = inFile.ReadLine())
                    outFile.WriteLine(line);
                inFile.Close();

                //Add our own category
                outFile.WriteLine(srvLine);

                //Close, swap, notify.
                outFile.Close();
                File.Delete(txtPathToELFolder.Text + "\\" + "servers.lst");
                File.Move(txtPathToELFolder.Text + "\\" + "servers.lst.new", txtPathToELFolder.Text + "\\" + "servers.lst");
                ShowConnectMsg("Created loopback entry.");

                //Add a new item to the list
                loopbackInfo = new ServerInfo();
                loopbackInfo.id = "loopback";
                loopbackInfo.configDir = "main";
                loopbackInfo.address = ELClientHook.SERVER_ADDR;
                loopbackInfo.port = ELClientHook.SERVER_PORT;
                cmbServers.Items.Add(loopbackInfo);
            }

            if (cmbServers.Items.Count > 0)
                cmbServers.SelectedIndex = 0;

            //Enable the remainder of our form
            cmbServers.Enabled = true;
            btnConnect.Enabled = true;
        }

        private void ConnectToServer(object sender, EventArgs e)
        {
            if (middleWare.Started || clientHook.Started || serverHook.Started)
                return;

            //Count
            packetCount = 0;

            //Get our currently-selected server and start it.
            ServerInfo srv = (ServerInfo)cmbServers.SelectedItem;
            middleWare.startAll(srv, loopbackInfo, txtPathToELFolder.Text, new ShowConnectMsgDel(ShowConnectMsg), new AddToConsoleDel(UpdateConsole), new IncrPacketCountDel(IncrPacketCount), Thread.CurrentThread.ManagedThreadId);
        }

        //Delegates and implementation details
        private void ShowConnectMsg(String msg)
        {
            Control ctl = txtConsole;
            if (ctl.InvokeRequired)
                ctl.Invoke(new ShowConnectMsgDel(ShowConnectMsg), msg);
            else
                ctl.Text = msg;
        }

        private void UpdateConsole(String msg)
        {
            Control ctl = txtConsole;
            if (ctl.InvokeRequired)
                ctl.Invoke(new AddToConsoleDel(UpdateConsole), msg);
            else
                ctl.Text += msg;
        }

        private void IncrPacketCount(int amt)
        {
            packetCount += amt;
            SetText(lblZero, packetCount + "");
        }

        private delegate void SetTextDel(Control ctl, String msg);
        private void SetText(Control ctl, String msg)
        {
            if (ctl.InvokeRequired)
                ctl.Invoke(new SetTextDel(SetText), ctl, msg);
            else
                ctl.Text = msg;
        }
    }
}

Right-click on the References section in your main Forms project and choose “Add Reference”. Then, in the “Projects” tab, choose “Eternal Lands Connector” and hit “Ok”. This will allow you to use our class library.

The source code is not too complex, and the effect is easy enough to describe: we simply intercept and then deliver all messages from the client to the server, and from the server to the client. At the moment, this is essentially a whole lot of work for nothing. However, now that all messages are passing through our program (our source code, you might say), you can easily  tag messages of a specific type, and prompt the user after further processing them. So, this was all “legwork”.

Our middleware in action, connecting to the PK server.

Congratulations, we have now bamboozled both end points. In the next section, we’ll consider how to filter messages without requiring a re-compile of the Connector each time we change our minds about which messages to filter. We’ll also get you started processing Eternal Lands’s images to brighten up our notifier.

Step 4: Filtering Messages (Outside the Core)

At this point, you should have a pretty good idea how to intercept messages of a certain type. Before you dash too far into it, though, may I remind you that one of the reasons we used C# in the first place was to avoid bugs that corrupted our communication layer. We should add the following to the communication core:

1. Allow our main Forms class to “register” any number of listeners with the core.
2. Before sending a message (to the client), check if it matches any of the listeners’ patterns. If it does, send a copy of the message to the matched listener.
3. Ensure that the core runs in a higher-priority thread than the listeners’ implementations. According to the rules of Windows thread priorities, this ensures that the listeners cannot starve the core for cycles.

This way, no amount of chicanery by the listeners can cause the core to malfunction. (And remember, malfunctions are BAD, since we’re not the ones paying for the EL servers.)

Open up ELMiddleware.cs, and add the following member variable to the class:

//An additional matcher

private MessageMatcher msgMatch;

We’ll define MessageMatcher later; for now, scroll down to the definition of startAll, and change it to read:

public void startAll(ServerInfo currServer, MessageMatcher msgMatch, ServerInfo loopback, String elClientDir, ShowConnectMsgDel ShowConnect, AddToConsoleDel UpdateConsole, IncrPacketCountDel IncrPacketCount, int hookThreadID)

Next, somewhere in the body of startAll, save the MessageMatcher that we passed in to the function:

this.msgMatch = msgMatch;

Make sure you start it too; after “base.startQueueing()” add:

msgMatch.startQueueing();

And finally, scroll down until you find the line:

client.sendData(msg.data);

…and add the following underneath it:

//Send the data to an additional thread for logging, parsing, etc.

msgMatch.AddPacket(msg);

That code should speak for itself; our MessageMatcher class will centralize message processing, and the AddPacket method is called for each server message. We do not filter client messages, because it seems that these originate mostly from the user’s actions (which are always known by the user –and must be confirmed by the server, regardless.)

Here, then, is the definition of the MessageMatcher class. Add it to your project (to the Eternal_Lands_Connector project, that is).

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Threading;

using System.Net;

using System.Net.Sockets;

using System.IO;

namespace Eternal_Lands_Connector

{

public class MessageMatcher : SleepQueue

{

/// <summary>

/// The core of our message matcher

/// </summary>

private Dictionary<Int32, List<MessageHandler>> handlers = new Dictionary<Int32, List<MessageHandler>>();

public void addHandler(MessageHandler h)

{

if (!handlers.ContainsKey(h.getProtocol()))

handlers[h.getProtocol()] = new List<MessageHandler>();

handlers[h.getProtocol()].Add(h);

Console.WriteLine(“Dictionary size: “ + handlers.Count + ”   entries[" + h.getProtocol() + "] –> “ + handlers[h.getProtocol()].Count);

}

public AddToConsoleDel UpdateConsole;

protected override void On_Message(PacketData msg)

{

if (msg == null)

return;

//Now, segment into multiple messages

for (int currID = 0; currID < msg.data.Length; )

{

int protocol = msg.data[currID++];

int size = msg.data[currID++] | (msg.data[currID++] << 8);

//Make a new data array

int count = size – 1;

byte[] b = new byte[count];

Array.Copy(msg.data, currID, b, 0, count);

currID += count;

//Match the first relevant handler

if (handlers.ContainsKey(protocol))

{

foreach (MessageHandler h in handlers[protocol])

{

if (h.performMatch(protocol, b, new Dictionary<string, object>()))

{

break;

}

}

}

}

}

/// <summary>

/// Somwhat less-interesting features

/// </summary>

private bool valid = true;

public bool IsValid

{

get

{

return valid;

}

set

{

valid = value;

}

}

//We make this its own class so we can change its priority.

public override System.Threading.ThreadPriority getInitialPriority()

{

return ThreadPriority.BelowNormal;

}

protected override bool isValid()

{

return valid;

}

protected override void On_Thread_Close()

{

//Nothing to do here

}

protected override void Pre_Init_Activity()

{

//Nothing to do here

}

}

}

The MessageHandler class hasn’t been created yet, but let’s just say that it matches all messages first on a “protocol” and then on a more complex “performMatch” query. All of these MessageHandlers are stored indexed by protocol in a hash table. This ensures that we can add lots of processing handlers without slowing down the pattern matching.

Finally, note that MessageMatcher is a SleepQueue, which means that even if we do add far too many handlers of the same protocol (say, one for each buddy in our buddy list?), only our matching will be delayed –the core won’t be.

You might be wondering about the structure of Eternal Lands‘s messages. From the processing loop in MessageMatcher, you can tell that every message contains a protocol byte as the first item in the message. The next two bytes describe the size of the remaining data. This is more important than you might think: Eternal Lands often sends multiple messages bunched together, and you can only tell where one ends and the next begins by processing the size bytes. Anything after the first three bytes is dependant on the protocol of the message. Also, for some unknowable reason, the actual size of the remaining data is size-1.

Now, make a class called MessageHandler, in the Eternal_Lands_Connector project:

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

namespace Eternal_Lands_Connector

{

public abstract class MessageHandler

{

//All message handlers match on a protocol

public abstract int getProtocol();

//Within a given protocol, does this handler match?

//  This should also perform an action

//  Decorations used to avoid multiple calculations

public abstract bool performMatch(int protocol, byte[] data, Dictionary<String, Object> decorations);

}

}

Were you to compile your project, you would notice that Form1‘s call of middleWare.startAll() is missing an argument: MessageMatcher. Before we go passing in an empty class, let’s try thinking about the types of MessageHandlers we might need. Well, the most basic type will simply match based on the protocol, and defer further processing till later. Then, we might want to match text messages sent from the server, using such things as the color of the text, the channel it was sent to, and a regular expression of what the text itself must contain.

First, make a folder (in the main Forms project) called matching. We will put three classes in there: Constants.cs, BasicHandler.cs, and TextHandler.cs. First, let’s cover Constants.cs:

public class Constants

{

public enum Protocols

{

RAW_TEXT = 0,

SEND_PARTIAL_STAT = 49,

GET_NEW_INVENTORY_ITEM = 21,

}

public enum Colors

{

c_green1 = 3,

c_red1 = 0,

}

public enum Channels

{

CHAT_SERVER = 3,

}

}

This class simply contains some useful constants used to identify channels, colors, and protocols. We’ve included only the bare minimum; please add any constants as your message matching needs expand.

Now, BasicHanlder.cs:

class BasicHandler : Eternal_Lands_Connector.MessageHandler

{

public delegate void OnMatchDel(byte[] data);

private int protocol;

private OnMatchDel OnMatch;

public BasicHandler(int protocolMatch, OnMatchDel OnMatch)

{

this.protocol = protocolMatch;

this.OnMatch = OnMatch;

}

public override int getProtocol()

{

return protocol;

}

public override bool performMatch(int protocol, byte[] data, Dictionary<string, object> decorations)

{

//Quick initial match

if (protocol != getProtocol())

return false;

//Always match

OnMatch(data);

return true;

}

}

The unfortunate reality is that this class is so simple that it makes understanding it tricky. It’s very simple:

1. The constructor takes a protocol value to match, and an OnMatch delegate to perform when that protocol matches. It saves these.
2. The getProtocol() method exists so that our MessageMatcher can store this MessageHandler in the correct hash table entry.
3. The performMatch() function always returns true, after performing the OnMatch delegate, assuming the protocol byte matches.

Without thinking too much about it, let’s add the code for TextHandler.cs and compare the two:

public class TextMessage

{

public int channel;

public int color;

public String text;

}

public class TextHandler : Eternal_Lands_Connector.MessageHandler

{

private const String KEY_DECORATIONS = “TextHandler_Decorations”;

public delegate void OnMatchDel(TextMessage m);

private Regex msgPattern;

private int msgColor;

private int msgChannel;

private OnMatchDel OnMatch;

//-1 for “all” on color or channel

public TextHandler(String msgPattern, int color, int channel, OnMatchDel OnMatch)

{

this.msgPattern = new Regex(msgPattern);

this.msgColor = color;

this.msgChannel = channel;

this.OnMatch = OnMatch;

}

public override int getProtocol()

{

return (int)Protocols.RAW_TEXT;

}

public override bool performMatch(int protocol, byte[] data, Dictionary<String, Object> decorations)

{

//Quick check; protocol

if (protocol != getProtocol())

return false;

//Get

if (!decorations.ContainsKey(KEY_DECORATIONS))

{

//Construct

TextMessage t = new TextMessage();

t.channel = data[0];

t.color = data[1] – 127;

//Parse Text

int count = data.Length – 2;

t.text = System.Text.ASCIIEncoding.ASCII.GetString(data, 2, count);

//Later, of course, we’ll have to deal with things like embedded color tags, etc.

//Save

decorations[KEY_DECORATIONS] = t;

}

TextMessage message = (TextMessage)decorations[KEY_DECORATIONS];

//Match -Stage 1

if ((msgChannel != -1 && message.channel != msgChannel) || (msgColor != -1 && message.color != msgColor))

return false;

//Does the text match?

if (!msgPattern.IsMatch(message.text))

return false;

//It matches!

OnMatch(message);

return true;

}

}

You’ll notice that I didn’t include the using directives or the enclosing namespace. Whatever Visual Studio provides as defaults will work in this case. You’ll have to add two additional using directives, however. Note the second one; it assigns an alias to an enum within our Constants class. This allows us to simply reference Protocols.RAW_TEXT, instead of the more cumbersome Constants.Protocols.RAW_TEXT.

using System.Text.RegularExpressions;

using Protocols = WindowsFormsApplication1.matching.Constants.Protocols;

We’re now ready to create some message handlers and log their output. Go to your main form’s ConnectToServer method, and add the following lines after your various checks for “Started”

//Make a Message Matcher to handle all our special cases

MessageMatcher match = new MessageMatcher();

match.UpdateConsole = UpdateConsole;

//Add some filters

TextHandler h1 = new TextHandler(“You stopped harvesting.”, (int)Colors.c_red1, (int)Channels.CHAT_SERVER, new TextHandler.OnMatchDel(On_Harvest_Stop));

match.addHandler(h1);

BasicHandler h2 = new BasicHandler((int)Protocols.GET_NEW_INVENTORY_ITEM, new BasicHandler.OnMatchDel(Got_Item));

match.addHandler(h2);

You will also need some using directives:

using WindowsFormsApplication1.matching;

using Protocols = WindowsFormsApplication1.matching.Constants.Protocols;

using Colors = WindowsFormsApplication1.matching.Constants.Colors;

using Channels = WindowsFormsApplication1.matching.Constants.Channels;

…the “WindowsFormsApplication1” should be the name of your project; you should be able to resolve naming conflicts like this on your own. Finally, add your MessageMatcher into the call to startAll:

middleWare.startAll(srv, match, loopbackInfo, txtPathToELFolder.Text, new ShowConnectMsgDel(ShowConnectMsg), new AddToConsoleDel(UpdateConsole), new IncrPacketCountDel(IncrPacketCount), Thread.CurrentThread.ManagedThreadId);

The only thing left to do is to implement On_Harvest_Stop and Got_Item. Here are the methods:

private void On_Harvest_Stop(TextMessage tm)

{

//Show message:

UpdateConsole(“\r\nYou have stopped harvesting.”);

}

private void Got_Item(byte[] data)

{

//Decode our message

int image_id = data[0] | (data[1]<<8);

int quantity = data[2] | (data[3]<<8) | (data[4]<<16) | (data[5]<<32);

int pos = data[6];

uint flags = data[7];

//Show message:

UpdateConsole(“\r\nYou got “ + quantity + ” item(s): “ + image_id);

The On_Harvest_Stop method simply informs the player that he or she has stopped harvesting. At this point, all we have to do is show a window and we’d be at the same point we were with the previous approach. (We’ll spice that up in the next tutorial). The Got_Item method gets the image_id, quantity, position, and flags for the item the user picked up (through harvesting, from a drop bag, whatever) and informs the user of these values. We’ll expand on this in the next section. How do we get this data? Read the Eternal Lands source code –it’s pretty easy to find what you want. For example, a little digging reveals the purpose of the flags byte:

is_resource        = binary(0010)  //Can be used to manufacture

is_reagent         = binary(0001)  //Can be used in magic

use_with_inventory = binary(1000)  //Item can be used with inventory

is_stackable       = binary(0100)  //The item is stackable

Moving on, compile and run your code, log in to the main server, and harvest something. Alt+Tab to our C# window and wait for the “You have stopped harvesting” message. Useful, eh? You’ll also see some “Got Item” messages.

Several Item and Harvest Messages

You should take some time to study these messages. First of all, you’ll notice that the quantity of a “got” item is the total number you now have, not the number you just picked up. Second, you’ll see that one of the items we were harvesting (id 35) was harvested at 2x the normal rate. This is because we were at one of the 2x spots for that item.

Step 5: Bonus! Fun With Images

At this point, we’re basically done with phase one of our project. We can hook the server and filter messages by a variety of criteria (without recompiling the connection code). But before we call it quits, there’s one thing that’s bothering me: what exactly is item 35? Unfortunately, there’s no easy way to tell, as the Eternal Lands client simply deals with images and identifiers. But maybe we can at least show a picture of the item we just picked up?

If you dig a bit in the source code, you’ll find that the following code is used for drawing items:

cur_item_img=item_list[pos].image_id%25;

u_start=0.2f*(cur_item_img%5);

u_end=u_start+(float)50/256;

v_start=(1.0f+((float)50/256)/256.0f)-((float)50/256*(cur_item_img/5));

v_end=v_start-(float)50/256;

//get the texture this item belongs to

this_texture=get_items_texture(item_list[pos].image_id/25);

get_and_set_texture_id(this_texture);

glBegin(GL_QUADS);

if(mini)

draw_2d_thing(u_start,v_start,u_end,v_end,mouse_x-16,mouse_y-16,mouse_x+16,mouse_y+16);

else

draw_2d_thing(u_start,v_start,u_end,v_end,mouse_x-25,mouse_y-25,mouse_x+25,mouse_y+25);

glEnd();

Converting this to pseudo-code, we get:

image_id = curr_item_img + (25*item_texture_num)

curr_item_img = u_comp + (5*v_comp)

u_start = 0.2*u_comp

u_end = u_start + 0.195  //Width is roughly in .2 increments

v_start = 1.00076 – 0.195 * v_comp

v_end = v_start – 0.195

A shrewd observer (or one accustomed to OpenGL) will notice that this code assumes Cartesian coordinates, so we’d have to flip the v_* variables.

Before we convert this to C#, browse to the Eternal Lands directory, and look in the “textures” sub-folder. Then, look at any of the “items*.bmp” files. Hmm… these images look mighty familiar…

A 5x5 Grid of Eternal Lands Items

Here’s the code to open one of these 5×5 items files, and read each separate image into a Bitmap object. Add this to your main Form’s ReadELFiles method, right before setting cmbServers.Enabled to true.

//Now load our bitmaps from the textures folder

for (int img_texture_id = 0; true; img_texture_id++)

{

//Load this image

String path = txtPathToELFolder.Text + “\\” + “textures” + “\\” + “items” + (img_texture_id + 1) + “.bmp”;

if (!File.Exists(path))

break;

Bitmap patch = new Bitmap(path);

//Loop through all possible images

for (int curr_item_img = 0; curr_item_img < 25; curr_item_img++)

{

//Construct a composite id

int composite_id = curr_item_img + 25 * img_texture_id;

//Get the sub-images’ co-ordinates

float u_start = 0.2F * (curr_item_img % 5);

float u_end = u_start + (float)50 / 256;

float v_start = (1.0f + ((float)50 / 256) / 256.0f) – ((float)50 / 256 * (curr_item_img / 5));

float v_end = v_start – (float)50 / 256;

//Convert to pixels, non-Cartesian

int xStart = (int)(u_start * 256);

int xEnd = (int)(u_end * 256);

int yStart = 256 – (int)(v_start * 256);

int yEnd = 256 – (int)(v_end * 256);

//Draw this into its own bitmap

Bitmap pic = new Bitmap(xEnd – xStart, yEnd – yStart);

Graphics g = Graphics.FromImage(pic);

g.DrawImage(patch, new Rectangle(0, 0, xEnd – xStart, yEnd – yStart), new Rectangle(xStart, yStart, xEnd – xStart, yEnd – yStart), GraphicsUnit.Pixel);

itemPics[composite_id] = pic;

}

}

The code is pretty self-explanatory; it simply reads all items*.bmp files until one doesn’t exist. For each of the 25 items, it applies the segmentation algorithm. It does require one more variable, added at the top of the class:

private Dictionary<Int32, Bitmap> itemPics = new Dictionary<int, Bitmap>();

Now, let’s use this! Add the following code to the end of Got_Item, replacing the existing call to UpdateConsole:

UpdateGotImage(image_id);

Now, implement our delegate and method call, as usual:

private delegate void UpdateGotImageDel(int image_id);

private void UpdateGotImage(int image_id)

{

if (justGotItem.InvokeRequired)

justGotItem.Invoke(new UpdateGotImageDel(UpdateGotImage), image_id);

else

{

justGotItem.Image = itemPics[image_id];

}

}

This references a GUI component that doesn’t exist yet. Add a PictureBox to your Form (anywhere it fits) and set its name to justGotItem and its size to 50,50. Sorry for approaching this in a rather roundabout fashion, but I’m sure you were able to keep up. Compile and run your code. Now, pay attention to your C# window every time you get a new item.

Harvesting Roses; Game Proves It's Working

Proof that this actually works is somehow so much more satisfying once images are involved. I’m at a loss to explain it.

The possibilities from here on in are really endless. What about catching Private Messages from players, and funneling them into GTalk through their API? How about updating your Eternal Lands blog with the item you are harvesting or making, in real-time? What about saving the value returned each time you use an Astrology Indicator, and then charting the results to see how your luck fares over time? All of these things are possible, and none (as near as I can tell) violate the terms of use of Eternal Lands. But make sure you post on the EL forum, if you’re not sure. It’s better to get permission rather than get banned.

I regret that I’ve only posted half the code I’ve written at this point. It just takes so long to re-write and re-test everything. I’m a bit tired of all this right now; next time I feel like writing 50 pages of text, though, I’ll finally show you how to soup up your message interface. Consider the following screenshots as teasers. Note the cool “dissolve in” effect, and the “maximize” and “close” buttons (I edited the image; normally they only highlight when you mouse over them.) All of this code works; I just need to document it, and hook up the notifications to real messages (no, I did not get three enrichment stones in a row. )

Nice System Tray Icon

Version 2.0 Has a Fancy, Animating Notification Stack

Final Discussion

Eternal Lands is one of my favorite online games, and yet like all cross-platform games, it doesn’t feel tightly integrated into Windows. One benefit of them opening the source to developers is that we can now hack around and improve the code as we see fit, distributing our changes to benefit everyone. In this article, I showed you how to enhance the notifications provided by Eternal Lands, to make harvesting of high-level items less tedious. By using a high-level language (C#) with low-level hooks into Win32 networking, we achieve both power and speed, with the potential to add nice animations and Windows-specific polish as well. Best of luck with your own hacks!

License – Preamble

The current code cannot be used to abuse the terms of use of Eternal Lands, and I’ve chosen a carefully-worded license to prevent further modifications of this code from being used for abuse. The Eternal Lands developers trusted us with their code, and I intend to honor that trust. Please do the same. The following license applies to source code, compiled binaries, and this article, describing how to make the program from scratch. All future source and binary releases (as well as updates to this article) will remain under this license.

License

Copyright (c) 2009 Seth N. Hetu

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The Software may not be modified to abuse the Terms and Conditions of the Eternal Lands game, as expressed in its license (http://www.eternal-lands.com/page/license.txt).

The Software may not be used to inject events to, nor remove messages from, the regular stream of packets sent between the Eternal Lands client and the Eternal Lands server.

All messages sent from the Eternal Lands client to the Eternal Lands server and vice-versa must arrive at their intended destinations without delay and without reordering or modifying them.

The Thread Priority of the event handler used by the Software may not be modified; moreover, any threads created that interact with the Software in any way MUST have a lower Thread Priority than that of the event handler.

If, at any time, an Eternal Lands moderator (as listed at: http://www.eternal-lands.com/page/developers.php?#mods) or Eternal Lands creator (http://www.eternal-lands.com/page/developers.php?#creators) requests that the Software or any specific derivative work be removed, for any reason, this demand must be met, and all source code and compiled code for the Software must be removed from public display. (Aside: We would ask moderators to be reasonable when demanding that the Software be removed.) The Software and its sources may be restored to public display only with the permission of an Eteranl Lands moderator. In the case of a dispute between moderators and creators, the creators are given preference.

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

[LZ-3] Intermediate Hyrule Magic

Blog Info: Staffman of the week goes to either Ubuntu or Fedora —whichever OS decided to corrupt the first block of this file, forcing me to re-type the entire thing.

Back-story: Sometimes there’s no quaint little project (like LZ-2) that gets you involved in the software; you just have to force yourself to accomplish stock goals. That’s what we’ll be doing in today’s lesson: solving a series of moderately difficult, realistically useful tasks related to Zelda hacking.

Goal: Walk through some typical usage scenarios with Hyrule Magic.

Don’t forget your hash:
1a74468291b02729329dd1357afb45af *Zelda3.smc

Table Of Contents

1. How Do I Edit “Link’s Graphics” In Hyrule Magic?

2. How Do I Save An Image in 16-Color Format?

3. How Do I Edit Other Graphics? How Do I Use Zcompress and YY-CHR?

4. How Do I Edit Overworld Tiles? What’s This 16×16 Tile Nonsense?

5. Lunch~

6. How Do I Change Flute Locations?

7. How Do I Edit The Game’s 3-D Graphics? How Complex Can These Graphics Be?

8. How Do I Change The Music For An Area?

9. How Do I Make An Enemy Drop a Key? How Do I Choose Which Enemy Drops the Key?

10. How Do I Add a Nifty Logo To The Title Screen?

11. (Bonus) How Do I Actually Manage a Large Project?

12. Addendum: How do I make a number of torches open a door when lit and make them burn out after a period of time if I do not do light them fast enough?

13. Addendum: How do I change were link lands when he falls in a hole to a lower level?

Question: How Do I Edit “Link’s Graphics” In Hyrule Magic?

One of the latest additions to Hyrule Magic is a “Link’s Graphics” tab which gives you access to the 4bpp image data associated with our hero. It’s a bit tricky to use properly, though.

Exporting Link's Graphics. Palette is Somewhat Messed Up, Since We're Using Wine. But It Should Still Work Ok.

The poses on the left are Link’s animation graphics, and the squares on the right are potential palettes. (You can click on various squares to mess with Link’s graphics, but make sure you click on the lower-left most square before doing any serious work.) To export graphics click the “copy” button, and then go to your favorite image editor and choose “paste”.

You might want to do this in Windows for two reasons. First, although Link’s palette (the bottom row) is shown properly in Wine, all other palette colors are messed up. Although the graphics editor doesn’t directly handle palette data, I personally wouldn’t risk having Wine corrupt your project; Hyrule Magic does enough corruption on its own. Second, one of the best free raster editing programs for Windows has only spotty success under Wine. On Linux, I’ve yet to find a native equivalent —I consider The Gimp to be terrible for pixel-based editing. Either way, you’re on your own for how you edit the copied graphics. Just make sure you save your file as a 4-bit (16-color) BMP file. If you’re using an editor that doesn’t support 4bit BMPs, just save it as a True Color BMP and then convert it later (see the next question).

Let’s give Link a diving-style helmet when he swims. Simply edit the three side-facing and two forward-facing heads with water around them. Give Link the helmet, and also remove the “mouth opening” graphics, like so:

Some of Link's Graphics, Edited to Add a Helmet. (Background Faded For Emphasis.)

Make sure that when you edit these graphics, you don’t use any new colors. Only use colors that are already in Link’s graphics. When you’re done, perform a “Select All” and then a “copy” from your image editor, then go into Hyrule Magic and choose “Paste” from the “Link’s Graphics” window. Save your game, and try it out.

Link Is Swimming With His New Safety Helmet On.

It’s not perfect (several more graphics need to be changed to make the animation accurate), but it shows the concept.

If Link’s colors are messed up, or if you got the following error when you pasted in Link’s graphics, then you probably failed to save the image in proper 16-color format. Please see the following question for information about this.

The Error You Get When You Try to Import a Picture With Too Many Colors.

Question: How Do I Save An Image in 16-Color Format?

Hyrule Magic is very picky about the format of its images. Personally, I think it should have exported and imported files instead of clipboard data, but whatever their rationale, the reality is that you often get a “Bad error” message or screwed up colors when you re-import your graphics. The best way to avoid this problem is at the very beginning: by saving the copied graphics properly as a 4bpp, 16-color bitmap before you perform any editing at all.

We’re going to use the free version of Graphics Gale, an excellent raster editing program. First, open the program. Then, choose “File->New”, and enter 128 for the width, 448 for the height, and 4-bit (16 color) for the bit depth. Choose “Ok”.

Now, open a clean Zelda3 ROM in Hyrule Magic, go to the “Link’s graphics” window, and select “Copy”. Then, go back to Graphics Gale. Click on the down-pointing arrow in the palette window, circled below. Then, choose “Load Palette”.

Graphics Gale Hides Its Menus In Non-Standard Context Buttons.

This will bring up a new menu; here, click on “File” and then choose “Import From Clipboard”. You’ll see the palette from Link’s graphics side-by-side with your current palette.

Copying Link's Palette In Graphics Gale.

Since we want to use all the new (left-hand-side) colors, click and drag to select all of them, and then click once on the first color of the right-hand palette to paste them in order. (You might also try the “All” button, though I’ve never used it.) Then, click Ok. Now, you can simply paste Link’s graphics into the main drawing window (Ctrl+V) and then save the file. At this point, you should be able to open this file using another program; most good editors won’t change the palette without asking you. If that doesn’t work, you can always edit your sprites in Graphics Gale; it’s a decent stand-alone editor.

I’ve heard that Graphics Gale works on Linux under Wine, at least well enough to perform the steps listed above. If you prefer to use another editor on any OS, here’s some instructions on how to save in 16-color format, in The Gimp, Photoshop, and other popular programs.

Remember, though, that saving in 16-color format won’t fix the error; you also have to copy it  in 16-color format. So you should always copy from Graphics Gale before pasting into Hyrule Magic. This is what makes Hyrule Magic difficult to use at times.

Question: How Do I Edit Other Graphics? How Do I Use Zcompress and YY-CHR?

Link’s graphics are unique in that they are encoded with 4 bits-per-pixel (a format supported by most major image editors) and they are (I believe) uncompressed. The remaining graphics are much harder to edit. You basically have to follow three steps:

1. Decompress the graphics from your ROM to a .bin file.
2. Edit the graphics using YY-CHR.
3. Recompress the graphics in your .bin file and insert them into the ROM.

This method is so general that it works for almost any game, so long as you have the right decompression tool. Lunar Compress can handle several popular formats. The only other thing you need to remember is how many bits-per-pixel each graphic is encoded in; please see the Perfect Guide for advice on this, or just look up the palette in Hyrule Magic.

All right, let’s edit Link’s shield graphics. First, open up a command prompt (Start->Run, type “cmd.exe” and hit Enter. On Vista just click Start, type “cmd” and hit Enter. On Linux… you should know how to do this. ) Browse to the directory you saved Zcompress to. Make sure you have also saved a clean Zelda3.smc ROM file there as well. Now, type:

zcompress.exe 0 Zelda3.smc Zelda3Gfx.bin

…on Linux, you’ll have to do:

wine zcompress.exe 0 Zelda3.smc Zelda3Gfx.bin

Now, open YY-CHR and select “File->Open”, then browse to your Zcompress directory. In the “File Name” box, type the following and press Enter:

*.bin

This allows you to see .bin files. Now, select your Zelda3Gfx.bin file. The window will load with apparent garbage. You can fix this by clicking on the “Format List” (currently, it shows “4BPP MSX/Genesis) and selecting “4BPP SNES”. Hit PgDown or PgUp to browse through the tileset; you’ll notice that the shapes are familiar but the colors are all wrong.

We've Found Mushroom Boy's Ghost!

It’s perfectly fine to edit the sprites using these colors, but you can make them much more accurate with just a little extra effort. The following is taken directly from the Perfect Guide:

1. Decide which sprite you want to edit, and locate that sprite in-game. We’re going to edit Link’s shield, so play through the game in Zsnes until Link’s uncle gives you his shield. Now, save a state (F2) and note the state’s id.
2. Close Zsnes, and locate the file that was just saved. For example, if you see “State 0 saved”, you will be looking for Zelda3.zst; if it’s “State 7 changed”, you’ll need Zelda3.zs7. Copy this file into the Zcompress directory, to make things easier.
3. Now, click on “Palette” then “Emulator State Load”. Browse to the Zcompress directory and load your .zs* file. (You might not be able to see it; just type *.zs* into the file window, similar to how you located the .bin file.)
4. Scroll to your sprite in YY-CHR (the shield is roughly 85% of the way down the screen). Now, click on the palette arrows (circled in red, below) to scroll through all possible palettes. You might not find the exact palette; just find one that’s close.

Link's Shield, With A Slightly-Distorted Palette.

Now, edit the shield. Make it bigger. As you do this, make sure that you only use the colors already in the palette. Stealing from colors further along is a bad idea, as it makes it much more difficult to edit those other palettes later. Just make something simple, then choose “save”.

After you’ve saved, close YY-CHR; our Zelda3Gfx.bin file now contains our new shield graphics. We just have to re-insert them. Switch back to your command prompt (in the Zcompress directory) and type in:

zcompress 1 87200 Zelda3.smc Zelda3Gfx.bin

This will re-insert all of your graphics (including the shield) –it will start doing so at 0×87200, which is where a normal Zelda3 ROM expects to see graphics data. Zcompress has the ability to export all graphics as separate files; in this case, you will have to figure out their insertion addresses yourself. But if you always use one big .bin file, the address should always be 87200.

Open your ROM, and play until you get the shield. Don’t your new graphics look nice?

We've Added a Blue Face To Link's Shield, and Enlarged It.

Of course, we can do better than that! Go into the palette editor in Hyrule Magic and scroll down to “Shield 0”. Change the dark blue to white, the light blue to navy, and the white to red. Now, re-arrange your graphics in YY-CHR to look spookier. Ooh~~~ skull shield!

We've Changed the Face Into a Skull! Note That Link's Uncle Uses the Same Shield As His Nephew.

Are we having fun yet? Keep in mind that some graphics are more stubborn than others. I still can’t manage to figure out how to change the treasure chest graphics. Also, some tiles are mirrored, so if you’re editing, say, tombstones, you don’t have to bother looking for the “right half” graphics —they’re simply mirrors of the “left half” graphics.

Question: How Do I Edit Overworld Tiles? What’s This 16×16 Tile Nonsense?

As mentioned previously, Zelda 3 is hyper-compressed. This makes overworld editing slightly more complex than it should be. Zelda3 enforces a three-tier hierarchy on overworld graphics, the primary purpose of which is to support complex dirt paths, mountain cliffsides, and rivers —all without wasting tiles. It’s pretty natural once you get used to it.

Each world-map “tile” is 32 pixels square. This is just a convenience measure; you cannot, for example, assign properties to any of these tiles. Rather, you assign properties to the four 16×16 tiles that make up one world map tile. 16 pixels square is the “logical” tile size in Zelda. (For now, we ignore “big” tiles like the 32×32 rocks Link can pick up.) Each of these, however, is made up four 8×8 tiles which act as a “palette” to construct the larger tile from. If you have, say, a sign which is symmetrical about the Y-axis, it makes sense to simply draw the left-hand-side, and then mirror that tile on the right-hand side. That’s what 8×8 tiles are used for.

Edit: February, 2010

After a bit of discussion with Math on Napkins, I can now explain to you why tiles are stored in a 3-tier hierarchy like this. First: map16 tiles. These make the most sense; you interact with them. Excluding big rocks, each “actionable” object is 16 pixels square. The game engine (that is to say, Zelda3′s software) understands these. However, the SNES is hardwired to output 8 pixel tilemaps. So, each tile is stored in memory (that is to say, it is stored in “RAM”, when you step onto that map) in the SNES. Finally, the Zelda designers realized that there was a lot of redundancy in the overworld. Thus, they decided to store groups of four map16 tiles as map32 tiles in ROM (that is to say, on the game cartridge itself, to save space by acting as a rudimentary form of compression). Now, map8 and map16 are fairly sensible. But map32 is fairly unique to Zelda; most other games use “palettes” of tiles if they run out of room in ROM.

From this, Hyrule Magic has two options for dealing with map32:

1. Force the user (that’s you) to create/place/manage map32 tiles.
2. Allow you to place/manage map16 tiles, and then automatically generate map32 tiles when you save the game.
3. Use its own custom “compression” format, and patch Zelda3 to load this instead of map32.

Needless to say, #1 is a lot simpler and safer (from the developer’s point of view) and that’s why you have to learn to deal with map32.

Just thought I’d update this while it’s still fresh in my mind. Happy hacking.

End Edit

We’re tired of drawing our own graphics, so let’s just re-use existing tiles to make something new. We’ll put the top of a bush onto the bottom of a rock, thus making a sort of “chia pet” tile. This tile is a rock, but it’s also easy to pick up; you don’t need any special gloves. We’ll put this tile outside of Link’s house.

That’s our goal; now let’s do it. Open a clean Zelda3 ROM in Hyrule Magic and go to overworld area 2A. Now click on the “movement” arrows (in red, below) to scroll around until you find Link’s House. Hint: Just move once to the right.

Navigating Is Done With the Four Arrows Circled Here.

Remember the room ID (2C) and you can just open it directly next time, or use the “jump to” button. Now, make sure the “Select” button in the top toolbar is depressed, and click/drag to select the two rocks in front of Link’s house. Then, right-click on this square to zoom to the “palette” tile on the right. (You actually don’t have to left-click drag, but I like to have the same tile highlighted on the left and right halves of the screen.) You’ll end up with the following:

Left-Click the Left and Then Right-Click the Left to Load the Right. Easy.

Double-click on the palette tile (right-hand side) and it will open in the 32×32 block editor. You can see that it is composed of four 16×16 tiles: two rocks, and two “jump down” cliff tiles.

The 32x32 Tile Editor.

Right-click on each tile to see its palette tile load in the right-hand side of the 32×32 block editor. Now, scroll down through the various 16×16 palette tiles until you find the dark-colored rock tile. You need to be careful when making new tile graphics; some of them are reserved, some cannot appear on every overworld screen, and some have hard-coded properties. So, we’ll just edit an existing tile that we know is safe: the Lv. 2 rock tile. Double-click on this tile to open it in the 16×16 block editor:

The 16x16 Tile Editor Requires Some Finesse With Palettes.

On the left is your tile, composed of four 8×8 tiles. On the right are your “palette” 8×8 tiles. Unfortunately, Hyrule Magic cannot know which palette your map will be using, so the colors are a bit screwy. Try typing various numbers into the “palette” dialogue until the graphics normalize. Palette number “5” or “6” is a good choice. If you double-clicked on an 8×8 tile, you could edit it further, but we want to just build our new tile from the existing “bush” and “rock” graphics.

However, there are several different bush/rock graphics, so which one should we copy? Close the 16×16 tile editor and right-click on the original (Lv. 1) rock. Then, double-click on its palette entry. Finally, right-click on each of the bottom two tiles. They are both palette 5, block 434, and one of them has the “X flip” attribute set. Cancel, and do the same for a “bush” tile from the world map. You’ll see that the top two tiles are both palette 6, block 384, and one of them has the X-flip attribute set. Now that we know this information, we can easily construct our new tile:

1. Re-open the “black rock” tile in the 16×16 block editor.
2. Enter “5” for the palette.
3. Type “434” into the “Block” window. Left-click on the lower-left tile to place this.
4. Click the “X flip” check box. Left-click on the lower-right tile to place this.
5. Uncheck “X flip” and enter “384” for the block and “6” for the palette. Left-click on the upper-left tile to place it.
6. Check the “X flip” box and left-click on the upper-right tile to place it.
7. Change the “Block type” to “39”, the same type as bushes, so that you can pick it up and slash it with your sword.

You should now have a proper bush-rock-combination tile. Click “Ok” and then left-click on your new tile in the palette window, then left-click on each of the two rocks to change them to chia tiles.

We've Successfully Merged Rocks and Bushes.

Then, click “Ok”. The bushes in front of Link’s house should now appear as rock-bushes. It is possible, however, that your new tile is all black. You’ll first notice this in the 16×16 block editor:

Sometimes The 16x16 Palettes Won't Refresh.

If this is the case, simply select “Ok”, and then look for the “Palette” box in the overworld map window. Currently, the entry is “0”. Change it to “5”, then change it back to “0” to fix the problem.

You will probably also get a warning message when you save the game:

Editing the World Map Can Lead To Difficulty Saving.

This is because several areas share some common tiles (like the dark black rock you edited). Personally, I’m wary of selecting “Yes”, since I don’t trust the editor to “fix” this problem. Clicking “No” has one obvious side-effect, though: it replaces all black rocks with your rock-bushes. I find this acceptable; we’re only testing out the area near Link’s house anyway.

Now, simply play the game, and enjoy your new bush-rocks.

The Bush-Rocks Work... Mostly.

There are a few minor issues, such as the fact that these new tiles occasionally drop bombs, or the incorrect “full rock” graphic that appears when you pick it up, but the general effect is nice.

Lunch~

It’s lunch time and we’re not even halfway done. Grab a sandwich (I recommend a Reuben) and get ready for more barrels of fun when we return.

Question: How Do I Change Flute Locations?

Changing the 8 flute locations (we ignore the special 9th location) is quite easy to do, but it requires two rather unrelated steps:

1. Change the location of the flashing number on the world map.
2. Change the location the bird drops you off at on the proper Overworld screen.

Neither step is difficult. First, open Hyrule Magic and click on “World Maps” and then “Normal World”. Change the selected marker from “Hyrule Castle” to “Flute Locations” —the very last item in the drop-down list.

Eight of the Nine Flute Locations, As Seen On The In-Game Map.

Personally, I’ve always found location 8 to be a bit lame. Let’s move it to a more useful place: the Great Fairy’s island. To do this, click on the “Move” button in the top toolbar, then simply drag the pink 8 to the center of the lake.

Drag the Flute Location to the Island.

Save, and close the world map editor. If you opened the saved game now, you’d see the flashing number 8 in its new location, but the bird would still bring you to the old location. To fix this, open “Overworld” area 3F. Click on the “Transport” button to show the whirlpools and bird locations, and then right-click on “Fly-8” and choose “Remove”.

Hyrule Magic Will Complain If You Don't Remove the Old Flute Point First.

Now, move to area 35 and right-click on the island, then choose “Insert bird location”. Congratulations! You’ve moved the drop-off point.

Adding a New Flute Location is Easy Once You Know Where to Look For It.

Save your map. Before testing it out, you should probably short-circuit a few of the doors in the castle, to make evacuating Zelda easier. Also, put the Flute in the treasure chest in Link’s house. This way, you can start a new game, rescue Zelda, and activate the bird in Kakariko Village without wasting too much time. The end result is exactly what we expected, except…

The Map Is Properly Up-To-Date.

Link Also Zooms In To the New Location Correctly.

Our rock-bush tile is still in effect from the pre-lunch hack. This has some unintended consequences:

Did Anyone Else Notice That Those Ice Men Were Super-Damaging?

But that’s nothing you can’t handle, right?

Question: How Do I Edit The Game’s 3-D Graphics? How Complex Can These Graphics Be?

The answer to the second question is: not very. The combination of points and faces is limited to 104 units in total, but I’m not sure how these are tallied. (My personal guess? 3 units for each point you place, and one unit for each point in a face.) Also, the triforce segments and crystal shards share the same total point set. For now, you’ll just have to experiment with the editor to see what its limits are.

To answer the first question, open a clean Zelda3 ROM in Hyrule Magic and click the “3D Objects” tab. Then click once on the black, gridded area to focus it. Now, you can rotate the scene (arrow keys) and zoom (num pad plus/minus). The current view, of the crystals, is somewhat difficult to grasp, so hit the right arrow in the upper-left of the screen to switch to editing the triforce shards.

Hyrule Magic's 3D Shape Editor is Impressive, But Slightly Tricky to Use.

You should then click on each point on the top surface of the triforce to see its location in 3D-space, which will appear in the upper-left corner of the screen.

Click On A Point To Show Its Position In 3D Space.

Here, we can see that this point is located at X:A8, Y:58, Z:76. Click on the remaining points and jot them down. These coordinates, of course, are hex values.

Face	X	Y	Z
Top Face	A8	58	76
	58	58	76
	80	A8	76
Bottom Face	A8	58	8A
	58	58	8A
	80	8A	8A

The triangle is 0×14 points thick, with the bottom face flat on 0x8A and the top face flat on 0×76. The remaining Cartesian co-ordinates are distributed as an equilateral triangle:

Our Equilateral Triforce Piece.

We would like to replace the triangle with a nice hexagonal shape. However, if you try to add a point, you’ll notice that we don’t have any more points to work with. So, click the left arrow (at the top-left of the window) to switch back to the crystals, and then click on any one of the points. Hit “Backspace” to delete this point —the cursor will then automatically jump to the next point. Continue hitting “backspace” until all points have been deleted. Then, hit the “right” arrow to go to the triforce triangle, and delete all points here too.

Now, draw a rough sketch of your hexagon; it should be roughly the same size as the triangle. Drawing a sketch is important, as it allows you to identify vertices easily. You should end up with something like this:

Our Hexa-Force is At Least Twice As Powerful As the Triforce!

Now we have our six points in a single z-plane:

(6C,58), (94,58), (A8,80), (94,A8), (6C,A8), (58,80)

You’ll notice that I wrote these out in a counter-clockwise order. The reason will be important later: Hyrule Magic creates an “outward facing” shape when points are connected counter-clockwise.

Now, we need to enter these points. To enter the first point, (6C,58), do the following:

1. Click the “Add Points” button, and click once on the black area. This will pull up a blue line and two green lines, indicating the temporary placement of your point.
2. Click on one of the green lines to place your point. Place it anywhere; we’ll move it manually to make sure it ends up at the exact position we choose.
3. Click the “Move” button, then left-click once on your new point. Now, press the A or Z key until the point’s “Z coordinate” is at 76. Then, use the numeric keypad (this part sucks on a laptop) to shift the point’s X/Y coordinates to 6C and 58, respectively.

Our First Point, Positioned Correctly In 3D Space.

Continue to add all the points; the order you add them in is not important (only the order you connect them in). The Z-component will always be 76, since we’re only building the top face for now. Then, after all points have been added, rotate the view to the origin (the easy way: save, close, and re-open your ROM in Hyrule Magic), click the “Add Faces” button and click on each point in order. It doesn’t matter where you start, so long as you go in a counter-clockwise direction, and end at your starting point. As you click on each point, a blue line will trace the current shape you’re creating. You should have something like this when you’re done:

Our First Hexagon Surface. 1-Dimensional.

Save your file and close the editor. Start up the ROM, and you’ll see some blinking hexagons:

Our 1-D Hexagons Often Vanish, But Prove the Concept.

This is fine; the back face of the hexagons is invisible; we’ll fix that later. If, instead of loading, your game shows some glitchy graphics and then freezes up, you probably edited the crystal’s graphics instead of the triforce’s graphics (d’oh!). Just delete all points, hit the “right” arrow, and start again.

Let’s add the second set of points. Load your ROM in Hyrule Magic and go to the 3D Objects editor, pressing “right” to go to the triforce graphics. You should see the hexagon facing you directly; hold “Down” to rotate the screen until you are seeing the back of the hexagon. Now, add six new points to the object. They have the same X and Y coordinates as the previous points, but their Z-coordinates are all the same: 8A.

After you’ve added these points, hit “Up” two or three times so that you can tell these points apart from the first set. Then, connect all six in a counter-clockwise direction using the “Add Face” button. You now should have two hexagons facing away from each other:

Our 2 Faces of the Coin Have Been Added. One Should Always Be Invisible.

There’s a small chance that you’ll corrupt your first face by adding the points for the second. In this case, the “Delete Face” button might not even work! You’ll have to delete one of the points contained in the face. (In my case, there was a weird triangular shape; I just deleted and re-added one of the points of that triangle.) In general, you’ll want to add all points before you connect any faces.

Either way, you only have one more task before we’re done: just add each of the six rectangles that connect the top and bottom two hexagons. Rotate the window as you go, and make sure you connect all points in a counter-clockwise fashion. When you’re done, you’ll have a nice, coin-ish object.

My Favorite Picture From This Entire Post: Your Complete Coin.

Save and exit, then load your game. My only regret is that I didn’t actually do the math properly, so our coin is a bit “longer” than it should be.

Our Coins Rotate Into Position. Hooray, 3-D Editor!

Does anyone else find it interesting that bsnes and zsnes display the opening zoom-in differently? Also, although it should be painfully obvious by now, you can change the palette used for the triforce just like anything else in the game:

Go Rainbow Laser Triforce!

Question: How Do I Change The Music For An Area?

I’m pitching a changeup with this one. Sorry, but I need an easy section for once. Changing the music for an area is just a matter of tracking down the right dialog box. The one you want is in the “Overworld” editor, called “Properties”.

Here's The Overworld Properties Dialogue.

Since I’m punting on this one, I’ll at least explain this box properly. Besides a section that lets you change the sign text (by ID), all eight options relate to music and sound. The four categories relate to the four stages of the game:

1. “Beginning” — From the game’s beginning until you get Zelda to the sanctuary.
2. “After rescuing Zelda” — From the sanctuary until you retrieve the Master Sword (regardless of if you’ve visited the Dark World yet).
3. “After getting the Master Sword” — From the time you get the Master Sword (Lv. 2) until the time you get teleported to the Dark World from Hyrule Castle.
4. “After beating Agahnim” — From the time you defeat Agahnim and are teleported until you beat the game.

The “music” option is simple: What music plays for this section of the screen during the designated time period? Try changing all “music” to “Forest” to see what happens. You should be able to guess. The “ambient sound” setting loops a background noise constantly when on. Try changing the sound to “Light Rain” for all four time slots. Now, when you enter this particular screen, you will always hear rain and the music from the Lost Woods. In the Dark World, there is generally only one time slice available. Try mixing and matching various sounds and music tracks to get the effect you want; actually changing the music itself will be covered in an “Advanced” tutorial later.

Thanks for the easy question~

Question: How Do I Make An Enemy Drop a Key? How Do I Choose Which Enemy Drops the Key?

This is a reasonable question to ask; although long-time Zelda players know that certain enemies drop keys in certain rooms, the Perfect Guide says to just “put the key at 00,3C,2” and pretty much leaves it at that. The actual method is slightly more involved:

1. Open a clean ROM in Hyrule Magic. Go to Entrance 04, the main entrance to Hyrule Castle.
2. Click on the “Sprite” checkbox, and then right-click anywhere on the map and choose “Insert An Enemy”. Scroll through the list until you find the “Key” entry and choose Ok. Then, click and drag that sprite to location X:00, Y:3C. Now, press the hyphen (minus) key to make it jump to “BG2”. Finally, press the period key several times until its property (P) value is 18.
3. Now, press the left arrow key several times, to move to the first sprite in the room. Hit the right arrow key to scroll through the sprites one-by-one, and pay attention to the order of the sprites. Your key sprite will appear directly after the enemy that drops it.
4. So, click on your key sprite and press “V” or “B” to move it forward or backwards in the order until it is directly after the enemy you want to drop the key. You might have to use the left/right arrow keys to check, in case you lose track of the current order.

Make sense? Here’s a picture of my screen after I’ve edited the key sprite and positioned it properly. Note the properties tab (circled) and the order of the sprites (drawn as a red connected path). According to this screenshot, you can determine that the key will appear after killing the “Green Soldier” enemy.

Annotated Screenshot Showing How Enemy Order Determines Who Drops the Key. Also Note the Object's Properties.

Question: How Do I Add a Nifty Logo To The Title Screen?

This question’s more a matter of patience than anything. The “Menu screens” tab in Hyrule Magic lists a series of single screens that you’ll recognize from the game. For now, open the last menu: “Destination List”. The player sees this menu whenever he starts the game any time after rescuing Zelda. There are several problems with the initial view, however.

The Destination Picker Window.

For one, the menu box and the fairy used to select your location are not visible, making it very difficult to decide where to place the logo. This occurs because such graphics are sprites, whereas the background is just a static image. We’ll get around this problem by using placeholders.

The next issue is that we can’t even see the palette tiles that make up the text! Click on the BG3 radio button; ah, there they are.

An altogether different sort of problem is one you should be used to by now: the palette’s miscolored. Keep trying numbers until you get something that looks good; I used 6.

However, we’ve now reached the main issue: editing the last five menu items usually doesn’t work. This includes the “Destination List”, of course. I’ve tried adding tiles on BG’s 1, 2, and 3, with or without the “In Front” property specified, all to no avail. There’s a chance that changing the palette would fix the problem, but I’d had enough by then. Close the “Destination list”; we’ll have to edit something else.

Double-click to open the “Title Screen”, which is much easier to edit. The only thing I haven’t been able to add yet is BG3 sprites. (Actually, I know they’re in there, since the Master Sword is obstructed by their shadows. But I can’t get them to show up.) This isn’t such a big restriction, since we’ll be drawing our own graphics anyway.

Menu screens are somewhat like dungeon screens, which makes it surprising that the same editor logic isn’t used. Instead, we’ve got a bunch of random keys which help us accomplish our tasks. Make sure the “Move” button at the top of the window is pressed down, and click once on the blue sky above the word “Zelda”. Notice how several tiles were selected? This is a common theme in menu screens: a “block” of editable tiles. You can drag a block of tiles with the mouse, and increase or decrease its size with the period and comma keys. Pressing m changes if “size” refers to width or height. If you want all tiles in a block to have the same value, press the hyphen (subtraction) key to switch between normal mode and “fill” mode. Note that dragging a block to the edges of the screen can sometimes cause it to auto-fill the entire width, just like in the Dungeon Editor.

Before we place our logo (which will be 3×2 tiles), we first have to place a dummy sprite and make sure it shows up in the right place. We’ll use two blocks of 3 tiles. Switch to “Move” mode, and switch to “BG1”. (Note: Click the radio button BG1, not the check box BG1. The check box just toggles the visibility, to make editing easier.) Now, right-click somewhere below the bottom of Hyrule Lake. A new block should appear. Hit the period key twice to make it expand.

Right-Click With "Move" Selected To Add A New Block. Then Resize It.

Now, switch to “draw” mode, and pick any colorful tile. Click on it in the palette on the right, then left-click and drag over your new block to fill all three tiles in that block with your new tile.

Left-Click and Drag to Color the New Block. Hit Detection Is Sometimes Dodgy Here...

Switch back to “Move” mode, and right-click one tile below the block you just entered. Press period twice to expand it to 3 tiles. Switch to draw mode and color it with another random tile from the palette. Now, switch back to draw mode and drag both of your new blocks until they’re in a sensible position.

Our First Tentative Placement of Logo Stand-In Graphics.

Now, save your game, and load it in Zsnes. Your faux-logo should be in the right spot; if not, go back to Hyrule Magic and move it until it shows up properly. In particular, if you can’t see it at all, something is overlapping it. Or, you’ve chosen a bad palette. Try re-drawing your title screen graphics after first changing the “palette” in the Menu editor. I used palette 4 pretty effectively.

After Some Fiddling and Re-Coloring, Here's Our Logo Placeholder Graphics in Zsnes.

Now that we’ve got the logo in place, we just have to edit its graphics. You should know how to save graphics in 16-color format (see question 2), and you should know how to edit Link’s graphics by exporting them (question 1) or with YY-CHR (question 3). Either of these will work. First, go back to your menu editor, and select the proper palette. Then, double-click on one of the tiles in the right-hand box that you previously added. A new window should pop up.

The Import/Export Window for Menu Tiles.

You should know what to do here: copy and paste these graphics into Graphics Gale and save it in 16-color format. Then, open Paint.NET and edit some of the tiles (using only palette colors) to create a desirable logo. Make sure you don’t edit any of the tiles that appear in the title screen. Some of the all-black tiles are probably safe to edit, but if you don’t mind messing up existing graphics in other menus, just edit tiles that you know for sure don’t appear in the title screen. Finally, copy/paste your edited spritesheet from Graphics Gale back into Hyrule Magic, and then switch to “Draw” mode and click on each of your six new tiles to insert them where your placeholder graphics used to be. Congratulations, you’ve made a new logo!

Our Completed Logo. Note: The Graphical Glitches Are Unrelated To The Logo; I Just Messed Up The Import.

Of course, if you’re willing to track down and manage the palette colors yourself, you can probably make a much nicer logo. We won’t cover changing the palette on image imports, as the Perfect Guide has a few nice examples on that.

Bonus: How Do I Actually Manage a Large Project?

Instead of giving you homework, I’ll give you my own personal bit of advice on how to manage a real project with Hyrule Magic. Learning the program is fairly easy, and coming up with creative dungeon ideas is either simple or impossible —it’s more of an art than a science. So the only real challenge is the organizational aspect. What if you want to steal a palette color from Link’s shield to use in his hat? Even if you set aside the same color in all three “shield” palettes, you’ll still have to jot this information down somewhere. What if you want to play around with the enemy palettes available in each dungeon, to make, say, crystal switches available everywhere? Hyrule Magic’s occasional corrupting of your ROM doesn’t make this any easier.

The key idea here is that you should somehow extract a “source” version of Zelda3, and then re-insert it into the ROM when you want to open it up in Hyrule Magic. The ZCompress tutorial I wrote is a good example of this. Some programs go even further, re-writing the game’s ASM to force you to change the ROM. Lunar Magic sometimes does this, relocating pointers and the like. However, I see no need to toss out Hyrule Magic for, say, Open Zelda. Can’t we find a way to perform our extract-edit-insert cycle without mangling the ROM so badly that Hyrule Magic can’t open it? That way, we can try out new ideas (and do dungeon design, etc.) in Hyrule Magic, and take advantage of all the work done by its developers on understanding the Zelda3 format. On a related note, can’t we add some capability for backup?

I forsee two possible solutions:

• Use a makefile to “build” your ROM from a series of text and binary files. As you decide to edit a new part of Zelda3, you can create a text file representing that particular aspect in a useful way. Then, you can define logic for importing it into the ROM. You’ll also need some way to “extract” dungeon rooms from Zelda3, and save them as .bin files (just the HEX values, really). That way, you can back up your work as you go. If Hyrule Magic corrupts your data, just delete the ROM and run a “make” command to generate a new one. This approach also allows you to throw your project into a repository, making it easier for others to help out with.
• Use Microsoft Excel and VBA macros. Although there are several great programming languages out there, Excel is one of the few good 2-dimensional ones. Consider: you write a macro to import a PNG of Link’s poses and display each one in a separate cell. Then, you generate cells representing each palette color. Another sheet can contain information on Zelda3’s palettes and how they relate. Your code can then check and tell you if you have any free palette colors available, or if you’ve overrun the limit. Linking this up with a program like Auto Hot Key can give you primitive control over importing data back into Zelda3.

Personally, I’d try to combine the two. Try to define text/image/binary formats for each of Zelda3’s various data. When possible, keep things “native” (e.g., PNGs) and use a common naming convention (e.g., “Link_4bpp.png”, “Weapons_3bpp.png”) to allow your tools to enforce rules about, say, bit depth, that are hard to enforce using general graphical formats. Then, make Excel macros to import this data and generate whatever views you need. Perhaps, for example, you actually prefer editing maps using Excel. So make a map editor tab. Then, use a Makefile to generate a Zelda3 ROM from your text/binary formats. This allows you to backup into an SVN repository. Moreover, it allows you to play around editing things in Hyrule Magic. I imagine Hyrule Magic will be faster to experiment in than anything you could come up with in Excel. So leverage that. Each tool has a use. To me, Hyrule Magic’s use is more geared towards general editing of Zelda3 than managing a new Zelda3 project. But if you’re content fiddling with it for all eternity, then by all means just skip this section —and make constant backups!

Happy Hacking!

Addendum

A few questions were asked after the initial post. They were pretty reasonable, so I pecked away at them for a few days and came up with some answers.

Addendum: Question: How do I make a number of torches open a door when lit and make them burn out after a period of time if I do not do light them fast enough?

First of all, have a look at this passage from the Perfect Guide:

$04F0-$04FF Timers for torches. Starts at #FF and counts down to 0. Setting it before the torch is lit is a bad idea. It will not cause a torch to light, nor will it brighten the room. Note that this range probably indicates we can have up to 16 torches in an area.

This is in the section on “RAM Data”, and it has some interesting implications:

1. Torches probably burn out on their own.
2. Apparently we could (with some ASM hacking) make a puzzle in which several torches burn out after different amounts of time. That would be a fun puzzle.
3. Seems that we can’t have more than 16 torches in one area/puzzle. Be careful! Multiple rooms might still be considered the same “area”; you’ll have to test this.

The Guide’s content for torches is a bit sparse (as always), so let’s look at a real example from the game. It’s been a while since I played Zelda3, so the only example I remember is that traumatic moving platform puzzle in Turtle Rock. Open up a clean ROM in Hyrule Magic, and browse to Entrance 35. Then, move up and then right until you get to the puzzle room.

This Puzzle Occasionally Appears In My Nightmares

Click on the “Torch” button in the lower-right, and then right-click on the map. It seems you can add and delete torches, but there’s no special property stating what these torches do.

Click on the “More” button in the top panel to view the map’s extended properties. Notice the tag “Light torches to open”. Click “cancel.

Note the Entry for "Tag 1"

Click on the “3” in the “Edit” box and then click once on the door at the top of the screen. You will see that its “Type” is set to “12”. Maybe Type 12 is a door that opens when all torches are lit? Let’s try it out.

Close the Turtle Rock dungeon window and click on Entrance 7D. Now, click on the “Torch” option, left-click on the existing torch, and drag it closer to the upper-middle-left door. Save.

Add a Torch in the Upper-Left Corner

Now, in the “Edit” box, click on layer “3”, and then left-click once on the door near the torch. Now press “M” several times until the “type” changes to 12.

The Door Is Now Locked

Don’t forget to set Tag 1 to “Light torches to open”. Otherwise, lighting the torches won’t affect anything.

Save the ROM, and try it out….  it doesn’t work. Hmm…. a bit of Googling finds this tidbit:

“In order for light torches to open door/get chest to work, you'll need exactly 4 torches in the room, no more, no less.” (Source)

Good to know. Add three more torches to our room. (Click on “Torch”, then right-click and choose “Insert a torch”.) Make sure you can access all four from the top half of the room. Note that you might have to delete three torches from some other room in the game before Hyrule Magic will allow you to save the game.

Our Three Additional Torches

Save, close, and try it out. Yes! It works!

The Door Behaves As Expected

Also note that, as expected, the torches extinguish after a certain amount of time.

Addendum: How do I change were link lands when he falls in a hole to a lower level?

There are two types of holes Link can fall in: “Entrances” from the overworld (like the one we just edited, below the bush in Hyrule Castle) and “Multi-Storey” holes inside dungeons. Well, there’s a third type of hole; one that damages you when you fall into it, but your question wasn’t really about that. Your question is about the second type, but I’ll cover both here.

Let’s start with Overworld hole entrances. First, you need to specify which entrance the hole will lead to. Open your Zelda ROM and click on “Overworld” then Area 1B. This is the main castle area. Now, click on the “Hole” toolbar item, then find the tag “7D – Hole” on the map screen. The “7D” in the hole’s name refers to the entrance it leads to; remember from the previous question that we opened Entrance 7D to edit the room where you find your uncle.

Click once on “7D – Hole” and then use the N and M keys to change which entrance this refers to. Change it so that that tag now reads “09 – Hole”. Then, save the map.

The Secret Garden Entrance, Modified

If you start Zelda3 and walk into that hole, you’ll end up in the Desert Palace. You will “fall” directly onto the main entrance. If you open up Entrance 09 in the Dungeon tab, go to the “Starting Location” section and change “Y” to, say, 400, then Link will fall in slightly above the entrance.

You Can Now "Fall" In To the Desert Palace.

However, he will also appear in this spot when he walks in from the outside door, which will look weird (try walking out then in). So how can we fix this? Well, we’d need to use two Entrances: one for falling in, and one for walking in. Notice how the Hyrule Castle area uses Entrance 7D for falling in, and Entrance 32 for walking in. They both lead to the same room, but with different “Starting Locations”.

So how about multi-level dungeons? The Perfect Guide has this to say about holes in dungeons, in the “Room Information and Properties” section:

Hole/Warp: Directs the holes or warp tiles. When you fall through a floor or warp to another room it will always be in the exact place in the new room as you left in the old.

To see this in action, open Entrance 33, the Tower of Hera, and click on the “More” button in the top panel.

The Tower's Map Properties Dialog

Notice the “Hole/Warp” box has a value of 167? This is where you land if you fall into the big jars in the top part of the room. Just to spice things up, let’s add a new hole to fall into. Click on the “2” in the “Edit” tab, then right-click near the entrance and choose “Insert an Object”. Browse to object 0A4, and click “Ok”. Then, drag it somewhere noticeable, and save.

Our Main Entrance Is Now Hazardous!

Click the “Jump” button and type in 167 to go to the target of the hole. Now, insert an object (any object) near the bottom of the screen, on layer 1, so that we can test if you fell into the right room. Save.

Add Anything, Just So We Can Tell If We Landed In the Right Room

Now, go back to overworld area 1B, and change the hole under the bush to lead to Entrance 33. Save, and start the game. Fall into the first hole (near the castle) to warp to the mountain palace…

The Shortest Distance Between Two Points Is a Modified Entrance.

…and fall into the second hole to test that you end up at the exact same location in room 167.

We Landed In Our Abandoned Room

So there you have it! Zelda3’s all about room/dungeon/nonsense management, and warp tiles/holes are just another great example of where a little knowledge goes a long way.

[RM-3] Speech Balloons That “Type” Their Message

Blog Info: I’m mostly done reading a great 65816 (SNES) assembly guide; this guide is so sensibly written that I think I’m actually ready to do a fun text hack for my favorite game of all time. So this week’s hack will help me prepare by introducing the algorithm in a high-level language. Plus, I think the RPG Maker community will like this one.

Back-story: Ok, our last attempt at speech balloons kind of flopped. They stretched text, they jittered, and they just popped into existence. Their worst sin, in my opinion, is that they didn’t “open” like the default RPG Maker ones, nor could they “type” their text out letter by letter. Let’s fix everything in one fell swoop.

Goal: Spruce up our speech balloon code, particularly in regards to opening/closing them when the player walks within range, and typing their message text out letter-by-letter.

You should read through tutorial RM-2 before starting this task. Instead of using a clean (new) project, I’m going to start where we left off after that tutorial. Don’t worry, I flushed my own project and copied it letter-for-letter from RM-2. One thing to note:

• Your “Talkbox_Window” module needs to be in a location that the interpreter can find. For example, add it below the “Windows” tab, or “Materials”. If you just add it anywhere (say, at  the end of the file), you might get an error when you try to run the program.

I added this factoid to the original article, too. People, please tell me if you can’t get your code to work following one of my guides; I’ll be happy to fix it.

Step 1: Variable Width and Jitter Reduction
Add the following line to “Main” right after Graphics.freeze:

$Debug = Window_Base.new(544, 0, 544, 416)
$Debug.windowskin = nil

This just gives us an “offscreen” area to calculate stuff on. If you’re combining tutorial RM-1 with RM-2, you can ignore this, and just use the regular $Debug window for offscreen stuff.

Replace the Talkbox_Window class with the following one:

#A slightly improved RM-2 talkbox
class Talkbox_Window < Window_Base
  #RPG Maker defaults
  MAX_WINDOW_WIDTH = 544
  MAX_WINDOW_HEIGHT = 416
  WINDOW_PADDING = 32

  #npc is a GameEvent
  def initialize(npc, txt)
    #Figure out a proper size, and initialize it offscreen
    my_size = $Debug.contents.text_size(txt)
    super(0, 0,
       my_size.width + WINDOW_PADDING,
       my_size.height + WINDOW_PADDING)

    #Put off setting this variable until later
    @offset_y = -1

    #On top and invisible
    self.z = 90
    self.visible = false

    #Draw the text once
    self.contents.draw_text(0, 0, draw_width, draw_height, txt)
  end

  #Call this to update the talkbox’s position over an NPC
  def update_pos(npc)
    #Store the npc’s height if we haven’t done so yet
    @offset_y = npc.sprite_height if @offset_y==-1

    #Update this talkbox’s position
    self.x = npc.screen_x - self.width/2
    self.y = npc.screen_y - self.height - @offset_y
  end

  #Get the width of the client area
  def draw_width
    return self.width - WINDOW_PADDING
  end

  #Get the height of the client area
  def draw_height
    return self.height - WINDOW_PADDING
  end
end

The MAX_WINDOW_WIDTH and MAX_WINDOW_HEIGHT variables just happen to be the window sizes RPG Maker ships with. I couldn’t for the life of me find these anywhere in the documentation, so I just stored my own values here. The DEFAULT_PADDING represents how much space is used as “border space” by any RPG Maker window —if you don’t add this value to each window you create, you may  find that everything you draw  is “cut off”.

We cache the return value of sprite_height because we’re worried it will affect performance. We’ll describe why later.

The remainder of the code is pretty simple to understand, even though not all relevant functions have been implemented yet. We’re basically resizing the talkbox to hold all of its text, and centering it above the NPC in question.

Now we need to add all those missing functions. Open Game_Event and add the following two member definitions:

  def sprite_width
    #Easy if this is a tile-based event
    return 32 if @tile_id > 0

    #Harder if this is a sprite
    unless $scene.is_a?(Scene_Map)
      return 0   #A generic Scene isn’t good enough
    end
    return $scene.get_event_width(self)
  end

  def sprite_height
    #Easy if this is a tile-based event
    return 32 if @tile_id > 0

    #Harder if this is a sprite
    unless $scene.is_a?(Scene_Map)
      return 0   #A generic Scene isn’t good enough
    end
    return $scene.get_event_height(self)
  end

There’s probably a better way to get this information, but this will have to do until I get a chance to read all the RMVX documentation. Basically, we just return the size of a tile or the size of a sprite depending on what this event is represented by. Of course, our caching code will position talkboxes improperly if you keep changing your event’s graphics from a Tile to a Chara representation. You can remove the caching if this bothers you, but I find it to be an unlikely edge case.

As you might have guessed, we now have to edit Scene_Map. Add the following functions anywhere:

  def get_event_width(event)
    return @spriteset.get_event_sprite(event).width
  end
  def get_event_height(event)
    return @spriteset.get_event_sprite(event).height
  end

Often, you’ll find that the Scene_* classes contain the down-and-dirty details of the game engine, while the Game_* classes contain a higher-level “logical” set of data points. Why, I’d bet we could fix the screen jitter we discovered earlier by simply updating talkboxes here instead of in Game_Map. This is completely correct. Find the following line in Game_Map’s “update” method:

    update_talkboxes

…and remove it. Then, add the following line to Scene_Map’s “update” method, right after “@message_window.update”.

    $game_map.update_talkboxes   #Update NPC talkboxes

That’ll fix the 1-pixel jitter we noticed earlier. Why? My guess is that “@spriteset.update” (which occurs after “$game_map.update”) does a minor pixel correction on every sprite that’s scrolling. For Game_Map’s objects, this tiny offset isn’t important.

By the way, a lot of game objects have both a Game_* and a Sprite_* representation. We might consider doing the same thing for our windows; then we wouldn’t need this minor hack. However, I don’t think it’s important.

Add the following function to Spriteset_Map. The explicit loop this function contains, by the way, is the reason we cache @offset_y.

  def get_event_sprite(event)
    for character in @character_sprites
      return character if character.for_event?(event)
    end
  end

Finally, add the for_event? function to Sprite_Character:

  def for_event?(event)
    return event == @character
  end

All this is rather hackish, but it’s self-contained, so I don’t consider it such a sin. More importantly, it makes our talkboxes work! Create a few NPCs and give them long-ish textboxes. Have them walk around if you want. Start a new game and enjoy the much, much nicer talkboxes (note the walking cow).

Our New Boxes Auto-Size and Don't Jitter When You Walk.

Don’t worry about the fact that boxes appear on the map for a split second before moving above their respective NPCs; we’ll eventually start them all “closed”, which will fix this problem.

Step 2: Opening Boxes When In Range

We’ve come a long way, but  our talkboxes still need some major work. For one thing, it doesn’t make sense to show them all on-screen at once. Besides introducing an unnecessary bottleneck, this is also rather boring to look at from the player’s perspective. So, let’s start all talkboxes “closed” and then “open” them when an NPC steps into the farthest visible tiles. (We’ll close them if he steps out of bounds).

This will require oodles of code to be written. Phase one will be to make the windows open and type their text; the second phase will invoke this code when an NPC steps into bounds. We will borrow most of our new code from Window_Message. Here’s our new class:

#A version of a Message which appears over a single
# NPC’s head. Supports most text tags, and auto-sizing.
class Talkbox_Window < Window_Base
  #RPG Maker defaults
  MAX_WINDOW_WIDTH = 544
  MAX_WINDOW_HEIGHT = 416
  WINDOW_PADDING = 32

  #Create a new Talkbox_Window.
  #  @param npc – The GameEvent above which we show our talkbox
  #  @param txt – A String (or array of strings) which
  #               represents our message. (Arrays aren’t yet
  #               supported).
  def initialize(npc, txt)
    #Figure out a proper size, and initialize it offscreen
    #  Height is +3 to allow for g/p/y/etc.
    my_size = $Debug.contents.text_size(txt)
    super(0, 0,
       my_size.width + WINDOW_PADDING,
       my_size.height + 3 + WINDOW_PADDING)

    #Put off setting this variable until later
    @offset_y = -1

    #On top and invisible
    self.z = 90
    self.visible = false

    #Fully shrunken, and neither opening nor closing.
    self.openness = 0
    @opening = false
    @closing = false

    #Pause after opening? Note that we can’t use
    # self.pause; that'll set a bouncing graphic
    @passive_pause = false

    #Reset variables used to track the state of the
    # drawing routine
    @text = nil       #Our text buffer
    @contents_x = 0   #Next character’s X
    @contents_y = 0   #Next character’s Y
    @line_count = 0   #Lines drawn so far
    @wait_count = 0   #How long to pause (for \., etc.)
    @line_show_fast = false

    #Save our text-to-draw
    @orig_texts = Array(txt)
   end

  #
  # New functionality for text processing
  # 

  #Frame Update – All opening/closing/typing happens here
  def update
    #Handle opening/closing
    super

    #All other actions must wait for a fully-opened window
    unless @opening or @closing
      if @wait_count > 0
        #We’re pausing _within_ the text stream
        @wait_count -= 1
      elsif @passive_pause
        #The window is fully open; leave it open
        # (optionally, we can add code to scan for Input)
      elsif @text != nil
        #There's text we haven’t shown yet
        update_message
      elsif continue?
        #Open the window and show its text
        start_message
        open
      end
    end
  end

  #Start a new line of text
  def new_line
    @contents_x = 0
    @contents_y += WLH
    @line_count += 1
    @line_show_fast = false
  end

  #Should we be displaying the next message?
  # For now, the answer is “always”, but you can add your
  # own control code if you like
  def continue?
    return true
  end

  #Align all our text for processing into a single string with
  # embedded control characters
  def start_message
    @text = ""
    for i in 0...@orig_texts.size
      @text += @orig_texts[i].clone + "\x00"
    end
    convert_special_characters
    new_page
  end

  #Replace all special letters with single-character
  # equivalents. (Copied verbatim from Message)
  def convert_special_characters
    @text.gsub!(/\\V\[([0-9]+)\]/i) { $game_variables[$1.to_i] }
    @text.gsub!(/\\V\[([0-9]+)\]/i) { $game_variables[$1.to_i] }
    @text.gsub!(/\\N\[([0-9]+)\]/i) { $game_actors[$1.to_i].name }
    @text.gsub!(/\\C\[([0-9]+)\]/i) { "\x01[#{$1}]" }
    @text.gsub!(/\\G/)              { "\x02" }
    @text.gsub!(/\\\./)             { "\x03" }
    @text.gsub!(/\\\|/)             { "\x04" }
    @text.gsub!(/\\!/)              { "\x05" }
    @text.gsub!(/\\>/)              { "\x06" }
    @text.gsub!(/\\</)              { "\x07" }
    @text.gsub!(/\\\^/)             { "\x08" }
    @text.gsub!(/\\\\/)             { "\\" }
  end

  #Start a new “page” of input on the current message
  # box’s surface. Right now, we only have one page of
  # input, but there’s no reason you can’t add more.
  def new_page
    #Clear the background bitmap
    self.contents.clear

    #Reset our drawing state variables
    @contents_x = 0
    @contents_y = 0
    @line_count = 0
    @line_show_fast = false
    @passive_pause = false

    #Reset our text color to the default
    contents.font.color = text_color(0)
  end  

  #Called when a message has been printed in its entirety
  # (used below)
  def finish_message
    @passive_pause = true
    @wait_count = 10
    @text = nil
  end

  #This is the main text processing loop of the talkbox window.
  # It works by slicing one character at a time off of @text,
  # and either reacting to it as a control character,
  # or displaying it.
  def update_message
    loop do
      #Get next text character
      c = @text.slice!(/./m)
      case c
        when nil
          #There is no text waiting to be drawn
          finish_message
          break
        when "\x00"
          #Our custom "newline" character
          new_line
          #Multiple pages; left in for your reference
          #if @line_count >= MAX_LINE
          #  unless @text.empty?
          #    self.pause = true
          #    break
          #  end
          #end
      when "\x01"
        #\C[n]  (text character color change)
        @text.sub!(/\[([0-9]+)\]/, "")
        contents.font.color = text_color($1.to_i)
        next
      when "\x02"
        #\G  (gold display) -ignore
        break
      when "\x03"
        #\.  (wait 1/4 second)
        @wait_count = 15
        break
      when "\x04"
        #\|  (wait 1 second)
        @wait_count = 60
        break
      when "\x05"
        #\!  (Wait for input)
        break
      when "\x06"
        #\>  (Fast display ON)
        @line_show_fast = true
      when "\x07"
        #\<  (Fast display OFF)
        @line_show_fast = false
      when "\x08"
        #\^  (No wait for input)
        break
      else
        #Normal text character
        contents.draw_text(@contents_x, @contents_y, 40, WLH, c)
        c_width = contents.text_size(c).width
        @contents_x += c_width
      end
      break unless @line_show_fast
    end
  end

  #
  # The rest of our functionality is unchanged
  #

  #Call this to update the talkbox’s position over an NPC
  def update_pos(npc)
    #Store the npc’s height if we haven’t done so yet
    @offset_y = npc.sprite_height if @offset_y==-1

    #Update this talkbox’s position
    self.x = npc.screen_x - self.width/2
    self.y = npc.screen_y - self.height - @offset_y
  end

  #Get the width/height of the client area
  def draw_width
    return self.width - WINDOW_PADDING
  end
  def draw_height
    return self.height - WINDOW_PADDING
  end
end

What a whopper! Fortunately, this is fairly easy to understand: update() determines whether or not we should be calling update_message(), which reads letters one-by-one. All other methods are helpers, and their functions are relatively simple. I’ve got to hand it to the RPG Maker VX team: their code might be a bit cluttered, but it’s very easy to follow and understand. I’m consistently amazed at how sensible it is to hack the RPGMVX engine.

Now, add the following code in Game_Event’s update() method, right below check_event_trigger_auto:

#Update NPC talkbox
@npc_talkbox.update if @npc_talkbox != nil

This is much easier to understand; we just needed to hook up our frame update method when the Game Event calls its update method. Finally, add a new event to your map, with the following “Script” command:

for event in $game_map.events.values
  tkb = event.npc_talkbox
  next unless tkb
  tkb.openness = 0
  tkb.start_message()
  tkb.open()
end

Since windows stay open forever, we need a way to “re-type” the window, to make sure our system is working properly. Since we didn’t bother writing such a method (we will later), we just hooked it up manually for now.

Run your program; talk to your new NPC any time you want to reset every window. Ah…. Isn’t that nice? As an added benefit, our windows no longer show up on the title screen, since their “openness” only gets updated in the main game loop.

These TalkBoxes Show Their Messages Just Like a Normal RPGMVX Show_Message Box.

Before we make these talkboxes appear on demand, we have to decide when exactly we want to show them. I chose to wait for a talkbox to be fully onscreen vertically, and halfway onscreen horizontally before starting to open it. I made this decision partly because boxes are centered over NPCs, so a talkbox will now appear slightly before or after its respective NPC becomes visible. Add the following function to your Window_Talkbox class; it checks whether or not this box should be opened:

  #Is this box within the vertical boundaries, and at
  # least halfway within the horizontal boundaries?
  def in_range?
    #Store some marker variables
    mid_x = self.x+self.width/2
    min_y = self.y
    max_y = self.y+self.height

    #Check bounds
    return true if min_y>=0 and mid_x>=0 and
                   max_y<=MAX_WINDOW_HEIGHT and
                   mid_x<=MAX_WINDOW_WIDTH
    return false
  end

As you should have noticed, this function relies on the update_pos() function being called first. And, if you think about it, update_pos doesn’t work unless called in the Scene_* routine; calling it in the context of Game_* is a mistake. So, instead of juggling a whole slew of function calls, we’ll just append our code to update_pos(). Your update_pos function should now look like this:

  #Call this to update the talkbox’s position over an NPC
  def update_pos(npc)
    #Store the npc’s height if we haven’t done so yet
    @offset_y = npc.sprite_height if @offset_y==-1

    #Update this talkbox’s position
    self.x = npc.screen_x - self.width/2
    self.y = npc.screen_y - self.height - @offset_y

    #Now, show/hide we show this box?
    if @onscreen != in_range?
      unless @onscreen
        retype_text()
      else
        close_window()
      end
    end
  end

This relies on a new variable, @onscreen. Add it to your init function (anywhere)

@onscreen = false  #In-range?

Finally, you’ll need the functions retype_text() and close_window(). Add them anywhere:

def retype_text
  @onscreen = true
  openness = 0
  start_message()
  open()
end

def close_window
  @onscreen = false
  close
end

This should do the trick. The only thing left to do is to find the following lines in update’s “elsif continue?” branch and comment them out:

  # start_message
  # open

…otherwise, the window will always open itself, regardless of its position onscreen. Run your code:

Our Boxes Now Open When In Range, Completing Our Original Goal.

The nice thing is, since we check this every tick (instead of just when the NPC moves) it should work properly for teleporting to a new map and teleporting NPCs around the map.

Note: The documentation states that if a Window’s “openness” value is less than 255, its “contents” Bitmap won’t be displayed. I am kind of assuming that, if its “openness” is 0, the windowskin also won’t be displayed, and processing will be minimal. I feel this is a safe assumption, but I want to make it clear that, if you experience a performance hit, you might want to look here first.

Possible Improvements

This tutorial was rather longish, so I’ll only give you two assignments this week.

• It’s well known that inheritance simplifies code reuse. Yet, rather than reusing the code in Window_Message, we just copied it and deleted what we didn’t want. Wouldn’t it be better to abstract it into a common superclass, and have both Window_Message and Window_Talkbox subclass this new class? For our tutorial: no. Extracting this code could have broken both normal messages and our talkboxes. Our lightweight class only risks breaking itself, which makes it ideal for development. Now that we’re done proving the concept, however, it would be a good idea to create a common superclass. That’s your first assignment.
• We copied the convert_special_characters() function directly from Window_Message, but closer inspection makes it clear that only a small number of backslashed “special letters” are used. Might we extend the list of possible escape sequences? Since we extract the text for our message balloon directly from a ShowMessage event, this provides a very elegant way of specifying additional properties of talkboxes. For example, we might use \{ACCEPT} to mean “this talkbox responds to the Accept key”, and \{EVENT:2} to mean “call event 2 when the user presses Accept”. Perhaps \{CLOSE:10} could means “close this talkbox 10 ticks after all text has been displayed” —indeed, we’ll need a lot more control if we’re to use these things for cutscenes.

[REV-1] Current Exciting Hacks

Blog Info: I’m in a bit of a rut, time-wise. My latest project just developed a huge bug, and my current research project is also eating up a lot of time. I could tell you my plans for future blog posts, but I’m notorious for changing my mind. Instead, I’ve got a much better idea!

Back-story: We’ve been hacking away for a while; it’s time to take a second and reflect on hacks that other people in the gaming community are working on. To me, this is always a rewarding experience: you’ll find a lot of really creative and exciting ideas out there, and you’ll be able to refine your own ideas a bit too. This week, we’ll focus on ROM hacks, particularly my penchants of Mario and Zelda. Where possible, I’ll provide videos.

Super Mario Odyssey
Summary: A thorough hack of Super Mario World that somehow presents itself as “just a simple hack”.

My Thoughts: Except for a certain two player modification, this has got to be my favorite ROM hack. The reason’s simple: it’s the only one that’s well designed! Level design on most hacks is idiotic; the goal seems to be to get the player lost, confused, and frustrated. Let’s not forget that one of the most fun parts of the original SMW was improving to the point where you could run through the early levels with ease. Super Mario World Odyssey is a nice mix of challenging, fun, and epic grandness. It’s a shame that the developer hasn’t spoken up in years; this is one of the few hacks I’m actually looking forward to.

The Legend Of Zelda: Lost Isle
Summary: Technically not a hack. Based off a Zelda engine that was first reverse-engineered and then enhanced. Very thorough, very pretty.

My Thoughts: Apologies if I appear nonplussed, but I find even the copyright holders mess up Zelda quite often -so how can hobbyists hope to get it right? The dungeons in this game are needlessly confusing, and the overworld is basically dead. That said, compared to Parallel Worlds, this game is not bad. Sure, it’s got a huge world map, but it also implemented a useful teleport system to ease the burden of long treks through the jungle. And the world really retains the “feel” of the Zelda universe, which is clearly a credit to the designer’s skill. No offense intended to the Parallel Worlds team, by the way; that hack is vast and professional, but its general design is uninspiring and amateurish. Consider it constructive criticism: the game would be an excellent first draft, in the sense that seriously re-tooling half of it would do wonders.

SMW Zelda (Un-Named Hack)
Summary: For completeness, this hack combines the previous two games! It’s a 2-D, Mario-ish Zelda game!

Important Note: For some reason, all links to the SMW Zelda hacks (indeed, all links to the author’s work, including his SMW Metroid hacks) have been removed from Youtube. I’m not sure what happened…. and if anyone has a copy, could you please send me a link? Until then, I’ve linked to a video of an “auto” Super Mario World level. I’ve always liked the auto levels; they showcase some really random happenstance, but are obviously planned for by the developer. I particularly like horizontal auto levels, as it’s much harder to make than a simple vertical drop.

My Thoughts: I’m cautiously optimistic about this one. I’ve always wondered what a 2-D Zelda game would be like, but I always thought it’d be more like Hannah and the Pirate Caves. Still, I like the hacker’s take on this one. The Deku Leaf is not just a weakly disguised cape; Deku Nuts (not shown) go straight forward but bounce at an angle, and the shield repels enemies. I found Legend of Princess to be a disappointing attempt at a 2-D Zelda (except fighting Shadow Princess; that was amazing), but so far, “SMW Zelda” seems like it’s blending both games to come up with something new. I really, really hope this one turns out good.

Super Mario World: The Second Reality Project 1
Summary: An “old-school” Mario World hack, which means the Overworld and the blockset haven’t been modified; only the levels have been changed.

My Thoughts: I dislike most Mario hacks. Super Demo World was nice to look at and modestly creative, but it gave me headaches. Various other hacks, requiring you to pull off such stunts as spin-jumping while holding a P-switch, were mere tributes to their designers’ egos. However, SRP1 offers something none of those do: an “easy mode” which is actually playable without savestates. It’s incredibly challenging, but so far (World 6) I wouldn’t say it’s unfair. (Editor’s note: Worlds 7+ are unfair.) And because it doesn’t use nonstandard hacks, I can take it on the go. If all developers took the time to step back and actually try their game out as a player instead of a developer or hobbyist, I think we’d have a lot more easy mode hacks, and homebrew would be all the better for it.

Various Spriter Sonikku Tricks (not a hack)
Summary: Some fascinating (but simple) ideas from a hacker.

My Thoughts: As a bit of a parting shot, have a look at this guy’s hack videos. It seems like he strives for simple, elegant hacks with lots of stopping power. Consider the video above, which allows bombs to destroy an on/off switch. I can immediately think of two clever tricks (and one gimmick) leveraging this ability. Although I don’t particularly like his games, Spriter Sonikku’s approach to development is geared towards maximum effect. Excellent~

Further Work
The Second Reality Project: Reloaded is a re-write of TSRP1 which changes very little in the way of level design and adds a much more coherent backstory.

I have a theory that good designers, when constrained, can sometimes produce better work than when they have full freedom. The Reloaded hack, for example, encouraged FPI productions to reuse older levels and eschew most custom blocks and scripts. In the end, I found this game better than both TSRP1 and TSRP2. So, in the spirit of being creative under extreme pressure, here’s your assignment for this week:

• Take a hack you enjoy somewhat, and pretend that your friend has asked you to spruce this up for a 1.1 release. Design (on paper; no code necessary) an improved hack which modifies very little from the original game but feels much more consistent overall. Try to write a few sentences describing the overall feel of the game, and then pinpoint parts of it that clash with this. Make sure you only suggest improvements which you know are possible. Try to limit yourself; pretend that you have a few friends to help you (for time-consuming tasks like replacing graphics), but you only have 6 months to develop the new version.

Design documents are due on my desk by Monday, although to quote a random internet source, I will also accept the excuse “I’m too lazy to do it”.

[EL-1] Game Event Notifier

Blog Info: I’ve made a habit of switching topics often on this blog. This might frustrate some of my readers, but believe me, it’s much healthier than constantly revising one project over and over. Besides, since none of these projects are real money-makers, we might as well build as broad a tool-set as possible. So in this guide, we’ll learn a bit about Win32, one of the oldest stable windowing toolkits in existence.

Back-story: Everyone has their MMO of choice. I play Eternal Lands because it’s very, very slow-paced, so I have no chance of getting hooked on it. It’s also quite balanced, and has beautiful scenery and a believable set of systems. You could really tell a story using that world. That said, some minor in-game features really bother me. For example, when you are gathering (“harvesting”) vegetables and minerals, you often unearth some nasty surprise that causes your character to pause. Since harvesting is meant to be the kind of thing you minimize the game for, this means that you’ll be constantly bringing the game back just to check if you’ve been interrupted.

Goal: Figure out a way to notify players when they stop harvesting. Perhaps a pop-up notification like Google Talk provides?

A note about Linux: Despite the fact that Eternal Lands is actually easier to compile under Linux, I’ll be developing this plugin under Windows. GTK fans should have enough knowledge to be able to port this example to Linux. My reason for doing this is simple: on my machine, Eternal Lands runs better on Windows. I’m all for equality, but I’m not going to handicap myself for the sake of fairness.

Getting the Source
Eternal Lands is rather unique in that the client is open-source. I find this rather trusting of the developers, although presumably they did it so that they could get features patched in by the user community. This makes our project a lot easier, so let’s grab the source right away. You’ll need Tortoise CVS. (Note that Tortoise CVS works fine on Windows Vista, as long as you don’t try to perform a CVS Checkout in a protected directory like C:\windows.) The source is located at the project’s BerliOS repository. To access it, create a folder called “EL Client” somewhere on your system. Open the folder, right-click, and choose “CVS Checkout…”. Copy and paste the CVS Root from the source page into the corresponding checkout field; it’ll look something like this:
:pserver:anonymous@cvs.elc.berlios.de:/cvsroot/elc
Then, enter elc for the module name. You can now click Ok:

Entering the CVS Root and the Module Name enables the "Ok" button.

The entire project contains lots of files, so you’ll have to wait a while for them all to download. While you’re at it, make sure you’ve downloaded and installed the Windows client of Eternal Lands from the main site. (You might consider the Down Them All plugin for Firefox, as the Eternal Lands site will occasionally yahtzee your connection.) This will make running the .exe you compile much easier.

Theory, Approach, and Not Getting Banned
While our downloads chug along their merry way, let’s talk design. First of all, what’s our end goal? That’s easy: “When the client shows the message You have stopped harvesting we should show a popup near the system tray if the Eternal Lands client is minimized.”

Next question: is this legal? Modification of the client code is fine, since the license is very liberal about the source. Actually using it to connect to the game requires that we subscribe to the terms and conditions of Eternal Lands. The only one that applies to us is:

[No] macroing, automating, scripting, exploiting bugs...

However, I don’t think we’re in violation of this. We’re not macroing, and the only thing we are automating or scripting is the use of Alt+Tab. I don’t think Entropy would insist that we constantly check up on our harvesting progress, especially since shell scripts in Linux to check the message logs have been received no hostility on the forums. In fact, this notification actually maximizes the ratio of player-controlled time to time wasted standing still.

Well, it’s nice that we won’t get banned for doing this, but how will we actually program it? There are two approaches I can see: either modify the game to display the window itself, or create a separate program that intercepts packets sent to Eternal Lands and determines if any of these packets represent the You have stopped harvesting message. There are pros and cons to each approach:

Modifying the Game Client
Pros

• You won’t lose any speed. Checking each message and responding only for a few specific ones will not affect performance.
• Modifying the code itself is easy; you won’t have to play with firewalls or packet interceptors.

Cons

• You’ll have to be able to compile the client. For some games (especially cross-platform ones like Eternal Lands) this can be a show-stopper; it’s difficult!
• You’ll need to re-apply your fix every time a new game client is released. This should be easy, but major releases (especially ones that use new libraries) might bring up the issue of compiling again.

Intercepting Packets
Pros

• No modification to the client is necessary. You can easily start the client with or without this feature enabled.
• When a new client is released, your code might continue to work! In fact, it will only fail if the connection protocol or information protocol changes, which is something the developers will probably avoid.
• You can develop in your favorite IDE; you don’t have to compile their code first. Consider: let’s say that we’re running a Windows game in Wine; compiling the client could be a nightmare!
• The result is slightly easier to port than modifying the code.

Cons

• The concept is more difficult; we’re dealing with packet nonsense instead of strings.
• If your code isn’t fast, it might cause your game to lag, since you’re dealing with all packets, not just the rare console message.

Since this is our first tutorial on Eternal Lands, we’ll modify the client code directly. In our next lesson, we’ll create a packet interception program, and polish up the interface so it looks real shiny-like.

Compiling the Code
The Official Eternal Lands Client Development Site has instructions for compiling the client using Dev C++. I followed these instructions and they worked perfectly. If this link is ever broken, you’ll have to figure it out on your own, or you can try compiling the client on Linux and re-working this tutorial from there.

You might want to remove debugging symbols from the compiled executable; this makes it about 77% smaller. To do that, add the line strip –s $(EXE) to the end of your $(EXE) target. In other words, in Makefile.win, find the following:
$(EXE): main.o $(TMP_LIB)
$(LINK) $(CFLAGS) $< -L. -lelc $(LDFLAGS) -o $(EXE)
..and replace it with:
$(EXE): main.o $(TMP_LIB)
$(LINK) $(CFLAGS) $< -L. -lelc $(LDFLAGS) -o $(EXE)
strip -s $(EXE)
Although the original 11.1MB isn’t too big, the resultant 2.56 MB is small enough to email to a friend.

To run the program, you should follow their instructions for running a local copy. In particular:

Copy the contents of the “Eternal Lands” directory from the official client release (e.g., C:\Program Files\Eternal Ladns) to a new directory called “Release” in the “EL Client” directory where you saved the source.

After you correctly compile the source in Dev C++, you’ll have a file el.exe in the “EL Client” directory. Copy that into the “Release” directory, over-writing the file that’s there.

Copy the files libcal3d-12.dll and alut.dll from C:\Dev-Cpp\dll and C:\Dev-Cpp\lib (respectively) to Release, over-writing the files there.

Run el.exe in the “Release” directory.

Wow! I don’t know about you, but for me, simply re-compiling the client caused it to run about 30% faster! This is particularly astounding, since I wouldn’t expect a naïve build to optimize better than the developer’s build. Someone should try compiling this thing in Visual Studio with PGO and seeing if we can make it even faster.

Our Client Compiles, And It Looks Great!

Adding a Window at Startup
The next logical step is to create a simple window as a proof-of-concept, and show it when Eternal Lands starts up. We’ll be using pure Win32 C to code this.
In Dev C++, click “File->New->Source File” and choose “Yes” when asked if you want to add this file to the current project. Then, immediately “save” the file as taskbaralerts.h, with a type of “Header Files”. Do the same for another file, taskbaralerts.c, of type “C source file”. These will contain the bulk of our implementation; it’s a very good idea to keep as much code as possible out of the official EL source files.

Enter the following into taskbaralerts.h:
#ifndef TASKBARALERTS_H
#define TASKBARALERTS_H

//Includes
#include <stdio.h>
#include <windows.h>

//Function prototypes
LRESULT CALLBACK AlertsWndProc(HWND hwnd, UINT msg, WPARAM wParam, LPARAM lParam);
void initAlertsWindow(HINSTANCE hInst, char *windowClassName);

#endif //TASKBARALERTS_H

The #ifndef/#define/#endif triplet is known as an Include Guard, and it prevents us from defining multiple versions of “initAlertsWindow”. Now, any file that wants to use these functions can simply include taskbaralerts.h and everything will run smoothly.

Of course, we need to actually implement these somewhere. Add this to taskbaralerts.c:

#include "taskbaralerts.h"

//Global variable
HWND alertsWindow = NULL;
HWND hStatic = NULL;
BOOL windowVisible = FALSE;
HINSTANCE myHInst = NULL;
HBRUSH bkgrdBrush = NULL;

//Message-handling callback
LRESULT CALLBACK AlertsWndProc(HWND hwnd, UINT msg, WPARAM wParam, LPARAM lParam)
{
switch(msg)
{
case WM_CREATE:
{
hStatic = CreateWindowEx(WS_EX_LEFT, “Static”, “”,
WS_CHILD | WS_VISIBLE,
5, 10, 180, 50,
hwnd, NULL, myHInst, NULL);
SetWindowText(hStatic, “EL Notification Area”);
break;
}
case WM_LBUTTONDOWN:
{
//Hide our window
windowVisible = FALSE;
ShowWindow(hwnd, SW_HIDE);
break;
}
case WM_CLOSE:
{
//Over-ride: Hide our window
windowVisible = FALSE;
ShowWindow(hwnd, SW_HIDE);
break;
}
case WM_DESTROY:
{
//Close the window for good.
PostQuitMessage(0);
break;
}
default:
return DefWindowProc(hwnd, msg, wParam, lParam);
}
return 0;
}

void initAlertsWindow(HINSTANCE hInst, char *windowClassName)
{
//Declare c-style variables
RECT r;
WNDCLASSEX wc;

//Save
myHInst = hInst;

//Necessary?
if (alertsWindow!=NULL) {
return;
}

//Make a brush
bkgrdBrush = CreateSolidBrush(RGB(0, 0, 0));

//Set a window class’s parameters
wc.cbSize = sizeof(WNDCLASSEX);
wc.style = 0;
wc.lpfnWndProc = AlertsWndProc;
wc.cbClsExtra = 0;
wc.cbWndExtra = 0;
wc.hInstance = hInst;
wc.hIcon = LoadIcon(NULL, IDI_APPLICATION);
wc.hCursor = LoadCursor(NULL, IDC_ARROW);
wc.hbrBackground = bkgrdBrush;
wc.lpszMenuName = NULL;
wc.lpszClassName = windowClassName;
wc.hIconSm = LoadIcon(NULL, IDI_APPLICATION);
if(!RegisterClassEx(&wc)) {
alertsWindow = NULL;
return;
}

//Get the desktop
GetWindowRect(GetDesktopWindow(), &r);

//Create a handle to the window
alertsWindow = CreateWindowEx(
WS_EX_TOPMOST | WS_EX_NOACTIVATE,
windowClassName,
“Eternal Lands Notification”,
WS_CAPTION | WS_SYSMENU, //Simple
r.right-200, r.bottom-100, 200, 100,
NULL, NULL, hInst, NULL
);
if (alertsWindow==NULL) {
return;
}

//TEMP: Show the window
windowVisible = TRUE;
ShowWindow(alertsWindow, SW_SHOW);
UpdateWindow(alertsWindow);
//TEMP end
}

That’s a lot to digest. Let’s start at initAlertsWindow(), which the main game should call when it first loads. First, we check if alertsWindow is not NULL, which could happen if, say, the game restarts itself by re-running main(). Next, we fill out the parameters of our WNDCLASSEX struct. (The MSDN has very good documentation (except, occasionally, when it’s totally wrong!) on its Win32 API, so I’ll be linking to that quite a bit.) The parameter of interest is lpfnWndProc, which points to the function which handles messages for this window. We set it to AlertsWndProc. This means that every time a message is dispatched to our alerts window, the AlertsWndProc function handles it.

Now, we try to register this window class. The function call might fail if, say, Eternal Lands has already created a window class with this name. At any stage along the way, if a function call fails, we simply return. Now we are ready to create the window. However, since we want to position it on the lower-right corner of the screen, we’ll need to know how big the desktop is. So, we call GetWindowRect(GetDesktopWindow(), &r), which stores the desktop’s dimensions in r. Now, we can call CreateWindowEx. Parameters of interest include:

• The dwExStyle is used to describe what type of window we want. WS_EX_TOPMOST means that we want our window to appear above all other non-topmost windows, and WS_EX_NOACTIVATE means that we want our new window to show, but not to take keyboard focus or have a section on the taskbar. This makes sense; if we are typing in Microsoft Word when we stop harvesting, we want the notification to appear above Word, but we don’t want it to take the focus away from word.
• The dwStyle parameter describes what our window looks like. We give it a WS_CAPTION, and add a WS_SYSMENU so that the user can “close” the window if he wants to. Nothing fancy here.
• The four paraemeters x, y, nWidth, and nHeight position our window on the lower-right corner of the screen. You can fiddle with it if you want to display it above the taskbar; it doesn’t really bother me.

That’s all there is to creating the window, and showing it is easy enough. The AlertsWndProc function simply switches through the variety of message types we can receive, returning “0” if it sees nothing of interest (which tells Windows to handle the message itself). In the WM_CREATE message, we add a static sub-window (called a “Label” in other languages) so that we can potentially display multiple messages in the future. The WM_LBUTTONDOWN message tells our window to hide itself; this lets the user simply click on the notification window to make it go away. We also intercept the WM_CLOSE message, because we don’t want the “X” button to actually close the window. WM_DESTROY functions normally for destroying a window, since closing Eternal Lands will also close all of its windows. (Don’t worry, this code won’t spawn any hidden processes that you’ll have to Ctrl+Alt+Delete).

There’s a few things left to do before we actually run this:

1. In Makefile.win, find the line that begins with COBJS= and add taskbaralerts.o in between tabs.o and text.o. If you were paying attention you’d notice that although we include taskbaralerts.h in several places, nothing links to taskbaralerts.c! This addition ensures that the linker can find your functions.
2. In main.c, add #include “taskbaralerts.h” at the top of the file. If you want, you can add an #ifdef check so that the client will still compile under Linux, but it’s not really necessary. Now, find the line init_stuff(); and add initAlertsWindow(hInst, “EL_Taskalert”); after it. This will initialize our taskbar alert code right before Eternal Lands starts rendering the main game.
3. Unfortunately, hInst isn’t defined. Find the line char *win_command_line; at the top of the file, and add HINSTANCE hInst; after it. Then, find the function APIENTRY WinMain and add hInst = hInstance; after int argc; —that was easy, right?

So, shall we run this code? Compile it, and copy el.exe into the Release directory. Run it, and you’ll see our window pop up about halfway through the boot sequence. If you’re in full-screen mode, this will be very obvious.

Our Notification Is Created When EL Starts

Intercepting Messages
Delete the lines we wrapped with the “TEMP…TEMP end” comments so that our window won’t show. It’s time to start fiddling with EL’s messages. In Dev C++ click “Search->Find In Files”. This allows us to search every file in our project for some text, and is indispensable when programming in C. Let’s try to find a message —any message— and go from there. What’s the most unique word in Eternal Lands? How about “radon”? Type it in and click “Ok”. The highlighted result looks particularly promising; double-click and you’ll be brought to the code that contains this —yep, it’s a big array of message strings:

We've Found The Radon Message (It's Misspelled As "Radeon" In One Location)

The name of the array is temp_event_string; scan the file a bit and you’ll find that this gets sprintf’d into search_str, replacing the %s with username_str. Well… that’s a tangential bit of useful information. Do a “Find In Files…” on username_str and you’ll see that it’s defined in interface.h. So, if we include interface.h, we can access our player’s name. Getting back to the problem at hand, do a “Find in Files…” on catch_counters_text (the name of the function which accesses search_str), and you’ll see that text.c calls this in filter_or_ignore_text(). We just completed a bottom-up search; if you did a top-down search starting from the TCP layer, you’d end up at the same place. Go to the top of the function, under #endif // NEW_SOUND, and put in a call to check_taskbar_alerts(text_to_add); —making sure to #include “taskbaralerts.h” somewhere at the top of the file. Now, switch back to taskbaralerts.h and add a function prototype:
void check_taskbar_alerts(char * text);
In taskbaralerts.c, add #include “interface.h” at the top of the file so we can access our player’s name. Now, before we go any further, we have a small problem. You see, the mess you receive isn’t exactly equal to “You stopped harvesting.” There’s a character at the beginning which specifies the color of the string. We could simply strcmp on “\x7FYou stopped harvesting.”, but this breaks our code if the color of this message changes. What we want is a function named “stringcontains”, which we’ll write next. (Of course, if a player now PMs you with the message You stopped harvesting, this will also set off your alarm. However, I think it’s better to react to this case then try to prevent it.) Add the following code at the top of taskbaralerts.c:

BOOL strcontains(char* src, char* pattern)
{
//Init some tracking variables
int i;
int len;
int patIndex = 0;
int patLen;

//Loop
len = strlen(src);
patLen = strlen(pattern);
for (i=0; i<len; i++) {
//Have we reached a non-match?
if (src[i]!=pattern[patIndex]) {
patIndex = 0;
}

//Continue tracking our pattern
if (src[i]==pattern[patIndex]) {
if (patIndex==patLen-1) {
return TRUE;
} else {
patIndex++;
}
}
}
return FALSE;
}

This function is simple enough; whenever it matches the first letter of the pattern with the current index in the string, it simply assumes it has a match, and resets to zero if that assumption is broken. This will fail if you’re looking for ABAC in the string ABABAC, so you can re-write it later (when we test for usernames) if your username is something funky like that.

Now, let’s hook this function up:

//Bring up our window?
void check_taskbar_alerts(char * text)
{
//Can we track this at all?
if (alertsWindow==NULL) {
return;
} else if (windowVisible==TRUE) {
windowVisible = FALSE;
ShowWindow(alertsWindow, SW_HIDE);
}

//Have we stopped harvesting?
if (strcontains(text, “You stopped harvesting”)==TRUE) {
//Prompt the user
SetWindowText(hStatic, “You stopped harvesting.”);
windowVisible = TRUE;
ShowWindow(alertsWindow, SW_SHOW);
UpdateWindow(alertsWindow);
}
}

Again, simplicity rules the day. This function checks each message as it is received. If the alertsWindow is still visible, that means the user forgot to close it, and we do that for them. Since users are prompted when they begin harvesting, this ensures that the window won’t give any false positives. We then use our new strcontains() method to check if our message looks something like “You stopped harvesting”. If so, we change the text of the window and make it visible. Compile the project and run it. It works! —but there’s one obvious flaw.

Only Displaying the Notification if We’re Not Viewing Eternal Lands
That’s right —the notification appears whenever we stop harvesting, even if we’re currently watching the Eternal Lands window. This is annoying as it grabs us out of full-screen mode. So what should we do? Well, the Eternal Lands window’s title bar contains the caption (Userame on main) Eternal Lands. The “main” changes if you switch servers, so let’s just use our strcontains function to search for the user’s name and the phrase Eternal Lands, and not show the window if both are present. Change our check_taskbar_alerts function to read:

//Bring up our window?
void check_taskbar_alerts(char * text)
{
//Can we track this at all?
if (alertsWindow==NULL) {
return;
} else if (windowVisible==TRUE) {
windowVisible = FALSE;
ShowWindow(alertsWindow, SW_HIDE);
}

//Have we stopped harvesting?
if (strcontains(text, “You stopped harvesting”)==TRUE) {
HWND foreWnd = GetForegroundWindow();
char wndTitle[200];
GetWindowText(foreWnd, wndTitle, 200);

//Does the window title contain the user’s name and the words “Eternal Lands”?
if ((strcontains(wndTitle, “Eternal Lands”)==TRUE) && (strcontains(wndTitle, username_str)==TRUE)) {
//We’re in the game! Do nothing
} else {
//Prompt the user
SetWindowText(hStatic, “You stopped harvesting.”);
windowVisible = TRUE;
ShowWindow(alertsWindow, SW_SHOW);
UpdateWindow(alertsWindow);
}
}
}

Try it out, and you’ll see that everything works just fine.

Possible Improvements
There’s a lot of potential for improvement, since we did such a hasty job designing our plugin. Here’s some particularly sore points; add your own and fix it up!

• If we stop harvesting, the message displays properly. However, it will disappear if another message occurs before we maximize the game client. Moreover, if we see the message and then open the client and tell our character to walk towards a storage (a common response to a full inventory) the message will remain until another message preempts it. Both of these glitches imply that we were wrong to assume that we should hide the alert window when a non-harvesting message appears; rather, we should hide it when the client gains focus again.
• It would be nice to have an icon in the status bar that our window is attached to.
• We should probably give our EL notification box a more EL-ish feel. We could do this with some simple GDI programming.
• We should be able to track more notifications, and display nice pictures related to their content. Like… if we find gold, etc. And, we could have our system tray icon flash bright red if we die (or green if we find a rare stone!).
• In fact, all of our suggestions are edging towards a general re-design of the plugin we made. But in that case, I’d rather not program it in pure Win32 C. If you’re of a different mind, by all means implement these changes. Me, I’ll be brushing up on .NET, Java, and Python. See you next week!

[LZ-2] Bow & Arrow Quest

Blog Info: These blog posts are supposed to take you about an hour to do, and me about three hours to write. That proved to be a gross underestimation –the prototype ROM for this post alone took at least three hours, and writing up step-by-step instructions has taken a few hours too. But I think we really accomplish something in this post.

Back-story: We decided in the last post to modify Zelda3 to make the bow-and-arrow the primary weapon for the first part of the game. But how do we actually go about doing this? And, how can we make the first part of the castle more challenging for advanced players? An arrow-related puzzle is in order! But this means we’ll have to start learning how to use (shudder) the Dungeon Editor. (There is no Dragon Editor).

Goal: Become proficient in using the Dungeon Editor, remove the sword, add the bow, and modify the first dungeon room you fall into to require the clever use of arrows to exit.

Step 1: Understanding Backups

Open a clean copy of Zelda3 in Hyrule Magic. Make SURE this is a copy, because I’m going to show you how simple it is to permanently muck it up. Expand “Dungeons” and click “Starting Location 03”, then click on the “More” button at the bottom of the dungeon screen (near “Starting Location”). See the radio button for “Upper Left”, next to “Horizontal Scroll”? Click that:

Clicking This Radio Button Will Definitely Crash Zelda3.

…and then click “Ok” and save. Start the ROM…. and notice that it crashes after the name select screen. Ok, go back to the “More” button and change the setting back to “Upper Right”. Save, start the ROM…. it’s still failing to work.

With one simple click, we have irreversibly corrupted our ROM. Sure, you can try to HEX it back to the status quo, but that could take hours. A better solution is to keep regular backups, and to do your most dangerous fiddling directly after you backup. Really, I’m not kidding when I say that you need to back up your working ROM far more often than usual.

Step 2: Understanding the Dungeon Editor

Open a clean copy of the Zelda3 ROM in Hyrule Magic. It’s time to play around with the dungeon editor. Open an MP3 player, too, for your sanity.

Expand the “Dungeons” tab and browse down to “Starting Location 04”. Double-click, and you should see a familiar room: the entrance to the sewers. In the top-left corner of the window it says “Room 81”. Next to this are some arrows to switch to the next room in each of the four directions. Finally, there is a “Jump” button which allows you to go immediately to a room by number. Click on the arrows a few times to see where it takes you, and then click on “Jump”, type in “81” and hit “Ok”. You’re now back at the throne room.

The Jump to Room Box

Close the dungeon view and scroll down to “Entrance 32” & open it. This is the room you fall into after pulling up that bush outside the castle. Actually, that’s somewhat inaccurate. This entrance only takes effect when you walk out of and then back into this room. It is the same room as the one you fall into (and the one you start from after saving) but depending on how you enter the room determines a number of minor things like where the camera starts. It’s important to remember this when you edit certain properties of this entrance. Make sure you always access it by double-clicking “Entrance 32”, unless I tell you to click “Starting Location 03”.

Look at the box in the lower-right corner of the Dungeon Window. There are several options:

• “1” – Layer 1 is where we do most of our editing. Doors can go here, too. Treasure chests often are placed here.
• “2” – Layer 2 is required for some editing tricks (like bridges) –I won’t be using it here. Doors can also be placed here.
• “3” – Pretty much only doors can go here. We’ll use this layer to make an entrance leading outside –that door has to go on this layer.
• “Sprite” – Things like enemies, crystal switches, and NPCs go here.
• “Item” – Items that can be placed on the ground or in pots go here.
• “Block” – This setting lets you organize your pushable blocks. We won’t be using this.
• “Torch” – This setting is for torches. We won’t use this, except to delete one torch.

Click on “1”, and then look for the corner jutting out on the left side of the watery place where you land. When you click on it, you will see the following information displayed in the lower-right:

Obj: 10D
X: 2D
Y: 11

Right-click on this object, and choose “Remove”. You’ve successfully deleted the corner.

Right-Click then "Remove" to Delete an Object

Now click on the piece directly south of the one you deleted. Click & drag it left until its X co-ordinate reads “2A”. Now move the mouse over the top edge of the same piece and click+drag to “extend” it until its “Size” reads “03”.

Unfortunately, the piece we moved is now under another piece. So, left-click on it, and then press “Ctrl+V” to bring it to the front. (Ctrl+B brings it to the back).

Right-click anywhere and choose “Insert an Object”. Scroll up to object 108, click it, and choose “Ok”.

Add an Object of Type 108

It will appear roughly where you right-clicked; drag it to 2A,0B. Makes a nice corner, doesn’t it?

Congratulations, we’ve made our first real change to the room! Save.

Move the Wall to the Front with Ctrl+V

We can describe what you just did using a simple table:

Comment	Before	After
wall	x:2D y:11	—
wall	x:2D y:15	x:2A y:0F size:3 front
wall	—	obj:108 x:2A y:0B front

That means “Delete object at location 2D,11”, then “Take object at 2D,15, move it to 2A,0F, resize it to 3 units, and bring it to the front”, then “Add a new object of type 108 at 2A,0B and bring it to the front”. It’s important that you understand this syntax; we’ll be modifying a lot of objects, and it would be horribly cruel to describe each one with a paragraph.

The “Comment” column is an informal addition to help you keep your place.

A note about resizing is in order. Sometimes, when you place a piece down (particularly one that stretches vertically) the piece will stretch to fill up all available vertical space. Please note that you can still resize it, you just have to grab the handle at the bottom and drag it all the way to the top. This is annoying, but not insurmountable.

Step3: Enabling The Bow

Back in Entrance 32, click on the “Sprites” radio button, then click on “Priest/Uncle”. Then, right-click on him, and choose “Remove”. Our Uncle will now no longer be there to give us the sword.

Close Entrance 32 and open Entrance 00. Click once on the chest with the lantern. You’ll see the word “Lamp” under the “Obj” section of the window. Press “+” (or “\”) to scroll through the list of possible treasure chests until you reach “Bow” (do not choose “Bow&Arrow” –it won’t display a message when you take it.) Then change the editing mode to “Item” and click on each heart, then using “N”, “M”, “J”, and “K” to switch the item to “Arrows”. Save.

You Can Change Treasure Chest Contents With + and "

Link now has access to the bow, 15 arrows, and not much else. Play through your game a bit, and you’ll notice that arrows are a bit hard to come by. Also, we’re stuck in the basement, due to that wall we placed in the last section. We’ll change that next.

Step 4: A Complete Roomlet

Modify the following in Entrance 32. Note that the “size” I specify might require you to stretch the component horizontally, vertically, or both.

Comment	Before	After
lights	x:1A y:09	size: 00
lights	x:1A y:15	size: 00
lights	x:2F y:25	x:30 y:09
lights	—	obj:80 x:3B y:11 front
wall	x:2A y:14	x:27 y:14
wall	x:18 y:09	x:25 y:14 size:01
wall	x:2A y:18	x:27 y:18
cement	x:18 y:18	—
cement	x:28 y:18	x:26 y:18 back
wall	x:2A y:1A	x:27 y:1A
wall	x:2D y:17	x:2A y:16 front
wall	x:31 y:17	x:31 y:16 size:00
stairs	—	obj:F9D x:2E y:16 front
wall	—	obj:113 x:33 y:16 back
wall	—	obj:004 x:33 y:17 size:01 back
water	x:30 y:0F	x:2E y:0F
water	—	obj:0C8 x:36 y:0F size:01 back
edge	x:30 y:0F	x:2D y:0F size:06 front
edge	x:30 y:0E	x:2D y:0E size:08 front
edge	—	obj:044 x:2D y:16 size:06 front
edge	x:30 y:17	x:35 y:17 size:00

Here’s what it looks like after you’re finished:

Our Entry Room to the Castle Basement is Complete

You’ve learned a lot about from this, I hope. For example, you’ve learned that tiles can overlay like crazy, and the movement rules are determined in a very sensible, visible fashion. (This becomes less intuitive once you start using two layers). You may have also learned that if you left click on an object, then right-click and choose “Add Object”, the default object will be the type of the first object you clicked on. This makes it really easy to insert a new tile if you can see a similar one on-screen. It’s a real time-saver.

Step 5: Doors and Staircases

We now move on to one of the most difficult parts of dugeon editing: doors and staircases. Actually, that’s a lie: everything’s amazingly difficult. But doors (and staircases) require actual explaining, not just hex mangling, to really understand the concept. And then there’s that mind-numbing number pad “feature”, which I consider to be pure evil.

Backup your ROM and open the working copy in Hyrule Magic. (We can switch to Windows for a while, yes?) Now, click on layer 3 in “Edit”, and then right-click and choose “insert a door”. Scroll up to door 000, click on it, and click “Ok”.

Add a New Door of Type 000

Now, left-click on the door you just placed. This is where Hyrule Magic gets evil. You need to press the arrow keys on the number keypad in order to move the door around. The regular arrow keys won’t work. (On a laptop, you’ll have to use the function key, probably). In my opinion, this is a pure sin of interface design —why would you ever assign different functionality to these two sets of keys. Even Ctrl+Arrow is a better solution.

Press the right arrow until the “Pos” of your new door reads “8”. This may look a bit odd for now, but we’ll fix that later.

Use the Number Pad's Arrow Keys to Move the Door

You can also add this door on layer 1. However, if you do that, the door on the left might disappear; you can manually add it back (it’s door type 001, but we’ll change that later.) A bigger problem is what happens if the entrance door disappears. If that is the case, you have to re-add it —and it can only be added to layer 3. Also, there are two doors which must be stacked to enforce the entrance. Do the following:

• First put down door type 02A and align it over where the exit was.
• Then, put down door type 033 and move it directly over the door you just placed.
• Now save. Your entrance functionality should have been restored. If not, make sure that you placed both doors on layer 3.

Now that you’ve fixed the entrance (or maybe you were lucky and didn’t have to) you should add another door, of type 016, and move it to position 7. Finally, click on the left-most door (the one that was already there) and move it to position 6. Right-click on it, select “choose an object”, and change it to object 000.

I recommend keeping all doors in layer 3, but it’s up to you. Either way, save your game.

Our staircase, by the way, already works. If you look at the very southern part of the room, you can see that we have to climb a staircase before exiting the “Dungeon”. This implies that you enter the dungeon on the “ground level”. In reality, though, staircases usually work so long as you place the start and end bits on valid walkable ground. (In case you haven’t figured it out yet, the “depth” in Zelda3 is just an illusion.) I don’t know enough about Zelda3 to know if you can just keep chaining staircases onto each other, but I’m going to try to stay consistent with the game until I feel adventurous. So, when I put that staircase in our small confined room, I knew that meant I was going to delete the other one.

Do that now: edit plane 1 and right-click on the staircase at 0E,34. Choose “Remove”. Save. We’ll fix the layout later.

Step 6: An Arrow Puzzle

We want to spice up our first Entrance with an arrow-only puzzle. My idea is to have a crystal switch the player sees as he falls in from the garden, and then have him walk around to the far door, then shoot an arrow over the gap at the crystal switch. The middle door is a red herring; there’s no way to get the big key yet. Maybe we can put a treasure chest there to reward the player later when he actually gets the big key.

Perform the following actions:

Comment	Before	After
Comment:	Before:	After:
wall	x:14 y:14	x:22 y:14 front
wall	x:14 y:18	x:22 y:18
wall	x:14 y:1A	x:22 y:1A
lights	x:11 y:1B	—
lights	x:0B y:1B	—
wall	x:15 y:11	x:18 y:18 size:00
wall	x:11 y:17	x:19 y:18
wall	x:0B y:17	x:14 y:18
light	x:1A y:15	x:17 y:1B
wall	x:0F y:17	—
wall	x:11 y:15	x:19 y:0E size:04
wall	x:0B y:0F	x:14 y:0E size:04
wall	x:0B y:0B	x:14 y:0B front
wall	x:0F y:0B	x:18 y:0B size:00 front
wall	x:2F y:0B	x:2D y:0B size:04
wall	—	obj:10A x:19 y:0B front
light	x:1A y:09	x:17 y:09
barrier	—	obj:069 x:13 y:0B size:0C
barrier	—	obj:069 x:1D y:0B size:0C
barrier	—	obj:069 x:29 y:0B size:00
barrier	—	obj:069 x:29 y:11 size:00
edge	—	obj:79 x:17 y:0F size:08 front
edge	—	obj:7A x:19 y:0F size:08 front
edge	—	obj:3F x:17 y:0E size:00 front
edge	—	obj:40 x:17 y:18 size:00 front
water	—	obj:0C8 x:17 y:0E size:02 back
treasure	x:30 y:2C	obj:F99 x:22 y:0C Item: 73:sword 1 front

Switch to “Sprite” mode, right-click and choose “Insert an enemy”. Choose “PegSwitch” and click Ok. Put this one at x:24 y:0C. Repeat, and place the second one at x:20 y:0C. Then, right-click on each of the “Knight” sprites below and choose “Remove”. Now Save.

The Top of Our Dungeon Room is Complete!

We’re beginning to get a feel for this dungeon: the player drops in, sees a switch, learns there’s no way he’s getting the big key, and then decides to hit the switch from the other side. Whew, what a lot of work! However, if you load the ROM in zsnes, you’ll notice a problem immediately:

Our Crystal Switches are All Shadowy

The problem is, all the enemies on one “level” use the same palette. The crystal switch is considered an enemy, so right now it is using a palette with nulled-out colors. Looking at the Perfect Guide, you can see that several palettes use PegSwitch: 15, 27, 26 and 10, to name a few. Open your ROM and change the “EnemyBlk” to “10”. Sometimes you’ll have to pair a palette up with a “blockset”. Eventually, you should be able to create custom groupings of blocks and colors for use in your dungeons.

By the way, change the “Blockset” from “16” to “1”. It’s not necessary, but it makes things look a lot more regal:

Proper Crystal Switches and a New Blockset

Step 7: Finishing Up the Arrow Puzzle

We need to edit the lower half of Entrance 32. We could simply render it as a big square, with some blue crystal blocks impeding our exit. But that’s boring, and will make our audience groan for its lack of creativity. Let’s think: what kind of building did we fall into?

• The original garden entrance was… some weird, useless storage shed.
• Now, we’ve got a locked door and a chest. (There’s a sword inside, but it could be changed to something more interesting… like flippers).
• So we’re in a… control shack for Hyrule River?
• Then the lower room should have lots of flowing water, and be a “public space” (no fancy jumping needed to get around).

Hmm…. maybe we can also be more creative with the crystal blocks. Let’s use them as bridges. Perhaps the big key door is locked because it’s a control room for the “bridges”. We’ll have to think of a better explanation later (water purification? Ph testing?) but at least that’s more interesting than a big rectangle.

Ok, make the necessary changes.

• Switch to level 3, click on the door at position 6 (the exit) and press right (on the numpad) twice. Now, click on the other door at position 6 and press numpad-right twice. We’ve now moved the exit doors.
• Switch to “Item” level, right-click on each “Magic” circle, and choose “Remove”.
• Switch to “torch” level, right-click on our only torch, and choose “Remove”. Switch back to level 1.

Comment	Before	After
Comment:	Before:	After:
pedestal	x:2C y:2A	—
wall	x:35 y:27	x:37 y:27
wall	x:38 y:28	—
wall	x:28 y:24	—
cement	x:3C y:24	—
wall	x:35 y:2B	x:37 y:29
wall	—	obj:106 x:3A y:2C back
wall	—	obj:001 x:3E y:2C
wall	x:35 y:35	—
wall	x:38 y:38	—
wall	x:26 y:38	—
cement	x:22 y:3C	—
wall	x:16 y:30	x:3A y:36
wall	x:1A y:30	x:3E y:36 size:01
wall	x:0D y:27	x:17 y:27 size:00
wall	x:2D y:27	—
wall	x:09 y:27	x:13 y:27
wall	x:13 y:2D	—
wall	x:13 y:31	—
wall	x:13 y:34	—
wall	x:09 y:2B	x:13 y:2B size:03
wall	x:0D y:34	x:15 y:33 size:04
wall	x:17 y:2D	—
wall	x:25 y:2D	—
wall	x:09 y:34	x:13 y:33 front
cement	x:1A y:34	—
wall	—	obj:10A x:18 y:27 front
wall	—	obj:10E x:18 y:2F
wall	—	obj:003 x:1C y:2F
water	—	obj:0C8 x:15 y:2D front
water	—	obj:0C8 x:15 y:31 size:04 front
edge	—	obj:079 x:15 y:2B size:08 front
edge	—	obj:07A x:19 y:2B size:05 front
grate	—	obj:FEF x:1B y:31 front
edge	—	obj:045 x:19 y:31 front
edge	—	obj:040 x:15 y:34 size:04 front
grate	—	obj:FEC x:16 y:28
barrier	x:09 y:38	x:13 y:2A size:06 front
barrier	x:12 y:38	x:13 y:2F size:08 front
barrier	—	obj:022 x:13 y:36 size:08 front
barrier	—	obj:069 x:13 y:27 size:00 front
barrier	—	obj:069 x:1C y:27 size:00 front
barrier	—	obj:069 x:13 y:2F size:04 front
barrier	—	obj:069 x:1E y:2F size:04 front
lights	x:11 y:25	—
lights	x:0B y:25	—
wall	x:16 y:34	—
wall	x:16 y:3A	—
wall	x:22 y:38	—
wall	x:22 y:30	x:20 y:3A
wall	x:22 y:34	x:20 y:3E size:01
wall	x:25 y:31	x:33 y:29 size:02
wall	x:25 y:35	—
wall	x:29 y:35	—
pot	x:33 y:31	—
pot	x:33 y:33	—
wall	—	obj:108 x:33 y:27 front
wall	x:37 y:27	front
water	—	obj:0C8 x:35 y:27 size:01 front
grate	—	obj:FEC x:35 y:27 front
barrier	—	obj:022 x:32 y:2E size:0A front
barrier	—	obj:022 x:32 y:37 size:0A front
barrier	—	obj:069 x:32 y:27 size:04 front

We're Nearly Done... Should We Add a Third Culvert?

Looks incomplete, no? At this point, I was going to add another minor waterway, but looking at the top screen, I noticed that it wouldn’t line up right. Originally, I was just going to say that some sewer line carried away the middle waterway –it looked pretty cool after all. But I think it’s actually better to leave this out.

Naturally, I create all of my tutorial ROMs twice: once to make sure the concept works and once to write out instructions for the blog post. But I want to stress the importance of reflecting on your design decisions: I had the third water-way in place on my first ROM, and decided not to add it on the second one.

Ok, let’s add the final pieces to our room:

Comment	Before	After
Comment:	Before:	After:
bridge	—	obj:0B9 x:13 y:2B size:05 front
bridge	—	obj:0B9 x:13 y:2D size:05 front
water	—	obj:0C9 x:30 y:2F size:0D front
water	—	obj:0C9 x:1F y:2F size:0D front
water	—	obj:0C9 x:23 y:34 size:06 front
wall	—	obj:107 x:2A y:3A back
wall	—	obj:062 x:2A y:3E size:01 back
bridge	—	obj:092 x:1F y:2F size:04 front
bridge	—	obj:092 x:21 y:2F size:04 front
bridge	—	obj:092 x:2D y:2F size:04 front
bridge	—	obj:092 x:2F y:2F size:04 front

And we’re done! (You guys didn’t notice that I forgot that corner, did you?)

Our Completed Room

Our puzzle is complete. Back up your ROM, please, since we’re going to be hacking the header next. When you run your ROM, you can solve the puzzle easily and exit the room. (There is a weird glitch where sometimes the screen won’t scroll properly after walking up the stairs for the first time. I’ve designed this room so that it won’t hinder your progress, but I’m not sure exactly how to fix that bug if it occurs again.) However, something’s amiss if you then re-enter the room. You’re in the wrong starting location!

Entering From the Original Location, Only Now the Door's Gone

Open Hyrule Magic (make sure you’ve backed up your ROM) and look in the “Starting Location” tab. You’ll see “Room”, which we won’t change, and then “X” and “Y” which we will. You can do the math to set these properly (each tile is 16 pixels wide and high) but I prefer to just switch to a room with a similar entrance and copy the values from there. We’ll also need to adjust the x-scroll value to start our camera further over to the right. And, we should fix the CX and CY values as described in the Perfect Guide. Change the following:

X = 376
xScroll = 256
CX = xScroll + 128 = 256+128 = 384
CY = yScroll + 112 = 272 + 112 = 384


Please click on the “More” button in the top part of the Dungeon editor, then click “Ok”. Then Save. Sometimes Hyrule Magic won’t persist room values if you don’t click the “more” button; I have no idea why.

Open your ROM, and re-test… whoah! You’re colliding with obstacles that aren’t there, and you’re jumping into ground that’s really water:

Jumping Into the Wall? Must Have Mis-Aligned the Camera

It’s almost like Hyrule Magic is ignoring the value we entered for xScroll. And… if you open your ROM and double-check, it is ignoring it! This is another frustrating feature of Hyrule Magic; sometimes it doesn’t really save your data. You’ll understand why after we cover the ridiculously complex way that Zelda3 saves the xScroll parameter. For now, it’s sandwich time. I recommend a BLT.

Step 8: Fixing the xScroll Wonkiness

You’ll need a hex editor. We’ll cover Windows first. I recommend using XVI32. Open the program, then open your Zelda3.smc ROM. We are now presented with the first few bytes of the file. They’re all header space, so they’re not too interesting. On the left is the hexadecimal view, and on the right is the ASCII view. Scroll down a bit and you’ll see what I mean:

A Typical Day With Your Hex Editor of Choice

In this example, we are in position 0xEB589, which has the value 4B. In ASCII, that would be the letter “K”, but probably this is actually just used as a number for something, not as a letter. A lot of the data in Zelda3 is compressed, so you won’t be able to find it even if you know what you’re looking for.

At the bottom of the screen, you can see that 0x4B is actually the number 75. Also, you can see that “Overwrite” is on. As you may recall from the early days of Microsoft Word, this means that typing a letter will erase the next letter, so that you never expand the length of the document. Unlike Word, you almost always want this on when hex editing. Otherwise, you’ll offset the entire ROM, and data won’t load where it’s supposed to. Don’t worry, you’ll notice if you do this when you open the ROM; it’s almost guaranteed not to load at all.

From the Perfect Guide, we know that xScroll for the dungeons starts at 0x14F45 and each value is 2 bytes in length. Since we want the data for Entrance 0×32, we do some simple math in Windows Calculator (hex mode) and get:

14F45 + 32*2 = 14FA9

Hitting “Ctrl+G” in XVI32, we click “hexadecimal” under “Go to”, ensure that our “Go mode” is “absolute”, and type in 14FA9. Click “Ok”, and our cursor jumps to address 0x14FA9.

The highlighted box reads 00, and the next box to the right reads 0A. Since Zelda3 stores bytes backwards, we read this as 0x0A00, or 2560 …which is not zero, as we might expect. Enter 2560 in Windows Calculator and click “Bin” to convert this to binary. Now, we have:

101000000000

Note the green bits. Zelda3 uses these for… something, and it prevents Hyrule Magic from saving the xscroll value properly. The red bits are the actual bits used for saving xscroll. So, if we want an x-scroll of 256, then we need:

101100000000

…since 100000000 in binary is 256 in decimal notation. (Generally, you won’t want to mess with the green bits).

If you enter 101100000000 into your calculator and then switch to hexadecimal, you get 0x0B00. Breaking apart and flipping the bytes, we get:

00 0B

So, in XVI 32, click once on the 0A next to the cursor to move it there, type 0B and then Save. Load the ROM in Hyrule Magic and you’ll see that our xScroll has, in fact, been updated. Load it in zsnes and you’ll notice that our scrolling problem is gone.

Hooray! Let's Go for a Swim!

Step 8.2: Fixing the Wonkiness on Linux

I’m still looking for a good hex editor on Ubuntu, so if you have any suggestions, please let me know. For now, you should use tweak, a console editor with a slight Emacs inclination.

sudo apt-get install tweak

Please read the previous section on Windows to understand how we do the math. Basically, we just want to replace the 0A at offset 14FAA with an 0B. To do this:

tweak Zelda3.smc
Then, press Ctrl+X then g
Then type 0x14FAA and hit Enter.
Then type 0B and press Ctrl+X then Ctrl+S
Then press Ctrl+X then Ctrl+C

Okay, we’ve edited our file, saved it, and exited. Re-run our game, and you’ll see that Link is back where he should be when entering the room.

Possible Improvement

Here’s a summary of some things I’d like to modify, but had no time to. You can play around with them, if you’d like:

• Change the opening message from Link’s uncle to make more sense.
• Re-arrange Entrance 00 (Link’s house, pre-Intro) to have a more intelligent layout. The chest should go in front of his bed, for example, or maybe he should have two chests; one with the bow and one with arrows.
• Change the messages from Zelda, and the message when you pick up your bow so that they make more sense.
• Advanced. You actually can’t save; it will drop you back off at the introduction. This is (I think) because the “Castle” save flag is set when your Uncle gives you your sword. Read through the Perfect Guide and MON’s Pinups until you know what has to be changed, and write a small hack which fixes this. (Maybe you can set the “Castle” save flag when Link falls into the hole?)
• If you play through the intro a bit, you’ll notice that arrows are in short supply, and that the ball-and-chain trooper goes down with a mere four arrows. Also, the lamp should be harder to get (maybe in that chest we put between the crystal switches?) and the boomerang should be acquired later. Fiddle around and fix these. Also, consider making more arrow-related puzzles in Hyrule Castle. Just remember that it is a castle, after all, so maintain a believable architecture.
• For the hardcore players, devise a way for them to get into the basement without even the bow. (Make it very tricky to figure out.) This means they have to beat the entire castle by throwing pots. Trust me, hardcore players will love you for this.
• Add a telepathic tile to the water control room which helps to clarify its purpose. (Hint: click on the “more” button on the top of the dungeon window).

Credits

In addition to the Perfect Guide, I relied heavily on MathOnNapkins’s “Zelda 3 ROM Map”, adding 0×200 to the offsets since I’m using a headered ROM. I generally refer to these as MON’s Pinups in this guide.

[LZ-1] Very Simple Introduction to Hyrule Magic

Blog Info: I wrote a short hack for Super Mario World a while ago using Lunar Magic. Like most hacks, it was unfairly difficult, gimmicky, and had rather long levels. That’s when I really appreciated the challenge of game design. Years, later, playing Portal, I was impressed by the impact of small, well-designed levels. And unlike my hack, the small levels had multiple solutions, so that lazy skilled players could innovate with some complex procedure while newbies had fun just figuring out the basic puzzle. And that’s when I turned my attention back to ROM hacking.

Back-story:  ROM hacking really constrains your options. Zelda 3 has a fully-featured map editor, so you’d think that you have the power to re-arrange and re-create rooms like a giant jigsaw puzzle… except that some of the pieces must be in one place, and some of them will automatically re-write themselves with something else. So today, we’ll just take a very shallow dip into the waters of Hyrule Magic, and end with an extra-long discussion section in preparation for next week.

Goal: Fiddle with the opening scene of LttP, changing graphics, text, and object placement.

Now, before we get started, let’s verify that we have the same file.

On Windows:

• Make a file named “Zelda3.hash” in the same directory as your ROM, with the contents:

       1a74468291b02729329dd1357afb45af *Zelda3.smc

• Make sure you switch Zelda3.smc with your ROM’s name. You can download “Checksum” from this link. Install it and double-click on your .hash file. You should get a message that the checksum is valid.

On Linux:

• Make a file named “Zelda3.hash” in the same directory as your ROM, with the contents:

       Jacksum: Meta-Info: version=1.7.0;algorithm=md5;filesep=/;encoding=hex;

       1a74468291b02729329dd1357afb45af Zelda3.smc

• (Make sure to replace Zelda3.smc with your ROM’s name).
• Install the “Jacksum” package:

       sudo apt-get install jacksum

• …and run this simple check from the command line:

•        jacksum -c Zelda3.hash

• You should get an “Ok” message.

Normally I’d just post the valid file, but I’m not allowed to distribute it unless you also own Zelda3. Sorry for the inconvenience. By the way, if your checksum fails, you can still try editing the ROM in Hyrule Magic, with some chance of success. (For example, your ROM may be headerless).

Please note that Hyrule Magic can royally screw up your ROM. Sometimes this is your fault (i.e., loading an expanded ROM), sometimes it is a bug in the editor, and sometimes it’s a feature of the Zelda3 ROM (e.g., “whoops, that tile can only be used for doors”) and you have no idea how to undo such changes. For this reason, it is highly recommended that you keep rolling backups of your project files, even moreso than you would normally. I’m still looking for a VCS that handles binary files well, so for now I’d just recommend daily backups to USB and server-side backups every time you accomplish anything time-consuming.

A Note About Linux

As you might have guessed, we’ll be using Linux for this tutorial. Why, you ask? Well, first of all, because we can, and I want to be fair to all homebrew developers. Secondly, I’ve got a grand goal of making a totally portable, OS-included thumb drive, and that requires me to learn Linux. Don’t worry, you can follow this tutorial on Windows (I’ve tested it on both). Here we go!
Moving Objects, Changing Message Text

Our goal today is a simple, shallow proof-of-concept hack. Copy the Zelda3 ROM to a new directory, then open Hyrule Magic, click “File -> Open”, and browse to the new copy. Once you open the ROM, a whole list of menu items will appear.

Tree-List of Options Available to You in Hyrule Magic

This list is a categorical view of all the data from the Zelda3 ROM that Hyrule Magic understands. Click on “Dungeons” and then double click on “Entrance 00”. Zelda3′s dungeons are set up in a grid-like fashion, and the first one just happens to be your Uncle’s house. You can click and drag various objects –try dragging the table in front of the door:

Dragging the Table in Front of the Door

Hit “Ctrl+S” to save your ROM and “Ctrl+F4” to close the dungeon editor window. Double-click on “Dungeons” to compress the list, then double-click on “Monologue” to edit the game’s text. Double-click on the line which reads:

[Name], I'm going out for a[2]while.  I'll be back by morning.[3]Don't leave the house.

This is the first thing your Uncle says to you after Zelda’s message arrives in your head. The [Name] will be replaced by your character’s name. The [2] and [3] indicate where the second and third lines begin. Change the text to read:

[Name], you sneak out too much.[2]I've blocked the exit.[3]You can't leave now!

Then click the “Set” button. In the same way, change message 32:

[Window 02][Speed 03]Help me...[2]I am in the dungeon of the[3]castle.[Waitkey][Scroll]I know there is a hidden path[Scroll]from outside of  the castle to [Scroll]the garden inside.

…to:

[Window 02][Speed 03][2]Zelda here... [3]didn't you learn how to jump?

Save your ROM (Ctrl+S) and close Hyrule Magic. You can now test the Zelda3 ROM you just hacked. Zsnes works on most Linux boxes:

zsnes /media/Windows/myProject/Zelda3.smc

…or just open Zsnes and browse to the ROM. Go through the opening sequence; boy you’re stuck now!

Link's Uncle Can Walk Over the Table, But Link Cannot

Palette Hacking

Well, that was fun; let’s do one more quick hack and call it a day. Open your ROM in Hyrule Magic, and expand “Palettes”, then “Clothes”, then double-click “Clothes pal 0”. Like most SNES games, Zelda3 stores each pixel it draws by reference to a “palette”. If you were to change, say, the “white” color referenced in the “sword” palette, that would affect every white pixel in Link’s sword. Right now, we’re going to change the color of Link’s clothes. You should see:

A Swath of Colors Used to Paint Link's Character

Count the colors from the left: “1, 2, 3…”. So color “5” is black. You can change a color by single-clicking on it and choosing a new one. Change the following:

       Color 8 from pink to yellow. (Did anyone else notice Link had pink hair?)
       Color 6 from orange to dark blue.
       Color 11 from green to matte blue.
       Color 12 from lime green to matte yellow.
       Color 2 from yellow to very dark red.
       Color 9 from green to a brighter green.
       Color 10 from green to a slightly lighter green than color 9.

This isn’t scientific; just pick colors you like, really. Now, save and load your ROM. Link’s makeover is complete:

Looks a Bit Like a Nightie...

You may have noticed that Link’s face gets all messed up if you pull something. This is not a bug; when we changed orange to dark blue, we affected more than just the trim on Link’s hat. If you were serious about changing Link’s color scheme, you’d probably have to edit the pixels which make up Link’s graphics in addition to the palettes.

Go make yourself a sandwich –or a coffee– we’ll be back after a short break.

Required Reading

You can skip this section if you like; it’s about game design, not hacking. But I think you should read it. Yes, you have to. That’s right.

While you eat your sandwich and drink your coffee, think about your target audience. Realistically, the only people who’d want to play your hack are ones who’ve already beaten Zelda3 multiple times. Probably, you’re one of these people too. So how can you cater to their particular needs?

Well, for one thing, the castle gets pretty boring after you’ve played the later dungeons.

We’re going to use a tried-and-true method of Zelda power-questers today: we’ll deny the players use of the sword for the first bit of the game. The lantern, unfortunately, is a pretty lousy weapon, so let’s give the player some arrows; that should allow for some interesting puzzle construction. But what should we do about Link’s uncle? I see two possible solutions:

The Hacker’s Solution: Scan the ROM and determine the code which gives Link the sword. Switch it so that Link’s uncle gives him the bow. Perhaps alter his sprite to be holding the bow, too.

The Storyteller’s Solution: Just give the bow to Link at his house and delete his uncle in the palace basement; the story can be molded to fit around this.

I’m inclined to go with solution number two —it opens up a lot of creative possibilities. Solution 1 is a bit show-offy, and it doesn’t really fit so well into the story for Link’s uncle to give him anything except a sword. At this point, it’s good to brainstorm and consider a few potential trip-ups, and some tentative solutions (lest we back ourselves into a corner).

Will Link Meet His Uncle Later? Maybe, but we won’t be able to have him in his “wounded” state giving Link anything, since this glitches the game after Zelda is rescued. We have his walkabout graphic; maybe we can replace one of the villagers with him. Or, we can have a mutant in the Dark World say he’s him. Or, we can change the ending credits from “Your Uncle Recovers” to “Your Uncle is Found”.

Why Does Link Have a Bow in His House? Maybe he’s a hunter? Actually, it makes a lot more sense for him to have a bow than a sword. Maybe we can change the general feel of Zelda3 to be more “woodsy” and less “running from the law”.

What About the Game Text? Link’s uncle can say anything; even the same message he says now. Zelda’s second message is easy to replace; something like “Half past the hour; all citizens to bed!”. The first message is tricky. We can have Sora Link monologue (“I’ve been having these… weird thoughts lately.”). Or, we can have him dreaming of his Uncle scolding him for playing around with his bow. Be creative.

Will Link Be Able to Get the Sword Later? This is a very good question, especially if your Uncle can never give you the sword. Can you put it in a chest? At the very least, you can get the Master Sword, so that’ll be our backup.

This was a bit of a digression, but I wanted to impress the importance of thinking creatively and creating a world of your own. The hacker’s solution is elegant, but it actually creates a game that “feels” wrong. The lantern was added to your house to make it feel like you were heading out into the rain, to rescue a princess from some unspecific evil. I’ve always felt this was poorly done in the original game. Grabbing your bow and rushing out into the rain is a lot more romantic. (Stumbling upon your uncle and nabbing his bow is… a non sequitor.)

Even if you were the best ASM hacker in the world, you still shouldn’t desire to change everything. At that point, you might as well just develop an entirely new game. Rather, you want to make a game that plays like Zelda3, but feels totally different. I always take the Quest for Calatia as my inspiration. Here are two innovations this developer makes:

• He uses different trees and houses to imply that Calatia is a colder country.
• He allows you to climb onto house rooftops and hookshot between chimneys.

Those two simple changes really expand the player’s mind as he is exploring. He feels like the fully-equipped link from Hyrule exploring a place which forces him to adapt his strategy.

Another example of great innovation uses ASM:

• Someone modified the game so that you can throw bombs in eight directions, and can throw them either a long distance or a short one.

This mod subtly changes the nature of bombs into something much more utilitarian. I’m sure you could come up with all sorts of puzzles involving bombs and moving platforms, or switches that must be bombed simultaneously.

Credits
My early blog entries on Hyrule Magic owe a lot to Orochimaru’s Perfect Guide to Zelda 3 Editing, an in-depth handbook to using this complex and sometimes non-intuitive editor. Feel free to refer to the Perfect Guide on your own time –it’s got lots of great info.

Post Posting Post: Looking at my dashboard, it seems that a lot of people are finding this post after searching for sprite editing. Like “how do I edit Link’s graphics”? instead of just his palette. Believe me, I understand the frustration of getting no good hits on Google. So, I promise to add some sprite hacking to my next tutorial. I can’t just tell you now, since a lot of the graphics in Zelda are compressed, so it’s more complex than that, and I’m in the middle of another RPG Maker article at the moment.

[RM-2] Speech Balloons for NPCs

Blog Info: I originally started this blog with The Legend of Zelda: A Link to the Past in mind. I got hooked after reading this hacker’s blog. Sometimes teaching is the best way to learn, so I decided to do a series of “mini-tutorials”, each building on the previous one. I later decided to post any tutorial fitting this rubric, since the number of Zelda3 hackers is so small (and since my expertise lies far outside of ROM hacking). I hope to eventually post tutorials on Hyrule Magic, Lunar Magic, and maybe even the Exult engine.

Back-story: Forcing the player to press “Enter” simply to hear what an NPC has to say is rather obnoxious. Wouldn’t it be better if NPCs simply “shouted” their messages out loud? Xenosaga Episode 3 did something similar; how hard can it be in RPG Maker VX?

Goal: Make a light-weight Window that appears over an NPC’s head and shows what he’d normally say using the “Show Text” command.

Let’s start with a basic Window subclass, and an NPC that uses it.

#Show above an NPC
class Talkbox_Window < Window_Base
  #npc is a GameEvent
  def initialize(npc, txt)
    super(0, 0, 100, 50)
    self.z = 90

    #Setup
    self.contents.draw_text(0, 0, 100, 20, txt)

    #Hide it
    self.visible = false
  end

  #We actually need to show the balloon a 
  #  small amount offset above the npc
  def update_pos(npc)
    self.x = npc.screen_x-47
    self.y = npc.screen_y-84
  end
end

For the Event (NPC), have page 1 be autorun. Give it the following commands:

Script:
  me = get_character(0)
  $myText = Talkbox_Window.new(me, "Howdy!")
  $myText.visible = true
  $myText.update_pos(me)
Set Switch 1: ON

Give the Event a second page, and make the condition be “Switch 1 is ON”. Give it no commands.

Start the game and talk to the NPC. The box appears to work, but as you approach the NPC, you’ll notice an obvious flaw: the box scrolls with the map!

Walking to our NPC causes his talkbox to scroll away.

A few interesting points, before we move on.

• $game_interpreter.get_character(ID) will return the event with the specified ID. If ID is -1, it returns the Player. If it is 0, the method returns the event which is currently being executed. Check the documentation for Game_Interpreter; there’s a lot of interesting stuff you can do!
• Recall that by using a dollar sign for $myText, we make it globally accessible.
• If you try to call update_pos(me) before you set $myText.visible to true, the window will show, but (under some conditions) the text will not. It’s a bit of a quirk, but easy to handle if you know about it.

Now, let’s fix up our program. The problem is, we’re using Windows in a way they weren’t designed for. Take a second and think: all other windows you see in the game are positioned statically at some point on the screen (not on the map, like NPCs are). This means we’re going to have to significantly alter the core coding of the game engine itself.

Don’t Run Away! This isn’t nearly as hard as it sounds. Just make sure you’re reading twice as much as you’re writing. For example, start by opening the source for Game_Map and reading the function update. See how it calls update_scroll, update_events, and so on? Presumably, the second of these will update every event on the board. So after that, we can re-assign the x/y co-ordinates of our new text boxes. Add a new function call to update_talkboxes right before the call to @screen.update.

Where to insert a call to our custom update function

Of course, now we have to define this function. Put it somewhere in the Game_Map file:

def update_talkboxes
  if $myText
    me = @events[1]
    $myText.update_pos(me)
  end
end 

This code checks if $myText exists, and then updates its position with that of event 1. You should only have one event on the map, so event 1 is guaranteed to be our NPC. Try it out.

Our textbox now scrolls if we manually update it.

Congratulations, it works! But this was just a proof-of-concept; clearly you can’t make a global variable for each NPC you want to vocalize. Also, you need some way of directly attaching talk boxes to NPCs, so that our update_talkboxes code doesn’t get too bloated.

There are several options. You could just assign NPCs to talk boxes inside some RGSS “init” code, but this is tedious. Wouldn’t it be better if you could use the GUI itself to do it? My solution was to name a switch:

Switch 2 : FLAG! NPC Talkbox

Now, make an NPC with two pages:

Page 1 has as its condition that Switch 2 must be ON. Give it one command, a “Show Text” with some short message.

Page 2 has no conditions and no commands.

Now, find the source for setup(map_id) inside Game_Map. At the end of this function, add the following source:

for event in @events.values
  event.init_npc_talkbox
end

This will call a function (init_npc_talkbox) for each map event when the map first loads. Go into the source for Game_Event, and add the following attribute reader in the “Public Instance Variables” section:

attr_reader :npc_talkbox

…and then add our new method:

def init_npc_talkbox
  page = @event.pages[0]
  if page.condition.switch1_id ==2
    #Steal the message box's text and init it
    if page.list.size>0 and page.list[0].code==101
      txt = page.list[1].parameters[0]
      @npc_talkbox = Talkbox_Window.new(self, txt)
      @npc_talkbox.visible = true
    end
  end
end

This code is very elegant. It simply checks if page 1 of this event requires switch 2 to be on. If so, it takes the first line of text from the first event command (which should be a “Show Text” command) and initializes a talk box with that text. This approach might seem hackish, but it was designed for stability. It is now impossible to accidentally link a talk box to an NPC which doesn’t exist, or vice versa. Moreover, any event that happens to use switch 2 on page 1 (and also has a “Show Text” command as the first one) will be an easy glitch to detect —you’ll see an unexpected talkbox above the NPC.

There’s only one thing left to make this trick complete: let’s change our definition of update_talkboxes.

def update_talkboxes
  for event in @events.values
    if (event.npc_talkbox)
      event.npc_talkbox.update_pos(event)
    end
  end
end

Very concise. Here’s a screenshot of our demo in action.

Automatically scrolling talkboxes look snazzy!

Possible Improvements: This system is quite nascent; there’s a lot you can do to improve or expand it. Be creative! Here are just a few ideas to get you started:

1. For some reason, the boxes “jitter” a bit when you walk up/down. This has to be an “off-by-one” error, since it fixes itself when you stop moving. Try to pinpoint this bug and fix it.
2. Talk boxes currently appear immediately after changing maps, which means they appear before the title screen has fully faded out. Find a solution for this.
3. Enhance talk boxes with Window_Message’s ability to detect symbols and color. (Start looking at convert_special_characters).
4. Enhance talk boxes with the ability to “open” and “close” like a normal window. For example, when you walk up to the person saying “Inn?” and hit Enter, their talk box should close and they should ask you if you want to pay 50G to sleep there.

Post-Posting Info:

A certain fact has come to light since originally posting this tutorial. Please update your projects if you receive this error.

• Your “Talkbox_Window” module needs to be in a location that the interpreter can find. For example, add it below the “Windows” tab, or “Materials”. If you add it on its own to the end of the file, you might get an error when you try to run the program.