Back-story: I was hard at work on my next Hyrule Magic blog entry, which was a lot easier thanks to all your excellent & intelligent comments. (Thanks readers!) But then, I stumbled across this post in byuu’s forum. I’ll explain it as we go along, since the linked post only contains half the story.

Super Mario Odyssey 2 — Now with MSU1 !

Summary: Years ago, BMF54123 suddenly stopped working on Super Mario Odyssey. This made me sad. Then, just last week, I found a post stating that he’d re-started it, and had utilized byuu’s amazing new MSU1 chip to put near-CD-quality audio into the hack.

Wait, that audio’s real!?!?

Technology: Yes, the sound on that video is exactly what you would hear if you plugged the Super Mario Odyssey cart into a SNES and turned it on. There are some caveats; for example, the MSU1 is only emulated by bsnes at the moment. Also, actually making a “Super Mario Odyssey cart” is not possible  –although byuu is making a hardware implementation of the MSU1, so it will be possible in the future. That said, most gamers only play hacks on emulators anyway, so the fact remains: if you play SMW Odyssey on bsnes, expect awesome music!

Other notes: I really like the short “intro” story, showing how Bowser might have ended up with all the crazy time-travel technology that caused the Odyssey in the first place. Also, I like how the game starts with a simple “intro” level (like Yoshi’s island), and I like the “File Select” menu. BMF54123 certainly fine-tuned his ASM skills in the intervening years. (According to the SMW Central forums, he decided to rewrite from scratch because his original code was an undocumented mess.)

What this means: I’d like to think that this hack will usher in a new era of good hacks with rich environments backed up by nice ambient music. However, it’s equally likely to generate hacks of poor quality that are now multi-megabyte patches due to useless MP3 from popular bands.

Why I’m Excited: I haven’t been this excited for a hack in a while –it rivals my Quest for Calatia obsession! In fact, the MSU1 has nothing to do with this  –I’m really just excited that Super Mario Odyssey’s excellent level design and intriguing world design will carry over into a new hack. I’ve always felt that BMF54123 understands what makes a good hack more than most people, and his inclusion of high-quality audio is just icing on the cake.

Your Homework: Watch the video again, and try to pay attention to the level design of the intro level. Even though this is likely just a demo level, it’s very balanced, and far above some hacks’ release-quality levels.

My Homework: Work on a blog post!

Back-story: When RPG Maker VX came out, it radically changed the organization of Ruby code internal to the engine. What it did not change was the fundamentals of the scripting system itself. Thus, any code in RPG Maker XP could feasibly run on VX –assuming that it made no reference to windows, sprites, etc. Unfortunately, porting this code was not always easy. One module that never made the transition was the RPG Maker XP Online System, programmed by username “Blizzard”. This system was a beautifully designed, fully-functional API for RPG Maker network play. Fortunately, Blizzard released the source code for this engine under the Creative Commons (NC-SA) license. This first entry will deal with the simplest –but most important– problem: using sockets. After that, we will slowly build up.

Goal: Create a simple proof-of-concept that TCP sockets can work on RPG Maker VX.

Table of Contents

Step 1: Use Case

Step 2: Borrowing Library Code

Step 3: Initializing & Testing the Connection with NPCs

Step 4: Taking a Step Back

Step 5: A Slightly More Impressive Proof-of-concept

Game Prototype 1: Autosave and an IP NPC

Game Prototype 2: An Online Community

Game Prototype 3 – Chatting with Other Players

Further Work

Step 1: Use Case

Let’s start with a brand-new RPG Maker project. Create a starting map, and put down some “land” tiles. Then, add an NPC with the following commands:

• Message Box: “Opening Socket”
• Script:

s = TCPSocket.open('127.0.0.1', 7689)
s.close()

• Message Box: “Success!”

Now, start your game and talk to your NPC. You should see the first message box, and then the game will crash.

No "sockets" library means no TCPSocket class, which in turn means that our two-line script will crash horribly.

What just happened? Well, first we tried to create a socket. This socket assumed that we have a server running on 127.0.0.1 (which means “this computer”) and that it is bound on port 7689. Since no server has been started, we expect a crash –however, we get the wrong kind of crash. If you look at the message box, you’ll see that RMVX had trouble finding the “TCPSocket” item. In other words, RPG Maker VX does not ship the default Ruby networking libraries! Oh no!

The thing is, RPG Maker XP also didn’t ship with networking support. So how does RMX-OS use it? If you’ve studied programming languages, you know that features such as networking are not considered part of the language itself; they’re usually linked up directly to the operating system after the language has been created. Doing something like that manually is far too difficult for the average programmer. Fortunately, we might be able to “borrow” the code that Ruby itself uses to connect to sockets.

If you want to follow along, you can download Ruby’s source (it must be version 1.8) from some online source –I recommend the Debian repository, with links below. However, the source really isn’t necessary, since I’ll be quoting samples from it when you need them. Moreover, I can’t seem to find the relevant socket code (although I did find useful Mutex code), so you probably don’t need to download the source unless you’re curious. Here are the links:

http://ftp.de.debian.org/debian/pool/main/r/ruby1.8/ruby1.8_1.8.7.249.orig.tar.gz

…or, more generally, from:

http://packages.debian.org/en/squeeze/ruby1.8

After downloading it, unzip it and remember where you saved the source. The only file we’ll need is thread.rb.

Step 2: Borrowing Library Code

One thing you may have noticed if you’re familiar with ruby is that RGSS (RPG Maker’s Ruby scripting system) doesn’t support the “require” primitive. This makes sense –since “require” searches the standard library path,  you might accidentally load Ruby library code which your playtesters likely won’t have. That said, RPG Maker is still perfectly capable of running native code from DLLs. For example, the following works just fine:

Win32API.new(DLL, 'connect', 'ppl', 'l').call(1, 2, 3)

Thus, we might consider chaining together DLL calls to create a kind of “fake” networking library. Eventually, we’ll probably try to migrate to a so-called “native” library, so that we can gain the benefits of low-level primitives. An example is provided in the win32utils:

http://rubyforge.org/projects/win32utils/

If you download and scan, say, the win32-nio library source, you’ll see lots of “require” tags at the top of the main library files. Be aware that you’ll have to essentially replace these with compatible functionality if you want to use the library.

Therefore, we won’t use win32-nio just yet. Instead, we’ll copy the Win32 API that “blizzard” put inside RMX-OS. He claims that it’s from Ruby’s 1.8.1 library source, but for the life of me I can’t find it online. That’s not really a problem, though, as the Ruby License allows verbatim copies of code and distribution in some format.

RMX-OS’s networking code is, by the way, built on top of DLL calls. Here’s the code you need to copy:

Code Segment 1 — Apologies for the ODT format, but PDF removed newlines and tabs.

In the Script Editor (F11), right-click on “Main” and choose “Insert”. Type “Ruby Library Code” in the “Name” box, and then paste the source code from segment 1 into the edit panel. Choose “ok”, then save your game.

If you run your code now, you’ll get some vague, disheartening runtime error. So, before you get depressed, delete your test NPC and add the following code directly into the Main script, below Graphics.freeze:

s = TCPSocket.open('127.0.0.1', 7689)
s.close()

Now, save and run the game. You’ll get the following error on startup:

This error means the library code loaded, then couldn't find the server. We are making progress!

Now that’s the kind of error we’re looking for! Congratulations, the basic structure of the TCP code you added was correct!

Step 3: Initializing & Testing the Connection with NPCs

We need to test an actual TCP connection –in addition, we should probably fix that bug where NPCs can’t seem to trigger our code. These are our next tasks.

If you try to track down the error from the NPC-enabled network code, you’ll reach the following line:

  def self.check
    errno = Winsock.WSAGetLastError
    raise Errno.const_get(Errno.constants.detect { |c| Errno.const_get(c).new.errno== errno })
  end

The error is located inside that particularly nasy “raise” statement. If you don’t understand Ruby’s “block” mechanism, you’ll probably feel like quitting right here. Instead, try this simple abstraction:

  def self.check
    errno = Winsock.WSAGetLastError
    constName = Errno.constants.detect { |c| Errno.const_get(c).new.errno== errno }
    raise Errno.const_get(constName)
  end

Run this code again, and the error’s still on the “raise” line… but now it’s superficially easy to understand. The original programmer was trying to look up Win32 error messages by ID, and he didn’t account for the possibility that a null socket might be returned with NO error code being generated in the process. Well, this is easy enough to fix:

  def self.check
      errno = Winsock.WSAGetLastError
      constName = Errno.constants.detect {|c| Errno.const_get(c).new.errno == errno }
      if constName
        raise Errno.const_get(constName)
      else
        raise "Unknown network error code: #{errno}"
      end
  end

Now, if you try to trigger a TCP connection from the NPC, you’ll see your “Unknown network error code” message.

At this point, I should probably mention that using TCPSocket.open() is not technically correct, since it expects to be passed a block of code to execute upon completion. Although that might be ok for simple socket-related tasks (like requesting the current time or weather conditions from a central server), we’ll need a persistent connection for our game engine. As you might expect, our code will overflow the limits of an NPC’s script command. So, change the NPC’s script code to read:

tcptest()

…and add the following to the “Main” module, directly above the first “begin” statement:

  def tcptest
    #Createa a socket
    s = TCPSocket.new('127.0.0.1', 7689)

    #Send a test message
    s.send("Testing...\n")
    #Receive a result from the server
    msg = ''
    while
      buffer = s.recv(1024)    #Read UP TO 1024 bytes
      buffer.gsub!(0.chr, '') #Remove null bytes
      msg += buffer            #Append received data
      break if buffer.count("\n")>0   #Stop if we've reached the newline
    end
    #Done; close the socket, print our message
    s.close()
    print "Received: #{msg}"
  end

Our code is fairly simple; we use “new” instead of “open”, and then rely on “send” and “recv” to send and receive our data. One problem with “recv” is that it won’t wait until the server has sent the entire message, so we have to chain together all of the bits and pieces of the message as they arrive. On the upside, one of the guarantees of TCP is that we’ll never receive messages out-of-order, so that’s one less potential headache.

Now all we need is the server. I find Ruby a bit distasteful for client/server code, so let’s try writing one in Java. One issue with Java is that compiling and running it can be a pain. Rather than give you a full tutorial on Eclipse or javac, I’m going to ask you to download and install TextPad.

Edit: Some users reported needing to download the JDK, too. Open a command prompt and type javac -version. If you get an error message, you’ll need to download the JDK. If all else fails, see the “source code” section for compiled versions of the server code.

Now, create a new file, and save it as SimpleServer.java. The name must be spelled exactly this way; such a naming convention is a feature of Java. Inside the file, add the following code:

import java.io.*;
import java.net.*;

class SimpleServer {
  public static void main(String argv[]) throws Exception {
    String clientSentence;
    String capitalizedSentence;
    ServerSocket welcomeSocket = new ServerSocket(7689);

    for ( ; ; ) {
      System.out.println("Waiting for connection...");
      Socket connectionSocket = welcomeSocket.accept();
      System.out.println("  >Socket connected");

      BufferedReader inFromClient = new BufferedReader(new InputStreamReader(connectionSocket.getInputStream()));
      DataOutputStream outToClient = new DataOutputStream(connectionSocket.getOutputStream());
      clientSentence = inFromClient.readLine();

      System.out.println("  >Received: " + clientSentence);
      capitalizedSentence = clientSentence.toUpperCase() + '\n';
      outToClient.writeBytes(capitalizedSentence);
      System.out.println("  >Replied: " + capitalizedSentence);
    }
  }
}

Pretty simple? We’ve ignored error checking in this code, so it’ll be fairly easy to crash the server. But for a quick demo, it’ll do. The server reads a sentence from the client, and capitalizes the alphabetic characters. It then returns this uppercase version of the sentence. Since we send “Testing…” to the server, we expect the result to be “TESTING…”.

To compile the server code, choose “Tools->External Tools->Compile Java” (or Ctrl+1).

Compiling Java code in TextPad

If your code contained no errors (and it shouldn’t if you’re copying and pasting) then you’ll see a message in the output window that reads “Tool completed successfully”. Now, choose “Tools->External Tools->Run Java Application”. At this point, a command window will appear bearing the message “Waiting for connection…”

If you see this text, then your server has been successfully bound to port 7689.

This means that our server is ready for the game client to connect to it. Run the game, then talk to the NPC. You’ll see a popup dialog with the response from the server.

Our RPG has successfully communicated with the server, sending it a message and receiving one back in return.

The server window will also have a few more messages:

Our server log proves that the communication was real, and also that Java and ruby can "speak the same language" using sockets.

Note that, since we closed the connection, the server is now waiting for another connection to be opened. We could talk to the NPC again, or simply close and re-open the game, then talk to the NPC again. Eventually, we will require each client to hold on to a connection from the server, but for now this proves the concept: networked RPGs in RPG Maker VX should work just fine. Press Ctrl+C to close the window; you might then have to type “Y” and press Enter to confirm that you want the server to halt.

Step 4: Taking a Step Back

At this point, it’s worth pausing for a second to consider what we’ve actually accomplished. There’s a lot of “glue code” (like reading lines of text from ruby) that hides what’s actually going on:

• We’ve created a server that can “receive” connections on port 7689, and a client that can request a “connection” on this port.
• Once connected, a client and a server can send and receive a sequence of characters.

That’s all there is to it. Adding multiple clients, some kind of synchronization, and partitioning our data into “messages” or “game events” are just additional details on top of what the tcp client actually is. For example, instead of sending strings of messages, we could send pairs of NPC x/y co-ordinates. In that way, we could treat “heroes” in separate RPGs as “npcs” in your RPG world. We could send “tiles”, and allow users to build their own houses (as maps) which they can allow their friends to visit. We can send custom avatar pictures for each player, or even co-ordinate some kind of battle system. All of these things can be accomplished simply by sending letters and numbers over a TCP connection.

Step 5: A Slightly More Impressive Proof-of-concept

Our “borrowed” Ruby source code is about 500 lines in total. This is only about 8% of the total RMX-OS code, which includes things like Scenes for connecting to the network and a password system to help protect users’ passwords and privacy. My point is, making anything interesting is going to be a huge undertaking. On the other hand, a simple message box usually isn’t enough to prove to your boss that you’ve accomplished a huge feat like networking a game engine that doesn’t normally expect to be multiplayer. What, then, might a nice, simple prototype look like?

• We don’t want to play around with Scene overloading. So let’s make an NPC that basically allows us to connect to a network, and manage our network configurations.
• When the game begins, the player can choose one of four starting heroes. This information is transmitted over the network.
• At this point, we should create NPCs for each of the other users online. These will just wander randomly for now; they won’t reflect the actual position of the other users.
• The hero can talk to these NPCs to receive “messages” that they have sent (like a chat). Only the most recent message is shown.
• When an NPC gets a new message to display, we can show a “balloon” animation above him –an exclamation mark or something similar.
• To send a message, the hero can talk to another NPC (like the connection NPC). We might use the hero name dialog, or we might capture key presses. In the worst case, we’ll just let the user select some pre-defined messages from a list.

There’s a lot of bullets (e.g., a lot of things to show off) but individually, each is not terribly difficult to accomplish. Let’s get started!

VERY IMPORTANT NOTE: Whenever you change some of your game’s code (scripts), you should re-test using a new game. If not, you risk loading old code from an old save file (it happened once to me, while testing this blog entry) and it is frustrating to track down an error like this. In my case, the initialize method for my Game_Map class was not re-called upon loading an existing save file, and I couldn’t figure out why some class variables didn’t exist anymore.

Game Prototype 1: Autosave and an IP NPC

Our first prototype will have no networking capability at all. We’ll need two maps; one for the player to select an Avatar upon starting a “New Game”, and the other for normal play. Create these two maps and call them “Player House” and “Character Select”. Make them both small (20 x 15 or so) and go to the Character Select map. Give it a fancy layout and set the hero’s starting position to be somewhere on the map.

Our spartan starting map.

Now, add a few NPCs (five would be good), each with the graphics of the first five heroes in the party. If you don’t like the default hero pictures, you should go into the database editor (F9) and change them. Just make sure they match the pictures of the NPCs you just added. While you’re in the database editor, you might want to change the Initial Party (in the System tab) to include only hero 6, and change hero 6‘s walkabout to be something more mysterious. Name him SELECT to avoid confusion.

Five possible Avatars on our starting map.

If that was all too much for you, you might consider reading through a few RPG Maker tutorials –this blog post would likely double in size if I tailored it to complete beginners. (If someone’s willing to do a voiceover for me, I’ll put up a video walkthrough on Youtube.)

Open up the left-most NPC in the Event Editor (double-click) and add the following on page 1:

Talking to an NPC activates this event, which asks the user if she wants this avatar.

To summarize that code briefly:

• Show a text box asking if the player wants this hero.
• Have a “choice” command (Yes/No). We only care if the user picks “Yes”
• If so, set variable 1 (which we call Avatar ID) to be 1.
• We then add the first hero (“Ralph”) to the party.
• We remove the SELECT hero.
• We transport to a random point on the Player House map.
• We then “auto-save” using the following script:

Scene_File.auto_save($game_variables[2])

Now, before we talk about auto-saving, copy and paste the event code from NPC 1 to each of the other 4 NPCs. When you paste the events, make sure to change the text in red to reflect avatar 2, 3, etc. Each player should introduce him or herself, ask the player to choose him/her, and then save his/her details into variable 1 and create a party with only him/her in it. Isn’t gender neutral language fun?

Let’s talk about the auto-save. You can probably guess that “auto-save” is (or will be) a static method in the Scene_File class, and that it will take a save slot ID (which we will store in variable 2. Open the script editor (F11) and browse to Scene_File. Add the following directly underneath Scene File < Scene Base:

  #Save to a slot directly.
  #Indexed from 1, not 0.
  def self.auto_save(file_index)
    fileName = "Save#{file_index}.rvdata"
    file = File.open(fileName, "wb")
    Scene_File.write_save_data(file)
    file.close
  end

We copied most of this code from the do_save method, making sure to construct the right file name. By naming our code as self.auto_save, we make the method static –the user code doesn’t need a (new) object to use the function. Unfortunately, this requires that  write_save_data also be static. Find the following line in Scene_File:

 def write_save_data(file)

…and change it to

 def self.write_save_data(file)

However, this will break normal saving, which we probably don’t want. Find the following function in Scene_File:

 def do_save
   file = File.open(@savefile_windows[@index].filename, "wb")
   write_save_data(file)
   file.close
   return_scene
 end

…and change it to:

 def do_save
   file = File.open(@savefile_windows[@index].filename, "wb")
   Scene_File.write_save_data(file)
   file.close
   return_scene
 end

This way, do_save will still work properly (by calling the now-static function). There are other ways to accomplish this, but this is the easiest to describe.

One tiny problem remains: how do we save the preferred autosave slot into variable 2? I see three possibilities:

• Just pick a slot (e.g., “the first save file”) and use that. This is too hot –it forces an arbitrary choice on the player.
• Give the player a choice box with options “1,2,3,4”. This is too cold –it’s a bit impersonal for players used to the nice save dialog of RPG Maker.
• Force the player to save the game once, and use that value. This is just right –it’s natural to the player, and shouldn’t be too hard to figure out from our end.

Needless to say, we’re doing option 3. Create a new event with no graphic in the top-left corner of the map and give it two pages. The second page should have the “Switch” box checked under “Conditions”, and have “Switch 1” selected. (We call this switch Chose Save Slot.) The first page should have its trigger set to “Autorun”, and have the following event code:

The code for our auto-save enforcing NPC.

In other words, we first tell the user to select an auto-save slot. Then, we show them the save menu. Now we need to check if the user canceled the menu. We do this by setting variable 3 (called TMP Save Count) to be equal to “Save Count” (listed under “Other” in the variables dialog). We then do a simple conditional branch: if this variable is greater than zero, we continue. The first thing we now do is execute a nifty script:

  #Set var[2] to the save slot
  slot = $game_temp.last_file_index + 1
  $game_variables[2] = slot

The variable last_file_index is already part of the game engine. It stores the id of the last slot saved to, starting from zero. (Since save slots are actually numbered from 1, we increment this value.) The default value for last_file_index is zero, by the way, so we can’t just check this to determine if the user canceled or saved.

Moving on, we then set switch 1 to ON. Next, we set the names of heroes 7 and 8 to 127.0.0.1:7689. This is not very important right now; just add the lines and we’ll get back to them later. Finally, we perform our “classic auto-save”

  #Save the game to slow (ID-1)
  Scene_File.auto_save($game_variables[2])

You should memorize this code fragment; we’ll be using it a lot.

At this point, you can run the game, and you will be prompted to save. Following that, you can choose a hero and then walk around the Player House map. Not too bad for a quick-and-dirty auto-save, eh? In fact, if you play around with quitting and reloading games, you’ll find that our code is actually pretty robust; cancel or close the program whenever you want and everything should still work fine when you reload it.

Our auto-save dialog performs impressively.

Go to the Player House map and create a nice, homely setting. Add some ambient NPCs if you want; just make sure there’s a scholarly-looking NPC somewhere near the middle of the map. We’ll use him for connecting to the server. Here’s just one possible setup:

Only the primary NPC matters, but feel free to make this map as picturesque as you like.

This NPC will have a whole lot of functionality wrapped up inside his persona. To make the code more manageable, we’ll put different functions into different common events. The core code for the map NPC looks like this:

The main code of our "Connect" NPC.

To summarize:

• This NPC can either introduce himself or call the “Connect to a server” Common Event, which has the ID of 1. The rest is just fluff.

Open the database editor (F9) and browse to the Actors tab. Name actor 7 SAVED_SERVER_1 and actor 8 SAVED_SERVER_2. Then browse to the Common Events tab. Name common event 1 “Connect to a server” menu. Name common event 2 Prompt for server address. Name common event 3 Prompt to Save Server. Now, for common event 1, add the following code:

CE 1 handles connecting to a server. It is currently incomplete.

This event is quite simple:

• First, it asks the user to choose a server. The first two options are the names of the SERVER characters that we just created. At present, nothing happens when these are selected. The third option allows us to add a new server. It calls the following two common events in sucession.

The second common event is also quite simple, if a bit repetitive:

CE 2 allows us to construct new IP addresses. It's just a little cumbersome.

It performs the following routine five times:

• Tell the user the name of hero SELECTION, then ask for a number. Store this number in variable 4 (TMP server part)
• Append this number to the name of the SELECTION hero using the following script:

    #Set actor's name
    actor = $game_actors[6]
    actor.name = "#{$game_variables[4]}."

  • …note, of course, that the “.” changes to a “:” for entry 4, and “” (nothing) for entry 5.

The third common event simply checks if this is what the user wants:

CE 3 just confirms the user's entry, saves it, and resets the SELECTION npc's name.

It saves, e.g., actor 7 by using the following script:

 #Finalize actor's name
 $game_actors[7].name = $game_actors[6].name

Of course, we reset the SELECT hero’s name to avoid confusion later. We then wrap it all up with an auto-save. And that’s it! We now have a simple way to enter server addresses without requiring any fancy interface or custom save files. Here’s a screenshot of our system in action:

The orange text makes it look at least ten times more professional.

Once the selection has been saved, it shows up on our options list:

The beauty of using an Actor's name to save the IP address is that you can use it in a Message Box without any fancy tricks.

It’s nearly time to move on to our second prototype (which will introduce network elements). Before we get too far ahead, I’d like to highlight some of the merits of our approach:

• We relied for the most part on local scripts, requiring only a single change to the game’s source code. In our defense, the function write_save_data should have been static anyway, and could have been fudged if we were so inclined to create a dummy object.
• By using several Common Events, we avoided the RPG Maker-exclusive temptation to create one messy NPC with all our code. In fact, nearly all events had single-screen pages of Event Commands.
• We used “special heroes” to store our IP addresses. This allowed us to stick with event commands like “Show Choice” instead of requiring custom Scripts for everything.
• By and large, we used mostly RPG Maker constructs, relying on scripts only for tiny globs of “glue code” where the interpreter fell short. Most intermediate RMVX users could probably figure out what our code does, and how to improve it.

Enough resting on our laurels; it’s time to notch up the interest level of our audience.

Game Prototype 2: An Online Community

We strive for incremental improvements, so let’s try adding only the most basic of network functionality:

• When a user Connects, we’ll add an (arbitrary) NPC to the player’s home map.
• When a user Disconnects or closes the game, we’ll remove this NPC.

This will require a lot more foresight than you might expect. First, we need to decide how the client should handle messages. After all, we don’t want our game engine to hang just because the network goes down. Following that, we’ll need a simple message format that is simple to expand with new data types. Finally, we’ll need a messaging protocol: how often should we send updates to users? For now, we’ll just assume “always send every update”, since we’re not tracking anything expensive like a user’s X/Y co-ordinates. For bigger games, the messaging protocol is the most important part of the network design. If your game’s lagging, try reading a few papers on interest management (PDF link) in online games.

First, let’s talk about threading. Like most modern high-level languages, Ruby runs inside a virtual machine, which itself is a single process in the Windows operating system. All ruby-related code in your game must therefore run in sequence. This includes all parallel or autostart events, common events, NPC interactions, and scripts. Unfortunately, this makes it very difficult to add networking. Let’s say that you check for messages from the server before you service all parallel process events. If your game has a lot of these events running at once, the network buffer might overflow, and you’ll lose messages. (Very Bad.) What we really want is a way of telling Ruby to check for network messages “every so often”. That’s exactly what threads were designed to do.

Consider the following Ruby code:

  #Code our thread will execute
  def network_thread
    while true
      read_server_messages()
      send_waiting_messages_to_server()
      sleep(1) #Avoid spinning too fast
    end
  end

  #Actually make a thread
  t1 = Thread.new{network_thread()}

Conceptually, the line with Thread.new tells Ruby’s virtual machine to make the network_thread() function run once inside a new thread. This thread will alternate with the main (RPG Maker) thread in the virtual machine. According to Ruby’s time-slicing policy, each thread will run for 10ms before the other thread gets a turn to run. Inside network_thread, we call a function which reads all messages that the server has sent to us. Note that we do not process these messages yet; we’ll do that in the main game thread. Instead, we simply get them out of the network buffer and into the game engine. Following that, we send any waiting messages to the server. Again, these messages are generated by the game engine; we don’t really care how they’re made. Then, we perform a call to “sleep”, which forces the current thread to suspend. That way, if we only used 2ms of our time slice, the main game engine doesn’t have to wait 8ms while we do nothing useful.

This is only one way of doing things. We might also have put read_server_messages and send_waiting_messages_to_server into their own separate threads.

Now, let’s talk synchronization. Where does read_server_messages save the messages it reads? Presumably, we’ll have an array to store them:

 read_messages = []

The network thread can then put messages into the back of this array, and the game engine can read messages from the front. The problem is, what happens if both try to act on the array at the same time? The answer: something Very Bad. To avoid this, we should synchronize access to this array. For example, the game engine might contain code like the following:

  GameMessage nextMessage = nil
  @lock.synchronize { 
    nextMessage = read_messages.shift unless read_messages.empty?
  }
  if nextMessage
    #....process nextMessage
  end

The @lock variable might be created with a call to:

  @lock = Mutex.new

It ensures that all the code inside the locked block executes before any other locked code. To avoid unnecessary slowdown, it is a good idea to keep the code inside a lock block as simple as possible –note that we store the next message in nextMessage, rather than processing it inside the locked section.

There’s just one problem: Mutex.new requires the ‘thread’ library. And I mean requires; that is, we don’t have access to it in RPG Maker code. (Thankfully, we do have access to thread). We can’t use the equivalent “monitor” class; it’s also required. Fortunately, there’s an easier solution. Remember the ruby 1.8 source code we downloaded earlier? Go to that directory, and then browse to the lib directory. You’ll see the thread.rb file which contains mutexes. Copy and paste the source code into your Ruby Library Code module (F11) at the very top. Segment 2 contains this code, minus most comments:

Code Segment 2 – Ruby code.

And that’s all we need to do to enable mutex creation:

The Mutex object was created properly. (Sample source code not included.)

We nearly hit a dead-end, but synchronization works. Bullet dodged; that could have killed the entire project. (Although not entirely: RMX-OS doesn’t use threads and achieves reasonable performance.)

All drama aside, you should now be able to envision how we can combine synchronization with threading to read messages from the server. But what is a game message? We want something easy to extend, and something that (preferably) doesn’t rely on newlines to specify message boundaries. Typically, we would use an array of bytes. However, the Ruby 1.8 library code we borrowed doesn’t support this, forcing us to read and write strings. There are some very good reasons why we don’t want to force byte arrays into strings:

• For some reason, I can’t get the Socket.read command to work. This would have allowed us to read X bytes from the socket, instead of recv‘s “up to X” bytes. (I had actually written half this guide based on read until I realized it didn’t work. Silly me.)
• If we use recv to read bytes, the default for “un-read” bytes is 0. But what if we actually wanted the number 0? We can get around this (add 1 to each byte sent, or initialize our buffer array with -1 instead of 0) but these “solutions” feel more like hacks –and although I derive great enjoyment from puzzling out another programmer’s creative solutions, I dislike writing obfuscated code myself.

So, we’ll make this simple: a message is just a string, of the format:

• TYPE:xxx:yyy:zzz:\n

In other words, a message is composed of a series of strings, separated by colons, with a newline designating the end of the string. The type of the message determines how many remaining segments are expected, and what they mean. Here are our message types, and their corresponding data.

The first type is CONNECT_REQUEST, sent when a client wants to log in to the server.

The second type is CONNECT_RESPONSE, sent back from the server whenever a client tries to log in.

The third type is USER_CONNECT, sent whenever another user has connected to the server to all other players currently online. A series of these messages is also sent from the server to a client that has just logged on, lest he think he logged in to an empty world.

The fourth message type is USER_DISCONNECT, sent when a user has disconnected.

The server manages all user connections, so things like timeouts, etc. are not the client’s responsibility. In some cases, the server may shut down; if so, it might send USER_DISCONNECT messages to all clients. Thus, if a client receives a USER_DISCONNECT message with its own user ID, it can assume that the server will send no further messages. Moreover, if the client sends a USER_DISCONNECT message to the server, it is probably trying to request a disconnection.

Let’s actually get some of this code working. First, we’ll need a Game_Shared class and its respective $game_shared singleton for containing all our shared objects (which require mutually exclusive access). Add the Game_Shared module below Game_Interpreter:

#==============================================================================
# ** Game_Shared
#------------------------------------------------------------------------------
#  Contains all objects that may be shared among threads, and contains mutexes
#  for controllling these accesses.
#==============================================================================

class Game_Shared
  
  #--------------------------------------------------------------------------
  # * Object Initialization
  #--------------------------------------------------------------------------
  def initialize
    @read_messages = []     # Just read from server
    @read_lock = Mutex.new  # Lock for read_messages
    @messages_to_send = []  # To send to server
    @write_lock = Mutex.new # Lock for messages_to_send
  end
  
  #--------------------------------------------------------------------------
  # * Public Accessor Functions --From Server
  #--------------------------------------------------------------------------
  def getNextServerMessage
    ret = nil
    @read_lock.synchronize { 
      ret = @read_messages.shift unless @read_messages.empty?
    }
    ret
  end

  def queueServerMessage(msg)
    @read_lock.synchronize { 
      @read_messages.push(msg)
    }
  end

  #--------------------------------------------------------------------------
  # * Public Accessor Functions --From Client
  #--------------------------------------------------------------------------
  def queueClientMessage(msg)
    @write_lock.synchronize { 
      @messages_to_send.push(msg)
    }
  end

  def getNextClientMessage
    ret = nil
    @write_lock.synchronize { 
      ret = @messages_to_send.shift unless @messages_to_send.empty?
    }
    ret
  end

end

To create the singleton, go into Scene_Title and find the function create_game_objects. Add the following text, right at the end of the function:

  $game_shared        = Game_Shared.new

Now, we will have access to our singleton object for most of the game. The shared object singleton allows us to read and write messages to and from the server, using arrays to queue the messages properly. It handles all locking internally, so we don’t need to worry about Mutexes outside of our Game_Shared class.

We’ll also need a Game_NetMessage class; add it directly below Game_Shared.

#==============================================================================
# ** Game_NetMessage
#------------------------------------------------------------------------------
#  A light-weight class used to store messages sent to/from the server.
#  Contains helper methods to construct most message types.
#==============================================================================
class Game_NetMessage
  
  #Message types
  def self.CONNECT_REQUEST  
    "CREQ" 
  end
  def self.CONNECT_RESPONSE 
    "CRES" 
  end
  def self.USER_CONNECT     
    "UCON" 
  end
  def self.USER_DISCONNECT  
    "UDIS" 
  end
   
  #Helper methods
  def self.Make_ConnectRequest(avatarID)
    "#{Game_NetMessage.CONNECT_REQUEST}:#{avatarID}\n"
  end
  def self.Make_ConnectResponse(playerNewID)
    "#{Game_NetMessage.CONNECT_RESPONSE}:#{playerNewID}\n"
  end
  def self.Make_UserConnect(playerID, avatarID)
    "#{Game_NetMessage.USER_CONNECT}:#{playerNID}:#{avatarID}\n"
  end
  def self.Make_UserDisconnect(playerID)
    "#{Game_NetMessage.USER_DISCONNECT}:#{playerID}\n"
  end  
 end

This class functions entirely through static method calls; you’d never need to create a Game_NetMessage object. Instead, you’d use it like this:

  msgToSend = Game_NetMessage.Make_ConnectRequest(3)
  $game_shared.queueClientMessage(msgToSend)

…to make a connection request message and queue it for sending to the server.

All we need now is a $game_network class which performs the following tasks:

• Create a network thread when the user connects to the server. Destroy this thread (safely) when the user disconnects, or when the server kicks him off.
• Maintain some statistics relating to the network: Is it connected? What’s the latency?
• Make sure that the network thread calls $game_shared.queueServerMessage() when it receives a message from the server. Make sure it sends data to the server whenever $game_shared.getNextClientMessage() returns something non-nil.
• Catch most bad behavior (like trying to connect to a server multiple times, etc.)

Given this brief description, we might create Game_Network (placed directly under Game_NetMessage) as follows:

#==============================================================================
# ** Game_Network
#------------------------------------------------------------------------------
#  This singleton class controls network access
#==============================================================================
class Game_Network
  
  #--------------------------------------------------------------------------
  # * Object Initialization
  #--------------------------------------------------------------------------
  def initialize
    @s = nil                # Our socket
    @loc = ""               # Help with debugging a bit...
    @network_thread = nil   # Thread is non-nil when we have a server connection
    @kill_thread = false    # Set to "true" to gracefully terminate this thread
    @server_message_buffer = [] # Buffer of bytes received from the server
    @my_id = -1 #Managed here, not in the game manager
    @connect_mutex = Mutex.new
    @discon_mutex = Mutex.new
    
    #Always start OFF
    $game_switches[2] = false
  end
  
  #--------------------------------------------------------------------------
  # * Main Functionality
  #--------------------------------------------------------------------------
  def handle_messages
    #Write while we have messages to send
    @loc = "handle_messages():send"
    while true #Can always send
      #Anything to send?
      @loc = "handle_messages():check to send"
      clientMsg = $game_shared.getNextClientMessage
      break unless clientMsg
      
      #Send it
      @loc = "handle_messages():send messages"
      @s.send(clientMsg)
    end
    
    #Read while we have data available
    @loc = "handle_messages():receive"
    partial_message = nil
    while @s and @s.ready? #Can only recv if the socket is ready
      #Read as many remaining bytes as possible
      #NOTE: Using -1 means that there will always be a '' entry if 
      #      our string originally ended in \n
      readStr = @s.recv(0x1024)
      readStr.gsub(0.chr, '')
      
      #TEMP
      #print "Read: #{readStr}" unless readStr.empty?
      
      
      #Continue?
      next if readStr.empty?
      lines = readStr.split("\n", -1)
      
      #Any partial message to complete?
      if partial_message
        lines[0] = partial_message + lines[0]
        partial_message = nil
      end
      
      #Is the final message partial? If so, defer processing till later
      partial_message = lines.pop 
      partial_message = nil if partial_message.empty?
      
      #Ok, we've got a series of messages. Queue them for the client
      lines.each{|message| 
        @loc = "handle_messages():process single message"
        
        #Split this one too; remove newline
        segments = message.rstrip.split(/:/)
      
        #TEMP: DEBUG
        #print "Message: #{message}"
        
        #We don't always post it
        if segments[0]==Game_NetMessage.CONNECT_RESPONSE
          id = segments[1].to_i
          @my_id = id if id > 0
        elsif segments[0]==Game_NetMessage.USER_DISCONNECT and segments[1].to_i==@my_id
          destroy_thread()
        else
          @loc = "handle_messages():try to queue"
          $game_shared.queueServerMessage(message)
        end
      }
    end
  end
    
  
  #--------------------------------------------------------------------------
  # * Thread access
  #--------------------------------------------------------------------------
  def connect(serveraddr)
    @connect_mutex.synchronize { #Keep re-entrant calls legal.
      #Error?
      raise "Network error: already connected" if @network_thread
    
      #Parse out the port from the server address
      serverparts = serveraddr.split(':')
      if serverparts.length != 2
        raise "Network error: cannot connect to \"#{serveraddr}\", bad address"
        return
      end
      serverparts[1] = serverparts[1].to_i
      
      #Create a network thread to connect and parse messages
      # Note that threads must handle their own exceptions
      @kill_thread = false
      @network_thread = Thread.new {
        begin
          #Connect
          @loc = "connect():init connect"
          @s = TCPSocket.new(serverparts[0], serverparts[1])
          $game_switches[2] = true
          
          #Send first message: whoami?
          @loc = "connect():send first message"
          avatarID = $game_variables[1]
          msg = Game_NetMessage.Make_ConnectRequest(avatarID)
          $game_shared.queueClientMessage(msg)
          
          #Process
          while true
            #Done?
            @loc = "connect():check if done"
            @discon_mutex.synchronize {
              break if @kill_thread
            }
            
            #Process any messages
            @loc = "connect():call handle_messages()"
            handle_messages()
            @loc = "connect():returned from handle_messages()"
            sleep(1)
          end
        rescue StandardError => err
          #Print an error, destroy this thread
          print "Exception in network thread(#{@loc}): #{err}\n"
          @network_thread = nil
        end
      }
    }
  end
  
  def ask_to_disconnect
    if @my_id != -1
      #Queue a closing message
      msg = Game_NetMessage.Make_UserDisconnect(@my_id)
      $game_shared.queueClientMessage(msg)
    else
      #Just kill the thread
      destroy_thread()
    end
  end
  
  def destroy_thread
    #Signal the thread to stop
    @discon_mutex.synchronize {
      @kill_thread = true
    }
    if @network_thread and @network_thread != Thread.current
      @network_thread.join
    end
    @s.close() if @s
    
    #Rest to nil
    $game_switches[2] = false
    @network_thread = nil
    @s = nil
    @my_id = -1
  end
  
end

Ok, that was far longer than I’d hoped, but that’s probably because my understanding of Ruby is pretty lousy. Before we analyze it, add the following to the end of Scene_Title‘s create_game_objects function:

  $game_network       = Game_Network.new

This creates our singleton object; now, let’s talk about the main code. First, some disclaimers: even though I’ve added a few mutexes, this code definitely isn’t thread-safe. If you call connect() and destroy_thread() multiple times and quickly, the code will likely destabilize. However, it probably won’t mess up under normal usage conditions. There’s only three  main differences between this code and the snippet we used in Step 3 to test our network connection:

• We borrowed the message loop from RMX-OS’s listen method, although we changed a few things to suit our purposes.
• Although the game engine handles most messages, our Game_Network class automates connection/disconnection requests and the storage of our own ID. We only have to call the connect and ask_to_disconnect methods (and read all other relevant messages in the game thread, of course) for communication to work.
• We use Switch 2, which you should now name Network ON, to store the status of the network. This makes it easy for NPCs to react to “being online” without requiring them to use any scripting.

The client-server communication is only halfway complete; we also have to create client messages to send and respond to server messages. This will occur inside the game engine. Briefly, the following needs to happen:

• Call connect and ask_to_disconnect when appropriate.
• Handle USER_CONNECT and USER_DISCONNECT messages (and discard or err on unknown messages) at some central point in the client code.

That’s it. Our client sends no outgoing messages that aren’t handled in Game_Network (for now). The question remains: where should we check messages? It should be somewhere central, and somewhere accessed often (but not too often). I recommend Game_Map´s update method. Add the following line to Game_Map´s update method, after update_scroll and before update_events:

  update_network

Now, somewhere in that module, add the code for update_network:

  #--------------------------------------------------------------------------
  # * Update Network: Read messages  (write as-needed elsewhere)
  #--------------------------------------------------------------------------
  def update_network
    #"Process no more than X messages"
    nextMsg = nil
    for i in (1..10)
      #Read the next message
      nextMsg = $game_shared.getNextServerMessage
      break unless nextMsg
      
      #Split
      segments = nextMsg.rstrip.split(/:/)
      
      #Process
      if segments[0]==Game_NetMessage.CONNECT_REQUEST
        next
      elsif segments[0]==Game_NetMessage.CONNECT_RESPONSE
          #Always refresh
          $game_map.need_refresh = true
      elsif segments[0]==Game_NetMessage.USER_CONNECT
        #Add an NPC to the map with the right picture and behavior.
        #In range (3,9) , (11,14)
        avtID = segments[1].to_i - 1
        plrID = segments[2].to_i
        if avtID < @max_network_npcs
          #Get the event, place it
          #print "Moving NPC #{@net_npc_start + avtID}"
          newNPC = @events[@net_npc_start + avtID]
          moveNPCRandom(newNPC, 3, 9, 8, 5)
          
          #Set its variable flag (for picture)
          $game_variables[@net_npc_picvar_start + avtID] = plrID
          
          #Always refresh
          $game_map.need_refresh = true
        end
      elsif segments[0]==Game_NetMessage.USER_DISCONNECT
        #Remove the NPC
        avtID = segments[1].to_i - 1
        @events[@net_npc_start + avtID].moveto(1, 1)
        
        #Remove its picture
        $game_variables[@net_npc_picvar_start + avtID] = 0
        
        #Always refresh
        $game_map.need_refresh = true
      end  
    end
  end

This code is not complicated, but let’s review it anyway.

• First, we read through all available messages. Since a very busy server will probably never stop spawning messages, we limit our processing to 10 messages per game “tick” (which should equate to 600 messages processed per second). We might have to change this later.
• Next, we react to each message:
  • When a user connects, we take a pre-defined npc and move it to a random location between (3,9) and (11, 14). We will create placeholder events later. We then set a pre-defined variable to be equal to the avatar’s ID. Presumably, our pre-defined event will have a page for each avatarID with its respective walkabout.
  • When a user disconnects, we teleport that NPC to the top-left corner of the map, and set its avatar ID variable to 0, thus making it invisible.
• A note about $game_map.need_refresh; this function basically forces several computationally-intensive checks of the map to re-occur on the next map update. In particular, we use it to make sure that each event is updated to the proper page when our NPC-Avatar ID variable is set.

This code requires the the function moveNPCRandom; for now, just add this prototype somewhere in the Game_Map class:

  #For now, just non-random is ok
  def moveNPCRandom(npc, xMin, yMin, width, height)
    for y in (yMin..yMin+height)
      for x in (xMin..xMin+width) 
        if events_xy(x, y).empty?
          npc.moveto(x, y)
          return
        end
      end
    end

    #If all else fails, just place it in the middle (there will be a conflict)
    npc.moveto(xMin+width/2, yMin+height/2)
  end

Re-using existing NPCs is not the best way of doing things; unfortunately, the Event class is internal to RPG Maker and cannot (easily) be changed. For our proof-of-concept, however, it will work fine.

Before we talk about the variables @net_npc_start and so on, let’s add the placeholder NPCs to the map. Create five npcs and place them somewhere near the top of the map, like so:

Put our five dummy NPCs out of range. Make sure they have increasing IDs.

You must create these NPCs in order, so that they have increasing IDs. Confirm that they do, indeed, have increasing events IDs (they should be named Event 006, Event 007, etc.). The first NPC should be set up in this way:

• Page 1 has no graphic, and is below the hero. (You might add a graphic for debugging purposes, just so you can confirm when this NPC moves.)
• Page 2 checks if variable 10 (called NPC1-Avatar ID) is 1 or above. It is a random-walk NPC at the same level as the hero, and has the first hero’s walkabout as its graphic. Its event code consists of two lines:
  • Set variable 9 (called CurrTalk NPC) to be equal to 6. (Or, whatever your NPC’s id is).
  • Call Common Event 4 (called Show NPC Message)
• Pages 3 through 6 are just like page 2, except all the red text is incremented by one. In other words, there is a page for each hero graphic.

The first NPC should look something like this, on the last page:

The last page of the first NPC in our series of dummy NPCs.

Now, alter the other NPCs, except increment the blue text by one. You’re basically creating a new NPC for each character in the virtual world, with a variable to change its appearance. The common event Show NPC Message will be used to centralize all communication.

Finally, go back to your Game_Map code, and put the following in the initialize method, directly before the call to create_vehicles:

  @max_network_npcs = 5 #How many max?
  @net_npc_start = 6 #Start at ID 6
  @net_npc_picvar_start = 10 #Start at ID 10

You might have to change @net_npc_start if your first NPC’s ID was something other than 6.

All that’s left is to hook up $game_network.connect() and the disconnect function. Open the database (F9) and go to the Common Events tab. Go to Common Event 1, “Connect to a server” menu, and add the following to the existing “Choice” command.

• For “When \N[7]”, add the script:

    npcName = $game_actors[7].name
    $game_network.connect npcName

• For “When \N[8]”, add the script:

    npcName = $game_actors[8].name
    $game_network.connect npcName

Now, go to the network NPC you created earlier, and add a new page with the same graphic. For this page’s “conditions”, select “Switch 2 (Network ON) is ON”. Then, add the following event commands:

Our "Connect" NPC will ask us to disconnect if already connected.

Described briefly:

• The NPC will ask if you want to disconnect. If you say “Yes”, he will call $game_network.ask_to_disconnect.

At this point, our prototype is complete. The only thing missing is the server. Our server will be written in Java, and it only performs three important functions:

• Track connection requests and responses, as well as disconnections and timeouts.
• Maintain a list of all online users and their avatars.
• Perform required bookkeeping; making sure that users receive all relevant messages.

Here’s the code for this in Java:

import java.io.*;
import java.net.*;
import java.util.*;
public class VXServer implements Runnable {
  //Data
  private int port;
  private Hashtable<Integer, GameConnection> players;

  public VXServer(String port) {
    this.port = Integer.parseInt(port);
    this.players = new Hashtable<Integer, GameConnection>();
  }

  //Get the next available ID
  private int getNextID() {
    for (int nextID=1; ;nextID++) {
      if (!players.containsKey(nextID))
        return nextID;
    }
  }
  //Repeatedly accept new connections
  public void run() {
    try {
      ServerSocket mainSocket = new ServerSocket(port);
      System.out.println("Server started on local port: " + port);
      for ( ; ; ) {
        Socket childSocket = mainSocket.accept();
        int id = getNextID();
        System.out.println("New player connected; given id: " + id);
        GameConnection gc = new GameConnection(childSocket, id);
        synchronized (players) {
          players.put(id, gc);
          gc.startThreads();
        }
      }
    } catch (IOException ex) {
      System.out.println("Main socket error: " + ex.toString());
      ex.printStackTrace();
    }
  }
  public void removeClient(String reason, int id) {
    String msg = "UDIS:" + id;
    GameConnection remPlayer = null;
    synchronized(players) {
      //Remove from hash
      System.out.println("Closing client[" + id + "] - " + reason);
      remPlayer = players.remove(id);
      //Send a message to all other players
      for (int x : players.keySet()) {
        players.get(x).sendMessage(msg);
      }
    }
    //Disconnect
    remPlayer.stopThreads();
  }

  //Main method; processing starts here
  public static void main(String[] args) {
    if (args.length<1)
      new Thread(new VXServer("7689")).start();
    else
      new Thread(new VXServer(args[0])).start();
  }

  //////////////////
  //Helper class
  //////////////////
  class GameConnection {
    //Data
    public int id;
    public int avatarID;
    private Socket s;
    private boolean active;
    private BufferedReader readStream;
    private BufferedWriter writeStream;
    private Thread readFromClient;
    private Thread writeToClient;
    private ArrayList<String> pendingToSend;
    public GameConnection(Socket s, int id) {
      this.s = s;
      this.id = id;
      this.avatarID = avatarID;
      try {
        this.readStream = new BufferedReader(new InputStreamReader(s.getInputStream()));
        this.writeStream = new BufferedWriter(new OutputStreamWriter(s.getOutputStream()));
      } catch (IOException ex) {
        throw new RuntimeException(ex);
      }
      this.pendingToSend = new ArrayList<String>();
      this.readFromClient = new Thread() {
        public void run() { while(active) { readLoop(); } }
      };
      this.writeToClient = new Thread() {
        public void run() { while(active) { writeLoop(); } }
      };
    }
    public void startThreads() {
      this.active = true;
      this.readFromClient.start();
      this.writeToClient.start();
    }
    public void stopThreads() {
      this.active = false;
      try {
        this.readFromClient.join();
        this.writeToClient.join();
      } catch (InterruptedException ex) {}
      try { this.s.close(); } catch (IOException ex) {}
    }
    public void sendMessage(String msg) {
      synchronized(pendingToSend) {
        pendingToSend.add(msg);
      }
    }
    private void writeLoop() {
      //Wait for a message (messages are NON_null terminated)
      String nextMsg = null;
      for ( ;nextMsg==null; ) {
        synchronized(pendingToSend) {
          if (!pendingToSend.isEmpty())
            nextMsg = pendingToSend.remove(0).trim() + "\n";
        }
        if (nextMsg==null) { try { Thread.sleep(10); } catch (InterruptedException ex) {} }
      }
      //TEMP
      System.out.println("Sent message: " + nextMsg.trim());

      //Send message
      try {
        writeStream.write(nextMsg, 0, nextMsg.length());
        writeStream.flush();
      } catch (IOException ex) {
        removeClient(ex.toString(), this.id);
        return;
      }
      //Quit?
      String[] segments = nextMsg.split(":");
      if (segments[0]=="UDIS" && Integer.parseInt(segments[3])==this.id)
        removeClient("Client requested disconnect.", this.id);
    }

    private void readLoop() {
      for ( ; ; ) {
        //Read a message
        String message = "";
        try {
          message = readStream.readLine();
        } catch (IOException ex) {
          //Close the server
          removeClient(ex.toString(), this.id);
          break;
        }
        //TEMP:Debug:
        System.out.println("Read message: " + message.trim());

        //Split
        String[] segments = message.split(":");

        //React to the message
        if (segments[0].equals("CREQ")) { //Connect Request
          //Save Avatar
          this.avatarID = Integer.parseInt(segments[1]);

          //Send this player a connect_response with his ID
          String msg = "CRES:" + this.id;
          this.sendMessage(msg);
          //Send this client's avatar ID to everyone ELSE
          msg = "UCON:" + this.id + ":" + this.avatarID;
          synchronized(players) {
            for (int x : players.keySet()) {
              if (x!=this.id)
                players.get(x).sendMessage(msg);
            }
          }
          //Send to THIS player all previously-connected players.
          synchronized(players) {
            for (int x : players.keySet()) {
              if (x==this.id || players.get(x).avatarID==0) //Might not have been initialized yet.
                continue;
              msg = "UCON:" + players.get(x).id + ":" + players.get(x).avatarID;
              this.sendMessage(msg);
            }
          }
        } else if (segments[0].equals("UDIS")) { //(Request for) User Disconnect
          //Can only disconnect self
          int newid = Integer.parseInt(segments[1]);
          if (newid == this.id) {
            //Send a disconnect message to this player
            // We close the connection after sending our own message
            String msg = "UDIS:" + this.id;
            this.sendMessage(msg);
          }
        }
      }
    }
  }
}

At over 200 lines, the server code is somewhat long, but that’s usually what ends up happening with Java. Our server functions in a relatively simple manner. At first, there is only one thread, containing the ServerSocket. This is basically the “connection” class, which will constantly wait for connections from our clients. Each time a connection is detected, the main thread creates a small helper class with the socket it just opened. This class spawns two more threads, one for reading from the socket and the other for writing to it. Thus, the total number of threads on the server at any one time is equal to 2 X number_of_clients + 1. There are better ways of doing this (e.g., using the NIO library), but they would require us to modify our client (RPG Maker) code in ways that are somewhat difficult.

Anyway, once the server is running, each client thread simply reads and writes messages as one would expect. Because the code for our server is contained within one file, we can access most functions without requiring any Interfacing hacks. All in all, it’s a rather tiny server.

Compile your server (Ctrl+1 in TextPad), and then run it (Ctrl+2). It will use the default port (7689) on localhost (127.0.0.1). Open RPG Maker and run your game. Talk to the Network NPC and select the first connection (assuming you didn’t modify it). Switch back to the server window; hooray! you’re connected.

Our server window has messages from both players.

Run another game, and you’ll see your player there. Disconnect and connect as many clients as you like. However, for now I recommend disconnecting by closing the game client; for some reason, using the NPC to disconnect will occasionally crash the server. (This is undoubtedly a flaw in my server code, not a bug with networking in RPG Maker games.)

Our NPCs connect, and they walk around randomly.

Everything works just fine; when you close the second window, its corresponding NPC will disappear from the first’s window. Isn’t that amazing?

Game Prototype 3 – Chatting with Other Players

We could stop here; our point has been proven. However, Step 2 was particularly time-consuming, and I don’t want you to think that network games are so tough. In fact, with the simple infrastructure we’ve developed for part 2, we can easily add the remaining functionality in a simple, modular way. (Of course, we’ll eventually reach the point where the network becomes saturated with messages. At that point, it will become very difficult to proceed, because we’ll have to start talking about interest management techniques.)

Let’s add client messages as a kind of “poor man’s chat” interface. Surprisingly, the networking is now the easiest part:

We will use the same network message to send a message request to the server, and to receive a message from the server (on behalf of another player). Add the following to Game_NetMessage:

  def self.CHAT_MESSAGE
    "CMSG" 
  end

And:

  def self.Make_ChatMessage(playerNID, msg)
    "#{Game_NetMessage.CHAT_MESSAGE}:#{playerNID}:#{msg}\n"
  end

Add the following to Game_Network:

  def sendChatMessage(msg)
    #Note that this MIGHT occur if we haven't heard back from the server 
    # yet. We might have to manually police this case later.
    raise "Cannot send message; you are not connected!" if @my_id == -1
    toSend = Game_NetMessage.Make_ChatMessage(@my_id, msg)
    $game_shared.queueClientMessage(toSend)
  end

Add the relevant message handling code to Game_Map, in the update_network function, right after the code for handling USER_DISCONNECT.

      elsif segments[0]==Game_NetMessage.CHAT_MESSAGE
        #Retrieve the message
        cmsg = segments[2]

        #Save the player's message
        avtID = segments[1].to_i - 1
        @events[@net_npc_start + avtID].chat_message = cmsg

        #Show an emotion icon
        @events[@net_npc_start + avtID].balloon_id = 1
      end  

This, of course, requires a new property to be added to Game_Event. Add the following to the Game_Event class, below the other attr_readers

  attr_accessor :chat_message             # Current message from other user

…and in the initialize method:

  @chat_message = ""

Thus, each NPC now has a property, chat_message, which is updated whenever a network message of type CHAT_MESSAGE. From here, the code for the Show NPC Message Common Event should be obvious. First, hit F9 and go to the Actors tab. Press “Change Maximum…” and enter a number like 15. Then, name NPCs 10, 11, 12, and 13 to CurrNPCTxt Line 1, CurrNPCTxt Line 2, CurrNPCTxt Line 3, and CurrNPCTxt Line 4. Now, edit the Show NPC Message Common Event:

Despite its length; this common event is actually quite simplistic in nature.

This is one of those “exploding events” that PRG Maker famously espouses. Briefly described:

• We first run a simple script that takes the current NPCs chat_message and splits it whenever there’s a tab (\t). (We can’t split on newlines because newlines denote new messages in our network protocol.) The reason our script is so complex is so that we can handle cases such as the empty string (“no message”) or a string with less than 3 tabs (messages with less than 4 lines).

   #What NPC is talking? What's he saying?
   currMsg = $game_map.chat $game_variables[9]
  
   #Set NPCs 10..13 to be the lines of text
   lines = currMsg.split("\t", -1)
   rep = lines.empty? ? 4 : 4-lines.length
   lines += [""]*rep
   for id in (0..3)
    $game_actors[10+id].name = lines[id]
   end

• Next, we run a much smaller script with sets the value of variable 8 (which you can rename to Curr NPC Avatar) to be equal to value of the avatar variable for the given NPC. I thought this kind of functionality was offered by RPG Maker’s standard events, but I couldn’t find it anywhere.

   #Set variable 8 to the current NPC's avatar
   index = $game_variables[9] - 6 + 10
   $game_variables[8] = $game_variables[index]

• Next, we have a big mess of If statements. Each “if” statement wraps the following text box command:

     \N[10]
     \N[11]
     \N[12]
     \N[13]

• …this uses the values we just stored in the Actors’ names to spell our out message. The only difference between each text box in the if statements is their face picture; we change this based on the value of variable 8 (Curr NPC Avatar).

The first script in our event will require the following method to be added to Game_Map:

  #Helper
  def chat(npcID)
    msg = @events[npcID].chat_message
    msg ? msg : ""
  end

The name was kept short to prevent overflowing the tiny script box inside RPG Maker’s Script Event Command. The reason we check for nil values and replace them with the empty string is that some of the functions that call chat will expect strings to be returned.

So now, messages work –presuming that we can actually send one. The problem is, text entry in RPG Maker is notoriously obnoxious. We could use the “Hero Name” dialog to spell out the sentence word-by-word, and then display the partial sentence like we did with the IP dialog. This would be clumsy at best. We could provide a set of partial phrases, and allow the player to select lines 1 though 4. This would actually be a minor pain too. From the user’s point of view it would be simplest to make our own Scene_* class that traps key events and lets them just type out the sentence. But that could easily double the length of our simple prototype. Since the problem cannot be solved simplistically, I’ll just punt on it: we’ll give the player a set of four predefined messages. First, add the following event command to the end of Common Event 4, Show NPC Message:

   Call Common Event 5

Name Common Event 5 something like Respond to NPC and give it the following code:

Event that allows a player to communicate with everyone else.

This Common Event is easy to understand, although it is clearly a too cold type:

• Ask the player if he wants to respond to that NPC.
• If so, let them choose a simple (one-line) message, a multi-line message, or just a glitchy message (to test what happens). I know, boring, huh?
• Then, simply use the function we put in $game_network to send the message.

    msg = "Hi there, friend!"
    $game_network.sendChatMessage(msg)

Pretty simple, although I think it’s a bit of a bad abstraction (since we talk to a single NPC to change our message to all NPCs. But that’s a minor point; we could change this easily later if we wanted to. Anyway, all that’s left is our server code. In the function readLoop, add the following case:

        } else if (segments[0].equals("CMSG")) { //Send a chat message to all clients
          //Can only send messages from yourself
          int fromID = Integer.parseInt(segments[1]);
          if (fromID == this.id) {
            //Send the same message to all other users
            synchronized(players) {
              players.get(fromID).lastMsg = segments[2];
              for (int x : players.keySet()) {
                if (x!=this.id)
                  players.get(x).sendMessage(message);
              }
            }
          }
        }

You’ll have to change the definition of GameConnection to include this “last known message” parameter:

    public String lastMsg;

…and, in the constructor:

      this.lastMsg = "";

You will, of course, need to add a similar broadcast to the UCON case. We can modify the last use of synchronized(players) to include this:

          //Send to THIS player all previously-connected players and their messages
          synchronized(players) {
            for (int x : players.keySet()) {
              if (x==this.id || players.get(x).avatarID==0) //Might not have been initialized yet.
                continue;
              msg = "UCON:" + players.get(x).id + ":" + players.get(x).avatarID;
              this.sendMessage(msg);
              msg = "CMSG:" + players.get(x).id + ":" + players.get(x).lastMsg;
              this.sendMessage(msg);
            }
          }

I’ve named this server VXServer2, and included its full listing in Code Segment 3, below, in case you’re not familiar enough with Java to insert the code yourself.

Code Segment 3 — Make sure this is in a file called VXServer2.java, or it won’t compile.

Anyway, compile it (Ctrl+1) and run it (Ctrl+2) then try out our nifty message code.

It looks like your friend has something to say to you.

The exclamation also appears at potentially inappropriate times, like when the NPC first logs on, but I don’t think that’s such a bad thing –it helps by alerting the player that a new NPC has connected.

A message which, if not user-defined, is at least user-chosen.

The server scales nicely, although if you’re running these examples locally, you’ll be limited by the fact that RPG Maker “pauses” all game windows except the currently-active one (so messages won’t be received). Nonetheless, the effect is impressive:

Several NPCs; we are talking to one while another indicates that he has a new message for us. Pretty cool.

Project

For the first time, I’ve decided to release the final project’s code as a sample. Here’s a zip containing everything you’ll need; make sure you read the article to understand how it works. As always, all code is in the public domain, unless it is borrowed from another author (in which case I have noted it) and is thus released under the license chosen by the respective author. All such licenses are open-source in nature.

At the moment, I cannot find a good site to host the project file. However, I posted the sample code on the RPG Maker VX Forums; you can download it from here (but you’ll need an account). If I can find a good hosting site, I’ll upload a direct link. Apologies for the inconvenience, readers (but it’s a nice forum anyway).

Further Work

I’m personally quite pleased with what we’ve accomplished; however, it’s by no means polished. There’s still bugs in the system, and there’s a great many improvements that could be made. Here’s some “homework” for you; I might get around to writing a post about one of these eventually, but for now I’m tired of writing server code.

• BUGFIX: Send a “glitch” message (option 3) and then try to read it with another player. Surprisingly, the client will crash! This was actually unintentional on my part, but it presents a nice, easy exercise for you. Figure out where the system is glitching up: server side, client side, or somewhere in between. Then fix it!
• BUGFIX: For some reason, closing the network connection from the NPC might crash the server. Also, hitting “F12” to go back to the title screen might have a negative effect on the connection. Re-write parts of the server and client to be more robust at handling this type of thing.
• FEATURE: Once connected, we should measure the average lag to the server, and print this in the lower-right corner of the screen. Perhaps use the Debug Window to design some kind of “connected” box which always hovers in the lower-right corner. Bonus points if you animate it with a spinning Internet icon.
• FEATURE: Combine this article’s source with the NPC Talk Boxes one, and make NPC messages appear above their heads whenever they change. You can still show the exclamation emote when they first connect.
• FEATURE: Our NPC code uses “sir” a lot, even if the connected player is clearly a female. Extend the game engine to handle strings like \sir, which expand into “sir” if the player is male and “ma’am” if the player is female.
• FEATURE: Add user accounts and passwords, and do this in a way that keeps users’ passwords safe and un-crackable –a salted hash comes to mind.
• PROJECT: Allow players to fight each other. This isn’t as difficult as it seems, since RPG Maker’s battle system is turn-based. You can extend Scene_Battle to expect network input and wait until both teams have made their decisions. Then, you might add an icon to show if the other player is “ready”, “disconnected”, etc. Finally, you’ll have to take the opposing player’s “stats” and somehow turn them into monsters which you place on the opponent’s screen.
• PROJECT: Assuming the previous project has been completed, add the ability for players to form “teams” and fight against each other at the party level. You can update your heads-up-display to show the moves your teammates have chosen, and you’ll probably want a team-chat and battle-chat feature (with real-time typing, not RPG Maker’s name entry dialogue). I’m giving this project a gold star, since this is the project I would most like to do myself.
• PROJECT: Extend our code to handle events arbitrarily. In other words, make it so that we can do something like $game_map.add_npc(Event(3, 5)) to create a new NPC at location (3,5). This is harder than it sounds, and really only makes sense when preparing for the next item.
• PROJECT: Track NPC movement in real-time. You might have to move our update_network code to a more prominent location –say, the place where RPG Maker handles user input. Regardless, the key challenge with this project is network congestion; sending 60 messages per NPC per second is a sure way to bog down your server. In other words, only update what’s necessary. You might, for example, send a USER_LEFT message when the player wants to move left. The server can then determine if moving left is allowed, and send back OK_TOGO_LEFT or NOGO_LEFT to inform the player. However, this will introduce some noticeable lag on the client side, so you might want to allow the client to begin moving left anyway, and just teleport him back if NOGO_LEFT was received. A careful consideration of when to attempt a move before receiving a response from the server is difficult, and will require extensive testing. On that note, you’ll have to test this code from multiple computers, since (1) there’s no lag otherwise and (2) RPG Maker pauses all but the top-most window. In fact, this pausing is a problem, since it interrupts the flow of messages. You’ll need to account for that somehow (perhaps force the game to remain full-screen? Or install a Windows hook that tricks RPG Maker games into thinking they’re always a top-level window?). Either way, this project will easily take weeks of your time to get right.

Happy coding!

Update: Phoenix Demo

Speed used the code in this post (along with tons of other cool scripts, including a mouse capture script!) to make a really awesome demo called Phoenix. Check it out here.

Some screenshots:

My character, loaded into the main game world.

Loading of a second character, tracking down the first.

New features added by Speed include:

• Characters are tracked by x/y coordinates, and across various maps.
• Full-featured chatting, account creation, password storage, etc. (He saves data in Text files).

…plus a lot more. Pretty cool to see my code in action. And now, back to listening to OC Remixes.

Quick Note: I’m still trying to fix some of the formatting, but the content’s all here, so I decided just to post it. I’ll re-tab the untabbed code later.

Back-story: Our previous notifier was pretty hackneyed. It was also heavily coupled with the client program’s source, which might not be such a good idea. (What if the developer released the next version in Haskell or Forth? What if he closed the client source for the next release? What if —and this is the most likely— we don’t want to make our notifier any fancier using Win32 C++?) We need a new solution, and that means it’s time to pull out the big guns.

Goal: Write a new notifier plugin. Make it operate external to the Eternal Lands program. Don’t slow down the client or the server. Don’t break any rules. Allow multiple types of notifications. Make it flashy.

Licensing Notice

This post provides a lot of code. While I normally provide all code in the public domain, I am restricting this post’s code with a license. Scroll to the bottom of the post to read it. This license is not for my benefit, but for the benefit of the EL servers and admins. Basically, it prohibits you from using this code to automate, or to modify it in any way that causes undue strain on the servers. You cannot inject events, nor can you delay sending them between the client and server. Finally, you cannot modify the thread priority of the event handler. The EL team has volunteered their time and creativity for your benefit, and I won’t have my code used to cause trouble for them.

Step 1: General Approach

Most of the problems in our last notifier stemmed from hastiness. In light of that, this section will take it slow and explain why we chose to do things the way we did. We’ll discuss some alternative approaches. But first, a general overview of what we want, and why our last plugin is no longer sufficient:

• Event Notification — We want to see a message when something happens in Eternal Lands. Our initial program tagged harvest completion, but that was all. (Bonus question: How can you stop harvesting without displaying the “You stopped harvesting” message? It’s a tricky one… if you’re not sure, try harvesting some Seridium until you run out of Matter Essences.)
• Notification Timeliness — We want to show notifications only when necessary, and hide them when they’re not needed anymore. The previous plugin showed them if the Eternal Lands window was not the foreground window, and hid them when a new message came in. Later, we found this to be unsatisfactory. Better to hide all messages when the EL window gains focus.
• Variety of Notifications — Seeing a non-descript “You stopped harvesting” is different from seeing a shiny “You found an Enrichment Stone!” or a red “You have died and gone to the underworld”. We’ll add multiple types of notifications, which will necessitate allowing them to stack, fade in, and fade out. If you guessed that we won’t be doing all this graphical work in Win32 C++, you guessed right!
• Separation from Client Code — We won’t be using C++, nor will we be building directly on top of the Eternal Lands source. Rather, we’ll build a separate module that will load the EL client, and then filter all messages sent to it. We also want a system tray icon, and we might even consider removing the Eternal Lands window from the taskbar entirely.
• Fast and Extensible — We want the code to be fast (sorry, Perl). We want it to be easy to add new filters (sorry, C). We want it to operate directly on the network data (sorry, Forth) and have a great graphics library. If it’s native to Windows, all the better. Our choice is C#.

Why not just read the client log file? If you browse to “C:\Users\User Name\Documents\Eternal Lands\main\chat_log.txt” —possibly replacing “main” with the name of the server you plan on, and “Documents” with “My Documents” on Windows 7— you’ll find a nice log of (most of) the chat window. Fiddle a bit more, and you’ll find that it’s updated in real time. Why, then, do we not simply tail this file? In Windows, this is pretty easy to do in Java:

public void run() {
  File f = new File("C:\\ ... \\chat_long.txt");
  long fPointer = ;/* Seek to the end of the file */
  for (;;) {
    Thread.sleep(100);
    long len = f.length();
    if (len < fPointer)  //Log was reset
      fPointer = len;
    else if (len > fPointer) { //File was appended to
      RandomAccessFile raf = new RandomAccessFile(f, "r");
      raf.seek(fPointer);
      String line = null;
      while ((line = raf.readLine()) != null)
      processMessage(line);
      fPointer = raf.getFilePointer();
      raf.close();
    }
  }
}

Don’t use this code; it doesn’t check for exceptions, and it isn’t mine (it’s part of the java-tail project). More importantly, it doesn’t really work if the file is being constantly added to. You could re-read the file after exceptions, but then you’re entering into the messy possibility of reading the same line twice.

On a totally different level, this is sub-optimal because the client log doesn’t contain everything. What if you wanted to write a filter that tells you if you’ve been interrupted from harvesting by a PKer instead of a sub-lethal cave-in? Battles don’t send messages to the console, unless you die. Wouldn’t you rather find out while you still have a fighting chance?

Why not use Java? This question applies to any other language, but I feel the most reasonable alternative is Java. (I consider Visual Basic .NET to be identical to C# for all such discussions, by the way.) Java is a very good language, and there is no reason not to use it. Your favorite language is also probably a good choice, although I might not have heard of it. My reason for choosing C# was that, of all the candidates I considered, it was capable of (read: “I knew how to use it to”) solving all the sub-tasks I needed. Feel free to follow along in Python or Ruby or Tcl/Tk, converting the code as you like. Ooh, or Delphi; I like Delphi. Regardless, I’m doing this in C#, and any intermediate programmer should be able to follow along.

Why not patch the client code? My earlier reasons for not simply extending on our earlier project were true, but also somewhat motivation by the fact that I knew I was going to use a much higher-level (and thus incompatible) language from the start. What if you started this project using C++? In that case, I’d still encourage you to keep your code totally separate. Besides all the previously-listed reasons, here’s one more: bugs. If you’re recompiling the entire Eternal Lands code base with each new notification, then you’re bound to slip up from time to time. This is part of development; you can release bug-free software, but your internal builds will be fraught with bugs. Now, given the nature of bugs in C++, this means that your code could easily corrupt some of Entropy’s code. If this results in sending a million messages a second to the server, “ya might get banned”. That said, if your bug is passive, and other people use your plugin, you start to run a real risk of bringing down the server. And even if your bug is harmless, it will still confuse developers trying to nail it down. Trust me when I say that tracking a bug to third-party patches does not make developers happy.

How will we catch messages? We’ll start our program first, which should give it priority over any child processes and their system resources. Then, we’ll connect our program to the Eternal Lands server, and connect the EL Client to our program (by adding a new server to the config file). Finally, as we ferry messages between client and server, we’ll send them off to a separate, low-priority thread where we can match then against an arbitrary number of filters, spawning pop-up windows as we identify messages of interest.

Step 2: Connecting to the client

The first thing we need to do it connect to the server. From the Eternal Lands server config file (D:\Programs\Eternal Lands\servers.lst), we know that the “main” server connects to port 2000 on game.eternal-lands.com. Open up Visual Studio (the free Express Edition for C# is pretty nice) and start a new project. Make it a “Windows Forms Application”. You’ll be presented with the design view for “Form1.cs”. In case you aren’t familiar with Visual Studio, take a minute to note the toolbox (circled in red on the left) and the Solution explorer on the right. Right click on Form1.cs in the solution explorer and note the two options shown circled in blue. “View Designer” will bring up the form designer, like you see now. “View Code” will bring up the code that runs everything behind the scenes. Note that C# supports “partial” classes, so the “View Code” option will not show you any code generated by Visual Studio, unless you are running a very, very old version of Visual Studio.

Visual Studio's "Hiding Tool Bars"

Open the toolbox (by clicking on it) and drag a “Text Box” control onto your Form. (This is under the “Common Controls” group). Now, right-click on this box and choose “Properties”. The most important property is called “(Name)”, because this is what we’ll use to identify the control in our C# code. The following table details which controls to add, and what properties to set for them. All other properties can be left at their defaults, or you can change them to your liking. For example, you might change the “font” for your controls, or the “BorderStyle”. Go for it.

*Note: If you installed Eternal Lands to a different location, set the Text property accordingly. Escaped backslashes are not necessary when entering this string in the visual editor.

Arrange these components like so:

Line Up Your Components Roughly Like This

A quick note on Hungarian Notation: in general, I do not support this hackneyed way of naming variables, especially in a strongly-typed language like C#. However, for GUI components, I often find myself saying “Ah, what was the name of that button I wanted?” By naming controls in an ordered fashion, I can type btn<Space> and get some help from Visual Studio’s auto-complete feature. Having large numbers of (essentially) global variables for your form controls is unavoidable in C#, and I find this approach scales really well, despite the frown that most people get when they see Hungarian Notation.

Compile your program (Build –> Build Solution) and then run it (Debug –> Start Without Debugging). None of the software logic is hooked up yet, but the general idea is that the user can change the client directory, then click “Read” to read the list of available servers. Next, the user can choose a server to connect to from the combo box, and click “Connect”. Log messages will be shown in our multi-line Text Box control, and the mysterious “Zero Count” will be used for something which will be explained later. Well, let’s start the magic.

Click on the “Read” button, and go to its properties. Notice that little lightning bolt, circled below in red? If you click that, you will switch to “Event” triggers. Click the button left of it to go back to properties view.

Bad Design Puts This Very Important Button in a Very Obscure Place.

Click the event button, find the event called Click, and type “ReadELFiles” in the box. Then, press “Enter”. Visual Studio will bring you to Form1.cs’s Code View, with a skeleton implementation of the ReadELFiles function used for your Click event delegate. Java users will be saying “What?”, as delegates and events make little sense from a minimalist point of view. We’ll have a good example of the power of delegates later.

Here’s the implementation of ReadELFiles:

private void ReadELFiles(object sender, EventArgs e)
{
  //Reload an entirely new list of servers.
  cmbServers.Items.Clear();

  //Don’t load anything if we specified an invalid directory.
  if (!File.Exists(txtPathToELFolder.Text + "\\" + "el.exe"))
  {
    txtConsole.Text = "Error: el.exe doesn't exist in EL folder";
    return;
  }
  else if (!File.Exists(txtPathToELFolder.Text + "\\" + "servers.lst"))
  {
    txtConsole.Text = "Error: servers.lst doesn't exist in EL folder";
    return;
  }

  //Open our servers file and read it.
  bool foundLoopback = false;
  StreamReader srvFile = new StreamReader(txtPathToELFolder.Text + "\\" + "servers.lst");
  while (!srvFile.EndOfStream)
  {
    String line = srvFile.ReadLine();
    if (line == null)
      break;

    //Coment? Empty?
    line = line.Trim();
    if (line.StartsWith("#") || line.Length == 0)
      continue;

    //Read:
    //ID, Config_Dir, Address, Port, Description
    String regex = "([^ \t]+)[ \t]+([^ \t]*)[ \t]+([^ \t]*)[ \t]+([^ \t]*)[ \t]+(.*)";
    Match m = Regex.Match(line, regex);
    ServerInfo sInf = new ServerInfo();
    if (m.Success && m.Groups.Count == 6)
    {
      sInf.id = m.Groups[1].Value;
      if (sInf.id.Equals("loopback"))
        foundLoopback = true;
      sInf.configDir = m.Groups[2].Value;
      sInf.address = m.Groups[3].Value;
      try
      {
        sInf.port = Int32.Parse(m.Groups[4].Value);
      }
      catch (FormatException)
      {
        //Don't add
        continue;
      }
      sInf.description = m.Groups[5].Value;
      cmbServers.Items.Add(sInf);

       //Loopback?
      if (sInf.id.Equals("loopback"))
        loopbackInfo = sInf;
    }
  }
  srvFile.Close();

  //Do we need to add our own local server?
  if (!foundLoopback)
  {
    //Copy existing server file
    StreamReader inFile = new StreamReader(txtPathToELFolder.Text + "\\" + "servers.lst");
    StreamWriter outFile = new StreamWriter(txtPathToELFolder.Text + "\\" + "servers.lst.new");
    for (String line = inFile.ReadLine(); line != null; line = inFile.ReadLine())
      outFile.WriteLine(line);
    inFile.Close();

    //Add our own category
    outFile.WriteLine(srvLine);

    //Close, swap, notify.
    outFile.Close();
    File.Delete(txtPathToELFolder.Text + "\\" + "servers.lst");
    File.Move(txtPathToELFolder.Text + "\\" + "servers.lst.new", txtPathToELFolder.Text + "\\" + "servers.lst");
    ShowConnectMsg("Created loopback entry.");

    //Add a new item to the list
    loopbackInfo = new ServerInfo();
    loopbackInfo.id = "loopback";
    loopbackInfo.configDir = "main";
    loopbackInfo.address = SERVER_ADDR;
    loopbackInfo.port = SERVER_PORT;
    cmbServers.Items.Add(loopbackInfo);
  }

  if (cmbServers.Items.Count > 0)
    cmbServers.SelectedIndex = 0;
}

Make sure you add the following using directives to the top of Form1.cs, otherwise you’ll get errors stating that “The name <something> does not exist in the current context”. You also won’t get auto-complete functionality until the proper directives are added. (Auto-imports is something I miss dreadfully from Eclipse.)

  using System.Text.RegularExpressions;
  using System.IO;

You will also need the following variables; add them as part of the class (e.g., one line after the opening brace following “public partial class Form1 : Form”)

public const string SERVER_ADDR = "127.0.0.1";
public const string SERVER_PORT = 5421;
private string srvLine = "127.0.0.1";"loopback        main            " + SERVER_ADDR
      + "               " + SERVER_PORT + "    Loopback adapter for use with the EL Notifier";
private ServerInfo loopbackInfo;

We also need this “ServerInfo” structure everyone’s been talking about. Add it as part of the namespace (not the class Form1).

public struct ServerInfo
{
  public String id;
  public String configDir;
  public String address;
  public int port;
  public String description;

  public override String ToString()
  {
    return description;
  }
}

The “ToString()” method is called by the Combo Box control to display the Object it contains; overriding it allows us to dictate that ServerInfo items are displayed by the description. There are better (but less concise) ways of controlling Combo Box output.

Note that structs in C# are very different than structs in C++. Indeed, you could have used a class here; the only difference would have been in its copy semantics, and possibly its memory footprint.

The only thing you’re missing is the ShowConnectMsg delegate, but for now just assume it’s a standard function call that does some non-essential logging. Let’s dissect the code we just wrote!

The SERVER_ADDR and SERVER_PORT variables help us to set up our program as a middleware between the EL Client and the EL Server. Our program has both client and server aspects; these two variables define access to the latter. We use srvLine to shim the servers list so that it can access our “Loopback” interface. Note that srvLine is effectively constant, even though it is not declared that way.

We are basically equivocating on the program’s socket expectations. From Wikipedia:

An Internet socket is characterized by a unique combination of the following:

• Protocol (TCP, UDP or raw IP).
• Local socket address (Local IP address and port number)
• Remote socket address (Only for established TCP sockets.)

The original connection from EL Client to EL Server has one socket: a TCP-enabled connection from 127.0.0.1:???? to game.eternal-lands.com:2000. This socket is bi-directional. We don’t know the local port, because local ports are almost always randomly assigned. We don’t technically know the server port either, but 2000 is what we connect to, so I’ll list that here.

Our shimmed middleware creates two bi-directional sockets:

TCP, 127.0.0.1:???? ? SERVER_ADDR:SERVER_PORT
TCP, SERVER_ADDR:???? ? game.eternal-lands.com:2000

Note that, as per the usual, we don’t know the local ports. Either way, we are guaranteed that they will be unique.

All that explanation for just three lines of code? Well, if you’re a non-network guy like me, figuring this out is one of the hardest parts of this project. Now, on to the rest of the code:

The ReadELFiles function is pretty easy to follow:

• Note that the function parameters “sender” and “e” are not used in our function. These are mandated by the .NET platform, but are not needed for our simple example.
• The foundLoopback variable is used to check each line read from servers.lst. If the id of that line is “loopback”, then we know a loopback entry exists. If, at the end of scanning, no such entry was found, we just add it to the end of the file. That way, when we start Eternal Lands, we can connect to the loopback server (instead of main or pk or testing) and fool the client into thinking it’s connecting to the real server, instead of our fake, localhost middleware.
• Note that we should check the loopback entry to ensure that its address and port parameters match ours. I’ll leave this as an exercise in regular expressions.
• Some regex pattern matching is used to filter each entry in our servers.lst file. We use regexes to check if a line matches a specific pattern, and we then use the Groups property to extract what was matched. If you don’t understand pattern matching, just skip the details. Learn it later, of course —it’s a fantastic way of dealing with text in almost any modern language.

We will now add the final missing piece of the puzzle. Add the following to the namespace (not the Form1 class):

public delegate void ShowConnectMsgDel(String msg);

Then, add the following function definition to the Form1 class:

private void ShowConnectMsg(String msg)
{
  Control ctl = txtConsole;
  if (ctl.InvokeRequired)
    ctl.Invoke(new ShowConnectMsgDel(ShowConnectMsg), msg);
  else
    ctl.Text = msg;
}

Our program will now compile. (I’ll explain delegates later, when it makes more sense to.) Run it, click the “Read” button, and you will see “Main game server” appear in the still-disabled Combo Box.

A Properly-Formatted Servers File Is Parsed.

If your program doesn’t compile, have a look at Listing 1 and try to figure out where you went wrong. (Apologies for the PDF; WordPress doesn’t like .txt files.) If it doesn’t load the server list, make sure you supplied the proper path to the EL Client folder.

Now, add the following code to the end of your ReadELFiles function:

//Enable the remainder of our form
cmbServers.Enabled = true;
btnConnect.Enabled = true;

That should enable everything you need. Now, in Design mode, click on the “Connect” button and give it a click Event called ConnectToServer. You’ll be given a function skeleton again; here’s what we’ll add to it:

private void ConnectToServer(object sender, EventArgs e)
{
  //Connect to our server
  UpdateConsole("Connecting.....");
  Thread t = new Thread(new ThreadStart(Check_Server));  
  t.IsBackground = true;
  t.Start();
}

This requires the CheckServers function, which will run in a separate thread:

private void Check_Server()
{
  //Our middleware acts as a client in this case.  
  TcpClient tcpclnt = new TcpClient();
  Stream srvStream;

  //Get our currently selected server and start it
  ServerInfo sInfo = (ServerInfo)cmbServers.SelectedItem;
  tcpclnt.Connect(sInfo.address, sInfo.port);

  //Display any messages we receive.
  srvStream = tcpclnt.GetStream();
  byte[] msg = new byte[1024];
  bool done = false;
  for (; !done; )
  {
    int amtRead;
    try {
      amtRead = srvStream.Read(msg, 0, msg.Length);
    } catch (IOException) {
      UpdateConsole("\r\nIOException; closing Server");
      amtRead = 0;
    }

    if (amtRead != 0)
    {
      //Echo to console
      String line = "MSG: [";
      for (int i=0; i<amtRead; i++)
      {
        byte b = msg[i];
        if (b >= 32 && b <= 126)
          line += (char)b;
        else
          line += '.';
      }

      line += "]";
      UpdateConsole("\r\n"+line);
    }
    else
    {
      UpdateConsole("\r\nRead zero; closing Server");
      done = true;
    }
  }

  //Must close BOTH
  srvStream.Close();
  tcpclnt.Close();

  //Ready to go again?
  UpdateConsole("\r\nServer Stopped");
}

Make sure you add the following using directives to the top of your file:

using System.Threading;
using System.Net.Sockets;

After that, you’ll only get errors for the UpdateConsole function, which we’ll implement now:

private void UpdateConsole(String msg)
{
  Control ctl = txtConsole;
  if (ctl.InvokeRequired)  
    ctl.Invoke(new AddToConsoleDel(UpdateConsole), msg);
  else
    ctl.Text += msg;
}

Add the following to our namespace, and we’ll be done:

public delegate void AddToConsoleDel(String msg);

Run the program, click “Read” to get a server list, then choose the “Main game server” from your dropdown list and hit “Connect”. You should see a nascent echo of the regular log-in message, and some unreadable garbage.

The "Garbage" Is Actually the Important Information, As We'll See Later.

If this just doesn’t work, have a look at Listing 2 to see where you went wrong.

If you scan the code, you’ll find everything in order. First, we create a TCPClient object to manage our TCP connection to the server. We then Connect it to the server, based on the item chosen in the drop-down list. Finally, we continue to read up to 1kb at a time from the server. To the best of my knowledge, the Eternal Lands server never sends more than a few hundred bytes of data at once, so we don’t have to merge messages.

Note that this all has to happen in a different thread, to avoid throttling the GUI dispatch thread. If you just ran a tight loop in the main Thread, your Windows Form would become sluggish and unresponsive. The C#-ism:

  Thread t = new Thread(new ThreadStart(Function_Name));

…capitalizes on delegates to encapsulate a function (Function_Name) in a new thread. This is a reasonably good example of a delegate, but we have a much better one: our UpdateConsole function.

The problem with ThreadStart is that it’s too easy to guess what it does, without actually understanding why we choose to do it this way. Instead, let’s have a look at UpdateConsole. I’ve highlighted it differently below:

private void UpdateConsole(String msg)
{
Control ctl = txtConsole;
if (ctl.InvokeRequired)
ctl.Invoke(new AddToConsoleDel(UpdateConsole), msg);
else
ctl.Text += msg;
}

First, our variables are declared in red. One is the value we wish to update the console with (as a String) and the other is a local variable used to reference the Text Box itself. Since all Text Boxes subclass Control, we can store this variable in a generic fashion. Note that the variable ctl was created to emphasize that this technique can be used on any part of our form. We could have put ctl as a second function variable, if we had multiple consoles.

Next, the blue text. The “InvokeRequired” property (think of it like a read-only variable) is a nifty shorthand which tells us if the given control was created by the thread we are currently running. As all you GUI programmers know, most actions that update GUI components should be done in the master dispatch loop of the thread that created the GUI. If, for example, this control was created in a random thread which is now finished running, then updating it from another thread will cause all sorts of exceptions at run-time. We could dispatch a new thread for each update operation, but the InvokeRequired property helps us to avoid spawning threads when we don’t need to.

The purple text is run if an Invoke is Required. It calls the Invoke method of ctl, which adds a new event to a thread guaranteed to be valid for modifying ctl. This thread is of type AddToConsoleDel, which we defined earlier as:

delegate void AddToConsoleDel(String msg)

…in other words, as a delegated function with one argument that returns nothing (void). This matches the function signature of our UpdateConsole function. Look back at the purple text, and you’ll see that we call new AddToConsoleDel(UpdateConsole) to  create a new delegate of type AddToConsoleDel that will run the UpdateConsole function when called. The second parameter to Invoke is the argument set that we intend to pass to our AddToConsoleDel delegate. Since UpdateConsole only takes one argument, we only pass in one (msg).

Which leads us to the green text: what happens if we are running in the right thread. If this is the case, we simply append the message string to the Text Box’s existing Text property.

So, the logic of our function is simple. If the thread is capable of modifying the control directly, it just appends the text. Otherwise, it creates a new delegate to the same function, passes along the same method arguments, and asks the given control to Invoke this delegate. That ensures that next time the function is called (by the delegate this time) it has the necessary permission to modify the control. This logic is a bit tricky to catch the first time you see it, but it’s one of the main design patterns of C#. And, it has the benefit of keeping the actual logic of the method in the same place as the invoke logic of the delegate. So we can just call UpdateConsole and not have to care about whether a thread is spun off to do the dirty work.

You can think of a delegate as a pointer to a function, with enhanced type-safety, if that’s any easier for you.
Take a break, grab a coffee, you deserve a break.

Step 3: Passing Messages Along Properly

Our middleware can locate the Eternal Lands server, and it establishes a TCP connection (proven by the fact that it receives two messages in sequence, not just two copies of the same message). We also jumped the gun and added a loopback entry to the client’s list of servers, to allow us to force the client to connect to our program instead of the real server. Now, let’s put that new entry to work, and tie the client and server together so that you can play Eternal Lands with our program as a local proxy.

Open your project in Visual Studio, and right-click on the “Solution” (top-most item) in the “Solution Explorer”. Choose “Add”, then “New Project”, then select “Class Library” and name it Eternal Lands Connector. Press “Ok”. Now, right-click on your main project (“Windows Forms Application 1”, in my case) and choose “Project Dependencies”. In the “Depends on” box, check the “Eternal Lands Connector” project. Click “Ok”.

The purpose of this secondary project is to isolate our code. Once we’ve written the C# necessary to tunnel messages from the client to the server, we’ll then be adding a whole bunch of filtering and display code. We don’t want to accidentally break our message passing code, so we’ll keep in a totally separate project. You could manage this project separately, and manually add the DLL to the main Windows Forms project each time you update it, but Visual Studio’s dependency tracker allows you to do this automatically.

Let’s get started. First, rename Class1.cs to ELMiddleware.cs. We’ll need another file, though (Right-click “Eternal Lands Connector”, choose “Add”, then “New Item…”, then select “Class”) called SleepQueue.cs. This queue manages a list of messages, and stops processing them when the queue is empty. It sets itself to sleep until a new message is received. This signaling approach is far more efficient than constantly checking if the queue has items (spinning). Here’s the code for this class:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Net;
using System.Net.Sockets;
using System.IO;

namespace Eternal_Lands_Connector
{
  public abstract class SleepQueue : IDisposable
  {
    //For threading: allow sleeping when no messages are received, by using an EventWaitHandle
    private Thread t;
    private EventWaitHandle wHandle = new AutoResetEvent(false);
    private Queue<PacketData> waitingMessages = new Queue<PacketData>();

    //Add a packet to the list of waiting messages. 
    // Wake the main thread.
    public virtual void AddPacket(PacketData p)
    {
      lock (waitingMessages)
        waitingMessages.Enqueue(p);
      wHandle.Set();
    }

    //What priority should the message queue thread be? 
    //  Sub-classes can override this to balance performance.
    public virtual ThreadPriority getInitialPriority()
    {
      return ThreadPriority.Normal;
    }

    //Dispose properly
    public void Dispose()
    {
      // Signal the consumer to exit.
      AddPacket(null);
     if (t != null)
        t.Join();
     if (wHandle != null)
        wHandle.Close();
    }

    //Start the queueing process.
    public void startQueueing()
    {
      t = new Thread(new ThreadStart(Handle_Queue));
      t.IsBackground = true;
      t.Priority = getInitialPriority();
      t.Start();
    }

    private void Handle_Queue()
    {
      //Do pre-inits
      Pre_Init_Activity();

      //Listen for queue updates
      PacketData msg = null;
      for (; this.isValid(); )
      {
        //Get the next task
        msg = null;
        lock(waitingMessages)
         {
          if (waitingMessages.Count > 0)
           {
             msg = waitingMessages.Dequeue();
           if (msg == null)
             {
             //Done.
             this.On_Message(msg);
             return;
             }
           }
         }

        //A bit confusing to have two null checks, but it makes sense.
        if (msg == null)
         {
          //No more tasks - wait for a signal
              wHandle.WaitOne();
         } else {
            this.On_Message(msg);
         }
       }

      this.On_Thread_Close();
    }

    //
    //Un-implemented functionality
    //

    //When does this queue exit?
    protected abstract bool isValid();

    //What should we do with each message?
    protected abstract void On_Message(PacketData msg);

    //What should happen when the thread closes? 
    protected abstract void On_Thread_Close();

    //What should happen before the message loop begins? (After the thread is started)
    protected abstract void Pre_Init_Activity();
  }
}

This depends on the PacketData.cs file, which is simply a wrapper on a byte array coupled with an id character:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace Eternal_Lands_Connector
{
  public class PacketData
  {
    public PacketData(byte[] data, char from)
    {
      this.data = data;
      this.from = from;
    }

    public byte[] data;
    public char from;
  }
}

The technical details of the Sleep Queue are not important, so long as you realize that:

1. Every message posted to the queue gets processed.
2. You can post null to stop processing and end the thread.
3. The thread takes up no CPU time when there are no messages waiting.
4. Sub-classes will have to tell the queue when to stop processing (naturally), what to do with messages that are received, and what to do directly before entering and after leaving the message loop.

We now have everything we need to implement the ELMiddleware class, whose purpose is to handle all messages between the client and the server.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
using System.Net;
using System.Net.Sockets;
using System.IO;

namespace Eternal_Lands_Connector
{
    //Global
    public delegate void AddPacketDel(PacketData msg);
    public delegate void ShowConnectMsgDel(String msg);
    public delegate void AddToConsoleDel(String msg);
    public delegate void IncrPacketCountDel(int amt);

    public struct ServerInfo
    {
        public String id;
        public String configDir;
        public String address;
        public int port;
        public String description;
        public override String ToString()
        {
            return description;
        }
    }

    public class ELMiddleware : SleepQueue
    {
        //Hooks for controlling the client and server.
        private ELClientHook client;
        private ELServerHook server;

        //Saved
        private ServerInfo currServer;
        private ServerInfo loopback;
        private String elClientDir;
        private ShowConnectMsgDel ShowConnect;
        private AddToConsoleDel UpdateConsole;
        private IncrPacketCountDel IncrPacketCount;
        private int hookThreadID;

        public override void AddPacket(PacketData p)
        {
            //Log
            IncrPacketCount(1);

            //Count
            base.AddPacket(p);
        }

        private bool started = false;
        public bool Started {
            get {
                return started;
            }
        }

        public ELMiddleware(ELClientHook client, ELServerHook server)
        {
            //Save
            this.client = client;
            this.server = server;

            //Hook
            this.client.PacketHook = new AddPacketDel(AddPacket);
            this.server.PacketHook = new AddPacketDel(AddPacket);
        }

        public void startAll(ServerInfo currServer, ServerInfo loopback, String elClientDir, ShowConnectMsgDel ShowConnect, AddToConsoleDel UpdateConsole, IncrPacketCountDel IncrPacketCount, int hookThreadID)
        {
            //Don't start twice
            if (started || client.Started || server.Started)
                return;
            started = true;

            //Save
            this.currServer = currServer;
            this.loopback = loopback;
            this.elClientDir = elClientDir;
            this.ShowConnect = ShowConnect;
            this.UpdateConsole = UpdateConsole;
            this.IncrPacketCount = IncrPacketCount;
            this.hookThreadID = hookThreadID;

            //Start the sleep queues
            base.startQueueing();
        }

        protected override void Pre_Init_Activity()
        {
            //Init Log
            ShowConnect("EL Notifier Init");

            //Start the server
            server.hookServer(currServer, UpdateConsole);

            //Start the client
            client.hookClient(loopback, elClientDir, UpdateConsole, hookThreadID);

            //Spin-wait for both
            while (!server.ReadyForData || !client.ReadyForData)
                Thread.Sleep(5);
            UpdateConsole("\r\nSpin-waiting done");
        }

       protected override void On_Message(PacketData msg)
        {
            //Count
            IncrPacketCount(-1);
            if (msg == null)
                return;

            //Forward this message to the appropriate third party
            if (msg.from == 'C')
            {
                //Send from the client to the server.
                server.sendData(msg.data);
            }
            else if (msg.from == 'S')
            {
                //Send from the server to the client
                client.sendData(msg.data);
            }
            else
                UpdateConsole("\r\nBad message code: " + msg.from);
        }

        protected override bool isValid()
        {
            return client.Started && server.Started;
        }

        protected override void On_Thread_Close()
        {
            //Make sure everything closed nicely
            if (!server.Done)
                server.Done = true;
            if (!client.Done)
                client.Done = true;

            //Done
            started = false;
        }
    }
}

Add two more classes to the project, called ELClientHook.cs and ELServerHook.cs, respectively. Mark each class public, but don’t add any implementation yet.

Let’s review the code for ELMiddleware. At the top, we have some global delegate definitions, presumably used for logging. We also have our ServerInfo struct —if you think this means we’ll be removing a lot of code from Form1.cs then you guessed right. Our ELMiddleware class extends the SleepQueue abstract class. It contains several delegate references, a mysterious hookThreadID, and a reference to our client and server hooks.

Most of the code after this is obvious, and should give you an idea what kind of functionality we’ll need in ELClientHook and ELServerHook. The onMessage() function simply checks the sender of the message, and then ferries the message across to the recipient. We will later add the ability to react to ‘S’ class messages.

Our client and server hooks will need to be able to “take” messages from the middleware, in addition to other miscellaneous features. First, let’s add ELServerHook. Its functionality will be very similar to the previous chapter’s exercises.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.IO;
using System.Net.Sockets;
using System.Threading;

namespace Eternal_Lands_Connector
{
    public class ELServerHook
    {
        private TcpClient tcpclnt;
        private Stream srvStream;
        private ServerInfo srv;

        //Callbacks
        private AddToConsoleDel UpdateConsole;

        private bool started = false;
        public bool Started {
            get {
                return started;
            }
        }

        private bool done;
        public bool Done {
            get {
                return done;
            }
            set {
                done = value;
            }
        }

        private bool readyForData = false;
        public bool ReadyForData {
            get {
                return readyForData;
            }
        }

        private AddPacketDel packetDel;
        public AddPacketDel PacketHook {
            get {
               return this.packetDel;
            }
            set {
                this.packetDel = value;
            }
        }

        public void hookServer(ServerInfo srv, AddToConsoleDel UpdateConsole)
        {
            //Don't start twice
            if (started)
                return;
            started = true;

            //Save
            this.srv = srv;
            this.UpdateConsole = UpdateConsole;

            //Start
            tcpclnt = new TcpClient();
            UpdateConsole("Connecting.....");

            Thread t = new Thread(new ThreadStart(Check_Server));
            t.IsBackground = true;
            t.Start();
        }

        public void sendData(byte[] data)
        {
            if (!readyForData)
                throw new Exception("Server is not ready for messsages");

            //Send the dat down this stream. Technically, this occurs in the middleware's thread,
            //   so it won't conflict with any data we're reading at the same time.
            try {
                srvStream.Write(data, 0, data.Length);
            } catch (Exception ex) {
                UpdateConsole("\r\nException writing to server: \r\n   " + ex.ToString());
            }
        }

        private void Check_Server()
        {
            //Connect to the server's TCP port.
            tcpclnt.Connect(srv.address, srv.port);
            readyForData = true;
            UpdateConsole("Connected");

            //Try reading a bit
            srvStream = tcpclnt.GetStream();
            byte[] msg = new byte[1024];
            done = false;
            for (; !done; )
            {
                int amtRead;
                try {
                    amtRead = srvStream.Read(msg, 0, msg.Length);
                } catch (IOException) {
                    UpdateConsole("\r\nIOException; closing Server");
                    amtRead = 0;
                }

                if (amtRead != 0)
                {
                    //Make a new packet, add it to the middleware's queue
                    byte[] pk = new byte[amtRead];
                    Array.Copy(msg, pk, amtRead);
                    PacketHook(new PacketData(pk, 'S'));
                }
                else
                {
                    UpdateConsole("\r\nRead zero; closing Server");
                    done = true;
                }
            }

           //Must close BOTH
            srvStream.Close();
            tcpclnt.Close();

            //Ready to go again?
            UpdateConsole("\r\nServer Stopped");
            Console.WriteLine("Server hook stopped");
            started = false;
        }
    }
}

The implementation is almost exactly what we had before. The call to srvStream.Read() is blocking, by the way —no Sleep Queue is necessary.

The ELClientHook implementation is much more elusive:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.IO;
using System.Net.Sockets;
using System.Text;
using System.Text.RegularExpressions;
using System.Diagnostics;
using System.Threading;
using System.Windows.Forms;
using System.Runtime.InteropServices;

namespace Eternal_Lands_Connector
{
    //Just a helper class; no threads.
    public class ELClientHook
    {
        //Const
        public const string SERVER_ADDR = "127.0.0.1";
        public const int SERVER_PORT = 5421;

        private ServerInfo loopback;
        private String elClientDir;
        private TcpListener listener;
        private Stream clientStr;

        private Form mainForm;
        private int hookThreadID;
        private bool elDone = false;

        private bool started = false;
        public bool Started {
            get {
                return started;
            }
        }

        private bool done;
        public bool Done {
            get {
                return done;
            }
            set {
                done = value;
            }
        }

        private bool readyForData = false;
        public bool ReadyForData {
            get {
                return readyForData;
            }
        }

        private AddPacketDel packetDel;
        public AddPacketDel PacketHook {
            get {
                return this.packetDel;
            }
            set {
                this.packetDel = value;
            }
        }

        //Callbacks
        private AddToConsoleDel UpdateConsole;

        public void hookClient(ServerInfo loopback, String elClientDir, AddToConsoleDel UpdateConsole, int hookThreadID)
        {
            //Don't start twice
            if (started)
                return;
            started = true;

            //Save
            this.loopback = loopback;
            this.UpdateConsole = UpdateConsole;
            this.elClientDir = elClientDir;
            this.hookThreadID = hookThreadID;

            //Start background client
            Thread t = new Thread(new ThreadStart(Start_Client));
            t.IsBackground = true;
             t.Start();
        }

        public void sendData(byte[] data)
        {
            if (!readyForData)
                throw new Exception("Server is not ready for messsages");

            //Send the dat down this stream. Technically, this occurs in the middleware's thread,
            //   so it won't conflict with any data we're reading at the time.
            try {
                clientStr.Write(data, 0, data.Length);

                //Logging code - uncomment for development
                /*StringBuilder sb = new StringBuilder();
                sb.Append("\r\nClient Write: [");
                foreach (byte b in data)
                    sb.Append(Convert.ToString(b, 16).PadLeft(2, '0') + ":");
                sb.Append("]");
               UpdateConsole(sb.ToString());*/
            } catch (Exception ex) {
                UpdateConsole("\r\nException writing to client: \r\n   " + ex.ToString());
            }
        }

        private void EL_Stopped(Object sender, EventArgs e)
        {
            elDone = true;
        }

        //Get our handle (no idea why this requires an invoke... maybe it creates it?)
        private delegate IntPtr GetFormHandleDel(Form f);
        private IntPtr GetFormHandle(Form f)
        {
            if (f.InvokeRequired)
                return (IntPtr)f.Invoke(new GetFormHandleDel(GetFormHandle), f);
            else
                return f.Handle;
        }

        private bool findMainForm(int[] allowedProcessIDs, String matcherRegex, int hookThreadID)
        {
            return true;
        }

        private void EL_Window_Activate()
        {
            Console.WriteLine("EL Client Window Activated");
        }

        private void EL_Window_Deactivate()
        {
            Console.WriteLine("EL Client Window Deactivated");
        }

        private void Start_Client()
        {
            //Start the socket
            listener = new TcpListener(IPAddress.Parse(SERVER_ADDR), SERVER_PORT);
            listener.Start();
           UpdateConsole("\r\nLoopback server running at: " + listener.LocalEndpoint);

            //Start the EL client program with the server specified as "loopback"
            UpdateConsole("\r\nStarting EL Client...");
            ProcessStartInfo ps = new ProcessStartInfo();
            ps.FileName = elClientDir + "\\el.exe";
            ps.Arguments = loopback.id;
            ps.WorkingDirectory = elClientDir;
            ps.ErrorDialog = true;
            Process p = Process.Start(ps);
            elDone = false;
            p.Exited += new EventHandler(EL_Stopped);

            //Get the full-screen window for this application
            mainForm = null;
            while (mainForm == null)
            {
                int[] validIDs = new int[p.Threads.Count];
                {
                    int id = 0;
                    foreach (ProcessThread t in p.Threads)
                        validIDs[id++] = t.Id;
                }

                if (findMainForm(validIDs, "Eternal Lands", hookThreadID))
                    break;
                Thread.Sleep(5);
            }
            UpdateConsole("\r\nClient spin-waiting done");

            //Wait for the client to try to connect to us
            TcpClient socket = listener.AcceptTcpClient();
            clientStr = socket.GetStream();
            readyForData = true;
            UpdateConsole("\r\nEL Client tried to connect to us: " + socket.Client.RemoteEndPoint);

            //Continually read from the client. This is the main client update thread.
            byte[] msg = new byte[1024];
            done = false;
            for (; !done; )
            {
                //Is the client still running?
                if (elDone)
                {
                    UpdateConsole("\r\nGame closed; closing Client");
                    done = true;
                }
                else
                {
                    int amtRead;
                    try
                    {
                       amtRead = clientStr.Read(msg, 0, msg.Length);
                    }
                    catch (IOException)
                    {
                        UpdateConsole("\r\nIOException; closing Client");
                        amtRead = 0;
                    }
                    if (amtRead != 0)
                    {
                       //Make a new packet, add it to the middleware's queue
                        byte[] pk = new byte[amtRead];
                        Array.Copy(msg, pk, amtRead);
                        PacketHook(new PacketData(pk, 'C'));
                    }
                    else
                    {
                        //We're done
                        UpdateConsole("\r\nRead zero; closing Client");
                        done = true;
                    }
                }
            }

            //Have to close all three
            clientStr.Close();
            socket.Close();
            listener.Stop();

            //Ready to go again?
            UpdateConsole("\r\nClient Stopped");
            Console.WriteLine("Client hook stopped");
            started = false;
        }
    }
}

You might get an error with the System.Windows.Forms using directive; simply right-click on the “References” section of your Connector’s Project file and choose “Add Reference”. Then, in the “.NET” tab, browse down to “System.Windows.Forms” and click “Ok”. By default, a library project does not have access to the Forms dll, for reasons that remain a mystery to me.

A quick browse through this code reveals that our fake middleware server will be situated at 127.0.0.1, on port 5421. The sendData() function is similar to the server’s copy; however, I’ve included some logging code for your convenience. If you uncomment this, it will print a properly-formatted hexadecimal representation of each message sent from the server to the client. This will allow you to manually debug unknown or tricky messages, and to determine how you should react to them.

The findMainForm() function is currently empty; it will later be used to determine which window is actually hosting the EL client. The only other function of interest is the StartClient() function. Note that we use a ProcessStartInfo object to handle starting a process:

1.We have to set the working directory to the Eternal Lands folder, otherwise the process won’t be able to find its resources (images, sounds, etc.).
2.We start the process el.exe. This, of course, is Windows-specific.
3.We start this process with the argument loopback. In other words, we call “el.exe loopback” from the command line, allowing us to choose which server to connect to.

The Thread-finding code which follows this is messy; just ignore it. Basically, our middleware program will have multiple running threads, and we need to figure out which one is hosting the EL Client’s window. That way, we can detect when it has focus, and avoid showing messages in that case.

Note that, like the server, the client continually reads from the middleware until no more data is sent. You’ll have to change Form1’s code to actually use this new middleware, of course.

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using Eternal_Lands_Connector;
using System.Text.RegularExpressions;
using System.IO;
using System.Threading;
using System.Net.Sockets;

namespace WindowsFormsApplication1
{
    public partial class Form1 : Form
    {
        //Count leftover packets. A good first-warning if something goes wrong.
        private int packetCount;

        //Our hooks
        private ELServerHook serverHook;
        private ELClientHook clientHook;
        private ELMiddleware middleWare;

        //Also...
        private ServerInfo loopbackInfo;
        private string srvLine = "loopback        main            " + ELClientHook.SERVER_ADDR + "               " + ELClientHook.SERVER_PORT + "    Loopback adapter for use with the EL Notifier";

        public Form1()
        {
            InitializeComponent();

            //Initialize connectors.
            serverHook = new ELServerHook();
            clientHook = new ELClientHook();
            middleWare = new ELMiddleware(clientHook, serverHook);
        }

        private void ReadELFiles(object sender, EventArgs e)
        {
            //Reload an entirely new list of servers.
            cmbServers.Items.Clear();

            //Don’t load anything if we specified an invalid directory.
            if (!File.Exists(txtPathToELFolder.Text + "\\" + "el.exe"))
            {
                txtConsole.Text = "Error: el.exe doesn't exist in EL folder";
                return;
            }
            else if (!File.Exists(txtPathToELFolder.Text + "\\" + "servers.lst"))
            {
                txtConsole.Text = "Error: servers.lst doesn't exist in EL folder";
                return;
            }

            //Open our servers file and read it.
            bool foundLoopback = false;
            StreamReader srvFile = new StreamReader(txtPathToELFolder.Text + "\\" + "servers.lst");
            while (!srvFile.EndOfStream)
            {
                String line = srvFile.ReadLine();
                if (line == null)
                    break;

                //Coment? Empty?
                line = line.Trim();
                if (line.StartsWith("#") || line.Length == 0)
                    continue;

                //Read:
                //ID, Config_Dir, Address, Port, Description
                String regex = "([^ \t]+)[ \t]+([^ \t]*)[ \t]+([^ \t]*)[ \t]+([^ \t]*)[ \t]+(.*)";
                Match m = Regex.Match(line, regex);
                ServerInfo sInf = new ServerInfo();
                if (m.Success && m.Groups.Count == 6)
                {
                    sInf.id = m.Groups[1].Value;
                    if (sInf.id.Equals("loopback"))
                        foundLoopback = true;
                    sInf.configDir = m.Groups[2].Value;
                    sInf.address = m.Groups[3].Value;
                    try
                    {
                        sInf.port = Int32.Parse(m.Groups[4].Value);
                    }
                    catch (FormatException)
                    {
                        //Don't add
                        continue;
                    }
                    sInf.description = m.Groups[5].Value;
                    cmbServers.Items.Add(sInf);

                    //Loopback?
                    if (sInf.id.Equals("loopback"))
                        loopbackInfo = sInf;
                }
            }
            srvFile.Close();

            //Do we need to add our own local server?
            if (!foundLoopback)
            {
                //Copy existing server file
                StreamReader inFile = new StreamReader(txtPathToELFolder.Text + "\\" + "servers.lst");
                StreamWriter outFile = new StreamWriter(txtPathToELFolder.Text + "\\" + "servers.lst.new");
                for (String line = inFile.ReadLine(); line != null; line = inFile.ReadLine())
                    outFile.WriteLine(line);
                inFile.Close();

                //Add our own category
                outFile.WriteLine(srvLine);

                //Close, swap, notify.
                outFile.Close();
                File.Delete(txtPathToELFolder.Text + "\\" + "servers.lst");
                File.Move(txtPathToELFolder.Text + "\\" + "servers.lst.new", txtPathToELFolder.Text + "\\" + "servers.lst");
                ShowConnectMsg("Created loopback entry.");

                //Add a new item to the list
                loopbackInfo = new ServerInfo();
                loopbackInfo.id = "loopback";
                loopbackInfo.configDir = "main";
                loopbackInfo.address = ELClientHook.SERVER_ADDR;
                loopbackInfo.port = ELClientHook.SERVER_PORT;
                cmbServers.Items.Add(loopbackInfo);
            }

            if (cmbServers.Items.Count > 0)
                cmbServers.SelectedIndex = 0;

            //Enable the remainder of our form
            cmbServers.Enabled = true;
            btnConnect.Enabled = true;
        }

        private void ConnectToServer(object sender, EventArgs e)
        {
            if (middleWare.Started || clientHook.Started || serverHook.Started)
                return;

            //Count
            packetCount = 0;

            //Get our currently-selected server and start it.
            ServerInfo srv = (ServerInfo)cmbServers.SelectedItem;
            middleWare.startAll(srv, loopbackInfo, txtPathToELFolder.Text, new ShowConnectMsgDel(ShowConnectMsg), new AddToConsoleDel(UpdateConsole), new IncrPacketCountDel(IncrPacketCount), Thread.CurrentThread.ManagedThreadId);
        }

        //Delegates and implementation details
        private void ShowConnectMsg(String msg)
        {
            Control ctl = txtConsole;
            if (ctl.InvokeRequired)
                ctl.Invoke(new ShowConnectMsgDel(ShowConnectMsg), msg);
            else
                ctl.Text = msg;
        }

        private void UpdateConsole(String msg)
        {
            Control ctl = txtConsole;
            if (ctl.InvokeRequired)
                ctl.Invoke(new AddToConsoleDel(UpdateConsole), msg);
            else
                ctl.Text += msg;
        }

        private void IncrPacketCount(int amt)
        {
            packetCount += amt;
            SetText(lblZero, packetCount + "");
        }

        private delegate void SetTextDel(Control ctl, String msg);
        private void SetText(Control ctl, String msg)
        {
            if (ctl.InvokeRequired)
                ctl.Invoke(new SetTextDel(SetText), ctl, msg);
            else
                ctl.Text = msg;
        }
    }
}

Right-click on the References section in your main Forms project and choose “Add Reference”. Then, in the “Projects” tab, choose “Eternal Lands Connector” and hit “Ok”. This will allow you to use our class library.

The source code is not too complex, and the effect is easy enough to describe: we simply intercept and then deliver all messages from the client to the server, and from the server to the client. At the moment, this is essentially a whole lot of work for nothing. However, now that all messages are passing through our program (our source code, you might say), you can easily  tag messages of a specific type, and prompt the user after further processing them. So, this was all “legwork”.

Our middleware in action, connecting to the PK server.

Congratulations, we have now bamboozled both end points. In the next section, we’ll consider how to filter messages without requiring a re-compile of the Connector each time we change our minds about which messages to filter. We’ll also get you started processing Eternal Lands’s images to brighten up our notifier.

Step 4: Filtering Messages (Outside the Core)

At this point, you should have a pretty good idea how to intercept messages of a certain type. Before you dash too far into it, though, may I remind you that one of the reasons we used C# in the first place was to avoid bugs that corrupted our communication layer. We should add the following to the communication core:

1. Allow our main Forms class to “register” any number of listeners with the core.
2. Before sending a message (to the client), check if it matches any of the listeners’ patterns. If it does, send a copy of the message to the matched listener.
3. Ensure that the core runs in a higher-priority thread than the listeners’ implementations. According to the rules of Windows thread priorities, this ensures that the listeners cannot starve the core for cycles.

This way, no amount of chicanery by the listeners can cause the core to malfunction. (And remember, malfunctions are BAD, since we’re not the ones paying for the EL servers.)

Open up ELMiddleware.cs, and add the following member variable to the class:

//An additional matcher

private MessageMatcher msgMatch;

We’ll define MessageMatcher later; for now, scroll down to the definition of startAll, and change it to read:

public void startAll(ServerInfo currServer, MessageMatcher msgMatch, ServerInfo loopback, String elClientDir, ShowConnectMsgDel ShowConnect, AddToConsoleDel UpdateConsole, IncrPacketCountDel IncrPacketCount, int hookThreadID)

Next, somewhere in the body of startAll, save the MessageMatcher that we passed in to the function:

this.msgMatch = msgMatch;

Make sure you start it too; after “base.startQueueing()” add:

msgMatch.startQueueing();

And finally, scroll down until you find the line:

client.sendData(msg.data);

…and add the following underneath it:

//Send the data to an additional thread for logging, parsing, etc.

msgMatch.AddPacket(msg);

That code should speak for itself; our MessageMatcher class will centralize message processing, and the AddPacket method is called for each server message. We do not filter client messages, because it seems that these originate mostly from the user’s actions (which are always known by the user –and must be confirmed by the server, regardless.)

Here, then, is the definition of the MessageMatcher class. Add it to your project (to the Eternal_Lands_Connector project, that is).

using System;

using System.Collections.Generic;

using System.Linq;

using System.Text;

using System.Threading;

using System.Net;

using System.Net.Sockets;

using System.IO;

namespace Eternal_Lands_Connector

{

public class MessageMatcher : SleepQueue

{

/// <summary>

/// The core of our message matcher

/// </summary>

private Dictionary<Int32, List<MessageHandler>> handlers = new Dictionary<Int32, List<MessageHandler>>();

public void addHandler(MessageHandler h)

{

if (!handlers.ContainsKey(h.getProtocol()))

handlers[h.getProtocol()] = new List<MessageHandler>();

handlers[h.getProtocol()].Add(h);

Console.WriteLine(“Dictionary size: “ + handlers.Count + ”   entries[" + h.getProtocol() + "] –> “ + handlers[h.getProtocol()].Count);

}

public AddToConsoleDel UpdateConsole;

protected override void On_Message(PacketData msg)

{

if (msg == null)

return;

//Now, segment into multiple messages

for (int currID = 0; currID < msg.data.Length; )

{

int protocol = msg.data[currID++];

int size = msg.data[currID++] | (msg.data[currID++] << 8);

//Make a new data array

int count = size – 1;

byte[] b = new byte[count];

Array.Copy(msg.data, currID, b, 0, count);

currID += count;

//Match the first relevant handler

if (handlers.ContainsKey(protocol))

{

foreach (MessageHandler h in handlers[protocol])

{

if (h.performMatch(protocol, b, new Dictionary<string, object>()))

{

break;

}

}

}

}

}

/// <summary>

/// Somwhat less-interesting features

/// </summary>

private bool valid = true;

public bool IsValid

{

get

{

return valid;

}

set

{

valid = value;

}

}

//We make this its own class so we can change its priority.

public override System.Threading.ThreadPriority getInitialPriority()