Emulating Classic Games in HD – Part 2

Last time, we learned the basics of the Game Boy hardware, and tried a simple strategy for rendering games with high-res graphics. Sometimes this simple approach worked well:

but other times it didn’t work at all:

In this part we’ll look at the timing of the Game Boy display hardware, and see how this interferes with our simple scheme.

Horizontal Blank

Computing the appropriate color for a pixel, given the background and sprites, takes time. As such, the graphics hardware does not compute colors for the entire screen simultaneously, but instead render pixels one at a time via raster scanning.

If we imagine how an old-fashioned CRT display scans out an image, there is an “electron gun” that scans each row of pixels from left to right, and proceeds down the screen through the rows. While the GameBoy uses an LCD screen rather than a CRT, it employs a similar notion of how the screen is scanned. Pixels are scanned from left to right in each horizontal line, and lines are scanned from the top down.

After the each line has been scanned, it takes time for the “electron gun” to move back to the start of the next line; this time is called the horizontal blank. At the end of the last line, it takes an even longer time to scan back to the start of the first line; this time is called the vertical blank.

During the two blanking periods, no rendering takes place. Because the rendering hardware is idle during this time, it is possible for software to safely change the values of hardware registers and affect how the next line will be drawn. For example, in Super Mario Land, the level scrolls from right to left as the player progresses, but the status/score display does not. In order to achieve this effect, the game changes the horizontal background-scroll register at the end of line 15.

A Less Simple Renderer

Now it should be clear why our simple rendering approach has problems: it only looks at the values of the hardware registers at the end of a frame (vertical blank), but those registers might have changed on each and every scan line (horizontal blank). Once we realize this, though, it is really quite simple to work around

First, we need to “snapshot” the state of the graphics registers at the end of each scan line. A clever implementation might only take a new snapshot when one or more registers actually change, or might store compact deltas, but my implementation takes a pretty naive snapshot at the end of each line.

Next, instead of rendering a textured quad for every tile, we can simply render a one-scan-line tall textured quad for every tile for every scan-line. On each line we render the tile using the state snapshot for that line.

This approach is somewhat wasteful of CPU and GPU time, but it rectifies the problems we saw before:

Try It Out!

I’ve gone ahead and thrown the code (such as it is) up on GitHub, just to prove I have nothing up my sleeves. You can also download a pre-built binary of the emulator for Mac (I haven’t had time to work on a Windows build yet, sorry).

Keep in mind that this is only a proof of concept; the controls are hard-wired, there is no support for save states or save slots, and it only supports a handful of Game Boy games.

Next Steps

My goal here was to show that this kind of graphics replacement was possible in theory, and I hope I’ve achieved that. I don’t have the kind of time it would take to turn this project into a real, usable emulator, so if you want to fork the code and run with it, feel free.

Scaling this approach up to a more powerful console would bring some interesting challenges, so I’ll throw out a few notes for anybody who might be interested in trying. Readers should feel free to bail out now if none of this makes sense:

  • I dealt with palettes in the easiest possible fashion: I use a 4-channel image to store weights for each of the colors in a 4-color palette (effectively treating the palette as a basis). For 16-color palettes this would involve 16-channel images, which is still pretty tractable. Beyond that, you’ll probably need a new technique.
  • Because the GameBoy is pure grayscale, it was easy to “project” arbitrary grayscale images into this basis by expressing them as a weighted sum of colors from the palette. For more complex palettes this runs the risk of being over- or under-constrained, so you’d likely want a tool to directly edit replacement images in a paletted fashion.
  • More accurate emulation is often required for more interesting consoles. In order to handle mid-scan-line effects on things like the NES, you might need to draw the replacement graphics on a per-pixel basis, rather than the per-line approach I use. In that case, it would help to detect when the graphics registers change, and only do the more expensive rendering when/where required.
  • I’ve ignored relative sprite priorities in my emulator, but I know that subtle priority details often have big effects (e.g., the clipping for item blocks in Mario 3). The key to getting priority right while still using replacement graphics is to cast the pixel-compositing operations usually applied into something like the Porter-Duff algebra, which will naturally scale to images and alpha transparency.
  • Many SNES games use HDMA to make rounded shapes (e.g., the Power Bomb in Super Metroid) or wavy distortion effects (e.g., Phantoon). If these kind of scanline effects get scaled up, they will become blocky. Ideally, an emulator should be able to interpolate between the state of registers at different scanlines to smooth out these shapes. The challenge then is figuring out when not to interpolate.
Advertisements

Emulating Classic 2D Games in HD – Part 1

Many emulators for 3D consoles support rendering games at higher resolutions than supported by the original hardware. Some emulators for older 3D consoles even support replacing the texture maps used in games with higher-resolution versions. Emulators for 2D consoles, such as the venerable Super NES, have no such functionality. Frames are rendered at the original native resolution, and then may be upsampled to higher resolutions using a handful of simple filters.

Why can’t we play classic 2D games in HD? Couldn’t we take the same concept of replacing low-resolution texture maps and apply it to inserting high-resolution sprites or background tiles into 2D games?

I’ve seen comments from emulator creators that this just isn’t possible, but I hadn’t really seen a solid technical explanation. So I decided to find out the answer for myself. Just as a little teaser, though, I’ll say that I was pleasantly surprised by what I could achieve (my own poorly-drawn programmer art aside):



Super Mario Land, now at 4 times the resolution!

The Game Boy

In order to understand both the opportunities and challenges, we need to cover some background on sprite-based 2D graphics hardware. To keep things simple, both for my implementation and for this article, I’ve chosen to focus on the original Nintendo Game Boy.

The Game Boy has a 160×144 pixel LCD screen, and uses 4 grayscale colors (2-bit color). Rather than store the image to be displayed in a frame buffer, the hardware renders the image by compositing three kinds of elements: the background, sprites, and window. For simplicity, I’ll only talk about the first two.

Background Tile Map

The background is a 32×32 grid of tiles, where each tile is an 8×8 pixel image. Thus the background defines a conceptual 256×256 pixel image.. We can represent the background as a 32×32 array of tile indices (the tile map) along with an array of the corresponding tile images.

uint8_t tileMap[32][32];
TileImage tileImages[256];

Since the background is larger than the screen, two 8-bit hardware registers are used to scroll the background and select what part will be visible. As we scroll past the edge of the background, it repeats. A game can replace tile indices that are out of view to implement a continuously-scrolling map.

Sprites

The sprites are defined by a structure called the Object Attribute Map (OAM). The Game Boy supports 40 sprites, and each sprite has an X and Y position, flags for horizontal and vertical flip, and the index of the tile image to use:

struct Sprite {
    uint8_t x;
    uint8_t y;
    bool flipX;
    bool flipY;
    uint8_t tileIndex;
};
Sprite oam[40];

Tiles

Both the background and sprite use tiles to define their image data. Each tile is a 2-bit-per-pixel paletted image (yes, even though the Game Boy only supports 2-bit color, it uses paletted images). An 8×8 pixel tile takes up 8x8x2=128 bits (16 bytes). There are distinct color palettes for sprites and the background:

Color bgPalette[4];
Color spritePalette[4];

In the sprite palette, color index zero is special: it indicates a transparent pixel.

Take-Away

There are, of course, a lot more details to the Game Boy rendering hardware, and other 2D platforms will of course have different designs. Still, these basic concepts – tile maps, sprites, and paletted tile images – are shared by many different 2D consoles.

Tile Replacement

In order to render a Game Boy game at higher resolution, we must find a way to map the existing 8×8 tiles to high-res replacements. A simple approach would be to hash the contents of the tile image, and use the resulting hash as a key for a table of replacements. In the case of the Game Boy, tile images are only 16 bytes, so we can just use the tile data itself as our hash key. Of course we also need the replacement images; for now I’ll be using my own programmer art:

On the left we have an original sprite from Super Mario Land, and on the right we have my replacement. The use of paletted images complicates this process somewhat, but let’s ignore those details for right now.

A Simple Scheme

At first blush it seems like we could accomplish our goal with a relatively simple renderer. Whenever a frame needs to be drawn we can:

  • Iterate over the background tile map, and draw one textured quad per background tile.
  • Iterate over the sprites, and draw one textured quad per sprite.

We need a cache that maps tile images to texture maps, but that can easily be folded into our tile-replacement step.

Let’s try this scheme out on a few games (note, I’m only replacing a subset of tiles right now – even programmer art takes time to create):




Link's Awakening

Those results seem good (my terrible art skills aside). Suspiciously good, in fact, if I’m claiming that this problem merits a whole article. And yes, if we look a little harder we can find examples where the results of our simple renderer are not as satisfying:




Super Mario Land again, not looking as good




Castlevania II looks like a complete mess

What is going on in these images that makes them fail? In Part 2 of this series, we will discuss the timing of sprite hardware, and see how to resolve these graphical issues.