Jump to content

Package plugins and Quake files now supported


Josh

2,117 views

 Share

Package plugins are now supported in Ultra Engine 1.0.2. This allows the engine to handle other package formats besides just ZIP. In order to test this out, I created the Quake Loader plugin, which currently supports the following formats used in the original Quake game:

  • PAK
  • WAD
  • BSP (textures)
  • SPR
  • LMP

Why Quake? Well, the original Quake game is what got me into game development through modding, but Quake is also great for testing because it's so weird. This game was written before hardware accelerated graphics existed, so it has a lot of conventions that don't match modern 3D standards:

  • All animation is vertex morphing (no skeletal animation)
  • Texture coordinates are in pixels / integers instead of floating point values
  • Three different types of "packages" (PAK, WAD, BSP), the latter two of which can be stored inside the first (packages-in-packages)
  • Image files are stored seven different ways:
    • LMP image files
    • SPR sprite files
    • Color pallette ('@')
    • Mipmapped texture in WAD ('D')
    • Console picture ('E')
    • Status bar pictures ('B')
    • Mipmapped textures stored in BSP

Each of the image storage methods store data in slightly different ways. Sometimes the width and height are packed into the image data, sometimes they are found elsewhere in a separate structure, sometimes they are hard-coded, and sometimes you have to guess the image dimensions based on the data size. It's a mess, and it seems like this was made up on-the-fly by several people, but this was cutting-edge technology at the time and I don't think usability outside the specified game they were working on was a big concern.

There are some interesting challenges there, but this forms the basis of several dated but influential games. As time went by, the design became more and more sane. For example, Quake 3 just stores textures as TGA images in a ZIP file with the extension changed to PK3. So I figure if I can make the plugin system flexible enough to work with Quake then it will be able to handle anything.

Originally I tried implementing this using HLLib from Nem's Tools. This library supports Quake formats and more, but I had a lot of problems with it, and got better results when I just wrote my own PAK loading code. WAD files were an interesting challenge. There is no real Quake texture format, so loading the raw texture data from the package made no sense. The most interesting breakthrough in this process was how I handled WAD textures. I finally figured out I can make the WAD loader return a block of data that forms a DDS file when a texture is loaded, even through the texture name has no extension. This tricks the engine into thinking the WAD package contains a DDS file which can easily be loaded, when the reality is quite different. This introduces an important concept, that package plugins don't necessarily have to return the exact data they contain, but instead can return files in a ready-to-use format, and the game engine will never know the difference.

The resulting plugin allows you to easily extract and load textures from the Quake game files. This code will extract textures from a single WAD file.

#include "UltraEngine.h"

using namespace UltraEngine;

void main(int argc, const char* argv[])
{
    // Load Quake file loader plugin
    auto qplg = LoadPlugin("Plugins/QuakeLoader");

    // Load FreeImage texture plugin
    auto fiplg = LoadPlugin("Plugins/FITextureLoader");

    // WAD to download
    WString wadfile = "bigcastle";

    // Download a WAD package
    DownloadFile("https://www.quaketastic.com/files/texture_wads/" + wadfile + ".wad", wadfile + ".wad");

    // Load the package
    auto wad = LoadPackage(wadfile + ".wad");

    // Create a subdirectory to save images in
    CreateDir(wadfile);

    // Read all files in the package
    auto dir = wad->LoadDir("");
    for (auto file : dir)
    {
        // Load the image
        auto pm = LoadPixmap(file);

        // Save as a PNG file
        if (pm) pm->Save(wadfile + "/" + file + ".png");
    }

    OpenDir(wadfile);
}

All the files found in the WAD package are saved as PNG images. BSP files will work exactly the same way.

image.thumb.jpeg.15c29a116ee568dbaf4bd439fc586213.jpeg

This code will load a Quake package, extract all the textures from all the maps in the game, and save them to a folder on the desktop. This is done by detecting packages-in-packages (WAD and BSP), which return the file type '3', indicating that they can be treated both as a file and as a folder. Since none of these package formats use any compression, pixmaps can be easily loaded straight out of the file without extracting the entire BSP. Since Quake PAK files don't use compression, the whole system just turns into a very complicated blob of data with pointers that store data all over the place:

#include "UltraEngine.h"

using namespace UltraEngine;

void main(int argc, const char* argv[])
{
    // Load Quake file loader plugin
    auto qplg = LoadPlugin("Plugins/QuakeLoader");

    // Load FreeImage texture plugin
    auto fiplg = LoadPlugin("Plugins/FITextureLoader");

    // Path to Quake game
    WString qpath = "C:/Program Files (x86)/Steam/steamapps/common/Quake";
    //WString qpath = "D:/SteamLibrary/steamapps/common/Quake";
  
    // Load game package
    auto pak = LoadPackage(qpath + "/id1/PAK1.PAK");

    //Change the current directory to load files with relative paths
    ChangeDir(qpath + "/id1");

    // Create a folder to save images in    
    WString savedir = GetPath(PATH_DESKTOP) + "/Quake";
    CreateDir(savedir);

    // Load the main package directory
    auto dir = pak->LoadDir("maps");
    for (auto file : dir)
    {
        // Print the file name
        Print(file);

        // Get the file type
        int type = FileType("maps/" + file);

        // If a package-in-a-package is found load its contents (BSP and WAD)
        if (type == 3)
        {
            auto subdir = pak->LoadDir("maps/" + file);
            for (int n = 0; n < subdir.size(); ++n)
            {
                // Load the texture (plugin will return DDS files)
                auto pm = LoadPixmap("maps/" + file + "/" + subdir[n]);

                // Save to PNG
                if (pm) pm->Save(savedir + "/" + subdir[n] + ".png");
            }
        }
    }
    OpenDir(savedir);
}

Here is the result:

image.thumb.jpeg.0b413ab419afe5946ace68c8ef4391d7.jpeg

Since all of these images can be loaded as pixmaps, does that mean they can also be loaded as a texture? I had to know the answer, so I tried this code:

#include "UltraEngine.h"

using namespace UltraEngine;

int main(int argc, const char* argv[])
{
    //Get the displays
    auto displays = GetDisplays();

    //Create a window
    auto window = CreateWindow("Ultra Engine", 0, 0, 1280, 720, displays[0], WINDOW_CENTER | WINDOW_TITLEBAR);

    //Create a world
    auto world = CreateWorld();

    //Create a framebuffer
    auto framebuffer = CreateFramebuffer(window);

    //Create a camera
    auto camera = CreateCamera(world);
    camera->SetClearColor(0.125);
    camera->SetPosition(0, 0, -2);
    camera->SetFov(70);

    //Create a light
    auto light = CreateBoxLight(world);
    light->SetRotation(45, 35, 0);
    light->SetRange(-10, 10);
    light->SetColor(2);

    //Create a model
    auto model = CreateBox(world);
    
    // Load Quake file loader plugin
    auto qplg = LoadPlugin("Plugins/QuakeLoader");

    // WAD to download
    WString wadfile = "bigcastle";

    // Download a WAD package
    DownloadFile("https://www.quaketastic.com/files/texture_wads/" + wadfile + ".wad", wadfile + ".wad");

    // Load the package
    auto wad = LoadPackage(wadfile + ".wad");

    // Load a texture from the WAD package
    auto tex = LoadTexture("sclock");

    // Create a material
    auto mtl = CreateMaterial();
    mtl->SetTexture(tex);

    // Apply the material to the box
    model->SetMaterial(mtl);

    //Main loop
    while (window->Closed() == false and window->KeyDown(KEY_ESCAPE) == false)
    {
        model->Turn(0, 1, 0);
        world->Update();
        world->Render(framebuffer);
    }
    return 0;
}

Here is the result, a texture loaded straight out of a Quake WAD file!

image.thumb.jpeg.a3ff0e25662f1bf5f9f612c9322f5b58.jpeg

To reiterate, testing with Quake files helped me to come up with two important design features:

  • Packages-in-packages, indicated with the file type '3'.
  • Packages do not necessarily have to return the same exact data they contain, and can instead prepare the data in a ready-to-use format, when that is desirable. The decision to base file type detection on the contents of the file instead of the file name extension worked well here, and allowed me to load the extension-less WAD texture names as DDS files.

Some of these images came from "bigcastle.wad". I don't know who the original author is.

If you are interested, you can read more about the weird world of Quake file formats here, although I must warn you that specification is not complete! :lol:

  • Like 4
 Share

10 Comments


Recommended Comments

Here we see the asset browser allows navigating through packages like folders, and even recognizes packages-in-packages (the BSP files). A texture is successfully loaded straight out of a BSP file without extracting it from the PAK. Thumbnails are not yet supported for the embedded files.

image.thumb.png.af786f7d8edd86b9aeacbadbf3ba9fdb.png

  • Like 1
Link to comment

And here you can see the zip package is displayed like a folder, but you can also see the zip file in the parent folder's contents, so you can still open it from there with the default program:

image.thumb.jpeg.ba49d9980c769338771a255d9fa051c6.jpeg

  • Like 1
Link to comment

Not yet. I have done earlier work loading Quake models, but did not load animations. It will also require support for vertex morphing.

  • Thanks 1
Link to comment

Here we can see the file time and size imported from the package. Labels for the file, folder, and package are displayed in the side panel.

image.thumb.jpeg.e96e9f50085941841710b1f4a5f0714d.jpeg

  • Like 1
Link to comment
Guest
Add a comment...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...