I put some time into getting caught up on maintenance tasks this month. They’re pretty boring to write about, so I’ll just be focusing on new things here. Besides that, the main addition this month is a routine scripting / macro system for actors.
Tiled Updates
Tiled 1.4 adds several new features. I upgraded from 1.3.1, and this version feels a lot nicer to use.
- Per-layer RGBA tint settings are neat, and are already supported on the game engine side, so I plugged that parameter in right away. I had some issues with the alpha parameter, so I’m leaving that alone, and using either the layer’s current opacity, or a custom ‘layer_alpha’ field property if it’s present.
- As with backgroundcolor, the tintcolor values are an HTML-style hex number in TMX format, and a table of numbers when exported to Lua. If alpha is 255, then that number is omitted from the table.
- Tiled’s world functionality is now expanded to a full editor. I tested it briefly and couldn’t quite get it working, but I’ll certainly be keeping an eye on this as anchoring maps together manually with plain function calls is really painful.
Behavior
Stage Geometry Search
Before the switch to multi-rooms, a scene context had one array containing general-purpose geometric objects (points, rectangles, etc.) which could be used for any arbitrary purpose, like identifying the destination point for a teleporter. Moving forward with multiple rooms, I had to decide how these shapes would work when rooms can appear and disappear. Unlike actors, I decided to keep shapes 100% tied to the rooms they’re defined in. They can never change room ownership, and if the room is deactivated, they should vanish.
This presents a couple of issues: shapes can exist outside of the room they belong to, and the shape-lists that an entity might want to search through will vary depending on the context. A typical opponent actor would likely only care about shapes that exist in its current owned room. On the other hand, when a scene is starting up, the action_manager actor needs to look for the starting checkpoint before any rooms are active at all.
So there are a few base functions for doing a linear search through a ‘geometry’ array, matching by type string, name string, or both. I made wrappers for these functions for the following situations:
- For an actor to search through shapes belonging to the room that currently ‘owns’ it from a resource management perspective. If there is no owning room or it has been deactivated, then the search always returns nil.
- For any given room.
- For all live, active rooms in a scene context.
- For all ‘model rooms’ in the scene’s list of sites.
There are many wrinkles to iron out here. Before multi-rooms, I made the assumption that lists of geometric shapes never change after the scene context is initialized, and that won’t hold anymore.
“Blackboards”
I started a module for attaching “blackboard” structures to a scene. This is in sarcasm quotes because nothing actually does anything with them yet. I’d like to put the player’s position and a direct link to its actor table there, so that other actors don’t need to search for the player actor. In any case, it’s off the todo list.
Callback functions for Rooms, Sites and Worlds
With multi-room scenes, I’m finding a need to implement “area” logic code at a finer level than just on a per-scene basis. For example, it would make sense to tie code that manages a custom background scrolling effect in one specific area to the room where it happens. That could be achieved by creating a special-purpose actor and placing it in the relevant room, but actors carry some risks: another actor could misbehave and remove() it, or I could accidentally delete the actor’s spawnpoint in Tiled without realizing what I had done until much later. (I seriously kept doing this at one point.)
I’ve added callback function support for every live room, site and world. Each of these entities can be assigned multiple scripts, and each script can fire callbacks when certain events occur:
CREATED ACTIVATED PER-TICK DEACTIVATED* DELETED* Room X X X X X Site X X X World** X X X * These fire in reverse of the order that they were defined. ** Worlds can't be turned on or off: switching worlds resets the whole scene context.
I’m not 100% decided on whether sites should have on_create and on_delete callbacks: these events would only happen when a world is loaded or unloaded, and sites don’t really do much right now other than serve as containers of rooms to be added or removed in batches from a scene without resetting state.
Routine Macro System
A debug visualization of a RoutineState structure.
This is the big one for the month. I mentioned in February that I’ve tried and failed to make some kind of scripting or state management system for actors. Since then, I’ve cleared a lot of things off the todo list, and this seemed like a good topic to return to. Due to scope creep, it became a much bigger job than initially planned.
Macro in this context means some way to direct the behavior of entities by stepping through a sequence of states over the span of multiple ticks. Up to this point, actors have had one main logic callback that runs every tick, and an array of ‘service’ callbacks that implement shared behavior, such as platformer physics and common opponent reactions. State management has been implicit, with a lot of ‘if data.state == “windup” then blahblah’ kinds of blocks, stuffed into either the main per-tick callback or one of the services. Explicit state structures would help in cases where you want a sequence of tasks to be executed over multiple ticks, each with their own timeout parameters, and possibly across multiple kinds of actors.
The entry for this in my todo list said “actor behavior trees“. I had difficulty visualizing how behavior trees would work in the current codebase, so I went for a more basic system where a state structure steps through linear sequences of states with timeout values and some flow control information.
Three modules implement the routine macros: the driver, which deals with run-time execution; the parser, which converts lines of text into a series of tables in the desired format during load-time; and a set of built-in commands which provide flow-control, ‘if’ statement support, counter variable assignment and other essentials.
Driver
I ended up going with the following set of structures, going roughly from bottom to top:
Command: A set of callback functions that are identified by a unique string
Statement: A table that pairs a command string with arguments
List: An array of statements
Label: A specific jump-point in a list, identified by a unique (to the list) string or integer
RoutineState: A structure that steps through a list, executing the current statement’s callbacks, and following flow control constructs.
Every actor now has an embedded RoutineState, inactive by default, and can be assigned a list that contains statements and flow control symbols which implement (or just indicate) the actor’s various states. RoutineStates can push new frames onto themselves in a stack fashion, then pop them off to get back to the previous list + position / state info. RoutineStates by themselves don’t assume anything about the entity they belong to, so they could be used in other things which tick besides actors.
Counters
Each RoutineState contains four general-purpose counters named ‘ra’ through ‘rd’. Each frame in a RoutineState also contains four counters, ‘fa’ through ‘fd’. Frames can mess with counter values belonging to the frame underneath it using the identifiers ‘pfa’ through ‘pfd’. Counters can be compared against or assigned the values of other counters which belong to the same RoutineState. Labels can be numbers, so that counter values can be used when jumping to a label.
Ported Variables
(I don’t know what to call these.) Ports are a set of getter / setter functions paired with a string identifier. They can be used as a way to read or write details external to the RoutineState. For example, you could set up a port to get/set the game’s current difficulty level.
Hierarchical Groups
Originally, commands, ports and lists were all stored in one table each, and addressed by string IDs. I split them all into groups, with an option to inherit fields from parent groups using Lua’s __index metamethod. Every RoutineState frame can independently select command groups, port groups and list groups at any time. I’m second-guessing this now, because it seems like there could be some really bad ramifications. It’s also possible that all the macro work required for this project may not need anything more than the default groups.
A confusing and now out-of-date diagram that I made while trying to figure this out.
Parser
Writing the routine tables by hand was getting annoying, so I started a parser to expand lines of text into tables in the desired format. Here is a short test list:
; Comments begin with a semicolon
* list_test ; List declaration
128 noop ; "Do nothing" for 128 ticks / 0.5 seconds
^ one ; (Test command -- just prints "one")
^ two ; (Test command)
.label ; Label declaration (defines jump-to points)
128 !norep three ; (Test command) !norep == run on_tick once
256 !norep set fa = 64 ; Set frame counter 'fa' to 64
^ !norep set rb = 64 ; Set RoutineState counter 'rb' to 64
pushList something ; Make new frame and point it to list 'something'.
; When that frame is eventually popped, this frame
; will continue execution on the following statement.
128 !norep four ; (Test command)
goStart ; Jump to the first statement in the list
There is some ZZT-OOP influence in this. I’ve written a lot of game scripting in ZZT, so for better or worse, I’m fairly comfortable with jumping around labels.
Here is the same list in table form that I was authoring by hand before:
routine.registerList( "list_test", {
{ name = "noop", time = 128, repeating = true, params = nil },
{ name = "one", time = 128, repeating = true, params = nil },
{ name = "two", time = 128, repeating = true, params = nil },
"label",
{ name = "three", time = 128, repeating = false, params = nil },
{ name = "set", time = 256, repeating = false, params = {"fa", "=", 64} },
{ name = "set", time = 256, repeating = false, params = {"rb", "=", 64} },
{ name = "pushList", time = nil, repeating = false, params = {"something"} },
{ name = "four", time = 128, repeating = false, params = nil },
{ name = "goStart", time = nil, params = nil }
})
Problems Encountered
This is silly, but one issue I didn’t even consider is that my ad hoc macro scripting doesn’t have proper syntax highlighting in any text editor. I’m looking into my options on this. Gedit seems to think that it’s Matlab-related code for some reason.
on_enter, on_tick, on_exit
Each command can have three callback functions. on_enter fires when a RoutineState first jumps to a statement, on_tick fires when the owning entity ticks, and on_exit fires when leaving a statement. In practice, it’s possible to jump from statement to statement, firing on_exit and on_enter but skipping the on_tick callbacks. This is intentional. I can’t necessarily fire on_tick during state changes because per-tick code depends heavily on the scene itself being mid-tick, which may or may not be happening when these functions are called. Neighbouring statements that handle flow control and variable assignment use on_enter callbacks to chain themselves together without yielding, and on_tick itself can kick a chained on_exit + on_enter sequence into motion.
on_exit is forbidden from returning flow control actions, as it would make any command capable of overriding any flow control action from any source.
Unintentional Recursion
Regarding when to yield control in a chain of on_enter callbacks: I did the initial implementation of flow control as plain commands. This worked, but the way it was written meant that chained on_enter statements would call the jump() function recursively. A loop at the list level is “flat”, but if all of those statements are chained together, then there’s potential for a Lua stack overflow.
The solution was to change the callbacks so that instead of invoking flow control functions directly, they return constants representing desired flow control actions, and those constants are passed to (or called within) a function which performs the work in a while loop, without spinning up a tower of stack frames.
Here is a quick diagram I made to visualize the problem. The recursive version is on the right, the looping version on the left:
Performance Concerns
Sitting on a per-tick callback is probably fine (and that was the original planned use-case), but while I haven’t done any profiling yet, I’m absolutely certain that chained statement execution (if statements, counter assignment, label-jumping) is hellishly slow. One potential workaround for specific problem points would be to create a one-off function that does all the logic work, associate it with a command ID, and run that command instead of the individual built-in commands.
Miscellaneous
Garbage Collection
I played around a bit with a semi-manual garbage collection module in the Batteries library by 1BarDesign. This function basically runs the GC cycle for a short amount of time every frame, instead of waiting for a larger amount of garbage to accumulate before collecting. It needs to be tuned to the needs of the project though, so I’m probably going to leave it commented out until I get a chance to test it under Windows.
Custom Iterator
I finally wrote a custom Lua iterator. This creates an alphabetized list of table keys and returns the keys and tables in order when called in a for loop:
function tableOps.alphabetize(tbl, sort_func)
local seq_az = {}
for k in pairs(tbl) do
seq_az[#seq_az + 1] = k
end
table.sort(seq_az, sort_func)
return seq_az
end
function tableOps.sort_upper(a, b)
return string.upper(a) < string.upper(b)
end
-- Usage: 'for key, tbl in tableOps.orderedPairs(tbl) do'
function tableOps.orderedPairs(tbl, sort_func)
if sort_func == "upper" then
sort_func = tableOps.sort_upper
end
local order = tableOps.alphabetize(tbl, sort_func)
local i = 0
local n = #order
return function()
i = i + 1
if i <= n then
return order[i], tbl[ order[i] ], nil
end
end
end
This assumes that all key values in the table are strings. One disadvantage of this iterator is that it creates and then discards an entire table as large as the input table, which is needed to keep track of the correct order of keys. I plan to only use this during startup, when saving config, and to help with generating debug reports.
Project Outlook
The routine macro stuff isn’t really finished, but I guess it’s good enough to plug into creature behavior and see what happens.
My list of missing pieces is getting pretty short now. I wish I had a better wrap-up for this post, but I’m unsure about the next thing to work on. I need to do some review. I’ll follow up around the end of August on whether the RoutineState stuff becomes an integral part of the engine, or if it just ends up being some cobwebbed thing that maybe one or two entities use.
Stats
Codebase issue tickets:
Project lines of code. I’ve decided to include the routine macro text files as part of the project LOC.
