Meta System
The Meta system controls the formats of almost all files in the Telltale Tool & Editor as well. You must read this if you want to mod properly and actually understand how everything works.
Overview
The Meta system is a reflection class system which lists a large array of class descriptions. These descriptors hold information about the classes such as their member variables in them, their flags and other various properties.
Classes are uniquely identified by their type name and their version index (also called version number). The type name is a platform agnostic (independent) type name string which identifies that class. For example, chore files (*.chore) all go under the class with the type name 'class Chore' and just 'Chore' in later games (keep reading for more information). The version number is used because there are a few instances within one game such that a class, such as Chore, has two or more different versions. One may have more member variables than the other, for example some meshes could contain an 'mName' member variable while there is another version exactly the same but without that. To cater for this, these version indices are useful. As you will see around the Lua API, whenever you try to search a class up or get the class of a member in another class, you will have to pass in or will receive two arguments: the type name string and the version index integer. The maximum number of versions of one type name is 10 (version index is only valid from 0 to 9). The default version of the class, the used in pretty much all of the files, if not all, should have version index being 0. Higher index values are used for outlier files which in some games do exist.
If you are curious why some files have different versions, this is because the file was made during development of the engine, and after the file was written the engine was updated and the class of that file (eg Chore) changed. So now we have a file which does not match the class currently in the engine (ie the one shipped that you run when you run the game). Therefore to account for this, version indices are here present. The engine internally fixes this by storing version files (with extension .vers) which contain the format of those classes and their members, so it knows which ones to skip and remap. These problems are only really found in older games.
Classes are registered in the Lua API, described below via functions such as MetaRegisterClass() and various others described in documentation below. You pass in Lua tables which describe the class and its member variables, and then the Telltale Editor will register it internally. Below is a table which shows the optional and required table values.
Name
The type name, eg "class String", "DCArray<bool>"
Yes
VersionIndex
Version Integer, eg 0 for default classes (most common).
Yes
Extension
File extension, eg ''vox" or "d3dmesh".
No
Flags
Flags Integer, eg kMetaClassNonBlocked.
No
Serialiser
Script function name (string), eg "SerialisePropertySet"
No
Members
Table of members, as an array where index = 1,2,3, ...
No
Normaliser
Script function name (string), eg "NormaliseScene"
No
Specialiser
Script function name (string), eg "SpecialiseScene"
No
The members table is optional and if it has members, then each member must be a table with the following format:
Name
The member name, eg "mName", "mbHasProps".
Yes
Class
The member class table, previously registered.
Yes
Flags
Flags integer, eg kMetaMemberSerialiseDisable.
No
EnumInfo
If an enum, table of tables with 'Name' and 'Value'.
No
FlagInfo
Same as above. Name is a string, Value an int.
No
Some member variables can be enums or flags. This means that they must be a integer type but can contain a list of string to integer pairs remapping numbers to string means.
For example, lets take a look at the class 'enum NavCam::Mode'. This is obviously an enum class. It has one member variable, 'mVal'. This member is an enum member, meaning it has a list of associated strings which give meaning the integer values which this value can take. When registering this class, this member table would have a table EnumInfo as a table of tables, with indices similar to Members in the class table being from 1 to N where there are N members. For example we could set EnumInfo[3] for the third enum to be a table of string Name being string 'eOrbit' and string Value being integer 3.
This works the same for flag types, although the values of flag types must be powers of 2, if you are unsure why; have a look online for information about using integers as flags.
The specialiser and normaliser script functions are only required for certain types. These functions normalise the given type into the common format using appropriate functionality. The point of this is because each game has a different format of different files and for the editor to be able to make sense of all of them it needs a common class and internal view of classes. This normaliser translates a class instance into the common format. The same goes for Specialiser, which is another function which does the opposite and takes in the common format and fills out a new instance which can then be written back to a file. See examples from actual game scripts for more information on how they work.
Creating & Managing Class Instances
OK, we have a big array of classes which we have registered. But what can we do with them? The first most important thing is, like in all OOP languages, to be able to create an instance of it (if you are not sure on this, read up on OOP, as you must understand it first - to a basic level). You can do this easily by the create instance function, which is described in documentation below. You can also create a copy of an instance, ie deep-copying the values of the members and their sub-members into a new instance. You can also move instances, which is equivalent to std::move in C++. This just means leaving the old instance empty, while moving everything to the new instance. The old instance is still valid but just contains values as if you just created the instance and had not set any of its value or serialised into it yet.
The lifetime of instances are managed by an internal reference counting system. When you use a function such as TTE_ReadMetaStream, it creates an instance of the class serialised in the file and returns it to you. This among a few other examples, is an example of when you own a strong reference to the instance. This means that as long as you somewhere in your Lua code has access to that instance of the class, it will stay alive - ie valid. However, most of the time you will be internally holding a weak reference to it: meaning that it is not guaranteed that your variable in Lua still holds a valid reference. The best example of this if when you use GetMember, or equivalent. This function, explained in the documentation below, returns a class instance to the given member in that class: for example the 'mName' string instance in the mesh class can be retrieved by using this function. However, you obtain a weak reference to this member. What does this mean? This means that when the parent class of that string instance, the mesh in this case, gets destroyed (ie nobody holds a reference to it anymore), calls to retrieve the value string of that member may return nil - as it would not make sense to hold a reference to mName if the mesh instance isn't alive anymore. There are functions which can be used to test if an instance is alive still, however if you know the instance is alive there is no point worrying as it is guaranteed to be alive if you know you hold a reference to it.
When you have a strong reference, the class instance you refer to is known as top-level. This simply means that it is not owned by any other class instance. Ownership of instances is very important: a weak reference infers that your instance your reference refers to is not top-level; it is owned by another class instance: for example when your instance was retrieved by GetMember, the class instance you got it from owns it. This means that internally, each instance has a list of its child instances, such that when that instance gets destroyed it invalidates all (weak) references of instances to any of its children. Internally, that class list is an array of strong references: they hold ownership of them, which makes sense, they are members of that class.
This child instance list can be enabled for any type when describing it in the Meta registration. It adds an overhead however, so this would be very bad for any intrinsic types such as integer! Only use these for large types where you know that it does require it, such as animations (store animated value) or meshes (embedded material property sets).
The one exception is PropertySets (.PROP). You must enable their child array in the registry as it is used for key-value pairs. Furthermore you must not create child instances under it! It will throw and error and not work, as it required for the key-value pairs. These are very standard files and don't change much at all so this won't be a problem.
What the meta system also exposes to you is a way to add and remove your own child instances to a class instance. What this means is that you can create instances in which their lifetime depends on another previously declared instance. This is extremely useful and is used in many classes, most namely the property set (prop files). Property sets at its core are just a map of property keys to values (basically a Lua table). The keys are symbols (think of these as a string, but faster internally). The values are an instance of any class in the meta system, eg another property set! How would we store these in the class for the property set through members? We would need an array of instances with different classes (ie a tuple). The property set stores each of its properties in the child instances array of itself. Where are the keys stored? Well the child array is actually map: it maps symbol names to instances. This lets you be able to give a name to each of the child instances too, useful not just for property sets but also where you want to store any other class instances in your class instance - you could give it an internal static name such as '__MY_DATA__', where you know that key won't be used anywhere else (it's totally unique). And in the property set example, what makes this even more ideal is that the property key-value pairs lifetime is dependent on the property set being alive.
To create an instance of a class, use the MetaCreateInstance function passing in the parent class which the lifetime of the returned instance depends on. You must also pass in the name (symbol) to name this new instance, as described above. You can use functions documented below to search for a child instance of a given instance given it's name in the future.
Serialisation of Classes
What makes this system so powerful is that these classes can be serialised immediately by definition after registration to the meta system. By calling functions such as TTE_ReadMetaStream, you can read any file which uses that class version from a file and load all its member values stored in the file into memory. However, sometimes more customisation is required for the serialisation, as serialising each member one by one is something not feasible. For example, meshes contain a large amount of information in the first part of the file, ie each of the members in the mesh class. However, a mesh by definition is a collection of vertices and lots of other associated data. It is a bad idea to store vertices in an array class, for example meshes don't just have a member called 'mVertices' being an array of Vector3 values. They are stored in large buffers which are uploaded to the GPU. So its quite obvious that we need a custom serialiser for this class.
Default vs Custom Serialisers
Before getting into custom serialisation routines, you should understand that if you don't use a custom routine then when serialising a class you are serialising it using the internal default serialiser. This is a simple serialisation routine: in the order that the members are declared in the class members table, they are serialised one by one. A block size is internally wrapped around each member before its serialised, unless the class of that member is instructed not to via the non blocked flag (see table below). This is a binary integer in the binary file which is the size of the serialised member in the file. This is useful for checking after the member has been read that we read the correct number of bytes. After the member has been read, we check the current number of bytes we are into the file and compare that to the initial number of bytes we were at before reading that member, plus the block size. This allows us to check if we have read too many or too little bytes, ie meaning we have the incorrect format OR the file is corrupt. This can save us from crashing or reading the file wrong generally. If this is detected, the serialisation fails.
All custom serialisation routines are written in Lua as they can be different for different versions of files. This is done by specifying the name of the serialisation function in the class table when registering the class. The name of the function must be unique and previously loaded using a require if needed. Internally it is compiled when you register it and saved to be executed when needed. This routine must take in there parameters in the order: meta stream, instance and is write. The first is the input or output stream of bytes which we are writing to or from (the file). The instance is, quite obviously, the instance of the class. You can use the get class function, see docs below, to get the version number of the current class if you want to use one serialiser function for multiple versions of the same class to save code duplication. The last argument is a boolean simply stating whether we are currently writing the instance to the output stream, or reading from the input stream into the instance.
Reading from and writing to the stream argument in a custom serialiser can be done using the MetaStreamXXX functions, all described below.
Almost every time there is a custom serialiser, you still want to first serialise the members and then your more complex file structure. To do this, just simply call the default serialiser, MetaDefaultSerialise. This will go through the members and serialise each as talked about above. If this fails, you should return false as well. After this, then you can serialise the rest of the file how it should be serialised. You can read integers, strings and other intrinsic classes right from the stream (write too) and serialise any child instances as required.
Available Class Flags
These are flag values which the class table key-value pair Flags can contain. To combine these flags, it is OK to just add them. However: be careful to not add them twice, and don't use any other operators like minus. This will break and internally it will read as a weird combination of possibly other flags. Use functions such as MetaFlagQuery to test if a flag is set before adding it if you don't know if its set yet!
kMetaClassNonBlocked
If default serialised, it won't automatically have a block size.
kMetaClassIntrinsic
Intrinsic class, meaning it won't be in the meta stream header.
kMetaClassAllowAsync
This class is large, and is allowed to be serialised async.
kMetaClassContainer
This class is a container class, used in MetaRegisterCollection()
kMetaClassAbstract
An instance of this class cannot be created, its abstract.
Available Member Flags
These are flag values which the member table key-value pair Flags can contain. Like enums, be careful when combining these but it is OK to just add them to combine multiple.
kMetaMemberEnum
This member is an enum, so table EnumInfo is required.
kMetaMemberFlag
This member lists flags, so table FlagInfo is required.
kMetaMemberBaseClass
While treated as a member, this member is a base class.
kMetaMemberMemoryDisable
Does not exist in memory for an instance of this class.
kMetaMemberSerialiseDisable
Do not serialise it in the the default serialiser.
kMetaMemberVersionDisable
Do not include this member in any version hashing routines.
Documentation
All scripting API for the Meta system can be found by generating Lua documentation. Run from command line with 'mkdocs', or do this from the editor application. Below are some of the most important functions. Like all scripting APIs, the prefix is all the same for all Meta system functionality starting with 'Meta', so type that in VS Code and then it should show you all available functions and their documentation.
This is used in the Games.lua script to register a game to the Meta system on initialisation. It takes in a table which must have keys string Name, string ID, bool ModifiedEncryption, table Key[string platform] to string hex key, etc. You could add your own game if you want to create one!
This registers intrinsic types required for the meta system for the current game. This should only be called a game meta classes registration script (eg WD100.lua).
Sets the function (pass the name not the function) which generates the version hash for the given class table (the function should return a number, see hashing functions below, and take in the class table). This should be set first, then call register intrinsics, then the rest of the classes. See existing game scripts for examples.
This registers a class to the meta system, passing in the information table. This must only be called in the initialisation. Table must contain the Name string, VersionIndex number, optional flags number, optional serialiser script name and optional members table. See example scripts from games for more information on its use with examples. You can register multiple classes with the same name as long as they are not exactly the same, ie the version hashes are different.
This registers a collection class to the meta system, only to be used in initialisation. Pass in the same table information as would be used in the normal register class. The table is the information about the collection class. The second argument is the key type value table, (previously defined and must have been passed into register class). This can be nil for non keyed collections, such as most arrays, but must be a previoulsy registered table otherwise (eg for Map classes). If this is a static array (SArray) class, then this should be an integer being the number of elements. The third argument must always be non nil and a table, which is the previously registered value type in the collection.
There is a special case where key or value type table can be strings. If you pass them as a string, this means they will be resolved once all classes have been registered. This allows forward declaration of a class inside a collection before its fully defined. Instead of padding in the type table you pass in the string name of the class it should be. Optionally you can end the string with a semicolon followed by the version index, eg 'class Hello;1', such that version index 1 is used in this example. By default version index 0 is used.
Returns true if the given instance of a class is alive. Returns false if the instance is invalid or the parent of the instance is no longer alive - meaning you cannot access the instance anymore as it does not exist.
Obtains a weak reference (instance) to the given member variable in the given class instance in passed. Pass in the name of the member variable as the second argument.
If the instance is an intrinsic type, it is converted to the equivalent Lua type. Strings, booleans, floats and doubles translate as normal. Integers, unsigned or signed, casted all to the same internal Lua number type, so this can be used for all integer types. Symbols are converted to symbol strings. Integers of width 64, signed and unsigned, are always converted to symbol strings as well, as the Lua number type cannot hold them to the correct value. If it is not one of those types, a normal weak reference is returned as it is a compound, normal class type.
Similar to the get class function. Sets the value argument. Ensure that the type matches what you are setting it to, as it won't be casted.
Creates an instance of the class specified by the first two arguments. The third argument is the name you want to give to the returned child instance. You can then access this returned instance later by finding the child instance by this unique name. Pass in a valid instance as the parent instance, which the returned instance's lifetime depends totally on.
Creates a new instance, copying everything from the 1st argument into the returned instance, leaving the first one unchanged. Rest of arguments are the same as CreateInstance. Parent and instance cannot be the same.
Creates a new instance, moving everything from the 1st argument into the returned instance, leaving the first one alive but empty like it was just created. Rest of arguments are the same as CreateInstance. Parent and instance cannot be the same.
Returns the class information of a given class instance. For a similar function to find the class information about a class serialised inside a meta stream, use MetaStreamFindClass.
Converts the given instance to a string. If the class of the instance has a defined to string meta operation, then that is called. All intrinsic types such as integer are converted to string by default. Higher level typed, is used defined typed, return a list of the member names and the instance values formatted nicely in a string.
Compares the two instances, using their less than meta operation. For intrinsic types such as floats and integers, this does the usual. Else this will return false for other types without a defined comparison operation.
Same as less than but returns true only if they are equal using the comparison meta operation.
Returns true if the given instance (1 argument) else is a collection class. If two arguments, pass in the typename and version number to test.
Returns the number of named children that the given instance holds. This does not include any instances returned with GetMember, but only MetaCreateInstance.
Returns a weak reference to the child instance associated with the given parent under the given name.
Releases the given child (identified by name) of the given instance. Use this to remove it from the internal map such that it will get destroyed after (unless a lower level C++ object still obtains a reference to it). Calls to get child after this will return nil.
Returns a table (indexed from 1 to N) of all of the children names in the given instance. Each value in the table is a string. Some children names may be hash strings (length 16, upper case hex) if the symbol could not be resolved. Internally child names are stored as symbols. Try to use global symbols, and use SymbolCompare to compare, so reduce this risk.
Reads a DDS file header returning a table with its information.
The returned table will contain the following:
Flags
Integer
DDS_HEADER::dwFlags
Width
Integer
DDS_HEADER::dwWidth
Height
Integer
DDS_HEADER::dwHeight
Pitch
Integer
DDS_HEADER::dwPitchOrLinearSize
Depth
Integer
DDS_HEADER::dwDepth
Mip Count
Integer
DDS_HEADER::dwMipMapCount
Format Flags
Integer
DDS_HEADER::ddspf ⇒ DDS_PIXELFORMAT::dwFlags
Format CC
String
DDS_HEADER::ddspf ⇒ DDS_PIXELFORMAT::dwFourCC
Format Bit Count
Integer
DDDS_HEADER::ddspf ⇒ DDS_PIXELFORMAT::dwRGBBitCount
Format Red Mask
Integer
DDS_HEADER::ddspf ⇒ DDS_PIXELFORMAT::dwRBitMask
Format Green Mask
Integer
DDS_HEADER::ddspf ⇒ DDS_PIXELFORMAT::dwGBitMask
Format Blue Mask
Integer
DDS_HEADER::ddspf ⇒ DDS_PIXELFORMAT::dwBBitMask
Format Alpha Mask
Integer
DDS_HEADER::ddspf ⇒ DDS_PIXELFORMAT::dwABitMask
Caps
Integer
DDS_HEADER::dwCaps
Additional Caps
Integer
DDS_HEADER::dwCaps2
If the FourCC is empty then the value was 0 and else its the string version of it, in little endian ie you don't have to worry about reversing the string characters.
If the FourCC is equal to 'DX10' then the table will contain (and must for Write below) the DDS_HEADER_DX10 information in the table below as well, which is in the same Lua table but just these extra values included:
DXGI Format
Integer
DDS_HEADER_DX10::dxgiFormat
Resource Dimension
Integer
DDS_HEADER_DX10::resourceDimension
Misc Flags 1
Integer
DDS_HEADER_DX10::miscFlag
Array Size
Integer
DDS_HEADER_DX10::arraySize
Misc Flags 2
Integer
DDS_HEADER_DX10::miscFlags2
Writes a DDS file header with the given information. It must include the same information as returned by the read version above.
Gets the number of bytes that would be written or have been read from the given DDS header table that either you contruct or was returned in the read function above. This is the number of bytes read or would be written.
Finds the class in the meta system associated with the given type symbol for the given meta stream. The symbol is either a symbol (16 byte hex hash string) or a string which will be hashed. Examples of use are passing in a symbol read from the meta stream when the meta stream contains a class determined by its symbol also stored in the meta stream. Returns the class type name and version number associated. The class is found by searching the meta stream header for the type name symbol, if it is found then the returned class is one with the same version hash as in the header of the meta stream. If it is not found in the header, then it may still be valid as intrinsic types are container types are not in headers. In those cases, the type with version number zero is returned - as those types don't have different versions and if they do they are never used.
Last updated