Spreadsheet Balancing: Application to the Game

In my previous post on spreadsheet balancing, I introduced my process of finding key values – in particular, how quickly resources need to drop in abundance for trading to be profitable – that will shape the first pass of balancing in Silkroads. While previous balancing, specifically regarding the spread of abundance across the map:

abundancespread1

… was done by eye, this likely won’t do. With some simple plotting in Microsoft Excel, I’m able to understand how this spread should look to produce profitable, unprofitable, and neutral routes.

Again, here’s the table of calculated Critical Delta Abundance (CDA):

excel1

I’ve added rough distance indicators to give a sense of the distances that these values represent. Remember, red cells (larger than 1) are impossible to trade profitably in any situation, on a base-infrastructure route. So, when the player starts the game, the most far-reaching resource routes are those of Silk, Silver, Gold, and Lapis Lazuli. They can be traded, in the best possible scenario, the distance between London and Kiev – roughly the length of Europe. The CDA of Silk is 0.04 delta-abundance per 100km.

I begin by putting these values into the game, along with the other key values I’ve been working with. The standard caravan can carry 100kg of cargo, for example, caravans travel at 500km per year, and it costs 5 monies to travel for a year.

CDA1.png

After adding some grid squares to make tweaking-by-eye easier, I create my first drop of abundance: the silk road itself. The distance between Merv and Bactra, I can guess, is about 400km. So I subtract 1.5 abundance between Bactra and Merv. In fact, I subtract a little more, since I want the silk road to be quite a viable, profitable route: silk abundance should drop fast enough for a profitable route to be quite comfortable.

I do the same between Merv and Nishapur, and so on …

Testing this route with a caravan produces some quite astonishing results …

play1play1and1halfplay2

Utilising the maximum carry load, the route is just profitable. This is precisely the desired effect. However, I do feel that the game should be more forgiving, and not expect the player to trade a perfect load each time. So I increase the delta-abundance a little more.

So, here’s the same technique with glass. Again, I start by identifying the production source. The levant was the famous birthplace of glassware in antiquity, with the Belus river – approximately near Tyre – being a notable site. Mesopotamia, too, had reasonable abundance of glassware. These wares were among the most common to be traded West-to-East in return for silk.

glass1

The distance between Tyre and Antioch is close to the distance between London and Newcastle: 400km, almost the length of England. I can see on my excel sheet that the critical spread drop-off for glass at this distance is 0.5. In other words, glass needs to drop by 50% abundance each Tyre-to-Antioch distance on the map.

glass2

I decide this is too fast, and I’m conservative with my drop-off. A quick playtest will show how this behaves.glasstrade1

GlassTrade2.pngGlassTrade3.png

Surely enough, the trip isn’t actually profitable. Is this acceptable balancing? Maybe. The player may need to upgrade infrastructure before trading glass becomes an option. Additionally, because buying on both trips can produce far greater profits (e.g. buying gold in Babylon so return to the markets of Tyre), a more complete maps might make the glass trade much more viable.

This is why even this careful tweaking on the first balance pass isn’t enough: playtesting is needed to truly highlight balance oversights.

Running through all eight current trade goods, and adhering roughly to the spreadsheet values, the map looks like this:

needsmoreincense

It doesn’t seem immediately unbalanced, overall. Most regions have some abundance to offer in trade to other regions. A major exception is the Arabian peninsula. To my knowledge, it wasn’t a production source of Silver, Gold, Lapis Lazuli, Nutmeg, or Cinnamon. I’m sourcing Ebony from Ethiopia, and want to leave the trade potential open for sea-routes across the Red Sea. Arabia needs its own luxury item. And it should be reasonably potent, given that Arabia was a nexus of trade from antiquity onwards.

Luckily, history has just such an example in the incense trade. So, I also add a new trading resource, and populate it across south Arabia.

firstpassbalance

And with some new non-gizmo UI, that’s the first-pass balance via spreadsheet. Its effectiveness will be of interest in the first round of playtests, which are soon to come. Hopefully, this method will have produced a map lacking in easy exploits, in which the player needs to build up their position by finding subtle profitable routes, before expanding infrastructure to support more lucrative trade.

Advertisements

Spreadsheet Balancing of Abundance

Key values are all in place. You can select trade routes – some of which will be more profitable than others. You can build infrastructure to make trading over long distances more feasible. Now what I need to do is ensure that abundance changes across the map – the rise and fall of selling-price for silk, spices, etc. – approaches a reasonable rate. This falloff needs to vary between somewhat profitable and very unprofitable: neighboring nodes with too high an abundance difference represent a dominant strategy, or an easy exploit which destroys the challenge of silkroads. If I can get rich trading between Babylon’s 1-abundance and neighboring Ur’s 0-abundance, what is the point of seeking out other search routes.

AbundanceSpread1.png

Many blog-posts ago, I explained my first pass at resource abundance distribution. Abundance – a value between 0 and 1 representing market saturation (0 sells high, 1 sells low) – can be seen for each resource in the bar-charts above. My first pass handled a few nodes strung across the silk road. Since then, I’ve added many more. All require more thorough treatment to create a balanced, challenging resource spread.

Managing these values by-eye and with trial-and-error would be possible, but extensive. When core values need to be quickly judged, designers employ a different tool.

Spreadsheeting

The task of showing numerous interdependent values – perhaps in the hundreds or thousands, allowing for tweaks, and with a customized display for easy reading, is perfectly handled by Microsoft’s spreadsheet software: Excel. I’ve been looking forward to using it to my advantage for the sake of predictive balancing.

To begin with, Excel can open my Goods.xml file directly:

spreadsheet1

From these values, I can determine a great deal. I’ll start with the abundance/valuePerKilo graph. A trade good’s value at any one node is given by the function:

localVPK = (-v + p) * a + v

Where v is the base valuePerKilo, p is prodCost, and a is local abundance (the x-axis). Writing this function into Excel’s cell function is easy enough, but I actually need it available in a lot of places. So, under the Developer tab, I can write it in Visual Basic as a custom function.


Public Function LocalVPK(vpk As Single, prodCost As Single, abundance As Single) As Single
LocalVPK = (-vpk + prodCost) * abundance + vpk
End Function

This was my first time using Visual Basic, but my experience with other programming makes these menial tasks easy to understand. This is just a custom mathematical function, which can give me the local value-per-kilo of any good, whenever I need it.

spreadsheet2

Making a table of possible values, between 0 and 1 abundance, I can graph the localVPK function for each good.

spreadsheet3

Again, this just demonstrates the value per kilo (v/kg) that a good would sell for at a node, for every possible abundance at that node. At 0 abundance, a good sells for its max price. At 1, it sells for its production cost.

Originally, the localVPK function I had in the game sold a good at 0 v/kg when at full abundance. This meant that at full abundance, a good would sell for nothing and could be purchased in infinite quantity for any amount of money, which didn’t make much mathematical or economic sense. The thinking behind this newer function is that, no matter how abundant a resource is, it will never be sold below its production cost, as the producer would be selling at an immediate loss. If it costs 1 value (1v) to mine 1kg of gold, selling that gold for 0.5v/kg is simply unthinkable.

Production cost – the lower limit of selling a good – is a fraction of maximum cost (base vpk). Currently, this is always 50%, but could easily be tweaked for more interesting market profiles.

Note that, despite this even ration, different resource graphs have different slopes: more valuable (per kg) resources, such as gold and silver, have steeper slopes. Less v/kg goods, such as glass and spices, have shallower profiles. This turns out to be very important.

Critical Delta-Abundance

The value I’m really interested in here is a complicated one. Assume I have a caravan travelling between two cities, 100km (1 Unity-unit) appart. Assume it can carry a maximum load of 20kg. Assume it’s trading silk, and that the first city has a higher abundance of silk than the second, so that it buys in cityA and sells in cityB. What is the minimum difference in abundance between cities A and B, needed to give the caravan enough profit to compensate for the cost of travelling.

This value is the Critical Delta-Abundance (CDA). It’s measured in difference in abundance per unit distance (da/100km). If the CDA is 0.3, and abundance at cityA = 1, while abundance at cityB = 0.7, the caravan can trade silk between the cities and make a net gain of exactly 0. If the difference in abundance between the two cities is less than the CDA, trading is not profitable. If it’s greater, trading is increasingly profitable.

Finding the CDA amounts to finding the delta-abundance from the delta-value (the value lost from travel expense), which is easily done with:

CDA = c / slope = c / (dx/dy)

where c is travel cost, and dx and dy are the differences in x and y. Since the graph is linear, dx and dy can be the entire width and height. Dividing the two yields the slope: 0 for flat, infinite for vertical, 1 for top-right diagonal.

The CDA should also take into account the quantity of goods being bought. This simply becomes:

c = q * (v – p)

q being quantity, v being base vpk, and p being production cost. This is important, because the slope of a graph for 1kg of silk is lower than the slope of a graph for 20kg silk.

Again, I’ve written this as a visual basic function:

Public Function CDA(distance As Single, amount As Single, vpkZero As Single, vpkOne As Single) As Single
CDA = (2 * distance) / (((vpkZero - vpkOne) * amount) / 1 - 0)
End Function

So now I can get the CDAs of each good, and at different distances.

spreadsheet4

Remember: these values are the minimum delta-abundance at given distances needed for a profitable trade route of each resource. It also assumes no infrastructure – a mechanic which decreases the cost of travel. Where CDA is 1 or larger, trading is outright infeasible even at the most ideal situation (since abundance cannot differ by more than 1). So, for example, trading silk over a distance of anywhere above 500km simply cannot be profitable (without infrastructure upgrades). If the player was trading silk from China, and it took over 500km for the abundance to drop to 0 (which is only the distance between the entrance to China’s northern trading pass, and Kabul in Afghanistan, for example), then trading would be unprofitable.

An extreme case can be seen with Cinnamon. It requires a full drop in abundance over a distance of 100km – roughly the distance between central London and Oxford. With its current rate, it’s very difficult to trade.

The only resources worth trading over very long distances are the more valuable ones, with the steeper v/kg/a slopes. Gold, in particular, is still profitable across 2,000km, about the distance between London and St. Petersburg.

Looking at the abundance vs. price graph, it’s clear that the slope of the graph indicates how viable long-distance trading is. A steeper slope means the good can e traded further. As a result, I’ve tweaked the production costs of Gold, Silver, and Lapis, making them more expensive to produce, and thus a little less profitable.

spreadsheet5

There’s much more to be done. It’s time to apply these important values to the game.

Map Projections and Mercator Distance

The systems I need to handle core gameplay are now almost in place. While approaching a final issue – calculating the cost of a journey by its distance – it occurred to me that now was the time to deal with something I’ve been thinking about for a while: geographic distance.

Maps are not perfect representations of the globe (and they can’t be, since it’s impossible to spread a sphere into a rectangle with no distortion). Instead, cartographers pick and chose the exact method they use to spread the sphere out, which has an effect on how the geography is rendered. Different projections are preferred in different situations. A very standard, scientific type of map projection is the equirectangular. This simply takes longitude and latitude, and turns them into x and y coordinates. It means that the map is always a circumference wide, and half a circumference tall, and that positions are handled in a mathematically simple way (making it useful for making calculations with). It also means that the northern and southern latitudes are stretched horizontally, making for a rather squat British Isles, for example.

(Circles – or Tissoy’s Indicatrix – represent local distortion, which is minimal at the equator, but quickly becomes noticeable in Scandinavia and Greenland)

Arguably even more common than the equirectangular is the Mercator. A common sight in geography classes, the Mercator does what the equirectangular does not: stretches the projection vertically as it approaches the poles, maintaining the aspect ratios of landmasses.

There are a lot of implications here. Most important to the Mercator’s success: directions remain consistent, and plotting a straight-line from Greenland to Madagascar maintains a straight line in real life, making the Mercator a favorite of navigators. Since poleward latitudes are being increasingly scaled, the distance between meridians (the vertical lines) shrink from several thousand kilometers, to a few kilometers, to a few meters around the poles. Vertical stretching means that the Mercator cannot render the north or south pole, as this would take infinite map (rendering a singular point with a “maintained scale” amounts to multiplying 0 into a larger number: impossible). And the real controversy with the Mercator is its shrinking of equatorial geography and growing of temperate regions, such that Greenland appears larger than South America (which it is not), and the European continent receives much more space than it deserves: a feature which many see as Eurocentric.

When I began Silkroads, I actually started by using an equirectangular. After reading about the projection, it seemed the obvious choice: it’s mathematically simple, making it easy for me to code around. However, a few months back, I just couldn’t get over the squashed Europe.

equirect

And the fact is, grand strategies like Europa Universalis IV seem to use some kind of Mercator-based map, providing the familiar proportions of poleward nations.

So I decided that Mercator was more aesthetically pleasing, and switched before I’d gotten too much of the map implemented.

Mercator.png

This distinction is important for the issue I’m about to describe, though, really, both projections suffer from the same problem. Non-geographic distance.

Mercator-Equirectangular Conversion

The simple case of as-the-crow-flies distance between two positions on a map is, actually, not so simple. While spatial accuracy is, of course, maintained on a globe, both Mercator and equirectangular gradually distort northern and southern latitudes more and more. If I wanted to check the distance between two equatorial cities on my map, say Kuala Lumpur and Mogadishu, I could use the standard Unity way of (positionB – positionA).magnitude and have pretty accurate results. But with any positions straying north or south – London to Istanbul, say – my results would be incorrect. Europe is much larger than it should be on a Mercator (and even wider than it should be on an equirectangular), so travel distance will be calculated as longer.

Luckily, this is a very common problem, and there are plenty of resources online explaining how geographic distance can be found with map coordinates. Among the most simple is equirectangular aproximation.

x = (long2 – long1) * cos ((lat1 + lat2) / 2)
y = (lat2 – lat1)
distance = SquareRoot(x^2 + y^2) * R

Where long 1 and 2 are the first and second horizontal positions, and lat 1 and 2 the vertical, in radians (1 radian = 1 radius, wrapped around the sphere, i.e. circumference / 2Pi, and 90 degrees/1 quarter way along the Earth’s circumference = 1.57 radians (1/2 Pi)). R is the radius of the map, if it were a globe with the same circumference as the map’s width, i.e. a globe at the correct scale as the map (my map is 400 Unity units wide, so its radius is 60.3 units). Notice that the last part of the equation is just the Pythagoras theorem. Also, be aware that this equation apparently best suits smaller distances (which is strange, because I found it accurate at half a circumference’s distance), and that it has a small degree of inaccuracy (which can be countered by more complex, but more intensive, equations).

Also notice that it’s an equirectangular equation. Herein lay an extra step of my issue: my map is Mercator, and so are the positions of its cities. So to use this nice and simple method, I had to convert from Mercator to equirectangular.

Again, this issue is reasonably common. However, it is more complicated, and did involve math which I frankly don’t understand. It took a lot of trial-and-error to get it right, in particular because math guides so often have implicit instructions (use radians, for example).

More commonly available is the equation for converting from longitude/latitude to Mercator:

x = R * long
y = R * Log(Tan(Pi / 4 + lat / 2))

This seems to be in radians, again, but I found that I had to tweak my input long/lat values to make it work. These equations tend to assume that you’re working with, as I said, longitude/latitude measurement-of-angles. This essentially is equirectangular, but the world-space coordinates need to be correctly converted to radians. After much fiddling, I found this to work:

// Convert an equirectangular map position to a mercator position.
public static Vector3 EquirectToMerc(Vector3 equirectCoords)
{
Vector3 mercCoords = new Vector3();

mercCoords.x = equirectCoords.x; // Longitude remains unchanged between mercator and equirectangular projections.
mercCoords.y = equirectCoords.y;
mercCoords.z = earthRadius * Mathf.Log(Mathf.Tan(Mathf.PI / 4 + (equirectCoords.z / (earthCircumference / 2) * (float)Math.PI) / 2));

return mercCoords;
}

Some testing confirmed that the equation was doing its job:

MercConversion1.png

For Mercator to equirectangular, the equation is:

long = x / R
lat = ArcTan(SinH(y / R))

Again, notice that it’s expecting an input of x,y coordinates (true, in my case), but outputting angles in radians. So once I have those radians, I just need to remember to convert them back to world position, a simple case of radians / (2 * Pi )* circumference.

// Convert a mercator position to an equirectangular position.
public static Vector3 MercToEquirect(Vector3 mercCoords)
{
Vector3 equirectCoords = new Vector3();

equirectCoords.x = mercCoords.x; // Longitude remains unchanged between mercator and equirectangular projections.
equirectCoords.y = mercCoords.y;
equirectCoords.z = Mathf.Atan((float)Math.Sinh(mercCoords.z / earthRadius)) / (2 * Mathf.PI) * earthCircumference;

return equirectCoords;
}

mercconversion3

MercConversion4.png

You might notice that the lines seem to be a little off. Mercator Denmark lands in equirectangular Oslo, and equirectangular Northern Ireland converts to the border between Northern Ireland and Republican Ireland on Mercator. These offsets are my fault, rather than the algorithms. The truth is that my Mercator is hand-crafted in Photoshop, as I couldn’t find a Mercator with high enough resolution, so I just adjusted my high-res equirectangular. It’s close, but there are small distortions. Still, this is fine for my purposes.

You can see more effective results when I import a true Mercator.

mercconversion5

Calculating Geographic Distance

With these functions ready, getting geographic distance is now very simple. I just write a function that takes two Mercator (world-space) positions, converts them to equirectangular, converts them to radians, runs the Pythagorean aproximation, and converts back to World-space distance.


// Returns the geographic as-the-crow-flies distance between two (Mercator) positions.
public static float GeoDistance(Vector3 posA, Vector3 posB)
{
// Convert the mercator positions to equirectangular.
posA = MercToEquirect(posA);
posB = MercToEquirect(posB);

// Convert the coordinates to radians. 1 radian = the length of the Earth's diameter. 1/2 the Earth's circumference = Pi / 2 radians.
Vector2 aAngles = new Vector2(posA.x / earthRadius, posA.z / earthRadius);
Vector2 bAngles = new Vector2(posB.x / earthRadius, posB.z / earthRadius);

// Pythagorean distance calculation.
float x = (bAngles.x - aAngles.x) * Mathf.Cos((aAngles.y + bAngles.y) / 2);
float y = bAngles.y - aAngles.y;
float distance = Mathf.Sqrt(x * x + y * y) * earthRadius;
return distance;
}

Now we can test the difference between two easy-to-locate by eye cities: London and Istanbul.

Distance3.png

GoogleMaps.png

Almost perfect, and remember that my hand-made Mercator is a little off.

So now I can use true distance as a basis for geographic movement costs. The further away a city, the more it should cost to travel there. Moving from Tyre to Jerusalem:

distance6

The actual distance between the two cities is 170km, so obviously I’m quite far off. This is actually because I’ve placed the cities by eye, and close inspection reveals that Tyre is too far South, while Jerusalem is too far North. This is partly so that I can get the cities nicely spaced, so that’s an element of realism I’m willing to sacrifice (Palmyra, too, should be further north than Tyre, but then it would be too close to Dura).

Testing with a larger distance:

distance7

Between Rhaga (a district of modern Teheran) and Nishapur (modern Neyshabur), I get a distance of 600km. It’s a little short of the true distance: 660km, but, again, these cities were placed by eye. What’s important is that different latitudes work.

So there you have it. With this feature in place, I can take the final step to adding a central element of challenge to the game: costly journeys, and building infrastructure to reduce cost.

The Final Word on Serialization

This project has seen me struggling to find a stable architecture for handling the serializing and deserializing (saving and loading) of important data types; something essential to any strategy game (for handling types of units, types of countries, types of resources, etc.). In some of my previous posts, I’ve explained my approaches to this problem with XML serialization and a custom serializable dictionary, and I followed each one, along with a handful of a few other attempts, with a triumphant “I’ve done it!”. As you might gather from this post, this was not the case, and it still may not be. Nonetheless, here’s the latest saga of Tom vs. Unity Serialization.

Recall the “Good” data type: this is the first (likely of many) data structures that I’ve created for my game. It describes trade goods: silk, gold, spices, and others.

public class Good : ILabel
 {
 [XmlAttribute("name")]
 public string iLabel = "Name";
 [XmlAttribute("valuePerKilo")]
 public float valuePerKilo = 1f;
 public Color guiColor = Color.black;

 // Empty constructor for XML serialization.
 public Good(){}

 // Constructor.
 public Good(string _label, float _valuePerKilo)
 {
 iLabel = _label;
 valuePerKilo = _valuePerKilo;
 }

 public string label
 {
 get{return iLabel;}
 set{iLabel = value;}
 }
 }

This code is nothing special. It’s just a simple list of attributes describing some things I might need to know about any particular trading good. With the use of an XML file, like this one:

xml

… and an appropriate Load() and Save() function, in the “Goods” class (which stores a list of Good objects):


// A list of all trading goods.
[Serializable]
[XmlRoot("Goods")]
public class Goods
{
[XmlArray("Goods")]
[XmlArrayItem("Good")]
public List<Good> goods = new List<Good>();

// Saves this goods class as an XML file.
public void Save(string path)
{
XmlSerializer serializer = new XmlSerializer(typeof(Goods));
FileStream stream = new FileStream(path, FileMode.Create);
serializer.Serialize(stream, this);
stream.Close();
}

// Loads a goods class from an XML file and returns it.
public static Goods Load(string path)
{
XmlSerializer serializer = new XmlSerializer(typeof(Goods));
FileStream stream = new FileStream(path, FileMode.Open);
Goods loadedGoods = serializer.Deserialize(stream) as Goods;
stream.Close();
return loadedGoods;
}
}

… and finally a Singleton accessor that ensures Goods is always loaded when something tries to access it:


// Singleton for accessing goods.
public class SGoods
{
// Singleton instance.
private static SGoods instance = null;
public Goods goods; // All goods.

// Constructor.
public SGoods()
{
// Load goods from XML.
goods = Goods.Load(Application.dataPath + "/Resources/Data/" + "Goods.xml");
}

// Singleton accessor.
public static SGoods Instance
{
get
{
if(instance == null)
{
instance = new SGoods();
}

return instance;
}
}
}

… I had a system that allowed me to easily edit the Good data, and that ensured the data was always available as soon as something needed it. It seemed to be working, too: my game could run in editor with no issue. A few suspicious errors would sometimes appear on loading Unity, though: “get_dataPath can only be called from the main thread”. Nothing to worry about, surely …

When I finally tried to build and run my game, however, I found that it either ran with errors, or simply crashed. Checking the build’s output log, I found the same “get_dataPath can only be called from the main thread” error, along with the tip:

“Don’t use this function in the constructor or field initializers, instead move initialization code to the Awake or Start function.”

“get_dataPath” is one of the tools I was using to retrieve the XML file in the Load() function. Unity can handle it no problem in a regular call: from the Start() function, or Update(). But apparently calling it in a constructor or field initializer is dangerous. I’ve heard that this is because Unity isn’t “thread-safe”. So where was get_dataPath being called in an initializer?


// A dictionary of values for each resource type.
[SerializeField]
public Supplies supplies = new Supplies();
Node node;

// The custom SDictionary needs to be extended into a type-specific class to work properly.
// This one uses Goods (trade resources), and values for each of them.
[Serializable]
public class Supplies : SDictionary<Good, float>
{
public Supplies() : base(SGoods.Instance.goods.goods)
{

}
}

… in the Serializable Dictionary’s application, the class that I’d worked so hard to create, and that was essential to a cleaner codebase for Silkroads. The Supplies object – derived from SDictionary – calls for SGoods when defining itself, since it’s a data structure that depends on knowing what the trade Goods are.

How was I doing to fix this once and for all?

The Return of the Centralized Script

After much head-scratching, I took my problem to one of my University’s programming lecturers, and he pointed me back to where I’d started: a centralized “World” monobehaviour script in each Unity scene, one which loaded the data I needed on Awake() or Start(). The problem with this kind of script, in the past, was that it could either manage functioning in edit-mode or on play, but had several issues when doing both (as when I tried to use [ExecuteInEditMode]).

With a simple script that loads any specified data class on Awake() (which I’ve called DataLoader), I can at least be sure that the data I need will be there as soon as possible. However, this doesn’t solve the main problem: the SDictionary-derivative Supplies class needs to know what the Goods are in its field initializer. This requires loading the goods, which requires calling the data path from the loading thread.

Instead of calling for SGoods, and hence asking it to load, on compile, I figured out that all Supplies really needs is the address of SGoods, and not the data itself. The actual data describing each good only needs to be loaded when the game runs, and when I’m performing certain operations in the editor.

This data address essentially already exists, in the form of the SGoods (Singleton-Goods) class. It can load the actual Goods data at any particular time. By giving the SDictionary a TA (Type-Address) generic parameter, I can ensure that it has this address.

public class SDictionary<TK, TV, TA> : Dictionary<TK, TV> where TK : ILabel
 {
// ...
 // The key stock address refers to a class - likely a singleton - referencing the key stock itself.
 // This way, the address of the keys can be given to this class before the keys themselves are loaded.
 protected TA stockAddress;

// Build the dictionary from the lists. Should be called on Start().
 public void CreateDictionary()
 {
 Clear();

// Get the actual keyStock from its address.
 if (stockAddress.data != null)
 {
 keyStock = stockAddress.data;
 }
 else
 {
 Debug.LogWarning("No data found in the data object: " + stockAddress + ". Make sure that the object's data is being loaded in a MonoBehaviour event.");
 }

for (int k = 0; k < keys.Count; k++)
 {
 Add(LabelToKey(keys[k]), values[k]);
 }
 }
 }

However, notice that, in this generic class, I need to specifically be able to reference the data that belongs to the TA. This is easily handled with another implementation:


// Interface for any core data types that, for example, need to be handled by SDictionary.
// Typical examples might be trade goods, weather, terrain, that sort of thing. Having this accessor
// allows this data to be used by SDictionary, and any other generic classes.
public interface IData<TD>
{
List<TD> data
{
get;
}
}


public class SDictionary<TK, TV, TA> : Dictionary<TK, TV> where TK : ILabel where TA : IData<TK>

Implementing this into the SGoods class …


// Singleton for accessing goods.
public class SGoods : IData<Good>
{
// Singleton instance.
private static SGoods instance = null;
public Goods goods; // All goods.

// Constructor.
public SGoods()
{

}

// Singleton accessor.
public static SGoods Instance
{
get
{
if(instance == null)
{
instance = new SGoods();
}
return instance;
}
}

// The list of goods: Silk, Gold, Ivory ... Objects of the Good class.
public List<Good> goodsList
{
get
{
// If the goods have not yet been loaded, load them. However, this should generally be handled by a
// loading MonoBehaviour script to keep the threads safe. The exception is when accessing goods in editor.
if(goods == null)
{
InitData();
}

return goods.goods;
}
}

// This accessor essentially just allows generic classes to access this class's data.
public List<Good> data
{
get
{
return goodsList;
}
}

// Load goods from XML.
public void InitData()
{
goods = Goods.Load();
}
}

So essentially, I’ve given SDictionary the address of the data, rather than the data itself. I’ve told SDictionary to access the actual data (which it can do, since I’ve made data into an IData interface, as I did with ILabel), but only when Start() has been called. I’ve left some of the load-on-access design in the SGoods class, allowing me to access it freely and safely from any code in the main thread (which is useful for retrieving data in edit-mode, for example), but I’m using DataLoader to make sure that the data is there when SDictionary starts up, although this final step is just a precaution.

This architecture worked well as I moved on to add my second data structure: Nations (representing countries and empires in the game). Again, this structure consists of Nation, listed in Nations, referenced in SNations. SNations implements IData, and can be used to retrieve the data safely and on-start by any generic classes.

Europa Universalis IV-Style Province Shader

Hello! After the winter break and a few weeks working on other projects, it’s time for me to return to my honours project; the Silkroads strategy game. And this time, it’s full-swing.

Over the next few weeks, I’ll be preparing the game for playtesting as soon as possible, since this is the cornerstone of my research process. The sooner I can observe players, the sooner I can draw conclusions about the game’s design. However, this term did commence with a bit of technical tinkering which I was determined to finish: province shading.

500px-provinceseuropa-universalis-iv-common-sense-tweaks-map-tech-gold-mines-483531-5

(Paradox’s Europa Universalis IV. Top: The province index-image. Bottom: A screenshot of how provinces look in the game.

Source: http://www.eu4wiki.com/images/thumb/1/13/Provinces.png/500px-Provinces.png
Source: http://i1-news.softpedia-static.com/images/news2/Europa-Universalis-IV-Common-Sense-Tweaks-Map-Tech-Gold-Mines-483531-5.jpg)

As has been mentioned before, Silkroads draws plenty of inspiration from Paradox’s brand of grand-strategy, perhaps most of all from the Renaissance-simulator Europa Universalis IV (EU4). The game is quite accessible for modding, and the wiki provides an introduction to the common modding procedure of creating a custom map. A few months ago, I took a look at this process, and was fascinated to see how Paradox utilise index-images to handle the process.

There are hundreds of provinces in EU4. They’re being edited all the time by the developers, they have to accurately represent the world’s historic regions and be well-designed for trafficking units between provinces without excessive choke-points. No doubt, a lot of iteration and a lot of time is spent on getting these hundreds of 1500s counties correctly shaped. I imagine this is why Paradox are using the index-image. Seen as the topmost picture, this map depicts each province in the game, drawn clearly (with no aliasing) in a unique RGB value. EU4‘s database has a text file for each province, containing a reference to this RGB value. Showing that a province is owned by France or England is a matter of displaying that province in blue or red, so, I have to imagine, the shader handling the way these provinces are drawn must look something like this:

// For each pixel in the texture "provinceIndexImage.png" ...
// What is this pixel's rgb value?
// Look up that value in the province database.
// Draw the pixel in the colour of the country that owns that province.

Again, the benefit of this approach is easy editing. Creating provinces is literally a case of painting them in an image editor. Getting their shape to be roughly accurate, to follow natural borders, to not border too many provinces, is much easier this way.

Recall that Silkroads uses a node-based system that is, essentially, identical to that of EU4:

supplyrates

Does it need provinces? Technically, not yet. Nodes can be selected by clicking on a small box collider, though this is much more awkward, and my users are so far having frustrating with selecting the desired node. You do not play a nation in the game, and you don’t own provinces so much as you have a presence in them (Genoa didn’t own the north coast of Africa, but they did have warehouses there. Venice didn’t have own Constantinople to enjoy trade endorsement: they were granted a whole slice of the city’s economic area). But I do hope to add nations as a mechanic that impacts your trade company, and I do think that provinces would look nice and be more easy to select. So, out of interest, I began fiddling with Unity shaders in the new year.

Once I’d figured out how to check for one colour and turn it into another, the solution seemed pretty easy:

// Note that I'm still new to shaders, and don't fully understand them.
// Unity shaders are written in shaderlab.

// A list of the index-colours for each province.
uniform float4 indexColors[256];

// Corresponding actual colours to be rendered in. This can be set externally.
uniform float4 trueColors[256];

// So, for example: Babylon's index col is r=1, g = 0, b = 0,
// at position [0] in indexColours.
// Once you have that index, get its trueColor.

// This part of the shader is what makes it a 'frag shader'. It
// does stuff one pixel at a time, returning the colour of the pixel.
fixed4 frag (v2f i) : SV_Target
{
// What colour is this pixel?
fixed4 col = tex2D(_MainTex, i.uv);

for(int i = 0; i &amp;lt; indexColors.length; i++)
{
// Once you find the correct index entry, give it its true color.
if(indexColors[i] == col)
{
fixed4 newCol = trueColors[i];
}
}

return newCol;
}

 


The problem here soon became pretty apparent. This isn’t the way to handle shaders. Since a shader is calculated on the GPU, rather than the CPU, asking it to perform checks and comparisons of most kinds is very intensive. It can do it, but even a single if-statement for every pixel, every frame, isn’t what shaders are designed to do. The for-loop above is basically a blunt for-loop search, something of an intensive search for any script, let alone one for every pixel, every frame. Shaders are supposed to be algorithmic: they should take data, manipulate it, and finish. A shader can easily display each colour multiplied by pi, for example.

So I had to figure out a way to take the index colour, and use it as a quick accessor to the true colour, in an algorithmic fashion. Was this possible? Apparently so.

My first thought was pretty simple. An image, to a shader, is basically just a big list of numbers: RGBA values between 0 and 1, representing fractions of 256. If you want, you can multiply the R (red) value of a colour by 255 (first-value-zero) to get its data in a 256-bit representation. The red value of a pixel, in other words, can be used to access a 256-long array. So:

fixed4 frag (v2f i) : SV_Target
{
// What colour is this pixel?
fixed4 col = tex2D(_MainTex, i.uv);

// If the r value was 1 in photoshop, make it 1 here.
int r = col[0] * 255;

// Use that value as the index.
return trueColors[r];
}

Thus: province 0 should be coloured (0, 0, 0). Province 1 should be (1, 0, 0). Province 255 should be (255, 0, 0).

nooffsetred

noOffsetRedUnity.png

Above, you can see the source index-map I used to do this, and the result in Unity. Setting these colors is as simple as using mat.SetVector(“trueColors” + i, new Vector4(whateverColor)); in an external C# script, and giving it a reference to the Material with the province shader. You can’t imagine how happy I was when this worked (I was certain that what I wanted to do was impossible).

So, with that, I could easily have extended the system to use green and blue channels also, giving me 3 x 256 potential provinces, more than I’d ever need (and slightly less than EU4). But I wasn’t done here. I’d set out to make province-drawing easy. However, since each province increased its r value by only 1, and since it’s smart to put provinces 0, 1, 2 etc next to each other, the result is areas that are too similarly coloured to properly distinguish. It can be worked around with the purple outline, as seen above, but it takes more time.

EU4 doesn’t have to deal with this. Provinces are assigned a random color, and can be looked up in a database. As established earlier, I can’t do this. But I don’t really need unique combos of rgb: what I need is a sequence of numbers that aren’t close together to be decodes such that they are close together. Enter the Offset series.

The Offset Series: Swapping Your First and Second Digit

After much pen-chewing, a solution did dawn on me. It was conceivable, even to my poorly-math’d-up brain, to shift all digits in the series of numbers (the series, as in 0, 1, 2, 3 …) into a series of numbers that were, say, 10 digits apart, and back again. It’s a bit like turning a 2D array into a 1D array.

Say we have the series (1, 2, 3 …). This is our indices of provinces in its normal order. Accessing it by an identical red value means that I have to draw my map using that series, as seen above. Now, a conceivable alternative might be:

offsetSeries.png

The first 10 values in the normal series (n-series) become the first values in each division-of-10 up to 99, in the new offset series (0-series). The second set of values – the elevens-and-teens – become the second values in each division of 10. Essentially, this is just swapping the digits around. But it works fine with divisions of not-10 as well.

If you can convert between these two series at will, you can take some well-spaced r-colours on an image file, and correctly use them to access an array in the right order. Province 1 can be represented by r = 10. Province 2 by r = 20. Province 12 by r = 21.

How do you convert between the n-series and o-series?

Where r is the red value in an image, and i is the index of the province we want it to represent:

[What order-of-10s is this o-series number in? Is it in the 20s? The 30s? 30 / 10 = 3, so this becomes the new single-unit.]

i = (r – r % 10) / 10 + …

[What’s the difference between this o-series number and the next multiple of 10? In other words, does it end in 2, 3, 4 … ? Multiply this by 10 as a fraction of the limit you want to stop at.)

… (r % 10) * (100 / 10)

More genrally:

The Offset Function

i = (r – r % m) / m + (r % m) * (l / m)

Where i is the index, r is the value (red), m is the multiple of offset (do you want to offset by 10? 16? 128?), and l is the max number before the sequence loops back and starts sorting the next order of numbers.

Magically, this equation works backwards, too. To get the o-sequence from the n-sequence, simply make r = n and i = 0.

Implementing this into the shader. Note that I went with an offset multiple of 32, allowing 8 offset steps before a reset to a similair colour. It’s uncommon for provinces to border more than 8 other provinces:

fixed4 frag (v2f i) : SV_Target
 {
 // What colour is this pixel in the index-map image?
 fixed4 col = tex2D(_MainTex, i.uv);

 // Take the value (r for red, but it could be red, blue, or green) that's being used for indexing.
 int r = (col[0] + col[1] + col[2]) * 255;

 // Clamp b and g channels to 1 and 2, respectively. Use that to increase the sequence by 256.
 // Note that these numbers should never both be above 0. This is just an algorithmic "or" statement that makes an assumption
 // about how the index-image is being drawn.
 int c = clamp(col[1] * 255, 0, 1) + (clamp(col[2] * 255, 0, 1) * 2);

 // Convert from offset-series to normal-series, add the 256 offset.
 int index = (r - (r % 32)) / 32 + (r % 32) * (256 / 32) + c * 256;

 // Access the array.
 fixed4 newCol = myArray[index];

 // Return the true colour.
 return newCol;
 }

This allows me to define provinces with much greater ease:

32offset.png

32offsetUnity.png

 

Shader source code:

Shader "Unlit/ProvinceTest"
{
// This shader colors provinces according to their index color, into their true color. It should be paired with a C# script,
// editing its array of true colours via Material.SetVector("myArray" + number, Color);
// It was adapted from an example shader, and there are some elements that may no longer serve a purpose.

Properties
{
_MainTex ("Texture", 2D) = "white" {}
}
SubShader
{
Tags{ "Queue" = "Geometry" "RenderType" = "Opaque" }
//Tags{ "Queue" = "Transparent" "IgnoreProjector" = "True" "RenderType" = "Transparent" }
ZWrite Off
Blend SrcAlpha OneMinusSrcAlpha
LOD 100

Pass
{
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
// make fog work
//#pragma multi_compile_fog

#include "UnityCG.cginc"

uniform float4 myArray[767];

struct appdata
{
float4 vertex : POSITION;
float2 uv : TEXCOORD0;
float4 color : COLOR;
};

struct v2f
{
float2 uv : TEXCOORD0;
//UNITY_FOG_COORDS(1)
float4 vertex : SV_POSITION;
//float4 color : COLOR;
};

sampler2D _MainTex;
float4 _MainTex_ST;

sampler2D _IndexTex;
float _Coord;

// Unsure of what this does.
v2f vert (appdata v)
{
v2f o;
o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
o.uv = TRANSFORM_TEX(v.uv, _MainTex);
return o;
}

fixed4 frag (v2f i) : SV_Target
{
// What colour is this pixel in the index-map image?
fixed4 col = tex2D(_MainTex, i.uv);

// Take the value (r for red, but it could be red, blue, or green) that's being used for indexing.
int r = (col[0] + col[1] + col[2]) * 255;

// Clamp b and g channels to 1 and 2, respectively. Use that to increase the sequence by 256.
// Note that these numbers should never both be above 0. This is just an algorithmic "or" statement that makes an assumption
// about how the index-image is being drawn.
int c = clamp(col[1] * 255, 0, 1) + (clamp(col[2] * 255, 0, 1) * 2);

// Convert from offset-series to normal-series, add the 256 offset.
int index = (r - (r % 32)) / 32 + (r % 32) * (256 / 32) + c * 256;

// Access the array.
fixed4 newCol = myArray[index];

// Return the true colour.
return newCol;
}
ENDCG
}
}
}

 

 

Fundamental Map Design, and the Core Verb

Following the completion of my serialized Dictionary, explained in my previous post, it was a quick job to finally assemble all the pieces I’ve been building into something playable (in a basic sense of the word). With a decent node system supporting pathfinding, quick data editing thanks to a custom inspector, and the use of Dictionaries for convenient accessibility of data, accomplishing my key gameplay milestone was a simple case of extending my Caravan class, and creating a simple menu UI piece for trading with nodes.

In previous projects, my first step has always been proving the core gameplay. Before I sink hours into programming a game, I like to make sure it’s actually enjoyable at a fundamental level. With some brainstorming and paper-prototyping, I can investigate the qualities of the system I’m thinking of turning into a project; normally validated when I feel that a prototype has a clear core game loop – the cycle of actions that will form the base of all secondary, or peripheral, gameplay. The prototype doesn’t need to be fun, but I need to be able to sense its capacity for depth, challenge, and extension. I actually did this with Silk Roads, drawing a map of the Middle East, assembling a network of trade nodes, setting local resource supplies, and briefly trying to make a profit by moving a caravan around.

silkroadspaperproto

In some cases, games are so simple, or so slow, that they can be accurately simulated on paper. However, often, and in the case of this strategy game, I’m not able to create a full paper representation without significantly altering the gameplay. For this reason, the challenging aspect of the game idea – the core gameplay – isn’t implemented until at least a few days of programming. Now, many strategy games have a dominant loop of gameplay: in Europa Universalis 4 (EU4), this is starting a war that you know you can win, winning the war, and annexing land to grow in strength. However, it’s very common for strategy games to have several important loops; this way, the player has a choice in their approach to the game, allowing them to strategize. Again, in EU4, countries can be expanded by dynastic inheritance, by offering vassalship, by colonization, and even regular warfare is informed by subsystems such as alliance-building, technology, and economy. As such, I feel that strategy games cannot have a single, simple core gameplay, unlike arcade games and puzzle games. The result of this required complexity is that I’m unable to prove the core gameplay at this stage.

I can, however, prove the core verb.

Core Verbs

Verbs are interactions available to the player: jump, shoot, build, conquer. In the case of Silk Roads, this is buying and selling resources. I can also approach the core gameplay itself, by implementing the simple dynamic (a dynamic is an element of the game’s behaviour) of buying resources where they’re cheap, travelling, and selling them where they’re expensive. Programming this kind of functionality has been my short-term milestone for the game’s preproduction. But, again, since this isn’t a challenging task, and doesn’t invoke meaningful decisions from the player, I don’t consider it to be proper gameplay, or a playable game.

Nonetheless, implementing the verb constitutes an important step, and I can use this kind of interaction as the basis of how I want my game to play. Allowing caravans to trade with markets involved writing some simple functions and basic math, where the value of a good is its base value-per-kilo multiplied by the inverse of its local abundance. Silver, with a value-per-kilo of 5, sells for full price where its supply is 0 (full demand), and half price where its demand is 0.5.

Writing a menu system has mostly been a case of creating reusable UI elements that can be easily constructed in code. These functions will serve to build other menus in the future. They currently include a menu, a text field, and a button.

silkroadsmarketmenu

Map Design

Though the game’s core mechanics still need to be developed properly, I’ve already placed some thought into how I build the map itself. With the idea of exploiting local demands as the core verb, it becomes clear that local demands themselves need to be carefully defined. If, for example, two neighboring cities have completely different supplies; one with high demand for silk, the other with high demand for glass, it’d be easy – and therefore boring – for the player to ping-pong between the cities making enormous profits. A rule of thumb is as follows: the further the player sends their caravan, and the more risk they encounter, the higher the reward they should receive. As such, local supply rates should only change dramatically over long distances. If the player wants to buy a good that’s rare in their home city, they should have to travel far for it. This is basically a high-risk high-reward system. It also follows the geographical reality that, say, silk would become more common as you head closer to China, while spices would gradually grow in abundance as you approach Indonesia.

My implementation of the supply rates – visible as color-coded bar-charts above each node, follows this thinking. I’ve used my historic knowledge to define epicenters of abundance, i.e. places where a good is being produced. Then I’ve gradually had the abundance fall with distance from the city. My classic example is silk (the leftmost crimson bar) coming from China, and glass (the light blue second-left bar) from the Mediterranean.

SilkRoadsGlassAndSilk.png

I’ve also used geography to inform supply spread. The blue lapis lazuli, rich in Afghanistan,  also spreads across the silk roads, while being slightly blocked by the mountains to the South, approaching India.

silkroadslapis

Cinammon and Nutmeg (the rightmost brown bars) spread from India and the Southeast spice islands. Transported by ships, these goods are more common in port-nodes such as Harmozia.

Ebony, a type of African wood, spreads from south Egypt and across Arabia. Silver flows from the Mediterranean. Gold, which I know was very common in West Africa, has been provisionally spread through wealthy trade centers.

When the game reaches a state that produces challenge and strategy, these distributions will have to be fine-tuned by playtesting. It’s important that I avoid the aforementioned situation – vast resource differences being too close together, resulting in a dominant strategy (trade only those resources and win the game). Meanwhile, if a player sends a caravan over a long distance, they should normally be rewarded with a large profit.

Map design also offers other opportunities for meaningful decision – classically, choice of route. There were several major trade routes, historically, some naval, some well-protected and taxed, some dangerous. While most traders might circumnavigate the Himalayas by heading north, taking the Gansu corridor and being received by Chinese officials, the more daring might pass through India and cross the Himalayas, reaching untapped markets more quickly and with less taxation. In short, routes themselves need to be balanced – there should be a reason for taking most routes, be that due to the player’s strategy, or the situation.

Finally, here’s a video of the core verb in action:

With the foundation for Silk Roads in place, I’m ready to focus on researching the design of strategy games.

The Serealizable Dictionary

The Dictionary is a type of data structure included in the .NET framework that works like a list, except that you can search through its elements using a ‘key’ rather than just a number (although you can use that too, with System.Linq’s ElementAt() function). And keys can be anything; objects of one of your own classes, for example, or enum types. As such, I can write:

// Declare the dictionary and add some elements.
Dictionary<Fruit, float> fruitSupplies = new Dictionary<Fruit, float>();
fruitSupplies.Add(apple, 5.0f);
fruitSupplies.Add(banana, 2.0f);

// Find an element by key.
Debug.Log("I have a banana supply of: " + fruitSupplies[banana]);

When I discovered this data structure at the start of my project, its application to a data-heavy grand strategy game seemed obvious. Say I have 200 provinces in the game, and each had an opinion (out of 100) of the other. It’d be compact and elegant if I were able to request, anywhere in my code:

// The lower rome's opinion of a state,
// the less efficient trade between the states.
rome.tradeEfficiency[carthage] = -rome.opinions[carthage] + 100;

Such cases where searching by my own key, be it a resource, a unit type, a map region, will be abundant in this type of game. So, it became clear to me from the outset that I’d be making heavy use of dictionaries.

I assumed this was a blessing on my progress. Here’s a neatly packaged data structure that does exactly what I need, ready for me to use. Maybe this won’t be so hard after all, I chortled. What a fool I was. The crucial caveat of the all-powerful Dictionary, I soon discovered, is that it cannot be serialised. In my previous post, I explained that serializing data means having it set and available before runtime, that ‘to serialize’ is essentially ‘to save’, and that I can either use Unity’s own API, JSON, XML, or any combination of the three, to successfully save and load – serialize and deserialize – my game data. But because Dictionaries are generic classes that can work with any object or data type – at least, I think that’s the reason – they cannot be serialized.

Supply and Demand Values

To introduce the specific problem I started off with, and how it was relevant to this dilemma: in my Silk Roads game, I will have a variety of trade goods, represented by objects of the Good class. Silk, Glass, Silver, Gold, Ivory, Lapiz Lazuli, Cinnamon, and Nutmeg, are the goods I’m using right now. No doubt I’ll add more. Every trading node in the game has a Market, and each Market needs to track the local supply/demand of each resource, currently represented as a number between 0 and 1. So, while the port of Harmozia is rich in Cinnamon from Southeast Asia, the markets of Antioch are more abundant with Roman Glass. Exploiting these local supplies and demands by buying where things are cheap and selling where they’re expensive is the game’s core mechanic, so in short, each market needs a number (a supply rate) for every trade good. Those numbers need to be specified before the game’s runtime, i.e. they need to be serialised.

All was not lost. While Dictionaries technically cannot be serialized, they can be reduced to two Lists on serialization – one for the keys and one for the values – and reconstructed into a Dictionary on deserialization. In fact, this is such a common issue that Unity provide their own mouthfull of a solution – the ISerializationCallbackReciever interface – for which the example of use is creating a serializable Dictionary.

*** I’m going to link to the documentation here because I’m sure it works for some people. But be aware, if you’re reading this in search of help, I’m about to explain why it did not work for me. ***

Interfaces

As mentioned, the ISerializationCallbackReciever is an interface (you can tell because it starts with ‘I’). Now if, like me a few weeks ago, you don’t know what an interface is; it’s basically a packet of functionality that you can tack on (or ‘implement’) to your classes. Say I have a Bomb class, and I want to give it timer functionality, because a bomb very much does contain a timer. I can create an ITimer interface, and implement it on the Bomb class. I can also add it elsewhere: clocks, microwaves, pulsars, and I can implement multiple interfaces on a single class, unlike inheritance. The syntax works like this:

// The interface.
public interface ITimer
{
    int time
    {
        get;
        set;
    }
}

// Interface is applied here with a colon.
public class Bomb : ITimer
{
    // Value to 'implement' or represent the interface's property.
    int iTime = 0;

    public Bomb(int _time)
    {
        iTime = _time;
    }

    // Getter and setter implementation of the property.
    public int time
    {
        get { return iTime; }
        set { iTime = value; }
    }
}

I’m still a little unclear on them, and don’t know why they have to have properties instead of variables, but they work.

Unity’s ISerializationCallbackReciever (I’m just going to call it ISerCallback for now) gives you two ‘callbacks’ (functions that are called automatically from elsewhere, that act like events for your class to respond to, like a Start() or Update() function). These are OnBeforeSerialize(), and OnAfterDeserialize(). As can be imagined, the idea here is to tell your Dictionary to become two lists when OnBeforeSerialize fires, and then to become a Dictionary again when OnAfterDeserialize fires. The basic code for Unity’s example looks like this:

// SDictionary inherits from a regular Dictionary.
// SDictionary takes a 'Type Key' TK and a 'Type Value' TV.
public class SDictionary<TK, TV> : Dictionary<TK, TV>, ISerializationCallbackReceiver
{
    public List<TK> keys = new List<TK>();
    public List<TV> values = new List<TV>();

    public void OnAfterDeserialize()
    {
        // For each key or value, add a key-value-pair to the dictionary.
    }

    public void OnBeforeSerialize()
    {
        // For each dictionary element, add a key and a value to the lists.
    }
}

At first, this seemed like the perfect solution. In fact, by making my keys and values serialize with [SerializeField], doing the same with the SDictitionary class’s instances, and making the class itself [Serializable], I was able to have my supply values appear in the inspector, ready for me to edit. With some fiddling around, I even customised the editor’s appearance to use sliders, and made gizmos to display the supply values as a bar chart.

custominspectorsliders

This type of easy editing is why I was so excited to spend time on this class: it allows me to easily edit the dozens of trade nodes in the game, and makes it easy to read how the supplies for different trade goods are spread out geographically.

Problems With Unity’s Serialization

If that solution was as stable as I’d assumed, I’d have ended the post here, and finished my game’s supply distribution two weeks ago. But I think my commit history following that first SDictionary attempt tells its own story:

silkroadscommits

It’s difficult to pinpoint exactly what the central issue is with this type of serialized Dictionary, because following this first triumph, it was one gruelling problem after another. Values I had set were on several occasions overwritten. References to the different types of goods were constantly lost, broken, or duplicated. From what I can gather after two weeks of firefighting, the ISerCallback interface is not very robust – especially when using in-editor scripts (which is precisely what I’m doing with all this serialization) – for two reasons.

  1. Unity serializes and deserializes very often. In all fairness, the documentation for ISerCallback does mention this. However, I wound up with serialization every frame, while deserialization didn’t seem to fire when it should at all. It was such a complicated mess, falling completely beyond my grasp, that all I can really say is this; you cannot know, at any time, if your SDictionary will be in a serialized state (where you should talk to its Key/Value lists) or its deserialized state (where you should talk to its Dictionary). The amount of NullReferenceExceptions is substantial and nerve-wracking. I could honestly never solve this issue.
  2. Serializing/deserializing breaks the reference value of your key. Values vs. references is a common trope in programming. Suffice it to say, somewhere in my dictionary’s serializing/deserializing, the trade good keys which were supposed to represent references to the central list of goods was broken, such that the key for “Silk” in Jerusalem no longer equalled the Silk in the static Goods class, so I couldn’t ask for Jerusalem’s Silk value anymore. See the diagram.
    silkroadintendedsilkroadincorrect

There may have been even more issues; after two weeks, I gave up. This technique is extraordinarily unstable, in my experience, and I’d advise against it if you’re after a serialized Dictionary that can be interacted with in the editor. I found another way, in some ways less elegant, but certainly more stable.

The Solution

I took a step in the right direction after finally asking myself “why do I even need to make a dictionary before runtime?” The alternative Dictionary seems simple at first: store the dictionary as lists in edit time, and do not consult Unity’s ISerCallback events for when to build the Lists into a dictionary. Only build the Lists into a Dictionary when I want – that is, on Start() or Awake(). There were, however, a few issues I had to solve.

The first is how to have an event fire on the game’s start. “Easy” I hear you cry, “use void Start().” void Start(), I retort, is only available to Monobehaviour classes. My SDictionary is already inheriting from a Dictionary, so it can’t inherit from a Mononehaviour too (this aside, that’d be unwieldy for a simple data structure). It is possible, however, to define your own callbacks – like Start() – using delegates. Normally, that’d be incredibly simple: create a singleton Monobehaviour called “EventManager” or something, give it a delegate, and have all of the SDictionaries ‘subscribe’ to the delegate. But since the SDictionaries will be attempting to subscribe to the delegate event on compile time, and a singleton in Unity seems to be a runtime type of pattern, this is a no-go. Instead, the final structure took three objects, and looked like this:

staticcallbackstructure

A static class stores the delegate subscribers (the CreateDictionary() functions). A Momobehaviour game script calls the static object’s events. It’s a little messy, but it shouldn’t cause any havoc down the line.

The second issue I had to solve was much more difficult. As mentioned earlier, deserializing a dictionary renders the key references broken, so that Silk in Jerusalem no longer refers to Silk in the rest of the game. I had to think of a way to turn goods, which are runtime objects, into keys, and back again, without losing any references. After much toil, I believe I found a sound solution, with the use of interfaces.

Essentially, by assigning every object that can be used as an SDictionary key with a special name, a “label”, that string can be used as the serialized key. Then, when the Dictionary is being built, it can find out what that string refers to by searching a provided ‘stock list’ of master-objects. Thus when Market.supplies enters runtime and builds its dictionary, it takes its list of strings – “Silk”, “Glass”, “Silver” … – consults a stock list that was given to it on creation – the static Goods.goods List – and assigns the appropriate keys. Since this is so mind-boggling, I’ll post the entire source code.

using UnityEngine;
using System;
using System.Collections;
using System.Collections.Generic;
using Trade;

namespace DataUtility
{
    // Interface for labels.
    // Labels are implemented on objects that can be used by the SDictionary as keys.
    // This is because the SDictionary turns object keys (references) into string labels (values) on serialization,
    // and turns them back on deserialization. Objects that need to be used as keys in an SDictionary should
    // implement ILabel, and typically just set the label to their name (e.g. the silk object is labelled "Silk").
    public interface ILabel
    {
        string label
        {
            get;
            set;
        }
    }

    // The custom dictionary class, that can be serialized as two lists, and builds a dictionary on runtime.
    // Note that it requires the keys to use the ILabel interface.
    [Serializable]
    public class SDictionary<TK, TV> : Dictionary<TK, TV> where TK : ILabel
    {
        // Serialized keys (uses key labels as strings).
        [SerializeField]
        public List<string> keys = new List<string>();
        // Serialized values.
        [SerializeField]
        public List<TV> values = new List<TV>();
        // The key stock is the reference used to convert strings back to the proper types on creating the dictionary.
        protected List<TK> keyStock = new List<TK>();

        // Constructor, which requires a specified keyStock.
        public SDictionary(List<TK> _keyStock)
        {
            EventManager.objectStart += CreateDictionary;

            keyStock = _keyStock;
        }

        // Build the dictionary from the lists. Should be called on Start().
        public void CreateDictionary()
        {
            Clear();

            for (int k = 0; k < keys.Count; k++)
            {
                Add(LabelToKey(keys[k]), values[k]);
            }
        }
        
        // Using the keyStock, find the key object which the stored string is supposed to represent.
        private TK LabelToKey(string key)
        {
            TK stockKey = keyStock.Find(k => k.label == key);

            if (stockKey == null)
            {
                throw new Exception("The label: " + key + " could not be found in the stock list: " + keyStock);
            }
            else
            {
                return stockKey;
            }
        }
    }
}

The plan following this design is to implement ILabel to any object that will be used as a key. These objects will be limited, but might include Countries, Languages, Technologies, Units, Currencies, and of course Goods. Then I’ll set up a central ‘stock’ list for these SDictionaries to refer to. The result is safe, robust data storage and retrieval.

Why does this require an interface? You’ll notice that generic classes, such as SDictionary, don’t handle a specific data types. This means, however, that you can’t access the data of your generic types within the class (because an int doesn’t have a Transform, while a Monobehaviour doesn’t have a .ToString()). You can use restraints, with the ‘where’ keyword, to tell your class that it can be sure of some type members being available. As such: “where TK : ILabel” is read as “where Type Key has a label”. There is not, to my knowledge, another way of doing this.

The Fruits of My Labour

Following this long-winded firefight, during which I introduced myself to abstract classes and interfaces, I received my reward: a dictionary that could be serialized. This made it very easy, ultimately, to set up my local supply values across the Middle East, in a matter of minutes:

supplyrates

The SDictionary has already proven useful elsewhere. I’m also using it as the ‘hauls’ variable for my Caravan units, describing how many of what Good is being carried in a Caravan, in kilograms. This structure, and similair structures, will prove invaluable as I build a complex system of interrelated values, allowing me to easily search by key through any “x for each y” collections. Hopefully, the time I’ve spent will be easily redeemed.