The Magic Payload Pattern

Introduction

This article introduces a pattern/concept that will assist with the storage and transfer of objects and data that don't perhaps need or deserve a full formal representation in your data repository or data/object transfer framework. I've used it in a plethora of projects and it is a useful trick to have, up your sleeve. This is a light read, nothing too taxing, in fact, perfect for your 11 O'Clock break!

Background

I'm sure there is a more formal or better known name for the pattern I am describing - I just haven't heard it... if you know it, please let me know in the comments! There is a lot more that you could do with the concepts discussed here, especially in relation to more efficient operation for example using reflection, however, I'll leave that up to you to implement. This article is about the concept :)

The pattern and variants of it are used frequently under the hood in the construction and management of NoSQL databases. In projects where you have data you need to store, or models and objects you need to transfer around, a common approach is to create a fixed structure to handle things. While this is all good, the problem for very dynamic applications, is that you can very quickly end up with a lot of weight of code to manage for very little data. The 'payload' pattern helps lighten the load.

Concept - flag & serialise

A common approach to storing structured data is to mirror a database record to a class structure type, or the other way around. This is fine for objects and entities where you will have large volumes of data, but bothersome and potentially quite wasteful for small handfuls of data storage, objects of which you could have many. An example of these kinds of things would be config type settings. I'll bet most of us have seen (or written!) systems where there were a pile of tables that were very lightly used and were in place primarily to store small bits of data that were seldom accessed or changed.

The concept works by doing away with multiple storage or access points, and instead, using object serialization and deserialization to form our objects into tidy neat packages that we can store and retrieve on demand.

The underlying pattern is really basic but very powerful - here's how it works.

When we want to store an object, we do two things. First, we get the string name of the object (this is our 'flag'), and second, we serialise the object into a string.
When we want to deserialise the object, we use the flag (string name) of the object to help us unwrap the object and cast it back to its original type.

Let's look at it in action.

Using the code

To demonstrate how it works, I have written a small console application. On run, it creates a test harness and populates it with sample data of different class types (remember, we want one single repository or access point for our data), it serialises the data, then deserialises it back out to prove everything remains intact.

The overall workflow..

namespace TestApp
{
class Program
{
static void Main(string[] args)
{
var Test = new TestHarness();
Test.Setup();
Test.ShowStorage();
Test.TearDown();
var inStr = Console.ReadKey();
}
}
}

Our test harness consists of two test classes that we will populate with sample data.

public class Person
{
public string PersonName { get; set; }
public int Age { get; set; }
public string Location { get; set; }
}
public class Country
{
public string CountryName { get; set; }
}

We, then, have a 'Storage' class that we use to 'flag and serialise' into. The 'type' field stores the type of the class (Person or Country), the 'payload' field stores the serialised object. How you serialise is your design choice. For this example, I have used JSON.

public class Storage
{
public string Type { get; set; }
public string Payload { get; set; }
}

In the test harness, we have a LIST of Storage into which we are going to populate our test objects.

public class TestHarness
{
public List<storage> Storage { get; set; }
public TestHarness()
{
Storage = new List<storage>();
}
...etc...
}

We also have one other core method which is used to take in an object, and return a Storage type, ready for serialisation as a collection. 'WrapPayload' uses NewtonSoft JsonConvert to Serialize the incoming object, take the JSON string representation of the object, and store it in the string field of the Storage object called 'Payload'.

public Storage WrapPayload(Object obj)
{
var rslt = new Storage();
rslt.Payload = JsonConvert.SerializeObject(obj);
rslt.Type = obj.GetType().Name;
return rslt;
}

Having setup the structures, we now put in some data to check everything works. For this, I have a method called Setup. First, it adds the data, then it wraps it in our Storage object, then it writes to the console what it has added.

public void Setup()
{
// setup test countries
var C1 = new Country();
C1.CountryName = "US";
var C2 = new Country();
C2.CountryName = "UK";
// setup test persons
var Person1 = new Person();
Person1.PersonName = "John";
Person1.Location = "Mumbai";
Person1.Age = 15;
var Person2 = new Person();
Person2.PersonName = "Mary";
Person2.Location = "Belfast";
Person2.Age = 47;
// serialise and wrap in payload storage
Storage.Add(WrapPayload(Person1));
Console.WriteLine("In: " + Person1.PersonName);
Storage.Add(WrapPayload(Person2));
Console.WriteLine("In: " + Person2.PersonName);
Storage.Add(WrapPayload(C1));
Console.WriteLine("In: " + C1.CountryName);
Storage.Add(WrapPayload(C2));
Console.WriteLine("In: " + C2.CountryName);
}

To confirm everything has serialised correctly, I, then, iterate through the storage collection and write it to the console as well.

public void ShowStorage()
{
foreach (var itm in Storage)
{
Console.WriteLine(string.Format("Type identifier: {0} Stored serialised payload: {1}", itm.Type, itm.Payload));
}
}

Finally, I have a TearDown() method that loops through the Storage and deserializes everything back out again to its original state.

public void TearDown()
{
foreach (var itm in Storage)
{
var newObj = new Object();
switch (itm.Type.ToLower())
{
case "person":
{
var PersonObject = JsonConvert.DeserializeObject<person>(itm.Payload);
Console.WriteLine("out: " + PersonObject.PersonName);
break;
}
case "country":
{
var CountryObject = JsonConvert.DeserializeObject<country>(itm.Payload);
Console.WriteLine("out: " + CountryObject.CountryName);
break;
}
}
}
}

Where is it useful?

So, where is this useful? ...well, for example, I have used this pattern to great effect in storing dynamic layouts for UI (web) forms that were constructed by users in a web browser, and then, dynamically displayed and used to collect data in a VS Cordova / TACO app. In addition to storing the metadata on the Server, I used the same concept to, then, construct UIs on demand in cross platform mobile apps. As the 'web forms' were simply serialized objects describing screen layour/field properties etc., it worked extremely well to wrap up a 'payload' in JSON format and zip this from Server to App as required.

I have also used this pattern to store a multitude of small collections of data in Azure table storage, where I didn't want to commit to really having any kind of formal structure. Anyway, it's been useful for me and I hope it finds a home in your toolkit too.

Big final note! .. As I used JSON to wrap/unwrap the payload, I was able to use it to convert data and objects between JavaScript on the front-end and C# on the back-end - very useful in a cloud/mobile orientated world :D

OK THEN, BREAK OVER - GET CODING! :D

1 Comment

Join the conversation! Your thoughts help the community grow.

Tom VS9y ago
Definitely can have its use, but it does feel extremely hacky to me, especially due to the switch on the type in the TearDown method. You really have to know every possible type there, which makes it something you have to carefully manage and maintain, which is exactly what you seem to want to avoid. That kind of ruins the whole use of the pattern, since you say you wish to use it where a "full formal representation" of the data is not present.