Post
Ask Question

Real World Cloud App From Start To Finish - The Data Layer

In the second article in this series, I laid out the architecture and design for this cloud solution. Now it’s finally time to get coding! In this article, I will discuss how I designed the data layer using Cosmos DB. To keep a separation of concerns, I will talk about the business entities which is part of the business layer.

Ever since Entity Framework was released I put the entities and classes to access the data in separate projects. The reason I do this is so that they can be easily reused by any assembly in the solution or other solutions. This way, client apps don’t need the data layer (nor should they) since all data access should go through the communication layer (discussed in a future article).

Read World Cloud App From Start To Finish - The Data Layer

This is the first time I have used Cosmos DB or any other document database framework. Let’s get going.

If you haven’t used Cosmos DB before, this is the description of it from Microsoft:

Azure Cosmos DB is Microsoft's globally distributed, multi-model database. With the click of a button, Azure Cosmos DB enables you to elastically and independently scale throughput and storage across any number of Azure's geographic regions.

Setting Up Cosmos DB

Creating a Cosmos DB is a lot easier than creating one in SQL Server and all of it can be done via the Azure Portal as you will see. This can also be done via code too.

Creating the Database

To get started, go to Create a Resource in the Azure portal and search for Cosmos DB and then click Create. You will be presented with the setup info as seen to the right.

For my database, I selected the following values,

  1. ID = addata: The ad document will be the main document for this database, so I just added “data” to the end.
  2. API = SQL: Cosmos DB also supports MongoDB, Cassandra, Azure Table and Gremlin (graph).
  3. Subscription: Choose the Azure subscription you would like to use.
  4. Resource Group: Here you can select an existing group or create a new one. I created a new resource group for this project called DotNetTipsAds. I will put all the resources for this project under this group.
  5. Location: I chose West US since that is the side of the country where I live.
  6. Enable geo-redundancy = true: This will allow faster access to different Azure regions.

Read World Cloud App From Start To Finish - The Data Layer

Ensure you have something else to do when selecting Create, it could take a while. 
 
Creating a Collection

Documents need to be added to a collection and that is easy to do too. It can be done via code, but I will show how to do it right from the portal.

Once the database is created, go to its summary and click on Add Collection. You will be presented with the setup info screen as shown to the right.

I added the following values,

  1. Database id = Ads
  2. Collection id = ads
  3. Storage capacity = Unlimited
  4. Partition key = /app: This is the way I am partitioning the documents based on the application id. Currently, I only have one but could add more in the future.
  5. Throughput = 1000: I sent this low since I don’t expect a lot of hits per second. The more throughput, the higher the cost, so, for now, I want to keep it as cheap as possible.

Read World Cloud App From Start To Finish - The Data Layer

After selecting Ok, the collection will be created and it’s time to start adding documents. I also added an adclicks collection with a partition key of /isoRegion.

Documents can be added and edited right from the portal, but we will do it via code coming up. But first, let’s talk about the entities.

Business Entities

The data that I need to be stored for this project is simple and currently, I have identified two main entities. All of these entities are in the project dotNetTips.App.Ads.Entities in source control.

Base Entity

Since I dislike code duplication, I created a base entity for Ad and AdClick to reuse the Id property. I can easily add more later. This is what it looks like. I’d like to point out that Cosmos DB always adds an Id property to every document. Id is typed as a string, but the value is a GUID.

  1. public class Entity  
  2. {  
  3.     [JsonProperty(PropertyName = "id", Order = 1)]  
  4.     public string Id {get; set;}  
  5. }  

Since the naming standards for JSON is different from .NET I am using the JsonProperty attribute to fix that.

Ad Entity

This is the main entity that stores the info for an ad. Below is the code for it.

  1. public class Ad: Entity  
  2. {  
  3.     [JsonProperty(PropertyName = "app", Required = Required.Always)]  
  4.     public App App {get; set;}  
  5.   
  6.     [JsonProperty(PropertyName = "image", Required = Required.Always)]  
  7.     public byte[] Image {get; set;}  
  8.   
  9.     [JsonProperty(PropertyName = "isoLanguage", Required = Required.Always)]  
  10.     public string ISOLanguage {get; set;}  
  11.   
  12.     [JsonProperty(PropertyName = "link", Required = Required.Always)]  
  13.     public string Link {get; set;}  
  14.   
  15.     [JsonProperty(PropertyName = "message", Required = Required.Always)]  
  16.     public string Message {get; set;}  
  17.   
  18.     [JsonProperty(PropertyName = "schedule", Required = Required.AllowNull)]  
  19.     public Schedule Schedule {get; set;}  
  20.   
  21.     [JsonProperty(PropertyName = "title", Required = Required.Always)]  
  22.     public string Title {get; set;}  
  23. }  

Here is the explanation of each property.

  1. App – Is the id of the app. Since currently I only have one app the value is 1 (since it comes from the App enum. Furthermore, an app id of 0 means that it applies to all apps.
  2. Image – This stores the image to be displayed in the app. I could have stored the image in an Azure blob and just link to it via a URL, but I decided not to. Reason being is that when I request available ads from the app, I want it to come down to it as one data blob. Chucky, not chatty is better for performance. I will need to write code later to ensure that any image used does not put the overall document size over 2MB.
  3. ISOLanguage – This will store the three letter ISO language that ad is for. Currently, all the ads will be in English, but that could change in the future.
  4. Link – This is the URL that the user will click on to find out more information about the ad.
  5. Message – This will be the message that will be displayed to the user for them to click on.
  6. Schedule – This is a class that holds the start and end dates of a scheduled ad. This is not required.
  7. Title – This store the title for the ad that will be displayed to view and edit ads in the management web site.

I might add more properties later, which is easy to do with Cosmos DB, but for now, this is all I think I need.

When the Ad document is inserted into Cosmos DB, this is what it looks like,

  1. {  
  2.     "app": 1,  
  3.     "image": "iVBORw0KGgoAAAANSUhEUgAAAUUAAAFFCAYAAAB7dP9dAAAAAIArs4c",  
  4.     "isoLanguage": "eng",  
  5.     "link": "http://bit.ly/dotnetdaverocks",  
  6.     "message": "Check out one of my conference sessions live & in person!",  
  7.     "schedule": null,  
  8.     "title": "Rock the World Ad",  
  9.     "id": "4baca6f3-8446-4729-b587-5e015dpmf12d",  
  10.     "_rid": "c-J6ALsystrongMIAAAAAA==",  
  11.     "_self": "dbs/c-J6AA==/colls/c-J6ALsystrongM=/docs/c-J6ALsystrongMIAAAAAA==/",  
  12.     "_etag": "\"0000d703-1100-0011-0000-5bb158660000\"",  
  13.     "_attachments": "attachments/",  
  14.     "_ts": 1538300158  
  15. }  

Id is automatically added by Cosmos DB along with the data properties that starts with an underscore.

AdClick Entity

Each time a user clicks on an ad, that event will be sent to Cosmos DB, via a Queue (this will be discussed in a future article). This entity properties are,

  1. public class AdClick  
  2. {  
  3.     [JsonProperty(PropertyName = "adId", Required = Required.Always)]  
  4.     public string AdId {get; set;}  
  5.   
  6.     [JsonProperty(PropertyName = "clickedOn", Required = Required.Always)]  
  7.     public DateTimeOffset ClickedOn {get; set;}  
  8.   
  9.     [JsonProperty(PropertyName = "isoLanguage", Required = Required.Always)]  
  10.     public string ISOLanguage {get; set;}  
  11.   
  12.     [JsonProperty(PropertyName = "isoRegion", Required = Required.Always)]  
  13.     public string ISORegion {get; set;}  
  14. }  

Here is what the JSON document looks like for this entity,

  1. {  
  2.     "adId": "4baca6f3-8446-4729-b587-5e015dpmf12d ",  
  3.     "clickedOn": "2018-09-30T23:52:42.1039781+00:00",  
  4.     "isoLanguage": "eng",  
  5.     "isoRegion": "USA",  
  6.     "id": "c6c13a3c-f185-4486-a17f-6f73fdpm13a1",  
  7.     "_rid": "c-J6AJXuBbUBAAAAAA==",  
  8.     "_self": "dbs/c-J6AA==/colls/c-J6AJXuBbU=/docs/c-J6AJXuBbUBAAAAAA==/",  
  9.     "_etag": "\"9d00825b-1100-0011-0000-5bb161ca0000\"",  
  10.     "_attachments": "attachments/",  
  11.     "_ts": 1530051562  
  12. }  
Data Layer

Next, I will describe how I coded the data layer that can be found in the dotNetTips.App.Ads.DataAccess project. Before I do, I will again state that, for many reasons, I try to avoid writing duplicate code and this layer is no different. In many of my conference sessions I state that 90% of all the code you write should be in reusable DLL’s. Once this project is completed, I have plans to move some of the base classes I’m writing for this solution into my .NET Standard open-source projects. I’m waiting until the end due to this since it’s my first Cosmos DB project.

DataContext<T>

This is the base class for the other data context classes that I will discuss next. I will go over the major methods of this class.

Constructor

The parameters in this constructor as in most I write is the data that the class needs to operate correctly. The description of the parameters is below,

  1. endpointUrl - This is the location of the database in Azure. Example: https://addata.documents.azure.com:443/
  2. databaseId - This is the unique id of the database. Example: Ads
  3. masterKey - This is the security key need to connect to the database. Example: NjpQtPF7s8zj9JAc2dpMrdVUqjAdpmialuCCu1vhw==

The constructor code is as follows,

  1. protected DataContext(string endpointUrl, string databaseId,   
  2.                       string masterKey)  
  3. {  
  4.     _databaseId = databaseId;  
  5.     _masterKey = masterKey;  
  6.   
  7.     _databaseUri = UriFactory.CreateDatabaseUri(_databaseId);  
  8.   
  9.     this._client = new DocumentClient(new Uri(endpointUrl), _masterKey);  
  10.   
  11.     this._client.CreateDatabaseIfNotExistsAsync(new Database   
  12.       { Id = _databaseId });  
  13.   
  14.     this._client.CreateDocumentCollectionIfNotExistsAsync(  
  15.       UriFactory.CreateDatabaseUri(_databaseId),   
  16.       new DocumentCollection { Id = CollectionId });  
  17.   
  18.     _databaseCollectionUri = UriFactory.CreateDocumentCollectionUri(  
  19.       _databaseId, CollectionId);  
  20. }  

This constructor stores all the data it needs to run along with creating the database and collection if they don’t already exist. If these are created in the constructor, be sure to go back to the portal if the default settings are not ideal.

UpsertDocumentAsync

In Cosmos DB, you don’t insert or update a document, that is done in just one method called Upsert.

  1. public async Task<Document> UpsertDocumentAsync(T entity)  
  2. {  
  3.     var result = await this._client.UpsertDocumentAsync(  
  4.     _databaseCollectionUri, entity);  
  5.     return result;  
  6. }  
RetrieveDocument

Retrieving a single document is easy too and looks like this,

  1. public T RetrieveDocument(string id)  
  2. {  
  3.     var options = new FeedOptions {EnableCrossPartitionQuery = true,   
  4.                                    MaxItemCount = 1};  
  5.     var result = _client.CreateDocumentQuery<T>(_databaseCollectionUri,  
  6.                    options).AsEnumerable().FirstOrDefault(p => p.Id == id);  
  7.     return result;  
  8. }  

When I was coding UpsertDocumentAsync, since I have been using LINQ since it came out, I coded it like this:

  1. var result = _client.CreateDocumentQuery<T>(_databaseCollectionUri,  
  2. options).FirstOrDefault(p => p.Id == id);  

But the line above would not work until I added .AsEnumerable() before FirstOrDefault(). Also, in this method, I am setting up options to enable cross partition query. I don’t plan on doing these types of queries, but this future proofs it in case I do.

AdsDataContext

Each collection in this design will have its own DataContext class. To do that, all I have to do is inherit DataContext and then set the CollectionId as shown below.

  1. public class AdsDataContext: DataContext<Ad>  
  2. {  
  3.    public AdsDataContext(string endpointUrl, string databaseId,   
  4.                          string masterKey) : base(endpointUrl,   
  5.                          databaseId, masterKey)  
  6.    { }  
  7.   
  8.    public override string CollectionId => "ads";  
  9. }  

From here I can easily add custom methods to retrieve and save data. For now, this is all I need to get my unit tests working. To upsert a document, all I need to do is create the Ad object and send it like this,

Document result = _adsDbContext.UpsertDocumentAsync(ad).Result;

To retrieve a document, all I need is its id and call the method below,

Ad doc = _adsDbContext.RetrieveDocument("4bdpm6f3-8446-4729-b587-5e015f8dpm2d");

Summary

That’s just about all for this article. You can find the source code by going here: http://bit.ly/RealWorldAppSource. Check back often to see how it's changing as I work on the rest of the layers and make improvements.

Cosmos DB tooling in Visual Studio has a long way to go to make it as easy to use as Entity Framework. I’ll keep my fingers crossed on that one.

Before You Start Using Cosmos DB

If you come from a relational database as I do, I highly recommend getting some training first. Relational databases and document databases really don’t have much in common and you will have a learning curve. I have done some training and read a lot to get this article completed, but I still have a lot of learning to do.

Since the learning curve for myself was high, I almost gave up and went back to SQL Server with Entity Framework (since I can do that quick). But I really wanted to learn Cosmos DB so I kept going with the help of some Microsoft Azure MVP’s.

Since Cosmos DB is so new, I would not recommend purchasing books on it since it still has a lot of changes going on. It changed quite a bit two weeks ago after announcements at MS Ignite and so far, most blog posts are out of date.

Changes

Here are some of the things I have changed since I released the first two articles:

  • Since Cosmos DB currently has a document size limit of 2MB, I had to move the ad clicks into their own collection. I hope they change this limit in the future.

I would like to thank Azure MVP’s Sam Cogan and Vishwas Lele for helping me with this article.

Resources Next Article

In the next article, I will be coding the business layer that will include using Azure queues, so check back here soon.


Similar Articles
McCarter Consulting
Software architecture, code & app performance, code quality, Microsoft .NET & mentoring. Available!