Post
Ask Question

Parser API Using .Net Core Web API 2.1 And Hosting In Raspberry PI

Introduction

This article explains how to create a generic parser API to parse video feeds from various sites that support RSS/Atom feed or scrape the content from DOM element and return the custom XML / JSON output based on the requested streaming box-supported format. The core parsing logic will be implemented in the base class and then custom formatting logic will be implemented in the appropriate child classes. Later in the article, we will also deploy this application into Raspian-based Raspberry Pi. We are using the latest .NET Core SDK 2.1 RC release for developing the web API. In my earlier article, I developed a similar application using NodeJS and Express and deployed it in Raspberry Pi.

Design

The below diagram depicts the architecture of Model and Service Components. In this article, I have only added support for feed-based service and I will be adding support to DOM-based extraction later and publishing it into GitHub.

Model Class Diagram

 

IParserModel

Interface for the model.

  1. public interface IParserModel  
  2.  {  
  3.      string RawContent { get; set; }         
  4.  }  

BaseParserModel 

Base class for the Parser Mode. RawContent will hold the Raw content of feed or DOM.

  1. public class BaseParserModel : IParserModel  
  2.     {  
  3.         [XmlIgnore]  
  4.         public string RawContent { get; set; }  
  5.     }  

FeedBaseParserModel 

The base class for RSS / Atom Feed based Parser Mode. This holds the SyndicationItems as a property to hold all the Feed Items from SyndicationFeed. For DOM based scraping, we will be using another class called ContentBaseParserModel.

  1. public class FeedBaseParserModel : BaseParserModel  
  2.     {  
  3.         [XmlIgnore]  
  4.         public List<ISyndicationItem> SyndicationItems { get; set; }  
  5.     }  

RokuParserModel 

This is the root class for Roku which holds all the properties and subclasses that are expected for Roku Streaming Box. I will be adding AndroidParserModel and IOSParserModel in the future to support other streaming boxes. The RokuParserModel also has sub classes to hold other properties such as video URL and ThumbnailURL. This can be customized in whatever way our streaming box expects the model to be.

  1. [XmlRoot(ElementName ="Feed")]  
  2.     public class RokuFeedParserModel : FeedBaseParserModel  
  3.     {          
  4.         [XmlElement]  
  5.         public int ResultLength { get; set; }  
  6.         [XmlElement]  
  7.         public int EndIndex { get; set; }        
  8.         [XmlElement(ElementName ="Item")]  
  9.         public List<RokuParserItem> ParserItems { get; set; }  
  10.     }  
  11.   
  12.     [XmlRoot(ElementName = "Item")]  
  13.     public class RokuParserItem  
  14.     {  
  15.         [XmlElement]  
  16.         public string Title { get; set; }  
  17.         [XmlElement]  
  18.         public int ContentId { get; set; }  
  19.         [XmlElement]  
  20.         public string StreamFormat { get; set; }  
  21.         [XmlElement]  
  22.         public string Synopsis { get; set; }  
  23.         [XmlAttribute]  
  24.         public string SdImg { get; set; }  
  25.         [XmlAttribute]  
  26.         public string HdImg { get; set; }  
  27.         [XmlAttribute]  
  28.         public string ThumbnailURL { get; set; }  
  29.         [XmlElement]  
  30.         public RokuMediaItem MediaItem { get; set; }  
  31.     }  
  32.   
  33.     [XmlRoot(ElementName = "MediaItem")]  
  34.     public class RokuMediaItem  
  35.     {  
  36.         [XmlElement]  
  37.         public string StreamUrl { get; set; }  
  38.         [XmlElement]  
  39.         public string ThumbnailURL { get; set; }  
  40.     }  

Service Architecture

 

IParserService 

Base Interface for Service

  1. public interface IParserService<T> where T : IParserModel  
  2.    {  
  3.        Task<T> ParseContent();  
  4.    }   

BaseParserSerivce<T>

Abstract Layer for Paser Service Class. This will have empty virtual method for now.

  1. public abstract class BaseParserService<T> : IParserService<T> where T : IParserModel, new()  
  2.   {         
  3.       public async virtual Task<T> ParseContent()  
  4.       {  
  5.           return await Task.FromResult(new T());  
  6.       }  
  7.   }  
RokuFeedParserService 

This is the service class for the Roku Format that holds all the core logic and extracts the content items from the feed. It takes the feedURL in constructor and overrides the parseContent method to implement Feed Based Parsing Service. I used the SyndicationFeed library from .net to parse RSS and atom feed. The base RokuFeedParserService will parse the feed and populate the list of items in SyndicationItems Property. Later, the child class that will be inhertiing from RokuFeedParserService will use the SyndicationItems values to create custom format for the xml / json output based on the streaming format requested.
  1. public class RokuFeedParserService : BaseParserService<RokuFeedParserModel>  
  2.     {  
  3.         public string FeedURL { get; set; }  
  4.   
  5.         public RokuFeedParserService(string _feedURL)  
  6.         {  
  7.             FeedURL = _feedURL;  
  8.         }  
  9.   
  10.         public async override Task<RokuFeedParserModel> ParseContent()  
  11.         {  
  12.             RokuFeedParserModel parserModel = new RokuFeedParserModel() { SyndicationItems = new List<ISyndicationItem>() };  
  13.             using (XmlReader xmlReader = XmlReader.Create(FeedURL, new XmlReaderSettings() { Async = true }))  
  14.             {  
  15.                 var reader = new RssFeedReader(xmlReader);  
  16.                 while (await reader.Read())  
  17.                 {  
  18.                     switch (reader.ElementType)  
  19.                     {  
  20.                         case SyndicationElementType.Item:  
  21.                             parserModel.SyndicationItems.Add(await reader.ReadItem());  
  22.                             break;  
  23.                     }  
  24.                 }  
  25.             }  
  26.             return parserModel;  
  27.         }  
  28.     }  

Ch9RokuParserService 

This is the child service class for the Channel9 feed and it will override the parseContent method to populate RokuFeedParserModel object based on SyndicateItems values. This will return the final output of the Roku based parser model object. We will be adding additional service classes for supporting other formats here.

  1. public class Ch9RokuParserService : RokuFeedParserService  
  2.     {  
  3.          
  4.         public Ch9RokuParserService(string _feedURL) : base(_feedURL)  
  5.         {  
  6.              
  7.         }  
  8.   
  9.         public async override Task<RokuFeedParserModel> ParseContent()  
  10.         {  
  11.             var parserModel = await base.ParseContent();  
  12.             parserModel.ParserItems = new List<RokuParserItem>();  
  13.             int currIndex = 0;  
  14.             foreach(var syndicationItem in parserModel.SyndicationItems)  
  15.             {  
  16.                 RokuParserItem parserItem = new RokuParserItem();  
  17.                 parserItem.Title = syndicationItem.Title;  
  18.                 parserItem.ContentId = currIndex;  
  19.                 parserItem.StreamFormat = "mp4";  
  20.                 parserItem.MediaItem = new RokuMediaItem();  
  21.                 parserItem.MediaItem.StreamUrl = syndicationItem.Links.FirstOrDefault(i => i.RelationshipType == "enclosure")?.Uri.ToString();  
  22.                 parserModel.ParserItems.Add(parserItem);  
  23.                 currIndex++;  
  24.             }  
  25.             parserModel.ResultLength = currIndex;  
  26.             parserModel.EndIndex = currIndex;  
  27.             return parserModel;  
  28.         }  
  29.   
  30.     }  

Controllers

BaseAPIController 

This is the base API controller and it will have the default annotation attributes for APIController and Route Actions. Note that I am using APIController Attribute that denotes a Web API controller class and it provides some useful methods and properties by coupling with ControllerBase method such as automatic 400 responses and more. I have also defined the default route [Route("api/[controller]")] at base class level so that I dont have to redefine this on every other controller.

  1. [ApiController]  
  2.     [Route("api/[controller]")]  
  3.     public class BaseAPIController : ControllerBase  
  4.     {          
  5.         public BaseAPIController()  
  6.         {  
  7.   
  8.         }  
  9.     }  

Ch9Controller

This controller will have all the GET methods for various streaming boxes and produces the xml or json output. Note that, I am using HttpGet(“Roku”) so that it allows me to have multiple GET methods on a single controller. You can also define your Routing by action like [HttpGet("[action]”)] and then you can call the API with the method name like /API/Ch9/GetRokuFormat

  1. public class Ch9Controller : BaseAPIController  
  2.    {         
  3.        [HttpGet("Roku")]   
  4.        [Produces("application/xml")]  
  5.        public RokuFeedParserModel GetRokuFormat()  
  6.        {             
  7.            var parserService = new Ch9RokuParserService("https://s.ch9.ms/Feeds/RSS");              
  8.            return parserService.ParseContent().Result;     
  9.        }  
  10.    } 

Startup.cs 

In my startup class, I have enabled both XML and JSON formatter to support both formats based on the request. You can also create custom formatter.

If you need anything other than XML / Json note that I  have enabled RespectBrowserAcceptHeader = true to support the XML output. Also, I have used the XML Annotation to change the element name and added the XMLIgnore Attribute to ignore from serialization.

  1. public class Startup  
  2.     {  
  3.         public Startup(IConfiguration configuration)  
  4.         {  
  5.             Configuration = configuration;  
  6.         }  
  7.   
  8.         public IConfiguration Configuration { get; }  
  9.   
  10.         // This method gets called by the runtime. Use this method to add services to the container.  
  11.         public void ConfigureServices(IServiceCollection services)  
  12.         {  
  13.             services.AddMvc(options =>  
  14.             {  
  15.                 options.RespectBrowserAcceptHeader = true;                  
  16.             })  
  17.             //support application/xml  
  18.             .AddXmlSerializerFormatters()  
  19.             //support application/json  
  20.             .AddJsonOptions(options =>  
  21.             {  
  22.                 // Force Camel Case to JSON  
  23.                 options.SerializerSettings.ContractResolver = new CamelCasePropertyNamesContractResolver();  
  24.             });  
  25.         }  
  26.   
  27.         // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.  
  28.         public void Configure(IApplicationBuilder app, IHostingEnvironment env)  
  29.         {  
  30.             if (env.IsDevelopment())  
  31.             {  
  32.                 app.UseDeveloperExceptionPage();  
  33.             }  
  34.   
  35.             app.UseMvcWithDefaultRoute();  
  36.         }  
  37.     }  

Program.cs 

I used the default settings to start the kestrel web server and configured to use the 5000 port for hosted applications to listen on that port. This will be used when we deploy the application Raspberry PI later.

  1. public class Program  
  2.    {  
  3.        public static void Main(string[] args)  
  4.        {  
  5.            BuildWebHost(args).Run();  
  6.        }  
  7.   
  8.        public static IWebHost BuildWebHost(string[] args) =>  
  9.            WebHost.CreateDefaultBuilder(args)  
  10.                .UseStartup<Startup>()  
  11.                .UseUrls("http://*:5000")  
  12.                .Build();  
  13.    }  

Now that we have done the coding for our channel9 feed Web API to support Roku XML format, let us run the application first using IIS express to make sure it works.

 

Deploying .Net Core Web API on Raspberry PI

Now that our application produced the xml output as expected, we will deploy this latest .Net Core 2.1 Web API in our Raspberry PI. Please note that .Net Core runs only on Raspberry PI 2 / 3. It does not run on Pi Zero. The Raspberry PI that I have is currently running on Raspian OS.

Before we deploy our application, as a first step, we have to install the .Net Core SDK and Runtime on Raspberry PI. In order to install the SDK, we will be executing the below commands on PI terminal window. I already have remote connection enabled from my laptop for my PI. I have also enabled network share from my PI so that I can publish the code later using windows file share. If you want to know how to enable to remote connection and file sharing for your Raspberry PI , visit the Dave J article about Beginner’s Guide to Installing Node.js on a Raspberry Pi and he explained all the steps in details.

Launch the remote connection and connect to the PI Server. Launch the terminal window and run the following commands.

  1. $ sudo apt-get -y update    
  2. $ sudo apt-get -y install libunwind8 gettext    
  3. $ wget https://dotnetcli.blob.core.windows.net/dotnet/Sdk/2.1.300-rc1-008673/dotnet-sdk-2.1.300-rc1-008673-linux-arm.tar.gz   
  4. $ wget https://dotnetcli.blob.core.windows.net/dotnet/aspnetcore/Runtime/2.1.0-rc1-final/aspnetcore-runtime-2.1.0-rc1-final-linux-arm.tar.gz    
  5. $ sudo mkdir /opt/dotnet    
  6. $ sudo tar -xvf dotnet-sdk-2.1.300-rc1-008673-linux-arm.tar.gz -C /opt/dotnet/    
  7. $ sudo tar -xvf aspnetcore-runtime-2.1.0-rc1-final-linux-arm.tar.gz -C /opt/dotnet    
  8. $ sudo ln -s /opt/dotnet/dotnet /usr/local/bin    

The first two commands are required for Raspbian for deploying .Net Core SDK and Runtime. These are some dependency modules that have to be added manually. For more details, you can check the official documentation here.

The next two WGET commands will download the latest DotNet SDK and Runtime (2.1 RC1) and then the following commands will be used to extract the output to /opt/dotnet folder and a symbolic link is created for dotnet.

If all of the above steps are done with no errors, .NET Core SDK is installed on PI. Just run the command dotnet –info to display the information about DotNet SDK and Runtime details.

Now, that we have installed the .NET Core SDK and Runtime on Pi, it is time to build and deploy the published code on Pi. As I mentioned earlier, I have the network shared drive enabled on my Pi to copy the files. You can also transfer to Pi via other methods like FTP.

As a first step, let's publish the application in Linux ARM architecture since Raspbian is based on Linux. Navigate to the project folder and execute the following command to publish the output.

  1. dotnet publish . -r linux-arm   

 

If you want to publish in release mode, you can add -c release attribute.

The code is published in Linux-arm\Publish folder. Now, I will create a folder in PI server called VideoParserAPI and copy all the files from linux-arm\publish folder and paste into VideoParserAPI folder in PI Server. Now the code is published into PI, we will just the run the application to start the service listening on port 5000. Remember in my startup class, I used the Port No 5000 to listen for network connection. Remember, this is my personal project and i will be using it only in my internal network and I have no intention to publish it on the internet. If you have the application that needs to be published on the internet, you may have to use reverse proxy like nginx to configure port 80 and reroute to kestrel web server to follow best practices.

 

Let's run the application to start the services. Open the Terminal Window on PI Server and execute the ./VideoParserAPI to run the application. It a few seconds, the service will start and listen on Port 5000.

Let's call the web API from my system to see the output.

http://pi:5000/api/ch9/roku

There you go. Our Web API Application developed in the latest .Net Core 2.1 is running on Raspberry PI.

Conclusion

I have created design pattern to handle all the core parsing logic in the base classes so that we don't have to rewrite the logic for every other streaming box. However, we can customize the logic to change the output xml / json content based on the streaming box in the appropriate child class. In the future, I will be developing an Android app to consume these web APIs to play the content in my mobile with Chromecast support. We can also extend this library to any other streaming box (Apple TV, Fire TV) . I also plan to deploy this web API app inside the docker in my Raspberry PI later.

I hope this will help you to get going with your crazy ideas. The entire source code is uploaded to github.

Happy Coding!!!


Similar Articles