Poor Man's Web Monitoring Tools

Background

This project was initially started back in 2012. Then, one day, my computer's hard drive crashed. I was lucky enough to be able to recover most of the files. After spending some time, I was able to get it to compile again. After that, I worked on it maybe once or twice a year. There is no doubt lots of improvements can be made to it. Lately, I've made some changes to the project and decided to share it now instead of letting it sit on the shelf for another couple of years. Hopefully, someone will find this project useful and continue to make enhancements to it.

Introduction

Like many other bloggers or website owners out there, most of us host our website on a shared server instead of dedicated server hosting due to the cost. The hosting company shall not be named here. Based on my own experiences, once in a while, some malicious folders and files/malware were being injected into the website. Every time I submitted a ticket about it, I would get a response like "Your computer was compromised", "you're using outdated software", "you need to change your password", etc. All kinds of this nonsense but the hosting company never took responsibility or did their due diligence. Some hosting companies even offered to clean up the vulnerability and monitor EACH site on the hosting for a yearly subscription. Imagine if you have more than one website. The cost will quickly add up due to someone else's negligence.

Since I refused to pay some company every year to monitor all my websites, why not build my own? The title says it all, "Poor Man's Web Monitoring Tools". If you're poor, then you need to work hard and put together all the available free tools yourself. It might look like a Frankenstein tool now, but I’m positive it will look better once we pour some more thought into it.

Shown below is a brief description of some of the components of the solution. In this article, I'll not go into every detail on how the solution was being implemented as I believe some more work is needed to optimize it. But I will share what each component will do and how to set it up.

1. DownloadIISLog
   • Download IIS Log files through FTP (WinSCP)
   • Insert the log file data into the SQL Database using Log Parser
2. SimpleMonitor.Web
   • Web UI to display the IIS Log data using jqGrid
   • Option to mark certain IP addresses "Banned"
3. SimpleMonitor.BlockIpAddress
   • Example of HTTP Module to block a banned IP address from accessing the Website
4. SimpleMonitor.ScanFile
   • Download latest files
   • Compare Baseline vs latest files using WinMerge
   • Send email notification of the results

DownloadIISLog

One of the responsibilities of this console application is to download the log files from the FTP Server using WinSCP. Please note that I only tested this on two different shared hosting environments. Here, what I would suggest is that if you get a permission denied error while accessing the log folder, first, log into your hosting account, navigate to the file manager, look for the logs folder, and assign read-only permission to the folder. The second option is to submit a ticket.

Shown in Listing 1 is the method to download the log files. The initial run will take a while, depending on the number of log files residing in the hosting log folder. The subsequent execution will be a lot faster because this method will download only files that were not previously downloaded. To achieve this, the method will first get the list of log file metadata from the server using the WinSCP ListDirectory method and store it in the directoryInfo object. Next, it will get the most recent log file name stored in the database. After that, the module will try to query the directoryInfo object for the LastWriteTime by the log file name. Finally, the module will download all the log files where the LastWriteTime is after the LastWriteTime that was determined previously.

static internal void WinScpGetLog() {
    using(Session session = new Session()) {
        // Connect
        session.Open(GetWinScpSession());
        
        string remotePath = ApplicationSetting.FtpRemoteFolder;
        string localPath = ApplicationSetting.DownloadPath;
        
        // Get list of files in the directory
        RemoteDirectoryInfo directoryInfo = session.ListDirectory(remotePath);
        
        // latest file name in the table by application name
        var latestLogFileInDb = Path.GetFileName(unitOfWork.IISLogRepository.LatestFileName(ApplicationSetting.ApplicationName));
        
        // get the date of the latest file from FTP
        var logFileDate = directoryInfo.Files
            .Where(w => w.Name.ToLower() == latestLogFileInDb.ToLower())
            .Select(s => s.LastWriteTime).FirstOrDefault();
        
        // Select the files not in database table
        IEnumerable notInLogTable =
            directoryInfo.Files
            .Where(file => !file.IsDirectory && file.LastWriteTime > logFileDate).ToList();
        
        //// Download the selected file
        foreach(RemoteFileInfo fileInfo in notInLogTable) {
            string localFilePath =
                RemotePath.TranslateRemotePathToLocal(
                    fileInfo.FullName, remotePath, localPath);
            
            string remoteFilePath = RemotePath.EscapeFileMask(fileInfo.FullName);
            
            // download
            TransferOperationResult transferResult =
                session.GetFiles(remoteFilePath, localFilePath);
        }
    }
}

Listing 2 shows the logic employed by the Microsoft Log Parser tool to query the log files and insert them into the database. The query used by the Log Parser in the CallLogParser()method is very straightforward. It will connect to the SQL database using the provided username and password. Then read all the log files in the folder and insert them into a table. The ApplicationNameparameter was added recently as an option to allow storing log files data for more than one website. If you have ten websites, you can duplicate this console application ten times and configure different ApplicationNamevalues in the configuration file.

static void CallLogParser()
{
    ProcessStartInfo startInfo = new ProcessStartInfo(ApplicationSetting.LogParserPath);
    startInfo.WindowStyle = ProcessWindowStyle.Minimized;

    Process.Start(startInfo);

    startInfo.Arguments = string.Format("\"SELECT '{7}', *,1 INTO {0} FROM {1}{2}*.log\" -o:SQL -createTable:OFF -server:{3} -database:{4} -username:{5} -password:{6}",
        ApplicationSetting.LogParserSqlTable, ApplicationSetting.DownloadPath, ApplicationSetting.LogFilePrefix, ApplicationSetting.LogParserSqlserver,
        ApplicationSetting.LogParserSqlDatabase, ApplicationSetting.LogParserSqlUserName, ApplicationSetting.LogParserSqlPassword,
        ApplicationSetting.ApplicationName);

    Process.Start(startInfo);
}

Figure 1 shows the data imported into the database table through the CallLogParser()method.

DownloadIISLog - Configuration

Table 1 shows the descriptions of the settings in the app. config.

SimpleMonitor.Web

The purpose of this web application is to display the log file data. Now that we have the data in the database, is up to you, what technology you want to use to interface with the data. This web application was developed by using MVC 5, EntityFramework v6.0, jqGrid, Bootstrap v4.0, and jQuery v3.3.