Recently, one of my friends, who works for an insurance company, buzzed me with this strange scenario he was encountering with the federal reserve site. There are a bunch of data files that are freely available on the site and the only catch is that they had to click on the "Accept Terms and Conditions" button for downloading.
After a few attempts, I found a way to use C# to download the file.
Get the Session Details
The below code first tries to hit the file that generates the session id and fetches the session id for that particular download call.
- static string GetJSession()
- {
- string jSession = string.Empty;
-
- HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create("https://www.frbservices.org/EPaymentsDirectory/fpddir.txt?AgreementSessionObject=Agree/EPaymentsDirectory/fpddir.txt?dataCaptureActive=true");
- httpWebRequest.CookieContainer = new CookieContainer();
- HttpWebResponse response = (HttpWebResponse)httpWebRequest.GetResponse();
-
- Uri uri = new Uri("https://www.frbservices.org");
-
- return httpWebRequest.CookieContainer.GetCookies(uri)[0]?.Value;
-
-
- }
Authenticate the Session
After fetching the session, it needs to authenticate the session cookie. In a nutshell, the below code is performing the same action as clicking on the "Accept Terms and Condition" button.
- static async Task<string> GetResponse(string JsessionID)
- {
- CookieContainer cookies = new CookieContainer();
- cookies.Add(new Cookie("JSESSIONID", JsessionID, "/", "frbservices.org"));
- HttpClientHandler handler = new HttpClientHandler();
- handler.CookieContainer = cookies;
-
- var client = new HttpClient(handler);
-
- var requestContent = new FormUrlEncodedContent(new[] {
-
- new KeyValuePair<string, string>("agreementValue", "Agree")
- });
- client.DefaultRequestHeaders.Add("Referer", "https://www.frbservices.org/EPaymentsDirectory/agreement.html");
-
- HttpResponseMessage response1 = await client.PostAsync("https://frbservices.org/EPaymentsDirectory/submitAgreement", requestContent);
- Uri uri = new Uri("https://frbservices.org");
-
- HttpContent responseContent = response1.Content;
-
-
- using (var reader = new StreamReader(await responseContent.ReadAsStreamAsync()))
- {
-
- Console.WriteLine(await reader.ReadToEndAsync());
-
-
- }
- return string.Empty;
- }
Download the file
The below code will download the file and store it in the same folder as exe.
- static void ExecuteRequest(string jSession)
- {
- HttpWebRequest httpWebRequest = (HttpWebRequest)WebRequest.Create("https://www.frbservices.org/EPaymentsDirectory/fpddir.txt?AgreementSessionObject=Agree/EPaymentsDirectory/fpddir.txt?dataCaptureActive=true");
- httpWebRequest.CookieContainer = new CookieContainer();
-
-
- httpWebRequest.CookieContainer.Add(new Cookie("JSESSIONID", jSession, "/", "frbservices.org"));
-
- httpWebRequest.CookieContainer.Add(new Cookie("abaDataCaptureCookie", "abaDataCaptureCookie", "/", "frbservices.org"));
-
- HttpWebResponse response = (HttpWebResponse)httpWebRequest.GetResponse();
- using (Stream s = response.GetResponseStream())
- {
- FileStream os = new FileStream("fpdir.txt", FileMode.OpenOrCreate, FileAccess.Write);
- byte[] buff = new byte[102400];
- int c = 0;
- while ((c = s.Read(buff, 0, 10400)) > 0)
- {
- os.Write(buff, 0, c);
- os.Flush();
- }
- os.Close();
- s.Close();
- }
Main Method Code
- tatic void Main(string[] args)
- {
-
-
- string jSession = GetJSession();
-
-
- GetResponse(jSession).Wait();
-
-
- ExecuteRequest(jSession);
-
-
-
-
-
-
- }
Hope this helps someone.
Happy Coding.