web URL scrapping ? ? ? advice also appreciated .

Question

Hi

 

 i intend to develop an application .. i just want to  know your view how you start the thing ..

 

My original requirement ... any ideas will be duly appreciated ty ..

 

 

Write an application which accepts requests to scrape web URLs and store the results.

The application should do the following :

1. Have an HTTP API to accept requests to fetch a particular URL

2. Assign a request ID to the request, and the response of this API should return the request ID

3. Process requests in batches in the background

4. The response of each request (the headers, the body, the time of request, the response size, errors if any) should be stored, which can be retrieved later using a status API

You are free to use any language, platform or database. Your answer will be judged on application and API design, clean code, maintainability, performance and efficiency.

 

 

My best wishes

Ty for your help