Hi everyone.
I am working on a project in which I want multiple urls to be scrapped by the same scrapper. For example: Let's say I want to scrape social media profiles. I want to scrape the name and the profile picture.
So I just define one scraper for this use case.
Let's say I have the profile urls of 10000 people. How can I scrape all of these urls without defining 10000 websites in the Django Administrator ?
Currently, what I see is that I can define one website with one hardcoded url and link it to a scraper and call the scrapy command tool with the website ID. But It doesn't give me any option to change the url dynamically.
I can't believe that there is no such option and that's why I am asking the community if there is such an option or if I can define this specific mechanism at the model level.
Thank you
Hi everyone.
I am working on a project in which I want multiple urls to be scrapped by the same scrapper. For example: Let's say I want to scrape social media profiles. I want to scrape the name and the profile picture.
So I just define one scraper for this use case.
Let's say I have the profile urls of 10000 people. How can I scrape all of these urls without defining 10000 websites in the Django Administrator ?
Currently, what I see is that I can define one website with one hardcoded url and link it to a scraper and call the scrapy command tool with the website ID. But It doesn't give me any option to change the url dynamically.
I can't believe that there is no such option and that's why I am asking the community if there is such an option or if I can define this specific mechanism at the model level.
Thank you