[question] best practice for searching same keywords on different sites

[Hans-Maulwurf]

Hey guys,

maybe someone can hint me to a good or best practice for my use case.

• I have a few keyword, e.g. product1, product2, product3
• I want to let webchanges look for changes on different urls for these keywords,
  ** e.g. site1.com?search=product1, site1.com?search=product2, site1.com?search=product3
  ** site2.com?search=product1, site2.com?search=product2, site2.com?search=product3

Now I ask myself, how ...

• how to define these keywords one time and re-use them in multiple webchanges-jobs?
• how to have some (random) delay between the requests, since I plan to have like 100 keywords (products) and 5 sites, one request per minute (to one site) would be enough. 100 requests in parallel (or within 10 seconds) would be too many I guess and would be rejected by the firewall

[mborsetti]

Hi @Hans-Maulwurf,

how to define these keywords one time and re-use them in multiple webchanges-jobs?

I'd probably write a script that writes a jobs.yaml file with the correct directives.

how to have some (random) delay between the requests, since I plan to have like 100 keywords (products) and 5 sites, one request per minute (to one site) would be enough. 100 requests in parallel (or within 10 seconds) would be too many I guess and would be rejected by the firewall

I would first try to limit parallelism by using the command line argument --max-workers. If setting this all the way down to 1 still isn't slow enough for the site(s) in question, please let me know.

[Hans-Maulwurf]

Thank you for your quick reply.

Writing a script that is creating the yaml-file sounds like a static way. I would like to maintain one file with keywords and this file should be read somehow into websitechanges. Maybe this can be done with execute or shellpipe?