hi all

urgent help

on scrapy

what happen

vii_majesty: theekkathir.in/News/states

I want to scrape this website

Note this website, there is no pagination support

I want to scrape by clicking load more button further scraping

beyond five-six list of news

have you tried curl

no

actually try wget

wget -r theekkathir.in/News/states

it downloads xml files

otherwise in your browser, go to the site

and press CTRL S

I saves as given 5-6 websites, while downloading

when I asked llama , It says

to use scrapy-selenium

any alternatives to scrapy-selenium

Why are you checking alternative to scrapy selenium is there any issues you're facing with it?

no hariharan

what is the purpose of alternative you're looking for?

selenium makes more challenging to write code to work with github actions

hariharan: I want to click load more button

to scrape more contents, which loads down

You have shared a github is that is used to scrape the contents?

yes hariharan

shall I share again

mohan43u: how to update the youtube live description with my blog post url can you able to update it ?

hariharan: I can update, send me the link