How to scrap data from a webpage with INFINITE scroll and closed API?
Same here, there are many such sites like twitter where the data is huge but the extractor gets only the top results in the run.
In the normal help sections theres an explanation that is quite elaborate:
I hope this helps!
Thanks Leadgen, unable to find the pagenum with twitter:(
Can you post the site you are trying to scrape so I can take a look at it?
I'm struggling with the same issue. I read the help section on Infinite Scroll but Twitter doesn't seem to use page numbers or php. Here's the Twitter feed I'm trying to scrape: https://twitter.com/ensbot?lang=en
Any help would be greatly appreciated.