The Met hides it's data well- I'm stuck trying to get past an infinite scroll


I’m trying to grab data from here:!?department=4&q=dagger&sortBy=Relevance&sortOrder=asc&page=1

I’m consistantly only getting 21 hits. I’ve tried to find the URL, and I think I found something that is relevant:

But the URL doesn’t work. I’m not sure how to get around this.


Not sure if you got help on this, but this is what I found out. The URL list is a little odd only because Page 1 has a somewhat unique address compared to others:!?department=4&q=dagger&sortBy=Relevance&sortOrder=asc&perPage=20&offset=1&pageSize=0

Notice the offset=1. All of the subsequent pages go from 20 incrementing by 20. So this is an example URL list and I was able to pull the data you needed. Hope this helps:!?department=4&q=dagger&sortBy=Relevance&sortOrder=asc&perPage=20&offset=1&pageSize=0!?department=4&q=dagger&sortBy=Relevance&sortOrder=asc&perPage=20&offset=20&pageSize=0!?department=4&q=dagger&sortBy=Relevance&sortOrder=asc&perPage=20&offset=40&pageSize=0!?department=4&q=dagger&sortBy=Relevance&sortOrder=asc&perPage=20&offset=60&pageSize=0!?department=4&q=dagger&sortBy=Relevance&sortOrder=asc&perPage=20&offset=80&pageSize=0

and so on…