The Met hides it's data well- I'm stuck trying to get past an infinite scroll


#1

I’m trying to grab data from here: http://www.metmuseum.org/art/collection/search#!?department=4&q=dagger&sortBy=Relevance&sortOrder=asc&page=1

I’m consistantly only getting 21 hits. I’ve tried to find the URL, and I think I found something that is relevant: http://www.metmuseum.org/api/collection/collectionlisting?artist=&department=4&era=&geolocation=&material=&page=3&q=rapier&showOnly=&sortBy=Relevance&sortOrder=asct

But the URL doesn’t work. I’m not sure how to get around this.


#2

Not sure if you got help on this, but this is what I found out. The URL list is a little odd only because Page 1 has a somewhat unique address compared to others:
http://www.metmuseum.org/art/collection/search#!?department=4&q=dagger&sortBy=Relevance&sortOrder=asc&perPage=20&offset=1&pageSize=0

Notice the offset=1. All of the subsequent pages go from 20 incrementing by 20. So this is an example URL list and I was able to pull the data you needed. Hope this helps:

http://www.metmuseum.org/art/collection/search#!?department=4&q=dagger&sortBy=Relevance&sortOrder=asc&perPage=20&offset=1&pageSize=0
http://www.metmuseum.org/art/collection/search#!?department=4&q=dagger&sortBy=Relevance&sortOrder=asc&perPage=20&offset=20&pageSize=0
http://www.metmuseum.org/art/collection/search#!?department=4&q=dagger&sortBy=Relevance&sortOrder=asc&perPage=20&offset=40&pageSize=0
http://www.metmuseum.org/art/collection/search#!?department=4&q=dagger&sortBy=Relevance&sortOrder=asc&perPage=20&offset=60&pageSize=0
http://www.metmuseum.org/art/collection/search#!?department=4&q=dagger&sortBy=Relevance&sortOrder=asc&perPage=20&offset=80&pageSize=0

and so on…