Python Scraping Data From the Web Additional Scraping Tasks Using Scrapers for Site Testing

Can you use selenium for scrapy spiders sraping js-based websites?

I am wondering how could you implement similar solution shown here for beautiful soup with scrapy. I am tinkering with a spider of my own and having a hardtime figuring out how to make my spider wait for js-cripts to load.

I disabled js in my browser while entering website of my choice and it turns out it doesn't show up so I guess this is the issue I am having. Not sure though. I saw people using not selenium but splash with scrapy. Anyone have any recommendations? Each and every comment would mean world to me :)