Welcome to the Treehouse Community

The Treehouse Community is a meeting place for developers, designers, and programmers of all backgrounds and skill levels to get support. Collaborate here on code errors or bugs that you need feedback on, or asking for an extra set of eyes on your latest project. Join thousands of Treehouse students and alumni in the community today. (Note: Only Treehouse students can comment or ask questions, but non-students are welcome to browse our conversations.)

Looking to learn something new?

Treehouse offers a seven day free trial for new students. Get access to thousands of hours of content and a supportive community. Start your free trial today.

JavaScript

Radha Bhambwani
Radha Bhambwani
10,182 Points

node js and mongodb

I'm new to node js and mongodb. Just testing out some ideas so I am trying to create a web scraper that adds data to a database in mongodb. I have a script that connects to mongodb and dynamically adds data to a database through a node js script. The script runs without any errors, but when I fire up the mongo shell to and run db.posts.find(), I get 0 records. Here's my script:

var MongoClient = require('mongodb').MongoClient;


//requiring the module so we can access them later on
var request = require("request");
var cheerio = require("cheerio");

MongoClient.connect('mongodb://127.0.0.1:27017/makeupdb', function (err, db) {
    if (err) {
        throw err;
    } 
    else {
        console.log("successfully connected to the database");

        //define url to download 
        var url = "http://www.nyxcosmetics.ca/en_CA/highlight-contour";

        var prodList = [];
        var priceList = [];
        var products = [];

        request(url, function(error, response, body) {
            if(!error) {


                //load page into cheerio
                var $ = cheerio.load(body);

                $(".product_tile_wrapper").each(function(i, elem) {
                    prodList[i] = $(this).find($(".product_name")).attr("title");
                    priceList[i] = $(this).find($(".product_price")).attr("data-pricevalue");
                });
                prodList.join(', ');

                for(var i = 0; i < prodList.length; i++) {
                    var prod = {
                        name: prodList[i], 
                        price: priceList[i]
                    };
                    products.push(prod);
                }

                console.log(products);

                //insert the prods into the database
                //in the 'posts' collection
                var collection = db.collection('posts');
                collection.insert(products);
                console.log("products inserted into posts collection");

                //Locate all the entries using find
                collection.find().toArray(function(err, results) {
                    console.log(results);
                });

            } else {
                console.log("We've encountered an error!")
            }
        });
    }

    db.close();
});
Jesus Mendoza
Jesus Mendoza
23,287 Points

I have never tried web scrappling beforfe but I think you are using the wrong selectors to find the product_name and product_price. Why are you using a dot in front? Is the data comming in a json file right?

Radha Bhambwani
Radha Bhambwani
10,182 Points

The data is not in JSON format, its in html. product_name and product_price are classes inside inside the html of the url, hence the dot usage.