Welcome to the Treehouse Community

Want to collaborate on code errors? Have bugs you need feedback on? Looking for an extra set of eyes on your latest project? Get support with fellow developers, designers, and programmers of all backgrounds and skill levels here with the Treehouse Community! While you're at it, check out some resources Treehouse students have shared here.

Looking to learn something new?

Treehouse offers a seven day free trial for new students. Get access to thousands of hours of content and join thousands of Treehouse students and alumni in the community today.

Start your free trial

JavaScript

Posted by Michael McDonald
Michael McDonald
2,877 Points

Is there a practical way to write a program to respond to updates in a webpage?

I would like to write a program to "watch" a webpage and notify me if it changes. I could resort to having a browser render the page and then scan video memory for changes I am expecting. That seems a bit extreme.

The portion of the website I am interested is produced by a javascript and I'm not sure what else I can offer about how it works. It draws an image of a calendar and places green squares where new available jobs have been posted. Poking on a green square brings up a description of the work but I only care whether there even is a green square or not.

I have novice level programming experience in several languages but advanced mastery in only Visual Basic. I'd be happy to learn a new language if it meant solving this problem.

Michael Bianchi
Michael Bianchi
2,488 Points

The languages that might work for your problem easily would be python or PHP, more so PHP. Many web crawlers (which is essentially what you need) are built in php.

1 Answer

Dino Paškvan
Dino Paškvan
Courses Plus Student 44,108 Points

Since the calendar is created with JavaScript, a simple script to download the page and compare won't work.

You'll need something that actually renders a page. PhantomJS is a headless WebKit with a JavaScript API, meaning it'll render the page, and then using some JavaScript magic you could check if there are green squares on the calendar.

It could looks something like this:

  • connect to the website
  • after it's rendered, select the calendar element
  • cycle through its children and check if the class for the green square is applied or not
  • print out which days have a green square

That's just a general suggestion since I have no idea what the website you want to query looks like.

Michael McDonald
Michael McDonald
2,877 Points

Thanks Dino. Does PhantomJS have a refresh method I can use to stay logged in to prevent automatic log out? I intend to log in and then run the code rather than specify credentials within the code. Lastly, is there a forum category tag for questions like this better than the Javascript tag I chose? Of course this isn't the kind of question this forum is built for and the majority of my posts are tightly related to the coursework. I just really like the community here.

Dino Paškvan
Dino Paškvan
Courses Plus Student 44,108 Points

The JavaScript tag is fine, as both the question and the answer seem to be centred around it.

How often do you plan on checking? Daily or?

If I was to write something like this, I'd make the logic of the script go something like this

  • log in
  • check the calendar
  • show output

And then set it as a cron task or something similar so it repeats as often as I need it.

If you want to run it manually, the same steps would still be fine. PhantomJS wouldn't stay logged in, each run would be like a completely new session. So, you'd just program in the log in logic to run before the calendar check.

Michael McDonald
Michael McDonald
2,877 Points

I was thinking to log in first and pass the url to the program then have the program loop: + wait 10 minutes + render page + check for green squares + if found, notify + loop