Avoid downloading csv, but parse through and save to database

You want to put a PHP script on your server that can be run from the command line. You then want to schedule that script to run at a regular interval with a cron job. In the PHP script, you can make a cURL request to retrieve the CSV.

Does any of that help? :~)

Sure does. Where can I learn more about what a cURL is and how it works?

Hey Randy Hoyt! Where can I learn more about how to use cURLs?

Grey Elerson - Check out this tutorial on using cURL with PHP

Thanks James. That's a really great resource. I'm having a little difficulty understanding what the cURL is, though. The way that the process would currently work, I would send the request via URL, and the browser would download a CSV. This would require me to interact with the server from my client computer in person for every round of data collection. How can a cURL eliminate the client, and allow my server-side scripts to interact with another server automatically?

I guess what I'm asking is this: Would I use the same URL that I have to enter by hand as the cURL URL? If so, how can I have it return the CSV to a variable or other type of input? Maybe an array?

It sounds like you are accessing the PHP script in the browser, right? The cURL command goes in the PHP script, and it accesses the URL of the CSV file. You can get it working in your browser first, but you eventually want it to run from the command line and then from a scheduled cron job that will run without your intervention.

Does any of that help?

Yes, I want to access the PHP result from the browser. Here's where we may be tripping over each other, though. The csv file doesn't have a http://website.com/some.csv url. Instead, the csv is created dynamically in response to the url request. Right now, it automatically downloads without the option of being displayed in the browser. From some digging around, I've been able to learn a fair deal about using cURLs (they seem really handy). Now, running the script from the command line is something that's completely new for me, as are cron jobs. Do you know of a good resource that can introduce me to running scripts from the command line?

I'm not following what you are trying to do. Is this the problem you are trying to solve?

"Right now, the user types in a web address for a CSV file and the browser downloads it. I don't want the browser to download it; instead, I want to display the data in the browser."

If so, here are some questions:

• Do these users work for your company? Or are they external customers or site visitors?
• Would they want to interact with the data in a spreadsheet program like Excel?
• Why don't you want them to download the CSV?

Ah. I gotcha. I'm really sorry that I'm not very eloquent with my technical description. The only user of the overall application will be me. I want the script to request csv data from another server using a URL. Right now, however, making the request via URL would require me to type it in manually (not efficient), and the csv automatically downloads (rather than being displayed in the browser).

If I can display the csv, then I know that I can parse all of that data into a PHP variable that I can process as I see fit. That's the reason that I wanted to show it in the browser. If there's another way, then I'm all ears.

There's not going to be any interaction with the data via excel spreadsheet right now. I want all of the data processing to be handled by PHP on the server. When all is said and done, I want only the metrics to be shown and styled in the browser.

My biggest issue is how do I get the csv file information over to my server where the php will run. Since the script will run numerous times per day (to capture real-time data), I don't want the csv to download or save. I just want to be able to capture the information and process it on my server.

Is that more clear?

I think we're getting closer. Do you need to save information from the CSV so that you can see how it changed over time? Or do you just need to be able to look at the data live?

Yes! Now we're getting somewhere! haha.

I do want to save the information, but to a mySQL database. If I'm not mistaken, though, that would be different functionality than the code I would need to access and process the data from the csv. Right?

All of the information that will be displayed will be pulled and calculated on the fly from data stored in the database. That information, however, will be updated many times throughout the day.

Cool. Are you going to save the history so that you can look at how the data has changed over time? Or are you going to save only the latest state of the data?

Yes, the different process functions will keep a log of anything noticeable within the data. I also want the code to keep logs at least throughout the first version so I can have more pointed and focused troubleshooting later on. That will also be held in a database.

As for the data itself, any new data from an incoming csv will be appended to the database that holds the historical data itself.

Check out this article (skip down to the "Running PHP from the command line" section):

• http://www.netmagazine.com/tutorials/how-use-php-command-line

Then check out this article (the first method should work):

• http://www.thegeekstuff.com/2011/07/php-cron-job/

This will give you a PHP script on the server that runs automatically without your intervention. You can write that script to retrieve data from the CSV using cURL.

Does that help?

It does. Thanks. Where can I find out how to retrieve the csv data that would automatically download?

You should be able to do that with cURL. I found a discussion on Stack Overflow with someone asking this same thing, and the first answer there has an example that should work for you:

• http://stackoverflow.com/questions/9304457/receive-csv-from-server-and-parse-using-php

How's that?

Randy, you are on your game. So far as I know, you've pointed me in the right direction for all of my concerns. Talk about service!

Listen, I work for a web service firm called Viss Media. We offer fully-supported Wordpress development and managed hosting. I tend to work on the administrative side, which is why I'm on Treehouse to learn more about the skilled labor aspect. If there is ever anything that we can do for Treehouse, please don't hesitate to ask. I'd be more than happy to make sure that you can get in touch with the right people right away.

Thanks again!

Glad to help!