Welcome to the Treehouse Community

Want to collaborate on code errors? Have bugs you need feedback on? Looking for an extra set of eyes on your latest project? Get support with fellow developers, designers, and programmers of all backgrounds and skill levels here with the Treehouse Community! While you're at it, check out some resources Treehouse students have shared here.

Looking to learn something new?

Treehouse offers a seven day free trial for new students. Get access to thousands of hours of content and join thousands of Treehouse students and alumni in the community today.

Start your free trial

Databases

Alex Flores
Alex Flores
7,864 Points
Posted by Alex Flores
Alex Flores
7,864 Points

Question about database design and efficiency?

I'm building a web app and I need to request data from another website (large volumes of it), parse it and then store it in the database.

In order to get all of the data I need, I need to make two requests to the outside website. I'm wondering if I should get all of my data first, parsed and ready to store OR if I should make the first request, parse it and store it and THEN make the second request and parse it and store it.

Is there a best practice for this? Is this called something?

1 Answer

Steven Parker
Steven Parker
231,128 Points
Steven Parker
Steven Parker
231,128 Points

I don' t think it makes much difference in efficiency, but I think your second approach might make it easier to deal with error scenarios. It could also help avoid any temporary memory capacity issues should the amount of data increase in the future.

Kevin Korte
Kevin Korte
28,149 Points
Kevin Korte
Kevin Korte
28,149 Points

I agree with Steven, from a speed perspective, two halves still make a whole, but there is benefit in handling it in smaller chunks for the reasons he mentioned.