How are you suppose to build an app with tons of users in MongoDb if the maximum document size is only 16MB

Question

I have an important question about MongoDB that’s been on my mind for quite some time.
Question: How are you suppose to build an app with tons of users in MongoDb if the maximum document size is only 16MB
Explanation:
Ok here is a simple example of what I mean. Lets say that you are building a ToDo List app,
Now the way the todo list app works is that every single day you have 5 things to do and those 5 thing that you are suppose to do gets saved in Mongo within a tasks document. So you can imagine a Mongo Schema like so,
javascript
var TodoSchema = new Schema({
    task: String,
    completed: { type: Boolean, default: false }
});

So now you the user creates 5 new tasks a day and each set of daily tasks gets saved onto a collection of ToDo tasks which end up looking like this,
javascript
{
  "_id": ObjectId("5872da60a6d290293f2f26db"),
  “task”: “Do the dishes“,
  “completed“: true
}
{
  "_id": ObjectId("5872da60a6d290292f2f26db"),
  “task”: “Study“,
  “completed“: true
}
{
  "_id": ObjectId("5872da60a6d2902452f26db"),
  “task”: “Work Out“,
  “completed“: true
}
{
  "_id": ObjectId("5872da60a6d2902dg3f2f26db"),
  “task”: “Feed the fishes“,
  “completed“: true
}
{
  "_id": ObjectId("5872da60a6d290293fgf26db"),
  “task”: “Sky Dive“,
  “completed“: false
}

Ok now here is the problem, lets say we have 100 thousand users or more using your ToDo list app and they love it. You end up having all 100 thousand users create 5 tasks a day WHICH GETS PERMANENTLY SAVED FOR THE ENTIRE YEAR. Meaning that through out the entire year of 2017 the users can go back and look at all their previous tasks.
As you can probably see this will pose to be a problem do to the fact that 100 thousand users times 5 tasks a day ends up being a lot of todo tasks that get saved. So now I’m stuck with this problem where the tasks document is only 16MB of maximum storage and have a lot more tasks than it can handle.
Which brings me back to my initial question, how can we build a full app of thousands of users if the maximum document size is only 16MB. I don’t see how I can have thousands of users on my ToDo list app if the maximum amount of tasks that the tasks document can take is only limited to 16MB.
@Andrew Chalkley (https://teamtreehouse.com/chalkers)

Ken Alger · Answer

@Lucas Santos (https://teamtreehouse.com/lucassantos) this is a great question that comes up quite a bit when discussing MongoDB. Let's chat about it for a moment.
First, you are correct that each document in MongoDB has a limitation to 16MB. Let's think of that in a bit of perspective, however, especially as it relates to your Todo List application. All of William Shakespeare's works come in at a little over 5MB, so you should have lots of room for your todo list in general.
From a schema design standpoint, in your above example, each of those todo's is a separate document in MongoDB having the 16MB constraint. All of those todo's together would combine into a single collection whose size is not limited, at least by MongoDB. Limits at that point would be dependent upon server hardware limitations. Again, based strictly on the way you are thinking of designing the documents above, I would recommend including something about which user each document is associated. Or as @Jeff Wilton (https://teamtreehouse.com/jeffwilton) mentioned, have each user's tasks associated with their own document.
Much of the schema and database design is going to depend upon how you will be using the data. Is the application going to be performing more read operations or will it be doing more write operations. What aspects of each document will be read each time. If it is a read heavy application what sort of indexes do you have on your collection?  
How quickly do you need historical data from the todo list? Is having a search result take a little longer acceptable if I am pulling up my todo list from 1983? Will there ever be joint todos between two or more users? How are those situations handled?
In my experience the 16MB per document limitation is not something most users of MongoDB are highly impacted by, with a properly thought out schema design. I know people who use MongoDB to handle client's corporate and personal tax returns without any performance issues at all and, in fact, picked MongoDB because it was more performant than other options. They have yet to run into an issue with the 16MB limitation, but they also have a well designed schema.
I hope that helps somewhat, but certainly would be happy to answer any further questions you may have.
Happy coding.

Jeff Wilton · Answer

I think what you are referring to is the BSON Max Document Size which would just a representation of the data retrieved from the mongo db. Each user would only retrieve a BSON document of their tasks, not everyone else's - and this limitation is purely for performance reasons. a 16MB BSON document would be pretty huge, and take a long time to parse through and display on a UI, so you certainly wouldn't want to exceed that. There are further techniques such as pagination that could be helpful if necessary.
Here is a helpful discussion about the size and scalability of Mongo DB (some popular websites use Mongo Databases that have more than a Terabyte of data).
http://stackoverflow.com/questions/7429663/what-is-the-maximum-capacity-of-mongodb-is-it-a-fully-scalable-db

Welcome to the Treehouse Community

Looking to learn something new?

Lucas Santos

Lucas Santos

How are you suppose to build an app with tons of users in MongoDb if the maximum document size is only 16MB

2 Answers

Ken Alger

Ken Alger

Jeff Wilton

Jeff Wilton