Bummer! This is just a preview. You need to be signed in with a Pro account to view the entire video.
Start a free Basic trial
to watch this video
Building Your Own Real-time API
26:07 with Roger StringerThe days of complex backend systems are at an end, developers can build complex apps now without the time it takes to devote to a full backend system thanks to APIs. In this talk, developers will get introduced to using LevelDB, Node.JS and Socket.io to build a real-time REST API that can be integrated into any any client-side app quickly. Once built, developers can use this API for apps of all sizes and platforms without any extra work.
-
0:01
[MUSIC]
-
0:15
All right. Welcome to Building Your Real-Time API.
-
0:19
AKA, how to get away with real-time.
-
0:24
I'm Roger Stringer.
-
0:26
I'm a DevOps engineer, author, speaker and
-
0:30
founder of Data McFly, a real-time app back end.
-
0:34
I work with a variety of languages, but the most common are JavaScript and PHP.
-
0:40
So today we're gonna cover the need for real-time data.
-
0:47
We're gonna compare a non real-time system and a real-time system,
-
0:53
and then we're gonna talk about what you need to build real-time APIs.
-
0:58
The need for real-time.
-
1:00
Real-time data, the notes information that is delivered immediately after collection,
-
1:06
you push data rather than request it.
-
1:08
Normal webpages are not real-time,
-
1:11
you have to refresh your page to see the content.
-
1:17
It's inefficient, if you're sending data from a web,
-
1:20
if from a webpage you want everybody to see it right away, and
-
1:23
you want to make sure you didn't just send something that's already sent.
-
1:29
For example, a help ticket system.
-
1:31
You're replying to a help ticket.
-
1:33
There's another support tech sitting next to you replying to the same ticket.
-
1:39
Real-time data gives your users a premium experience.
-
1:42
Real-time data is the best data.
-
1:46
The help ticket system is again, that sample.
-
1:51
Twitter, heavy users are real-time, most people found out about
-
1:56
real-time web design when Twitter started instantly updating the page.
-
2:01
GitHub does the same process.
-
2:04
Uber makes heavy use of real-time for
-
2:08
their come get me, pick me up, drive me home.
-
2:13
And Google Docs, if you've ever worked with Google Docs.
-
2:16
With other people, they will see what you're typing, and
-
2:19
they will see your name as you're typing it.
-
2:24
Which can be handy so you're not editing the same paragraph.
-
2:28
And Facebook.
-
2:29
Everybody loves sharing photos of their cat.
-
2:32
And of course here, News Feed has the update automatically.
-
2:35
So these samples.
-
2:40
They have a lot of engineers.
-
2:41
Most companies don't have as many engineers, so
-
2:44
how do you build real-time without a lot of engineers?
-
2:49
Ask anybody and they'll tell you that real-time data is hard.
-
2:53
They'll tell you that building systems to sync between that iPhone and an iPad and
-
2:57
a desktop and a laptop, it's hard, it's difficult.
-
3:02
It's really not.
-
3:06
HTTP is request response.
-
3:08
So like I said, the client's respond is blind to updates on the server,
-
3:13
but that's what we're learning to get around.
-
3:20
So there are three data transport solutions.
-
3:22
We have polling, that's your standard AJAX, it's been around for years.
-
3:27
It will send out a regular interval to update the page.
-
3:29
It's not really real-time.
-
3:35
It's not efficient, it doesn't scale,
-
3:36
if you have a lot of people hitting the server, that way, what's gonna happen?
-
3:40
The server's gonna have traffic issues.
-
3:43
Server sent events.
-
3:45
Server sent events works well.
-
3:47
It's kind of Web Sockets, without web sockets.
-
3:50
But it mostly only works in Chrome, and I think Opera supports it, but
-
3:55
it doesn't work with any other browser at all.
-
4:00
They're trying to make it work but it doesn't.
-
4:04
So we have Web Sockets.
-
4:05
With Web Sockets we have two way data push, and it works great for transport,
-
4:09
you can pass data of lots of sizes.
-
4:13
You still need to store your data somewhere.
-
4:17
So data transport is only one tiny part of our problem.
-
4:20
Real-time message passing does not equal data sync.
-
4:25
You still need to have a database, so you'd still need a service for that.
-
4:30
So a quick scenario.
-
4:32
Let's build a non real-time system.
-
4:36
Normal non real-time system can involve a lot of different pieces.
-
4:43
Jason is on his desktop, he refreshes the page to see new posts.
-
4:47
Sorry I wrote that before I knew you were gonna be in here.
-
4:50
>> [LAUGH] >> You
-
4:55
get a lot of people doing the same request response,
-
4:58
it's more traffic to the server, it's not real-time.
-
5:03
Everybody has to refresh to see it.
-
5:05
Sorry, that got cut off.
-
5:12
Then you toss in your databases, your servers.
-
5:15
How are you supposed to keep track of that?
-
5:17
I've been developing and doing DevOps for 15 years.
-
5:20
To me this, I wanted a better solution,
-
5:24
I wanted to work myself kind of out of a job for all this stuff.
-
5:29
Nothing's in sync.
-
5:29
Devices aren't kept up to date with each other.
-
5:31
Rest has its place, and it's great.
-
5:34
But it's not for keeping clients in sync in real-time.
-
5:37
So, let's take a look at our real-time system.
-
5:41
A real-time system still involves a lot of pieces.
-
5:43
You still have your database.
-
5:44
You have your web sockets.
-
5:46
Sometimes they can be separate servers, but usually they're not.
-
5:49
The trick is making everything work together seamlessly.
-
5:52
So Rick is on his desktop.
-
5:54
You see his new post up here, instantly.
-
5:56
So does the guy on his iPhone.
-
5:58
So does the guy on another computer.
-
6:03
They're working seamlessly.
-
6:04
This is real-time.
-
6:07
With a real-time architecture,
-
6:12
you can use web sockets to push and pull data from your database or
-
6:16
your API, and nobody ever sees the difference.
-
6:21
Everybody sees a new update.
-
6:23
Your friend uploads a latest picture of a cat named Tim, and
-
6:28
everybody sees it right away and comments, and everybody sees the comments.
-
6:32
You can have multiple clients, push and pull, they work great.
-
6:38
That's not hard is it?
-
6:41
It's almost too easy, so why does everybody say real-time is hard?
-
6:46
It's all in the planning.
-
6:47
If you don't have a plan for the proper tools,
-
6:51
what are you gonna do with your API?
-
6:56
So, what tools do you need to properly build a good real-time API?
-
7:00
You need a language.
-
7:02
Node tends to the be the preferred.
-
7:05
Ruby can also handle real-time pretty well.
-
7:09
Databases, do you have LevelDB, MongoDB, and Redis?
-
7:14
We're gonna talk more about LevelDB, rather than the other two.
-
7:18
You need a real-time transport, that's web sockets.
-
7:22
And you need something for security, so you have API keys, JWT tokens,
-
7:27
and IP domain white listing.
-
7:30
Doing all this through clients means that you can,
-
7:33
if you're doing something internally, you can say this is only accessible
-
7:37
within my IP, within my sub domain.
-
7:40
Within my company and nobody else can do remote access to it.
-
7:46
And you need a client side library which Browserify gets handy with that.
-
7:51
So why would you use Node?
-
7:54
Node is JavaScript.
-
7:57
JavaScript's everywhere.
-
8:00
If you have a website open on your computer, you're using JavaScript.
-
8:04
Our presentation uses JavaScript.
-
8:08
It's code reuse is minimal.
-
8:11
Same programming culture client and server, and
-
8:13
there's a lot of JavaScript programmers.
-
8:17
It's a vibrant community, there's over 60,000 packages on NPM, which NPM
-
8:22
is kind of like Gem for Ruby and Composer for PHP.
-
8:30
Almost any solution you're looking for is on there.
-
8:33
Top corporate sponsors and there's a lot of code.
-
8:36
Do not ever make the mistake of searching just JavaScript on GitHub.
-
8:41
GitHub does not like that.
-
8:43
They will probably send my an angry email for suggesting you do that.
-
8:47
Some sweet spots.
-
8:50
Node is perfect for real-time, high concurrency apps.
-
8:52
It scales well.
-
8:54
It's perfect for building API pages, single page rich clients.
-
9:01
It's got service orchestration.
-
9:04
And it works great with JSON document databases.
-
9:08
And JSON is kind of the standard for
-
9:10
how you store your data, with most databases in Node.
-
9:13
Not so sweet spots.
-
9:16
Frameworks, they vary.
-
9:18
You've got Express.
-
9:19
You've got Happy, which isn't very happy.
-
9:25
There's Eskimo.
-
9:31
In the end, most people build their own or go with Express.
-
9:36
It's not as fast for CPU intensive stuff.
-
9:40
But it works well for doing clustering and breaking off children,
-
9:45
forking off children to run in the background and do work.
-
9:51
So, some really basics with Node.
-
9:54
It runs on the V8 JavaScript run time engine,
-
9:58
which Google developed, and which is.
-
10:01
Really fast.
-
10:03
It just helps you're absolutely fast, it's event driven.
-
10:06
Non blocking standard libraries, most APIs speaks streams.
-
10:12
The, so instead of just grabbing a variable called name,
-
10:17
you can stream your input and it helps for larger input.
-
10:22
Someone's passing a 20 page document they wrote in Google Docs.
-
10:27
Using stream grabs that very easily and
-
10:30
helps you parse it without running out of memory.
-
10:33
A lot of the extensions have been built with C,
-
10:36
C++ to help get into the byte code and help be faster, and
-
10:40
the package manager It helps find solutions really fast.
-
10:46
What node is not.
-
10:47
Node is not a framework.
-
10:50
It's not like Django, or Rails, or Laravel.
-
10:56
It's a language.
-
10:57
Node is just a name for server side JavaScript.
-
11:00
It's not a new programming language.
-
11:02
It's older than most programming languages actually.
-
11:05
JavaScript's been around longer than PHP, all those.
-
11:11
Standard libraries by default is asynchronous, which works well for
-
11:16
dealing with your event room clients.
-
11:21
When you make a call to just slash API slash.
-
11:25
Users or slash followers, you're only dealing with that code.
-
11:28
But you can get into callbacks.
-
11:30
Low level, HTTP is by default, there.
-
11:36
So then, we look into doing sockets.
-
11:41
You have developing and debugging.
-
11:42
You have a lot of different choices.
-
11:45
Nodemon, node inspector, and surprisingly Microsoft Visual Studio Code,
-
11:52
the new IDE that Microsoft released, has a really, really powerful debugger for
-
11:57
node built in, which was surprising.
-
12:01
But it's definitely a really good editor for Java Script.
-
12:07
Common modules, prevents pollution of the global scope.
-
12:11
You have two styles of module development.
-
12:21
When we talk about Browserfy, you'll see that you can build your code and node,
-
12:25
run Browserfy, and you're running it in JavaScript for the client.
-
12:29
It's really handy.
-
12:30
NPM, you have node's package manager, some of the ruby gems.
-
12:34
Handles and resolves dependencies.
-
12:38
There's over 60,000 modules, there's more being added everyday.
-
12:44
It's really handy, you just go install and
-
12:46
it installs your module, then you can build your apps.
-
12:49
So this is a web server and node.
-
12:58
It uses a module called Express, it assigns it to port 5,000 and
-
13:03
displays a page called Hello World.
-
13:06
So that's how quick it is to build a web server.
-
13:12
Persistence.
-
13:12
So to be a real time back end, you need data.
-
13:17
You need to store your data somewhere.
-
13:19
You need a database.
-
13:21
A database is a tool for interacting with structured data,
-
13:24
externalize from the core of our application.
-
13:27
Popular node.js options.
-
13:29
You have SQL libraries that are supported, but they're not that popular and
-
13:33
node kind of wraps around a whole no SQL style of data basing.
-
13:39
There's a lot of community support around postgres because
-
13:42
postgres has really support for JSON.
-
13:46
Document stories are more popular.
-
13:48
Good fit for JS.
-
13:48
Document stories are no SQL.
-
13:51
So Mongo DB, LevelDB or Coach DB are popular primary data stores.
-
13:59
Reddis is also popular for in memory data storage, but
-
14:02
Reddis likes to expire your data, so it's not a permanent data store.
-
14:07
A lot of people use them for caching.
-
14:13
What we need from a data base, we need persistence, performance.
-
14:15
We need to be able to quickly access complex data.
-
14:21
We need to be able to store JSON as JSON is hyper flexible, and
-
14:24
can be used anywhere.
-
14:26
Sometimes we need shared access, and sometimes we need scalability.
-
14:31
So, don't tell anybody, but my secret ingredient for
-
14:35
building quick databases in node, is a module called Levelup.
-
14:39
It's a wrapper for LevelDB.
-
14:41
Everybody in this room, if you're using Chrome or most web browsers,
-
14:46
you're using LevelDB right now.
-
14:47
It's a library that was developed by Google.
-
14:52
It's used for index DB,
-
14:53
local storage, all that stuff that's built into your browser it uses Level DB.
-
14:58
So it's highly in use, it's fast, there's no server involved.
-
15:04
Just a key value store that can contain any date.
-
15:11
LevelDB is a fast key value storage library, written at Google,
-
15:15
that provides an ordered mapping from string keys to string values.
-
15:19
Server less, stores JSON.
-
15:22
So basic features.
-
15:23
This image pretty much shows what the data base looks like in LevelDB.
-
15:27
T1 references one base of data.
-
15:30
T2, rather not showing data, T3 is a smiley face.
-
15:37
It's happy data.
-
15:39
Basic features for LevelDB, you have get, you have putt,
-
15:42
you have delete, you have batch, and you have an order iterator.
-
15:47
Your data goes in the same way you enter it.
-
15:49
It helps to add a number sometimes to that.
-
15:51
i keep going the wrong direction.
-
15:57
So, LevelUp is the binary library for LevelDB.
-
16:01
So let's look at how you build a database in JavaScript.
-
16:05
This is building a database in JavaScript.
-
16:08
VarDB level your database.
-
16:13
DB putt user one for the ID, post Roger.
-
16:21
And then we did a DB putt user two, Martin,
-
16:23
and then we told it to do a DB get on user one.
-
16:26
And return that.
-
16:29
That's an entire database now, and note what LevelDB.
-
16:35
There's no x or query in it, it's very basic key value.
-
16:38
One other feature that LevelDB,
-
16:42
LevelUp has that's handy is, Readstream.
-
16:47
So Readstream is kind of like a select all.
-
16:50
When you create a Readstream, you tell it to start by key.
-
16:53
So you can start at user underscore one and
-
16:57
end at user underscore ten, and it will do a stream.
-
17:01
It's an asynchronous stream In this example,
-
17:05
I'm doing an alert for the JavaScript, for the JSON string.
-
17:09
But you can have it process.
-
17:11
You can have it do whatever.
-
17:12
You could even have it delete that user if you wanted to.
-
17:17
Strings let you iterate through multiple records,
-
17:20
perform actions based on their value.
-
17:25
LevelUp also has a very strong ecosystem.
-
17:29
So you have a lot of tools, packages that are growing every day.
-
17:34
I mean you can take LevelUp and build your own graph database.
-
17:37
If anybody has ever heard of graph databases.
-
17:39
It kind of is the no SQL style of relational databases.
-
17:43
Be in a relational database.
-
17:50
So, then we want to make our database real time, so we use Socket.io.
-
17:56
Socket.io is a wrapper around web sockets,
-
17:59
which are supported by every browser and true libraries supported in iOS, Android.
-
18:06
All you have to do is make a connection.
-
18:08
They work in tandem with the web server.
-
18:10
The Socket.io documentation kind of sucks.
-
18:16
If you've ever tried to read documentation at the Socket.io webpage, don't.
-
18:20
Go to the wiki instead, it works better.
-
18:24
Key features, client server push.
-
18:27
Isn't that what we wanted with realtime,
-
18:29
it broadcasts, we get client session information.
-
18:33
We can do name spaces which kind of translate to like chat rooms.
-
18:37
So you could have Slack, slack uses Socket.io with name spaces for
-
18:43
each channel, and I'm sure a few of you in here have used Slack 1 or 2.
-
18:49
Then you have just your plain old WebSockets API which is,
-
18:54
like I said built into every web browser.
-
18:56
So server, a little bit of code again.
-
19:00
I know I did promise someone there wouldn't be much code today,
-
19:02
but I may have lied.
-
19:04
So with your server it's just your straight forward again, node server.
-
19:07
The difference is at the bottom, it listens for a socket connection.
-
19:11
The reason why sockets work so well for real time, is because once you open
-
19:15
that connection, the connection is opened until the browser or the app is closed.
-
19:21
The long pooling that I mentioned with Ajax,
-
19:23
that connection had to open every ten seconds.
-
19:26
So with sockets once you connect, you stay connected,
-
19:32
unless you decide to kick everybody off.
-
19:36
It's handy for doing all your real time stuff.
-
19:40
You can pass your data.
-
19:41
I've never hit any limits to how much data you can pass through a socket,
-
19:46
but I'm sure there is somewhere.
-
19:49
Then you have client, so the client site is just an HTML page.
-
19:53
It tells it to connect to local hosts, and then it says, okay,
-
19:57
once we're connected do it dot on.
-
20:00
Dot on is.
-
20:02
Unfortunately some of the wording isn't as good, but
-
20:04
dot on basically means performed as an action.
-
20:07
Oh sorry, listen for this action.
-
20:10
It's a listener, so you're listening for when news comes back.
-
20:12
And then on the server side, when the connection happens, we do a dot emit news.
-
20:17
So we're sending back hello world.
-
20:21
And then once we receive that we send an email, which is a trigger,
-
20:25
emitters are basically this is how we send data over a web sockets.
-
20:30
We're sending an emit for my other event which contains the data we just received.
-
20:36
And then on the server side we have another listener for my other event which
-
20:41
takes the data and displays it these two work this in real time.
-
20:46
So then we wrapped up with level DB, and when we can receive my other event,
-
20:51
we can take that data, store it in a database.
-
20:54
You build a real time API.
-
20:58
And now, let's look at clients.
-
20:59
So, Browserify Is really handing because with Browserify,
-
21:06
if you write code for Node in JavaScript, you require files.
-
21:12
You can require Socket.IO client.
-
21:14
You can require all that stuff.
-
21:16
You hit Browserify, it'll convert the code that you're running as a Node client.
-
21:22
Into a JavaScript file for your browser.
-
21:25
It lets you make use of MPM modules.
-
21:30
It's, like I said, best to use for building
-
21:34
code that you can use everywhere and don't want to have multiple code bases.
-
21:42
Why would you want to use Browserify?
-
21:44
Helps you write modular code.
-
21:46
No futzing with client side script loading.
-
21:50
Share client and server code.
-
21:52
Have the same code style package management and module system as server.
-
21:57
Why have to different paradigms?
-
21:59
Some companies it may be fine if you have a large number of developers.
-
22:04
But if you're just a couple of developers Browserify works great.
-
22:11
Actually, I work with a company with almost a hundred developers and
-
22:14
we use Browserify all the time.
-
22:15
So, large developers use it too.
-
22:19
Fun features It provides browser shims for most of the node core.
-
22:25
It can provide modules with alternate entry points when used by browserify.
-
22:29
It creates source maps.
-
22:31
One word of warning, if you don't use, use stuff to minify your code, it'll get big.
-
22:38
And security.
-
22:39
So security.
-
22:41
There's lots of ways you can secure an API.
-
22:44
I generally go with API keys and secret keys so
-
22:48
they can only be used by authorized users.
-
22:53
And can be used to set permissions, read only, read write.
-
22:56
You can let them view all the data they want, they can't delete anything.
-
23:01
I had one person once request to be able to
-
23:04
have a key that could only delete stuff and not read anything.
-
23:08
We never set a key up that way.
-
23:11
You can use JSON Web Tokens, also called JWT.
-
23:16
They're kind of like a session.
-
23:17
It's basically an encrypted JSON A JSON document that gets
-
23:23
passed between client and server, that contains information about that client.
-
23:28
It works on by using public and private keys, so
-
23:31
only your client can decode that key, that was assigned by the server.
-
23:40
And the server can look at it.
-
23:41
And so
-
23:41
it really works as kind of a hash that tells you this user's allowed to do this.
-
23:49
And then you can waitlist IPs and domains.
-
23:53
This one works really well.
-
23:54
If you When you're building APIs, you can do this accidentally.
-
23:59
You can run into this all the time, because of something called CORS,
-
24:02
Cross-Origin restrictions in browsers.
-
24:07
So certain API requests through a browser will fail, unless you it has CORS-enabled.
-
24:14
And when you enable that, you can allow cross-origin by all domains.
-
24:18
Or certain domains and unless you want everybody using your code,
-
24:23
then it was for internal purposes.
-
24:25
And so you say just use this on my website.
-
24:29
And of course, log all access to your API.
-
24:33
Actually, log all access to your website anyway.
-
24:37
So, want to know a little secret?
-
24:42
Having a universal real time back end saves someone time.
-
24:45
You don't have to worry about the actual backend,
-
24:48
you have your developers focus on the app.
-
24:52
The way to do that is you can either setup a global backend
-
24:56
that all your apps talk to or you can have one that you just install for every app.
-
25:02
It also helps promote consistency by not having to maintain dozens of different
-
25:06
systems and it's real-time.
-
25:10
But REST still has its places.
-
25:13
So I'm not saying don't use REST, I'm saying use REST with real-time.
-
25:17
Just don't use REST for web pages.
-
25:19
And, I had to include that.
-
25:25
>> [LAUGH] >> So
-
25:28
>> There's a link at the bottom which
-
25:32
actually will download a real time API that was developed just for
-
25:38
this conference called Feel Real, Future Insight's live real time API.
-
25:58
Don't ask for a fil.
-
25:59
F I L is how it's pronounced, but fil.
-
26:01
That's it.
You need to sign up for Treehouse in order to download course files.
Sign up