[MUSIC]

All right. Welcome to Building Your Real-Time API.

AKA, how to get away with real-time.

I'm Roger Stringer.

I'm a DevOps engineer, author, speaker and

founder of Data McFly, a real-time app back end.

I work with a variety of languages, but the most common are JavaScript and PHP.

So today we're gonna cover the need for real-time data.

We're gonna compare a non real-time system and a real-time system,

and then we're gonna talk about what you need to build real-time APIs.

The need for real-time.

Real-time data, the notes information that is delivered immediately after collection,

you push data rather than request it.

Normal webpages are not real-time,

you have to refresh your page to see the content.

It's inefficient, if you're sending data from a web,

if from a webpage you want everybody to see it right away, and

you want to make sure you didn't just send something that's already sent.

For example, a help ticket system.

You're replying to a help ticket.

There's another support tech sitting next to you replying to the same ticket.

Real-time data gives your users a premium experience.

Real-time data is the best data.

The help ticket system is again, that sample.

Twitter, heavy users are real-time, most people found out about

real-time web design when Twitter started instantly updating the page.

GitHub does the same process.

Uber makes heavy use of real-time for

their come get me, pick me up, drive me home.

And Google Docs, if you've ever worked with Google Docs.

With other people, they will see what you're typing, and

they will see your name as you're typing it.

Which can be handy so you're not editing the same paragraph.

And Facebook.

Everybody loves sharing photos of their cat.

And of course here, News Feed has the update automatically.

So these samples.

They have a lot of engineers.

Most companies don't have as many engineers, so

how do you build real-time without a lot of engineers?

Ask anybody and they'll tell you that real-time data is hard.

They'll tell you that building systems to sync between that iPhone and an iPad and

a desktop and a laptop, it's hard, it's difficult.

It's really not.

HTTP is request response.

So like I said, the client's respond is blind to updates on the server,

but that's what we're learning to get around.

So there are three data transport solutions.

We have polling, that's your standard AJAX, it's been around for years.

It will send out a regular interval to update the page.

It's not really real-time.

It's not efficient, it doesn't scale,

if you have a lot of people hitting the server, that way, what's gonna happen?

The server's gonna have traffic issues.

Server sent events.

Server sent events works well.

It's kind of Web Sockets, without web sockets.

But it mostly only works in Chrome, and I think Opera supports it, but

it doesn't work with any other browser at all.

They're trying to make it work but it doesn't.

So we have Web Sockets.

With Web Sockets we have two way data push, and it works great for transport,

you can pass data of lots of sizes.

You still need to store your data somewhere.

So data transport is only one tiny part of our problem.

Real-time message passing does not equal data sync.

You still need to have a database, so you'd still need a service for that.

So a quick scenario.

Let's build a non real-time system.

Normal non real-time system can involve a lot of different pieces.

Jason is on his desktop, he refreshes the page to see new posts.

Sorry I wrote that before I knew you were gonna be in here.

>> [LAUGH] >> You

get a lot of people doing the same request response,

it's more traffic to the server, it's not real-time.

Everybody has to refresh to see it.

Sorry, that got cut off.

Then you toss in your databases, your servers.

How are you supposed to keep track of that?

I've been developing and doing DevOps for 15 years.

To me this, I wanted a better solution,

I wanted to work myself kind of out of a job for all this stuff.

Nothing's in sync.

Devices aren't kept up to date with each other.

Rest has its place, and it's great.

But it's not for keeping clients in sync in real-time.

So, let's take a look at our real-time system.

A real-time system still involves a lot of pieces.

You still have your database.

You have your web sockets.

Sometimes they can be separate servers, but usually they're not.

The trick is making everything work together seamlessly.

So Rick is on his desktop.

You see his new post up here, instantly.

So does the guy on his iPhone.

So does the guy on another computer.

They're working seamlessly.

This is real-time.

With a real-time architecture,

you can use web sockets to push and pull data from your database or

your API, and nobody ever sees the difference.

Everybody sees a new update.

Your friend uploads a latest picture of a cat named Tim, and

everybody sees it right away and comments, and everybody sees the comments.

You can have multiple clients, push and pull, they work great.

That's not hard is it?

It's almost too easy, so why does everybody say real-time is hard?

It's all in the planning.

If you don't have a plan for the proper tools,

what are you gonna do with your API?

So, what tools do you need to properly build a good real-time API?

You need a language.

Node tends to the be the preferred.

Ruby can also handle real-time pretty well.

Databases, do you have LevelDB, MongoDB, and Redis?

We're gonna talk more about LevelDB, rather than the other two.

You need a real-time transport, that's web sockets.

And you need something for security, so you have API keys, JWT tokens,

and IP domain white listing.

Doing all this through clients means that you can,

if you're doing something internally, you can say this is only accessible

within my IP, within my sub domain.

Within my company and nobody else can do remote access to it.

And you need a client side library which Browserify gets handy with that.

So why would you use Node?

Node is JavaScript.

JavaScript's everywhere.

If you have a website open on your computer, you're using JavaScript.

Our presentation uses JavaScript.

It's code reuse is minimal.

Same programming culture client and server, and

there's a lot of JavaScript programmers.

It's a vibrant community, there's over 60,000 packages on NPM, which NPM

is kind of like Gem for Ruby and Composer for PHP.

Almost any solution you're looking for is on there.

Top corporate sponsors and there's a lot of code.

Do not ever make the mistake of searching just JavaScript on GitHub.

GitHub does not like that.

They will probably send my an angry email for suggesting you do that.

Some sweet spots.

Node is perfect for real-time, high concurrency apps.

It scales well.

It's perfect for building API pages, single page rich clients.

It's got service orchestration.

And it works great with JSON document databases.

And JSON is kind of the standard for

how you store your data, with most databases in Node.

Not so sweet spots.

Frameworks, they vary.

You've got Express.

You've got Happy, which isn't very happy.

There's Eskimo.

In the end, most people build their own or go with Express.

It's not as fast for CPU intensive stuff.

But it works well for doing clustering and breaking off children,

forking off children to run in the background and do work.

So, some really basics with Node.

It runs on the V8 JavaScript run time engine,

which Google developed, and which is.

Really fast.

It just helps you're absolutely fast, it's event driven.

Non blocking standard libraries, most APIs speaks streams.

The, so instead of just grabbing a variable called name,

you can stream your input and it helps for larger input.

Someone's passing a 20 page document they wrote in Google Docs.

Using stream grabs that very easily and

helps you parse it without running out of memory.

A lot of the extensions have been built with C,

C++ to help get into the byte code and help be faster, and

the package manager It helps find solutions really fast.

What node is not.

Node is not a framework.

It's not like Django, or Rails, or Laravel.

It's a language.

Node is just a name for server side JavaScript.

It's not a new programming language.

It's older than most programming languages actually.

JavaScript's been around longer than PHP, all those.

Standard libraries by default is asynchronous, which works well for

dealing with your event room clients.

When you make a call to just slash API slash.

Users or slash followers, you're only dealing with that code.

But you can get into callbacks.

Low level, HTTP is by default, there.

So then, we look into doing sockets.

You have developing and debugging.

You have a lot of different choices.

Nodemon, node inspector, and surprisingly Microsoft Visual Studio Code,

the new IDE that Microsoft released, has a really, really powerful debugger for

node built in, which was surprising.

But it's definitely a really good editor for Java Script.

Common modules, prevents pollution of the global scope.

You have two styles of module development.

When we talk about Browserfy, you'll see that you can build your code and node,

run Browserfy, and you're running it in JavaScript for the client.

It's really handy.

NPM, you have node's package manager, some of the ruby gems.

Handles and resolves dependencies.

There's over 60,000 modules, there's more being added everyday.

It's really handy, you just go install and

it installs your module, then you can build your apps.

So this is a web server and node.

It uses a module called Express, it assigns it to port 5,000 and

displays a page called Hello World.

So that's how quick it is to build a web server.

Persistence.

So to be a real time back end, you need data.

You need to store your data somewhere.

You need a database.

A database is a tool for interacting with structured data,

externalize from the core of our application.

Popular node.js options.

You have SQL libraries that are supported, but they're not that popular and

node kind of wraps around a whole no SQL style of data basing.

There's a lot of community support around postgres because

postgres has really support for JSON.

Document stories are more popular.

Good fit for JS.

Document stories are no SQL.

So Mongo DB, LevelDB or Coach DB are popular primary data stores.

Reddis is also popular for in memory data storage, but

Reddis likes to expire your data, so it's not a permanent data store.

A lot of people use them for caching.

What we need from a data base, we need persistence, performance.

We need to be able to quickly access complex data.

We need to be able to store JSON as JSON is hyper flexible, and

can be used anywhere.

Sometimes we need shared access, and sometimes we need scalability.

So, don't tell anybody, but my secret ingredient for

building quick databases in node, is a module called Levelup.

It's a wrapper for LevelDB.

Everybody in this room, if you're using Chrome or most web browsers,

you're using LevelDB right now.

It's a library that was developed by Google.

It's used for index DB,

local storage, all that stuff that's built into your browser it uses Level DB.

So it's highly in use, it's fast, there's no server involved.

Just a key value store that can contain any date.

LevelDB is a fast key value storage library, written at Google,

that provides an ordered mapping from string keys to string values.

Server less, stores JSON.

So basic features.

This image pretty much shows what the data base looks like in LevelDB.

T1 references one base of data.

T2, rather not showing data, T3 is a smiley face.

It's happy data.

Basic features for LevelDB, you have get, you have putt,

you have delete, you have batch, and you have an order iterator.

Your data goes in the same way you enter it.

It helps to add a number sometimes to that.

i keep going the wrong direction.

So, LevelUp is the binary library for LevelDB.

So let's look at how you build a database in JavaScript.

This is building a database in JavaScript.

VarDB level your database.

DB putt user one for the ID, post Roger.

And then we did a DB putt user two, Martin,

and then we told it to do a DB get on user one.

And return that.

That's an entire database now, and note what LevelDB.

There's no x or query in it, it's very basic key value.

One other feature that LevelDB,

LevelUp has that's handy is, Readstream.

So Readstream is kind of like a select all.

When you create a Readstream, you tell it to start by key.

So you can start at user underscore one and

end at user underscore ten, and it will do a stream.

It's an asynchronous stream In this example,

I'm doing an alert for the JavaScript, for the JSON string.

But you can have it process.

You can have it do whatever.

You could even have it delete that user if you wanted to.

Strings let you iterate through multiple records,

perform actions based on their value.

LevelUp also has a very strong ecosystem.

So you have a lot of tools, packages that are growing every day.

I mean you can take LevelUp and build your own graph database.

If anybody has ever heard of graph databases.

It kind of is the no SQL style of relational databases.

Be in a relational database.

So, then we want to make our database real time, so we use Socket.io.

Socket.io is a wrapper around web sockets,

which are supported by every browser and true libraries supported in iOS, Android.

All you have to do is make a connection.

They work in tandem with the web server.

The Socket.io documentation kind of sucks.

If you've ever tried to read documentation at the Socket.io webpage, don't.

Go to the wiki instead, it works better.

Key features, client server push.

Isn't that what we wanted with realtime,

it broadcasts, we get client session information.

We can do name spaces which kind of translate to like chat rooms.

So you could have Slack, slack uses Socket.io with name spaces for

each channel, and I'm sure a few of you in here have used Slack 1 or 2.

Then you have just your plain old WebSockets API which is,

like I said built into every web browser.

So server, a little bit of code again.

I know I did promise someone there wouldn't be much code today,

but I may have lied.

So with your server it's just your straight forward again, node server.

The difference is at the bottom, it listens for a socket connection.

The reason why sockets work so well for real time, is because once you open

that connection, the connection is opened until the browser or the app is closed.

The long pooling that I mentioned with Ajax,

that connection had to open every ten seconds.

So with sockets once you connect, you stay connected,

unless you decide to kick everybody off.

It's handy for doing all your real time stuff.

You can pass your data.

I've never hit any limits to how much data you can pass through a socket,

but I'm sure there is somewhere.

Then you have client, so the client site is just an HTML page.

It tells it to connect to local hosts, and then it says, okay,

once we're connected do it dot on.

Dot on is.

Unfortunately some of the wording isn't as good, but

dot on basically means performed as an action.

Oh sorry, listen for this action.

It's a listener, so you're listening for when news comes back.

And then on the server side, when the connection happens, we do a dot emit news.

So we're sending back hello world.

And then once we receive that we send an email, which is a trigger,

emitters are basically this is how we send data over a web sockets.

We're sending an emit for my other event which contains the data we just received.

And then on the server side we have another listener for my other event which

takes the data and displays it these two work this in real time.

So then we wrapped up with level DB, and when we can receive my other event,

we can take that data, store it in a database.

You build a real time API.

And now, let's look at clients.

So, Browserify Is really handing because with Browserify,

if you write code for Node in JavaScript, you require files.

You can require Socket.IO client.

You can require all that stuff.

You hit Browserify, it'll convert the code that you're running as a Node client.

Into a JavaScript file for your browser.

It lets you make use of MPM modules.

It's, like I said, best to use for building

code that you can use everywhere and don't want to have multiple code bases.

Why would you want to use Browserify?

Helps you write modular code.

No futzing with client side script loading.

Share client and server code.

Have the same code style package management and module system as server.

Why have to different paradigms?

Some companies it may be fine if you have a large number of developers.

But if you're just a couple of developers Browserify works great.

Actually, I work with a company with almost a hundred developers and

we use Browserify all the time.

So, large developers use it too.

Fun features It provides browser shims for most of the node core.

It can provide modules with alternate entry points when used by browserify.

It creates source maps.

One word of warning, if you don't use, use stuff to minify your code, it'll get big.

And security.

So security.

There's lots of ways you can secure an API.

I generally go with API keys and secret keys so

they can only be used by authorized users.

And can be used to set permissions, read only, read write.

You can let them view all the data they want, they can't delete anything.

I had one person once request to be able to

have a key that could only delete stuff and not read anything.

We never set a key up that way.

You can use JSON Web Tokens, also called JWT.

They're kind of like a session.

It's basically an encrypted JSON A JSON document that gets

passed between client and server, that contains information about that client.

It works on by using public and private keys, so

only your client can decode that key, that was assigned by the server.

And the server can look at it.

And so

it really works as kind of a hash that tells you this user's allowed to do this.

And then you can waitlist IPs and domains.

This one works really well.

If you When you're building APIs, you can do this accidentally.

You can run into this all the time, because of something called CORS,

Cross-Origin restrictions in browsers.

So certain API requests through a browser will fail, unless you it has CORS-enabled.

And when you enable that, you can allow cross-origin by all domains.

Or certain domains and unless you want everybody using your code,

then it was for internal purposes.

And so you say just use this on my website.

And of course, log all access to your API.

Actually, log all access to your website anyway.

So, want to know a little secret?

Having a universal real time back end saves someone time.

You don't have to worry about the actual backend,

you have your developers focus on the app.

The way to do that is you can either setup a global backend

that all your apps talk to or you can have one that you just install for every app.

It also helps promote consistency by not having to maintain dozens of different

systems and it's real-time.

But REST still has its places.

So I'm not saying don't use REST, I'm saying use REST with real-time.

Just don't use REST for web pages.

And, I had to include that.

>> [LAUGH] >> So

>> There's a link at the bottom which

actually will download a real time API that was developed just for

this conference called Feel Real, Future Insight's live real time API.

Don't ask for a fil.

F I L is how it's pronounced, but fil.

That's it.