[MUSIC]

[MUSIC]

[MUSIC]

[MUSIC]

[SOUND].

[MUSIC]

Thanks everybody thanks for hanging around I know it's dinner time so,

I'm gonna talk really fast cuz you're probably hungry I'm probably hungry plus I

don't, I only have one gear.

So, I can only talk one speed I want to talk about scale-oriented architecture so,

we really like repurposing acronyms at Run scope because so

service-oriented architecture is boring and enterprise and so 2005.

And so, like, we thought man We could really spice it up by redefining it

okay let me tell you a little bit about what Run Scope does because it's, it's

actually sort of important to understand what we're building and the tools that we

offer at a very high level I'm not going to try and sell you anything.

To understand why we built the infrastructure behind it the way we did so

we let you do three things, log, monitor, measure your API usage to

solve API problems fast and so what we're trying to do is help developers who

consume APIs understand the performance and characteristics of those API calls.

So they could build better applications that's essentially what we do

there's three ways we do that logging, Keith's just showed you that.

If you were in this room you can get a nice log for all the traffic that, for

all the calls you're making monitoring, we do ongoing API

monitoring through product called Onescope Radar that hits your API as often as

every minute from nine locations around the world or from behind your firewall.

And then metrics which is our newest usage and

performance reports for the API consumption that your app uses so

over 20,000 developers use Runscope we launched just about 18 months ago.

So it's kind of crazy 18 months for, for traction and

it's been growing very quickly in fact, here's our, our graph for traffic.

I used to show this graph and I used to cheat, and I used an [UNKNOWN] graph,

which is really nice cuz it, it's always up in the right no matter,

you know how little traffic you had it's always going that way.

This is now actually our weekly total graph so you can see that

in the last couple months traffic has really, really started taking off.

And, and, and

all of these sort of early infrastructure decisions that we made are really starting

to come into focus very quickly as we've been starting to grow very quickly.

Let's go back to the very beginning of Run scope so

it was 2000, late 2012, early 2013.

My co-founder and I, we both worked together at Twilio.

He was on the API team I was on the evangelism and pr, developer experience

team and we started building Run scope and what we started out with was essentially.

If you can read this two components to start with so there was OneScope.com which

was the dashboard for where everybody would interact with their account data.

And then there was these Run Scope urls.

So Run Scope url is a special url that every traffic-

or every request you make through it gets a snapshot taken by Run Scope but

then sent to the upstream API and then, on the way back, we capture the response.

And then back to your app.

So you don't make an app changes,

all you do is swap out his host name and you can start getting an inspection for

any API, any language or framework, any environment, that sorta thing.

Well, these two components obviously needed to talk to each other.

Once the Run Scope URL captured traffic, we needed to get it into.

The Run scope dashboard and so, we started building some services and

the first two that we built are Identity and Request Vault.

So, Identity keeps track of people and Request Vault keeps track of requests so,

that's all of the API track that's going back and forth.

Excuse me and so when we first started building this you know Frank was

working on [UNKNOWN] and I was working on Dashboard.

We needed to start talking to each other and

I actually really, really hate databases, like a lot I mean,

I've been writing SQL since SQL Server 6.5 and I can, you know?

You know, sling SQL with you know the mediocre best of them [LAUGH] and

so, but I just really hate databases.

But I really Like API's in fact, in 2010 when I started this or 2008,

2009 when I discovered Twilio it really like altered my view on

how software should work and how different components should take to each other.

And I really fell in love with that really simple unified interface that HP gives us

with a really well designed API and so, what we decided to do was for

each of these two services we were just gonna put an API in front of them.

So, we had an API service for identity and

an API for request fault Identity was backed by PostGres and

Request Vault was backed by Retis at them time which we'll come back to in a moment.

This worked really well I got to build dashboard I can actually sort of

build you an API contract that we agreed to ahead of time I didn't have

to wait to, till the data was in there you know we just sort of figured out what

the interface should be and started talking back and forth.

And I built Run Scope again, only using API calls and to this day almost two

years later from that point Run Scope still only makes API calls it's

never talked to a database directly but then we started adding components.

So we added a public API so you could interact with your data [UNKNOWN] so

that started talking to these two services as well use the exact same thing,

identity and Request Vault.

If you imagine that there's arrows between all of the things but

it gets kind of crowded if I put it on a slide.

But imagine there's another set of arrows there between API.Run Scope .com and

the ones in the middle and then we added a new service to handle customer

lifecycle events and that was just another API that talked to everything.

And pretty soon we just started adding services and adding services.

Each one with a small job to be done, right so Courier sends

emails externally FileCabinet manages file storage for, for data.

[UNKNOWN] Service eventually took over a bunch of parts from Identity and

handles things like SAML, that sort of stuff so we have all these services and

then we started adding more consumers like our internal admin.

We added a remote proxy so

you can actually Run Scope urls in on premise in your behind your firewall.

We also expanded to run it from one amazon service region to nine around the world so

that piece was like needed to be very independent.

We added something for handling web hooks.

And then we ended up doing another un-promised test runner pretty soon

the whole thing had really blossomed into like I guess, sort of behind without us

really even paying attention to it, it really turned into something big.

In fact, to this day, err, now we have over 40 internal services and

only seven engineers.

Our service to engineer ratio has been greater than four at

almost the entire length of the company time the company has existed.

So we have lots and lots of little services,

all focused around each one does one job to be done, and does it well, and

uses the technology behind it to sorta that best serves that service's job.

All right so let's talk about,

I wanna talk about two ways that we took this infrastructure and we used it to,

to build scale or infrastructure the first one is one scaling the technology,

the next one we'll cover in a little bit is scaling the team so when we scale

the stack there was a bunch of benefits that we started to get from this so

called micro service architecture.

I know micro services has now reached peak buzz word where it

means everything to everybody and nothing to everybody all at the same time and

so when I say it I think what I just described to you is that loose network of

small services all serving a single job.

That's how we talk about micro-services, and so

when I say that, that's what we mean.

So we started to get some really good benefits for,

from this micro-service architecture and

the first one was independent deployability so the previous,

one of the previous jobs I had was a large monolithic Ruby application.

Something you're probably all familiar with, and every time we wanted to play

a change, somebody would literally yell deploying deprod now, and the whole office

would know to stop committing changes for a moment to master, and then deploy.

How, how, how familiar does that sounds to people like, maybe it's not a verbal yell,

but maybe a hip chat or a slack message that says hey, stop shipping stuff right?

And we've never really done that all of our sort of we're deploying now

have been informative and

not so much a warning shot to not screw things up or to not break the build.

Uh,one of the first things I also told Frank when we started was as much as I

hate databases, I hated deploying from the command line I'm like an okay developer.

I just don't feel like I should be a command line expert in order to push code

and so Frank built a service called Prometheus.

And so Prometheus is our internal deploy tool and

there are a couple of interesting things on this slide here

you can see there's a list of all of our GitHub branches.

And we're in our realm, out test realm here, that we run as a mirror staging and

then the branch is here, so we can deploy any branch.

You can see build status for any given branch.

And the that's summarized up here.

And you can see what the last deploy was, who did it, what got deployed.

And we'll get a deploy history in a moment but

I have never deployed from the command line so I've,

I've only ever used Prometheus to deploy Run Scope tools and what I can do is,

I can go in and say all right, I need to ship a new email template and currier.

I commit that to currier.

I go on to Prometheus, deploy currier to test.

Test it, deploy to [UNKNOWN] and nobody else knows the difference nobody else had

to stop what they were doing cuz they weren't likely working on

email template code and we can ship that very frequently in fact.

Across everything we deploy more than 30 times a day.

We've deployed over 15 times a day from ba, going back to when we were two people.

So we've sort of maintained that velocity as we've,

as we've gotten bigger but Prometheus gives us a lot of other nice things too.

So you can see when somebody else is deploying, so as it's in progress so

you don't both sort of step on each other.

It actually prevents you from doing anything dangerous now so

if I hit deploy while one is running it'll just say, sorry there's one running.

You have to wait for that to finish and then the other thing we,

you can see the console output as its deploying.

And that shows host output from all of the EC2 instances that, that is being deployed

to and so, if something goes wrong it's really nice to have the raw console log.

To really dig into what happened in,

in the deploy we also have like one-click re-deploy as a roll back.

So, if we deploy something that breaks.

We just, we have like this company policy that as soon as you

think something's wrong.

You should immediately roll it back and then figure out what happened.

Don't try to assess the situation with something that's potentially broken in

prod but you can go back to any point in time essentially for any deploy.

And commit that we've ever made in GitHub and

roll back to that at any given point all right.

So we've got really good independent employability.

This allowed us to move faster.

We also started to get really good service-level isolation.

So this used to say modularity in you know,

I got dinged by people who were like hey.

My DLLs are modular, and hey.

My JARS are modular, and and then they are, and your gems are too,

and I don't wanna take anything away from that.

So what I'm saying when you get isolation, you get to do things with your sort of

modules that you wouldn't necessarily be able to do with a code base module.

So we swapped out a major component that we were running that was written

in Python, we rewrote it in Go, and now we've gotta go to production with this,

and we, we consider ourselves infrastructure.

And so we can't go down we're trying to achieve 100% up time SLI across all of

our, our proxies.

So shipping a new huge change like that is not something that we

can just say alright we're gonna do a scheduled maintenance window.

Your API trap is gonna break for 20 minutes while we,

you know, try to deploy this new go thing it had to be done, gradually worked in.

Because this change was sort of isolated at the service level and, and

encapsulated behind a HTB interface, we could use existing HTB tools that

are really good at sort of phasing things in like load balancers.

So what we did is we brought up the new, the new nodes with the new code on it.

And then behind a load balancer we would bring up just 5% of traffic and

then shuttle that over the go instances and make sure that was working.

All right, that worked, let's do it in another region so

we then we ended up with 5% across all of our regions.

And then we just slowly brought the traffic up watching,

making sure that we were getting the exact same responses and

the exact same characteristics that we were expecting from the old code.

We ran them both for

about a couple days after we got them both up to 100% just load balancing across them

all they did on the backend was talk to an API so it really didn't matter.

There was no sort of work to for getting data into the system once we collected it.

Once everything was good, we just pulled Pythons one out,

nobody knew the difference.

So we did a completely transparent major rewrite of our most important component,

by keeping that, those services isolated, and

letting us move it in sort of in pieces without affecting everything else.

[SOUND] We also get really nice Independent Scalability.

So if you go back to the database thing, so our first sort of

naive implementation of request fault was just dumping everything in Rattus.

Rattus is good for a lot of things but, pretty soon you start trying to do

relational things with it and it doesn't really work for

that so well since it's not really what it was designed for.

And so what we did is we actually realized that Postgres would be good for it, but

we just had to run Postgres with a little more horsepower than we were used to.

So not only did we rewrite request fault to switch over to Postgres, once we

did that we were able to scale up request fault independent of anything else,

right so we put a little bit more horsepower behind it.

Identity is now our most popular internal service it makes 100 of millions of

calls a month, mostly all internal.

And we're able to scale that without scaling courier along with it

courier makes a couple thousand calls a month right?

And so we throw more instances, more hardware at the services that need it and

we don't have to worry about how that like wasting resources on,

on part of the stacks that don't need it.

[BLANK_AUDIO]

All right. We also get network resiliency.

So, a lot of the services are distributed across availability zones in Amazon.

So, again using HP load balancing it's really nice that you

can spread out the load for the given work.

Across even boundaries like service providers.

So, we run a couple of proxies in rack, in Rack Space.

And if all of Amazon's service regions went down we

would still be proxying your traffic through Rack Space.

And we hope to add, you know, Azure and Google Cloud Compute Engine when ever it's

called and Soft Layer and Digital Ocean and all these others in the future and

get complete resiliency across service providers.

Even though it's unlikely that Amazon, Amazon's never lost two regions at once.

[SOUND] So, my ops guy will, you know, be upset that I jinxed that into happening.

All right so we get a lot of sort of like technical benefits from the ser,

micro-service architecture, that's allowed us to keep moving quickly, and

to keep up with the load as we're going.

But we also get a lot of human benefits sparkles and

we've been able to scale the team a lot faster as well.

So we're 13 people now but we, you know 18 months ago we were two people.

And we wanted to make sure that we kept that sort of initial startup

velocity as we kept adding people, and as there were more moving parts, and

as there were more code and more component scheme ship.

So here's a couple benefits that we've got on the, on the people side.

So the first one is that it puts us in to a network mindset.

So we work with APIs and the first rule of distributive systems isn't probably this

officially, but the network is fallible, right the network is unreliable.

And that has never been so true in our

lives until we started building all these api German applications.

I'm gonna talk more about that in a little bit,

but what when you start building behind all

these services you're constantly thinking this call is going to fail.

It's not, not when it fails or it might fail it's going to fail at some point.

And so you start building your applications with this sort of fallible

network in mind and would build a lot more resilience in the system by default.

When we talk about smart client later on,

that's one of things that is really stems from our thinking that every call we

make t to a service could fail at any given point, and we wanted to like,

build a client that thinks about failures as well as successes sort of equally.

The next one is it helps us isolate breakage so if I break courier, if I ship.

Bad code to courier which is again making a few 1000,

sending a few 1000 emails a month.

Its not gonna take down the whole system.

In fact all of the dependents servers to courier will just send a message to it and

be like if courier is broken they'll just keep retrying until it works again.

And so when I break code, cuz as the CEO who

still codes I'm the most likely culprit for breakage, it really

minimizes the impact that I'm going to have across this system, especially once

you've sort of built all the systems with that network fallibility in mind.

Some of our new employees who are a little more junior were really,

really scared about their first deploys.

And we were like you know what don't worry about it like you can't break everything.

It would be very difficult to break 40 services with one deploy.

In fact in our infrastructure it would be nearly impossible.

So the first time I, I gave this talk I used this term human modularity and I

promised I would never use it again but I haven't come up with anything better yet.

So this is the third time I've, I'll say that I'll never use this term again.

It's been easier to organize our team and tasks around services instead of ,.

[UNKNOWN] around like having to understand the whole system.

The amount of information you need to keep in memory in your human memory not

computer memory in order to fix a problem is much smaller when,

when the problem is boiled down to go add this feature to this service,

instead of go try figure out the dependency chains for all the calls across

all of the you know methods and all of the consumers of this module.

It really like reduces the amount of,

of in process things that you have to remember in order to ship a change.

And so, we can actually also structure teams around the most popular services so,

our calculon test runner, it processes all the test data.

We basically have a team that sort of sits and works on that, and

as long as they maintain their API contracts and their performance SLAs.

It doesn't really matter what changes they make behind the scenes and they, they

don't have to worry about how that effects the rest of the services around it.

So this, I mean, obviously it aligns very well with the tech benefits of isolation,

but you can also you know, group people around those problems and

services as well.

And my favorite one, going back to why, about hating data, databases,

is you get a uniform interface so no matter where I am working in the code,

I know that if I want to send and

email all I need to do is make an http post to courier and it will get sent.

I don't need to know a bunch of different database,

you know query languages I don't need to know the protocol for

redress versus postgress versus lasic search versus dynamo db.

All I need to know is how to make a HP request and

I can interact with any part of this system from any other part of the system.

And so there's a lot less to learn again for new developers because all

they have to know is one way to access and write, to read and write data.

Also later on as we have devs that only work higher in the stack, this is like,

this just removes a huge amount of cognitive overhead for them to try to

understand how to ship a new feature or what technology to, to use to build it.

All right, so we spend a lot of time investing infrastructure between

Prometheus, between all the, [UNKNOWN] orchestration system,

our atlas realm, config centralization, that sort of stuff.

And without that we really wouldn't have been able to get all the benefits that we

got from these micro-services.

So I would say if you're not willing to invest in the infrastructure to

make this possible then don't invest in micro-services.

Most of the complaints I hear are, I don't know how to deploy 40 service

using [UNKNOWN] Isn't there a lot of management overhead for that?

Isn't there a lot of operations overhead for that and there absolutely is so,

if you're not willing to invest in automating that and

abstracting that away from, from your developers.

You're probably gonna have a harder time getting value out of the micro-services.

All right, so, so far I've talked about internal API's for Run Scope.

But if you look at an application from a broader perspective,

that's really just one component of what we do right so

I've mentioned this courier service bunch that talks to mandrale.

We have a bunch of other services that talk to other third parties.

And ultimately we have all of these different pieces that sort of make up

our application experience right?

We have the third party APIs we depend on we have our we don't have native apps but

imagine you know, we were shipping a customer [UNKNOWN] stuff.

We'd have mobile apps, we'd have desktop apps, we'd have a website.

We'd have sensors and hardware dumping data in the system.

All of these things are talking to each other over APIs and

it's really like these APIs oh, not the slide I thought it was gonna be.

Anyway, get to watch the animation again.

So all of these pieces really encompass what the,

the technical components or pieces of the actual user experience.

It, with any one of these pieces not talking to each other or

not working well with each other, you get a bad user experience.

So this was never more true for me than when I worked at if this than that.

So if connects services you use you can do things like when the U,

team USA wins the gold medal then you know, turn on a light, a disco ball, and

play music, right?

You can do all sorts of hardware hacking in combinations you can

combine different services and ,.

If we got bad data from the Olympic API for that integration,

then somebody's lightbulb doesn't get lit up in the service.

We look bad because we promised somebody that you would be able to do

fun things like that right?

So, all these pieces combined really comprise your application experience.

So, one, once you have this sort of API driven application or

distributed application.

You run into sorta these greater challenges.

So I'm gonna talk about three and

yeah, we'll just start with the first one, all right.

[LAUGH] The first challenge is getting a complete picture of your app.

So we run all those internal services, and I am ashamed to tell you that I

still don't, can't tell you how many API calls we make on the whole every month.

So we're, we're working towards that it's hard to

use our own tool on itself sometimes and, and so we get in these looping situations.

But our goal is to make it so that you will always know how many calls you make,

and to get a, make it more easy to get a complete picture of your app.

Remote services tend to have like an ad,

a really strong adverse effect on your user experience so I mentioned the one

about, you know, not turning on the lights when the USA wins a gold medal.

We had another integration with a social network, every other Friday night at about

11 p.m the responses would start going to about 15 to 20 second response times but

then it would work.

So it, it wasn't like downtime, downtime is super easy to spot right?

Like we, tried to make the call and

we go an exception, it was broken slow is way harder to find.

And so, because this is one of our most popular integrations at Ift.

It would basically lock up all our workers for 15 to 20 seconds at a time.

Something that was taking like normally a couple of 100 milliseconds.

And it would just grind the whole system to a halt and

then all our queues would back up and then everything would break.

And it took us a really long time to identify what it was in fact,

right as we identified what it was the service provider fixed it.

[LAUGH] So,.

But we didn't have any information or really good complete picture of

all the calls going back and forth to even go to them and say hey here's the pattern

we're seeing that its this every other week thing at 11 o'clock on Friday.

So, getting that complete picture is still difficult but

we're, we're trying to make that easier.

Also like traditional monitoring tools don't really get into this very well.

So a lot of like APM tools will tell you that something was slow or broken.

But they don't really consider like what the data was on the wire.

So if you've ever gotten in your bug exception tracker a like, JSON parse

exception that's probably the most common API downtime notification that exists.

JSON Par's exception says nothing about the network or

why that JSON couldn't par us.

But usually what happens is the API you're trying to hit is returning a 503 which

is an eight, in like, typically the default like engine x or the Apache page

which comes back as HTML and your JSON Parser can't do anything with it right?

But you're thinking my adjacent code is, my adjacent parsing code, or my adjacent

library is broken when in fact the remote service is sending you invalid data.

A lot of the APM tools don't really and the error log,

you don't really describe them very well.

So it's important to try to get a better grasp on like, what the actual traffic is

and what the actual responses are and in both good and bad situations.

So our strategy for that is that we watch and

log everything so we do this everything third party, we log.

Thankfully we have a really great tool for doing that.

but, every call we make outbound to an API we, we shuttle that through Run scope so

we can see what's going back and forth.

We were getting ran, really weird what we thought were weird MailChimp errors and

then, cuz we got 500s back and we went and looked and

it turns out MailChimp turn, returns 500 for basically any error.

[LAUGH] If you saw King's talk just before this,

it would be a classic example of how not to use status codes.

All of our inbound web hook requests are also run through Run Scope URL.

So if our server for some reason couldn't process it, we could go to Run Scope and

we can click retry and rerun that web hook and we don't miss anything.

This has really sort of altered how we view APIs even though we

knew going in that this was like really important.

Actually having done it now for all of our third party service integrations has

really made us a lot more skeptical of even the providers that we've chosen who

had good reputations or that we thought had really good reputations.

All right, next challenge in, in dealing with distributed applications is

managing change so, if you've been doing software for more than five minutes you

know that this is not a problem that is limited to API driven applications.

But I'm gonna talk about it from the perspective of, of using APIs.

API version changes are very difficult to change, right, or to track.

Largely you have some HP servers that may or may not be using like versions in

the resources or version content types or media types is unlikely to be notifying

you every time they add something and your app is like dependent on,

on all of these changes not being, being made in a broken fashion.

A lot of changes are made accidentally so a popular social network API used to

accidentally capitalize an attribute name temporarily until they realized that

whatever was generating their objects was doing this.

And that would just cause, I mean, one letter changing you think would not be

the end of the world and it would hose up our entire application cuz-

We were looking for essentially the wrong key and

couldn't find that data but all the rest of the data was there right.

So, that's a change that they didn't intend and that's sort of

like the bad change but there are a lot of other subtle changes too that

companies sort of accidentally introduce.

It turns out so SDK's which is sort of like the way that API providers will

tell you that they are getting around these sort of.

Changing problems by abstracting away the underlying API actually only attend up.

End up exacerbating the problem in the long run instead of solving it.

So, this is probably my favorite about API's I hate SDK's.

They are terrible they're not terrible for everything, but

they're terrible for maintaining software or writing maintainable software.

Which is where you spend about 99% of the time with software that gets out of

the development stage, right.

Is maintaining it.

So I wanna give you a couple reasons why you

should think twice about taking a dependency on an SDK.

okay so let's look at the first case which is managing some version changes.

So this is typical conversation between, your code and an API in SDK.

API might be on version one and the API SDK might be on version two.

And now you're now, now you're watching two versions, right?

So, the underlying API may get new features, but

now you've gotta go check to see if your SDK supports those new features.

And a lotta cases they don't right away.

So now you're tracking the state of two versions.

Oh, I thought there was another part of that slide.

But anyway, you're starting to get compounding version management, and

compounding things that you have to watch in order to,

to, to keep up to date with that API.

This is just case if you're using one.

So if you're using two you're actually introducing a third layer of

version management that you definitely don't wanna be dealing with.

So in a lot of languages there aren't that many HTTP clients or

adjacent parsers or XML parsers.

They're sort of like these core typically open source or

in the core standard library.

Things that everyone depends on, right an HP client is a, is a classic one.

So your code is dealing with APIs.

We would have a version and then SDKs which have another version which may or

may, may not be up to date with the API version.

They are also depending on versions of these dependent pieces that

get shared across a lot of different SDK libraries.

And so what happens is what happens when you have an HTTP client.

With two consuming SDK's and one requires version one and

one requires version two of that client.

Now if you're in node you're probably lucky because you can

do multiple versions, dependencies with different versions.

If you were in Python or Ruby or

pretty much anything else or .net you're kinda screwed here.

So that underlying SDK changes what its depending on,

you now have to go research whether or not.

You can bring, the SDK version up to date, its dependency up to date.

And now this is again, code you didn't maintain.

You're now multiple levels removed from code that you actually care about and

are writing yourself.

And you've gotta start making decisions on can we submit a pull request to

get this updated can we run it with the wrong version can we

override the hard link that it has to a specific version?

Again, none of this is delivering any value to your customers or

your users of your application, right?

You're way down the dependency chain here dealing with problems that you don't

want to deal with.

And I thought this was sort of a rare problem because at IFT we

had 65 API integrations in a GEM file a mile long and

it turns out it doesn't take very long, especially in something like PYTHON or

RUBY before you start running into this problem.

Also what happens when something goes wrong right?

When that conversation starts breaking, is it between your code and the SDK?

Is it it between the SDK and the API?

This is actually very hard to determine in a lot of cases so

we've tried to make it possible to use our tools to see what was on the network so

you can start removing variables.

But this is I can't tell you how many times I've been trying to

debug like where is the breakage happening, is it after I got the data or

is it before the data reached my coder, or which layer is it in?

All right, so SDKs cause lots of problems,

especially if you use more than one, but there are times that they're okay.

So, one time that it's really okay is if you're doing prototyping.

Right? So you just wanna figure out what is

the capability of this API that I'm using, and

will it solve this problem that I have, right?

So if that's the case, sure, pull it in, prototype it.

But I would do that with like a very temporary mindset and

I know there is no temporary in software.

The, you know, temporary tends to be permanent very quickly when you're writing

an application, but if you're prototyping, it can be ok to use an, SDK there.

Another good time to use an SDK is if you don't have a good HP client.

So this used to be like one of the biggest reasons why people made SDKs is because if

you looked at languages like I don't know,

let's say dot net that did not have a very good HP client for a long time.

You didn't want to have to tell your customers of your API, yeah you have to

use this really horrible HP web request API in order to talk to our API, right?

You wanted to sort of make a smoother experience there.

That's why I wrote Rasharp for .NET, that's why I wrote the Twilio C Sharp SDK,

we were basically hiding away HP web requests.

Thankfully .NET has a much better HP client now.

And so that's less important there but there are other languages that

continue to lack good, good HP clients in the standard library.

Another good time to see if you're building an entire client.

So one of the, sort of the modern like restful API STK really got popular.

With things like Twitter.

And the reason it did is because people were building Twitter clients and

they needed every API call to exist.

So if you're building something that is gonna use 100% of an API then sure.

Go ahead and use an SDK cuz you're gonna end up building that anyway.

If you're using some portion of it you probably don't need to carry the weight

around in order to make that valuable or to make it work not using an SDK.

If you're using complex APIs like something that does sync or

something that's not over like HP, you know, thrift web sockets,

that sorta thing, sure, go ahead and use one for that.

Or if you're using native APIs on a mobile app it ties in like, you know,

photo library, camera, or hardware sensors, or that sorta thing, then sure,

pull in the SDK for that, cuz you probably don't wanna write that yourself.

The danger zone is when you're using more than one because of

the dependency management problem.

If they're community-built because community-built SDK's tend to

go stale very, very quickly.

And again if this is like, core to your product and

core to your experience, you don't wanna be reliant on some open source.

Developer who may not care that your app is no longer working, and

that, that doesn't care that an underlying dependency changed.

It's actually easier going back to community-built, and

it's actually easier for somebody making an open source SDK to pin to specific

versions of dependencies to minimize the amount of support that they get.

But they're really just sho, they're basically distributing the,

the, the who has to deal with that to somebody else.

Another time that it's dangerous,

if it has lots of dependencies we covered that or if it's inactive.

So if you go to API provider and the SDK hasn't been updated in I'd say,

you know, the last 90, si, 90 days or

six months I'd be very weary about pulling that into your application.

In some cases that can mean stable and

in other cases it could just mean like, not up to date.

So it's hard, you have to make that, that judgement call for yourself.

If you're an API provider, you actually must create SDKs like,

you cannot produce a public API these days that does not

have good SDKs if you expect developers to get up and running quickly.

And that first five minutes a developer reaches your site is the single most

important time for

their entire success on your platform, and an SDK gets them over that hump quickly.

So, if you're going to build them.

And you should if you're providing an API build them for

as many platforms as possible, as native as possible, they should feel like

the language they're written in don't have a PHP developer write a C Sharp SDK.

And make sure they're as well documented, because what you're saying is hey,

we have an API.

We need SDKs, but now you have 17 APIs.

And if you don't believe me, I enumerated here on this slide so

this is the big six current sort of landscape for which languages you need to

support because those are where all the developers are.

Here are all the version within that that you need to support.

Thankfully RUBY 186 is dead now you don't need to do that anymore so

let's go with 16 and then something like Golang comes along and

now you've got to learn a new language.

And you're gonna have a lot of developers clamoring for a wrapper there.

And if every one of these is not documented as well as your rest API,

then you've created a horrible first run experience for

somebody and you're going to lose that customer.

So, good luck with that.

All right.

It's not fun. I mean,

that was part of my job at Twilio, was maintaining the SDKs and it was.

More time spent managing version and

dependencies than actually adding value to the SDKs.

So I'm gonna breeze through this really quick cuz I'm running out of time

this is how we do it at Run Scope.

So we wrote a smart thing we call Smart Client, and

the goal is to get around some of these sort of problems.

Now, we use this mostly with internal services.

We do a little bit of this with external services,

depending on how the API is designed but in our case what we're trying to do

is avoid hard coded sort of paths to services in there.

You can see the EC-ugly, EC-2 url in there.

We're also trying to make the most common error handing cases sort of work

by default so what we do is we have something called smart Client and

it does service locations so

this ties in with a lot of infrastructure investments that I was talking about.

So that whatever realm I'm in this knows where to find it.

If I'm local it knows it's on local it's 5003 If I'm in prod it knows it's at this,

this DNS entry which is a load balancer in front of like seven instances.

We also try to make as many things as we can item potent I really like that word so

I'm gonna say it about three more times.

So what this does is say hey, if this is down or we couldn't get that resource,

we're gonna retry it smartly, to make sure that we can get that resource

without having to worry about writing a lot of repetitive retrial logic.

And so we like to do [UNKNOWN] creates, all of our creates.

Internally are done through puts instead of posts, so

that we can reissue it as many times as we want, and

the put will take effect when that API responds properly and it doesn't matter if

we sent it more than once, so we get really nice retries on creates.

Our sign up page is our most important page in our entire company, and

if for some reason somebody hits sign up and it doesn't work,.

It's bad, but

if it takes five seconds that's can be, that's like a less bad, right?

And we're, we want that sign-up to go through sorta regardless of

what the underlying problems are.

Then we put these in thin wrappers.

The thin wrappers thing is something that we definitely apply to third-party API's

so that we can swap out underlying integrations.

If we do a new prototype that uses an SDK we try to put a thin wrapper around that.

So when it's time to pull it out and

we're gonna make calls directly, we don't have to change the the calling code.

All right.

So challenge number three is, is testing and monitoring.

So APIs have this unique property that you can test them,

you can monitor them the same way you test them.

Does anybody run unit test on production servers in your production environment?

Right. And nobody does that right.

You run it on your staging server, or

you run it locally, you run it on your CI server.

But once it goes to prod you're not like doing ongoing monitoring or

ongoing testing using your unit tests or functional test or any of that right.

You gotta test it externally somehow.

Well API's are nice cuz you can apply the exact same way that you would test it

which is making HTTP requests against it.

To your local environment, to your staging environment,

to your production environment all using the exact same test plan.

And so, what we did, is we built a product that tries to address both of these cases

and removes this arbitrary dotted line and makes it easy to test in development and

production looking for functionality and correctness in development, but

then in production looking for performance and availability, right so.

Locally, when we're building an API we wanna make sure it's,

it's right, that we're getting the right data out of it.

And then when we go to production using that exact same test plan,

we wanna make sure that it's performing properly and that it's up.

And so we took all these ideas from how to do API testing in a way that you

couldn't really do previously, and we made a product out of it I'll skip that point.

Called Run Scope Radar, and it does all of those things.

It runs from nine locations around the world, or

behind your firewall, or on your local machine.

You can take one test plan and run it against multiple environments very easily.

So, if you're looking to get a, sort of a better grasp on your internal services and

just understanding are things correct, or

are things up, Radar is really a great tool for that.

We run it against all of our services we have 100 and 100 of these radar tests, and

we know very quickly if a deploy heard something or spoke something down.

All right so, I'd love for you to try Run Scope,

you can actually try Run Scope now without even giving me your email address.

You go to our homepage and click the link that says try it now sign up later,

you can get a free one day account that's completely anonymous.

If you do sign up and you do give us your email address I will mail one of

these Everything is going to 200 OK T-shirts.

Just sign up and email me at john@runscope.com and

we'll send it to you.

And if you like working on distributed systems and

you like API's and you like building great developer tools and

helping developers be more productive, runscope.com/jobs.

We're hiring for dev ops engineers and.

Sales people and product engineers and pretty much everything.

We're really fun.

I promise.

[LAUGH] We just played croquet the other day and it was amazing.

[LAUGH] Anyway so, I've got a minute 16 left for questions.

>> Well let's all take a deep breath.

>> [LAUGH] >> Holy crap.

And let's put our hands together for John too.

[APPLAUSE]