So good afternoon, my name is Jonathan Karon there are actually

two Jonathan Karon's on the internet, hate to admit that, one in

Massachusetts who's a lawyer and one in Oregon who's a software person.

I'm the software person and I hate to admit it

but the lawyer actually has me beat on page rank.

So, you can help me out by just going and linking

to Tapester.com right now, that'd be great, we'll solve that, I'll wait.

No?

Okay.

So clearly I don't blog enough but that's not why we're here.

It's Friday afternoon and you've all graciously stuck around for

the end of the conference so that's kind of exciting.

If you're someone who's into mobile applications

or understanding and tinkering with like big

and complex systems then I think this is gonna be a really good talk for

you ,I'm excited, if you're not into that stuff then you should probably just, like,

enjoy Vegas and, you know, all of that, so, no one's leaving, that's good.

So did anyone catch Carl's keynote yesterday?

On the lemonade stand and all of that fun stuff?

[COUGH] this will make sense in a minute.

I've talked a bunch about monitoring

mobile apps and software tuning and performance.

And one day I was out walking my dog in the woods, like Carl told us we should.

And I realized that I'm not talking about the right thing.

That, you know, there's so much more

important to mobile application performance than just

writing optimal code, So I'm excited to give you this talk because it's super new.

I haven't given it before, and it's kind

of like the biggest, newest idea in my head.

So hopefully, this will be like catching a

good punk band before the producers mess everything up.

so, I'm here to talk about mobile performance, performance of mobile apps.

My background is more in native mobile apps but

I think this will apply to pretty much everything.

The first time I did a talk on

mobile performance, a bunch of marketing folks showed up.

And they were trying to figure out how to improve their

campaign conversion rates and deliver more push messages to your pockets.

So that kind of bummed me out, cuz that wasn't what I was talking about.

So let's start with what do I mean by performance?

To me, performance is the creation of a frustration-free user experience.

it's, it's pretty simple really, right, our jobs as app developers is to

make people feel like they are having a really good time using our apps.

so, you know, this gets into how people are happy

with our apps and how not so let's review some statistics.

98% of users say that performance matters.

And the apps where it matters most for those people are things like banking,

mobile payments, e-commerce, and shopping and those

are where all the money is, right?

So the more important the app the more important the

performance is and that shows in the way people get frustrated

by these things.

79% of users will give your app a bad review for freezing up.

And one in five will delete the app forever if

they are left there sitting for more than five seconds.

That is like, some serious performance anxiety, because five seconds is

not a long time, and our users are very, very mercurial.

So looking at retention and good will,

you're actually better off if your app crashes.

Then if it locks up and makes someone wait for something to happen.

Because it makes people feel like you're wasting their time, right?

And there's nothing worse than sitting there,

waiting for your, like, you know, dopamine cycle

to kick in because you got your little treat in the app, and it not happening.

The, you know, but reflecting on these

statistics, I started hearing a common theme.

They don't really have anything to do with code optimization.

Right, as long as we don't exhaust available

memory, people don't care if our apps are hogs.

They don't care whether we're using one core, or six cores, or whatever

the phone has, or if you spend a week hand-coding assembly to do something.

They care about a smooth experience, where

their expectations are met and things happen.

They're not sitting there waiting.

So our job as craftspeople is to prevent hiccups.

Basically to keep things moving at a nice pace.

But how do we know what's important and

what's misdirected effort when we're building out an app?

Right?

Eventually we'll get there with testing and feedback and all that stuff.

But early on is where, you know, you listen to all

of the rules about design up front and you know, test

early and often and, and you know it's way, way cheaper

to make changes early on in a product lifecycle than later.

So when you're designing a product, you have all these great

techniques of like, research and user fast, focused design and, right?

And you can figure out how to deliver the right

experiences for people so that they can get the job done.

We spend describing what an interaction with our software should accomplish.

Right, we create these user stories.

And if you caught Luke's talk this morning, you saw what

an amazing difference of good, thoughtful user experience can make, right.

Fewer taps on the screen, fewer times the keyboard comes up and you

gotta switch from thumb to two fingered mode and all of that stuff.

All right, so the design process and really

thinking through the interaction can make a big difference.

But, as you probably learned, implementing that stuff is non trivial.

Right?

I've been building these connected location

aware mobile apps for 17 years now.

Which is ridiculous.

The first one was an Apple Newton with a cell modem, and I still struggle

with these things right, every time I have an app there's something in it that

I don't realize is gonna be a problem and one person out of 20 calls

me up and says hey, this is stupid and I spent 20 minutes not doing anything.

So even if you're a really great software architect.

You, and you know the mobile devices

really well, you're still gonna get into trouble.

And you're gonna spend all of your time

trying to fix problems that you can't reproduce.

All right, when you could be building something awesome like the

next new feature or going to future insights and hanging out.

So we get to the point.

Mobile apps are a bricolage of different pieces that we put together with code.

You can optimize the code all day long but if you

start thinking about that too early you're just doing everything wrong.

Right, because most of the time the slow things aren't under your control.

You can't speed them up.

But you can orchestrate them and make them feel better.

So there are tricks to eking the last nugget of speed out of a phone, right?

You can optimize your code and do this or that, and think about all

sorts of technical things, but we can get to those another day right, the

first job is to solve the 80% problem of accommodating how things are going

to work under most cases and not worrying too much about those last 20% speed.

So instead of that I'm gonna talk about orchestrating

the symphony, right, you have all these components that play

in the app and your job Is to create some

kind of really tremendous first class experience out of that.

So there's time when an app is slow because someone didn't write good code.

That's happened.

But a lot of the problems are things you can't really solve.

You just have to learn how to dance with that.

You know, the analogy that I sometimes use is that it's like a ugly house next door.

You can do landscape design and kinda hide it and people won't really notice it.

But then the analogy starts to fall apart because mobile

issues don't really sit still like a house next door.

It's more like you've got fifty houses scattered over your lawn.

Some of 'em right in front of your front door.

And they tend to jump around and change shape and color.

And sometimes they knock on the door at 2:00 AM

yelling like, hey I just came back from Vegas, what's up?

[LAUGH] So, a simple app, right?

A simple, simple mobile app.

Can rival the complexity of an entire data center.

This is an app that we built for a conference called Future Stack last year.

Which has nothing to do with Future Insights.

We just overloaded the future thing.

If you step back far enough, and think

about this stuff like a systems management person.

This is what your app kind of looks

like right, it's, it's this broad, distributed, connected system.

And in systems management you start by optimizing for the

through put and the performance of the median case, right?

If 50% of my users are having a good experience at this level,

awesome, that's great, our servers are running,

and our network is not bogged down.

But as we all know in mobile you tend not to talk about the typical user experience.

You talk about the one person who had a really horrible time and is

trashing your brand on the app store and Twitter and all of these places.

Right because it really matters, like each one of

those people as individual and their experience is vastly different.

And you can't just log into their phone and

tune the database or figure out why things aren't working.

So let's zoom in and consider that single

copy of the app, right, running on one phone.

The way the app performs for individual users is really variable.

Well why is that?

The bricks that make up the app have a range of characteristics.

Right, you know, on the one hand you have these, like multi-core

high performance CPU's, that are really

great at crunching through certain data sets.

And, you also have three year old commodity phones,

that have a tenth of the power of that, right.

So there's a, there's a huge breadth in how quickly

you're gonna be able to accomplish any data processing task.

And then you have fairly slow flash storage and phone

components like the camera and the screen, which have inertia.

Right.

They take time to, to come up to power and to update.

And then at the very far end, you've got really slow cellular network connections.

Right, you've got Bluetooth.

And then you end with these large data sets

that you have to deal with because your building

modern connected apps and people are sharing video and

downloading thousands of bits of content and this and that.

And these aren't things you can tune in your code right?

They're just facts of life and bits of reality.

But they are things you can reason about and design for.

So how do we do that?

Most of this stuff doesn't really measure consistently.

Right, you can build a great user experience, but how do you compensate

for something that could take a 100 milliseconds, or it could take 30 seconds.

That's not the same thing as tuning code.

My answer lies in this discipline called systems theory.

And the way it's applied in practice is typically called systems modeling.

So, Russell Ackoff, who is an operations theorist.

Once said that, managers don't actually solve problems, they manage messes.

And I have a suggestion.

Which is that this holds the same for mobile developers.

That, as app developers, what we do, is we manage messes of pieces.

So, as a software engineer, you're conditioned to

think about processing data with this reliable, testable process.

You know, predictability and building things

that are easy to understand and test.

But if you're like most mobile developers I

know, that's like the first half of your job.

All right, the second half involves tweaking your order

of operations of things, and fiddling with poorly documented API's.

And generally poking your app with a stick to try and make it do what you want.

You want consistent and reliability out of your flow

of events but that's never ever what you really get.

So that's what i mean by a mess.

Does anyone here now about Six Sigma?

Okay, it's a management practice.

And there are all sorts of courses and books and things that

you can take to you know, learn about it, but I am

gonna reduce it to two really simple goals and probably get my

butt kicked by the Six Sigma black belt guy in the back.

first, reduce the standard deviation and then increase the mean.

So what does that mean?

It means first focus on reducing the variability in your process.

Then once you've achieved consistency focus

on increasing the speed of the process.

Because you can't reliably improve something that's

out of control .That's just optimizing the mess.

It doesn't work, it ca, it just, you know, you spend

a lot of time and energy and at the end of

the day there are still people out there who are just

frustrated because things don't work the way you think they do.

So this is where systems theory comes in.

Using systems modeling, we can address the variability problem head

on before we start actually designing our code and writing things.

So systems theory is a science for

modeling and understanding pretty much any dynamic system.

It turns out that you can model almost

anything natural mechanical whatever with four basic elements.

You've go stocks, flows, feedback and delays.

[BLANK_AUDIO]

So let's step back and talk about systems.

Every system has a purpose, right?

A restaurant is a system intended to put food

in your mouth, or put you to sleep at lunch.

A football team is a system intended to score goals and enrage fans.

And the mobile app, I'll call it a collection of systems.

You know, each of those systems is intended

to deliver some bit of functionality to your user.

You know, there's a system for every user story, right?

This, we call them flows for a reason.

And using systems models, you can help build reliably performant mobile apps.

But first let's simplify things a little bit, because it gets complicated.

Feedback and delays, we can let those happen

during testing and, and user acceptance and revisions.

So we won't worry about those in our initial modeling.

Which leaves us with stocks and flows.

So what are these things?

Stocks are the content that our app processes.

Okay, essentially buckets full of data.

Flows are the algorithms, or the services, or the system components,

or the IO operations that act on the stocks of data.

Think of them as pipes right, they handle the inflow of data

into a stock and the out flow of data out of a stock.

A thin pipe will take a long time to fill or

drain a big bucket, whereas, a collection of thick diameter pipes.

Can move that data much more quickly.

So we can model a user activity in an app as a system of stocks and flows.

Why?

The goal is to estimate the overall relative complexity of

each piece so that we know how it's gonna impact the overall experience of the user.

We can represent the complexity of data with different sizes.

The stocks the data we're processing are defined by the

size and complexity that we expect that data to have.

Alright, so text, a pretty small thing.

A text message is what?

A few hundred bytes.

It's simple, a video is gonna be big, it's gonna have a lot of inertia, it's

gonna take a long time to move through our app if we're doing anything to it.

And then something like tweets are pretty small,

right, but they're structured data, there's a lot

of meta data in there and when you go and fetch a list of Tweets from Twitter.

You may get hundreds or thousands of them.

Right?

That implies more work for us than just a simple block of text.

So we'll kind of put that in the middle as sort of a medium sized bit of data.

And then we represent the flows with thickness, like pipes.

And parallelism as multiple pipes.

In a, in a group.

So things like system API's, and data processing, and communication that

we orch, orchestrate, right, these are all different kinds of flows.

And we can size them appropriately.

You know, cellular connections, especially when

you're on marginal, or 3G, or whatnot.

They're really small, right?

You can sometimes get three, maybe four concurrent network operations in place

before things really start to collapse and they can take a long time.

Lot of latency.

File I/O, kind of in the middle, right?

Writing to a file can be pretty quick, but you can't do

a lot of it at the same time or you start thrashing.

And then CPU well, you know, these chips

that we have these days can be pretty fast.

So, you know, we'll model a four core CPU as

something with a lot of ability to more data thru it.

So let's deconstruct a simple user flow.

The act of taking and tagging and sharing a photo, something that a lot of apps do.

In this flow the user takes a picture, adds a title, selects a

place where they are, and then shares it to a website or something.

All right, but there's a lot more going on underneath this.

In there you know, you're gonna use a

GPS to get your location, we're gonna connect out

to Foursquare to get a list of the places

so the user doesn't have to type everything in.

We're gonna maybe apply an image processing filter to the photo and then

you've got to upload it which we know could take a long time intuitively.

So the stocks in this case, right, you've got the title of the photo,

you've got the place name, those are small things, the image could be pretty big.

And then this place list, right as I was saying you know,

that structured data, you could get a hundred places back from the API.

So we'll make that kind of a medium sized thing.

And then the flows.

All right, again the flows are the ways we move that data through our application.

So there's camera access, which.

Is sometimes kinda bulky and slow, but usually

not too bad, especially on a newer phone.

GPS takes a long time to lock, right?

And so remember that the thicker this pipe, the faster it's gonna execute.

Place look-up is a network request, could take a little while, but it's

a small amount of data and, you know, Foursquare tends to be pretty quick.

Image processing, well we've got the whole CPU and we've probably

got a system library already that's pretty optimized for doing that.

And then the image upload again, probably gonna take a while because

it's a big image trying to move it over a slow cell connection.

So when you put all that together this is what our image sharing flow looks like.

Right?

I know some of you heard the phrase image processing and

you immediately started thinking like oh man, that's gonna take a while.

We're gonna have to work on that.

And it might.

Right?

But is that the right thing to focus on first?

Let's first start by improving this map.

Right?

And make it easier to reason with.

What I want is more like a visual timeline.

That gives me a reference check for how long this stuff is gonna take.

So we'll do a little research and measure you know, how long

does it take for a camera to start up on the phone.

How long does the GPS lock take under different scenarios?

And then i'll pencil in approximations for how long those

things will take and scale the widths of those flows accordingly.

All right, and what I realize is that this

is a really long, slow, and frustrating process for someone.

And I won't stand here and act out the whole

minute and a half of it cuz then you'll just leave.

Or beat me up.

[BLANK_AUDIO]

so, I've proven that this user experience is probably gonna suck.

Right.

The good news is we've done that in ten minutes

instead of going out and writing code for ten hours.

But then what do we do, right?

So I can quickly look at the model

and start reasoning through the trouble spots, right.

The image is big, it's way at the

beginning of this flow, that means I'm gonna have

to move it through a lot of parts in

my application and it's gonna take a long time.

But I also need to pay attention to that places list.

It's gonna take a while to get that on to the phone.

Off of the network and the GPS and the upload flows are gonna take a long time.

And in this case you know, I don't really need to

worry about this image processing step cuz I can barely see it.

Even though it's half a second.

All right, it's not the low hanging fruit in this case.

So now that I can visually reason about the system I can start improving it.

How do I do that?

The system model makes it easy to map out the dependencies in this flow.

So let's break it down into dependency chains and

start seeing how the different parts of this flow interact.

first, there's the image pipeline, which I'll code in blue.

And I apologize if anyone is color blind.

And if we had some of those cool color blind-correcting glasses

they were talking about this morning, I would hand them out.

We've got the camera, the image

processing, the uploading of that data, right?

Those all have to happen in order.

So they are dependent on each other and they're operating on that big blob

of image data so i worry about that whole chain taking a long time.

Then there is the place selection process.

Which i'll code in green here.

You've go the GPS and the Four Square

requests and those have to happen one after another.

And then the user can't actually pick a location.

Until we have that stuff.

And the photo title can stay off by itself because

that's just gonna be a text box that someone types in.

So by separating these out into

independent sub-processes, I can start optimizing things.

In fact, I may be able to start some things,

like the GPS, before the user even enters this flow.

Right, you can start getting a GPS lock as soon as they

launch the app, and because of the way people use apps it's

more than likely that you know, if they don't actually share anything,

okay, you know, we got a GPS lock, and throw it away.

If they do share something they're not gonna have moved very far from

where they fired up the app 'cause they'll close the app in the meantime.

If they're actually going to a different place.

So in this process, we've also discovered some hidden dependencies, right?

There are stocks of data that are orphaned in this flow, right?

The title and the location don't have any way of getting up to our server.

So, with a system model its easy to reason about those and I can add a

couple of little short upload requests to make

sure that data hits the server or I can

go one step better and bundle the title in with the place name and do one network

request that does both of those things and

initiates the share if the image is already uploaded.

And something magical just happened here.

That I didn't even anticipate the first time I went through this process.

Because we just defined the server API that our app

needs in order to share a photo with your friends.

Right?

We didn't have to think about it, it just happened,

because we can see which data goes into which network request.

And when it's gonna hit the server in the life cycle of this.

So, this isn't always a necessary process.

Right?

You're an experienced software developer, you can probably figure

out most of the bottlenecks in this image sharing flow.

But not all performance problems will show up in a model like this.

But our brains are really not good

at thinking about and reasoning about software performance.

It's, it's really easy to build software with emergent

properties that are so complex no one sees them coming.

And your user stories may be more complex than you will.

You know, than you think and as they get more and more

complicated you're less and less likely to anticipate where the problems are.

So this, this is, I call this story my coffee debugging story.

I was building a mobile app that had a you know, friends feature in it and what

we decided to do was use Facebook Connect

for our directory list so you'd log into Facebook.

And the app would pull down all of your friends, and then it

would make sure that it had a local cache of all those cache.

All right?

And spend a bunch of time optimizing the,

the database I/O and all of that process, right?

So that you log in, the app goes and fetches your current friends

list from Facebook, see if there's any

changes to people, like, they've, you know,.

People have been removed or added or they have a new, like, picture or whatever.

Right?

And we synced all those changes to the little database on

the phone and then went about your business of doing stuff.

And because we knew that it could take time to go out

and fetch your friend's list and process that, it happened in the background.

And the only part of the app that didn't work until

that was done was the send a message to someone feature, right?

That just, we decided to have that sort of disabled until that process was complete.

Cuz we wanted to make sure you were up to date on who you could send a message to.

So what happened?

We did this app it was really cool, we had a bunch of beta testers.

One of them was this really important guy in

San Francisco, who was really excited about the project.

One day he called me up and he was like, hey this app sucks.

And I'm like, what do you mean?

Well, the first time I installed it and run it, it's great.

But then I close and I run it again and I can't share a message for like 45 seconds.

And that's stupid because the whole point is like I wanna send someone something.

So we did the like, okay, uninstall it, reinstall it.

First time great, after that.

Took forever.

okay, log out, log back in, right?

Nothing worked, and we did some server side data

logging and I even sent him, like, an app build

that was, had all of this custom instrumentation in it

and it would send me logs of everything that happened.

And it turned out yeah, it was taking like 45 seconds.

It wasn't taking 45 seconds to go and get data from Facebook, that was really quick.

Right, it was taking like 45 seconds after it had his friends list before

it would unlock the share feature and I happen to be flying down to San

Francisco to do some other stuff so I called him up, I'm like hey can

we just get together and maybe look at your phone and I'll figure something out.

And so that's exactly what we did, right?

Flew down to San Francisco, did stuff, texted him a bunch.

We figured out a time when we could meet.

Met at a coffee shop, sat down, I plugged his

phone into my laptop, fired up the app, and boom.

The answer was right there in front of my face.

Then I turned to him and I said so.

How many Facebook friends do you have?

What do

you think?

Guesses?

10,000?

That's a pretty good guess.

10,000 friends on Facebook.

I didn't know it was possible, but apparently it is.

And so you know it turns out that statistically if you have 10,000 friends

on Facebook, every hour there's gonna be hundreds of profile updates

from all those people and so every time he launched that app

it was going and getting a summary of all of his Facebook friends.

And then the app was sitting there going, well I got 300 friends, I've got

to synchronize I'll insert an update, insert

an update, insert an update, insert an update.

Right and those were like, they were pretty fast.

You know, we were really proud of our 50 millisecond, like record

update speed on that stuff, cuz that stuff can be really hard.

But there was no way to do that at the

scale that this particular user's data was talking about, right?

And that was the catch, was we just didn't

realize the complexity of the data that we were handling.

So, you know, fixing this wasn't that hard.

We were like.

Well.

It's really fast to just insert a bunch of bold records,

so why don't we delete everything that's changed and reinsert it, right?

And it went down to like a half second.

Great.

But that on top of like pissing off this beta tester and flying

to San Francisco and spending an hour in traffic and all of that.

Not the best time.

And so the point is that you really, you'd

never know where you're gonna get hung up and

it can be the stupidest things, but unless you're

modeling out like it's, it's like risk modeling, right?

This bucket, well, you know, it's my Facebook friend's

list so it's kinda structured and it's small, except

that like you know, some people have a lot

of friends, so maybe we should make it big, right?

Then it's up there, with like, the videos and

other big chunks of data that your app is

gonna have to move around, and it shows up on the map, and it's easy to reason about.

And you can say, huh.

This could be something we need to worry about.

[BLANK_AUDIO]

So, the practice also leads to a lot of awesome stuff.

We saw that it makes API design easy.

Because you can see what things need to hit your API at the

same time and what things need be different and if there's two things

running in parallel, you know, you gotta figure out how to trigger the

right server side action after the second one lands, whichever one that might be.

All right, and that's not an easy problem but even knowing

that you have that problem can be hard to find out.

But what about other things?

What about fault tolerance?

Right?

Modeling what happens if a network request fails.

What do I do then?

Or working out the dependency analysis for complex activities.

Right?

Or you know, you can compare these.

Right?

You can take this and you can resize

things to look at the range of possibilities, right?

I could look at a Nexus 5 CPU VS a 3

year old you know little throw away android phone and say.

Hm, these pipes are a lot smaller on an older device I better think about how that

data is gonna get out of that particular

stock and through the rest of the user flow.

And once you have that complexity and the

performance modeled down to the component level, you can

wander down all these great avenues of thinking

about how to arrange things and orchestrate that, right.

And create the dance that makes people happy to use your app.

[BLANK_AUDIO]

Hm.

Technical failure.

[LAUGH] Too much fancy stuff.

So to summarize, you'll begin with the user stories, right?

Begin with the things that describe how your app is going to work.

What it's going to do.

And every designer I know sits there and hammers into me, like you need to do

user-centric design before you start writing any code, or designing screens.

And then model the stocks and the flows in each of those user interactions.

And identify the data and the processes that are gonna happen to them in a way

that allows you to visually reason with them

easily, rather than keeping it all in your head.

And then finally focus on understanding and accommodating

the very ability that's inherent in those systems.

So once you've done that.

Once you've reduced the variability, accommodated it

and created something that's really awesome, then

you can start poking the app with a stick when you need to, right?

You may find that yeah, we're really allocating too

many objects and we're gonna have to create an

object pool or, you know, we've got all of

these thread synchronization problems that are stalling things out.

We're gonna have to re-architect that internal code.

But until you've gotten rid of the concerns about unknown variance,

that's all just you know, you're, you're just optimizing the mess.

Right.

And if you find this systems theory stuff

interesting, I cannot recommend this book highly enough.

Thinking in Systems by Donna Meadows.

Its a really quick read and it will make you start seeing systems everywhere.

Literally everywhere.

Everything from pollution, to traffic to software engineering to why

the executives at your company do the things they do.

It all comes out of this systems modeling stuff.

And then, we get to the feedback part.

I would really love to hear whether this is something that helps you out.

Cause as I said it's a pretty new idea in my head, I'm really excited about it.

yeah, I hope it's gonna be a great thing.

And so, thank you very much.

We obviously have a little bit of time extra,

so any questions, thoughts, complaints about mobile app development?

[BLANK_AUDIO]

Yeah?

Hm?

All right.

No thumbs down.

[SOUND] That's excellent.

Thank you very much.