• Home
  • >
  • DevOps
  • >
  • The 5-Year Retrospective – InApps Technology 2022

The 5-Year Retrospective – InApps Technology is an article under the topic Devops Many of you are most interested in today !! Today, let’s InApps.net learn The 5-Year Retrospective – InApps Technology in today’s post !

Read more about The 5-Year Retrospective – InApps Technology at Wikipedia

You can find content about The 5-Year Retrospective – InApps Technology from the Wikipedia website

Charity Majors

Charity is the co-founder and CTO of Honeycomb.io, a tool to help software engineers understand what happens when their code meets production. Charity ran infrastructure at Parse and was an engineering manager at Facebook. She started Honeycomb to bring engineers the observability tooling they need for the era of distributed systems. Charity is the co-author of Database Reliability Engineering (O’Reilly), and is devoted to creating a world where every engineer is on call, and nobody thinks on call sucks.

Two years ago, I wrote a long retrospective of observability for its third anniversary. It includes a history of instrumentation and telemetry, a detailed explanation of the technical spec, and why the whole “three pillars” thing is nonsense. At the time, it’s what was needed to steer conversations away from silly rabbit holes about data types and back to what matters: how we understand our systems.

Two years later, observability rhetoric has become even more overheated. Every other day, I hear about a new “observability for x” startup getting funded. Which is cool, in a way, but I worry about the devops’ification of the term. “Observability” is starting to be used to mean anything to anyone, just a label you slap on the side of your product to fit in with the zeitgeist.

O’Reilly said it best this year when they summarized learning trends in 2021:

Observability saw the greatest growth in the past year (128%), while monitoring is only up 9%. While observability is a richer, more powerful capability than monitoring — observability is the ability to find the information you need to analyze or debug software, while monitoring requires predicting in advance what data will be useful — we suspect that this shift is largely cosmetic. “Observability” risks becoming the new name for monitoring. And that’s unfortunate. If you think observability is merely a more fashionable term for monitoring, you’re missing its value. (Emphasis mine.)

We can’t lose sight of that value. We can’t afford to. This isn’t just a tale of vendors arguing to define marketing terms for their own benefit. The pain and suffering that people endure every day because they can’t understand their own damn systems is too real. The long hours, the toil, the greasy hacks moldering away into technical debt, the late nights, the missed sleep, the burnout. The pain is real, and the solutions are specific. We need specific, meaningful technical terms to help users navigate the future and find their way to those solutions.

Christine and I started Honeycomb after trying every monitoring and logging solution under the sun and realizing that they were fine for answering simple known-unknowns, but wretchedly horrible for answering unknown-unknowns. Observability changed our lives (and sleep schedules) dramatically, transformed how we used systems, made us better engineers, and made us want to give that to our entire industry.

Now it’s time for another retrospective. What have we learned and where are we, as an industry, going wrong?

From the Mouths of Tweeters: Observability vs. Monitoring

I opened Twitter the other day to find Corey Quinn dangling some tweet bait in my face, with this quoted retweet simply saying, “Discuss!”:

Annnd boy, did we. Cue a flood of definitions for observability (and squabbling over each other’s definitions).

First, I stated that observability was defined by structured events.

Then, Alex Hidalgo countered that it wasn’t.

The perspective that Alex appears to be echoing is one where observability is just a generic synonym for telemetry or insights. Any old graph, dashboard or measure you happen to have lying around? That’s observability! That janky old nagios3 instance you still haven’t spun down? Observability!

In that scenario, observability is just another generic word for “understanding systems.” A closely related view is that observability has three pillars: metrics, logs and traces. If you also accept that definition, then I can understand why you might gripe that using observability to define a technical set of capabilities that go well beyond monitoring would be erasing “decades of work.”

I don’t accept the three-pillars definition. The world doesn’t need another classification word for existing data types. And it doesn’t need another synonym for “telemetry” or “insights.”

But there is a coherent set of practices that started to emerge about five years ago, in response to changes in how we build modern systems — particularly microservices. Those practices are a fairly discontinuous leap from their heritage of using metrics and logs. In fact, some of the best practices are exactly opposite those of monitoring, and the respective data formats used are entirely incompatible. Those practices do need a word.

It’s been a while since I waded into the definition wars. And I told myself I wouldn’t do this anymore. But it seems we can’t stop getting lost in the weeds, and we won’t get things back on a useful track unless we do.

Let’s do this.

A Historical Taxonomy of Definitions for Observability

1960-2015: To mechanical engineers and control systems theorists, observability means the mathematical dual of controllability — how well can you understand the inner workings of a system simply by observing it from the outside? The term was rarely heard in the computer industry, with the notable exception of Twitter’s “observability engineering” team. However, they used it as a generic synonym for telemetry.

2015: We started building Honeycomb and wrestling mightily with how to talk about what we were building. In July of that year, I googled the definition of observability, and it resonated with me powerfully. It was exactly what we were trying to build — the ability to understand any inner system state, with no prior knowledge or warning, simply by interrogating it from the outside! At the time, I wrote a great deal of content about the technical prerequisites and implications of this definition.

Read More:   Containers Return Work-Life Balance Back to Developers – InApps 2022

2017: After attending a tracing summit, Peter Bourgon wrote an extremely influential blog post claiming that observability has three pillars: metrics, logs and traces.

2018: Vendors from logging, metrics, and tracing companies (unsurprisingly) loved this definition and adopted it enthusiastically. “Three pillars” marketing content pummeled you with targeted ads. I wrote my retrospective. Ben Sigelman wrote an excellent rebuttal too. One large incumbent monitoring vendor launched their “observability” platform and the gold rush began. Vendors in the metrics, logging, tracing, monitoring, and APM categories — even certain database companies — all began rebranding themselves as observability companies. Companies were acquired, products were consolidated on the backend, and the marketing dollars just flowed.

2020: I personally resolved to stop talking as much about the technical definitions of observability and instead focus on the capabilities. What matters is how the technology changes our practices as an industry, right?

2021: The present. Where a lot of people are yammering on about observability with very little conceptual coherence.

The ensuing Twitter thread above shone a light on some of the many conflicting and/or complementary definitions for observability — some of which I can totally accept and buy into. I think others are (at best) useless or (at worst) harmful to users.

Let’s examine some of the definitions for observability offered in this very thread. Cue my inner pedant!

(a since deleted tweet): Anything that gives you insight into a system counts as observability.

No. This bar is way too low. Anything that offers insights? You mean my copy of “Advanced Programming in the Unix Environment” counts as observability? Try again.

Lorin Hochstein: Observability is about fault localization.

True! Observability is not about debugging your code. It’s about figuring out where exactly in the system to look for the code you need to debug. This succinct take is correct, but it could use more detail.

@austinlparker: Monitoring is the monitor telling me the baby is crying but observability is telling me why.

Great (if weird) analogy! It nods at the perspective shift where monitoring is primarily running third-party checks, while observability is a first-person narrator.

Sam Coren: Observability is building out systems and processes to ensure accurate and actionable data exists to measure performance and diagnose issues. It’s about training people to know how to instrument their apps or how to create good workflows that helps them solve actual problems.

Sure, I guess. But definitions like this are also harmful. In practice, how is this any different than monitoring?

Doug Odegaard: Observability is a way of correlating to reduce the amount of time for human correlation (poring over by hand or grepping logs). Find one issue and dive down on the other two. It’s a data solution and UX as well.

Ooh, sounds like Doug has spent time with a real observability tool! This is indeed one of the most common motions of observability in practice.

@Network_Guy: Observability allows me to understand what my application is doing at a much deeper level then single entity monitoring.

True. But … how? Why?

Kevin Parsons: Monitoring is associated with boring old things like Nagios. Observability is much shinier and cooler. They are both buzzwords, I don’t see any meaningful difference between them.

This is a terrific example of why I needed to write this article. Kevin, I hope you’re reading!

And then Liz Fong-Jones adds to the mix.

Liz Fong-Jones: Observability is not a binary state; it requires work over time. It’s the ability to swiftly surface relevant data to solve perf regressions, understand how your system works or your code behaves, and resolve incidents. Observability uses instrumentation to help provide context and and insights to aid monitoring. While monitoring can help you discover there is an issue, observability can help you discover why.

Honors class answers, Liz!

(a since deleted tweet): Observability is not about any particular tool or data structure, it is about gaining insight into systems.

It’s this last one I particularly want to speak to. Because it’s true. But it’s also ridiculously vague. Observability is about successfully gaining insight into systems. And it’s the how of what we do to get there — that’s what matters about observability.

Observability as Defined by What You Can Actually Do with It

Reading the various takes above was actually encouraging to me, because a bare majority of responders have internalized two important things:

  1. Observability is materially different from our monitoring heritage, and
  2. It is different because it helps you explain and understand the inner workings of the system.

There are dozens, maybe hundreds, of different ways to create nuance around that to arrive at some definition that works for you. Similarly, DevOps (to this day!) struggles with the fact that there is no real agreed-upon definition of the term. Therefore, one team’s practice is DevOps just as much as another team’s practice (that looks nothing like the first team’s) is still DevOps — much to the bewilderment and confusion of anyone on the outside looking in. Vendors can then easily slap a DevOps label on just about anything and no one is apparently the wiser. After all, it’s still DevOps!

Rather than describing all the different facets and implementations, which can be either unhelpfully vague or more harmful than helpful, how about we focus on the capabilities instead? What you can materially do with observability that you can’t do with another practice like monitoring?

Here’s my definition of observability, for what it’s worth:

Observability lets you find answers to application issues that are unknown-unknowns. You have observability if you can ask any question of your system from the outside, to understand any state the system has gotten into, no matter how bizarre or novel, without shipping any new custom code to get answers.

I like this definition because it hews closely to the definition from mechanical engineering and control theory:

Observability is the ability to measure the internal states of a system by examining its outputs. A system is considered “observable” if the current state can be estimated by only using information from outputs.

Imagine: You just deployed, and something weird is happening. The anomaly doesn’t look like any issue you’ve ever encountered before. Can you swiftly figure out what’s going on? Can you do that without SSH’ing into the machine? Can you do that without adding any new custom metrics? Can you do that without spraying a bunch of Hail Mary log lines around the new change and redeploying?

Read More:   Machine Learning Engineer vs Data Scientist - Detailed Comparison

If the tools you’re using to understand that system are metrics, static dashboard and log aggregators, then probably not. Those tools are built to measure your system against known good or bad states and tell you if good or bad things are happening in production right now.

They’re not built to debug entirely new issues you didn’t even know were possible by methodically following clues, by slicing and dicing telemetry data to compare it across multiple dimensions, and by digging through what’s happening in real time to identify outliers and find correlations.

What I like about focusing on the capabilities that define observability is that, if you accept the definition about finding answers to unknown-unknowns, then you inevitably must also accept that there are certain technical prerequisites for achieving that goal. In order to understand any unknown internal state, any observability tool must also support:

  1. Arbitrarily wide structured raw events: a building block so granular that you’re able to slice and dice your telemetry data across any dimensions needed to find the tiniest commonalities across system anomalies.
  2. Context persisted and ordered through the execution path: you need to instrument your code so that you are persisting the unique request id, trace id, and any important context from service to service, so that you can trace the requests and so that later, while slicing and dicing, you will be able to correlate patterns between requests.
  3. Without indexes or schemas: because you can’t predefine optimizations for things you didn’t know you were going to look for in the future — data retrieval must be fast, at all times, for all data. (You pretty much have to have a columnar store to achieve observability.)
  4. High-cardinality and high-dimensionality: so that there are effectively no limits to how you’re able to slice and dice your data.  You must be able to work with your data without the constraints imposed by most traditional databases.
  5. Client-side dynamic sampling: so that for the largest use cases at scale, where the typical trickle of telemetry data can quickly become a torrential flood, you can manage the tradeoff of keeping enough observability data for proper debugging without killing your underlying systems.
  6. An exploratory visual interface that lets you arbitrarily slice and dice and combine dimensions: humans are visual creatures and we’re wired to find patterns amongst a sea of noise. Visual interfaces help us quickly sift out signals that matter when exploring unknown issues.
  7. In close to real time: you have to get answers within seconds, because no one has time to wait several minutes for results when production services are down.

I’ll be the first to admit: Maybe I’m wrong! Maybe you CAN get answers to unknown-unknown questions without high cardinality. Maybe you DON’T have to use the arbitrarily wide structured data blob in order to correlate details from context between a set of events. Maybe YOU know how that’s done.

Great! I would LOVE to hear about it! Come find me on Twitter.

Wouldn’t THAT be a more helpful argument to have? How can we achieve these kinds of results? How can we build more on top of these new ways to understand our systems?

Observability Must Be a Clear Concept

Lots of people seem to have gotten the memo that observability is about understanding your systems. Great! Now they just need to understand enough about the required capabilities so that they can see through people selling them snake oil with an “observability” label on the bottle and no material way to actually achieve it.

(I’m also looking at you, “observability engineering” teams that spun up Prometheus, added some logs and dashboards, and called it a day. I hear from your software engineers every week in despair. Just because you call it observability doesn’t make it so.)

When talking about observability as a set of capabilities — understanding any new, bizarre, and previously unknown failure simply by analyzing the data that the system emits — you can start to see why observability has exploded in popularity.

Systems used to fail in fairly predictable ways. It didn’t matter that metrics stripped away all the context before storing them or that you couldn’t dive down into a dashboard and ask “…and then what? And then what?” You didn’t need to dig that deeply to understand where failures could be occurring. You could just examine problems by monitoring conditions from the outside.

You had The App, The Web, and The Database. The overwhelming majority of complexity lived in The App. If all else failed, you would attach gdb to The App and step through it. Back then, we knew where to look for issues.

But microservices pushed much of the complexity out of the app and into the system space. Modern systems are now a soup of databases and storage types, serverless functions and third-party APIs. Novel and bizarre partial failures aren’t fascinating rarities in today’s application architectures. You have unknown-unknowns for breakfast.

And in today’s world, we need to change how we think about systematically finding the real sources of issues. For decades, we told ourselves that’s what we were doing. But it turns out that most of our “debugging” was done by intuition, or by reasoning through the maybe half dozen components where a failure could be lurking.

Now? Forget attaching gdb to The App. You don’t even know which of the dozens of ephemeral app instances in this cluster could even be experiencing the problem. Or if it’s one of the hundreds or thousands of instances upstream or downstream from this one.

That’s why “observability” matters. We need to have a term that encapsulates all of these newer and firmly modern tools and best practices, so that people can find the tools they actually need. We don’t need yet another synonym for telemetry, or monitoring, or for any of the tools we’ve had for decades that are failing the problems of today.

InApps Technology is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Honeycomb.io.

Photo by Dylan from Pexels.

Source: InApps.net

List of Keywords users find our article on Google:

useless synonym
boring synonym
satisfied synonym
horrible synonym
insight synonym
silly synonym
similarly synonym
minimum viable product charity
overwhelming synonym
admit synonym
rush synonym
capability synonym
notable synonym
incompatible synonym
unknown synonym
failure synonym
analogy synonym
anomaly synonym
emit synonym
charity synonym
despair synonym
influential synonym
promptly synonym
confusion synonym
suffering synonym
fail synonym
nonsense synonym
sucks synonyms
bare synonym
havent synonym
harmful synonym
retrospective synonym
withstand synonym
arbitrary synonym
unfortunate synonym
encapsulating synonym
honeycomb io
trace synonym
dedication synonym
heritage synonym
suck synonym
build a team synonym
container synonym
flood synonym
reasonable synonym
tweet synonym
monitoring thesaurus
e-commerce synonym
mathematical synonym
greasy synonym
ran synonym
lurking synonym
e commerce synonym
correlation synonym
dedicated synonym
emphasis synonym
observable synonym
arrive synonym
explode synonym
reasoning synonym
acquired synonym
endure synonym
majority synonym
vague synonym
majority syn
interrogate synonym
worry synonym
emerge synonym
bait synonym
ran away synonym
deeply synonym
ideology synonym
couldnt wait synonym
agreed upon synonym
storing synonym
pushed through synonym
mechanical synonym
digging synonym
years later synonym
leap synonym
rebuttal synonym
creature synonym
bewilderment synonym
rarity synonym
coolina usa
correlate synonym
echoed synonym
predictable synonym
synonym for systematically
bizarre synonyms
founding member synonym
narrator synonym
hospitality synonym
synonym for creature
synonym for monitor
capability syn
branch synonym
actionable synonym
agreed-upon synonym
bunch synonym
nuanced synonym
synonyms for bewildered
summit synonym
aquired synonym
resolve synonym
succinct synonym
granular synonym
viable synonym
constraints synonym
exception synonym
synonym unknown
we understand that synonym
ephemeral synonym
examine synonym
representative synonym
constraints synonyms
failing synonym
customer service representative synonym
derive synonym
distinct synonym
devoted synonym
coherent synonyms
responsive synonym
successfully synonym
thomas comes to breakfast wiki
arbitrary wikipedia
simply three wikipedia
“modern systems”
richer sounds trustpilot
the five year engagement wiki
trickle up synonym
closely related synonym
happening a lot synonym
powerfully synonym
gotten better at synonym
deeper level synonym
hews twitter
janky synonym
sift heads wikipedia
adhering to synonym
learn deeply synonym
lose sight synonym
cooler synonym
logo wawa cosmetics
stripped back synonym
another synonym
synonym for wholly
wouldn’t synonym
overwhelmingly synonym
pushed out synonym
couldn’t synonym
damn synonym
heated conversation synonym
synonym for gotten
touched upon synonym
wide fit flatforms
amongst synonym
lying definition synonym
waded synonym
another word for fashionable
fault synonym
haven definition synonym
we were able to synonym
building upon synonym
countered synonym
anomalies synonym
digging out synonym
nobody synonym
overheat synonym
specs synonym
swiftly synonym
synonym for insightful
talking nonsense synonym
ability to speak well synonym
analogical synonym
closely synonym
impressed synonym
leaping synonym
drip synonym
retrospective definition
synonym sucks
what is the synonym of impressed
build upon synonym
engineering synonym
innerworkings synonym
synonyms for sucks
wrestling synonym
charity case synonym
hews hack
synonyms of retrospective
trace back synonym
anomalous synonym
customer is not defined in company code
honeycomb.io jobs
pillar synonym
synonym for actionable
synonym for charity
synonyms lurk
timeframe synonym
anomalous synonyms
control myself synonym
couldnt synonym
cue synonym
synonym cooling
synonym for founding
synonym for viable
vendor is not defined in company code
as agreed upon synonym
beyond the limits synonym
pedantry synonyms
pushed synonym
rebut synonym
synonym of professionalism
a bunch synonym
contradictory synonym
development synonym
hailed synonym
hopeless synonym
inner workings synonym
internalised synonym
it is unfortunate synonym
retrospect synonym
synonym feasible
synonym for recruit
synonym for sucks
to be in debt synonym
vendor not defined in company code
wasn’t synonym
end suffering synonym
lying synonym
synonym for ideology
velocity synonym
bewildered synonym
emphasise synonym
interrogated synonym
pillars synonym
ran syn
viable synonyms
alight synonym
arbitrary synonyms
bare minimum synonym
utterly synonym
another word for stripped away
branch out synonym
endured synonym
endures synonym
methodical synonym
monitor closely synonym
synonym viable
aggregator synonyms
analogous synonym
bizarre synonym
building block synonym
coherent thesaurus
happening now synonym
recruitment synonym
synonyms for suck
touch synonym
correlates synonym
gold mine synonym
leap synonyms
stripped synonym
another word for trickle
best case scenario synonym
built upon synonym
correlations synonym
define endure synonym
extremely happy synonym
i missed you synonym
imagine synoym
newer synonym
outsystems log in
persisting synonym
synonym monitor
synonym of pillars
synonyms for despair
synonyms for toil
synonyms for useless
ensuing synonym
horrible synonyms
imagine synonym
in coherence synonym
nagios partner
niche synonym
nuance synonym
observing synonym
resolved synonym
resonate synonym
succinct synonyms
wretchedly synonym
can’t get enough synonym
co founder synonym
encapsulates synonym
hotline synonym
incumbency synonym
narrating synonym
professionalism synonym
synonym insightful
synonym of intuition
synonyms for slice
arbitrarily synonym
correlating synonym
cosmetic synonym
failures synonym
lose hope synonym
rubbish synonyms
synonyms of bewildered
when does wawa stop serving breakfast
beforehand synonym
co-founder synonym
complexity synonym
exploded synonym
incumbent synonym
keeping track of synonym
lose synonym
missed out synonym
mobility synonym
overwhelmed synonym
slicing synonym
synonym for correlation
synonym for rhetoric
vague thesaurus
wawa soup of the day
bottle label design templates
changed a lot synonym
effectively managed synonym
monitoring synonym
nagios vs
rubbish thesaurus
stop talking synonym
swiftly synonyms
synonym for merely
synonym of reliability
synonyms for coherent
tale synonym
thread synonym
tradeoff synonym
advanced programming in the unix environment
emits synonym
encapsulate synonym
lost hope synonym
narration synonym
realizing synonym
silly soup game
synonym dedication
synonym for coherence
synonym for impressed
synonym for mobility
what is a synonym for bewildered
why did the narrator resolve to forget the address
capabilities synonym
capability synonm
consolidated container company jobs
flooding synonym
mine spec hire
outlier synonym
predictably synonym
synonym for overwhelm
synonym for unhappy
synonym overwhelm
as we agreed synonym
building blocks synonym
closely similar synonym
ephemeral syn
nuances synonym
resonates synonym
synonym dual
synonym for coherent
synonym for niche
terrific synonym
designing synonym
encapsulated synonym
fascinating synonym
focusing synonym
interested synonym
investor synonym
not satisfied synonym
strengthen synonym
synonym for pedant
tell tale sign synonym
era synonym
hail synonym
persists synonym
apm vs observability
coherent synonym
common sight synonym
dna synonym
localization synonym
memo synonym
mightily definition
summarized synonym
binary synonym
failure synony
granularity synonym
granularity synonyms
low observable jobs
meets requirements synonym
synonym for arbitrary
synonym for era
christine quinn wikipedia
feasible synonyms
i hope you understand synonym
synonym analogy
high productivity synonym
observability vs apm
outsourced synonym
retrospect definition
synonym for apparently
synonym for overwhelming
synonym for partial
boring synom
engagement synonym
exploratory synonym
feasible synonym
generic synonym
implications synonym
missed synonym
parse synonym
succinctly synonym
suspect synonym
synonym emerge
synonym of coherent
attach synonym
erase synonym
merely synonym
nagios core requirements
synonym for ephemeral
dedication synonyms
engineer synonym
fashionable synonym
outsystems architecture
persist synonym
resonated with me definition
ridiculous synonym
succinctly synonyms
synonym of dedicated
Rate this post
Content writer

Let’s create the next big thing together!

Coming together is a beginning. Keeping together is progress. Working together is success.

Let’s talk

Get a custom Proposal

Please fill in your information and your need to get a suitable solution.

    You need to enter your email to download

      Success. Downloading...