I updated the DNS haiku, y'all:
everyone forgot
about the certificates
they changed and things broke
#certificates #devops #softwareengineering
Remote
drmorr
@drmorr@hachyderm.io
mastodon
4.6.6
Optimizer. Photographer. In disbelief that computers actually work. Accidentally writing a role-playing game. (he/him)
0
Followers
0
Following
41
Posts
Joined January 26, 2023
GitHub:
Company:
Posts
Replying to
@mattblaze@federate.social
@mattblaze@federate.social the other difference is that nobody talks on the phone anymore. I went all day as a T-Mobile customer and had no idea they were having any sort of outage.
Replying to
@Migueldeicaza@mastodon.social
@Migueldeicaza@mastodon.social lol it got half of the things incorrect, despite me not taking any particularly extreme measures to protect anything.
Replying to
@b0rk@social.jvns.ca
@b0rk@social.jvns.ca (I'm really sorry, I was making a 12yo joke 💩 , I'll get out of your mentions now)
Replying to
@lindsey@recurse.social
In case you missed it, this is my SRECon recap, featuring all of my favorite talks!
I stayed up late to (finally) finish stranger things season 5. "it'll be just after midnight," i told myself. "not that much later than I normally stay up, it'll be fine."
I did not factor in the extra thirty minutes I would need to stop bawling my eyes out 😭😭😭
I believe.
Replying to
@drmorr@hachyderm.io
"No one ever made a decision because of a number. They need a story" Daniel Kahneman.
#srecon
Replying to
@drmorr@hachyderm.io
How to get better at storytelling? Lots of practice! If you're involved in incidents you'll have lots of practice, and you can tell them to people at SRECon :D
#srecon
Replying to
@drmorr@hachyderm.io
(Is this a good time to plug ACRL's one public postmortem? 🤣 🤣 🤣 🤣 🤣)
https://blog.appliedcomputing.io/p/postmortem-intermittent-failure-in
#srecon
Replying to
@drmorr@hachyderm.io
"So what, Lorin, you told us all this stuff, what am I supposed to do with it?"
One take-away: when you do your incident writeups, tell it as a story.
#srecon
Replying to
@drmorr@hachyderm.io
A third story told by Diane Vaughan said that challenger was a story about normalization of deviance.
If you tuned a noisy alert, that is normalization of deviance: if the alert is firing, but the system is healthy, we make the alert less noisy! We do this all the time.
#srecon
Replying to
@drmorr@hachyderm.io
Another story, told by Edward Tufte (of Visual Display of Quantitative Information fame) said this was about "poor information presentation and bad visualization".
Not surprising that an info-vis guy said it was an info-vis problem 🤣
#srecon
Replying to
@drmorr@hachyderm.io
The details of a story depend a lot on the perspective of the storyteller.
Example: the Challenger disaster. Feynman wrote an appendix to the "official" synopsis of the accident. He said this was a story about "management underestimating risks"
#srecon
Replying to
@drmorr@hachyderm.io
When it comes to incidents, there are different kinds or styles of stories you can tell:
- "The horror story": we failed over and the problem followed us to the new region! 😱
- "The morality tale": the engineer ignored the failing test and the bad code made it to production 😡
Replying to
@drmorr@hachyderm.io
We're discussing how this applies to the Therac-25 story.
The simple story is that "it's about race conditions", but the real story is a lot more complicated.
One example: the machine would frequently error and the errors were harmless, and in this case the error meant "the patient's dose is too high" but the operators didn't have that info.
#srecon
Replying to
@drmorr@hachyderm.io
- The story needs to be immutable (this is kindof a weird term to use): important details are preserved as the story gets passed on.
Replying to
@drmorr@hachyderm.io
- The story needs to be anomalous -- something in the story needs to disrupt your mental model of the world. There needs to be a disconnect between your belief and reality. Example: "This should never happen!" which means "I never expected that to happen!"
Replying to
@drmorr@hachyderm.io
Claim: good stories have two properties to be "useful" for some nebulous definition of "useful"
#srecon
Replying to
@drmorr@hachyderm.io
We're comparing "nursing" to "SRE" -- for example, a patient is deteriorating but it isn't showing up in their vitals yet. In engineering, this is like the system is in bad shape but no alerts have fired yet.
#srecon
Replying to
@drmorr@hachyderm.io
"Only a fool learns from his own mistakes, a wise man learns from the mistakes of others" -- unknown
We've also talked a lot about vicarious learning, e.g., learning from watching or listening to someone else.
Example: Shoulder-surfing while someone is looking up dashboards.
#srecon
Replying to
@drmorr@hachyderm.io
"Stories" stick in your head, which is helpful for getting information into the heads of people who need it.
#srecon
Replying to
@drmorr@hachyderm.io
The incident prevention grand challenge is "How do you get the right information into the heads of the right people at the exact moment when they need it?"
#srecon
Replying to
@drmorr@hachyderm.io
When something negative, surprising, or unexpected (like an incident) happened, the human response is "How could this bad thing have happened????????"
Stories are important because they are a tool that humans have developed to make sense of senseless events.
#srecon
Last talk of this conference is "The Power of Stories" by Lorin Hochstein, aka @norootcause@hachyderm.io
Replying to
@adrian@discuss.systems
@adrian@discuss.systems yea, will do it again next week I think. Or maybe I'll try shepherder's bread again.
Replying to
@adrian@discuss.systems
@adrian@discuss.systems yesterday I made challah, it was going to be really good, and then I popped it in the oven, didn't set a timer, and promptly forgot about it for the next hour and a half 😭😭😭😭
Replying to
@saraislet@infosec.exchange
@saraislet@infosec.exchange this feels like a @hazelweakly@hachyderm.io toot.
Gonna liveblog @Rejekts@hachyderm.io over here! Follow along if you want.
This is a post by Anastasjia Uspenski about how the "myth of the 10x engineer" is actively harmful (the post is itself based on a talk by @mipsytipsy@hachyderm.io at the Craft Conference in Budapest):
https://shiftmag.dev/10x-engineers-charity-majors-5755/?ref=cloudnative.now
"If there is a single non-negotiable truth in Charity’s worldview, it’s this: shipping is how engineers learn. Not architecture diagrams. Not months of planning. Not 400-comment design docs. Shipping."