On the Origin of the Speciated Conference

I go to so many conferences! It’s an awesome and amazing part of my job. I speak at them, but I also attend them. I sit in the front row and live-tweet. I attend talks. I participate in unconference sessions. I talk to people in lines, and at lunch, and at the afterparty. I give out stickers and I say hi to the vendors. Conferences are something I’m an expert at. And when I’m not doing technology stuff, I am support crew for science-fiction conference runners.

Given that, I’m surprised that I haven’t seen a taxonomy of technical conferences, something that would help you understand which flavor of conference you’re about to go to. As near as I can tell, about 10 years ago, there was a great flowering of conference types.

Originally, we had The Technology Conference. People paid a lot of money, they sat in large conference rooms (over 100 people, say), and they listened to industry experts. Some/many of the industry experts were also vendors giving pitches.

Then, a decade ago, many people decided that this method was not meeting their needs, and they wanted more interaction, more peer contact, more connections. We ended up with some new species of conference:

  • The regional variant of a language conference. No longer just PyCon, but PyConAU, and EU, and the same for JSConf and Ruby. It was cheaper to get people to someplace close to them, the conferences were smaller, the odds of meetings speakers and experts was higher.
  • Single-track conferences with registration caps. Write the Docs, The Lead Developer, and (I think) Monitorama use this method. Everyone attends the same talks, but the registration cap means that it’s still possible to identify and talk to a speaker. A well-run single-track conference allows a lot of time between talks so people can mingle and talk.
  • DevOpsDays. The DoD format is flexible, but tends toward the single-track morning, unconference afternoon. They also work really hard to fit into budgets that allow people to attend on their own, with lowish registration fees and locations all over.
  • No Fluff, Just Stuff. The first unsponsored conference I’ve spoken at. No vendors, and a high rate of repetition for speakers and a lot of tracks, so odds are good that you will be in a small group.
  • Birds of a Feather. Not unique to any one conference organizing system, but a way for people interested in a similar problem to find each other and do collaborative learning. Mostly these happen during non-programming time.

All of these conference styles prize collaborative learning over authoritarian instruction. If you’re a speaker coming from a more authoritarian background, I have to imagine the change is a bit of a shock. I know I have felt weird when presented with a large audience that I can’t see. I don’t want you to think that one way or the other is better – depends on what you need. 50k people wouldn’t go to AWS Reinvent if there wasn’t a value to be found in it. And Reinvent has small, unrecorded sessions as well as the massive keynote sessions.

Hotel ballroom filled with a few hundred people, all facing toward a podium and the camera.

DevOpsDays Toronto

So who are the stakeholders for running a conference?

  • Attendees
  • Sponsors
  • Speakers
  • Organizers

When you maximize the happiness or utility for one group, the utility for other groups goes down, or may go down. There are some overlaps. Attendees want content that answers their questions. Speakers want to provide content that is new and promotes their personal brand. Organizers want to select speakers who bring good value and are reliable. Sponsors want their speakers selected because talking about a product drives sales. Attendees, on the whole, don’t want sales-pitch talks. You see the problem!

As a speaker, I prefer single-track conferences. That way, I never miss other people’s talks! The talks are also usually very highly curated, since a day-long conference might only have 7 speakers, so it’s pretty darn flattering to get picked. As an attendee, I like conferences that are sized so each speaker ends up talking to about 50 people. It’s small enough that I feel engaged, and big enough that the speaker doesn’t feel like they have to stop to take questions. As a sponsor, I want multiple tracks with large spaces where people have to walk past my booth to get caffeine. As an organizer, well, I’m still working on that.

I’m thinking about this because LaunchDarkly is assembling our first conference this year (2019), in the spirit of Gremlin’s Chaos Conference and Honeycomb’s o11ycon. What do we want to give people, how many people do we think we’ll have, and how do we make the experience useful?

In the spirit of testing in production, we’re going to try a combination of things – keynotes will be one-track, so everyone has a common thing to talk about, and then we’ll split into other configurations in the afternoon.

We’re looking for people who want to join us on April 9 at Trajectory to talk about feature flagging, trunk-based development, devops tools, testing in production, blue-green deployments, and other ways to speed up your development and delivery…safely.

https://www.papercall.io/trajectory

If you want help with your pitch, or want to noodle around an idea, let me know. I’ll be back at work on the 7th and ready to think it through with you! (Yes, we’ll do bigger announcements later!)

Book Review: Accelerate

Accelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology OrganizationsAccelerate: The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations by Nicole Forsgren
My rating: 5 of 5 stars

This book is an excellent summation of the overall findings of the State of DevOps reports and what the implications are for organizations trying to persuade themselves into DevOps transformation.

When I was reading it, I realized that it was everything you would learn if you went to a couple dozen DevOps days and listened to all the transformation stories, and it can be summarized as:

Go faster, be safer.

This is such a counterintuitive thing to say, but I have an analogy for you: When you are trying to teach a kid to ride a bike, they’re like “You’re hurling me at the asphalt and yelling at me to GO FASTER?!?!”. You know, even though they don’t that they will be safer if they go faster, but they feel like it’s very dangerous. Increasing your release cadence feels like that – someone you trust is telling you to do it faster, but it’s very scary to change your thinking about ASPHALT.

Forsgren, Humble, and Kim unpack the scientific analysis of why it works that way, but I find it really compelling to look at the correlation of speed, psychological safety, and business value.

Part 2 of the book is a very wonky, nerdy analysis of the academic methods, which gives the rest a lot of validity, but I give you permission to skip it if you would just like to take their word for it that the math is sound.

Part 3 is a case study of a company currently practicing a lot of the principles mentioned in the book. It may not be relevant to everyone, but it was a nice unpacking of how it could work.

Read if:
You are trying to convince your organization to become more devops-y, or if you want to understand what the benefit of that would be. Also if you would like some eyebrow-massive correlation numbers to use in arguments.

Skip if:
You will only pine because you can’t do this kind of transformation.

Also read:
The Phoenix Project
The DevOps Handbook

View all my reviews

Documentation for DevOps: A DevOpsDays Open Space Writeup

I’m borrowing this image from my friends at Chef, because it’s really funny to pretend that it’s possible to just sprinkle on “devops”, something that is about cultural change at every level.

At every DevOpsDays I go to (and that’s quite a few), I propose/wrangle/run two open spaces. One is about feature flags/canary launches/hypothesis-driven development, and the other is this one, which I usually refer to as “Docs for DevOps”.

DevOps documentation is difficult because it’s not intended for external audiences, so it’s hard to make a case for hiring a writer, but it’s also mission-critical, which means that it can’t be ignored or neglected.

I also have a talk that I give on this topic. You can find recordings from DevOpsDays Minneapolis and NDC Minnesota here. but this is specifically about the open space. I am indebted to a number of people in the last year who have come to open spaces, shared their experiences, asked questions, and challenged my thinking and assumptions.

I’ve grouped the questions I hear most often into these categories:

  • How can I find anything?
  • How do I get people to read the docs?
  • How do I get people to write the docs?
  • My team is small, do you have particular advice?
  • My team is very large, what should we do?
  • Distributed team best practices?
  • Why is sharepoint?
  • What tools are most useful?

How can I find anything?

Frequently, the first problem that people have is not that nothing is written down, but that lots of things are, and the problem is discoverability and accuracy. It’s one thing to have instructions, and quite another to have instructions that you can find when you need them. Many people have their operations documents spread across one or many wikis, ticketing systems, and document storage platforms.

The short answer to this problem is “search”. That is also most of the long answer. The thing I think we overlook is that we don’t have to use the search engine that came with our wiki. We could put something else in front of it. With the demise of the physical Google Search Appliance, you still have a number of service options. Most of them will give you the option to index and search multiple sites.

As a consultant who helped companies structure documentation so it was usable, I have a lot of specific advice on how to change the architecture and the culture, but honestly, if you could get your company to hire someone to help with that, you could probably get them to hire a writer and solve the problem another way.

The last part of this answer is that you could get some consulting dollars and hire an archivist/librarian to help you come up with tags that help identify the types of data and make it easier to search for. When we’re trying to recall data, we frequently do so by keyword or tag, and an archivist can help you make a canonical list of what people should use for that.

How do I get people to read what we have?

First, see above, make it easy to find.

Second, make it as easy as possible to read. Sit down in a group and hash out what type of information has to be included to make a topic/article/page useful to others. Once you know what you need to have included, you can make a template so that people remember to include it all.

The next problem is to figure out how to point people to the documentation without being dismissive or making them feel like you are too good to answer their questions. This might be a good place for a chatbot, or a single web page full of resources.

Many people would rather be able to find an answer themselves than bug a coworker to get it, so if they’re not reading the documentation, there’s probably a systemic problem blocking them.

How do you get people to write the docs?

Do you know any sysadmin or SRE who has ever been fired for lack of documentation?

Do you know of anyone who has gotten promoted solely on their ability to write internal documentation?

Most of us don’t. No matter how much organizations say they care about documentation, they are not putting anything of meaning behind those words. It’s not malicious, but it does tell you that documentation is not actually something they care about.

One of the ways to fix this is by using a reward system. Buy a package of Starbucks cards, or awesome stickers, or ice cream certificates, and give them out every time someone completes a documentation task, immediately. Adding a tally mark for their eventual review is much less motivational than immediate rewards for work. You don’t have to be a manager to do this – anyone can motivate their co-workers. The reward doesn’t have to have cash value, but I do think it’s the easiest way to indicate your appreciation.

Another thing to pay attention to is how easy or hard it is to write the documentation. If it’s a complicated system, the barrier to entry is high. If it’s in a tool the team already uses, in a language they already know, it’s much easier to persuade them to drop in a few lines. if there is a form for them to fill in so they don’t have to figure out what they are supposed to include, that’s even better. Lowering the cognitive barrier to entry is as important as the tool barrier to entry. After all, we’d have a lot more trouble writing bug reports if we had to write them out by longhand without any fields to fill in. Documentation is the same way.

My team is small

Consider using just one single page of truth. No, seriously. If you’re working with a team under 8, and especially if they’re distributed, consider just putting everything on one single web page and searching it when you need to find something. The overhead of managing a documentation system or wiki is not something you need when your team is that small. As you write new things, they go at the top and older stuff gets pushed down to the bottom and becomes obviously less important or relevant. If you need to find something, you can easily search the page. No one has a secret pile of documents that the rest of the team doesn’t know about. It’s not an elegant solution, but it’s hella utilitarian.

My team is large

Hire a writer. And/or a librarian. I was talking to someone whose IT organization was over seven thousand people. They obviously do need some structure, some tooling, some information architecture. Rather than spend internal IT cycles on it, I suggest hiring experts. There are thousands of people in this world who are trained to manage, collate, write, and wrangle large sets of documents, and it’s wasteful to try to do it by yourself.

What are the best practices for distributed teams?

Pretty much the only effective way to have a distributed team is to change the team culture to one where you write down questions and answers, instead of popping by to share them. It’s important to make sure that key data is always written down and not passed on by word of mouth, because as soon as you have an oral culture, you are leaving remote members out.

It’s useful to have at least one team member working remotely who has distributed team experience. You can ask them for what has worked for them in the past and enlist their help in iterating your practices. For example, is it hard for the team to remember to set up the call-in for meetings? Change the meeting template to include a shared meeting room automatically. No one gets a distributed team right immediately, anymore than we do a colocated team. It’s just that it’s easier for distributed teams to end up in a broken state before someone notices.

Why is SharePoint?

Because the paradigm and core analogy is filing cabinets. “What if I could see inside the Operations filing cabinet?” Because it thinks of itself as a way to organize pieces of paper, it’s not very good at documents that need to be shared and altered simultaneously across organizational boundaries. It was a marvel for its time, a way for non-technical users to get their documents out of shared drives and file cabinets and give other people regulated access, but it is fundamentally hostile to searching.

What tools are most useful?

It depends, right? Tools exist on a sliding scale between absolutely zero barrier to entry on one end and and taxanomic usefulness on the other. Every team must find their own best place, which may not be the same across the company. But if a team dislikes a tool or finds it burdensome, they just won’t use it.

The best tool is the one you can get people to adopt. If your team is 100% people who know how to use LateX and created their resumes and gaming character sheets in it, then it would be a fine tool. If you had a team composed entirely of people under 20, maybe a a youtube channel would be the way to go. Since none of us have completely uniform teams, the best thing we can do is to find a tool that everyone can grudgingly agree only sucks a little bit.

In conclusion

  • Use a better search engine
  • Hire experts whenever possible
  • Make it easier to write things by using templates
  • Iterate and keep improving your devops docs process. There’s no solution, just a fix for now.

Talk write-up: Choose Your Own Deployment

Yesterday, I was in Phoenix for their first DevOps Days.


The interesting thing about doing this talk in Twine instead of my beloved Google Slides was how much I had to learn to make it look anything like I wanted. There’s a lot of CSS and Twine-specific syntax. At first, it felt like I was wasting my time and being slow because I know other people know this better than me, but as the project went on, it was honestly delightful to learn something new and get my pages to look like I wanted. There was a lot of fist-pumping success in getting a font to work.

It’s still not perfect, and I need to do some pretty drastic revisions for version 2, but now I know which of the resources are most useful to me and I have a conceptual model that I didn’t have when I started, so I think it will be easier to learn the parts I still don’t have.

It’s been a long time since I’ve had to learn something that had immediate tangible results, instead of concepts and pitches and taking what I already know and distilling it down. It was good to do that, and I should remember to schedule it into my life sometimes.

Why on earth would developers WANT to write documentation?

This week I was at DevOpsDays Minneapolis, which was an amazing, warm, and inclusive event. There were at least three open spaces on documentation, communication, and related problems, and I didn’t propose any of them!

Minneapolis DevOpsDays 2017 logo

The question the devops world keeps asking, over and over again is,

How do I get developers to write anything?

There aren’t a lot of easy answers for this.

The first one is that if you care about developer-produced documentation, you should hire for it. But almost always, the people hiring are not the same as the people feeling the pain of missing documentation, so that may not happen.

The second is that you should hire someone other than the developers to write, if that’s important to you. I’m going to mention that all the times I’ve interviewed at Google, it was to be as an internal systems writer, documenting how things in the company work for other people in the company. Big, smart, companies hire writers for internal documents. But that is a tough budgetary sell.

The third, partial, answer, is that you make it significantly easier for developers to write. You give them forms and templates to fill in instead of expecting them to invent documentation theory. You specifically allocate writing time that can’t be budged for “one more tweak to the feature”.

Here’s the key problem though:

No developer has ever been fired for lack of documentation.

No developer I know of has gotten a bonus for documentation.

There is no stick. There is no carrot. There are a lot of competing demands on a developer’s time. Why on earth would they go do something that is not in their core competency just because you want it?

My instinctive response, whenever my kids want something, is to say, “It’s good to want things.” I think that is the instinctive response of most developers who are asked to do extra work. It’s not malicious, it’s just true. There is no reason they should do it – they won’t lose their jobs, the way they might if they prioritized writing over code. They won’t derive any benefit except the long-term and intangible one of being asked fewer questions. If they like being the source of knowledge, they’d be giving that away. If they like coding, you’re asking them to do something other than what they want. There is no upside.

So knowing that, how can we encourage developers to write documentation that we need?

First, reward it. Give people prizes for writing docs, or for throwing away or revising old docs. This needs to be something outside their usual compensation structure. Gift cards, time off, merit badges, whatever would be meaningful to that person. Ask them. They know what rewards them, because they use it to reward themselves.

A variety of rewards

Second, require it. I don’t mean “You have to write some docs or I will, uh, be disappointed in you.” I mean “The presence of documentation is a quality gate and you cannot release without it.” I mean “QA is using your docs as the test story.”.

Companies are never altruistic, humans are never purely motivated. We need to accept this and stop expecting people to do extra work because it’s the right thing to do, and start making the work not-extra, and the reward tangible.