How does the DevOps role help/damage team cohesion?

Very excited about this book!

I feel a lot of us from R&D lost a lot of touch with the way modern production is handled. I know it's a heavy burden to lay on one book though...

My question is this: I work a lot on production debugging. So pretty excited about tools like kubectl debug etc.
How do you mitigate the security risk of giving developers access to tools like that?

Well, let me interject and say that allowing developers direct access to production resources is a Very Bad Practice, whether you're doing it the DevOps way or no.

I have spent most of my career having the ability to do things that average devlopers aren't allowed to do. I spent over half a decade in the OS Support Group for a major mainframe system. My first rule has always been to never give myself permission to do anything I don't want to get yelled at when it gets damaged or hacked. And by "permission", I don't mean self-discipline I mean hard constraints like security system rules.

When I build a webapp, there are no references to production files or databases coded into the app, They're all externally injected as part of the deployment process. And while I also set up the deployment process, that doesn't mean that I actually deploy production. That's handed off to the properly authorized person or department in the Operations staff. Because there are no internal changes, however, if a production system goes down, I can often diagnose without access to the production code or data because the exact same WAR file is used for development, testing, and production with only the files and databases swapped out; again, via external configuration, not changes to the WAR. There are always harmless test versions of the files and databases for the development and support side so that access is not generally needed to production. I don't need to know secure data and I can't be blamed if it gets damaged.

I aid and abet this process by putting lots of logging code into the apps with appropriate log levels, so that if things do go sour, we can turn on fine-grained tracking to get a better view of the fault. Writing "anti-bugging" code into the app helps as well.

Of course, sometimes all that isn't enough. As I said, I'm generally entrusted to be able to walk into the computer room and lay hands on directly. But it's a last resort and as much as possible, I get things done through the operations staff. I don't hack the database if I can submit a request to the DBA, I don't jam in an experimental WAR without sysadmin support (including fallback processes).

One reason sysadmins like me is because I don't just throw loose code and vague instructions at them, I provide a complete standardized installation process that makes their job easier. That is in large part what DevOps is geared towards for everyone.

I'll offer a diametrically opposite perspective based on my own experience supporting a mission critical P1 application. It was a P1 application because it was supporting part of the business that had to be able to respond very quickly to Zero-Day security vulnerability events, sending updates to affected customers worldwide. The working definition of DevOps for our team was the one that simply states that "The developers carry the pager." And we did. That meant that we were doing development and operations. Corporate operations was still a thing, and they were in charge of the network and infrastructure, basically the platform, which included WebSphere servers, load balancers, the Oracle database servers, etc. Everything else fell to our team of seven, which included a DBA and tester.

We tested in production (something that many people consider bad practice, but it depends)
We debugged in production
We were a full stack team even before the term "full stack" was even a thing (before 2008)

Was what we were doing Very Bad Practice? We didn't think so. To us, it was necessary to work this way in order to support the business and their needs for quick response to very rapidly changing and developing needs of customers. Were we as agile as we could have been? We were agile enough to support a very time-sensitive business function. Zero-Day SLAs are no joke and any delays had significant impact to reputation, not to mention the financial penalties for failure to deliver on the SLA.

I wrote:
We tested in production (something that many people consider bad practice, but it depends)
We debugged in production

In case anyone takes that the wrong way, let me be clear:

While we tested in production, we also tested the heck out of our software prior to releasing to production. I'm proud to say that the last few years our team was in charge of that application, we had ZERO defects. Production issues were platform related, usually something that corporate operations or corp DBAs screwed up.

While we debugged in production, we also tested the heck out of software prior to releasing to production. Again, we had ZERO bugs escape from us. I credit that stat to our practice of Test-Driven Development and collaborative development (pairing/mobbing) and having a strong refactoring discipline. As the tech lead, I was the main driving force behind that but I couldn't have done it without our whole team buying into the practices and whole-heartedly toeing the line.

Full disclosure: I said ZERO bugs escape. That doesn't mean there weren't any known bugs in the system, because there were. They were just in an area of the codebase that even we didn't dare touch for fear of breaking something. The bugs were not significant enough of an operational threat to warrant any of the significant effort required to hunt them down and eliminate, however, so we just learned to live with the occasional annoyance they caused. Here, monitoring and observability was key to keeping our app's availability at the required level. You can't win them all. (shrug)

As to the question of "How does the DevOps role help/damage team cohesion?" I would say this:

There is no DevOps role in my book. There's a DevOps way of working, a culture, a mindset, if you will. But saying there's a specific DevOps role seems to me like an indication of a misunderstanding of what DevOps is. Like I said, for our team it basically boiled down to "The developers carry the pager." One might play a specific role in an organization that has adopted the DevOps model but DevOps is about collaboration across the board and breaking down silos between different functions of the organization, primarily bridging the divide between development and operations, hence the portmanteau "DevOps".

I think that has to do with the size of your team. You can — often must — adopt a more freewheeling approach when you are a small group with many hats.

I've never worn a pager, though. Not even when I was in the mainframe OS support group. THAT, admittedly is not normal, but my boss had become fed up with too many call-ins and insisted on a policy where systems were well-vetted and he was supposed to be the first one called when something blew out.

And while we didn't wear pagers, since our most critical stuff happened in the wee hours, we were generally by the (land-line) phone. And the third-shift lead operator could be annoyingly cheerful when he called us!

Oh, and over the years more than once we did the day-to-day mortgage loan data processing for about two thirds of the USA and Puerto Rico. So most of the really large lending institutions were clients of ours. In a later stint there, I wasn't as close to the daily processing, but, to my shame, I supported the group that made all that pre-2009 merger and acquisition activity possible, to my shame. It was our calculations that caused the creation of institutions that were "too big to fail", as well as the backing data for over-extending credit for mortgage-backed securities.

Then I went to work for the largest privately-held bank in Florida and the auditors would have had kittens if we'd been caught playing games with production systems, emergency or no. More importantly, so would the Federal Government. So we had a very tightly-controlled change control process with specific duties - and access rights - for different parties involved. Every change had to include well-documented recovery and rollback procedures and excepting dire emergencies all changes were scheduled for a particular day  a week in advance.

To me, DevOps basically meant bridging the divide between Development and Production, and I stood across that divide since long before the term (or the neat tools we now have) were a thing.

The one modern trend that does make me nervous is Continuous Integration. CD/CI is fine by me for tweaks to the CodeRanch. It's OK, I suppose if you're Amazon and you need to make changes to the online shopping software. But I dig my heels in when you log in to the money-handling and personal secrets sides of the system. Some things require, in my opinion, more deliberation.

Tim Holloway wrote:
The one modern trend that does make me nervous is Continuous Integration. CD/CI is fine by me for tweaks to the CodeRanch. It's OK, I suppose if you're Amazon and you need to make changes to the online shopping software. But I dig my heels in when you log in to the money-handling and personal secrets sides of the system. Some things require, in my opinion, more deliberation.

Very interesting thoughts. I'd like to get more clarity on your definition of CI and your concerns about it. For me CI is very closely related to Trunk-Based Development (TBD) and how well team members are collaborating with each other as they work on the same codebase. For me, CI is mainly a way to get faster feedback about changes that are happening to the same area of the codebase that you're working on.

I'm not sure where the "money-handling and personal secrets side of the system" comes in. Can you elaborate a little more about the concerns around these with respect to your definition of Continuous Integration?

Interesting thoughts from everyone here. FYI when I talk about debugging in production I'm mostly referring to tools such as Lightrun which provide the means to do this securely and at scale.

There are cases where one needs to go in with kubectl debug and deal with the low level nuts and bolts. This is a more difficult case and a very risky one so I very much get all the stances. I'm actually happy to see there's no consensus on this since I can't even get that equilibrium in my own brain.  

Our debugging in production was very low-tech and low-fi : all we had to lean on was Log4j and going through the logs in WebSphere (blech!) We'd set environment variables to modify log levels so we could get more fine-grained log messages to help track down problems in production.

We never did get to a point where we were doing Blue/Green deployments so we had as close to production like environments on which to debug things.

Tim Holloway wrote:I've never worn a pager, though.

The last time I wore a pager was in the mid 90s for a job I had in Singapore. I don't think I've ever literally worn a pager for work here in the US. At the job I mentioned previously, we had an enterprise paging system that would send SMS and/or email notification to whoever was registered as being on call for the week. That usually meant a free $100 for whoever was on call, because we hardly ever had any unscheduled outages.

Junilu Lacar wrote: I'd like to get more clarity on your definition of CI and your concerns about it. For me CI is very closely related to Trunk-Based Development (TBD) and how well team members are collaborating with each other as they work on the same codebase. For me, CI is mainly a way to get faster feedback about changes that are happening to the same area of the codebase that you're working on.

I'm not sure where the "money-handling and personal secrets side of the system" comes in. Can you elaborate a little more about the concerns around these with respect to your definition of Continuous Integration?

Disclaimer: I've never worked in a CI/CD shop myself. But as I understand the concept, it sort of takes Agile one step further. If Agile is a process where you pass through successive refinements to deliver progressively more functional releases (tailored by interim user feedback), then to me, CI/CD would be a never-ending "Agile" where once the project is "finished", it sees regular ongoing change. Something that you wouldn't want from, say Accounts Payable, where the majority of changes might be annual tweaks to accomodate new tax laws. but which are very important if you want to continue to be a "hip" online presence on the Social Web.

The "Money handling and social secrets" side is different. You may have the world's fanciest online catalog, but once someone actually takes their shopping card to the checkout, the rules change. At most sites, in fact, the actual payment handling is passed off to a third party (such as PayPal or a credit card processor). Hackers can only do a limited amount of damage if they can spy on your shopping cart, but the payment system has to have things like your credit card or bank account credentials, as well as possible hooks to fraud abuse and other security features. You do NOT want that information leaking out, nor do you  want false orders being made. So my preference would be to test that stuff extra hard before it enters production and to avoid meddling with it any more than absolutely necessary once it works.

If you'll note, I too use logging (usually log4j) with controllable log levels (and loggers) as a primary production debugging tool. I think that the difference here is that I'm not letting developers alter code to change log levels, but more often making the log control be one of the (secured) sysadmin features. So I can tell the sysadmin to turn logging on and off on the failing code.

Actually, way back when, computers often had a set of binary switches on the operator's console (sense switches). The exact number and use varied on the manufacturer and model, but it was not uncommon that you could have the operator flip a sense switch to turn debugging on and off. Some Unix-style apps use a software equivalent to this even today.

The IBM System/360 could even be run in single-step diagnostic mode from the console but that was an extreme measure. When you're running an increasingly complex multi-user OS, parking the entire box to debug one program via hardware is prohibitive. I've never seen it done myself.

Junilu Lacar wrote:

Tim Holloway wrote:I've never worn a pager, though.

The last time I wore a pager was in the mid 90s for a job I had in Singapore. I don't think I've ever literally worn a pager for work here in the US. At the job I mentioned previously, we had an enterprise paging system that would send SMS and/or email notification to whoever was registered as being on call for the week. That usually meant a free $100 for whoever was on call, because we hardly ever had any unscheduled outages.

It's extremely uncommon for mainframe OS support people not to have at least a duty pager. Ironically, in the same shop, there were 2 such pagers passed out every weekend to applications developers.

Tim Holloway wrote:
If you'll note, I too use logging (usually log4j) with controllable log levels (and loggers) as a primary production debugging tool. I think that the difference here is that I'm not letting developers alter code to change log levels, but more often making the log control be one of the (secured) sysadmin features. So I can tell the sysadmin to turn logging on and off on the failing code.

I was starting to look at JMX and the feasibility of introducing MBeans into the legacy app's design, precisely for allowing us to control log levels via a JMX console among other things. Never got beyond investigating and conceptualizing it though since the app was scheduled to End Of Life in a couple of years at that point and it wasn't worth the time and effort. We normally had sudo access rights to the pre-production and production servers our app components were deployed to (we had a few Linux systems that provided supporting services to the web app).

JMX is an interesting option. I think that its main disadvantages are these:

1. It requires access to the a custom (JMX) port of the JVM, if memory serves.
2. If you're running a paranoid shop, it's just another way for developers to touch production OR you have to train operators how to use the JMX console app (lots of luck with that around here, where cheap operators are more important than skilled ones).

But JMX is easy to implement. The early versions of JBoss were all based on JMX, I think, although it had its own UI to control the MBeans.

Overall, I'd find it easier just to add a sysadmin web page if it's a webapp. My really complex ones frequently have both sysadmin and useradmin security roles (and often look-but-don't-touch auditor roles).