DevOps Is Bullshit: Why One Programmer Doesn’t Do It Anymore

I’ve always been handy with hardware. I was one of “those kids” you hear about that keeps taking things apart just to see how they work – and driving their parents nuts in the process. When I was a teenager, I toyed with programming but didn’t get serious with it until I decided I wanted to get into graphic design. I found out that you don’t have to write HTML yourself, you can use programming to do it for you!

But I never stopped tinkering with hardware and systems. I used Linux and BSD on my desktop for years, built my LAMP stacks from source, and simulated the server environment when I couldn’t – when I used windows for work, and when I eventually adopted Apple as my primary platform, I first started with cross-compiled versions of the components, and eventually got into virtualization.

In the early days (maybe 10 years ago) there seemed to be few programmers who were like me, or if they were, they never took “operations” or “sysadmin” jobs, and neither did I. So there was always a natural divide. Aside from being a really nice guy who everyone likes, I had a particular rapport with my cohorts who specialized in systems.

I’m not sure exactly what it was. It may have been that I was always interested in the finer details of how a system works. It may have been my tendency to document things meticulously, or my interest in automation and risk reduction. It could have just been that I was willing to take the time to cross the divide and talk to them, even when I didn’t need something. It may have just boiled down to the fact that when they were busy, I could do things myself, and I wanted to follow their standards, and get their guidance. It’s hard to tell, even today, as my systems skills have developed beyond what they ever were before, but the rapport has continued on.

And then something happened. As my career progressed, I took on more responsibilities and did more and more systems work. This was partly because of the divide widening to some extent at one particular job, but mostly because, I could. Right around this time the “DevOps Revolution” was beginning.

Much like when I was a teenager and everyone needed a web site, suddenly everyone needed DevOps.

I didn’t really know what it was. I was aware of the term, but being a smart person, I tend to ignore radical claims of great cultural shifts, especially in technology. In this stance, I find myself feeling a step or two behind at times, but it helps keep things in perspective. Over time, technology changes, but true radicalism is rare. Most often, a reinvention or revisiting of past ideas forms the basis for such claims. This “DevOps” thing was no different. Honestly, at the time it seemed like a smoke screen; a flashy way to save money for startups.

I got sick of tending systems – when you’re doing it properly, it can be a daunting task. Dealing with storage, access control, backups, networking, high availability, maintenance, security, and all of the domain-specific aspects can easily become overwhelming. But worse, I was doing too much front-line support, which honestly, at the time was more important than the programming it was distracting me from. I love my users, and I see their success as my success. I didn’t mind it, but the bigger problems I wanted to solve were consistently being held above my head, just out of my grasp. I could ignore my users or ignore my passion, and that was a saddening conundrum. I felt like all of the creativity I craved was gone, and I was being paid too much (or too little depending on if you think I was an over paid junior sysadmin or an under paid IT manager with no authority) to work under such tedium. So I changed jobs.

I made the mistake of letting my new employer decide where they wanted me to go in the engineering organization.

What I didn’t know about this new company was that it had been under some cultural transition just prior to bringing me on board. Part of that culture shift was incorporating so-called “DevOps” into the mix. By fiat or force.

Because of my systems experience, I landed on the front line of that fight: the “DevOps Team”. I wasn’t happy.

But as I dug in, I saw some potential. We had the chance to really shore up the development practices, reduce risk in deployments, make the company more agile, and ultimately make more money.

We had edicts to make things happen, under the assumption that if we built it, the developers would embrace it. These things included continuous integration, migrating from subversion to git, building and maintaining code review tools, and maintaining the issue tracking system. We had other, less explicit responsibilities that became central to our work later on, including developer support, release management, and interfacing with the separate, segregated infrastructure department. This interaction was especially important, since we had no systems of our own, and we weren’t allowed to administer any machines. We didn’t have privileged access to any of the systems we needed to maintain for a long time.

With all the hand wringing and flashing of this “DevOps” term, I dug in and read about it, and what all the hubub was about. I then realized something. What we were doing wasn’t DevOps.

Then I realized something else. I was DevOps. I always had been. The culture was baked into the kind of developer I was. Putting me, and other devs with similar culture on a separate team, whether that was the “DevOps” team or the infrastructure team was a fundamental mistake.

The developers didn’t come around. At one point someone told a teammate of mine that they thought we were “IT support”. What needed to happen was the developers had to embrace the concept that they were capable of doing at least some systems things themselves, in safe and secure manner, and the infrastructure team had to let them do it. But my team just sat there in the middle, doing what we could to keep the lights on and get the code out, but ultimately just wasting our time. Some developers starting using AWS, with the promise of it being a temporary solution, but in a vacuum nonetheless. We were not having the impact that management wanted us to have.

My time at this particular company ended in a coup of sorts. This story is worthy of a separate blog post some day when it hurts a little less to think about. But lets just say I was on the wrong side of the revolution and left as quickly as I could when it was over.

In my haste, I took another “DevOps” job. My manager there assured me that it would be a programming job first, and a systems job second. “We need more ‘dev’ in our ‘devops'”, he told me.

What happened was very similar to my previous “DevOps” experience, but more acute. Code, and often requirements, were thrown over the wall at the last minute. As it fell in our laps, we scrambled to make it work, and work properly, as it seemed no one would think of things like fail over or backups or protecting private information when they were making their plans. Plans made long ago, far away, and without our help.

This particular team was more automation focused. We had two people who were more “dev” than “ops”, and the operations people were no slouches when it came to scripting or coding in their own right.

It was a perfect blend, and as a team we got along great and pulled off some miracles.

But ultimately, we were still isolated. We, and our managers tried to bridge the gap to no avail. Developers, frustrated with our sizable backlog, went over our heads to get access to our infrastructure and started doing it for themselves, often with little or no regard for our policies or practice. We would be tasked with cleaning up their mess when it was time for production deployment – typically in a major hurry after the deadline had passed.

The original team eventually evaporated. I was one of the last to leave, as new folks were brought into a remote office. I stuck it out for a lot of reasons: I was promised transfer to NYC, I had good healthcare, I loved my team. But ultimately what made me stick around was the hope, that kept getting rebuilt and dashed as management rotated in and out above me, that we could make it work.

I took the avenue of providing automated tools to let the developers have freedom to do as they pleased, yet we could ensure they were complying with company security guidelines and adhering to sane operations practices.

Sadly, politics and priorities kept my vision from coming to reality. It’s OK, in hindsight, because so much more was broken about so-called “DevOps” at this particular company. I honestly don’t think that it could have made that much of a difference.

Near the end of my tenure there, I tried to help some of the developers help themselves by sitting with them and working out how to deploy their code properly side-by-side. It was a great collaboration, but it fell short. It represented a tiny fraction of the developers we supported. Even with those really great developers finally interfacing with my team, it was too little, too late.

Another lesson learned: you can’t force cultural change. It has to start from the bottom up, and it needs breathing room to grow.

I had one final “DevOps” experience before I put my foot down and made the personal declaration that “DevOps is bullshit”, and I wasn’t going to do it anymore.

Due to the titles I had taken, and the experiences of the last couple of years, I found myself in a predicament. I was seen by recruiters as a “DevOps guy” and not as a programmer. It didn’t matter that I had 15 years of programming experience in several languages, or that I had focused on programming even in these so-called “DevOps” jobs. All that mattered was that, as a “DevOps Engineer” I could be easily packaged for a high-demand market.

I went along with the type casting for a couple of reasons. First, as I came to realize, I am DevOps – if anyone was going to come into a company and bridge the gap between operations and engineering, it’d be me. Even if the company had a divide, which every company I interviewed with had, I might be able to come on board and change things.

But there was a problem. At least at the companies I interviewed at, it seemed that “DevOps” really meant “operations and automation” (or more literally “AWS and configuration management”). The effect this had was devastating. The somewhat superficial nature of parts of my systems experience got in the way of landing some jobs I would have been great at. I was asked questions about things that had never been a problem for me in 15 years of building software and systems to support it, and being unable to answer, but happy to talk through the problem, would always end in a net loss.

When I would interview at the few programming jobs I could find or the recruiters would give me, they were never for languages I knew well. And even when they were, my lack of computer science jargon bit me – hard. I am an extremely capable software engineer, someone who learns quickly and hones skills with great agility. My expertise is practical, however, and it seemed that the questions that needed to be asked, that would have illustrated my skill, weren’t. I think to them, I looked like a guy who was sick of systems that was playing up their past dabbling in software to change careers.

So it seemed “DevOps”, this great revolution, and something that was baked into my very identity as a programmer, had left me in the dust.

I took one final “DevOps” job before I gave up. I was optimistic, since the company was growing fast and I liked everyone I met there. Sadly, it had the same separations, and was subject to the same problems. The developers, who I deeply respected, were doing their own thing, in a vacuum. My team was unnecessarily complicating everything and wasting huge amounts of time. Again, it was just “ops with automation” and nothing more.

So now lets get to the point of all of this. We understand why I might think “DevOps is bullshit”, and why I might not want to do it anymore. But what does that really mean? How can my experiences help you, as a developer, as an operations person, or as a company with issues they feel “DevOps” could address?

Don’t do DevOps. It’s that simple. Apply the practices and technology that comprise what DevOps is to your development process, and stop putting up walls between different specialties.

A very wise man once said “If you have a DevOps team, you’re doing it wrong“. If you start doing that, stop it.

There is some nuance here, and my experience can help save you some trouble by identifying some of the common mistakes:

  • DevOps doesn’t make specialists obsolete.
  • Developers can learn systems and operations, but nothing beats experience.
  • Operations people can learn development too, but again, nothing beats experience.
  • Operations and development have historically be separated for a reason – there are compromises you must make if you integrate the two.
  • Tools and automation are not enough.
  • Developers have to want DevOps. Operations have to want DevOps. At the same time.
  • Using “DevOps” to save money by reducing staff will blow up in your face.
  • You can’t have DevOps and still have separate operations and development teams. Period.

Let me stop for one moment and share another lesson I’ve learned: if it ain’t broke, don’t fix it.

If you have a working organization that seems old fashioned, leave it alone. It’s possible to incorporate the tech, and even some of the cultural aspects of DevOps without radically changing how things work – it’s just not DevOps anymore, so don’t call it that. Be critical of your process and practices, kaizen and all that, but don’t sacrifice what works just to join the cargo cult. You will waste money, and you will destroy morale. The pragmatic operations approach is the happiest one.

Beware of geeks bearing gifts.

So lets say you know why you want DevOps, and you’re certain that the cultural shift is what’s right for your organization. Everyone is excited about it. What might a proper “DevOps” team look like?

I can speak to this, because I currently work in one.

First, never call it “DevOps”. It’s just what you do as part of your job. Some days you’re writing code, other days you’re doing a deployment, or maintenance. Everyone shares all of those responsibilities equally.

People still have areas of experience and expertise. This isn’t pushing people into a luke-warm, mediocre dilution of their skills – this is passionate people doing what they love. It’s just that part of that, is launching a server or writing a chef recipe or debugging a production issue.

As such you get a truly cross functional team. Where expertise differs, first, there’s a level of respect and trust. So if someone knows more about a topic than someone else, they will likely be the authority on it. The rest of the team trusts them to steer the group in the right direction.

This means that you can hire operations people to join your team. Just don’t give them exclusive responsibility for what they’re best at – integrate them. The same goes for any “non deveoper” skillset, be that design, project managment or whatever.

Beyond that, everyone on the team has a thirst to develop new skills and look at their work in different ways. This is when the difference in expertise provides an opportunity to teach. Teaching brings us closer together and helps us all gain better understanding of what we’re doing.

So that’s what DevOps really is. You take a bunch of really skilled, passionate, talented people who don’t have their heads shoved so far up their own asses that they can take the time to learn new things. People who see the success of the business as a combined responsibility that is eqully shared. “That’s not my job” is not something they are prone to saying, but they’re happy to delegate or share a task if need be. You give them the infrastructure, and time (and encouragement doesn’t hurt), to build things in a way that makes the most sense for their productivity, and the business, embracing that equal, shared sense of responsibility. Things like continuous integration and zero-downtime deployments just happen as a function of smart, passionate people working toward a shared goal.

It’s an organic, culture-driven process. We may start doing continuous deployment, or utlize “the cloud” or treat our “code as infrastructure” but only if it makes sense. The developers are the operations people and the operations people are the developers. An application system is seen in a holistic manner and developed as a single unit. No one is compromising, we all get better as we all just fucking do it.

DevOps is indeed bullshit. What matters is good people working together without artificial boundaries. Tech is tech. It’s not possible for everyone to share like this, but when it works, it’s amazing – but is it really DevOps? I don’t know, I don’t do that anymore.

This entry was posted in commentary, culture, devops, editorial and tagged , . Bookmark the permalink.

93 Responses to DevOps Is Bullshit: Why One Programmer Doesn’t Do It Anymore

  1. lol says:

    I’d like to read your article, but it’s unreadable for me due to the light gray text on a white background.

  2. lol says:

    For anyone else who can’t read it… open your js console and do: = “black”

  3. Torkel Lyng says:

    I agree with much of what you’re saying. I can’t say that it’s bullshit, but it’s mostly interpreted wrong by many people (or by me). DevOps is first DEVELOPER, hence the Dev prefix and second Operations. It’s easier to train a developer to do sane operating procedures than to teach a sysadmin the developer mindset (testing, revision control, breaking down to logical units and similar). DevOps is making the old SysOps positions deprecated. SysOps require manpower or attention to do something, at a general term atleast. While in DevOps we strive for an fulle automated system, from the application level down to bare metal, which is out of scope for any normal system administrator. This will become more evident in the years to come when microservices get more traction. With microservices we get a more volatile, highly dynamic environment where manual labor is not an option. Another evidence that the traditional system administrator is getting deprecated is container (docker and similar). Container put the entire OS in the hand of the developers, all required from system administration is a kernel which support containers and maybe storage solutions. A small team of 5 to 10 persons can easily manage 10 000 servers in such a setup (not at containerlevel, but container-host level).

    Those who believe DevOps is a developer and a sysop do not see the entire picture, sadly.

    We’ve had this t-shirt available for years: , a more modern version would have said something like:
    “I will replace your infrastructure with code”, hence infrastructure as code.

    • dotnetchris says:

      If your microservices themselves are volatile you are doing it wrong. Services need to be reliable and consistent. The code inside the services should be volatile, if it’s not, why are you encapsulating it in the first place? But service contracts and deployment are mostly rigid.If you continuously change these clients will continually break. If you’re breaking clients, you’re failing.

      • Torkel Lyng says:

        Volatile as in may go up and down at anytime, move from one datacenter to another etc, not volatile in API contracts.

      • Torkel Lyng says:

        Or similar to what netflix has done, introducing a Chaos Monkey to their environment that intentionally crash and burn different parts of their production environment. With the correct strategy and infrastructure it should be possible to achieve five nines service times even with seriously broken software.

    • matthewrdunbar says:


      As an experienced SysAdmin/Systems Engineer who *does* not only understand but *promotes* testing, revision control, modularization and reusability, iterative-design-life-cycle refactoring, rapid test rapid fail, ci/cd, et al not just as buzzwords, but as practical and pragmatic practices in ever-shifting operational environments, who also frequently has to rework or completely rewrite scripts delivered by developers who think they know something about system administration but really fail to grasp both the broader and deeper aspects of how system elements are inter-related and can have much farther reaching consequences if not taken into account, I think you may have a somewhat one-sided perspective.

      There are a lot of smart folks in different roles with varied backgrounds. There are a lot of folks who happen to be in Ops-oriented positions with broader backgrounds in computer science than you may have run into in your experience. By the same token, there are dev folk who not only fail to look at the broader environment and architecture into which they’re introducing their code, but fail to see why they should, because they don’t think beyond the black box. Don’t get stuck on roles or the titles when really it’s about people and abstract systems, and, as you touched on, education and mindset.

      Strong collaboration, leveraging team members individual strengths, and continuous learning are what drive good development and good deployment practices, regardless of the labels and metaphors folks may be using at any given time.

      That being said, I too agree with a lot of what’s here, and at least part of what I think you’re saying. I think though that the issue isn’t whether someone happens to be Ops or Dev that’s the differentiator between who grasps the potential advantages of the advancement of DevOps as a concept, but to what degree they grok modern design patterns and how they apply to systems implementation as a whole as an expansion of the concepts beyond individual software projects.

      Older ‘traditional’ concepts of what constitutes an operations environment are changing, but changing technology has always been the case. Organizations that fail to adapt and take advantage of the new technologies that are rapidly maturing into useful tools aren’t going to continue to be competitive nor cost effective. Good operations teams have to varying degrees insulated the development side of the house to one degree or another from the negative effects of the actual scope of those changes while striving to provide access to the advantages offered by new advances. Now, more ever, it is not change, but the *rate* of change combined with ever increasing scale without ever increasing manpower (who are we kidding, for most organizations that’s with increased Ops staff reductions) that’s driving both automation and abstraction.

      If you look at the technologies that are now driving these changes – virtualization, infrastructure as code, Docker, et al, those are *not* simply efforts driven by developers who want more control of their operations environments. They’re driven by folks faced with the challenges of doing operations in a modern landscape of Large Installation System Administration who want smarter, saner, more streamlined delivery of support, who want better integration and deployment methodologies — by folks who straddle the divide between ‘traditional’ developmental and operational roles already — by ops guys who code, and developers who understand operational requirements and maintain systems — and by collaboration toward common goals.

      Docker, by the way, won’t (simply can’t) replace your SysOps role. You still have to have someone building your environments and setting up systems level automation for your containers. And, you have to have someone who understands all of the various the underlying requirements to make that work. Those role differentiations developed not simply because of some perceived hardware/software duality, but because there’s a reasonable (still artificial) division between folks focused on specific, relative, deliverables addressing a specific requirement (traditional dev), and those focused upon broader, common, universal, underlying services (traditional ops). However you choose to label it, or parse out the responsibilities, someone on your team still needs to be looking at systems and services as well as application level security, patching, data protection (security, integrity, backup), regularity compliance if appropriate (SOX, HIPPA, PCI, et cetera), log management, storage (local/physical, networked, and near-line), and so on. Whether you label that team member an Ops guy or a Dev guy or a DevOps guy, the elements of the role are still there. Same with NetOps and broader architectural considerations. Maybe you can offload your entire de/qa/staging/production environments to AWS, but you’ve still got some local physical assets that somebody’s got to deal with. (Hardware is an aspect of a Role, whether or not that’s SysOps.)

      There are a lot of elements here, and as the OP has observed, how organizations grow, adapt and change to take advantage of these new developments are all going to differ. Bolt on, whole-cloth, adoption isn’t likely to work for larger organizations. Arbitrarily relabeling a team or a role and trying to make it work simply by paying it lip service isn’t going to be effective. Dysfunctional organizations are likely to remain dysfunctional. Organizations that are already highly collaborative who are also used to periodically advancing their tools sets will have far less trouble. For some, it’s a radical departure from past practice, for others it’s just another logical step in evolutionary progress.

      ‘DevOps’ as a concept, a holistic view of CIS, without silos, is a lot easier to implement from ‘the ground up’ when you’re starting from scratch. Few folks have that luxury. What’s essential is to find the right pragmatic balance for your organization. Take what’s useful for you. Run with it as far as it’s helpful, and don’t waste cycles trying to shoehorn every last thing into it all at once. Take a chunk, factor, review, adapt, refactor. There are different approaches and different balances for different situations and organizations. There are a lot of potential benefits that come from these concepts. I’m a proponent of a holistic view and highly collaborative teams. We just shouldn’t get hung up on labels, or the newest metaphors or buzzwords, or think that there’s only one solution that’s going to fit every organization or circumstance.

      • Torkel Lyng says:

        I agree. In my environment I’ve not seen that mindset from our system administrator at the time being, neither from most of the developers that being said. DevOps require some apsect of a developer oriented mindset when it comes to abstractions and other integration routines (testing, ci, coverage and similar) which the system administrators I know don’t have to a satisfying degree. This also requires the developers to have sysops oriented mindsets and skills when it comes to different tasks such as storage, database, replication and failover etc. The only thing I know for certain is that in the future a tighter collaboration between the ‘roles’ are required (from my experience / environments). Todays system administrators need to be more tightly included in the development of different products/services and developers need to be more aware of the actual environment the application is running in and how it operates. All parties are required to participate with actual code. With time we would not be able to differentiate between the roles.

    • I haven’t met a single developer in my 25 year sysadminning career who I would leave in charge of any complex environment. It is not easy to tech developers sysadmin skills, primarily because most don’t want to know. It takes a lot of experience to appreciate why one does things a certain way, why never to do things a certain way, etc. Developers always take the easiest way, which is almost always the wrong, wrong, wrong way. A look at any enviroment managed by “devops” usually reminds me of looking at a junior developers laptop. You treat systems that way and it is a disaster. I suspect strongly that the immense popularity of docker is simply because none of these people know how to integrate multiple versions of software into a system and still keep it maintainable (and updateable). As a result, “devops” has brought the entire industry to the brink of catastrophe, where no one has any clue what software is in their own stacks, where everyone is pulling unsigned code directly into them with no chain of responsibility or authority. I can’t wait until the next industry disaster caused by a compromised node module or ruby gem. You folks haven’t the faintest clue as to what you’re doing. Security is thrown out the window, because it is boring, and difficult. This is the situation linux distributors solved in the 1990’s. It is the reason Linux became so popular. Devops has brought it down. We are all now where windows was 20 years ago. Congratulations!

      • unbeleevabal says:

        egad… I was just out googling for a drift management tool, not “the truth of the matter”..

      • Deix says:

        KISS/YAGNI are your answers. Some procedures even the term “procedure” start to be OBSOLETE now in modern indus management (IT or not). Freedom spirit is what bring back everyone to earth with pragmatic approaches.

        Nostalgy will however stay for ever for some…

      • Mark B says:

        I entirely agree. Devops has done nothing but totally confuse the role of developers (who don’t want and shouldn’t need to understand operations), and destroyed the carefully detailed practices of operations people tasked with keeping a 99.97 uptime. Leaving no one capable of running any robust system, that can handle any functional change, that can be secure and run with any sense of ongoing uptime. It’s a shit fight where a majority of poor IT management who couldn’t run a successful IT shop threw everything in the air in the hope that by some random chance it would fall back to Earth in a working configuration (devops pffft).. Sadly, with 25 years of hard fought experience running pristine production systems with brilliant uptime levels and the ability to manage daily change I’m now incapable of getting a job paying more than a graduate cause no one left understands the need for properly run IT systems. Thinking Devs can dump it all in docker and it’ll just work and that systems management can be handled by people who never understood it to begin with. Well, take a look around CIO’s. Maybe ask some customers. Most people on the floor in the business think their IT is fucked, is down more than its up and that things ran so so much better 20 years ago when IT was in house and the focus was on the business goals. Rather than every IT shop being full of outsourced monkeys concentrated only on the marketing and buzz words of the day. Devops will be looked upon as the biggest IT abortion of all time in years to come. Meanwhile great Devs are being sidelined by OPS requirements they could care less about and great admins are unemployed, having tried several times to integrate into Devops before realising devops indeed is bullshit and that the only place left for great Sysadmins is to get the fuck out of IT, cause it’s all gone to hell and it shows in the failed end products made of endless non standardised languages, ideas and random guesswork of a group of people’s who’s days consist more of chaos than productivity.

    • darkfader says:

      A lot of uninformed bs from someone who has (opposed to the blog owner) never sat down to learn about ops.
      FYI: LinkedIn Ops happily managed a fleet of real servers much larger than your container example with a smaller team since years.
      And the “dev mindset” being about testing is quite funny too when it’s been decades where Ops tried to explain to their dev colleagues how to test stuff and deliver quality.
      I also love how you forgot about backup and monitoring and all that.

      finally, the infra as code thing is, like devops just a bastardized version of a sysadmin term / concept, named infrastructure administration.

      All I see in your dev mindset is ignoring advice for years, pushing away those who try to help and whom your “output” hurt day and night. Then at last grasp it and run around preaching what awesome idea *you* came up it, and still being an ass about it.

      The best way to do that is to shit all over the people who came up with the ideas and push them out as fast as possible so you when some kid wipes out your consul data you can just keep up pretend noone could have _ever_ foreseen, write a really great root cause analysis to put on twitter and never be held accountable for never having stopped to learn from people you don’t respect.

    • Jean-Phillipe Ennui says:

      “It’s easier to train a developer to do sane operating procedures than to teach a sysadmin the developer mindset (testing, revision control, breaking down to logical units and similar). ”

      People keep saying this over and over and I’m sorry, after the last few years of watching developers completely fuck up operations and stubbornly reinvent the wheel over and over for problems solved 20 years ago, it isn’t true. At ALL. Competent sysadmins have always coded and have always automated systems since forever. Devops is bullshit because it is just sysadminning, you’re just using python instead of perl. Except that the people who came up with the word “devops” hated sysadmins. Grasping the git workflow and developer practices are EASY. Sysadmins wanted something like git forever. We did not use source control, generally, because it was horrible, not because we have some genetic aversion to source control.

      • valdukas says:

        Completly agree. And I thought that sysadmins always did and do automation, and now it happens that only DevOps do it. Sysadmin is (was?) said to be good when from side view he is doing nothing great deal of the time – thats primary due to automation, clever infrastructure and wise choices of hardware and software. And the title Systems Administrator is very good and relevant because its the system that produces results not the individual components. Now DevOps – while it maybe started more like developers can do their own operations due to cloud and virtualization technologies, nowadays most DevOps jobs are just Sysadmin working in Software Develpment enviroment. Some employers think that it could be Developer doing DevOps but its rarely true. Of course due to lack of proper definition DevOps could be anything related to computer systems and this resulted in hudreds of articles and discussions about this term. It’s a distraction.

  4. Anne says:

    Reblogged this on Running Agile and commented:
    Interesting read. Feel like there’s much to ponder here.

  5. Pingback: The better Internet moments |

  6. dotnetchris says:

    I think you’ve missed out on something very important in your career. You own your resume. You own your experience. To a reasonable extent you even own your job titles as you include on your resume.

    If you found it difficult to showcase as yourself as a developer to recruiters it’s because you failed to showcase yourself as a developer on your resume. For example, if your job title was DevOps Engineer you’re just as reasonably able to rebrand yourself as Development Engineer, Software Engineer and so forth. Just because an organization anointed you with some jumble of words doesn’t mean that’s branded on your forehead. The important part is to know where latitude / creativity ends and lying begins. If you’re going to stick “Director” or “Manager” on your resume and it wasn’t part of your previous title you better have a powerful argument for it.

    There’s also a corollary, I’ve actually recommended to a person previously to remove “Director” from their resume. To insulate themselves from being associated with the organization’s failure which they clearly could have had no part in for how short their tenure was, but it wouldn’t stop a person from laying undo blame.

    If you want to be a developer you dial down your operations talk on your resume and turn up the software side or vice-a-versa.

    • Wilfred Godfrey says:

      I think perhaps he ran into this issue – that DevOps is such a sought-after role recruiters were honing in on any DevOps related experience.

      I’ve had this before. I have some Rails experience but nothing to write home about. Several recruitment agents narrowed in on that, and pursued it with as much enthusiasm as they did my 10 years of experience in Javascript, interface design and full-stack development. Most ignored the title on my resume.

      So when a recruiter wants a certain type of person to fit a role, they tend to go out of their way to make applicants fit that role. It’s no longer about what the applicant wants necessarily. If the role is in high-demand then that’s going to skew things.

      • dotnetchris says:

        So delete it from your resume. You can always speak of it in an interview if it’s relevant or just bring it up. Most interviews always have the question “so what else do you do” generic questions and that’s a great time to bring up “well actually i’ve looked into ROR…” blah blah blah

      • dotnetchris says:

        I wish there was an edit button on wordpress comments…

        Choosing what you delete from your resume is just as equally important as what you choose to place in it.

      • jjmojojjmojo says:

        I actually included things like my blog, github link and The Johnson Pyramid of Programmer Greatness when I talked to recruiters – they would routinely omit them when they’d pass my resume on to the hiring manager.

      • jjmojojjmojo says:

        As for omitting things, a 3-5 year gap is hard to explain. Besides that, I’m proud of what I’ve accomplished in these so-called “DevOps” jobs. The “DevOps” itself may have been an epic fail, but I’ve learned so much, and it’s made me, overall, a better engineer.

    • jjmojojjmojo says:

      You make a valid point, but it’s a little misplaced here.

      Even if I were inclined to spin what I did in these so-called “DevOps” jobs (and I’m not), it’d be hard to make it sound like I was doing software when I just wasn’t. And even the software I was writing, was to automate systems and manage other software – it was very operations heavy as well.

    • @dotnetchris is right. You sound like someone who is drifting at sea, waiting to be rescued by people with the right intentions. You can resolve this situation if you want to. Decide what you want to be doing and do that. Don’t settle (within reason). You should not be taking jobs that involve using languages that you do not prefer to code in. Imagine a classical piano player taking a job as a trombone player in a marching band because a recruiter asked him to. You may simply need to do more planning and research before you act.

      • jjmojojjmojo says:

        You sound like someone who didn’t read all of my post. I’m not adrift, I’m in a software engineering job now, and we’re doing “devops” right.

        Careers are never a straight line, no matter how much you plan and research. You, and @dotnetchris’ attitudes are such a cop out. “You should learn how to write a resume” “You’re just not trying hard enough.”… Give me a break!

        This attitude is a deflection of all the real issues with DevOps that my post brings up. This was not a post about recruiters failing or my career not working out the way I want. I mention my experiences because they truly reflect the state of “devops” in our industry. Those experiences have helped shape my current views and methods.

        Read the comments here, on Hacker News and (to a lesser extent because they are meanie-pants) Reddit. I’m not the only one who has had this sort of experience. I’m sure we’re all not just fussy prima donnas who don’t know how to tailor a resume to a job.

  7. Nathan Reese says:

    I’m on the other side of the chasm: I love working with developers and users. I’ve been a unix Sys Admin for 20 years now facing my job being torn to shreds by management hipsters who think they need DevOps because they read it was the thing startups in San Francisco need. God only knows why.

    Slowly being forced to do it, or forced to go jobless as everybody abandons traditional Operations roles in favor of what amounts to, in the long run, an often unnecessary role in the organization.

    While you miss being a coder, I miss being a Sys Admin. Your post hurt me to read. I thank you for bringing this to light, and hope most people will find some good takeaways here.

    • dotnetchris says:

      Retool your angle to be the security and IT regulatory compliance side.

      No matter how DevOp-y a person is, there is not going to be a developer that has the slightest interest in intrusion detection systems, HIPAA compliant datastorage, etc. Even configuration management, merely keeping developers from manipulating production databases as is required by numerous regulatory authority.

      • dotnetchris says:

        Also make sure you get on board with the cloud. Show you’re role is just as relevant with the cloud as it was before (because it certainly is).

      • Nathan Reese says:

        Already fully on-board with AWS and VMWare, have been for years. I’ve written PCI and SAS70 compliance manuals over 200 pages. But my larger point is this:

        I can’t be the only one feeling the pinch because I don’t find Jenkins (which I know anyway) fits in to the “what I want to work on all day” aspect of having a job in Ops. I also recognize that times and industries change. We have to change with them, even when it aint no fun. ;)

    • One of the biggest hurdles I have faced (I too am a now-graybearded Ops guy) is that 90% of the time I am pitched a ‘DevOps’ position, it is, as the OP states, a ‘config mgmt in the cloud’ position. The really dangerous ones are those where the organization wants or needs to run their own metal, but refuses to or cannot understand that the actual ‘ops’ part of the job is now no longer ‘redundant’ (not that it ever really was) but is critical. Many orgs just refuse to grasp the level of abstraction that AWS and its ilk provide, and how much effort that abstraction takes to design, build and maintain.

  8. Anand Jeyahar says:

    Reblogged this on Just another complex system.

  9. I think you are responsible for correcting management that does know any better; especially if you have background in something they are trying to put together. If you already made a ‘DevOps’ group before figuring out that you don’t have access to infrastructure; something does not add up.

    Different groups within ‘engineering’ do not separate people on a smaller scale. Unless those groups proactively decide to do so which is counter productive . When company grows, you can usually avoid this by splitting ‘DevOps’ to different parts of the architecture (hopefully it’s not a giant brick) together with other engineers.

    They still function as a ‘single’ entity with a ‘single’ goal: supporting vital of your product.

    The only reason for ‘DevOps’ name is to draw a fine line on duties and responsibilities. Nothing else; it should not be impacting day to day life or culture. IE: Making an engineer a release manager does not really pull him away from core group and goals.

    Your post states:

    > We had edicts to make things happen, under the assumption that if we built it, the developers would embrace it.

    Followed by:

    > What matters is good people working together without artificial boundaries.

    The former will never produce latter; work with engineers and keep them involved. Don’t assume their needs for them ;)

    • jjmojojjmojo says:

      “I think you are responsible for correcting management that does know any better”

      I’ve never laughed so hard in my life. Thank you for that. :D

      “The former will never produce latter; work with engineers and keep them involved. Don’t assume their needs for them ;)”

      The point is that it’s better if they decide what their needs are, not management, and not a silo of people who aren’t involved in the day-to-day engineering work (e.g. planning and writing code).

      Having an engineer with an ops background on the team is great (if not ideal), but even integrated into a team it’s too easy to become artificially isolated if you treat something as a special task that only one person is capable of performing. You don’t want the person with the ops experience to be the person that “does” DevOps on your team.

      You can sometimes get away with that in software engineering (maybe you have your front-end devs, your DBAs, your Big Data experts, etc) but with the things DevOps does, like building environments, source control, deployment, continuous integration/delivery/deployment and such, it’s not advantageous to getting things done efficiently.

      IMHO those concerns are developer concerns, and everyone on a development team needs to be able to contribute to them. For example, I should be able to make a Jenkins job, not have to submit a ticket, or pull someone off of some other work to do it for me (yes, I’ve experienced that first hand).

  10. For most organizations, if they can enforce standards and configuration by convention devops done well should be a short lived endeavor, that should run on autopilot after that. Once version control, CI, code review and deployment scripts are set up, there should be left nothing left to do, except for occasional patching and upgrades, if future projects use convention so that everything does not have to be reconfigured. Not everyone is an Amazon or a Netflix.

  11. I stopped reading at “DevOps” team. What these guys did in that organization seems to have little to do with DevOps. You can’t have a DevOps team as much as you cant have an Agile team. It’s a mindset and not a job-title. And of course DevOps should never come from only one direction (e.g. only from the top down or from the bottom up) but needs awareness and support from all parts in the organization. A good indicator if DevOps would be a success is the Netflix ChaosMonkey. If you could get everyone including CEO on board that the ChaosMonkey will improve quality then your odds are good. Its the same with agile all over. You cant have middle/top management continuing as before but the whole mindset needs to shift in order to support it. And the bigger the organization the harder it will be to implement this (no surprises there).

    • dotnetchris says:

      Ah the tried and true “No true Scotsman” fallacy when it comes to “agile”.

      Chaos monkey has nothing to do with agile. If you expect the chaos monkey to live in your shop you’re going to need alot more than just DevOps.

    • jjmojojjmojo says:

      You should finish reading :)

      IMHO any cultural change (be it “Agile” or “DevOps” or whatever) should come from all directions, with leeway from management to allow for adaptation and evolution of the culture to suit the overall goals of the organization and the people doing the product work.

      With “Agile”, I’ve seen the opposite problem from what you mention – typically it’s management that have “drank the kool-aid” and try to dictate how every team implements Agile practices. This is counter to notions like “Individuals and interactions over processes and tools”. A world of pain.

  12. Ivan says:

    I totally know this recruiters thing to view people from an angle. If you have a ‘DevOps’ on the resume and they’re looking for a ‘DevOps’, they’ll be focusing on that aspect. Cool hack is to have something like “I’m not a DevOps person and I’ll gladly prove why” – this bypasses simple filters :)

    I also feel pain because of last minute hurdles and I thought probably there are a few factors contributing to this. Thus if we can fix the problem by attacking them:
    * People can’t find docs describing which to think about while designing (maybe nobody even though of how to document that, also it seems that having someone consulting teams starting on new products / services works even better)
    * Some aspects of development are thought of as a responsibility of another team
    * There are no tools / processes know to people to self-serve their needs
    * Lack of back-pressure :)

  13. Stu says:

    “You can’t have DevOps and still have separate operations and development teams. Period.”

    Not so. The more prominent examples of Devops cultures (Etsy, Google) actually do have separate teams and roles. They don’t have rigid structures and have strong collaboration between teams. Forcing roles onto the same team is one approach to foster collaboration (e.g. Amazon, though even they have a separate ops team for the lower layers of the architecture)…. but it’s not the only approach, and not one that works well without extremely strong senior management support.

    “Another lesson learned: you can’t force cultural change. It has to start from the bottom up, and it needs breathing room to grow.”

    Bottom up culture change doesn’t work either unless you have top management on board.

    Having witnessed some several company wide cultural changes in transportation, telecommunications, and even technology companies, I have seen cultures changed through relentless senior management that will fire (or make life hell) for anyone that doesn’t get with the program, in tandem with strong on-the-ground influencers running the day-to-day. This isn’t dissimilar from bone marrow transfusion in a leukemia patient, you’re basically killing off all of the old stem cells in the organization for new ones. Not great for morale, but, yes, it changes the culture.

    “Things like continuous integration and zero-downtime deployments just happen as a function of smart, passionate people working toward a shared goal.”

    This is the fundamental fallacy of your post. You’re basically saying that nothing new about process or method can be learned unless people already know it.

    Lots of smart and passionate people have failed in their mission. Nothing “just happens” in an organization except non-performance, unless the ingredients for emergence are there; passion and intelligence are necessary but not sufficient. You need knowledge about culture, automation, feedback loops, metrics, collaboration. More specifically, you need management to know these things.

    Devops is a professional movement to codify that knowledge. It gets misinterpreted. Doesn’t mean it’s bullshit.

    • jjmojojjmojo says:

      “The more prominent examples of Devops cultures (Etsy, Google) actually do have separate teams and roles.”

      First, roles and teams are different things. I was talking about teams, not roles, and specifically a single split between “operations” and “development”. There are plenty of practical reasons to have both. There are plenty of practical reasons to not pretend to “do DevOps” as well.

      Google and (especially) Etsy have teams that are highly collaborative, with dedicated contributor roles that float and disperse amongst the other teams. Etsy and Google also are very different companies and do things very differently. I’m not even sure if Google would really call what they do “DevOps”.

      The common denominator there is that both companies apply the practices and technology that comprise what DevOps is to your development process, and stop putting up walls between different specialties, which was the first thing I said just a few short lines before the line you quoted.

      “Bottom up culture change doesn’t work either unless you have top management on board. “

      Um, see that part where I said “it needs breathing room to grow”? that’s where management comes in. What did you think that meant?

      “This is the fundamental fallacy of your post. You’re basically saying that nothing new about process or method can be learned unless people already know it.”

      That’s a complete misrepresentation of what I said.

      What I’m basically saying is what I actually said, but let me restate: talented people, who give a shit about what they’re doing (aka are passionate), aligned to a common goal (like say, making the company money), will do things to make themselves better able to achieve that goal (e. g. continuous integration, zero downtime deployments, etc).

      This is why methods and process exist – they were made by talented people who were passionate about solving a problem.

      The things great people will do to achieve goals includes acquiring “… knowledge about culture, automation, feedback loops, metrics, collaboration.”. Effective, engaged people do this without prodding from management. As such, it just happens. hence my phrasing.

  14. I think the mistake is mentioned and begins very very early. If we look back at the old UNIX days, there was no such thing as a dev vs a sysadmin. Too many sysadmins don’t know development and therefore are bad sysadmins. Too many developers don’t know systems administration and therefore are bad devs. Bad as in performing and contributing business value way below their full potential.

  15. sean says:

    Great read. This is valuable insight – you should consider turning this into a talk at a conference. It’s basically textbook “What DevOps is not”. There is no such thing as a DevOps engineer … it’s a set of principles. This essay reminded me of an old zen saying “If you see the buddha on the road, kill him.”

  16. sean8sean says:

    You should consider turning this into a talk for a conference, under the banner of “How not to do DevOps, or If you see the buddha on the road, kill him”

  17. Wilfred Godfrey says:

    I agree with what you’re saying. That works for someone like me who happened to work on a couple of Rails-backed Angular apps. I’d be impressed if the author could pull off leaving out or editing out his years of DevOps experience though. At some point you almost have to try and re-engineer the truth rather than remove it completely. Messy.

  18. Wilfred Godfrey says:

    Wow wordpress comments are bad. How does anyone have a good conversation?

  19. Greg says:

    WAAAAAAH, I’m a beautiful and unique snowflake whose skill-set noone really wants.
    WAAAAAAH, I’m too young to have heard of configuration management.
    WAAAAAAH, I ramble on and on at length about how I’m a frustrated Help Desk worker.
    WAAAAAAH, I’m so self-important, and I can’t understand that poor management is why devs and ops don’t work together well.

    You’re a woman.

  20. Barry Alistair says:

    It strikes me that when one considers the changing landscape of software application delivery, the ops role has changed so much more than the developers – perhaps the culture shift would be better labelled OpsDev.

  21. Pingback: Daily Hacker News for 2015-03-09 | Xucifer

  22. Adam Yuret says:

    Devops doesn’t exist. It’s a social construct.

    • Fooman Baz says:

      Wow. That’s so insightful. Now I realize that marriage and jobs don’t exist either, they’re just social constructs!

      In case you’re a bit slow (and apparently you are), “just” being a social construct is pretty small grounds for saying something doesn’t exist, and is largely irrelevant. HR departments are scanning resumes for DevOps. People are hiring for DevOps. Just like this purely arbitrary social construction of marriage (who would look for a marriage partner if we didn’t have the social construct of marriage?), being a social construct doesn’t mean we can dismiss it.

  23. Pingback: This is How We DevOps |

  24. Pingback: Required Reading – 2015-03-21 | We Build Devops

  25. Pingback: This is How We DevOps | Smart Everythings

  26. Chris Calloway says:

    When I saw this headline in Plone News, I knew who wrote it before I read it.

  27. Joseph says:

    This appears like analytic assessment of a synthetic issue. Systems theory and its application to IT or any other complex of elements in interaction (Bertalanffy, 1934) is science not Bullshit. Coining a phrase like DevOps (Debois, 2008) for what its worth may help people operating in silo’s realize the barriers existing between them. The barriers are the bullshit and are ultimately managements responsibility to identify and improve upon (Deming, 1993) Genuine interest in creating a generative culture (Westrum, 2004) of continuous improvement (Shewhart, 1924) in an end to end in a chain of value (Akoff,1972) is not. Please take some time to explore the foundational science of systems before you accept jjmojo’s assertion:

    Ludwig Eduard Boltzmann (1870) Statistical Mechanics
    Walter A Shewhart, (1924), Statistical Process Control
    Karl Ludwig von Bertalanffy, (1934), General Systems Theory
    Joseph Schumpeter, (1939), Business Cycles
    W Edwards Deming, (1950), Mount Hakone Speech
    Russ Akoff (1972), Purposeful Systems
    Taiichi Ohno (1948 – 1975), Use Case – Toyota Production Systems
    W Edwards Deming, (1982), Out of Crisis
    Eliyahu M. Goldratt, (1984), The Goal, A Process of Continuous Improvement
    W Edwards Deming, (1993), New Economics
    R. Westrum, (2004), A typology of organizational culture

    Respect the shoulders of the giants you are standing on rather then calling bullshit on something with which you may have only a limited functional understanding.


    • jjmojojjmojo says:

      So what you’ve done here is mentioned a large stack of books, and warned people to be cautious about my assertions. Based on nothing but the title of my article. Good job.

      For pretending to be so well read, it’s sad to think you can’t be bothered to actually read a measly article before you warn people of its content – what’s here is a deeply personal story about one person’s journey through the current broken state of the tech industry, watching companies fail in vein attempts to embrace things they either don’t really understand, or aren’t mature enough to integrate with. I talk about barriers and my experiences with how they’ve impeded progress. Spoiler alert: in the end, the protagonist ends up embracing DevOps, and sees the barriers and poor management as the problem. You’d know that if you had any integrity. How dare you skim my work so thoughtlessly and accuse me of not understanding what I’m talking about. My personal insight is not a “synthetic problem”.

      Being familiar with Bertalanffy and Deming myself, and knowing that most of the other works you cite are heavily plagiarized (er.. sourced) in most books and articles about “devops”, your comment is especially disappointing – we could have a real discussion here but instead you went out of your way to show us how smart you are, while providing no actual content. You are part of the bad culture that makes DevOps bullshit. Congratulations.

      I almost declined to approve your comment, but you’ve provided a decent reading list for anyone interested in process, management, or systems theory, so I’ll let it stand.

  28. Joseph says:

    For those of you who don’t have time to locate the works above I have attempted to summarize (some are speeches and videos).

    Ludwig Eduard Boltzmann (1870) Statistical Mechanics
    Botlzmann is recognized as the father of statistical mechanics. His seminal work on statistics took place at a time when atomic units were considered non-existent. Boltzmann was attempting to quantify closed systems at the atomic level. The entropy calculation S = kB log W was memorialized on Boltzmann head stone and has provided the foundation for establishing the link between entropy and probability. S = Entropy or Randomness, kB = Boltzmann’s constant, W = Probability. When applying this to systems theory we find that closed system entropy can be characterized, but cannot be reduced. This is expressed in many closed systems and is the foundation of many engineering rules including the second law of thermodynamics. In open systems with inputs and outputs we can leverage the concepts provided by Boltzmann to remove variables or probability thus eliminating entropy as we eliminate variability in the system.

    Walter A Shewhart, (1924), Statistical Process Control (SPC)
    In Shewhart’s seminal work on SPC developed for Bell labs in the production of the US telephone system. Shewhart postulated that an open system can be continuously improved by the elimination of variability. The process falls directly into the physics provided by Boltzmann and was leveraged in manufacturing for the first time with Shewhart. Shewhart leveraged Boltzmann’s constant as well as the Boltzmann’s laws of entropy to develop Plan Do Study Act (PDSA). These concepts have provided the foundations of many quality initiatives leveraged to to date. This reference goes into greater detail.

    Karl Ludwig von Bertalanffy, (1934), General Systems Theory
    Bertalanffy is recognized as the father of General Systems Theory. Bertalanffy’s seminal work in systems theory expands the concepts provided by Boltzmann who was predominantly focused on closed systems. Bertalanffy applied his open systems concepts in biological systems. The main premise behind his seminal work was the initial definition of the system. Bertalanffy contended that a system as a complex of elements in interaction. Bertalanffy went on to postulate that the sum or system is greater then all of the constituent parts. His further contention was that the properties and modes of action of the higher levels are not explainable by the summation of the properties and modes of action of their components as studied only in isolation. However, if the practitioner can characterize all the components brought together and all the relationships existing between them, then and on then can the higher level system be derived from its components.

    Joseph Schumpeter, (1939), Business Cycles
    Shumpeter is recognized as the father of the term “Creative Destruction”. In Shumpeter’s seminal work on economics of a capitalist society he developed many of the foundation driving theories around expanded markets and cycles associated with the advancement of economic times. In 1939 Schumpeter provided that innovation and invention differed in that invention focused on advancement of a particular technology or discovery. This invention didn’t focus on the external factors of providing goods or services. Schumpeter went on to postulate that innovation surrounds inventions, yet specifically in the context of doing things better. Schumpeter’s views described the entrepreneur as the agent of change advancing innovation. This innovation was segmented into five types in the Schumpeter model. These segments including creation of new products or modification of an existing product, the creation of a new production methods or sales campaigns, creation of a new market, development and acquisition of new source of raw or semi-finished goods, and finally the creation of new structures of industry, which create or destroy monopolies. Schumpeter’s seminal work has been the basis of work in both disruptive and value based innovation, which surround the systems theory of continuous improvement.

    W Edwards Deming, (1950), Mount Hakone Speech
    Deming is recognized as the father of Statistical Product Quality Administration. In his seminal speech in 1950 Deming lays out the foundation to what has been historically referenced as the miracle of Japan. Deming, being the thought leader behind building the industry to support World War II is sent to by the department of treasury Japan to help the Japanese leadership rebuild. Deming trained a prestigious group of managers including Taiichi Ohno in statistical process control. Deming had also previously worked with Dr. Walter Shewhart the seminal author of Statistical Process Control. After teaching this course Deming was asked to speak in front of 80% of the wealth of Japan. In this historical setting Deming lays out that the problems facing manufacturers can be solved through cooperation, despite differences (Later defined as the Aim of the system). That marketing is not “sales,” but the science of knowing what people who buy your product repeatedly think of that product and whether they will buy it again, and why (the foundations of analytics). That in the initial stages of design, you must conduct market research, applying statistical techniques for experimental and planning and inspection of samples (Basis of Statistical Process Control). He also states that you must perfect the manufacturing process (Continuous Improvement). In the speech he mentions the Shewhart Cycle, known as the Deming cycle in years to follow in Toyota Production Systems (TPS).

    Russ Akoff (1972), Purposeful Systems
    Akoff is recognized as one of the founding father of the purposeful systems. Akoff postulated that synthesis rather then analysis is the preferred mechanism for defining purpose. Akoff went on to describe system analysis into 3 phases, which starts by separating an object into its parts. In the second step of analysis we attempt to identify the function of the parts taken separately. In the final step of analysis we attempt to aggregate the parts in an attempt to understand the system as a whole. In the first step of synthesis you ask what is this a part of and identify the containing whole, which the object is a part. In the second step of synthesis we attempt to identify the behavior of the containing whole. In the third step of synthetic thinking we disaggregate the thinking of the containing whole by identifying the role or function of the component in the whole. The system as a whole as defined by its function in the larger system of which it’s a part. Every system is contained in a larger system, and its role or function in that larger system is what defines it. In order for a system to function it requires parts, which are required for its function, yet they are not sufficient. Therefore, an essential property of a purposeful system is that it cannot be divided into independent parts. The purpose system is derived from the interaction of its parts and not the actions of its parts taken separately. When we attempt to conduct analysis and separate a system into its parts we loose the systems essential properties and so do the systems parts. A system is never the sum of its parts. A system is rather the product of the interaction of the component parts. Adapted from this speech:

    R. Westrum, (2004), A typology of organizational culture
    Ron Westrum’s seminal work on organizational culture provides an in-depth analysis of the relationships of communication throughout an organization. Westrum provides that often time’s organizations communicate in crisis in a very similar fashion to how they act during normal day-to-day activity. Westrum provides chilling evidence of this in several case studies with its implication on safety. His Ground Zero investigation between the NYPD and the NYFD illustrated the results of misaligned organizations and how communication focused on the goal should be optimized despite organizational separation. He proposed that there were three main typologies for a system. His contention is that organizations fall into pathological, bureaucratic, or generative cultures.

  29. Joseph says:

    Here are some more pragmatic observations from the summaries above:

    1. IT and for that matter the entire organization is basically an open system (inputs and outputs – the end result of all of the inputs and outputs is the aim or goal of the system)

    2. The more variables you add to an open system the greater the amount of entropy (Randomness) exists in the system – the result is more defects, longer time to recover, more outages, lower quality, slower time to market, etc, etc

    3. When a company creates silos they are being analytic rather than synthetic, which is counter to systems theory –

    4. Deming (Probably one of the greatest hero’s of our time) stated that transformation is EVERYONE’S responsibility and that EVERYONE must know the system, the aim of the system and their role in accomplishing the aim. Think of a how a submarine operates where everyone knows everyone else’s role (Crew members are actually taught and required to be signed off on all tasks by the task specialists) in case an issue arises and the specialist isn’t available.

    5. Organizational typology is based on the way the organization transmits and uses information (not how the organization is structured): Pathological = Fiefdoms || Bureaucratic = Silos || Generative = Shared risk and responsibility focussed on the aim of the system

    Ok now add all that up:

    If your company creates a silo to address change (DevOps or whatever) your organization has basically added more variables to the system, which adds more entropy thus making the problems worse.

    • jjmojojjmojo says:

      So you’ve summarized what I said. Thanks. Have you actually read this article yet?

      Your evangelical approach masked in “science” is symptomatic of what’s wrong with DevOps as a culture.

      Digging into the background information you mention is a chief reason why I came to my conclusions, both about failed implementation of DevOps principals, and the broken nature of the whole DevOps scene: there’s too much cherry picking of ideas here. You can’t bring up Deming and ignore the conundrum that his work incites: it’s very difficult, if not impossible, to get a software system under statistical control. Where do you measure? What do you measure? Software engineering teams are more like artisan guilds, not assembly-line factories: they produce a series of one-of products. Every application, every feature, every bug, every release. Every one is different. Trying to shoehorn them into a consistent model doesn’t work. It’s slightly easier with infrastructure, since we can encapsulate infrastructure into repeatable units, but there are limits, and the day-to-day work is typically just as one-of as software engineering.

      It’s easy to print out the “Seven Deadly Diseases” and staple them on your boss’ door like a “DevOps Martin Luther”, but there’s real work involved in applying the principals behind Deming’s observations in a completely different context – having tried to do this myself, I know it requires a lot of pretense and ignorance, and will fail. Ultimately Deming wasn’t a philosopher, he was a statistician. Ignoring that aspect of his work, what made it all real, is egregious.

      This is just one example, using one source I’m particularly familiar with. With some time I can find many more, because a pedantic approach to DevOps is nothing but self delusion. In trying to convince people that DevOps is not bullshit, you are just re-enforcing my assertion.

      • Joseph says:

        Lets make a wager first… 1$ or a cup of coffee – your choice

        Which shall it be?

      • Joseph says:

        I’d like to address a few of your points and at the same time add value to your readers. Based on your silence from my gentleman’s wager request and your post above I have a feeling that your pride won’t allow you to listen to reason, science, or proof. Please prove me wrong and respond with an open mind and drop the emotion.

        Agile and Microservices are both attempts at what you have stated is nearly impossible in your artisan example (Applying science (statistically significant findings) to software development). Both Agile and Microservices leverage the science in my previous references, which you state is “cherry picked”. Its baffling that you call it “cherry picking” even though the references are in chronological order and build upon each other. In fact Dr. Shewhart, Dr. Deming, and Dr. Ackoff all worked with each other through the genesis of their lives. I’m also forced to point out you yourself have “cherry picked” Deming and admit you can’t respond to the rest of the references until you have “some time”. I must give you some credit for seeing that infrastructure and platform have a direct relationship to systems science references I have provided.

        Unfortunately, your “With some time” statement also proves you aren’t as prepared to be taken authoritatively on the topic as you let on.

        How could you be?

        An understanding of the foundational science is required for successful implementation. If you knew what you were doing you would have been standing on your desk at your first job. FYI A practitioner becomes proficient in Theory first and THEN Practical Application.

        Its like doing heart surgery without learning anatomy and yes both are pedantic in application. In such that both have specific steps required for success.

        Doing it in the opposite order (Practicing then learning the Theory), which you described in your novel is counter intuitive and could be considered somewhat disingenuous. Yes I understand that you were forced into your first position and that you attempted to read some theory afterwards, but because you don’t understand something (which you have exposed that you still don’t completely grasp the foundations from your comments above) does’t equate to it all being bullshit.


      • jjmojojjmojo says:

        You accused me of not understanding source material. I chose, as an example, one particular work that I’m very familiar with (as I stated), to point out that the overarching DevOps community, represented by you, are a cargo cult. I’ve never claimed that the whole of DevOps is bullshit. This is clear if you actually read what I wrote. By ignoring my points, and twisting my words, you’ve done nothing but continue to prove every point I’ve raised.

        Please stop.

        You’re just being nasty at this point, and I won’t tolerate it. You’re as bad at the chap who equated me to a baying woman. But what you’re saying is far worse, because you’re attempting to wrap insults about my intelligence and qualifications in pseudoscientific nonsense. At least what the other guy said was funny.

        Playing the “who’s a hypocrite now” game is adolescent and boring. I suppose my statement about cherry picking hit a little too close to home. In any case, I prefer to talk about things I understand, so I chose a work I was intimately familiar with. I used it to explain how you and other “devops” evangelists are broadly applying a work to a topic that doesn’t warrant it – it was one example, I chose because I don’t like to talk out of my ass, and pretend I’ve read things I haven’t. I can’t say the same for people like you.

        This is a fundamental problem with “devops” that I didn’t bother raising before.

        It’s like anyone who recently “discovered” a new ideology. They proclaim, “LOOKIT!, I found ancient hidden knowledge! It’s a revolution! Come worship at the feet of my saviors! How could you be so stupid as to not receive this communion of life changing wisdom! REPENT! REPENT! VIE ON THEE, EVIL HEATHEN TRANSGRESSORS!”

        They squash down anyone with a dissenting opinion, and destroy open dialog by cramming out of context quotes from their holy texts down the throats of anyone who has lived with the ideology and found it wanting. The true believers don’t really understand what they’re saying, they just parrot what others have told them, and try to prop that up with citations. This is endemic in the so-called “devops” community, and leads to the sort of failed “lets do devops” misfires that I discuss in my article. Outside of the evangelical fringe, nobody bothers to read the source material, because of the reality that actually reading it makes DevOps look less credible. It’s easier to just buy what Gene Kim is saying at face value and not look any deeper.

        But I was hesitant to address this in any depth in my article, because I do have some respect for the academic side of this community. Or at least, I did, before you wandered over to save my 40 readers a day from the content of my article – which is, as is obvious from the title, NOT A SCIENTIFIC WORK.

        I think anyone listening to you rambling would easily understand your point: there’s a lot of fundamental principal behind what people shallowly implement as “devops”. We get it. But I have shed light on the fact that the foundation is often shaky at best. Instead of addressing that accusation, you attack my credibility. And then you deflect with more nonsense. I seriously can’t believe you said the words “microservice” and “agile” in this context, implying there’s “science” behind either, that they have anything to do with DevOps, and the source of that science is in the works you cite. I’m sure you didn’t mean to imply they have their roots in Out Of The Crisis, but the way you structured your diatribe makes it sound like you did. You are obviously out of your depth here.

        Again, where I wasn’t calling “bullshit” on DevOps as a collection of principals and concepts, here you are turning them all into bullshit by appropriating concepts out of context and turning them into religion, then trying to use them to discredit the non-believer. That’s hardly scientific.

  30. Here’s what I wager is a common situation in agile organizations, which I feel is a huge barrier to the methodology being actually implemented in a serious manner, and it has to do with devops.

    In my organization, were we to try to use devops strictly as a method, not a role, it would never work, because developer time would be immediately consumed by the monthly sprint, leaving sysadmins to generally implement their own solutions. What has actually happened in most environments that I see is that devops is a group or role itself consisting of primarily or only developers with just enough systems knowledge to do the job, with a lot of direction, yet there is no direction. The result is that a lot of cloud environments appear to have been created by junior admins. The feeling I’m getting from other sysadmins is that this practice is becoming a pandemic.

    I read a lot of comments about devops in general and it seems to me most people have glaring misconceptions about what a sysadmin does. Setting up a system is not what we do, it’s the result of what we do. An environment isn’t just some random collection of devices, virtual or otherwise. It’s a set of processes, rules and standards into which one integrates said things. If you think you can have the former without the latter first, you are going to have a mess, very quickly. I’m surmising that the notion that technologies such as docker are making sysadmins obsolete stems from this; or the notion that just because an environment consists of ephemeral devices means one needn’t manage them. What it really means is that you aren’t doing the latter. It is irrelevant that devices are ephemeral. We’ve always been dealing with that, only the timeline has been sped up. Hardware is ephemeral as well. We change it out all the time. If you don’t have someone with experience to ensure your practices are secure and don’t hinder upgrade pathways or organizational growth, you are still making a mess. How many devops do you think even have a cursory familiarity with PCI-DSS?

    The cloud doesn’t change what sysadmins do at all. We can’t effectively be replaced by any code, because code doesn’t think

    • Joseph says:

      Great post!

      IMHO you are absolutely correct…

      Functional requirements often pass directly from the business to the development teams and the non-functional requirements (Infrastructure, Platform, Compliance, Security, Data Protection & DR, Workload Management, Maintenance, Patching, Facilities, Separation Restrictions, Air Gap Restrictions for Classification, Encryption, etc, etc, etc) to support the functions of the applications require a great deal more complexity then many of the functional requirements teams (devs) I have worked with and been a part of want to have visibility into.

      If you look at government, defense, pharma, banking, retail (or any highly regulated enterprise) these operational non-functional requirements are show stoppers.

      Governance in these environments whether they are in the cloud or on premises ALWAYS apply! As you state clearly PCI-DSS, 8500-8510, FDA, etc, etc, etc aren’t eliminated because we are using AWS or Docker.

      Code in infrastructure and platform will assist in mandating certain aspects and provide visible audit/compliance, but the architecture cycles / care and feeding of the systems to address the ever changing mandates and keep the functional aspects functional can’t be dismissed.


  31. jdumars says:

    Thank you for an excellent article. My experience is that most people who speak positively about DevOps have never lived with it/managed it in practice and have no idea how destructive it can be to people, culture and companies, especially at scale. I’m putting on my flameproof suit, because I am going to take on DevOps as a concept in our industry. It’s time we change the conversation because it should never have been about DevOps in the first place, and all of the sophistry about DevOps being “collaborative culture” is not fooling anyone.

  32. Pingback: Operation DevOps | Skankworks Agile

  33. Pingback: To be or not to be DevOps | Tibileo's Blog

  34. Pingback: DevOps Is Bullshit: Why One Programmer Doesn’t Do It Anymore | jjmojojjmojo: In Effect « The Wiert Corner – irregular stream of stuff

  35. jeff says:

    I believe that for both DevOps and in the same vein Full Stack Devs – very very few developers can handle that kind of responsibility and scope. If they do, they basically a crappier/shallower version of the specialists that focuses on their speciality. Ops is a lot more than “deployment”. It means understanding network at such a level that developers don’t understand at all. And if you’re talking about AWS Ops, you’re talking about a speciality within the “Ops” field that frankly, doesn’t care and know a lot about what the traditional ops people need to know. I will also assert that the same talented person that *can* handle DevOps duties will be a *much better* specialist of either. That’s the nature of needing to know too many thing to such minute detail.

    DevOps is business people doing this:

    Except instead of hiring 2 people, they are hiring one.

  36. Pingback: Five Blogs – 23 August 2016 – 5blogs

  37. Pingback: Bridging the DevOps Divide – Simple Talk – Cloud Data Architect

  38. Holly says:

    Nice article…this recruiter does like to understand the technical as much as she can. I will tell you that some recruiters do push DevOps programmers as programmers because they are programmers. It’s often the hiring managers that are pushing back on this and not considering their experience as applicable. How can I help get them over that hurdle?

  39. faraco says:

    Reblogged this on faraco .

  40. Pingback: The Developer Tool Revolution — Sway Ventures

  41. I know this is a old post, but I totally agree with you. The big reality is that DevOps steams off of AWS and there Cloud Platforms. Oh sure, there are companies that LOVE the cost saving idea that AWS tries to lure them in.. but as you stated, it all falls a part. I’m older than you.. I go back into the early 1980’s and I’m still in Information Technology and Services. Back then everything was even “more” separated one group for desktop, one group for servers, for programming, for networking Ect.. with “no certifications”. Then after big M$, you had certifications.. then you had “oh well you can MULTI-TASK doing two or more job roles / positions! ” “Multi-Disciplined in many I.T. Fields” The whole AWS DevOps is just for companies trying to thin there workforce to save money.

    • jjmojojjmojo says:

      I’m with ya, but you can’t blame AWS for this… wait… maybe you can… wow… My understanding is that Amazon just facilitated the cultural shifts (aka ‘trying to thin the workforce’), making this possible – do you have a source for what your saying here? I haven’t thought to dig into the history of DevOps in relationship to AWS. Did Amazon shape that approach beyond providing a platform to implement it? Could be fascinating stuff to investigate.

  42. passerby says:

    first of all, comment box should also be put right after article for ease of access.
    back to the article, it is basically a bunch of superhuman who can do from coding until managing operation by themselves. I think it is preferable to have such a team for several critical projects.

  43. Matt Booth says:

    I’m currently working in DevOps for a company that created the role not so long ago. I turned down a job within a company with an existing DevOps pattern (they had a contractor and the CTO had setup the systems, considered himself DevOps).

    What a mistake.

    No-one here knows what DevOps is and everyone has different ideas about what the role should do. My line manager is holding me to too high of a standard considering the monolithic and archaic system they have in place (and has expected me to understand the system in and out within 3 months, not taking into account the Christmas period of absence).

    My role seems to be “do the deployment when that needs doing, and then fill in odd-jobs in the meantime”.

    Whereas I’ve already identified loads of areas for improvement within the Git system they’re using, the lack of automation and Continuous Integration and the deployment process (I.E., deploying requires the live system to go down and I find this mindbloggling).

    I had to wrestle the access details from one product owner (no idea why he was doing this) for domain names and SSL certificates. Managing the Windows Server Domain is done by someone else. The lead developer takes all the server specification and management decisions. The scrum master is a person (who’s doing great) who’d been out of employment for 15 years, previously a programmer and spent one afternoon watching one other company do Scrum. I’ve been requesting to take me out of sprints entirely as most of what happens within it has nothing to do with me and I then have some pressure to arse about doing code on the main system (whereas I’d rather be spending time programming a bespoke data reporting solution to our main system that highlights operational data and issues).

    I could go on, but I won’t. Lesson in life learned. I should’ve gone for the employment where everything was already in place and they were willing to take the time to bring me up to speed before letting me loose with the role and responsibilities.

    I’m paid well, but I’m not happy.

  44. Pingback: What Comes after DevOps? Two Possibilities -

  45. Sang R Kang says:

    I sort of gave up all aspect of IT management. This was when, I started to realize, the corporation, the management, they are all prioritize to please the client/profit and not the quality of the infrastructure or how well oiled it is.

  46. Sergey says:

    “The developers are the operations people and the operations people are the developers”

    That thing work for people who are ready to study whatever and don’t have a solid vision of their career plan (that incorporates educational process also).
    I am not sure whether an awesome developer or an analyst with a great skill-set would like to learn new skills about CI/CD or maintaining the system or somewhat else similar.
    Especially if that person has been working on medium/big size projects for their big part of career.
    That stuff is so freaking boring. As for me I don’t even take it into account as an opportunity for learning. I had an experience of having a part-time work as a 3rd line support – thank you, but no. Idk what skills you might get from that kind of job – you don’t even see a whole picture.
    Knowing how an IT system maintaining a business process from a technical side – OKAY, but it does not call any shots.
    Lastly, I think, the knowledge of a system is not a competitive advantage.
    Why? I might explain in extra. Speaking so shortly – you should rather need to know the concepts that this system is incorporated.
    I don’t have anything bad to do with the OPS /the maintainers/IT support.
    That part of my opinion might hurt anybody, but remember – it’s my opinion.

    The second thing –
    yes, as it was mentioned above, people from OPS or from the field that is responsible for an IT support, usually have a very very different mindset (and skillset) and that process of sharing knowledge becomes so difficult for the both sides. Why?
    1) after the N-iteration of the same thing explanation you would come with a thought that your students have a lack of even more ground knowledge like a theory of databases etc.
    Idk maybe someone is ok with it and ready to teach people in infinite way mode, but that case lowers a motivation of a teacher (that’s my own experience).
    2) a derived case from the 1st – you start losing man-days that you could use in more proper useful way. Let’s say, to fix some stuff from your technical debt.
    3) if your team is unbalanced skills-wise then for the most skilled persons there won’t be any challenge and motivation to learn stuff from the others , especially of they are “snobs” (see the 1st paragraph above).
    and so on.

    it’s so sad that all that new stuff like DevOps or DevOps X.0 or even Agile is always given like a panacea or a silver bullet against all the pain that poor organizations are suffering from without testing in on different projects in different industries.
    Other smart guys who wanna surf on the waves of hype start writing their own books based on those great ideas.
    But no, that’s not the end.
    Then all that stuff is start feeding some consultants who has become the great transformers of business in terms of changing the culture with IT etc.
    And lastly all that crap being applied in an organization starts to live in conditions of continuous political wars (often).
    As a result – you having an agile mindset (or say DevOps it you want) and knowing how the thing are working in certain project conditions , have to follow the prescribed path instead of working more efficiently.

    Such a life.

  47. john zajac says:

    – Using “DevOps” to save money by reducing staff will blow up in your face.
    – You can’t have DevOps and still have separate operations and development teams. Period.
    ^^^that alone – 100 points for you. Especially the second one – separation – mercy on those souls who refuse to accept that.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s