Architecture and Design/Engineering

Today I would like to discuss with you the differences between architecture and design. Although it might sound like a highly academic discussion I would like to clarify their differences. They can lead to confusing expectations of both architects and people who do engineer/design the solutions.

We can see a strong analogy with the roles used in construction building. There the architect shapes the way the construction looks, and more generally covering the functional needs of the people (stakeholders) who hired the architect to architect the building. The construction workers and building companies involved will take the architecture plan and will decide for themselves, within the constraints given by the architecture, on how the construction is realised. This detailed way on how to construct is not an architect activity but a engineering/design activity.

In the world of Enterprise Architecture the separating roles of architecture and design/engineering can be made in a similar way. That doesn’t mean that they to consist of separate organisations, teams or even people. That is how you implement (assign) the roles, but the role differences are essential. 

Architecture is an activity on a more abstract level, design is much more focussed on detailed level. Architecture may select a class of solutions while a design defines one solution. Architecture is about fitness for purpose expressed by the needs of the stakeholders, while design is more about the engineering optimising while taking in account the architectural constraints.

Hope it gives some direction.

 

Joost Bleijenberg

Why do we need SysAdmins?

Currently I am working on a new deployment concept for a customer. We are developing a complete new way to deploy a web landscape. The major goal is to have a faster deployment model and to move into the land of continuous deployment.

Today the environment is setup in a pretty conservative deployment model. Linux boxes are build from scratch and although some automation is in place, it is pretty labour intensive.
After the build process of the system OS different additional packages are installed for backup, monitoring, scheduling, etc. Then the applications are deployed (web apps) and configuration of the app takes places. After that is done the Linux Ops department (SysAdmins) take over and full control is with them. Everything from backup/restore, full filesystems, crashing daemons is in their control.

Now I am investigating a model to automate the whole lot, but that is not the big issue. Part of my idea is to move into a more DevOps approach and that brings up the discussion of management. From a DevOps perspective we would like to deploy in a quick way to for instance a User Testing Lanscape. To do so I would like to have a clean environment and deploy the application to it, configure them and run User tests. Preferably under full version control and completely automated. To get a clean environment the plan would be to completely rebuild the landscape automatically (using VM’s and automation like puppet, cfengine ) every time we do a new deployment of the app. The advantage would be predictability. The ultimate goal would be to do the same in the production landscape.

However there is a lot of resistance from the Sysadmins to this approach. Since they are not in control anymore of the formal acceptance process that they normally run for every deployed system. I think that is a normal response from a team in a very silo-ed organisation. But it is not this that i am focussed on, it is more the SysAdmin work done today. Why do we do the things we do?

Today I think we have an approach of fixing things in a adhoc way instead of fundamentally. And i don’t meant that we should implement more monitoring and be more proactive. I mean if the filesystem is running full you should not implement more monitoring you should fix the application that fills up the file system. If the daemon crashes don’t try to fix it, do a clean install and redeploy the app. And of of cause you should investigate the problem but when you fixed it you apply the fix to the image that was used for this system. The image is under version control and a redeploy can now take place on other systems which had the old version still installed. New systems will get the the new version directly.

My vision is that all web landscapes that we have move into a platform (paas model?), framework whatever you call it. In the past i was developing code for embedded systems and we take inspiration from that. Embedded systems have scarce resources which forces the developers to use resources properly. The more resources the are the less a developer focuses on it. Examples are firmware in an embedded system for instance your wifi router, or IOS in your i* devices. Did you ever had a filesystem full message, a SNMP trap or did you need to login as root to recover “something”.
We need to change our SysAdmin role to a more engineering rol where they should engineer this platform instead of maintaining it. I call them SysEngi’s. This move gives the SysEngi also a better context with developers. Developers in their place should develop for platforms, with clear constraints.

And you already see the move into this direction, just think about the the vmware appliances that enable you in no time with functionality. Restriction for this approach is that wou have data-less systems and most of our web landscapes should be. Well if you have implemented SOA at least.

Joost Bleijenberg

Self service storage example

This example is how an existing IT operations process, in this case storage provisioning, can be automated and self serviced. It is a very typical process which is often used when building unix or windows systems. And every time there is more storage needed, as a result of data growth, it is executed again and again. So improvements will save resources not only during the first deployment of a of system but also throughout it’s lifetime.

This currently existing process roughly looks like this :

20130127 172302

As you can see it is an manual process between the unix/windows admin, the requester, and the storage admin who is the receiver. The process contains a lot of sequential steps with lots of waiting points. For instance a mail request can wait for attention for days in the recipients mailbox. The process style is push and we know from the kaban theory that that is not the most efficient process style.

Now let’s change the process into a pull process style. So lets see what happens when we fully automated the process, supposing that there is an automated process inside your organization to deploy a windows, or unix servers. This could be done using product from Microsoft, hp, IBM or great open source solutions like chef and cfengine.
In such case you can hookup the storage provisioning part as a sub process. The result would look like this :

20130127 210204

An other way to optimize is to make the provisioning process self service, this of course when fully automating is not possible. This self service should enable the unix/windows admin to pull the storage towards herself without any involvement from the storage admin. The result could look le this :

20130127 210620

With a simple web request form, or script the unix admin can pull the storage towards herself.

All of these ideas are not very difficult to apply in practice and will increase your service time-to-market, and even better it will release your admin staff so they can keep themselves busy with more cool automation stuff.

Joost Bleijenberg

intro : continuous delivery, continuous integration, continuous ….

When your IT organization consists of both developing software and managing the infrastructure behind it you might enter into a very interesting domain of managing the overall success and delivery of code and business functionality.

Currently there is a lot of attention on the SCRUM method which fills in the need for a better fit between the business functionality needs and software development. The method is incremental based and has an very elegant way of making the fit between chaos and order. It’s purpose is to deliver, big or small required functionalities in small steps frequently.

Now that also means that new functionalities and new code is delivered more frequently from the development factory, however that doesn’t mean that the operations organisation can deploy and integrate it at the same speed.

What is often seen is that when moving it into acceptance or production environments it delivers all kinds of problems in the context of configuration, integration, scalability requirements etc.

That is not really strange because with the increased pressure from agile development the flexibility that is needed can not be delivered by operations. The nature of operations is stability not hosting new code every 2 weeks. The nature delivers procedures, strict processes and operator mentality. That means the models for these two domains conflict by nature, and a big red wall forms between them. Culturally it could be perceived as the cowboys from development, and the others, the police from operations.

Screenshot 1 26 13 10 06 AM 102

It could be the case that before you enter production you might want to do User Testing, Integration testing, Load Testing etc. For that you might need different environments and deployment and constraints become a hug issue. This because it is not only about the code anymore, it is about the setup of the whole infrastructure landscape. So it is about, software distribution, configuration of web servers, configuration of web services, setup of firewalls, setup of web service security etc. The use of SOA will even make things more complex, because the small change of a service might impact other non changed parts of your environment.

Well although solving all these issues will cover many areas of the DevOps arena the Continuous * concepts focuses on the way to increase the agility of code delivery between the development and operations landscape. So what can the Continuous * concepts do for you?

Continuous Delivery/Deployment is based on the Agile Manifesto and it is meant to delivery code fast from the developers desk into your production landscape. It is focussed about building a stronger collaboration between software development and IT operations (DevOps), it focusses on processes, and high degree of automation for deploying, integrating and testing your code into a production stage. CD is taking the Continuous Integration concept further.

Continuous Integration is meant to integrate the whole development process where every change of code that is done by a developer will be compliled, unit tested, and will do a complete build of the application and even do tests on the application level. This will create a better application quality instead of developers working in isolation on certain parts of the code while the over all application breaks or doesn’t deliver the proper functionalities. In non CD environments it could take until a very late stage to find out that the combination of pieces don’t work together. It might take considerable time and resources to debug to the core of the problem in such case.

Both CD and CI concepts consists of ecosystems (process, automation, policies, etc) to support them.

 

Joost Bleijenberg

Automate or self service process?

When it comes to making decisions on automating, or self service processes there are some considerations to make. If found the following rules working in practice (in order of priority):

The process selecting what to do with a IT process could be :
1. Analyse the process and get rid of it if you can, if that is not possible ….
2. Automate the process completely, that means it becomes totally automatic, no manual handling, or request. If that is not possible …
3. Self service the process, that means making it a pull type and give the self service capability to the requester. If this is not possible …
4. Optimize the process by semi automation, so automate parts of the process.

@1 When you manage to get rid of process completely that is the best solution. Some processes are there because of reasons from the past, or because of silo-ed organizations, or because of no reason at all. Consider when you look at a process if it creates value, or is it just a bunch of steps to do. Value needs to be considered as business value. If there is none you might wonder what it is about.

@2 Automation is beautiful as it will remove human intervention completely, automation means that the process will be executed possible as part of another automated process. Disadvantage is that the automation itself needs to be managed or engineered. But it can be perceived as moving to a next level of devops or Datacenter CMM.

@3 Self service, is a great way to empower the requester, in stead of an question-answer approach with wait time involved it give the requester the possibility to self service herself. It still could be that a manual step is needed for instance to initiate the request, like filling in a form, running a script etc. But after that all requested is coming automatically.

@4 Semi automation is the last possible step to take, sub steps of the process can be automated to reduce the manual interventions needed. It is not the most sexy alternative but sometimes it is the only one left.

When using these rules you have to thing out of the box, existing situations should be questioned and some analysis could help to get a clear picture on the value that can be gained from doing things differently and simpler.

Joost Bleijenberg

What data center orientation do you have?

Why do you do the things you do? It is an elementary question and relevant in the DC context.

Most data centers are part of a company and have either internal or external customers (in the case of outsourcing). For both you have to task yourself the question who are my customers and what are they expecting of our service offerings?
In oder to get some context on that you can simply use the SLA or SLO and think that that should answer the question. Well in some way it does. However when it comes to delivering real value to customers might get another context.

I would like to hand you the idea of a separation in types of customer needs and want you to think about how this could affect your way of looking at both product development within IT (for developing new services for instance).

In most cases we try to make all products the same but we might have lost the reason for the why.

From marketing strategy we can learn that there are 3 types of orientation when it comes to value proposition : customer intimacy, product leadership and operational excellence.

NewImage3

 

Operational excellence

  • Your customers just want a good product at the lowest possible price. New bells and whistles aren’t particularly important – it’s price that counts.
  • You’re not trying to come up with new or better products; you just want to produce more volume at a lower cost.
  • You focus on creating operational efficiencies to keep your costs low.

Product leadership

  • Your customers care most about the product itself – they want the best.
  • Your team and company culture is completely dedicated to innovation and quality.
  • You’re constantly working on improvements and innovations that you can bring to market.
  • You know what your competitors are doing and are completely focused on staying one step ahead in order to capture a greater share of your market.

Customer intimacy

  • Your market is flooded with products or services like yours at all ends of the price spectrum. But your customers want more than a standard product or program; they want customized solutions.
  • Your mission is to know as much as possible about your customers’ businesses and build successful relationships so you can deliver the correct products, services and/or solutions over time.
  • You know you can’t just say “We offer great service.” Instead, your team just does it – in every interaction they have with prospects and customers.

 

When you use this orientation you might understand in which part your customers are located. In most cases not all your (internal) customers fit one of the 3. Let me give you an example.

When I worked for a customer who had an internal IT operations they were both responsible for the IT of the ongoing business and the e-commerce activities. The ongoing business consisted of systems that had to be running but no specifics we needed just that they had to be as affordable as possible. These systems were running mail&calendaring, SAP HR and file services. Clearly this part of the company needed Operational excellence from their IT. And IT operations was fully focused on this.

However there was also the e-commerce customers, highly driven by sales&marketing and demanding a totally different model from IT, there need was clearly a need for innovation from IT. But IT operations had the focus on Operational Excellence. This didn’t fit at all. The marketing and software development part of the company was pushing for new innovative technologies to support their E-commerce drive but IT operation couldn’t deliver. Often you hear excuses like they are not agile, we don’t align. Well in my opinion it is just a orientation awareness from both sides. It is clear that a different structure (architecture if you want) is needed to solve this. Because the characteristics that belong to a Product leadership are very different from the operational excellence. But that doesn’t mean they cannot co-exist in a company, they very well can. It is a matter of segmenting your service offerings and develop them differently, their needs are different and should be handled differently. So the one size fits all approach doesn’t make it here, you might even need different teams working on these different orientations.

Now go back to your organisation and take a look with these 3 orientations in mind. I hope it helps you a bit!

 

Joost Bleijenberg

Agile vs. Waterfall non is better just different

Much is said about agile approaches mostly in contrast of conventional waterfall methods. This blog is just a quick side to side comparison, but I would like to show you on when to apply them and when not. Both waterfall and agile approaches can be used for software development, infrastructure engineering and even any other project which needs to manage problems into solutions in a controlled way.

Both of the method types can be used with great success and don’t believe in the fact that one is better then the other, it is just the environment that might prefer one above the other.

 

Waterfall

Waterfall is a method where all steps are done in a sequence, these phases are defined and when one is done the other takes of. Of course they can be used with techniques as iterations to go back to earlier phases to refine the work, but once the phase is closed the next one starts. Normally in organisation different departments sign of these phases to all commit on the contents. Basically the process moves from collecting requirements, to specification and design into the solutions area for implementation and operating. Each step moves more towards realisation and operating, or from problem definition into solution.
Waterfall is a great approach when both the problem and solution area are predictable and stable, that means that with defined steps it is possible to properly select solutions. This approach requires environments where problem areas are also stable, meaning that requirements won’t change over time. The picture below shows the approach in the dimension of problem and solution area.

20130129 232003

When the solutions area is known (meaning we have experience, done it before) and we are working inside a problem area which is known (been there done that) waterfall might be a good idea. Experience with both problem and solution areas are essential when selecting the waterfall model, that also means that lessons learned from the past need to be included to make it a “known” area. If a problem or solution space is an unknown territory it is clear that risk is introduced, either consciously or unconsciously. 

One of the most risky things about Waterfall approach is that risk needs to be managed properly, this as much of the previous phases are closed and therefor all the next phases will just be a further realisation of the previous phases. Risk might therefore be building up. If things change over time, like requirements, or for instance aspect in the design are overlooked there might be issues introduced into future phases. It is clear that in such cases each phase will just be an add up of all these risks leading to huge problems during for instance implementation or operation. And this is where agile comes in.

 

Agile

Agile is a method where there is much more interaction between the requirements and realisation phases, it could be seen as small iterations where the amount of changes is very small. That is the elegance of Agile, make things small, small requirements and small towards realisation. By doing this this approach is much more open to changes made by stakeholders, even late in the process and the risk is managed by the reduction of changes. With small iterations the process progresses towards the solution. Since both problem and solution spaces are much more unknown areas of work it is a good approach to limit the requirements to cover in each process iteration. This way risks on both problem and solution areas are limited for each iteration in stead by letting them building up over time, with huge impact for implantation or operations. Graphically the agile approach might look like this:
20130217 232811

These types of projects require high involvement of stakeholders from both the problem area and solution area because for each iteration they decide where to go next. Scrum is of course on of the most likely methods to select as an agile method.


Conclusion

Depending on the circumstances you might select the waterfall or agile model approach for your projects. As a guideline you could use the following criteria.

Waterfall application characteristics

  • Stable requirements
  • Low technology risk/predictable technology
  • Experience in both problem and solution area is high (been there done that)


Agile application characteristics:

  • Customer responsiveness/business responsiveness
  • Project uncertainty
  • Technology uncertainty
  • Innovation cultures

 

Hope it helps you to identify the method and to recognise the problems you face in your environment and the possible methods to solve them.

 

Joost Bleijenberg

Little’s law and Continuous Deployment

In this article I would briefly like to give some background of the reasons why Continuous Delivery and Integration are a success in reduction of wait time.

From the theoretical background on queue theory (Little’s law) we can derive that the wait time is depending on the queue size and the processing rate.

In the case of integration and deployment of code the processing rate doesn’t change, however if we perceive the queue size as a package of developed code (or change) the wait time will reduce if we manage to get the size of the work down. 

NewImage2

 

As the batches (change or queues as Little states it) grow larger a number of issues occur :
• Increases cycle time
• Increases variability non-linearly as 2^n
• Increases risk (it contains more potential issues)
• Reduces efficiency 
• Limited by its worst element

(source : Principles of Product Development Flow, Don Reinertsen)

The solution is to bring change with smaller batches/steps(queues) to reduce the issues mentioned above.

Welcome in the agile world.

 

Joost Bleijenberg 

 

 

 

The Agile principles

Just in case you might wonder what is behind this agile culture drive I’ve made a copy of the principles behind it.

Principles behind the Agile Manifesto

We follow these principles:
Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.

Welcome changing requirements, even late in
development. Agile processes harness change for
the customer’s competitive advantage.

Deliver working software frequently, from a
couple of weeks to a couple of months, with a
preference to the shorter timescale.

Business people and developers must work
together daily throughout the project.

Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done.

The most efficient and effective method of
conveying information to and within a development
team is face-to-face conversation.

Working software is the primary measure of progress.

Agile processes promote sustainable development.
The sponsors, developers, and users should be able
to maintain a constant pace indefinitely.

Continuous attention to technical excellence
and good design enhances agility.

Simplicity–the art of maximizing the amount
of work not done–is essential
.

The best architectures, requirements, and designs
emerge from self-organizing teams.

At regular intervals, the team reflects on how
to become more effective, then tunes and adjusts
its behavior accordingly

Source: http://agilemanifesto.org/

Joost Bleijenberg