May, 2016.

Introduction

Tableau and Spotfire are both great visualization tools. We will work through creating a set of visualizations combined into one dashboard using both these tools. This will showcase the similarities between both products, but it should be noted that the deeper you dive into the products (the more specific demands you have of the product) the more the differences will be apparent. We will work with data from opendata and the Fraser Institute to create the dashboards. The idea is to take school data for Calgary and create a simple list showing the types of schools and the quantity in each category. The next step is to show a list of schools filtered by menus. The school scores will be shown to the right of this list when available (the ratings are not Calgary specific and don’t cover all the schools).

Step 1: School Type Bar Graph

The first step in this dashboard is to create a bar graph that represents the amount of schools per type of school.

Tableau

The first step is to create a connection to the data in question. In tableau, you connect to the csv directly and for our purpose, we will not be joining the data. The first connection is to the open data csv that contains all the Calgary school information that we are interested in.

To start the visualization, we will need to drag the Type dimension into the column section and the number of records measure into the row. This will create our bar-chart.

The next step is to format the chart.

Right clicking on Type and selecting sort will give us a menu which lets us select how we want to show the information. In this case we want to show the greatest selection on the left and the smallest selection on the right. Select Descending by the Field “Number of Records” with aggregation sum. (Taking the average or max/min will only net us 1 when we need everything in the groups to add up).

Next we want to show the actual number on the top, so while holding the control button, drag the Type dimension from the columns pane onto the label button.

Finally, right click on the labels at the bottom of the bar-graph and select Rotate Label.

We now have a working bar-graph showing us the different number of schools of each type in Calgary.

The first step is to connect to the data. Just as we did with Tableau, connect to the csv to get started. Select the information that we are interested in and then select ok.

Here is where we see the main difference between the tools. In Tableau, you just start manipulating your data on the fly. In Spotfire, you select the kind of graph you would like to see before going any further. At the top, select the bar-chart button as show below.

It will try to auto select what Spotfire determines or estimates to be a good set of data that would correspond to a bar-graph. In the columns selector, use Type and as the height of the bars use Row Count.]

Just like we did above, right click on the labels and select vertical labels.

The next step is to sort it correctly.  To do this, right click in the visualization and select sort bars by value.  This will sort the bars in the exact manner we wanted.

The last step is to show the labels.  Right click on the visualization and select properties.  Then select the labels tab on the left and ensure the show labels for has All selected.  Unselect the bar segments and we should have a very similar graph to the one that was created in Tableau.

Step 2:  School Names

Our next step will be to create a list of the schools available in Calgary.  This list will be our master list whereas the list of ratings will be a smaller list (since not all schools have ratings in the information that was used).

Tableau

In a new sheet, drag the Name dimension to the rows pane.  Then drag another copy onto the labels button.  This will create two identical lists.  Right click on the Name dimension in the row pane and unselect Show Header.  This will hide the first list and should only show a list of all the schools in Calgary.  To prepare for filtering later, we will create a new calculated field that renames the null values of school boards into ‘No School Board’.

Right click on the whitespace in the Dimensions or Measures pane on the left of the screen, and then select Create Calculated Field.

Call this calculated field School Board and write in this code for the calculation:

If isnull([Board])
then ‘No School Board’
else [Board]
end

All this calculation does is check to see if in the Board field there are null values.  If there are, it replaces that value with No School Board.

Next drag the newly created School Board into the filters, and repeat the steps for Type and Grades.  (Always select all data for now).

Spotfire

Just as before, select the appropriate visualization (in this case, select the table icon as show on the left).  This will auto select the current data source that was imported and show all the columns.  To show the only the columns we want, right click on the visualization and select properties, then select columns on the left hand side.

Select all the columns that we are not interested in (all but Name) and click on the remove button.  This will now show us a list of all the Calgary schools available.

Step 3:  Ratings

The final visualization that we would like to show is the ratings visualization.  This will be a list of schools with their attached rating.  This information is found in a different csv, this time from the Fraser Institute (the information was in a table that was copied and pasted into a csv).

Tableau

As we did initially, we will add another data source to Tableau.  Click on the top left Data menu and select New Data Source.

On the left side, click on Text File and select the csv when prompted.  Now that we have this information, we need to correctly link both data sources together.  Typically, if there are field names that are the same, Tableau will automatically create a link (for blending purposes) between them.  In this case, none of the field names are similar so we will need to connect them.

Select Data -> Edit Relationships.

Select custom so that we can add a relationship and then click on Add as shown below.

Match School Name to Name and then click ok to create this link.  All this will tell Tableau is that the Name from one of the data sources is the same as the School Name in the other data source.  There is no other information that is passed along.

Now to create the visualization:

Click on the Calgary School Locations data source on the top left.  Drag the Name dimension from the left to the Row pane.  Next, click on the Rankings data source on the top left.  Drag the School Name next the Name dimension on the Row pane.  There should be a small database icon with an orange check mark next to it.  This means that that data source is being blended into the visualization.

Next drag the School Name to the filters pane.  We will remove all the null values to ensure only the schools present in both lists are shown.

Select None and then select Null on the top left of the window.  Ensure that Exclude is selected at the bottom right of the window.

The next step is to get rid of the Name dimension entirely.  We only needed it there to ensure that the Calgary School Locations was the primary data source.  This is really important for when we are pushing information from one visualization to the next.

Right click on the Name dimension and select remove.

Drag the Rating Measure from the bottom left to the text icon as shown on the left.  This should display all the ratings each school has had.  Since it is possible to get two ratings for a school (elementary and high school) we need to switch the aggregation level from sum to average.

Right click on the Sum(Rating) in the marks pane and hover over Measure (Sum). Change this to

Average.

Next we should standardize the number format to have 2 decimals. Right click on Avg(Rating) in the

marks pane and select format.

Select Number (Custom) and ensure that 2 is in decimal places section, then click ok.

The last step is optional. Above the ranking, there is nothing indicating what it is, so this step rectifies it. Drag Measure Names to the columns pane. Right-click on it, and select Edit Alias. Rename it to Ranking.

To add a new connection with Spotfire, click on the Add Data Tables icon on the top left as show here. Click Add -> File Select the csv file.

As we did with Tableau, we will create a relation between the two tables. Right click on the Rankings data table at the top left and select Data Table Properties. Select the relations tab and click on Manage Relations.

Match the School Name with Name and click ok (example below)

Click on the cross table icon on the top bar as shown on the left side.

There are a few important pieces on the cross table. First ensure that the data table is pointing to the Calgary School Locations. Drag Names to the left selector. On the bottom drag the Rating from the  Rankings data table and ensure the aggregation is set to average. A warning message at the top indicates that column matches may not be working correctly. Click on this icon.

This message should pop up. Click on the blue hyperlink at the bottom of the warning.

Ensure you match Name to School Name so that Spotfire can set up the data appropriately.

To better format the visualizations, right click anywhere in the visualization and toggle the legend and axis selector in the Visualization Features section.

Step 4: Putting it all together

The final step in this process is to ensure the filtering is working appropriately between the visualizations.

Create a dashboard, drag the School Types visualization from the left to the main window. Next select the School Names visualization and drag it underneath the current window. The bottom half should darken to represent where the visualization will land.

Repeat this process for the last visualization and place it on the bottom right.

The next step involves getting the filters to work properly. Click on Analysis in the menu at the top and select Filters -> Grades, School Board and Type. (You will need to do this sequentially) This should add filters on the right hand side.

Now we need to edit the look of the filters. Click on the down arrow next to the Type filter and select Single Value (Dropdown)

Do the same things with the School Board filter, this time selecting Multiple Values (Dropdown) and ensuring that the Only Relevant values option is selected. This demotes one filter to a secondary filter where the first (primary) filter feeds results into the secondary filter (cascading filters).

Do the exact same process as above with the final filter (Multiple Values(List) needs to be selected instead of Multiple Values (Dropdown))

Next, go to the School List worksheet. Right click on the filter and select apply to Worksheets -> Selected Worksheets. Check the box for Rankings. Do this with all the filters on this workbook.

Go back to the dashboard and select Dashboard from the menu at the top and select Actions.

Ensure the select icon is selected on the right. Select School names as the source sheet (the top selection) and Rankings as the Target sheet (the bottom selection).

As long as the All Fields option is selected, select ok. This completes the Tableau Dashboard.

Select the filter icon in the top menu (looks like a funnel).

Right click on the new filter pane and select organize filters. Uncheck all filters but Type, Board and Grades. Click Ok.

Right click on the Rankings visualization and select Data on the left side. To filter this table by selecting an item on the list, check the Limit data using markings option. Go to the properties of the top visualization and ensure it isn’t checked as well as any filtering scheme is not checked either (you will need to scroll down).

Right click on the filters again and click on Filtering Scheme Properties. You should see a list of all the filters. Click rename on all the filters we are currently using and rename them appropriately. This completes our dashboard for Spotfire.

Above is the dashboard for Tableau and below is the dashboard for Spotfire. They both are quite similar in looks.

There are a few differences between these dashboards. The way we built the Tableau dashboard, only the Schools that have a ranking will be shown. Whereas with the Spotfire dashboard, all the schools will be shown and if there is a ranking, it will also be shown. This is a slight difference, but if there is a preference, the same technique can be applied to Tableau to free up the list. Restricting the data on Spotfire is more complicated than Tableau. Since the data is being blended, you cannot restrict the data based on the blended data. Workarounds can be quite complicated but it will not be covered in this tutorial.

December 16th,  2015.

Business Process Management (BPM) can be described as a discipline that implements capabilities to manage complex process workflows, where coordination of the different elements that constitute a business process is required: human resources, documents, databases and internal / external information systems. It generally has high durability, from minutes to months; and high state management requirements, while implementing persistence of the state of each one of its activities.

• BPM can be implemented leveraging open standards such as Business Process Model and Notation (BPMN), with tools that provide capabilities to design and automate business processes defined in such standards. In addition, modern BPM tools offer capabilities such as user portals, business process optimization and prototyping, and a subset of capabilities from the Business Intelligence (BI) and Business Activity Monitoring (BAM) disciplines.
  The different stages of the BPM lifecycle are different depending on client needs, and also differ depending on the team implementing the technology; but the lifecycle is outlined usually as follows:
• Business Process Identification: in this stage, the focus is to identify business process that are high priority to the business, and at the same time identify processes that can be candidate for automation. This stage also focuses on analyzing the spectrum of selected processes, and identification of any duplication of effort in the process activities, or duplication of processes themselves.
• Definition of the As-Is Business Process Models: once the candidate processes have been selected, this stage focuses primarily in outlining the different activities that constitute each one of the processes, for end users to get a grasp of the complexities and identify areas for improvement.
• Business Process Re-engineering: once the areas of improvement of the process have been identified, this stage focuses on the analysis of different areas of the organization that the process has influences in; in order to redesign how the work is accomplished, with the intention of better meet the organization’s vision and reduce costs.
• Design of the To-Be Business Process Models: after the redesign of the business processes is completed, the design of to-be business model is intended for it to be documented in BPMN, with automation in mind. It is important to note that although not all activities within a process can be automated in a cost effective manner; the to-be business process may include checklists or notification activities in order to ensure that human tasks continue to take place and are tracked accordingly.
• Business Process Management Tool Selection: the organization must make a decision at some point as to what BPM technology vendor to select for implementation. Obviously, different vendors offer different basic BPM capabilities, and some other capabilities that will enhance the final experience, such as a customizable user portal, scorecards, dashboards, business process inspection and drill-down, activities status, mobile capabilities, live user interaction, single sign-on, among others.
• Business Process Prototyping: this stage focuses on the implementation of a prototype business process, in order to demonstrate the capabilities of the selected BPM toolset. Usually, the business process at this stage does not include an end-to-end process; however, it might include a few activities that take place across different functional areas of the process. Prototyping also helps identify the different roles that will be involved in the process, alongside specific milestones and most importantly, any sub-processes that may be candidate for re-usability in other related processes.
• Business Process Development and Automation: once prototyping ends and the capabilities of the BPM toolset are understood, development of the end-to-end automated process will commence; alongside any other capabilities that may be required for visibility and automation, such as Single-Sign-On (SSO), scorecards, dashboards, and user portal customization; in order to enhance the user experience.
• Business Process Integration: during the development and automation stage, one key aspect that enhances the user experience and proper management of complex processes, is integration with existing databases, applications or systems, such as Data Warehouses, Data Marts, Enterprise Resource Planning systems, Customer Relationship Management software, and even Cloud-based systems.
• Business Process Quality Assurance: this stage focuses on ensuring end users get familiar with the nature of the BPM solution, and this is normally accomplished by leveraging an iterative approach in the form of ‘play-backs’, which are intended to showcase how the automation and development of the process is evolving. Normally, and in most Software Development projects, quality assurance takes place right after development is completed; however, it is important to engage the Quality Assurance (QA) teams early in order to refine any issues that end users may encounter, and to test all the capabilities that the BPM solution will eventually implement.
• Business Process Execution: once the quality assurance stage has been delivered to customer satisfaction, and development has been signed-off and completed; the deployment of the business process to the Production environment would take place, and the execution stage will commence; and users will be able to complete their work leveraging the new BPM solution. Because of the nature of the quality assurance stage; and the iterative nature of the play-backs; end users will already be familiar with the behavior of the BPM solution.
• Business Process Monitoring: this stage focuses on monitoring the behavior of the automated business process, in order to identify areas where activities bottlenecks are taking place, and to delineate a remediation of performance issues, bug fixes, alongside the identification of future enhancements to the BPM solution.
• Business Process Optimization: this stage refers to ensuring that remediation of issues identified during the        monitoring stage, are addressed; and that any areas of improvement in the process are modified in order to        meet business needs.

It is possible to leverage Software Development Lifecycle methodologies in order to develop BPM solutions, and by evaluating specific client needs in order to deliver projects in either an iterative or sequential manner. Groundswell is proud of implement BPM solutions in a variety of toolsets, while following an implementation approach that is mostly technology-independent; until the time that the Development stage of the project commences, allowing clients to determine the best BPM technology stack for their very specific needs.

July 29th, 2014.

For the second installment of this month’s blog series on Competency Centres, Enabling Governance as the First Stage of an Integration Strategy, we’d like to touch on further key components of Competency Centres and what makes them successful in implementation. As a preface for this post, it may

For the second installment of this month’s blog series on Competency Centres, Enabling Governance as the First Stage of an Integration Strategy, we’d like to touch on further key components of Competency Centres and what makes them successful in implementation.

As a preface for this post, it may help to backtrack and re-read our first blog posted last week which serves as a ‘Competency Centre 101’ of sorts.

As you will recall, we capped off that post by sharing the foundational components of a Competency Centre: Governance Committee, Integration Inventory, Governance Process, Reference Architecture, Design Patterns, Development Guidelines & Methodology. However, it is important to note that additional components can be added to the Competency Centre in subsequent phases, as required.

Governance Committee

A Governance Committee needs to be established early, and it will:

• Be constituted by stakeholders representing different Information Technology (IT) business-facing groups.
• Meet regularly to review integration progress, challenges and exceptions to any of the foundational components of the Competency Centre.

Integration Inventory

The Integration Inventory is a very important component of the Competency Centre, since it will allow:

• Identification of existing integration patterns, technologies utilized, documentation availability.
• Clarity around technical and functional ownership and the business data objects being integrated.
• Consistency of information, since as integration projects are implemented or retired, the inventory requires to be updated.

Governance Process
A Governance Process is crucial for the proper execution of the Competency Centre, and must be created to adapt to the current processes of the organization. The process will ensure that all the components of the Competency Center depicted here work in unison, and that benefits are measured when the process is completed. This process is usually depicted in a Cross Functional Flowchart, or utilizing Business Process Management Notation.

Reference Architecture

The Reference Architecture is intended to become an abstract representation of the permissible patterns within specific domains (i.e. EI, ETL, SOA, BPM). This deliverable must be technology independent, and must be updated on a regular basis in order to identify new or deprecated domains. Additionally, any pattern identified within the organization that does not belong to a domain; must be considered an anti-pattern.

The Reference Architecture will also include a set of Principles to be followed for all integration work across the organization. Alongside these Principles, a set of Policies will also be outlined in the Reference Architecture for proper Governance to take place.

Design Patterns

The Design Patterns derive from the domains outlined in the Reference Architecture, and are also technology independent. Each domain defined in the Reference Architecture, will have a permissible set of Design Patterns; that are intended to provide consistency across the organization. These Design Patterns will serve as the foundation for all integration work performed within the organization.

Development Guidelines

The Development Guidelines are technology specific, and must be derived from the Design Patterns. Each one of the technologies aligned to each domain depicted in the Reference Architecture, must have a set of Development Guidelines. These Development Guidelines will serve as the baseline for all code/configuration reviews that will take place through the integration development lifecycle, as defined in the Governance Process.

Methodology

The implementation of a Methodology is also essential, and it will dictate the type of deliverables expected throughout the entire integration development lifecycle. The Methodology will provide each team with guidance, and the proper templates to be utilized to complete a project. The Methodology can be utilized or merged with any other existing methodologies in the organization.

Some of the templates provided by the Methodology are:

• Project Assessment
• Requirements Document
• Architecture Impact Assessment
• Risk Assessment
• Architecture Document
• Detailed Design
• Architecture Fit Assessment
• Test Plan and Strategy
• Test Cases
• Code Reviews
• Deployment
• Cut-Over
• Decision Records
• Technical Recommendations

Governance Process

A Governance Process is crucial for the proper execution of the Competency Centre, since it will:

• Confirm that the proper Methodology is followed for all integration projects.
• Make sure that compliance with the Reference Architecture takes place.
• Ensure that the proper Design Patterns are followed in all integration projects.
• Ensure that the code or configuration created for integration projects meets the proper Development Guidelines and standards, in order to ensure its quality.
• Make certain the Governance Committee is engaged when necessary.
• Confirm that the process to ensure Governance is followed and takes place for every project.
• Ensure the Integration Inventory is kept up to date with new, changed and retired integrations.

The Governance Process, as mentioned before; must be created with the specific needs of the organization in mind. An overly complex process may be too overwhelming and not be followed, and a process that is too simple may not be respected.

July 22nd, 2014.

As outlined in a previous Groundswell blog post, we identified how organizations continue to adopt technology to accomplish their daily processes; while enabling employees to perform tasks successfully, and effortlessly. We also identified how modern integration technologies allow the seamless addition or decommission

As outlined in a previous Groundswell blog post, we identified how organizations continue to adopt technology to accomplish their daily processes; while enabling employees to perform tasks successfully, and effortlessly. We also identified how modern integration technologies allow the seamless addition or decommission of applications, and how an Integration Strategy will provide the necessary initial steps to implement Governance, usually in the form of a Competency Center; along with the acquisition and regimented implementation of those integration technologies.

Before we begin with the Competency Center aspects derived from an Integration Strategy, let us make a few clarifications to our previous blog, since we received quite a bit of feedback on it.

• When we speak about modern integration technologies, we are referring to those Enterprise Tools that provide capabilities for Enterprise Integration (EI), Service Oriented Architecture (SOA), Extract Transform Load (ETL), Business Process Management (BPM), amongst others; and not necessarily to those implemented by more conservative organizations where information is shared, and processes triggered (ie: putting a Post-it note® in someone else’s screen or desk).
• The depiction of technology between people and processes in our graphics is not intended to say that technology creates a barrier between the other two. Please consider these a graphical representation only to illustrate our discussion points.

So, let us begin with a Competency Center definition:

“A governance body that provides guidance and expertise within an organization, in order to coordinate, support, and act as the repository of knowledge, for a specific discipline; in order to gain efficiencies and effectiveness for a competitive advantage.”

Essentially, Competency Centers originated from the need to have proper Governance and guidance within an organization, in order to avoid proliferation of competing technologies, architectures and design patterns; and with the intention to regulate growth of a certain discipline. In this case, let’s consider this discipline is Integration, since Competency Centers can be redesigned to include Governance for Business Intelligence, Business Process Management, Enterprise Content Management, and so on.

It is important to note Competency Centers are commonly established before, or at the same time when a new integration technology is acquired; in order to early assess, document, define and guide the existing and future integration components that support the business processes which bring revenue to the organization.

That said, let’s go back to the organization we depicted in our previous blog: an Integration Strategy is in place, and the decision to acquire a new integration technology has been made.

However, the acquisition of such integration technology is not just about the purchase, proper Governance in the form of a Competency Center must be in place ahead of time in order to avoid common pitfalls in the future, such as: competing technologies, disjointed processes, unknown data quality, unsustainable point to point interfaces, and complex systems architecture. In our experience, organizations that do not implement a Competency Centre usually have the following symptoms:

• Tendency to implement the new technology in a disorganized way; with disparate architectures and competing design patterns.

• Failure to keep track of old integrations that require to be replaced, and implement new integration projects without following a process.
• Propensity to accumulate competing and disparate technologies, architectures and patterns that will lead them back to the original problem they were trying to solve.
• Disorganized in keeping a proper knowledge base about their integrations, with no documentation to enable sustainment.
• Susceptibility to integrated data ownership problems, which will lead to inefficiencies and ineffectiveness; affecting the organization’s competitiveness in the marketplace.
• Major impacts to their net profit due to data discrepancies that lead to higher costs of ownership.

The establishment of a Competency Centre within an organization helps solve and alleviate most of the issues and risks outlined before. However, like anything else, a Competency Centre will not be a ‘silver bullet’ that will immediately resolve those issues and mitigate the derived risks. A Competency Centre is designed to establish processes through technology, enabling people to leverage integration in the proper manner. As the organization matures their approach to integration, by following and enforcing Governance, and once the different foundational components of the Competency Center are clearly delineated and valued, risks will be addressed and mitigated, and issues will eventually be resolved.

In Groundswell’s experience, the minimal foundational components of a Competency Centre are as follows:

• Governance Committee
• Integration Inventory
• Governance Process
• Reference Architecture
• Design Patterns
• Development Guidelines
• Methodology

It is important to note that additional components can be added to the Competency Centre in subsequent phases, as required.

This is the first of a three-post blog series. Stay tuned for our next post which will cover key components of Competency Centres, including Governance Committee, Integration Inventory, Governance Process, Reference Architecture, Design Patterns, Development Guidelines and Methodology.