May, 2016.

Introduction

Tableau and Spotfire are both great visualization tools. We will work through creating a set of visualizations combined into one dashboard using both these tools. This will showcase the similarities between both products, but it should be noted that the deeper you dive into the products (the more specific demands you have of the product) the more the differences will be apparent. We will work with data from opendata and the Fraser Institute to create the dashboards. The idea is to take school data for Calgary and create a simple list showing the types of schools and the quantity in each category. The next step is to show a list of schools filtered by menus. The school scores will be shown to the right of this list when available (the ratings are not Calgary specific and don’t cover all the schools).

Step 1: School Type Bar Graph

The first step in this dashboard is to create a bar graph that represents the amount of schools per type of school.

Tableau

The first step is to create a connection to the data in question. In tableau, you connect to the csv directly and for our purpose, we will not be joining the data. The first connection is to the open data csv that contains all the Calgary school information that we are interested in.

To start the visualization, we will need to drag the Type dimension into the column section and the number of records measure into the row. This will create our bar-chart.

The next step is to format the chart.

Right clicking on Type and selecting sort will give us a menu which lets us select how we want to show the information. In this case we want to show the greatest selection on the left and the smallest selection on the right. Select Descending by the Field “Number of Records” with aggregation sum. (Taking the average or max/min will only net us 1 when we need everything in the groups to add up).

Next we want to show the actual number on the top, so while holding the control button, drag the Type dimension from the columns pane onto the label button.

Finally, right click on the labels at the bottom of the bar-graph and select Rotate Label.

We now have a working bar-graph showing us the different number of schools of each type in Calgary.

The first step is to connect to the data. Just as we did with Tableau, connect to the csv to get started. Select the information that we are interested in and then select ok.

Here is where we see the main difference between the tools. In Tableau, you just start manipulating your data on the fly. In Spotfire, you select the kind of graph you would like to see before going any further. At the top, select the bar-chart button as show below.

It will try to auto select what Spotfire determines or estimates to be a good set of data that would correspond to a bar-graph. In the columns selector, use Type and as the height of the bars use Row Count.]

Just like we did above, right click on the labels and select vertical labels.

The next step is to sort it correctly.  To do this, right click in the visualization and select sort bars by value.  This will sort the bars in the exact manner we wanted.

The last step is to show the labels.  Right click on the visualization and select properties.  Then select the labels tab on the left and ensure the show labels for has All selected.  Unselect the bar segments and we should have a very similar graph to the one that was created in Tableau.

Step 2:  School Names

Our next step will be to create a list of the schools available in Calgary.  This list will be our master list whereas the list of ratings will be a smaller list (since not all schools have ratings in the information that was used).

Tableau

In a new sheet, drag the Name dimension to the rows pane.  Then drag another copy onto the labels button.  This will create two identical lists.  Right click on the Name dimension in the row pane and unselect Show Header.  This will hide the first list and should only show a list of all the schools in Calgary.  To prepare for filtering later, we will create a new calculated field that renames the null values of school boards into ‘No School Board’.

Right click on the whitespace in the Dimensions or Measures pane on the left of the screen, and then select Create Calculated Field.

Call this calculated field School Board and write in this code for the calculation:

If isnull([Board])
then ‘No School Board’
else [Board]
end

All this calculation does is check to see if in the Board field there are null values.  If there are, it replaces that value with No School Board.

Next drag the newly created School Board into the filters, and repeat the steps for Type and Grades.  (Always select all data for now).

Spotfire

Just as before, select the appropriate visualization (in this case, select the table icon as show on the left).  This will auto select the current data source that was imported and show all the columns.  To show the only the columns we want, right click on the visualization and select properties, then select columns on the left hand side.

Select all the columns that we are not interested in (all but Name) and click on the remove button.  This will now show us a list of all the Calgary schools available.

Step 3:  Ratings

The final visualization that we would like to show is the ratings visualization.  This will be a list of schools with their attached rating.  This information is found in a different csv, this time from the Fraser Institute (the information was in a table that was copied and pasted into a csv).

Tableau

As we did initially, we will add another data source to Tableau.  Click on the top left Data menu and select New Data Source.

On the left side, click on Text File and select the csv when prompted.  Now that we have this information, we need to correctly link both data sources together.  Typically, if there are field names that are the same, Tableau will automatically create a link (for blending purposes) between them.  In this case, none of the field names are similar so we will need to connect them.

Select Data -> Edit Relationships.

Select custom so that we can add a relationship and then click on Add as shown below.

Match School Name to Name and then click ok to create this link.  All this will tell Tableau is that the Name from one of the data sources is the same as the School Name in the other data source.  There is no other information that is passed along.

Now to create the visualization:

Click on the Calgary School Locations data source on the top left.  Drag the Name dimension from the left to the Row pane.  Next, click on the Rankings data source on the top left.  Drag the School Name next the Name dimension on the Row pane.  There should be a small database icon with an orange check mark next to it.  This means that that data source is being blended into the visualization.

Next drag the School Name to the filters pane.  We will remove all the null values to ensure only the schools present in both lists are shown.

Select None and then select Null on the top left of the window.  Ensure that Exclude is selected at the bottom right of the window.

The next step is to get rid of the Name dimension entirely.  We only needed it there to ensure that the Calgary School Locations was the primary data source.  This is really important for when we are pushing information from one visualization to the next.

Right click on the Name dimension and select remove.

Drag the Rating Measure from the bottom left to the text icon as shown on the left.  This should display all the ratings each school has had.  Since it is possible to get two ratings for a school (elementary and high school) we need to switch the aggregation level from sum to average.

Right click on the Sum(Rating) in the marks pane and hover over Measure (Sum). Change this to

Average.

Next we should standardize the number format to have 2 decimals. Right click on Avg(Rating) in the

marks pane and select format.

Select Number (Custom) and ensure that 2 is in decimal places section, then click ok.

The last step is optional. Above the ranking, there is nothing indicating what it is, so this step rectifies it. Drag Measure Names to the columns pane. Right-click on it, and select Edit Alias. Rename it to Ranking.

To add a new connection with Spotfire, click on the Add Data Tables icon on the top left as show here. Click Add -> File Select the csv file.

As we did with Tableau, we will create a relation between the two tables. Right click on the Rankings data table at the top left and select Data Table Properties. Select the relations tab and click on Manage Relations.

Match the School Name with Name and click ok (example below)

Click on the cross table icon on the top bar as shown on the left side.

There are a few important pieces on the cross table. First ensure that the data table is pointing to the Calgary School Locations. Drag Names to the left selector. On the bottom drag the Rating from the  Rankings data table and ensure the aggregation is set to average. A warning message at the top indicates that column matches may not be working correctly. Click on this icon.

This message should pop up. Click on the blue hyperlink at the bottom of the warning.

Ensure you match Name to School Name so that Spotfire can set up the data appropriately.

To better format the visualizations, right click anywhere in the visualization and toggle the legend and axis selector in the Visualization Features section.

Step 4: Putting it all together

The final step in this process is to ensure the filtering is working appropriately between the visualizations.

Create a dashboard, drag the School Types visualization from the left to the main window. Next select the School Names visualization and drag it underneath the current window. The bottom half should darken to represent where the visualization will land.

Repeat this process for the last visualization and place it on the bottom right.

The next step involves getting the filters to work properly. Click on Analysis in the menu at the top and select Filters -> Grades, School Board and Type. (You will need to do this sequentially) This should add filters on the right hand side.

Now we need to edit the look of the filters. Click on the down arrow next to the Type filter and select Single Value (Dropdown)

Do the same things with the School Board filter, this time selecting Multiple Values (Dropdown) and ensuring that the Only Relevant values option is selected. This demotes one filter to a secondary filter where the first (primary) filter feeds results into the secondary filter (cascading filters).

Do the exact same process as above with the final filter (Multiple Values(List) needs to be selected instead of Multiple Values (Dropdown))

Next, go to the School List worksheet. Right click on the filter and select apply to Worksheets -> Selected Worksheets. Check the box for Rankings. Do this with all the filters on this workbook.

Go back to the dashboard and select Dashboard from the menu at the top and select Actions.

Ensure the select icon is selected on the right. Select School names as the source sheet (the top selection) and Rankings as the Target sheet (the bottom selection).

As long as the All Fields option is selected, select ok. This completes the Tableau Dashboard.

Select the filter icon in the top menu (looks like a funnel).

Right click on the new filter pane and select organize filters. Uncheck all filters but Type, Board and Grades. Click Ok.

Right click on the Rankings visualization and select Data on the left side. To filter this table by selecting an item on the list, check the Limit data using markings option. Go to the properties of the top visualization and ensure it isn’t checked as well as any filtering scheme is not checked either (you will need to scroll down).

Right click on the filters again and click on Filtering Scheme Properties. You should see a list of all the filters. Click rename on all the filters we are currently using and rename them appropriately. This completes our dashboard for Spotfire.

Above is the dashboard for Tableau and below is the dashboard for Spotfire. They both are quite similar in looks.

There are a few differences between these dashboards. The way we built the Tableau dashboard, only the Schools that have a ranking will be shown. Whereas with the Spotfire dashboard, all the schools will be shown and if there is a ranking, it will also be shown. This is a slight difference, but if there is a preference, the same technique can be applied to Tableau to free up the list. Restricting the data on Spotfire is more complicated than Tableau. Since the data is being blended, you cannot restrict the data based on the blended data. Workarounds can be quite complicated but it will not be covered in this tutorial.

December 16th,  2015.

Business Process Management (BPM) can be described as a discipline that implements capabilities to manage complex process workflows, where coordination of the different elements that constitute a business process is required: human resources, documents, databases and internal / external information systems. It generally has high durability, from minutes to months; and high state management requirements, while implementing persistence of the state of each one of its activities.

• BPM can be implemented leveraging open standards such as Business Process Model and Notation (BPMN), with tools that provide capabilities to design and automate business processes defined in such standards. In addition, modern BPM tools offer capabilities such as user portals, business process optimization and prototyping, and a subset of capabilities from the Business Intelligence (BI) and Business Activity Monitoring (BAM) disciplines.
  The different stages of the BPM lifecycle are different depending on client needs, and also differ depending on the team implementing the technology; but the lifecycle is outlined usually as follows:
• Business Process Identification: in this stage, the focus is to identify business process that are high priority to the business, and at the same time identify processes that can be candidate for automation. This stage also focuses on analyzing the spectrum of selected processes, and identification of any duplication of effort in the process activities, or duplication of processes themselves.
• Definition of the As-Is Business Process Models: once the candidate processes have been selected, this stage focuses primarily in outlining the different activities that constitute each one of the processes, for end users to get a grasp of the complexities and identify areas for improvement.
• Business Process Re-engineering: once the areas of improvement of the process have been identified, this stage focuses on the analysis of different areas of the organization that the process has influences in; in order to redesign how the work is accomplished, with the intention of better meet the organization’s vision and reduce costs.
• Design of the To-Be Business Process Models: after the redesign of the business processes is completed, the design of to-be business model is intended for it to be documented in BPMN, with automation in mind. It is important to note that although not all activities within a process can be automated in a cost effective manner; the to-be business process may include checklists or notification activities in order to ensure that human tasks continue to take place and are tracked accordingly.
• Business Process Management Tool Selection: the organization must make a decision at some point as to what BPM technology vendor to select for implementation. Obviously, different vendors offer different basic BPM capabilities, and some other capabilities that will enhance the final experience, such as a customizable user portal, scorecards, dashboards, business process inspection and drill-down, activities status, mobile capabilities, live user interaction, single sign-on, among others.
• Business Process Prototyping: this stage focuses on the implementation of a prototype business process, in order to demonstrate the capabilities of the selected BPM toolset. Usually, the business process at this stage does not include an end-to-end process; however, it might include a few activities that take place across different functional areas of the process. Prototyping also helps identify the different roles that will be involved in the process, alongside specific milestones and most importantly, any sub-processes that may be candidate for re-usability in other related processes.
• Business Process Development and Automation: once prototyping ends and the capabilities of the BPM toolset are understood, development of the end-to-end automated process will commence; alongside any other capabilities that may be required for visibility and automation, such as Single-Sign-On (SSO), scorecards, dashboards, and user portal customization; in order to enhance the user experience.
• Business Process Integration: during the development and automation stage, one key aspect that enhances the user experience and proper management of complex processes, is integration with existing databases, applications or systems, such as Data Warehouses, Data Marts, Enterprise Resource Planning systems, Customer Relationship Management software, and even Cloud-based systems.
• Business Process Quality Assurance: this stage focuses on ensuring end users get familiar with the nature of the BPM solution, and this is normally accomplished by leveraging an iterative approach in the form of ‘play-backs’, which are intended to showcase how the automation and development of the process is evolving. Normally, and in most Software Development projects, quality assurance takes place right after development is completed; however, it is important to engage the Quality Assurance (QA) teams early in order to refine any issues that end users may encounter, and to test all the capabilities that the BPM solution will eventually implement.
• Business Process Execution: once the quality assurance stage has been delivered to customer satisfaction, and development has been signed-off and completed; the deployment of the business process to the Production environment would take place, and the execution stage will commence; and users will be able to complete their work leveraging the new BPM solution. Because of the nature of the quality assurance stage; and the iterative nature of the play-backs; end users will already be familiar with the behavior of the BPM solution.
• Business Process Monitoring: this stage focuses on monitoring the behavior of the automated business process, in order to identify areas where activities bottlenecks are taking place, and to delineate a remediation of performance issues, bug fixes, alongside the identification of future enhancements to the BPM solution.
• Business Process Optimization: this stage refers to ensuring that remediation of issues identified during the        monitoring stage, are addressed; and that any areas of improvement in the process are modified in order to        meet business needs.

It is possible to leverage Software Development Lifecycle methodologies in order to develop BPM solutions, and by evaluating specific client needs in order to deliver projects in either an iterative or sequential manner. Groundswell is proud of implement BPM solutions in a variety of toolsets, while following an implementation approach that is mostly technology-independent; until the time that the Development stage of the project commences, allowing clients to determine the best BPM technology stack for their very specific needs.

August 1st, 2014.

For the final installment of this blog series on Competency Centres, we would like to finalize by talking about the different challenges that a governance body would address, and the long term benefits that it will bring to the organization. As a preface to this blog, please refer to the

For the final installment of this blog series on Competency Centres, we would like to finalize by talking about the different challenges that a governance body would address, and the long term benefits that it will bring to the organization. As a preface to this blog, please refer to the first blog, and to the second blog.

The Challenges of Governance

The following are some of the challenges that Governance, in the form of a Competency Centre; helps solve:

• The most challenging part of every organization: People. With people, politics materialize. Corporate politics is always the elephant in the room. Governance protects people from themselves. In heavily bureaucratic organizations, where resistance to change is substantial; proper Governance can help, since it will handle progressive change and implementation of new processes and systems.

•  It is human nature to first implement technologies, and then implement the rules (i.e.Governance). We first invented cars… and then the traffic laws, right? Establishing Governance early helps establish the necessary policies, principles and guidelines that the organization will follow.

• Proper communication between business units is crucial for the successful establishment of Governance; otherwise, noncompliance will propagate across the organization.

• Proper Governance will also address any challenges that today’s multi-cultural and multi-vision organizational environments bring, and avoid the classic: “I will put this code I just created in production by myself, that’s how we do it where I come from… and, there is no better test than production.”.

• Nowadays, many leaders and managers within large organizations try to evade confrontation for many different reasons; all of them creating bureaucracy and overkill: hopes of organizational advancement, corporate politics avoidance, etc. However, this lack of decision making can be handled by a Governance Body with established mechanisms and processes within the organization, and not having to rely on a single individual.  Additionally, proper Governance will not let certain groups get their own way; bringing consistency across the organization and reducing enterprise debt.

Benefits of a Competency Center

The establishment of a Competency Center does:

• Enable and maintain communications across different teams and stakeholders.

* Provide Governance for new and existing integration projects.

* Keep a consistent record of the existing integrations across the organization.

• Provide expertise around system integration, assisting in the elaboration of technical recommendations, and issuing decision records when compliance does not take place.

• Provide guidance with Development Guidelines, standards and best practices, along with enforcing adherence to an established Reference Architecture and its derived Design Patterns.

• Provide guidance around integration-specific Methodologies, and contains all necessary templates for proper execution of integration projects.

• Deliver an escalation mechanism through the Governance Committee.

Our Approach

Groundswell’s Competency Center methodology allows our clients to establish the proper Governance mechanisms, in order to properly implement and sustain modern integration technologies; while allowing people to leverage technology in order to perform their processes, bringing revenue to the organization. Groundswell works with clients in order to properly customize a Competency Center, aimed at Integration, Business Intelligence, Business Process Management or Enterprise Content Management; that will allow them build an effective and progressive organization.

July 22nd, 2014.

As outlined in a previous Groundswell blog post, we identified how organizations continue to adopt technology to accomplish their daily processes; while enabling employees to perform tasks successfully, and effortlessly. We also identified how modern integration technologies allow the seamless addition or decommission

As outlined in a previous Groundswell blog post, we identified how organizations continue to adopt technology to accomplish their daily processes; while enabling employees to perform tasks successfully, and effortlessly. We also identified how modern integration technologies allow the seamless addition or decommission of applications, and how an Integration Strategy will provide the necessary initial steps to implement Governance, usually in the form of a Competency Center; along with the acquisition and regimented implementation of those integration technologies.

Before we begin with the Competency Center aspects derived from an Integration Strategy, let us make a few clarifications to our previous blog, since we received quite a bit of feedback on it.

• When we speak about modern integration technologies, we are referring to those Enterprise Tools that provide capabilities for Enterprise Integration (EI), Service Oriented Architecture (SOA), Extract Transform Load (ETL), Business Process Management (BPM), amongst others; and not necessarily to those implemented by more conservative organizations where information is shared, and processes triggered (ie: putting a Post-it note® in someone else’s screen or desk).
• The depiction of technology between people and processes in our graphics is not intended to say that technology creates a barrier between the other two. Please consider these a graphical representation only to illustrate our discussion points.

So, let us begin with a Competency Center definition:

“A governance body that provides guidance and expertise within an organization, in order to coordinate, support, and act as the repository of knowledge, for a specific discipline; in order to gain efficiencies and effectiveness for a competitive advantage.”

Essentially, Competency Centers originated from the need to have proper Governance and guidance within an organization, in order to avoid proliferation of competing technologies, architectures and design patterns; and with the intention to regulate growth of a certain discipline. In this case, let’s consider this discipline is Integration, since Competency Centers can be redesigned to include Governance for Business Intelligence, Business Process Management, Enterprise Content Management, and so on.

It is important to note Competency Centers are commonly established before, or at the same time when a new integration technology is acquired; in order to early assess, document, define and guide the existing and future integration components that support the business processes which bring revenue to the organization.

That said, let’s go back to the organization we depicted in our previous blog: an Integration Strategy is in place, and the decision to acquire a new integration technology has been made.

However, the acquisition of such integration technology is not just about the purchase, proper Governance in the form of a Competency Center must be in place ahead of time in order to avoid common pitfalls in the future, such as: competing technologies, disjointed processes, unknown data quality, unsustainable point to point interfaces, and complex systems architecture. In our experience, organizations that do not implement a Competency Centre usually have the following symptoms:

• Tendency to implement the new technology in a disorganized way; with disparate architectures and competing design patterns.

• Failure to keep track of old integrations that require to be replaced, and implement new integration projects without following a process.
• Propensity to accumulate competing and disparate technologies, architectures and patterns that will lead them back to the original problem they were trying to solve.
• Disorganized in keeping a proper knowledge base about their integrations, with no documentation to enable sustainment.
• Susceptibility to integrated data ownership problems, which will lead to inefficiencies and ineffectiveness; affecting the organization’s competitiveness in the marketplace.
• Major impacts to their net profit due to data discrepancies that lead to higher costs of ownership.

The establishment of a Competency Centre within an organization helps solve and alleviate most of the issues and risks outlined before. However, like anything else, a Competency Centre will not be a ‘silver bullet’ that will immediately resolve those issues and mitigate the derived risks. A Competency Centre is designed to establish processes through technology, enabling people to leverage integration in the proper manner. As the organization matures their approach to integration, by following and enforcing Governance, and once the different foundational components of the Competency Center are clearly delineated and valued, risks will be addressed and mitigated, and issues will eventually be resolved.

In Groundswell’s experience, the minimal foundational components of a Competency Centre are as follows:

• Governance Committee
• Integration Inventory
• Governance Process
• Reference Architecture
• Design Patterns
• Development Guidelines
• Methodology

It is important to note that additional components can be added to the Competency Centre in subsequent phases, as required.

This is the first of a three-post blog series. Stay tuned for our next post which will cover key components of Competency Centres, including Governance Committee, Integration Inventory, Governance Process, Reference Architecture, Design Patterns, Development Guidelines and Methodology.