Open Source AddOn for deduplication, data modelling and advanced HTML reporting

Introduction

In one of the Sparx sites I am supporting an urgent need existed for deduplicating elements in a large EA repository. In the previous period a large number of Architects worked in the repository and there was no procedure for preventing duplicates. To improve the content of the repository the fist step is to deduplicate content based on scripts.

Installation of IDEA

Installation is relatively easy with the following steps:

 

  • Download the setup file  and run the setup as administrator (right click on the file and select run as asministrator
  • When the installation does not work please register the DLL which is stored in dll.zip based on the steps described on the sparx website http://www.sparxsystems.com/enterprise_architect_user_guide/9.3/automation/deployingaddins.html Please note that you should run the regasm step in the directory of the actively installed DotNet version (last version folder with regasm available as an executable.
  • Import the duplicatiereport.XML file in the resource as a user report template (including the report fragment). Use The transfer -> Import Reference Data function in EA
  • When you want to generate HTML reports with this DLL please extract the and adapt the template to your own company brand style, a sample is included in htmltemplate.zip in the installation zip
  • When you want the tool to generate PDF files within the HTML generator please extend the resources in EA with extra rtf templates as configured in the htmlgenerator.xml report definitions (please import these in the resources module in EA, use The transfer -> Import Reference Data function in EA)

Functionalities

The IDEA AddOn is used for generating layers in an integrated data architecture and for creating mappers and mergers for XSD schema's etc. Please notice that also scripts are available for this project in a sample repository. See the screenshots below:

Mapper screen

Refactor screen 

Usage

Usage of this extension for mappers and generating data layers has the following logical steps:

  1. Select in the project browser the package, diagram or element (and eventually subpackages) you want to generate mappers, mergers or layers for
  2. Right click the package and select extensions -> IDEA Browser Helper
  3. Select the elements which you want to generate and select the relevant attributes when relevant.
  4. Press the Generate button
  5. Please notice that a number of forms have tabpages for more functionalities

Deduplication

Deduplicating elements is merging two or more elements to one where the child entities of this element are also merged. A duplicatie validation is based on the following:

  • Name
  • Object_type
  • Stereotype
  • Version

When these all match an element is considered a duplicate. The elements need to be exactly the same. Especially for the name this can be a problem. For example eBS and E-BS are not the same and are not consided duplicates

When a deduplication is done the following child elements are merged:

  • Properties or attributes
  • Methods or operations
  • Notes
  • Requirements
  • Scenarios
  • Tagged values
  • Linked files

It is easily possible to extend this list since the code to do the child element transfer from duplicate to original, see therefore the code within the AddOn

Screen shots

Screen 1 Validate

 Before we do the duplication we have the possibily to generate a report of the duplicate elements or we can get a summary overview within this extension (see screen 3).

Screen 2 Deduplicate

This is the tabpage with selecting the merge aspects for the duplicated elements. On the left hand side you see which child entities will be merged. On the righthand side you can select a number of functionalities

  • Create a folder where the processed duplicate will be transferred to
  • Close this window when the processing is ready
  • Do a recursive processing for the different subpackages of the selected package

Screen 3 Validation result

This screen shows the result f the validation list option.It gives a list of the duplicate including the name, the package and the author of the duplicate. This gives you an option to modify the elements before you merge them with the deduplicate routine.

Release management

In a situation where multiple (scrum) teams are developing on the same model in EA, releases can be quite challenging. Especially when each team has a specific release rhytm. 

When a diagram driven modelling approach is followed it can be of interest to create a release manager that automates a number of steps for the model manager. Because transferring parts of the model in a secure way automated support of this release process can be very helpful and prevent the release process from introducing mistakes.

The steps to be taken will be:

  1. Activate the Model Producing environment (development)
  2. Go to the package with the diagrams of the team(s) that want to produce their models
  3. Collect all the elements of the diagrams in the selected package
  4. Create a baseline
  5. Export the data via XMI to a file in the filesystem
  6. Transfer all the elements back to the original package(s)
  7. Activate the Model Consuming environment (production)
  8. Go to the team package (for consuming the modified model)
  9. Create a baseline
  10. Import the XMI file from the filesystem

These steps are all included in the release manager. In the picture below you get an idea of the user interface for this functionality

  

BizzDesign Converter

For modelling Enterprise Architectures organisations can choose for multiple tools. In the Netherlands numerous organisations use BizzDesign as modelling tool. However license costs are relatively high and therefore some of the EAxpertise customers are considering to migrate to Sparx. 

The functionality offered by both tools overlap to a great extend and especially in data modelling Sparx has more functionality than BizzDesign. For one of our customers we did a research on the migration options, there are three models in the enterprise architecture:

  • Enterprise Architecture modelled in ArchiMate 3 models. This can be migrated very well with the Model Exchange Format standard in the ArchiMate 3 definition.
  • Business processes modelled in BPMN models This can be migrated with the BPMN exchange format 
  • Data  Models modelled in the Enterprise Data Model. For this part of the model there is no exchange format so we had to develop  something for this.

For the last part of the enterprise model we had to develop an extension. The rest of this article described this extension.

Migration extension

We first tried to develop a solution based on a script but we needed a simple user interface to control the steps so we decided to extend the IDEA DLL with this functionality. In the image you get an idea of the functionality

In the screenshot you see the following functionalities:

  • Options to load the data from an excel file. BizzDesign has the possibility to define excel files for the entities that can not be exported by the stanrd XML formats like MEF and BPMN. This applies to the EDM entities
  • Migrating the Entities data from the excel file to the repository of Sparx
  • Migrating the Associations from the excel file to the repository of Sparx
  • The transfer of the Attributes to Sparx
  • The last option needs a little explanation. BizzDesign has the option, just like Sparx to add hyperlinks in the notes that link to other elements in the repository. However they use a different internal format for these hyperlinks. This functionality transforms the format of the hyperlinks to the Sparx format.

An extra remark is needed for the counter under the load data button. This is used for loading the worksheet with that ordinal position in the workbook file.

Search Helpers

One of the challenges of a teambased architecture repository based on the ArchiMate language is to reduce the number of duplicate entities. With duplicates I mean an Archimate concept with the same name and stereotype that has two or more instances in the repository.

Especially in large repositories with a complex project browser configuration and a large number of modellers the changes of duplicates are substantial. Therefore mostly a procedure is introduced based on the steps that a modeller has to do a repository search before he or she can actually add an element from the toolbox and modify it

Search

For searching for elements in the repository there is a search dialog availabe that opens by default in a simple common search that searches for a text in all the items. This works fine but has a number of limitations for an ArchiMate modeller:

  • The search is relatively slow
  • The modeller is not interested in the full content but only in ArchiMate elements of a specific type.

Luckily we can define our own searches in EA and an example of a ArchiMate specific search is shown bin the image below:

In this image you see that we can search for application components only and the list is by default sorted by objectname. This will help us in a later stage in user friendliness. The SQL statement is given below.

 

SELECT t_object.ea_guid AS CLASSGUID, t_object.name as ObjectName, t_object.Stereotype, ParentPkg.Name as ParentName, t_object.ModifiedDate, t_object.Author, t_object.[Version], t_object.[Alias], t_object.Object_Type as CLASSTYPE,t_object.Note as Notes

FROM t_object, t_package as ParentPkg

WHERE (t_object.name LIKE '#WC##WC#' OR '=')

AND t_object.stereotype IN('ArchiMate_ApplicationComponent')

AND t_object.Package_ID = ParentPkg.Package_ID

ORDER BY t_object.Name  

The trick is in two parts of the select statement:

  • The first one is that we filter on the stereotype of the element so we only see a limited number of elements based on that stereotype
  • The second is that we do a trick with the search term. We combine a search term search with a full search so you can see all the elements when the search term is empty

Model view

 

This helps us a lot in finding the relevant elements relatively fast however the search screen has a number of limitations in which selecting all the time the right query is relatively user unfriendly. However luckily Enterprise Architect has another screen for implementing this user friendly approach in a convenient way: the model view.

In the modelview you can define your own search view and link these views to a search in the search component of EA. See the picture below

In this screen you see a sample of a number of ArchiMate specific searches based on their stereotype. When you expand one of these searches you get the list of elements based on this stereotype. Since we have a combination of a keyword search and a full search (on the stereotype) the modeller can select the approach that best fits his needs. In the view you can select an element by typing in the first characters another option is to open the conext menu and go to the search function in EA and now we open the richt search immediately.

Download of archisearch

These searches can be extended and modified quite ease. However there are a number of files in the download zipfile included which you give you a jumpstart:

  • ArchiSearch.Xml can be uploaded via import in your search window in EA
  • ArchiView.Xml can be uploaded via import in your modelview window in EA

When you want to have more information on this search function or when you want to have assistance from us with the introduction of a specific search for your organisation, please feel free to contact eaxpertise.nl

 

Usage

Usage of this extension for deduplication has the following logical steps:Select in the project browser the package (and eventually subpackages) you want to deduplicate

  1. Right click the package and select extensions -> TenneT Browser Helper
  2. Validate your pacakges, this means that you get an overview of which elements in the repository are considered as duplicate (screen 1)
  3. Evaluate the results in your report or screen (screen 2)
  4. When you want you can check the content of this list and modify elements when you want
  5. Go to the duplicate tab (screen 2) and configure which child entities you want to merge and select in the right hand box the required functionalities
  6. Press the Deduplicate button and the merge is performed. 
  7. Go to the deduplicate folder and see which elements are merged to an original and are available for delete or archive
  8. Delete or archive the (empty) duplicate elements

Usage of the HTML publication has the following logical steps

  1. Select in the project browser the package (and eventually subpackages) you want to publish
  2. Configure the HTML generator for the
    • HTML file location
    • HTML template
    • Report name within the EA document generator
    • Coverpage name within the EA document generator
    • Generate the HTML site and use the result in your browser.

Formal aspects

This AddOn is developed by Eaxpertise for the IDEA community in the Netherlands. The source code can be downloaded here. The AddOn is available under the EUPL license

For support and adapations of the AddOn you can contact the participants in EAxpertise. However when you want to extend the product by yourselve please feel free to do so. You can find the Visual Studio project here. Sharing your adaptations in this AddOn is stimulated and will be published in the eaxpertise website.