data mapping Archives

Tag Archive for: data mapping

Automate Data Transformation with FlowForce Server

February 5, 2013/2 Comments/in Data Integration /by David McGahey

Altova designed FlowForce Server to provide comprehensive automation, management, and control over data transformations performed by dedicated high-speed servers. FlowForce Server Beta 3 is currently available to users of MapForce Enterprise and Professional Editions at no charge during the beta test period.FlowForce Server can provide hot folder automation of data mappings and maintains a detailed activity log users can monitor remotely in a Web browser window. The screenshot below shows the log for FlowForce Server running the MapForce data mapping CameraLogToGPX we wrote about in the blog post titled Process Multiple Input Files in a Single Data Mapping. This mapping used wildcards to specify multiple input files for processing.
FlowForce Server job log

It only takes a few minutes to set up, run, and review the results of jobs like this on FlowForce Server. Wildcards or Hot Folders?Wildcards and hot folders increase the complexity of a data transformation workflow, and using them successfully requires careful planning. Let’s take a minute to look a little deeper at the scenario we want to implement.Assume we are the IT department in a company that publishes nature and hiking guides. We employ photographers who go out trekking and record their routes as they go, using the GPS tracking feature of their digital cameras. We want to convert the camera GPS log files to XML-based .gpx format for mapping and other processing.We will publish a folder on our network where photographers can drop off their GPS log files. This will be the hot folder FlowForce Server watches for new files to supply as input to the CameraLogToGPX mapping. We only need to process each input file once. So, after the data transformation is complete, we can remove the input file from the hot folder. We also want to place the output file in a separate folder. This suggests the following FlowForce Server job steps:

Look in the hot folder to see if new a input files has arrived
Perform the data mapping on the input file and place the output file in a separate folder
Move the input file to a permanent location

The diagram below shows a folder structure we can use for the workflow, with files ready to drop into the hot folder for processing:

Hot folder structure

The hot folder is C:CameraGPShotFolder and the generated .gpx files will be placed in C:CameraGPSoutputFiles. When the data mapping is done, input files will be moved to C:CameraGPScompletedInput. Deploy the Mapping on a ServerMapForce Beta is a component of FlowForce Server that works just like MapForce, and adds a feature to deploy mapping files to a FlowForce Server. Existing data mappings need little or no special preparation for deployment.The only thing we need to think about are the filenames of the input and output files. We will instruct FlowForce Server to provide the input filename as a job parameter as new files arrive in the hot folder for processing. The original mapping assigned a wildcard filename for the input component, which is no longer needed. We will also want FlowForce to specify the location of the output file.We can open the mapping in MapForce Beta, remove the filename from the input component, and add a remove-folder filepath function to the output file so FlowForce Server can set the destination. The screenshot below shows the new filename definitions in the mapping.

Mapping file names

In the MapForce Beta dialog to deploy the mapping, we can choose to immediately open the mapping in a FlowForce Server job definition window in a Web browser to finish defining the job operations.

Deploy mapping dialog

Defining the Job in FlowForce ServerThe screenshot below shows the complete job steps defined in a FlowForce Server job properties window:

FlowForce Server job definition

The job trigger is defined at the bottom of the window. Every 30 seconds, FlowForce Server will check the hot folder. If the contents have changed, FlowForce Server will execute the job steps. Each Execution step could be a deployed MapForce mapping, a system step, or even another FlowForce Server job.The name of each new file entering the hot folder becomes the parameter called {triggerfile} that we use in the mapping step as the input filename, and in the move step as the name of the file to be moved.The Working-directory parameter in the mapping step defines where the output files will be placed.FlowForce Server also includes features to set automatic run and stop times for jobs, user permissions and roles, and Queue settings to define minimum time between job runs and maximum parallel instances of a job.In our scenario, we are likely to receive multiple input files in groups as they are copied from photographers’ memory cards. Multiple parallel runs can greatly improve throughput. As a rule of thumb, you might want to match the number of cores or CPUs in the machine running FlowForce Server.

FlowForce Server job queue settings

Exploring the Job Log

Each time FlowForce Server runs our job, six lines are added to the Log View shown in the illustration at the top of this post. The first and last lines record the start and completion of the job, and each Execution step generated its own start and completion messages. The phrase “completed with status: 0” means the step was successful with no errors.

FlowForce job log

We can click the more links for a detailed report on each Execution step. The screenshot below shows the message for the MapForce mapping Execution step:

FlowForce job detail

Each file dropped into the hot folder generates an individual FlowForce Server job instance, even if multiple files are added as a group. This makes it easy to track any individual input file that generates an error.When we dropped four files into the hot folder, FlowForce Server ran four jobs, and the contents of the output folder looked like this:

Job results folder

Click here to read more about FlowForce Server Beta 3 at the Altova Web site, or visit the FlowForce Server Beta 3 download page to get started automating data transformations in your data center!

Tags: data mapping, FlowForce Server, MapForce

Process Multiple Input Files in a Single Data Mapping

January 29, 2013/3 Comments/in Applications, Data Integration /by David McGahey

Typical data conversion tasks require processing numerous input files that arrive in batches. Altova MapForce includes features that let you handle groups of files with minimal intervention.For instance, we recently copied a set of files from the memory card of a digital camera with GPS support. Each .LOG file is a CSV containing GPS coordinates for a single route.

Directory list of source files for data conversion

We quickly designed a mapping to convert the CSV data to XML-based .gpx format and processed all three files to generate three output files in a single execution:

List of automatically generated output files

Tags: data mapping, MapForce

Expect the Unexpected – Altova MissionKit Solves a Number Format Mystery

January 10, 2013/in /by David McGahey

Every time you receive data from an outside source, there is a chance it won’t arrive in the form you expect. This can require special accommodations for the rare and unlikely to make a real-world data mapping and transformation solution robust and reliable.

We processed literally dozens of .gpx files, containing hundreds of coordinates each, through the MapForce mapping we wrote about in the blog post Web Service as a Look-Up Table to Refine GPS Data. Then one day we ran a new file and encountered the error below, which caused the mapping to fail:

Error message during mapping of Web services data

Reaching into the Altova MissionKit to combine features of MapForce and XMLSpy, we quickly diagnosed the issue and developed a solution we can also reuse in future mapping projects.

Tags: data mapping, MapForce, MissionKit, XMLSpy

Stored Procedures in Database Mappings

September 25, 2012/in Applications, Data Integration /by David McGahey

In response to user requests, the new release of MapForce 2013 includes robust support for stored procedures in database mappings as:

input components (for stored procedures that provide results)
output components (for stored procedures that insert or update data in a database)
or, stored procedures may be inserted as a function-like call (allowing users to provide input data, execute the stored procedure, and read/map the output data to other components)

The screenshot below shows the mapping of a stored procedure in SQL Server to create an XML file.

A MapForce database mapping with a stored procedure

Tags: data mapping, MapForce, v2013

Web Service as a Look-Up Table to Refine GPS Data

June 20, 2012/1 Comment/in Data Integration /by David McGahey

Elevation data recorded by GPS devices is notoriously inaccurate, especially in hilly terrain like the Russian River Valley example from our earlier post.

The final elevation track plotted from the Russian River Valley GPX file is suspicious for several reasons. First, the graph shows we descended almost 50 feet below sea level. That’s hard to believe, since we were travelling along the bank of the river, only about 10 miles from the Pacific Ocean.

Altova StyleVision Chart Secondly, we were headed mostly west, following the river downstream, but the track shows a predominantly uphill trend.

We can evaluate the recorded GPS elevation data by comparing it to information available from the United States Geological Survey (USGS). The USGS operates a Web service that accepts latitude and longitude coordinates, and returns elevation data measured by NASA and assessed for accuracy based on over 13,000 control points in the continental United States.

Using the elevation Web service in an Altova MapForce mapping will let us extract each point from the GPX file, send the coordinates to the USGS Web service, and build a new GPX file with corrected elevation data.

Intuitive User Interface Features

January 23, 2012/in /by David McGahey

Here at Altova we like to share user interface features across products to give the MissionKit a consistent look and feel, and to help users leverage experience gained using one tool to get up to speed more quickly with the others. In version 2012 we introduced new visual alignment guides in both UModel and MapForce to make it easy to work very rapidly, yet still produce a neat, organized result that communicates effectively with other team members. As users drag elements in the diagram window, alignment snap lines appear automatically to allow any component to align with any other component. In the UModel class diagram shown above, the developer has collapsed the properties and operations compartments to concentrate on class associations. As the CheckingAccount subclass is dragged upwards on the screen, a visual alignment guide appears and offers instant snap-to alignment with the other subclasses of the Account class. Complex data mappings in Altova MapForce might include dozens of components, functions and constants, and visual alignment guides can help organize the mapping view. In the screenshot below, the developer used alignment guides to collect split-name functions together, while concat functions are aligned in a separate group. Constructing the mapping diagram this way can greatly clarify the developer’s intent when multiple complex string manipulations are needed! Of course, aligned components also improve the value of printed diagrams that become part of the permanent documentation for UModel and MapForce projects. We also understand that not everyone likes to work the same way. If you prefer not to see guide lines and to turn off automatic alignment, this feature can be deselected in the Tools / Options menu selection in both UModel and MapForce. If you’d like to see for yourself how intuitively tools in the Altova MissionKit 2012 work together, click here to download a free trial!

Tags: data mapping, MapForce, UML, UModel, user interface, v2012

Announcing Altova FlowForce® Server Beta 1

December 15, 2011/in /by David McGahey

Altova FlowForce® Server Beta 1 is an exciting new tool for execution of automated data mappings designed to provide comprehensive management and control over data transformations performed by dedicated high-speed servers, virtual machines, or even regular workstations, depending on the size of the task. To gather user input and accelerate product development, Altova is offering Altova FlowForce Server Beta 1 as a free public beta test to all licensed users of Altova MapForce® 2012 Enterprise Edition and MapForce® 2012 Professional Edition. Users employ Altova MapForce data mappings for two different types of data transformations:

One-time data conversions
Data transformations that are repeated by date or time, when new source data is available, or based on some other external event

While command-line execution, royalty-free code generation, and the MapForce API can assist with automation of repeated transformations, FlowForce Server Beta 1 provides much greater power and flexibility. FlowForce Server Beta 1 is a server-based tool with a Web interface that makes it much easier to implement, manage, or modify data transformation jobs in a busy data processing environment. FlowForce Server Beta 1 can administer multiple transformation jobs simultaneously, lets users define and adjust a variety of job triggers and actions on the fly, can perform housekeeping tasks like moving output files or cleaning up intermediate work, records detailed logs of all activity, and much more. FlowForce Server Beta 1 consists of four components that work together as illustrated in the diagram below. The FlowForce Server continuously checks for trigger conditions, starts and monitors job execution, and writes detailed logs. MapForce Server is an implementation of the MapForce Built-in execution engine that executes mapping packages previously deployed via the MapForce graphical environment. The FlowForce Web Administration Interface is a standalone web application that runs in an internet browser and provides the front-end of FlowForce Server. MapForce Beta is an enhanced version of the Altova MapForce application with an integrated deployment feature to deploy MapForce data mappings to a FlowForce server package. Altova FlowForce Server Beta 1 is available immediately as a free public beta test to all licensed users of Altova MapForce® 2012 Enterprise Edition and MapForce® 2012 Professional Edition. Visit the FlowForce Server Beta 1 page at the Altova Web site for more information!

Tags: data integration, data mapping, FlowForce Server, MapForce

Processing the Groupon API – Epilogue

November 28, 2011/in Applications /by David McGahey

Rare edge cases can derail loosely coupled data mapping applications. This is especially true when you are consuming large datasets available over the Internet and have little or no influence over the source data. In this article we describe a debugging technique that lets developers working on data mapping and transformation projects quickly identify and accommodate unexpected data in a stream from a remote source. The Problem Last summer we wrote a series of blog posts describing how to work with the Groupon API to retrieve a subset of offers in all Groupon cities and format the list for a web browser or mobile device. We concluded with a command line to run a MapForce data mapping that calls the Groupon API over 150 times — once for each Groupon city, then filters the data to extract deals sold on the Internet instead of a physical location, and formats the results in HTML using StyleVision. Every morning we run the command line in a batch file that saves the HTML output on a local server so our colleagues can check it out with any Web browser to find interesting offers from all over the country. The mapping ran fine for more than two months until one day it failed with this error message: “Source-value “” of type dateTime could not be converted into target-type dateTime.” The specific explanation is that somewhere in the mapping where we expected a dateTime, we received an empty value. On a more abstract level, the error suggests a potential defect in the logic of our mapping strategy. Every time we call the Groupon API we receive a well-formed XML data stream enclosed in a <response> element, but the API specs do not include an XML Schema defining the data that may be returned. When we developed our mapping we needed to analyze the raw data and select the output we wanted, so our first step was to call the API to capture all the Groupon deals for one large metro area. We assumed we would get a large enough data sample to include every possible option in the API response. After our mapping ran successfully for two months, the API finally delivered a rare edge case that did not fit the pattern we expected. Debugging Tools MapForce provides debugging help. We can run our data mapping using the MapForce built in execution engine to see more details in the Messages window. The lines labeled Related location are hyperlinked back to components in the mapping where the error occurred. Clicking on the result error takes us to a format-dateTime function. We can either click the “” error or trace the value connector to identify the input element to the format-dateTime function. Either way, we locate the element that triggered the error. The suspect element resides in the input component that captures all the data returned by our calls to the Groupon API before any filtering or conversion takes place. When we designed the mapping, the endAt element in our sample data always reported the ending date and time for each Groupon offer, but for some reason we must have received an empty value in this field. If the error had occurred by running a local input file we could simply examine the file contents, but in this case the data came from multiple URLs, and is only held temporarily until it is mapped to the output component. Fortunately, we can apply a trick to easily modify the mapping and preserve all data received from the Groupon API. We simply copy the input component and paste a duplicate into the mapping. We can connect the response element from the original to the duplicate, which simultaneously maps all the child elements between the components. Our original input component is now connected to two output components. We can select which output component will be generated by the MapForce built-in execution engine by clicking the eye icon at the top right corner of any output component. The new output component simply saves a copy of everything in the input component. When we examine the raw data using XMLSpy, sure enough we find an empty element where we expected a date and time: The Solution Now that we know an offer might have no specific end time, we can plan for that possibility in the mapping. In the revised treatment of the endAt element, we do an if-test before the original format-dateTime function and provide an alternate output when the endAt element is empty. We had to work fast because all Groupon data is time sensitive. The edge case would eventually expire and disappear from the data stream. This experience showed us how important it is to have powerful debugging tools and to use them creatively, even after you think a data mapping project is running successfully! Altova MapForce is available in a free trial – the next edge case you solve could be your own. Editor’s Note: Our original series on mapping data from the Groupon API ran in three parts you can see by clicking the links here: Part 1 of Processing the Groupon API with Altova MapForce describes how to create dynamic input by collecting data from multiple URLs. Processing the Groupon API with MapForce – Part 2 describes how we filtered data from the API and defined the output to extract only the most interesting details. Processing the Groupon API – Part 3 describes formatting the output as a single HTML document optimized for desktop and mobile devices, and reviews ways to automate repeat execution.

Tags: data integration, data mapping, MapForce, StyleVision, XMLSpy

Analyze Football Statistics using the Altova MissionKit

November 22, 2011/1 Comment/in /by Niki Devgood

In this article we use stats from NFL.com and ESPN.com to show how easy it can be to process and analyze online data in new ways – even when it uses different metrics and is only available in textual format. We have seen in previous blog posts how easy it is to gather data from the Internet that is widely available in XML formats. But what about interesting data that is available online but not in an XML format, or data that is buried in legacy data processing systems and only available in textual report format? One such example involves quarterback ratings. The NFL has used a Passer Rating that rates quarterbacks solely based on a passer’s completions, attempts, touchdowns, and interceptions. ESPN introduced a new rating system this year called the Total QBR (Quarterback Rating). The Total QBR incorporates more data, including an expected points average and a clutch play index, that ESPN claims gives a more accurate measure of a quarterback’s performance. Let’s compare the rankings that these system produce to see if we can garner some useful information. For this example we’ll be using the data importing and analysis tools of the Altova MissionKit to compare the ratings. If you want to try this out yourself, the MissionKit is available to download for a 30 day free trial from the Altova web site. You can access the files used in this example here. The first thing we need is the raw data to analyze. Let’s use the entire 2010 season as a data source. We can get the table with Passer Ratings from NFL.com and then copy and paste it as a new text file. We can access a similar table of Total Quarterback Ratings from the ESPN web site and create a second text file. We now have two text files with tables of data in different orders. The next step is to combine the tables into one file and generate charts. First, we need a schema file for the destination of the data. In XMLSpy, we can create an XSD file quickly, and graphically, to contain a series of QB nodes with child nodes of first and last name, team, passer rating and rank, and total QBR and rank. Now, in MapForce, we open the text documents and use FlexText to parse the text and change it into a list of categories. We then build a mapping file in MapForce to map the data from the text files to the destination XML file. Built-in functions make it easy to extract the first and last names from the Player string, and a value-map will change the team abbreviation to a string (ARI is changed to Arizona Cardinals, ATL to Atlanta Falcons, etc.). We set the Priority Context in the test of our filters to make sure we get the correct set of data for each unique quarterback. Once we execute the mapping, we can save the resulting XML data file and use it as the source file in StyleVision to design a stylesheet. In this stylesheet, we create a table of the top ten ranked passers and charts showing the Passer Rating and the Total QBR graphically. Now that we have a visual representation of the rankings of the two rating systems, we can examine their differences and try to see which works better. For example, Peyton Manning was tenth in passer rating, but was second in Total QBR. This can be explained by the Total QBR taking clutch points into account and knowing that Peyton Manning had a few late game comebacks in the 2010 season. Since we now have a collection of files (the XSD file built in XMLSpy, the FlexText and mapping files from MapForce, and the stylesheet design created in StyleVision), we can update the text data files easily to analyze new sets of quarterback data. Later in the season, we can update the text tables with 2011 data, and allow the data to flow through the mappings and into the stylesheet to update the charts and see the rankings for the current season. This example focuses on numbers from the NFL, but this method can easily be adapted to other data sets and data sources that are accessed as text files as well as in other formats. You can learn more about how to use the products in the Altova MissionKit by taking our free online training courses.

Tags: charts, data mapping, FlexText, MapForce, XML charts

Mastering Paid Keywords

November 17, 2011/in /by David McGahey

Anyone who manages paid keyword search knows it is hard work! You can look at vast reports of raw statistics and quickly get lost in trivia. At Altova we designed a better way to analyze and manage the performance data for our Google Adwords campaigns. We can creatively query the numbers to: · Quickly aggregate results for subcategories of campaigns, for instance by product, geographical region, or any other grouping · Easily identify trends over time The chart below illustrates these advantages by collecting data for a single Altova product – SemanticWorks – from multiple campaigns over six individual months. Starting Out Like many keyword advertisers, we were viewing statistics in Adwords, downloading CSV files, then spending hours massaging and manipulating the data in spreadsheets to identify and format the information we required. We wanted more immediate and in-depth reporting of keyword performance while retaining full control of the process and managing everything internally. SQL queries of a database of keyword statistics offer a powerful and flexible alternative. In the remainder of this post we explain how the database design, data mapping, and reporting features of the Altova MissionKit can be applied to create an architecture to efficiently track paid keyword performance. Database Design Our choices were to implement a keywords database on an existing database platform already running in the company, an express edition of a commercial database, or an open-source database, since the Altova MissionKit works with SQL Server®, MySQL®, Oracle®, IBM DB2®, PostgreSQL®, Sybase®, and Microsoft® Access®. We chose SQL Server for our database platform. We connected with DatabaseSpy and used the graphical database Design Editor to create the table shown below. Most columns correspond to fields in a keywords report. In order to store multiple rows for each individual keyword – one row for every month of statistics – the table also includes columns for the month and year. Populating the Table The Google Adwords online interface lets users create reports of keyword statistics of specific date ranges and download them as CSV files. We downloaded individual CSV files containing our performance data for each unique month. We used MapForce to map values from the CSV files to columns in the database table and insert the month and year data for each row. The string functions at the bottom center of the mapping diagram remove percent signs and commas from fields we want to treat as numerical data. By doing this in the mapping, we don’t have to massage the columns of data in the CSV files before importing them. Since the CSV files for each month all have the same structure, the mapping needs only minor revisions to import each new month’s data: update the constants at the top that define the starting row id, month, and year. MapForce processes the mapping with its built-in execution engine, reading the CSV input and generating SQL INSERT statements for each row of data. MapForce then allows users to execute the entire generated SQL script by clicking a toolbar icon or from a selection in the Output menu: Querying the Database Back in DatabaseSpy, we can query the database from the SQL Editor window. This query reports the top ten performing keywords for SemanticWorks in October 2011. For data privacy, some fields in the Results chart are hidden. To get additional interesting results, the SQL statement can be easily modified. For instance, the ORDER BY line can sort for highest cost, most clicks, or any other characteristic. The WHERE statement combines data from multiple campaigns. The LIKE keyword treats the percent signs around SemanticWorks as wildcard characters to match any campaign with SemanticWorks anywhere in its name. Other queries could add a geographic identifier such as US or EU, or match on an entirely different column such as adgroup. Of course, all these options depend on a consistent and predictable campaign and adgroup naming system. We created a DatabaseSpy Project to collect all our favorite SQL queries for sharing and convenient reuse. Here is the query we used to generate the chart right in DatabaseSpy that appears at the top of this post: This query goes beyond simple SQL reporting to perform calculations on a subset of the data and format the results. Database Reports We designed reports for the executive team using Altova StyleVision, based on the queries and charts we had already designed in DatabaseSpy. We simply copied our queries from the DatabaseSpy SQL Editor window and added them as sources in the StyleVision Design Overview window. Saving our report design in a StyleVision SPS stylesheet makes it is easy to regenerate an updated version every month. Here is the HTML output for a SemanticWorks Keyword Trends report based on the query above, displayed in the StyleVision Preview window: If you follow the conventional wisdom for building your own paid keyword campaigns, you will develop segmented campaigns with many small, highly specialized ad groups, and you may also find yourself overwhelmed by the data in Adwords reports. If you’d like to try managing your own keywords the way we describe here, a fully functional trial of the Altova MissionKit is available.

Tags: data mapping, database charts, database reports, DatabaseSpy, MapForce, StyleVision

January 1 HIPAA Deadline Has Wide Impact

October 31, 2011/1 Comment/in /by David McGahey

Every day tens of thousands of encrypted data transactions occur between health care providers, pharmacies, insurers, medical billing services, and employers who provide worker health coverage. Everyone in the United States covered by health or dental insurance depends on the automation, accuracy, and security of these largely unnoticed data streams to verify eligibility and process payments for nearly all medical services. Definition of messaging standards for these transactions are mandated by HIPAA (Health Insurance Portability and Accountability Act), passed in 1992 and administered by the Department of Health and Human Services. By January 1, 2012, all health care enterprises covered under HIPAA are required to transfer data in version 5010, the latest HIPAA standard, based on ANSI X12.

Migrating Existing Data Formats To and From the Latest Standards

Altova MapForce is a powerful and flexible tool used by developers and IT professionals in many industries to automate today’s complex data conversions. The screenshot below shows a portion of a real-world example of a complex HIPAA data mapping in Altova MapForce from one state’s ANSI X12 eligibility file to integrate membership file data for public sector healthcare.

Altova MapForce 2012 enables visual mapping to and from all transaction sets that are required to comply with HIPAA regulations. If you are a developer working for an organization in the healthcare industry, you can map HIPAA messages to or from XML, databases, flat files, Excel 2007+, and Web services, then generate royalty-free code in Java, C#, or C++ for your data transformation to integrate internal healthcare data formats and the HIPAA standards. MapForce includes an example mapping from a HIPAA message to XML format, along with a sample data file and XML schema, so you can generate XML output using the MapForce built-in execution engine. The screen shot below shows the MapForce example mapping, including the pop-up help available by rolling the cursor over any HIPAA message element. MapForce allows users to apply mathematical, string, or logical functions, and other conversions, as data is transformed between the source and destination, illustrated by the parse-date function at the bottom center of the mapping image. Of course, privacy issues forbid distribution of an actual patient health care record, so the MapForce example HIPPA message content shown below is describing a fictitious person, but the message format is valid and the example is an effective demonstration of mapping functionality. When you click the Output button at the bottom of the mapping window, MapForce extracts the selected fields from the sample HIPAA message and transforms them to the XML version. For one-time data conversion needs, MapForce lets you save the results displayed in the Output window. MapForce also integrates with Altova StyleVision to apply style sheets that format data mapping output to HTML, RTF, PDF, or Microsoft Word formats.

HIPAA Message Formats Supported by MapForce

MapForce supports version 5010 for all the following key X12 transaction types included in the HIPAA January 1, 2012 mandate:

270-B1 — Health Care Eligibility Benefit Inquiry
271-B1 — Health Care Eligibility Benefit Response
276-A1 — Health Care Claim Status Request
277-A1 — Health Care Information Status Notification
277-B3 — Health Care Claim Acknowledgement
278-A1 — Health Care Services Review – Request for Review
278-A3 — Health Care Services Review – Response
820-A1 — Payroll Deducted and Other Group Premium Payment for Insurance Products
834-A1 — Benefit Enrollment and Maintenance
835-W1 — Health Care Claim Payment/Advice
837-Q1 — Health Care Claim – Professional
837-Q2 — Health Care Claim – Dental
837-Q3 — Health Care Claim – Institutional
999-A1 — Implementation Acknowledgement for Health Care Insurance

You can also use the example mapping to generate code and compile it to experiment with interfacing a data mapping application to your existing health care records infrastructure. HIPAA data mapping is only one application for the any-to-any graphical data mapping, conversion, and integration functionality of Altova MapForce. If you are interested in trying MapForce for your next data conversion project, you can click here to download a fully functional 30-day trial.

Tags: code generation, data integration, data mapping, MapForce

Processing the Groupon API – Part 3

August 23, 2011/in XPath /by David McGahey

Concluding the series in this post, we will apply a stylesheet to transform the XML data created from our mapping of the Groupon API into HTML. Here is an example of the XML output from the data mapping we created last time:

Assign a Stylesheet to Transform XML The Component Settings dialog for the output component of the MapForce mapping allows us to assign a stylesheet created with Altova StyleVision.

Assigning a stylesheet to the data mapping output component integrates the operations of MapForce and StyleVision, and a new series of buttons appears at the bottom of the MapForce mapping window for HTML, RTF, PDF, and Microsoft Word formats. (You must have both MapForce and StyleVision installed on your computer.) When you click any of these output format buttons, MapForce executes the data mapping exactly as we saw in the previous post. MapForce seamlessly passes the XML output to StyleVision, where it is transformed to the selected format. MapForce then displays the formatted document in the Output window. Here is the MapForce Output window for HTML, based on the StyleVision Power Stylesheet assigned above: The MapForce Output menu lets you save the XML data mapping output or the HTML document formatted according to the stylesheet. How to Make a Stylesheet We designed a stylesheet for the Groupon API data mapping using Altova StyleVision, based on the XML Schema for the MapForce output component. The intuitive StyleVision interface and powerful data access and manipulation features make it easy to create attractive documents in HTML, RTF, PDF, and Microsoft Word formats from XML files. The screenshot below shows the StyleVision Design View of the Extreme Groupon stylesheet. The blue numbered circles identify the location of each design feature listed following the image. Features of the SPS file

User-defined html specifies the viewport meta tag for mobile devices. This lets us design one stylesheet to generate a single HTML file for computers and mobile devices
Document title with customized font, size, and color
An autocalc element uses the XPath count function to count the number of deals in the XML input document and inserts the total
The date timestamp is placed at the top of the document even though the date element occurs at the end of the XML input data file
Images from the Altova and Groupon Web sites are referenced by hyperlinks, not as inline image data
Horizontal rules set off each individual deal. For HTML documents, the rules automatically fit the browser window width
Customized fonts and sizes assigned to different elements
A two-column table organizes each deal description
URLs in the XML file are dynamically assigned as hyperlinks for embedded images, fixed text, and dynamic data

StyleVision Power Stylesheets can combine multiple .xsd files, existing .css stylesheets, database schemas, XBRL taxonomies, and more to produce richly formatted reports that can even include automatically generated charts in various styles. You can also use StyleVision to define e-forms with data entry fields, drop down menus, radio buttons and other advanced features. Previewing Stylesheet Transformations StyleVision lets you assign a working XML file to preview your output as you design the stylesheet, and the buttons along the bottom of the Design window make it convenient to display the formatted working file as you refine your design. We saved the XML output of the MapForce mapping and assigned it as our working document. When the stylesheet was complete, the HTML Preview in StyleVision was identical to the MapForce HTML Output window shown above. To view the document on a mobile device you can either deploy the HTML as a page on a Web site or email it as an attachment. In addition to the stylesheet itself and formatted versions of the working document, StyleVision lets you save generated XSLT files to transform other XML files using your stylesheet design outside the StyleVision application. Automation Next Time In the future when you want to re-run a data mapping and refresh the HTML document with up to date data, there are two ways to automate the process:

You can run MapForce from a command line with parameters to name the mapping definition file and even call StyleVision to create formatted output
You can generate royalty-free code for the mapping in XSLT, Java, C++, or C# to combine with the XSLT code from StyleVision to build your own end-to-end application

XMLSpy, MapForce, and StyleVision are all available together in the specially priced Altova MissionKit. See for yourself how easy it is to use the MissionKit to convert data from a Web API — download a free 30-day trial!
Editor’s Note: Our original series on mapping data from the Groupon API ran in three parts you can see by clicking the links here: Part 1 of Processing the Groupon API with Altova MapForce describes how to create dynamic input by collecting data from multiple URLs. Processing the Groupon API with MapForce – Part 2 describes how we filtered data from the API and defined the output to extract only the most interesting details. Processing the Groupon API – Part 3 describes formatting the output as a single HTML document optimized for desktop and mobile devices, and reviews ways to automate repeat execution.

Tags: Altova XMLSpy, code generation, data integration, data mapping, MapForce, MissionKit, StyleVision, xpath, XSLT

Processing the Groupon API with MapForce – Part 2

August 16, 2011/6 Comments/in Data Integration /by David McGahey

In Part 1 of this series we described how to connect Altova MapForce to the Groupon API. We queried the API for a list of Groupon divisions, then used the list to create API queries for all the current deals from every division. In this part, we will execute the /deals queries and filter the response for the most interesting data. The list of /deals queries we built previously looks like this: To process all the queries, we can connect the list as a dynamic file input to a new mapping component. When we needed a new component last time, we dropped an API /divisions query into the mapping, and let MapForce create an XML Schema automatically. We could do the same thing here by dropping in an API /deals query as an XML input file. There’s just one small issue — although the Groupon API online documentation clearly describes the queries we can make, it is vague about the information that will be returned. Before we send dozens of queries to the API for all the current deals, we probably want to know a little more about the data that will come back.

Let’s Make a Deal Like Yogi Berra said, you can observe a lot just by looking. Let’s start by running a /deals query in XMLSpy. That will let us examine the response to a query for one division before we pull in a potentially unwieldy volume of data. The XMLSpy File / Open menu includes the same Switch to URL option we used in MapForce in the earlier post. If we enter the /deals API query for a division that covers a large metro area – say Dallas – we are likely to get enough deals instances to extrapolate the characteristics of the entire data set. XMLSpy opens the response to the /deals API query in Text view just as if we opened a local file: As expected, we got quite a bit of data when we requested all the deals for a single division! A fast way to analyze the structure of this data is to use the XMLSpy DTD / Schema menu option to generate an .xsd file from the xml. Shown below is a reduced view of the entire generated .xsd file based on the response to the /deals query for Dallas: We can dig even deeper, following Yogi’s advice like déjà vu all over again. Expanding all the elements to review the XML Schema reveals some curious anomalies. For instance, there are two elements named redemptionLocation with different definitions. The first contains a sequence of child elements: And the second is defined as a simple string: Going back into the xml data for Dallas and searching for redemptionLocation displays these examples: And: And: Now this is really interesting, because redemptionLocation = ”online” identifies deals that can be redeemed from anywhere, instead of by a visit to a bricks and mortar location in the division where they are advertised. What if we ran the /deals API queries for all divisions and extracted a list of all the online deals? That would be one extreme Groupon! Only Ask for What You Need The Groupon /deals API query supports an optional parameter called &show= that allows users to limit the data returned. Applying this parameter can save bandwidth and reduce processing time for the data transformation by removing unwanted data from the API response. We can also simplify our final result by including only the most interesting information, including the link to the Groupon web page for each deal. After we remove unwanted elements from the generated Dallas schema, our final version for the summary of online deals looks like this: When we add the &show= parameter to our MapForce mapping to request only the elements included in the simplified XML Schema, the queries look like this: Now we can drop the revised .xsd file into the mapping and connect the list of API /deals queries as dynamic input. We don’t need to delete the text file we used to collect the list of queries — that might continue to be helpful for future debugging. These changes complete the input side of the data mapping. Defining the Data Transformation Output Back in XMLSpy we can make a couple more revisions to the input XML Schema to design a new version for output: We discarded the response element since it doesn’t add any value, and eliminated the redemptionLocation element that we don’t intend to include in the output. We also added a date element for a timestamp, because our output file will be a snapshot of data that is constantly changing. After saving this version of the .xsd file in XMLSpy, we can drop it into the MapForce mapping. Shown below is the output side of the mapping with the output component partially connected. The filter at the top reads the redemptionLocation element to select only online deals and the now function inserts the date: The last revision we made in the output XML Schema was to change several element types from dateTime, Boolean, and integer to the string data type to allow more descriptive text Here is the complete definition of the mapping with the final connections to the output component: Now for the Payoff When we click the Output button MapForce processes the entire mapping from beginning to end using the MapForce Built-in execution engine. Here’s a breakdown of the steps:

Run the /divisions query to get the current list of divisions
Concatenate strings to build the list of /deals queries for all divisions
Run the /deals queries to create dynamic data for the input component
Filter for online deals to generate the output component, execute the remaining mapping functions, and add the timestamp after all the deals are processed

MapForce takes only a few seconds to complete all those steps and generate an output file with a series of deals that look like this: In part 3 of this series we’ll design a stylesheet to automatically transform the XML output of our mapping into html for attractive presentation in a web browser and on mobile devices. See ya at the ballpark, Yogi! XMLSpy and MapForce are available together in the specially priced Altova MissionKit. See for yourself how easy it is to use the MissionKit to convert data from a Web API — download a free 30-day trial!
Editor’s Note: Our original series on mapping data from the Groupon API ran in three parts you can see by clicking the links here: Part 1 of Processing the Groupon API with Altova MapForce describes how to create dynamic input by collecting data from multiple URLs. Processing the Groupon API with MapForce – Part 2 describes how we filtered data from the API and defined the output to extract only the most interesting details. Processing the Groupon API – Part 3 describes formatting the output as a single HTML document optimized for desktop and mobile devices, and reviews ways to automate repeat execution.

Tags: Altova XMLSpy, data integration, data mapping, MapForce, MissionKit, XML Editor, XML Schema, XMLSpy

Processing the Groupon API with Altova MapForce

August 2, 2011/2 Comments/in Data Integration, JSON /by David McGahey

We often think of a data integration project as a translation from one singular data input file to some other data set, but Altova MapForce lets you greatly expand the concept of an input file. For instance, the MergeMultipleFiles.mfd example installed with MapForce illustrates how you can use a filename with wildcard characters to merge multiple input files into a single output. A MapForce mapping input doesn’t even need to be a physical file – it can be a URL that returns predictable structured data, like the APIs for popular Web sites like Groupon and many others.

In this blog post we’ll describe how to use Altova tools to retrieve, filter, analyze, and present data available from a Web-based API, using Groupon as an example. If you want to follow along yourself, you will first need to visit http://www.groupon.com/pages/api to request your personal Groupon API client key. The Problem: All Deals Are Local The Groupon Web site and email subscriptions are great for finding deals in your local neighborhood, but what if you’re looking for a deal to use on an upcoming vacation, or for a gift for friends or family across the country? Sure, you could enter each location manually at the Groupon Web page, but that’s so last century. Let’s use the Altova MissionKit to automate things. The Groupon API offers two URL queries that return data in .json or .xml formats: the first returns a list of all Groupon localities (called divisions), and the second returns current deals information for one named division. If we want to see all the deals for more than one division, we need to resolve multiple URLs and aggregate the data into a single result. Yes, MapForce can do that! First We Need a Schema The Groupon API documentation describes the elements that will be returned by our requests, but doesn’t provide an XML Schema. That’s okay, we can use MapForce to generate one. All we have to do is open a new mapping design and choose Insert XML Schema/File, then click the Switch to URL button. Now we can enter the URL to retrieve the Groupon divisions list: When we click the Open button MapForce offers to generate the schema: When we click Yes, the File / Save dialog opens. I saved the schema as divisions.xsd, and the mapping with the new XML Schema inserted looks like this: And the Properties dialog for the XML Schema component automatically contains the API /divisions URL as the Input XML File: Check the Work We want to filter the Groupon divisions data to build a list of id names to use for deal queries for each locality. But before we go any further, now might be a good time to apply the text file trick from the Quick Solution for Complicated Functions blog post to look at the id values. When we insert the text file and connect the divisions and id schema elements, the mapping looks like this: We connected the division element to Rows in the text file in order to generate a new row in the text file for each unique division, so that Field1 in each row will hold the id. Clicking the Output button now generates this result: All we need to do is apply the concat string function to build the list of /deal URLs for all division IDs. The next step in the mapping looks like this: Rolling the cursor over the constant connected to value1 of the concat function displays its full definition: When we click the Output button to execute the mapping, the Output file now looks like this: As a further review, we can open the generated XML Schema in XMLSpy and display it in graphical Schema View: So far we have:

built a MapForce mapping that queries the Groupon API for all divisions
extracted the division id fields
and built a list of URLs for API queries to get the deals in each division

In the next post in this series we will process the list of deal queries as the input for a new mapping component and filter the output for some interesting information. Find out for yourself how easy it is to apply MapForce to convert data from a Web API! Download a free 30-day trial of MapForce.
Editor’s Note: Our original series on mapping data from the Groupon API ran in three parts you can see by clicking the links here: Part 1 of Processing the Groupon API with Altova MapForce describes how to create dynamic input by collecting data from multiple URLs. Processing the Groupon API with MapForce – Part 2 describes how we filtered data from the API and defined the output to extract only the most interesting details. Processing the Groupon API – Part 3 describes formatting the output as a single HTML document optimized for desktop and mobile devices, and reviews ways to automate repeat execution.

Tags: Altova XMLSpy, data integration, data mapping, MapForce, MissionKit, Web services, XML Schema

Use Built-In XPath Functions

July 12, 2011/in XPath /by Erin Cavanaugh

In developing one of the Altova Online Training courses, I sorted a list of books by the authors. I realized that my author field was a string of the author’s full name, so the books were sorted by the first letter of the string, or the author’s first name. It did not fit into the course to fix the sorting, but you can easily extract the last name from a string and use it for the sorting key using XPath functions. If you then use the books’ titles for a secondary sort key, you run into an issue with titles that start with “A”, “An”, or “The”. I want to use the title for the secondary sort key, but ignore a leading definite or indefinite article. Output the book list with a sort corrected using XPath expressions Let’s take a look at how we created this XSLT code.

This article was written using XMLSpy as the platform, but the same XPath expressions can be used inside MapForce or StyleVision to achieve similar results. We can start with a simple XML book list. We have 4 books with author and title nodes. An XSLT to create a list of the books would look like this: This will generate the following output: The books are output in the order they appear in the original data file. If we add xsl:sort to the xsl:for-each loop, we can arrange our output in other ways. This will generate a sorted list, but not sorted properly. Sorting author as a string, results in “Jules Verne” appearing ahead of “Mark Twain”. Also, “A Connecticut Yankee in King Arthur’s Court” appears ahead of “Adventures of Huckleberry Finn”. We want to ignore the indefinite article, “A”, so that “Adventures of Huckleberry Finn” appears ahead of “A Connecticut Yankee in King Arthur’s Court”. We can use XPath expressions to extract the sorting keys we want. Let’s examine the code before we look at the output. We replace “author” with “reverse(tokenize(author, ‘ ‘))[1]”. Tokenize breaks the author string into tokens using a single white space as the break point. So, “Jules Verne” is tokenized into “Jules” and “Verne”. Reverse reverses the order of the tokens to “Verne” and “Jules”. The one in square brackets chooses the first item in the list, “Verne”. This is the value that is used in for the xsl:sort function to arrange the books. This is not the perfect solution, but it works in our case. The title looks convoluted, but the logic is straightforward. The “tokenize(title,’ ‘)[1]” expression extracts the first word of the title. So, the first if test is “Is the first word of the title the word “A”? “. If it is, then we return the substring of the title that starts with its third letter, thus eliminating “A” and the space. If the first word of the title is not “A”, then we need to test it again to see if the first word of the title is “The”. If it is, we use the substring of the title starting with its fifth character, thus eliminating “The” and a space. If we fail both tests, then we just pass the title along as the sorting key. We could add another test to our code to see if the first word is “An”, but it is not needed for this data set. Executing this last XSLT, we get the following output. “Mark Twain” is now ahead of “Jules Verne”. “Adventures of Huckleberry Finn” appears ahead of “The Celebrated Jumping Frog of Calaveras County” and “A Connecticut Yankee in King Arthur’s Court”. The flaw in our approach to the author string is that we want “Jules Verne” to be treated as “Verne, Jules” for the sort, so that if we had a book by “Jimmy Verne”, the sort would treat them as different authors. Our code does not. Using “concat(reverse(tokenize(author, ‘ ‘))[1], reverse(tokenize(author, ‘ ‘))[2])” would sort “Jules Verne” and “Jimmy Verne” correctly, but this solution only will work with 2 word names. If an author had a suffix (“Martin Luther King, Jr.”) or multiple words (“George Herbert Walker Bush”), the code would fail. There are many exceptions to the general rules on alphabetizing names, and the code to allow for all variants goes far beyond the scope of this article. What we wanted to show was the ability to manipulate XML data on the fly using XPath expressions. We do not always have complete control on the format of our data sources, but using the power of XPath expressions, we can transform the data into the format that we need. A copy of the files used in these examples is available here.

Tags: Altova Online Training, Altova XMLSpy, data mapping, education, learn xml, XML Editor, XMLSpy, xpath, XSLT

A Quick Solution for Complicated Functions

June 27, 2011/in /by David McGahey

Some data conversions require multiple steps to satisfy today’s complex data communication requirements. Altova MapForce 2011 lets you graphically combine a wide variety of mathematical, logical, string, and other specialized functions to build complicated "data equations" to get the results you need. If you’re working on a big, complicated mapping, if you’re using an unfamiliar function for the first time, or when your function creates an intermediate result that needs further processing, it is helpful to test your work each step of the way as you build. You can use a simple text file as a temporary target to see the output of your function under construction. Let’s say we want to add a date and time stamp to an existing mapping of an output file to record the time the data was generated. We can easily experiment with the MapForce now function in a new mapping that simply connects now to a text file. Clicking the Output button at the bottom of the mapping window shows us the immediate result:

Okay, that works, but maybe the consumer of the output file would like to see a more elegantly formatted version. We can work with various datetime and string functions and build a complicated function until our mapping looks like this: Or we could simply use the format-dateTime conversion function with a string constant to define the format: As we work, we can press the Output button at any time, until we are satisfied with a result that looks like this: If you don’t work with dateTime data types every day, it’s hard to write a format string on your first try that generates the desired result! After several attempts, variations, and consultations with MapForce Help, here’s the format we developed and stored in the string constant connected to the format input of the format-dateTime conversion function: So far we used a text file that was defined as a basic CSV file with only a single field, but you can open the file Properties dialog and add fields or create a more customized output file. One advantage of using a text file as the output target is there is no type checking of the function result, so the text file target lets you examine and verify the output of any function. In the screenshot below we created a mathematical function and added a second field to the text file target to map the result. Now the output is a single-row, comma separated result like this: If our ultimate target is going to be a database field or an XML element with an integer data type, then we can immediately see this mathematical function doesn’t produce the expected datatype. When you have perfected your function in the test mapping, you can simply select, copy, and paste it directly into your larger mapping. You can then connect the output of the function to your ultimate target. In the screen shot below our format dateTime function is mapped to an element in an XML schema that is defined as a string. If this is a function you are likely to want again in the future, you can convert it to a User-Defined Function and save it in your library for convenient reuse. You can even use a text file as a temporary target to build and test a complicated function right inside your larger mapping. In that case, your mapping includes multiple possible outputs. The eyeball button at the top right of each output file selects which output file is displayed in the Output window. When you select the text file as the Output preview, the MapForce execution engine does not process the larger mapping, only the function connected to the text file. This can be a huge time-saver when you just want to examine the function result and your larger mapping processes a large volume of data, or if you don’t want to risk inserting invalid data into a database, and in many other situations. Find out for yourself how easy it is to apply MapForce to your own data mapping projects! Download a free 30-day trial of MapForce.

Tags: data integration, data mapping, MapForce, Version 2011

Switch Statement vs. Look-up Table in MapForce

May 17, 2011/6 Comments/in Data Integration /by David McGahey

One of the great things about working with software developers is you not only get to create new things that never existed before, you also get to see how other peoples’ minds work when they discover alternate solutions to any design challenge. We received a comment from a software developer on our recent post titled Expandable If-Else Works like a Switch Statement in MapForce regarding one of the examples we used. The reader suggests that our second example illustrated a problem that would be more elegantly solved in Altova MapForce with Value-Map than by our Expanded If-Else statement. Here was the original example that received the month as a string of characters and needed to generate the corresponding number: A Value-Map in MapForce is an alternate solution that functions as a look-up table, whereas an Expanded If-Else acts like a switch statement. Here is how our mapping would look with a Value-Map in place of the Expanded If-Else: Yep, that’s it. Rather than copying, pasting, and modifying sets of elements the way we built our original Expanded If-Else, a Value-Map lets us easily create the entire look-up table in its Properties dialog: We accept the commenter’s point — Value-Map definitely works better for the problem we chose because it’s much quicker and easier to create! The table from the Value-Map properties is also more concise and easier to interpret in MapForce-generated mapping documentation than our original Expanded If-Else structure. Of course you can’t always replace an Expanded If-Else statement with a Value-Map. Data entering the Value-Map must equal a single value in the input table to generate a specific output, whereas Expanded If-Else lets you set up a series of conditions with different logical tests. Sometimes the exact nature of a data conversion project makes it a judgment call to use a switch element vs. a look-up table. Let’s say your project receives input as a number that represents a wavelength of the electromagnetic spectrum and you want to handle ultraviolet, visible colors, and infrared energy individually. In that case we could use an Expanded If-Else to test for ranges of input values. The Expanded If-Else section of the mapping might look like this: If the input is an integer, you could also create a solution using Value-Map, but you would need to build a very long look-up table. And then what happens later if the project requirements change and the input becomes a decimal number, or you need to filter each visible color separately by name? Essentially Altova MapForce is a really cool graphical representation of a complete software language toolbox that insulates you from detailed programming language syntax, with a rich collection of components you can assemble creatively to solve your own data mapping, conversion, and integration challenges. Find out for yourself how easy it is to apply MapForce to your own data mapping projects. Download a free 30-day trial of MapForce.

Tags: Altova, Blog, data integration, data mapping, legacy data, MapForce, software tools

Expandable If-Else Works like a Switch Statement in MapForce

April 26, 2011/in Data Integration /by David McGahey

In response to several user requests, the if-else component in MapForce has been enhanced in MapForce 2011 Release 2 to accommodate any number of variables. This feature, which is equivalent to a switch statement in many programming languages, enables you to easily control the flow of data in a mapping project by matching a value to a selected criterion. For example, the screenshot below shows the if-then component used in combination with other components to analyze temperature data and produce the following results:

If temp is greater than 20, true is passed on to bool1 and "high" is passed on to value-true1.
If temp is less than 5, true is passed on to bool2 and "low" is passed on to value-true2.
Only one of the value-true parameters can be true for any given temperature, and that is the one that is passed on to the desc attribute in the target component.

The screenshot above is taken from the sample file Classify Temperatures.mfd included during the MapForce installation, so you can immediately process the sample data or modify the mapping to see the output for yourself. The new expandable if-else functionality helps developers build data mappings more rapidly, creates a mapping diagram that displays the conditional test more effectively, and facilitates testing and debugging because the expanded if-else function is much easier to read than a cascading structure of individual components. This is even more obvious when you have to evaluate a large number of conditions. The screenshot below shows a conversion from string to number where the string consists of a three-letter abbreviation of the month. (To save space here, this if-else function only processes strings for the first six months of the year.) In a copy-paste oversight we forgot to update the output constant for May from the value 4 to 5. That’s a much easier error to find and fix when you can see all the logic in one view!(UPDATE: We originally created this example to illustrate the advantages of Expanded If-Else over nested or cascading If-Else mappings that can quickly grow too large to view on a single screen. For a better design for solving this and other similar conversions using a look-up table, see the Switch Statement vs. Look-up Table in MapForce blog post.) Find out for yourself how easy it is to use expanded if-else functions in your own data mapping projects. Download a free 30-day trial of MapForce.

Tags: data mapping, MapForce, v2011r2

Harness the Power of Chained Transformations in MapForce

March 28, 2011/in XQuery /by David McGahey

Altova MapForce includes the ability to define chained transformations. Chained transformations let you create complex mappings where the output of one mapping becomes the input to another. In other words, two or more components can be directly connected to a final target component.

Hot off the Press!

March 17, 2011/in /by Niki Devgood

The industry is abuzz with the latest news announcing our release of the MissionKit Version 2011 Release 2. The release is loaded with new features for chart and report creation, enhanced data mapping capabilities, new XML Schema editing functionality, support for the latest version of BPMN, and a really cool new feature for comparing and merging Microsoft® Word documents. Dr Dobb’s and SQL Server magazine are just a few of the industry publications and blogs that covered the launch. Read what the industry is buzzing about and then download a free 30-day trial of the MissionKit and check out for yourself all the powerful new features now available in our suite of XML, database, and UML tools!

Tags: Altova XMLSpy, Blog, BPMN, data mapping, database charts, database reports, diff merge tool, DiffDog, file comparison, MapForce, MissionKit, reporting tool, software tools, SQL Server, UML, v2011r2, XML charts, XML Editor

New Software Release : MapForce 2011

September 27, 2010/2 Comments/in Software /by Niki Devgood

On September 8, Altova released v2011 of the MissionKit with powerful new features across the entire product line. MapForce has benefitted from the addition of advanced data integration functionality like chained transformations, integration with StyleVision for rendering transformation output, new formatting functions, and extended support for the UN/EDIFACT and ANSI X12 EDI formats. Let’s take a closer look at the exciting new features in MapForce 2011 including:

Chained transformations
Integration with StyleVision for output components
Formatting of numbers, dates, and times
Extended support for EDIFACT
Extended support for X12

Chained transformations Support for chained transformations lets you create complex mappings where the output of one mapping becomes the input of another. This is a powerful feature that adds to MapForce’s ability to execute fully automated transformations. Each chained component becomes a modular entity in an interdependent transformation sequence, allowing for conversions to be made on-the-fly. You can create chained transformations using any number of mapping components connected to a final target component. Preview and code generation features can be displayed/generated for intermediate components, as well as for the final mapping result. Intermediate mapping components also now have a "pass-through" button in the title bar that lets you define the set of data that gets passed on to the following component.

For example, the mapping above shows a chained mapping where the pass-through button is active and sample data is assigned to component A. The output preview offers two separate sets of data: XML that conforms to the Contacts.xsd schema and incorporates a subset of data mapped from the Employees.xml sample file, and XML that conforms to PersonList.xsd using data from the Employees sample file. In both cases, the data has been amended with the relevant constants and functions that are shown in the MapForce design pane. Integration with StyleVision for output components Support for Global Resources throughout the MissionKit has already offered you some integration between Altova software applications… But this feature takes that integration to a whole new level, allowing you to render XML and XBRL output from your data mappings into HTML, RTF, PDF, and Word 2007+ using an associated StyleVision template design. This essentially combines MapForce’s any-to-any data mapping capabilities with a sophisticated rendering engine, meaning that you can now automate report generation from virtually any data format that can be mapped to an XML Schema or XBRL taxonomy. In the example below, a StyleVision design has been associated with the mapping by simply right-clicking the output file and browsing to select a suitable SPS file using the Component Settings dialog. Once the SPS file is entered or selected, you will be able to simply click on the relevant format tab (i.e., HTML, RTF, PDF, or Word 2007+) to view rendered data. Formatting of numbers, dates, and times Probably one of the most requested features, this lets you easily convert numbers, dates, and times to their string representations using several different formatting functions. For example, the simple format-number function below converts a numerical input (e.g., 12345) to a monetary one in with a USD prefix, comma separating the thousands digits, and two decimal places (e.g., $12,345.00). This post really just scrapes the surface of what the new MapForce release has to offer -download a free 30-day trial of Mapforce today to experience all of these new features and more!

Tags: chained transformations, data integration, data mapping, MapForce, StyleVision, v2011, Version 2011

Updated Industry Schema Library

May 26, 2010/in XBRL /by Niki Devgood

“The nice thing about standards is that there are so many to choose from.” – Andrew S. Tanenbaum (attr.)

Maybe we can help. Altova’s updated schema library is a collection of over 100 industry and cross-industry XML Schema and DTD specifications – all in one central location. XML standards (and technology standards in general) are used to promote information sharing and interoperability across disparate software and systems. In a perfect world, this would translate to a global network of data being exchanged seamlessly between information partners… in a perfect world. However, there are some industries that are reaping the rewards of standardization, and hopefully these efforts will pave the way for more. Very recently we have seen an increased interest and even some actionable mandates in XBRL for financial data, HL7 for exchanging healthcare messages, and NIEM for inter-agency communication within the United States government. Of course, if you do find yourself working with some conflicting technology standards, you should probably take a look at Altova MapForce. With native support for visually mapping pretty much any data formats you can think of (XML, databases, flat files, EDI, Excel 2007+, XBRL, and Web services), easily adding data processing functions, and a whole bunch of automation options, you may even find that creating data integration solutions is well… kind of fun. Check out Altova’s Industry Schema Library, or download a free 30-day trial of MapForce today!

Tags: Altova, data mapping, EDI, HL7, NIEM, XBRL, XML Schema

Altova MapForce Brings Powerful Data Transformation Capabilities to Quark Dynamic Publishing Solution

April 28, 2010/in Data Integration /by Niki Devgood

This week Altova announced the integration of Altova MapForce, its conversion and data mapping tool, with Quark® Dynamic Publishing Solution. Through Automation Services™, a component of Quark’s dynamic publishing software, MapForce transformations translate XML into Web publishing formats, transform QuarkXPress® layouts for digital publishing, and convert XML to print pages, providing multi-channel publishers with a complete, automated XML-based publishing solution. This solution highlights two powerful features of MapForce that have demonstrated benefits to users with a wide range of data conversion needs: The MapForce visual data mapping interface supports integrating data as the source or destination in virtually any format, including XML, databases, flat files, EDI, Excel 2007+, XBRL, and Web services. MapForce includes the ability to automate data integration and repeatable business processes through an automation interface that lets developers access its advanced features programmatically. Users can also automate MapForce through the command line interface, or by using MapForce generated royalty-free code in XSLT 1.0/2.0, XQuery, Java, C++ or C#. “Having the ability to transform and publish data to multiple channels using an automated process helps users improve productivity and cut costs,” said Alexander Falk, President and CEO for Altova. The Altova Web site features a series of MapForce flash videos that each demonstrate a significant feature in 4 minutes or less. The Altova Online Training page offers three free MapForce self-paced training modules for Beginning, Intermediate, and Advanced data integration needs. Find out for yourself how convenient and cost-effective can be to solve your data conversion requirements. Click here to download a fully functional, free 30-day trial of Altova MapForce 2010!

Tags: Altova Online Training, C#, data mapping, demo videos, MapForce

HL7 and XML Healthcare Data Integration at HIMSS10

February 25, 2010/in Data Integration /by Niki Devgood

Altova is kicking off the 2010 trade show season with our first participation in the annual conference of the Healthcare Information Management Systems Society (HIMSS). The HIMSS10 conference will be held on March 1-4 at the Georgia World Congress Center in Atlanta. We’re looking forward to demonstrating the functionality in XMLSpy 2010 for editing, analyzing, and validating HL7 healthcare data. We will also show how MapForce 2010 can map older EDI-based legacy healthcare records to the latest XML-based HL7 v3.x standard. The example below shows a simple MapForce mapping to update an HL7 v2.6 message to v3.x. This will be our first opportunity to demonstrate the new Altova MissionKit 2010 Release 2 with significant speed and performance enhancements across the product line. Come see the new 64-bit versions for yourself, or download a free trial today. If you are one of the expected 30,000+ attendees traveling to Atlanta for HIMSS10, stop by and see us at booth 5744. Or click here to get more information on the Altova MissionKit and HL7.

Tags: Altova events, Altova XMLSpy, data mapping, EDI, HL7, MapForce, v2010

MapForce v2010r2 New Features

February 19, 2010/in Data Integration /by Niki Devgood

2010 has already been a very busy year at Altova, with our v2010r2 release now just three days old. Continuing on the Most Wanted theme of our v2010 launch, we have added several great new features to MapForce in response to customer requests (just in case you thought we weren’t listening) – but first let’s talk a bit about 64-bit versions. MapForce, and all of the other tools in the MissionKit, are now available in 64-bit versions for the same price as the 32-bit downloads (SMP subscribers can use their current keycodes for 64-bit versions). Customers running 64-bit operating systems will now be able to take full advantage of much faster processing speeds for integrating large files (100 MB+) in MapForce. For the developers, we have also added support for external C# and Java calls in mapping transformations, so that you can get an accurate preview of your mapping in the output window. We’ve also developed a new process for adding C# and Java libraries – by simply selecting the .NET assemblies of Java class files to import. For EDI specialists, we’ve added automatic generation of X12 997 functional acknowledgement messages to confirm message receipt or relay transaction errors. And… for everyone else, we’ve added three new string functions: tokenize, tokenize-regexp, and tokenize-by-length. As you can see, there’s a little something for everyone here, so download a free trial of MapForce today to check it out – or, if you have SMP, take advantage of your free upgrade!

Tags: 64-bit, Altova, altova product upgrades, data mapping, MapForce

MapForce v2010 – “Most Wanted”

November 10, 2009/2 Comments/in Technology /by Niki Devgood

As a frequent attendee at the Altova booth at tradeshows, I have to say that this v2010 "Most Wanted" release has been one of my favorites since I’ve been with the company. Rather than centering the release around a marquee technology like we have in the past with XBRL, OOXML, etc., this time we have added to the MissionKit a collection of over 70 (not a typo) of the features that our customers have requested the most. Many of these requests have come directly from the tradeshow floor, and some I even recall scribbling down myself. Others have come from through our Support Center, and still more from Altova’s online user forums. So keep those requests coming, and we’ll keep on listening! In this post I’ll outline a few of the new features added to MapForce below, and be sure to look out for our future posts where we will spotlight individual features in more detail. Processing data from/into multiple files MapForce users have always been able to map data explicitly to and from many different components at the same time. This feature takes that ability much further, letting you implicitly process files, for example to/from a file collection or directory using a variety of different methods including wildcard values, database tables, auto-number sequences, and more. For example, the screenshot below shows files from a directory being mapped into a single target file using a wildcard (?) value. The output file generated from this mapping can be saved to any location from the Output Preview window. If you’d rather separate the output results into two separate XML files, you can just add a connection between the two file items at the top of each mapping component. Using file names as parameters As a complement to this functionality, you can now use file names as parameters in your mappings – an extremely useful feature for real-time transformations when this information may not be known until run time. In the example below, this is accomplished using an input parameter and connecting it to the file item node in the source mapping component. Support for WSDL 2.0 Like XMLSpy, MapForce v2010 has added support for Web services based on WSDL 2.0 in addition to WSDL 1.1. When you are building or connecting to Web services, MapForce automatically recognizes the syntax of WSDL 2.0 documents and applies appropriate processing rules. This feature gives MapForce users the flexibility to work with either version of the W3C format. A number of other features have been added to the "Most Wanted" release of MapForce including: · Support for xsi:type in XML Schema · EDI file validation in generated code · Support for additional EDIFACT messages So… be sure to download a free trial of MapForce v2010 – or if you have active SMP, download your upgrade today!

Tags: data mapping, MapForce, v2010, WSDL 2.0

HubKey Case Study

September 28, 2009/in Hardware /by Niki Devgood

Overview HubKey is a technology company offering e-commerce solutions and services to small and mid-sized organizations. Their flagship product, ILXA, is a hosted application that uses the document and workflow management capabilities of Microsoft Sharepoint, combined with the power and flexibility of Altova MapForce and XMLSpy, to deliver a scalable, end-to-end, business-to-business (B2B) solution for outsourced EDI. ILXA builds an intuitive user interface and superior content management controls and functionality around e-commerce/EDI data sources, giving customers the freedom to quickly and accurately process electronic transactions without the need for costly software and hardware systems. The Challenge The HubKey team wanted to build a comprehensive EDI management and translation system that would give their clients the ability to track their EDI transactions across a customized workflow and also transform the messages into virtually any other data format. EDI systems are required to handle a large and constant flow of transactions going back and forth between trading partners. While the actual volume of the data being transmitted is often very small, the amount of individual communications can be overwhelming. HubKey ILXA contains the chaos of EDI automation by giving organizations the ability to view and manage tasks and processes in batches or on-the-fly. Recognizing an increasing demand for B2B integration systems that comply with both EDI and XML, HubKey decided to build a platform that had powerful support for both data formats and could generate application code to automate these translations. Complex EDI-XML and XML-EDI mappings would run behind-the-scenes, but users would be able to access these mappings, as well as the raw data, for quality assurance and error fixing. The Solution HubKey ILXA is a hosted e-commerce solution that gives non-technical end users the ability to seamlessly manage their EDI transactions without being exposed to complex data syntax. To easily manage the document workflow, HubKey chose to build their system on top of the Sharepoint platform, creating a customizable .NET application with advanced functionality for a collaborative workflow environment. The ILXA system draws upon the versatility and quality control capabilities of XML, with EDI-XML conversion powered by Altova MapForce. The end result is sophisticated translation software combined with validation and workflow management capabilities, all in one easy-to-use system.

Translator – performs virtually any to any document translation for integration with backend ERP and accounting systems or trading partner requirements
Tasks Engine – gives users a Web-based interface through which to view and orchestrate document management tasks
Business Rules Engine – handles data validation, alerts, substitutions, and more through generic or specific processing rules devised by the user

ILXA is delivered to customers in the form of a personalized, secure Web portal. Connections are easily set up between backend accounting/ERP systems (including technologies from Microsoft, Sage Software®, Exact®, and Intuit®) and member trading partners for sending and receiving messages via virtually any connectivity option (AS2, VAN, HTTP, etc.). The modularity of the ILXA system, combined with the experienced solution providers at HubKey, work together to make setting up the system a seamless process that can be implemented and up and running within 24 hours.

Translator Using the data transformation and code generation capabilities of MapForce, ILXA translates raw EDI data into XML based on generic XML Schemas (developed using Altova XMLSpy) stored on the system. Once in XML, the data becomes much more usable within the Sharepoint environment, enabling users to work easily with messages and respond to tasks. Non-technical users can create workflows, assign tasks, and send and receive messages within the ILXA interface without even seeing a line of EDI or XML code. However, if changes or adjustments need to be made, ILXA customers are given the freedom to apply these in-house using the data maps saved in the SharePoint document library. MapForce’s intuitive graphical interface enables users to redraw connections, add functions from the function library, and apply filters to the mappings. Any saved changes cause compiled code to be saved back to the system and will affect ensuing automated transformations. MapForce generates program code in Java, C++, and C#. HubKey opted to have code generated in C# to make it easily consumable by the Sharepoint platform. Once the data has been translated, users can even launch XMLSpy to view and make changes directly in the XML. The generated XML displays the original EDI format in the file for an easy comparison. Using MapForce, ILXA can also be easily configured to convert data into other EDI specifications, databases, flat files, Microsoft Excel 2007, and Web services. Tasks Engine The ILXA Tasks Engine enables users to manage advanced workflows, as well as track and resolve processing and validation errors that occur during the translation process. Users can view all of the documents in their workflow and take actions based on their status. Once an issue has been resolved, the document translation automatically resumes. The ILXA Tasks Engine builds upon the advanced capabilities of Sharepoint to make a document management and publishing system that is ideal as an EDI/e-commerce solution. Users can easily assign, resolve, and review tasks in a secure setting based on assigned permissions and defined roles. The ability to streamline these vital business processes in one central application helps increase the quality and consistency of error-prone large-scale message translation and transmission. Business Rules Engine HubKey’s patent-pending Business Rules Engine provides customers with a powerful tool for implementing specific rules based on documents, trading partners, and/or date/time parameters. This gives organizations the ability to apply filters to transactions based on customized definitions and constraints that apply to a particular situation. The HubKey team offers its customers the option to have their business rules set up and implemented, or the training to do so in-house.

The Results ILXA breaks down the barriers to costly EDI implementation, giving organizations an affordable, flexible, and reliable alternative to fully outsourced solutions through a modern, Web-enabled, component-based application. By combining content management functionality with age-old e-commerce business process requirements, HubKey is able to offer its customers a centralized EDI management application with resources and personalized services customized to meet any level of e-commerce data expertise. Altova MapForce and XMLSpy provide the translation and XML structure behind-the-scenes, but are also available to more technical users to make adjustments and confirmations at the source. This gives HubKey the ability to offer a flexible and changeable solution to their end users, giving them the power to decide upon hands-on EDI management, or an assisted solution that still falls within their budget. Find out how MapForce and XMLSpy can add functionality to your business applications. Download a fully functional free trial of the Altova MissionKit today!

Tags: Altova, Altova XMLSpy, Case Study, data mapping, EDI, MapForce

Adding z-segments to HL7 Mapping Components

July 20, 2009/in /by Niki Devgood

When mapping HL7 EDI components, it is often necessary to add locally-defined information, or z-segments, to accommodate additional fields not included in the standard. Following is a simple walkthrough that will demonstrate how to add z-segments to the HL7 configuration files that are available as a free download with MapForce. In the example below we will be adding a ZLR segment to an HL7 2.3 Observation Results Unsolicited (ORU) message. The ZLR segment is commonly used for adding additional information for legacy laboratory-based reporting. ZLR attributes are provided below:

SEQ	LENGTH	OPT	DATA TYPE	ELEMENT NAME
1	106	true	XAD	Ordering Provider’s Address
2	90	true	XON	Ordering Facility Name
3	106	true	XAD	Ordering Facility Address
4	40	true	XTN	Ordering Facility Phone
5	20	true	SN	Patient’s Age
6	40	true	XPN	Next of Kin/Associated Party Name
7	40	true	CE	Next of Kin/Associated Party Relationship
8	106	true	XAD	Next of Kin/Associated Party Address
9	40	true	XTN	Next of Kin/Associated Party Phone

The ZLR segment must follow each OBR (Observation Request) segment, and there can only be one ZLR per OBR. 1. Go to C:Program FilesAltovaMapForce2009MapForceEDIHL7.v230 to access the MapForce configuration files for HL7 version 2.3. 2. First, locate the message configuration file in question, ORU_R01, and open it in XMLSpy – or any text editor.[i] Add a ZLR just below OBR. 3. Save this file as ORU_R01_ZLR (or any unique name you choose). 4. Now open the EDI collection file and add the new message to the list.[i] 5. Next, simply open the HL7 SEGMENT file to add the segment details to the GENERATE DATA section as provided above.[i] 6. Finally, scroll down to the GENERATE SEGMENTS section and add the following: 7. Now, let’s access our newly customized HL7 EDI mapping component in MapForce. Open MapForce and choose Insert > EDI. In the Browse EDI collections dialog, select HL7.v230 and scroll down to select ORU_R01_ZLR. Press OK to insert. 8. Your new mapping component will appear in the MapForce design pane with the new ZLR segment included. Now you can complete your data integration design by inserting another source or target data structure(s) and dragging lines to connect nodes. MapForce supports mapping to/from XML, databases, flat files, EDI, XBRL, and Web services. [i] If you are working in XP, you will have to unclick “read-only” in the Properties dialog. Vista users will need to copy the file to another location before editing – you can then copy the file back to the appropriate HL7 collection directory. For more information about mapping HL7 and other EDI formats, please see the MapForce feature pages – or download a 30-day free trial of MapForce today!

Tags: Altova, data mapping, HL7, MapForce

XBRL… so much more than compliance

July 13, 2009/3 Comments/in XBRL /by Niki Devgood

Having recently returned from a short visit to the 19th XBRL International Conference in Paris, I can’t help but think that many organizations are simply missing the point – and that perhaps the SEC mandate is partly to blame for this. One would think (well, I thought, anyway) that in the year following the issuance of XBRL reporting requirements by the world’s largest economy, that this conference would be overflowing with company representatives eager to learn more about how, and best of all, why they should mark up their financial data in XBRL. But alas, this was not the case. I can only guess that the meager attendee numbers – especially from the United States – have to do with the fact that organizations are still viewing XBRL as singularly a compliance issue and are continuing to outsource the “tagging” of their financial statements to financial printers or other EDGAR filing entities. So, is XBRL a compliance issue? Well, of course it is, but it is much more than that. I can tell you this for certain because I work for Altova and we simply do not focus on compliance software. We build tools that help businesses to maximize the efficiency of their internal processes with an eye toward reducing the overall time and cost of the data management workflow. And this is well within the realm of possibility for any company using XBRL – but it means taking a proactive look at the way you manage your data. “Tagging” implies that financial statements are drawn up in the traditional manner in a spreadsheet or accounting program and then retroactively and meticulously marked up with XBRL tags to make them compliant. Ugh… no wonder compliance has such an ugly ring to it! Haven’t we all got enough work to do? And wait, isn’t this just adding one manual task on top of another – doubling the chances of human error? I’m not sure exactly when this word “tagging” became so popular for describing XBRL implementation, but all it has done is succeeded in oversimplifying something that was not very complicated in the first place (admittedly, it was probably coined by someone in the marketing tribe – of which I’m a member). Anyway, let’s put this idea of tagging aside and see if we can come up with something a little more dangerous. Let’s say that all of your company financial data resides in some sort of backend repository, a database, accounting/ERP system, XML, or even some combination of these. What if you could simply map your data to XBRL in-house instead of having external consultants comb through reports and tag each line item? What if you could even reuse this mapping the next time you had to produce a similar financial report? And what if you could even have your IT department automate your XBRL filing processes?

XBRL Mapping Altova MapForce is an enterprise-level data integration tool that lets you do just this. It is used at a high level by developers and application architects, but its easy-to-use graphical interface makes MapForce accessible to anyone with an understanding of the data that needs to be mapped. Let’s look at a partial example to illustrate how easy this can be: The first step is to load insert the source data component – in this case a database – into the MapForce design pane. Note that the mapping component is a representation of the tables and columns in the database, the underlying data can, therefore, change at any time and the mapping itself will not be affected. The same is true for any mapping structure that you use in MapForce – XML, databases, flat files, EDI, Excel, XBRL, or Web services. Next, we’ll add our target mapping component, in this case a basic XBRL extension taxonomy built on top of US GAAP – Commercial and Industrial. Now, we can simply begin the mapping by connecting lines to associate line items. There will be some cases when we need to apply data processing rules to slightly modify the format, filter data, or even add constants for XBRL reporting requirements that do not exist in the database. All of this is very easily done by dragging and dropping intermediary functions from the MapForce library in the sidebar. Let’s say, for example, that your database automatically requires a datetime format to record any accounting period. Since XBRL reporting only requires a date, we need to strip the time out in our mapping. So, simply drag a date-from-datetime function from the library and connect the lines between your database and XBRL component. Of course, you’ll probably also need to add a variety of other math, logical, or other types of functions to your data, and you will find a long list of these already available in the function library.

You can also easily add custom functions, if needed, using a graphical function builder. In the end, your mapping will look something like this: Now just hit the Output tab to see what the XBRL looks like. And there you go… a reusable, extensible data mapping that you can run any time you need to submit XBRL data. You can even integrate the mapping interface into another application, or ask IT to generate code that will automate the XBRL file generation each time a report is due. For a more detailed overview of how XBRL mapping works in MapForce, check our Altova’s XBRL tutorial.

So, here we have a very quick example of generating XBRL directly from an accounting system – no need for re-keying information, no need to create a set of traditional financial statements, and certainly no need for “tagging”. And best of all, all of this can easily be done in-house and at a fraction of the cost. Now don’t get me wrong, outsourcing could very well have a place in your company’s XBRL implementation. Building an XBRL extension taxonomy, for example, could very well be something that you feel more comfortable leaving to those who have years of experience working with XBRL syntax and other complexities. But putting your organization’s financial data into XBRL… shouldn’t that be left to those who know the data best? For more information on the Altova MissionKit tools for XBRL – which includes support for XBRL mapping and automation, XBRL validation and taxonomy creation, and XBRL rendering – please visit https://www.altova.com/solutions/xbrl-tools.html …or download the Altova MissionKit today!

Tags: Altova, data mapping, MapForce, XBRL

SOA and Cloud Services Within Your Budget

March 26, 2009/in Hardware /by Niki Devgood

The hardships affecting today’s economy present new challenges for organizations. Interdepartmental budgets are being cut, and large purchases are being carefully scrutinized. Costly enterprise software and mainframe computing systems that once held promise are being reconsidered on a global scale in favor of more agile, component-based systems that cut costs and increase efficiency with forward-thinking concepts like Service-oriented Architecture (SOA) and cloud computing. These architectural concepts incorporate modern technologies and object-oriented approaches to solve real-world technology issues in complex environments while decreasing maintenance, integration, and deployment costs with modular design and component re-use. The Altova MissionKit is a highly affordable toolset uniquely suited to address this shift toward more flexible and lightweight infrastructure. With strong support for XML, UML, databases, and data integration technologies, the MissionKit offers all of the tools necessary to build agile architectures replete with repeatable services, reusable components, and scalable resources.

SOA & Cloud Computing

SOA and Web/cloud services are two of the strongest buzzwords in technology today. Though they have some clear differences, both of these concepts represent a paradigm shift from large-scale enterprise systems to service-based architectures built on modular components and reusable functionality. The SOA approach aims to help organizations respond more quickly to business requirements by packaging processes as a network of interoperable and repeatable services. This modularity creates system flexibility and gives developers the agility required to build new capabilities into the current system as needed – without reinventing the proverbial wheel. SOA is essentially a series of interconnected and self-contained services, the functionality of which is dynamically located and invoked based on certain criteria, communicated in messages. At the heart of SOA is a high level of component reuse that drives down costs and increases efficiency in a fully scalable architecture. Cloud services build upon the concept of interoperable services, adding a virtualization component to help relieve internal servers from being overtaxed by the constant reuse of these services within the system. This paradigm uses the Internet and Internet-enabled technologies to increase performance and processing speed by storing information permanently in the "cloud" and caching it only temporarily on client machines. Cloud computing implementation is a powerful option for increasing system capacity and capabilities by leveraging next-generation data centers in combination with the World Wide Web. Both SOA and cloud computing seek to alleviate problems created by inflexible architectures that rely heavily on tightly coupled enterprise application infrastructure. This focus on interoperability and independent software services reveals a distributed solution that is event-driven, flexible, and cost conscious in almost any setting.

Anatomy of a Service-based Architecture

Since their inception, XML and Web services have been continuously gaining notoriety as the standards of choice for secure, efficient, and platform-independent data exchange between software applications and over the Internet. XML provides the foundation for the protocols that power Web services infrastructure: WSDL (Web Services Description Language) and SOAP, an XML-based messaging standard. Web services are hardware, programming language, and operating system independent, meaning that they are duly amenable to the seamless and interoperable exchange of data over a network and uniquely suited to component-based systems. Web services architecture Both SOA and cloud-based architectures generally rely on WSDL to describe interaction and functionality and locate operating components within the system. WSDL works hand-in-hand with SOAP, a messaging protocol used by the client application to invoke the methods and functions defined in the WSDL description. The example below is the stock quote example used in the W3C WSDL specification and describes a simple, single operation service that retrieves real-time stock prices based on ticker symbol input. Of course, most services that exist within enterprise architectures are far more complex. Graphical WSDL editor Take, for example, the publicly available Amazon Web services, which provide accessible Cloud services and infrastructure to a growing number of companies worldwide, including Twitter, SmugMug, and WordPress.com. These services essentially allow independent organizations to rent some of the immense power built into the Amazon distributed computing environment and add the same scalability, reliability, and scalability to their online presence at a fraction of the price. The much anticipated Windows Azure from Microsoft® operates on a similar model, giving developers the opportunity to build and deploy cloud-based applications with minimal on-site resources. Amazon provides a WSDL file that contains the definition of the Web service, the requests that the service accepts, and so on. Developers can then write a SOAP-based client application that invokes the Amazon Web service for the functionality it provides. (At this time Amazon provides a number of Cloud-based services for application hosting, backup and storage, content delivery, e-commerce, search, and high-performance computing.)

Altova MissionKit

Recently named "Best Development Environment" in the Jolt Product Excellence Awards, the Altova MissionKit is a diverse set of software tools that provides scalable options for leveraging your current software assets in an SOA or cloud-enabled environment. Strong support for XML, Web services, data integration, process automation, and databases, as well as accessibility to powerful APIs give developers flexible options for creating service-based solutions and an affordable alternative to costly consultant fees, extract/transform/load (ETL) tools, and/or enterprise service bus (ESB) products. The Altova MissionKit* supports end-to-end Web services development and includes a graphical WSDL editor, visual Web services builder, advanced capabilities for managing WSDL and other XML file relationships, a SOAP client and debugger, WSDL data integration, code generation, and more. Together, all of these features provide a robust solution for integrating disparate services and systems in a distributed computing environment, whether the components be in-house, network, or Cloud-based.

WSDL Editor

The XMLSpy XML editor provides a graphical interface (GUI) for designing and editing WSDL documents. The structure and components of the WSDL are created in the main design window using graphical design mechanisms (with tabs allowing users to toggle back and forth between text view), and additional editing capabilities are enabled from comprehensive entry helper windows. Users can easily create and edit messages, types, operations, portTypes, bindings, etc., inline. In addition, publicly maintained WSDL files like the Amazon Simple Storage Service, or Amazon S3, (below) can be opened instantly using the Open URL command in XMLSpy. WSDL editor Amazon Web services XMLSpy’s WSDL editor gives developers a sophisticated environment for rapid Web services development, managing WSDL syntax and validation through an intuitive, drag and drop graphical interface. The addition of a documentation generation feature makes it possible to share the complete details of a Web service interface with non-technical stakeholders in HTML or Microsoft Word.

SOAP Client

SOAP requests can be manually created in XMLSpy’s SOAP client based on the operations defined in the WSDL. Once an operation is selected, XMLSpy initiates the request based on the connections provided in the WSDL and displays the XML syntax of the SOAP envelope in the main window. The message can then be sent directly to the server for an immediate response. SOAP client for Web services

SOAP Debugger

XMLSpy also includes a SOAP debugger, which acts as Web services proxy between client and server, enabling developers to analyze WSDL files and their SOAP message components, single-step through transactions, set breakpoints on SOAP functions, and even define conditional breakpoints that are triggered by a stated XPath query. SOAP debugger

Building Web Services

Once a WSDL definition is complete, it can also be visually implemented using MapForce, Altova’s any-to-any data integration tool. MapForce gives users the ability to map data to or from WSDL operations and then autogenerate program code in Java or C#. Tight integration with Visual Studio and Eclipse makes it possible to then compile the code within either of these IDEs and deploy the service on the client machine. When you create a new Web service project by specifying a Web services definition file (WSDL), MapForce automatically generates mapping files for each individual SOAP operation. MapForce project The SOAP input and output messages can then be easily mapped to other source data components (XML, databases, flat files, EDI, XBRL, Excel 2007) to create a complete Web services operation. Data processing functions, filters, and constants can also be inserted to convert the data on the fly. Web services mapping MapForce can autogenerate Web services implementation code in Java or C# for server-side implementation, and it is also accessible for automation via the command line.

File Relationship Management

For complex Web-based applications that include a large number of disparate files and project stakeholders, the MissionKit offers an advanced graphical XML file relationship management tool in SchemaAgent. SchemaAgent can analyze and manage relationships among XML Schemas, XML instance documents (SOAP), WSDL, and XSLT files. The client/server option enables any changes to be visualized in real time across a workgroup. Managing XML files This gives organizations the ability to track and manage their mission critical SOA files as reusable individual components, reducing development time and the occurrence of errors.

Data Integration

A key factor of any SOA is the ability for disparate systems to communicate seamlessly via automated processes. As an any-to-any graphical data integration and Web services implementation tool, MapForce facilitates this undertaking with support for a wide variety of data formats including XML, databases, flat files (which can be easily parsed for integration with legacy systems with the help of the unique FlexText™ utility), EDI, XBRL, Excel 2007, and Web services. MapForce data mapping in Visual Studio MapForce supports complex data mapping scenarios with multiple sources and targets and advanced data processing functions. Transformations can easily be automated via code generation in C#, C++, or Java, or the command line. Full integration with Visual Studio and Eclipse also makes this an ideal development tool for working in large-scale enterprise projects – without the heavy price tag. This gives developers a flexible and agile middleware component that can work in virtually any service-based architecture. The ability to integrate disparate data in on-the-fly is a key requirement in real-world enterprise and cross-enterprise systems where legacy systems and other less flexible formats co-exist with XML and other modern, interoperable standards.

Database Management

Even in the rapidly evolving semantics-driven macrocosm that is Web 2.0, most companies still use one or more relational databases to store and manage their internal data assets. The Altova MissionKit supports working with the most prevalent of these systems (see listing below) in a wide variety of different ways. Database support is offered in XMLSpy, MapForce, StyleVision, and, of course, DatabaseSpy.

Microsoft® SQL Server® 2000, 2005, 2008
IBM DB2® 8, 9
IBM DB2 for iSeries® v5.4
IBM DB2 for zSeries® 8, 9
Oracle® 9i, 10g, 11g
Sybase® 12
MySQL® 4, 5
PostgreSQL 8
Microsoft Access™ 2003, 2007

DatabaseSpy is a multi-database query, editing, design, and comparison tool that allows users to connect directly to all major databases and edit data and design structure in a graphical user interface with features like table browsing, data editing, SQL auto-completion entry helpers, visual table design, content diff/merging, and multiple export formats. In a service-based architecture, the ability to compare and merge data directly in its native database format is an enormous asset to developers who need to locate changes, migrate differences, or synchronize versions of database tables across test and live environments. Database tool and SQL editor As a component of the MissionKit, DatabaseSpy gives disparate groups within organizations the flexibility to work with data from multiple databases in one central interface simultaneously. Whether this data is eventually integrated into other systems or applications or lives permanently in the database, DatabaseSpy provides a simple and flexible solution to managing and maintaining massive data stores.

Single Source Publishing

In today’s world of highly automated data transfer and management, it is still necessary for human readers to ultimately consume the data in some format or other. Of course, the problem that organizations often run into is what format to publish to. XML and single source publishing have revolutionized content management, document exchange, and even multilingual communications by separating content structure from appearance. An XML-based documentation system can greatly reduce costs through facilitating ease of conversion for delivery to many different data formats and types of applications. The single source concept ensures that workflow processes (i.e., conversion, edits, etc.) do not have to be repeated or reworked – that all content in the repository requires only minimal restructuring and promotion before being loaded to respective applications for delivery. Altova StyleVision is a graphical stylesheet design tool that enables users to easily apply single source publishing to XML, XBRL, and database content, without having any affect on the source data. In this way, companies can create reusable template designs for data that can then be rendered automatically in HTML, RTF, PDF, Microsoft Word 2007, and even an Authentic e-Form for immediate publication to any conceivable medium without any process disruption – resulting in the presentation of accurate, consistent, and standardized information in real-time. StyleVision stylesheet designer Single source publishing gives organizations the ability to add a human component to their highly automated data processing workflows, enabling them to view transmission reports at any stage. For example, in a world where compliance management plays such a large role in day to day enterprise operations, StyleVision can be integrated into any SOA to provide a sort of visual audit trail for manually reviewing XML, XBRL, and database transactions. StyleVision’s template-based approach to stylesheet design makes it an ideal addition to a distributed development environment, where repeatable processes are an integral part of the system’s overall efficiency.

Conclusions

Financial downturns can make investing in technology a difficult decision. However, forward-thinking organizations will find that focusing on restructuring the legacy assets they already have in place, automating internal processes, and adding virtualization layer to their application infrastructure can lead to increases in efficiency, speed, and potentially enormous ROI. The Altova MissionKit gives businesses all of the tools that they need to augment their enterprise architecture with iterative, process-driven solutions that will recover costs through the reuse of current assets and the ability to deliver Web-driven automation within and across organizations on a global scale. The MissionKit is a highly affordable solution that offers developers, software architects, and IT users all of the tools they need to build flexible and powerful technology solutions and efficiencies that advance component-based service-oriented infrastructure – without breaking the budget.

Tags: cloud services, data mapping, database tool, legacy data, MissionKit, single source publishing, SOA, software tools, XML Editor

What's New in MapForce 2009?

March 5, 2009/in Data Integration /by Niki Devgood

It feels like we’ve been writing about XBRL and HL7 for months… Let’s move away from that topic for the moment and talk about additional essential new functionality added to MapForce in the v2009 release. Granted, some of these features have been added as a necessity for mapping to/from multi-dimensional XBRL data, but they also transcend well beyond interactive financial data reporting to the other aspects of mapping XML, database, flat file, EDI, Excel 2007, and Web services data in MapForce.

Custom Functions for Data Mapping

MapForce functions add a custom data processing layer to your mappings, letting you reformat output structure and even manipulate content on-the-fly. With v2009 we have added a grouping, distinct-values, and a predicated position function to the already well-shelved MapForce function library. Grouping functions can be selected for code generation in XSLT 2.0, Java, C#, and C++ and let you effectively reorganize source data into based on common values:

group-by – reorganizes data in the output document based on a specified common value, or grouping key
group-adjacent – applies grouping to an alternating sequence of items, assigning items that have a common value to the same group
group-starts-with – begins a new group based on a specified pattern and puts all subsequent items into the same group until another item matching that pattern is found
group-ends-with – ends a new group based on a specified pattern and puts all subsequent items into the same group until another item matching that pattern is found

The distinct-values function is a data filtering operation that, simply put, automatically ignores duplicate input values when writing mapping output results. MapForce distinct-values function MapForce also now includes a predicated position function that lets you filter out data based on its context position in the input document. For example, the mapping below will return data for only the first two people listed in the source document. MapForce position function

Extended Database Support

For users creating database mappings, MapForce 2009 provides new native support for additional databases: Microsoft SQL Server 2008, Oracle 11g, and PostgreSQL 8. It also now supports mapping XML fields in SQL Server. The complete list of natively supported databases is:

Microsoft® SQL Server® 2000, 2005, 2008
IBM DB2® 8, 9
IBM DB2 for iSeries® v5.4
IBM DB2 for zSeries® 8, 9
Oracle® 9i, 10g, 11g
Sybase® 12
MySQL® 4, 5
PostgreSQL 8
Microsoft Access™ 2003, 2007

Data Mapping Documentation

The ability to generate data mapping documentation makes it much easier to collaborate on large data integration projects, which often include a variety of designers, developers, subject matter experts, and stakeholders. Generate mapping documentation

Find Dialog for Identifying Nodes

And now for a personal favorite: a new find feature. Yes, this may sound very simple and mundane – unless you have had the pleasure of mapping large and complex multi-layered data components like XBRL and EDI (there we go again!). Consider, for example, digging through an HL7 ADT A05 transaction to find the second CWE identifier field under the fifteenth PR1 segment. Find dialog Ahhh… much easier!! Take a look at the full list of new features in MapForce v2009 – and, as always, keep in mind that Altova adds new functionality to all of the MissionKit tools based on user requests… so keep ‘em coming!

Tags: data mapping, MapForce, v2009

HL7 Data Integration

February 26, 2009/in Data Integration /by Niki Devgood

By now, you may be aware of the global push that is being made for data transparency – both in the realm of financial reporting* with a recent XBRL mandate in the U.S., and electronic health records – and that these efforts are focused on the creation and maintenance of XML standards. Of course most of you are among the XML savvy and can feel free to please join me in a resounding “duh” to the rest of the world that is only now beginning to realize the value of XML data in reducing errors, lowering costs, and generally increasing the overall efficiency of data management. But for now, let’s focus a bit on healthcare data and standards. Both HL7 and the HIPAA mandated X12N formats healthcare data exchange have traditionally been EDI-based, but the newest version of HL7 (version 3.x), released in 2005 is XML-based and constrained by a formal framework (HDF) that allows for an evolving data model within a carefully defined development methodology. Yes people, standards – bring it on!! Well, of course there is a need to map this data from the HL7 EDI to HL7 XML, to and from backend systems, to Web services and beyond. So what now? Do you need to become an expert in all of these formats? Weren’t standards supposed to make things EASIER? Please ladies and gentleman, return to your seats! Let me draw your attention once again to MapForce, the coolest data integration tool on the market, with support for mapping and converting data to and from XML, databases, flat files, EDI (including HL7, X12, and EDIFACT), Excel 2007, and Web services. The shot above shows a simple graphical mapping updating an HL7 v2.6 message to v3.x. Altova MapForce is an any-to-any visual data mapping tool that supports mapping HL7 data, in its legacy EDI or newer XML-based format, to and from XML, databases, flat files, other EDI formats, and Web services. Mappings are implemented by simply importing the necessary data structures (MapForce ships with configuration files for the latest EDI standards and offers the full set of past and present HL7 standards as a free download) and dragging lines to connect nodes. A built-in function library lets you add advanced data filters and functions to further manipulate the output data. MapForce can also facilitate the automation of your HL7 transaction workflow through code generation in Java, C#, or C++ and an accessible command line interface. Additional support for mapping HL7 data to and from Web services gives healthcare organizations the ability to meet new technology challenges and changing enterprise infrastructures as they unfold within internal and external provider domains. Read more on our new HL7 tools page in the Altova Solutions Center. *The Altova MissionKit has been infused with XBRL support to meet financial reporting mandates.

Tags: data mapping, HL7

New MapForce Online Training – Code Generation

January 9, 2009/1 Comment/in Data Integration /by Niki Devgood

I’m pleased to announce that the next module in the Altova Online Training Series on the MapForce data mapping tool is now available. MapForce Code Generation is an advanced-level course that provides step-by-step tutorials for generating program code (including C# , C++, Java, XSLT, and XQuery) based on graphically defined data mappings and integrating that code into your own applications (royalty-free). Detailed tutorials also walk you through how to add custom XSLT and XQuery functions, add function libraries, and process mappings with multiple files.The MapForce Code Generation module is available on-demand, so you can learn when your schedule allows, and, like all Altova Online Training courses, there is no fee or registration required. Please let us know what you think of MapForce Code Generation – comments and suggestions are appreciated.

Tags: Altova Online Training, data mapping, MapForce

Next Altova Online Training Module is Available

November 20, 2008/in /by Erin Cavanaugh

We recently blogged about the return of free Altova Online Training, with its new, more convenient, on-demand format. The first module available was Introduction to MapForce, and we’re pleased to announce that the next module in the series is now available. MapForce Data Sources and Targets is aimed at the intermediate MapForce user and provides students with step-by-step tutorials for mapping XML, databases, CSV, EDI, text, and Excel 2007 files in the data mapping tool. Detailed tutorials also walk you through how to process and map legacy text files using MapForce FlexText. This module also covers the methods of mapping and allows you to practice each method, including source-driven, target-driven, and copy-all mapping. MapForce Data Sources and Targets is in BETA status, and we hope you’ll give us your feedback and suggestions so that we can continue to improve. Please respond either using the survey included in the class or by leaving a comment on this blog.

Tags: Altova Online Training, data mapping, MapForce

Case Study: Wrycan, Fitz & Floyd, MarketLive

August 7, 2008/in /by Niki Devgood

wrycan Fitz and Floyd is a leader in design and manufacture of hand painted ceramic gift ware. In 2007, they approached Wrycan, an Altova partner focused on content-centric XML expertise and related software development, for help creating a solution that would allow Fitz and Floyd to interface their existing CRM system to their new Web-based storefront application from MarketLive, the leader in e-commerce software solutions. Fitz and Floyd had already purchased a license for the Altova MissionKit software suite, so Wrycan was able to jump right in and start mapping data from Fitz and Floyd’s Oracle database to MarketLive’s proprietary schema using Altova MapForce. Wrycan assigned the project to a Principal Consultant, who had plenty of previous experience with XML technologies (including XSLT and XML Schema) as well as with large-scale databases, but who had never before used MapForce, Altova’s data conversion, transformation, and integration tool.

The Challenge

Fitz and Floyd required a solution that would automatically synchronize data from their Oracle database to MarketLive’s storefront application. It needed to perform the following functions: inventory updates, product updates, and order status updates. This way, when a customer ordered a Fitz and Floyd product via the MarketLive interface, they would be getting real-time information about the company’s inventory. The solution needed to be simple to use, easy to maintain, cost effective, and completed on time, so they could put their new storefront into production promptly. Fitz and Floyd’s existing data was housed in an Oracle 8.0.5 database and was organized according to internal requirements. In order to transform their data into a format that would work with MarketLive’s storefront application, Fitz and Floyd’s data needed to be mapped to MarketLive’s XML Schema. In addition, there needed to be a system in place to track and log any transaction errors that occurred.

The Solution

Because of MapForce’s ease-of-use, the Principal Consultant was able to get started using its intuitive features right away. Wrycan used MapForce to map the transformation from Fitz and Floyd’s Oracle database to the XML Schema definition (XSD) instance provided by MarketLive. Using the database as the source component and the XSD as the target, the following mapping was produced: MapForce mapping transparent In order to map to some XML Schema entities that were not explicitly defined in the original MarketLive schema, Wrycan used Altova XMLSpy’s graphical XML Schema editor to fill in the gaps, adding attributes to the schema that had not previously existed and thus ensuring that all necessary Fitz and Floyd data would be mapped to the MarketLive Web interface. An example of the schema modifications is shown below: XML Schema modifications Wrycan used MapForce’s unique code generation capabilities to automatically produce a Java applet that was used to update Fitz and Floyd’s product, inventory, shipping, and order status information programmatically. This specialized applet was then packaged along with Wrycan’s proprietary Transaction Manager. MapForce made it very easy to update and redeploy the data mapping requirements as they changed throughout the project. Because of MapForce’s ease of use and built-in code-generation capabilities, less technical users can also update the data mapping when there are changes.

Simple Web-based Transaction Manager

Utilizing open source Java technologies such as Apache Tomcat and Quartz Enterprise Job Scheduler, Wrycan was able to create a simple transaction manager that allowed the transactions handled by the MapForce-generated, Java-based data integration applet to be scheduled, processed, and logged. The Transaction Manager is a custom software application made specifically for Fitz and Floyd by Wrycan, but built in such a way that it can be reused for future clients. It consists of several components:

User interface – allows the integration of MapForce-generated Java code
FTP interface – adds the ability for files to be downloaded for transformation from Oracle database format to the eCommerce platform XML format or vice versa
Scheduler – allows the automation of the data migration
Reporter – stores transaction results in XML files accessible in the user interface and also has the ability send emails in case of exceptions

The Transaction Manager’s user interface is the point of contact for Fitz and Floyd to control and schedule any data transformations. Because Wrycan wanted to be able to reuse the Transaction Manager, they chose to generate the MapForce code in Java, a platform-independent programming language. (MapForce can also generate application source code in C# and C++.) This code is an integral part of the Transaction Manager, as it dictates the data mapping process, allowing Fitz and Floyd’s internal information to be accessed via the MarketLive interface. The FTP interface is a simple way to manage the transfer and delivery of files from within the Transaction Manager once the MapForce-generated Java applet has transformed the data according to the MarketLive schema. A built-in batch scheduler allows Fitz and Floyd to automate the data migration operations by content type (i.e. order, inventory, product, etc.). Batch jobs The reporting component allows the result of each transaction to be logged in XML. Because of this, if any transaction errors occurred, Wrycan was able to use Altova XMLSpy to analyze and debug the issues.

The Results

Fitz and Floyd now has an easy to use data integration layer that is extensible by adding new MapForce transformations, and they can easily adjust their current transactions. Any updates made to the Fitz and Floyd Oracle database are automatically transferred to the MarketLive application in a format that it can readily understand. Log Details Because the Transaction Manager application is based on platform-independent Java code (generated by MapForce), Wrycan also has a reusable application that can be used as an asset by any online retail company. Wrycan is now able to approach potential clients with a proven data integration layer product that provides job scheduling, email notification, and FTP integration and can utilize any database or schema output via a custom Altova MapForce transformation. When speaking about this project, Dan Ochs, the principal consultant at Wrycan involved with the Fitz and Floyd application stated “MapForce has proven to be an easy-to-use, effective tool for making the data integration and mapping process much easier and faster to implement.” This and many other customer case studies involving Altova solutions are available in the Altova library.

Tags: Altova, Case Study, data mapping, MapForce, MissionKit, XMLSpy