Tag Archive for: FlowForce Server

Transforming and Converting Protobuf


MapForce supports mapping protocol buffers (Protobuf) to and from other structured data formats as mapping sources or targets. In the constant quest for more efficient ways to transfer, manipulate, and manage large structured data sets, Google has created a language- and platform-neutral data format similar to XML, but smaller, faster, and simpler than even JSON data. Tools are available to generate and work with Protobuf using Java, Python, C++, C#, Ruby, and other programming languages.

The structure of any Protobuf message is defined in a .proto file that defines each field name and value type. Altova MapForce lets users drop these .proto files into a data mapping as a source or target along with any other data, including XML, JSON, relational databases, Excel, flat files, REST and SOAP web services, and others.  .proto files versions 2 and 3 are supported.

A MapForce data mapping creates compatibility between existing XML, JSON, database or legacy data formats and new applications leveraging the efficiency of Protobuf.

Read more…

Tags: , , , ,

How to View Workflow Automation Reports on FlowForce Server


Altova FlowForce Server is a high-performance workflow engine for automating enterprise-level data processing, data integration, and ETL jobs.

FlowForce Server includes a helpful web interface for managing and monitoring all aspects of data processing jobs, including in-depth logging functionality and a complete visual dashboard. Let’s take a look at how you can take advantage of FlowForce charts and statistics to monitor the progress of FlowForce Server jobs, as well as performance of the server itself, in great detail.

Article about FlowForce Server
Read more…
Tags: , , , ,

New XML Grid and More in v2021r2


In the latest release of Altova desktop developer tools and server software products, we’re introducing a completely rebuilt XML Grid View, support for XSLT3 for XML data mapping, statistics and charts for monitoring FlowForce Server, and much more. Let’s take a look at the highlights of Altova Software Version 2021 Release 2. 

New features in Altova v2021r2
Read more…
Tags: , , , , , ,

Transitioning Data Mapping Projects from Development through Testing and Production


Data mapping projects often mirror software development efforts with distinct phases for design, testing, and deployment. This is especially true for ETL (Extract Transform Load) projects when repeated data mapping execution is required as new data becomes available, and the stakes increase higher with large data sets. The Altova MissionKit and Server Software products provide Global Resources to define configurations for each project phase and smoothly transition between them.

Let’s take a look at an example based on a MapForce data mapping from a source file to a database.

Read more…
Tags: , , , , ,

Handle HTTP Errors During Automated Data Integration


Data analysts and other professionals often need to generate real-time data through automated execution of data mappings that request Web services and save the results. During automated execution it’s important to gracefully handle any unexpected HTTP error rather than terminate the integration task.

In an earlier post we discussed conditional processing of a REST Web service response to handle HTTP errors, where separate output files were generated for a normal response and an error. Now let’s look at a revised mapping solution for the airport status example to generate a single mapping result file that contains either the requested airport status or a description of the error.

Read more…
Tags: , , , , ,

Handling HTTP Errors in Web Service Data Mappings


Data integration projects that include information from external Web services may be vulnerable to HTTP errors when retrieving remote data. When data mappings run under automated control it’s especially important to detect and report errors even if errors only occur very rarely.

A MapForce data mapping can include Web service calls and output the result directly to a file or database, or combine it with other inputs for further processing. Regardless of the final output, an HTTP Web service error encountered in a REST Web service request puts the mapping at risk.

MapForce includes features for handling HTTP errors instead of simply aborting execution of a mapping. Developers can configure the body of a REST Web service call to handle and report exceptions based on the HTTP status code returned.

Let’s look at an example.

Read more…
Tags: , , , , ,

Job Distribution on FlowForce Server


FlowForce Server is Altova’s high-performance engine for automating workflows of XML processing, data integration, report generation, and more. It integrates with other Altova server software products to automate their functions, such as executing complex data integration processes, including ETL projects, designed in MapForce; running  StyleVision report generation jobs; or validating XML, XBRL, or JSON files with RaptorXML Server.

Starting with Version 2019, FlowForce Server offers new options for distributed execution and load balancing to improve availability and performance. Let’s take a look at how configuring multiple FlowForce Servers to run as a cluster can help improve data throughput and provide redundancy.

Job distribution for high availability on FlowForce Server

Read more…

Tags: , , , ,

New Data Integration Whitepaper Now Available


The data engulfing enterprises today comes from many seemingly unrelated sources. Collecting, refining, and assimilating all this information, and being flexible enough to accept new data formats and resources, is critical to success in the 24/7 connected world.

We are proud to announce a new whitepaper titled Data Challenges and the MapForce Platform for Data Integration that addresses these issues, now available for free download from the Altova Reference Library.

Mapping_mixed_content

 

 

This data integration whitepaper gives business executives, data analysts, data-focused developers and others an overview of current trends and issues in data integration, then discusses how Altova MapForce combines with Altova Server products to create a highly automated enterprise data integration platform that is configurable, scalable, and cost-effective.

To cover all the relevant, up-to-date information, the whitepaper even includes a section titled Data Integration for Mobile Devices that discusses how Altova MobileTogether easily integrates to bring data integration results to mobile devices, providing business intelligence dashboards, elegant enterprise forms, and much more in today’s cross-platform BYOD (bring your own device) workplaces.

Click here to download your copy today!

The MapForce Platfrom for Data Integration whitepaper

Tags: , , , , ,

Data Integration for Mobile Devices – MobileTogether Works with FlowForce Server


When an offsite manager can read and reply to company email on a mobile phone, it’s only a short step to wanting access to all up-to-the-minute company data from the most convenient device.

Previously we’ve written about using FlowForce Server to generate HTML output that communicates data integration results. And yes, HTML results can be made available to Web browsers on mobile phones and tablets, but that’s an inferior solution for several reasons – the data will be cumbersome to view because it is not formatted for each device, and there is no interactive support for user interface controls built into mobile operating systems like buttons, combo boxes, edit fields, etc.

Fortunately, FlowForce Server jobs can be triggered by Altova MobileTogether solutions. The results are delivered directly to the mobile device and presented in a beautiful native-client form.

Altova MobileTogether is a unique cross-platform mobile development solution that includes functionality to work with FlowForce Server, and by extension MapForce Server, to bring data integration results to mobile devices to integrate business intelligence dashboards, elegant enterprise forms, and much more.

Here’s how a month-to-date sales report can appear on a mobile phone as a MobileTogether solution:

Month-to-date sales report on a mobile device, created with MobileTogether

When the solution launches, the sales manager selects the region and product, the report is generated by execution of the FlowForce Server job running on a server back in the enterprise data center, and the results are displayed.

Read more…

Tags: , , , ,

Automating Data Integration Workflows – Download the E-Book


We’ve published an e-book titled Automating Data Integration Workflows with Altova FlowForce Server that is now available for free download from the Altova Web site.

Here is a brief description: FlowForce Server is a highly-customizable platform to automate data transformations defined by MapForce data mappings, report and document generation, and other tasks on dedicated servers, virtual machines, or workstations scaled for the scope of the project. FlowForce Server empowers data architects, analysts, and other IT professionals to efficiently complete enterprise-level data integration tasks.

Automating Data Integration Workflows with Altova FlowForce Server provides an overview of the Altova FlowForce Server platform through a series of real-world data integration tasks and examples.

Automating Data Integration Workflows with Altova FlowForce Server

The new e-book collects updated information from the Altova Blog, the Altova Web site, and some all-new material, organized in one location to help new FlowForce Server users get started and to suggest new applications and efficiencies for experienced users.

Read more…

Tags: , , , , , ,

Installing Altova Server Software in the Cloud


Cloud-first is becoming the new normal. At recent events we have frequently been asked about using Altova Server tools in the cloud. The answer is definitely, go for it. The installation is easy. In fact, we use Altova Server Software products ourselves for an internal reporting application, installed on local virtualized servers and on an AWS cloud instance. The charts below were generated by StyleVision Server running in the cloud to quickly communicate information about changes in dynamic data.

StyleVision Server is based on the built-in report and document generation engine developed for StyleVision and renders .SPS stylesheets originally designed in StyleVision, including features like a rich variety of charts to visually represent data.

In this post we will walk through the installation of FlowForce Server, MapForce Server, StyleVision Server, and RaptorXML Server for a complete data integration solution in the cloud.

Read more…

Tags: , , , , , ,

FlowForce Server Job Cache Enhancements


FlowForce Server has new enhancements that add functionality to job caching. Job caching allows administrators to schedule execution of time consuming jobs, store the results, and deliver output instantaneously in response to HTTP requests from end users. Our earlier post titled Result Caching Accelerates Application Response Time described how to configure a job to take advantage of caching.

151346252_700px

Read more…

Tags: ,

Deploy Data Mappings and Report Designs for Automated Processing


Deploying data mappings created in MapForce and report designs created in StyleVision for automated processing by Altova server products is straightforward and quick.

The File menu in MapForce includes two options to optimize, preprocess, and deploy data mappings for MapForce Server and FlowForce Server. Preprocessing enables faster performance and reduced memory footprint for most data mappings.MapForce menu option to deploy to FlowForce Server Read more…

Tags: , , , ,

Result Caching Accelerates Application Response Time


Rapid response to user input is critical to the success of any Web application. FlowForce Server administrators can leverage result caching, a new feature introduced in version 2014, to deliver nearly instantaneous results to users running FlowForce Server jobs in a browser window as HTTP services.

A FlowForce Server job running as an HTTP service

Configuring a FlowForce Server job to take advantage of result caching is a simple two-step process. As an example, let’s look at the job we created recently in FlowForce Server Jobs as HTTP Services. We defined a StyleVision Server transformation as the last step in a FlowForce Server job and made the job available as an HTTP service that delivered the HTML result to a Web browser window. Read more…

Tags: , ,

FlowForce Server Jobs as HTTP Services


FlowForce Server administrators can define jobs as HTTP services to empower end users to execute the job on demand, as easily as opening a Web page.

A FlowForce Server job as an HTTP service with a URL

When a FlowForce Server job runs as an HTTP service results are delivered back to the Web browser. If the last step is a StyleVision Server transformation, the job can create a rich HTML-based Web page.

Even better, the same result is simultaneously saved in the enterprise workflow.
Read more…

Tags: ,

Filesystem Commands and More Wizardry with FlowForce Server Built-in Functions


In our earlier posts on FlowForce Server, we described jobs that execute MapForce Server for data transformations, StyleVision Server for report and document rendering, and RaptorXML Server for XML processing. In each scenario we also used built-in system functions to copy move, or delete files, as is often required in real-world workflows.

95662684_700px

In this post we’ll take a look at more of the built-in functions automatically installed in the FlowForce Server system container. You can use these commands as execution steps to automate the file housekeeping so often required in enterprise production.
Read more…

Tags: , , ,

Constant Quest for Efficiency


In last week’s blog post titled FlowForce Server Supports RaptorXML, we created a FlowForce Server job that defined a RaptorXML execution step to validate XML files, and we called that job as a step at the end of our job that processes camera GPS data, as a final check on the output. That was a quick way to demonstrate integration of FlowForce Server and RaptorXML Server, but for real-world production we would want to perform the same task more efficiently.

RaptorXML as an execution step in a FlowForce Server job

If we insert the RaptorXML validation function at the top of the job, right in front of the On error definition as shown above, we can apply the same error handling steps for failures of either the data mapping or the validation step. In other words, an error in any one of a series of steps before the On error definition forces the job to take the error path.
Read more…

Tags: , ,

FlowForce Server Supports RaptorXML


Altova FlowForce Server supports RaptorXML Server and RaptorXML+XBRL Server.

Log details of a RaptorXML job step

Altova RaptorXML is the third-generation, hyper-fast XML and XBRL processor from the makers of XMLSpy. RaptorXML is built from the ground up to be optimized for the latest standards and parallel computing environments. Now FlowForce Server jobs can include steps to validate XML, transform XML with XSLT, execute XPath and XQuery, and even perform complex XBRL operations for financial reporting including validating XBRL taxonomies, and validating XBRL instances against XBRL taxonomies with support for XBRL Dimensions and XBRL Formula, which define new, compatible functionality to extend XBRL 2.1.

For example, we can return to the FlowForce Server job described in Taming Bad Input Data with FlowForce Server and add a RaptorXML operation to validate the XML files created from raw GPS data captured by a digital camera.
Read more…

Tags: , ,

Download Example Files for FlowForce Server, MapForce, and StyleVision


In response to an interested reader’s suggestion, the Altova MapForce mapping files and StyleVision stylesheet we deployed to FlowForce Server for the job described in the blog post Taming Bad Input Data with FlowForce Server are now available for download on the Altova Web site at www.altova.com/documents/AltovaBlogExampleFiles.zip

Altova MapForce and StyleVision sample files

Simply unzip the archive into a new folder and you’ll have all the data mappings, the stylesheet, and other supporting data files all in one place. A ReadMe file explains the contents. You can download fully functional free trials of Altova MissionKit and FlowForce Server at https://www.altova.com/download.html and implement and test FlowForce Server yourself.

Or, execute the data mappings in MapForce and the stylesheet in StyleVision to see how easy it is to extract meaningful information from the GPS data recorded by your own digital camera or GPS device. The example files in the download were also used in Web Service as a Look-Up Table to Refine GPS Data, XPath Enhances XML Reports, and others in our series on working with XML and Global Positioning Systems.

If you’re already an Altova MissionKit user, you can download these files with examples of Web services and user functions for MapForce, and XPath calculations and chart features for StyleVision, and add them to the extensive libraries of MapForce and StyleVision samples installed with Altova MissionKit tools.

Tags: , , , ,

Taming Bad Input Data with FlowForce Server


Whenever you accept data from an outside source you risk encountering errors. We have blogged about this phenomenon in the past in Expect the Unexpected – Altova MissionKit Solves a Number Format Mystery and in the series of posts on Processing the Groupon API.

Bad data in an input file can cause the data transformation step of a FlowForce Server job to fail. When a FlowForce Server Job fails, further execution steps will not be performed. FlowForce Server is designed this way to prevent an error in one job step from cascading into a series of additional invalid results. Happily, FlowForce Server also includes features to help you recover from errors and keep production flowing.

In this post we will further extend the data mapping and report rendering job described in Customizing a FlowForce Server Job to gracefully handle bad data in an input file.

FlowForce Server New Job Steps
Read more…

Tags: , , , , , ,

Come see us at TechEd in New Orleans!


clip_image004Altova will be exhibiting in New Orleans, LA from June 3-6 at Microsoft TechEd in booth #429. Come on down to the Morial Convention Center and speak with our experts about our new line of cross-platform server software products: FlowForce Server, MapForce Server and StyleVision Server. clip_image002If you are attending TechEd, stop by our booth to see first-hand how these new products offer high-speed automation for projects designed using familiar Altova MissionKit developer tools. While you are at our booth mention this blog post to receive a special giveaway. We can’t wait to see you next week in New Orleans!

Tags: , , , , ,

Customizing a FlowForce Server Job


In our earlier post titled Automate Data Mapping and Transformation with FlowForce Server, we created a job called SimpleMapAndTransform to automate data mapping with MapForce Server and creation of html reports by StyleVision Server. After the FlowForce Server job ran several times, we have accumulated many output files in the same folder we use to process input files, as well as temporary intermediate files in the workFiles folder, as seen in the image below.

FlowForce Server job execution log and working folders

In this post we will enhance the job to create more orderly results and remove unneeded temporary files.
Read more…

Tags: , , , ,

Automate Data Mapping and Transformation with FlowForce Server


Altova FlowForce Server, launched on April 29, 2013, includes numerous enhancements over earlier beta releases and one of the most exciting new features is the implementation of StyleVision Server. Now a FlowForce Server job can automate a complete data transformation workflow by executing MapForce Server for data mapping and pipelining results to StyleVision Server to render a variety of output formats.This post describes a straightforward FlowForce Server mapping and transformation job as illustrated in these messages from the FlowForce Server Log, with the most recent step at the top of the list:

FlowForce Server Job Log showing MapForce Server and StyleVision Server job steps

We’ll start with the GPS log files created by a digital camera. We wrote about these files last January in the post titled Process Multiple Input Files in a Single Data Mapping. We’ll use the mapping output with a StyleVision SPS stylesheet adapted from the XPath Enhances XML Reports post to produce a time and elevation report for each file.A FlowForce Server For-each job step repeats based on the result of an expression. We can use For-each to build a list of files in a folder, then repeat one or more steps for each file. Here is how it looks in the job configuration page:

FlowForce Server data mapping job step

The line labeled Execute function defines the mapping to be used by MapForce Server, and the input parameter {file} refers to each file in the list C:CameraGPSexample*.LOG.We can add an execution step to instruct StyleVision Server to perform the transformation:

FlowForce Server transformation job step

The data mapping creates output files by adding .gpx to the name of the input file, and now we can define the transformation input using the {file} variable with the new file suffix. We chose to create .html output, but we could just as easily create other formats for a multi-channel publishing implementation.The transformation working directory is the location where StyleVision Server unpacks the contents of the .pxf file containing the stylesheet, XML Schema, and other needed components. Using a dedicated working folder will keep the workflow more organized.We want to allow network users to drop new .LOG files into the C:CameraGPSexample folder and we want run the FlowForce Server job on a regular schedule, but we don’t want to process the same files over and over. We can define one more job step to move the processed file to a different location:

FlowForce Server file move job step

The complete FlowForce Server job is a series of three steps that loops for each .LOG file found in the folder. We can set up a repeating trigger for the workweek or any other appropriate schedule:

FlowForce Server calendar-based job trigger

Here is a portion of a .LOG file created by the camera that is an example of one input file:

.csv input file example

We can drop this file into the C:CameraGPSexample folder, where it will be processed based on the FlowForce Server job trigger:

Workflow folder structure

When the timer triggers execution of the FlowForce Server job, the Web interface Job Log page displays this series of messages for the complete job:

FlowForce Server job log for complete job execution

The contents of the C:CameraGPSexample folder now look like this:

Completed work files after FlowForce Server job execution

We can examine the 121130.LOG.gpx file in XMLSpy:

XML file created by MapForce Server job step

And we can open the .html file in any Web browser:

.html output from StyleVision Server job step

In future blog posts we will enhance this FlowForce Server example to illustrate jobs with error handling and more complete cleanup of working files.FlowForce Server is available for Windows, Linux, and soon for Mac OS platforms. To get started yourself, click here to download a free trial!

Tags: , , , , ,

New Server Products for Data Transformation


Altova Server ProductsOn the heels of our RaptorXML announcement earlier this month, we are very excited to tell you about the availability of three more new products that round out the Altova server software product family.

These high-performance server products are optimized for multi-CPU servers and are highly complementary when used together for defining and managing information pipelines. Cross-platform support is immediately available for Windows and Linux, with MacOS support expected just around the corner in May. Supported languages include English, German, Spanish, and Japanese.

Let’s take a look at each of these new products in detail.

FlowForce ServerAltova FlowForce Server

After receiving valuable customer feedback during the beta period, we have announced general availability of FlowForce Server, a powerful new tool for managing multi-step, enterprise-level data aggregation, processing, and reporting tasks. The straight-forward FlowForce Web browser interface makes it easy to configure and monitor jobs, whether they require a single step, or multiple cascading steps starting with data aggregation and conversion, and completing with multi-channel report generation.

FlowForce Server Job Log

A flexible and highly customizable system of job triggers lets you schedule FlowForce Server job execution based on each job’s unique requirements. Jobs can be triggered by date and time, by the arrival of files in hot folders, or by http command – and you can define multiple triggers per job. To take full advantage of server resources and meet the demands of busy data transformation workflows, multiple jobs – even multiple instances of the same job – can run simultaneously on FlowForce Server.

FlowForce Server Job Triggers

FlowForce Server provides strong security measures through user, role, job credential, and permission definitions that enable multiple simultaneous jobs while preventing unauthorized access to restricted data repositories or system resources.

So, what kind of jobs can you set up on FlowForce Server? Most customers will want to use it in conjunction with MapForce Server and/or StyleVision Server for high-performance automation of data integration and multichannel publishing projects that doesn’t require writing or compiling any source code.

FlowForce Server enterprise workflow

MapForce ServerAltova MapForce Server

You may already be familiar with MapForce, Altova’s desktop developer software for aggregating, mapping, and converting data. We describe it as an any-to-any data mapping tool because it lets you move between any combination of XML, SQL database, flat file, EDI, Excel, XBRL, and Web services data. With the introduction of MapForce Server, it’s now easy to fully automate those transformations that have been defined in MapForce.

MapForce Server performs data transformations based on preprocessed and optimized data mappings stored in MapForce Server Execution files prepared by MapForce and uploaded over a network. Preprocessing enables faster performance and reduced memory footprint for most data mappings.

When operating under the management of FlowForce Server, MapForce Server is a powerful transformation engine suitable for the high-volume, quick turn-around demands of today’s leading edge data centers. The standalone version is suitable in a department level or other small environment that does not require the extensive multi-tasking, job scheduling, and security features of FlowForce Server.

StyleVision ServerAltova StyleVision Server

StyleVision Server provides high-performance automation of report and document generation tasks based on templates designed in Altova StyleVision. A graphical XSLT stylesheet and report design tool, StyleVision helps you create one design to render XML and XBRL data in Web and print formats including HTML, Word, RTF, and PDF, simultaneously. Using StyleVision Server, you can generate reports from XML and XBRL data on a recurring basis faster than ever before.

Like MapForce Server, StyleVision Server is an important option for document production by FlowForce Server, and the standalone version can be run from the command line and is suitable in smaller deployments that may not require all the features of FlowForce Server.

This new line of Altova server products represents a huge leap forward in effective automation of essential business processes, whether they require a single step, or multiple cascading steps starting with data aggregation and conversion via MapForce Server and completing with multi-channel report generation via StyleVision Server. These high performance workflow servers help customers design information pipelines quickly and easily, and then deploy them in a flexible way, utilizing the power and speed afforded by multi CPU servers. We’re excited to offer this new family of server products to our customers developing data integration and reporting projects using Altova MissionKit developer tools, as well as welcoming new customers who need a multi-platform, lightning fast orchestration engine.

Download a free fully-functional 30-day trial!

As with all Altova tools, we invite you to try these new server products before making a purchasing decision. Download a free, 30-day trial now!

Tags: , ,

Web Interface Simplifies Automation of Data Transformations


FlowForce Server is a new product designed to provide automation of data transformations performed by dedicated high-speed servers. FlowForce Server can start jobs based on a variety of triggers, runs multiple jobs simultaneously, and can even run multiple instances of the same job, depending on workflow.Monitoring all these complex activities is critical to success in a busy production environment. The FlowForce Server Web interface includes customizable views into operations, simplifying management from anywhere on the network.

FlowForce Server viewed in a Web browser window

The screen shot above shows the Home page of the FlowForce Server Web browser interface, displaying all currently running jobs and active triggers. Six instances of the gpxElevationUSGS job are running, each identified by a unique job ID. Four job triggers are also active, three watching hot folders, and one based on a timer. Each FlowForce Server job automates a MapForce data transformation.The orange arrows below each grid are clickable update buttons, and the job names link to the definition pages for each job.The blue headings at the top are also clickable buttons. The Log link displays the Log View, a detailed history of all system activity, shown in a truncated version below.

FlowForce Server Log View Shows History of Operations

Each message line describes one step in one instance of a FlowForce Server job, and may have links to more information, as we described in our earlier blog post titled Automate Data Transformation with FlowForce Server. If the status is anything other than 0, the more link opens a detailed error message.

Description of a single FlowForce Server job step

In a busy environment, all this can be just too much information. In that case, the Log View offers several alternatives to help you find critical information quickly. The options bar above the list lets you filter items displayed by a particular job or Message severity.

Filter FlowForce Server Log messages by date, by job, or by severity

If you only want to know if anything unexpected occurred, select Warning as shown above to hide all normal Info messages.You can also sort the list by any of the column headings Date, Severity, Module, User, or Instance ID in ascending or descending order. When the FlowForce Server is running multiple simultaneous jobs and instances, it’s very likely individual steps from different jobs will be shuffled. Sorting by Instance ID can let you more easily review the job history.Click here to learn more about FlowForce Server Beta 3 at the Altova Web site, or visit the FlowForce Server Beta 3 download page to get started automating data transformations in your data center!

Tags: , ,

Automate Data Transformation with FlowForce Server


Altova designed FlowForce Server to provide comprehensive automation, management, and control over data transformations performed by dedicated high-speed servers. FlowForce Server Beta 3 is currently available to users of MapForce Enterprise and Professional Editions at no charge during the beta test period.FlowForce Server can provide hot folder automation of data mappings and maintains a detailed activity log users can monitor remotely in a Web browser window. The screenshot below shows the log for FlowForce Server running the MapForce data mapping CameraLogToGPX we wrote about in the blog post titled Process Multiple Input Files in a Single Data Mapping. This mapping used wildcards to specify multiple input files for processing.
FlowForce Server job log

It only takes a few minutes to set up, run, and review the results of jobs like this on FlowForce Server. Wildcards or Hot Folders?Wildcards and hot folders increase the complexity of a data transformation workflow, and using them successfully requires careful planning. Let’s take a minute to look a little deeper at the scenario we want to implement.Assume we are the IT department in a company that publishes nature and hiking guides. We employ photographers who go out trekking and record their routes as they go, using the GPS tracking feature of their digital cameras. We want to convert the camera GPS log files to XML-based .gpx format for mapping and other processing.We will publish a folder on our network where photographers can drop off their GPS log files. This will be the hot folder FlowForce Server watches for new files to supply as input to the CameraLogToGPX mapping. We only need to process each input file once. So, after the data transformation is complete, we can remove the input file from the hot folder. We also want to place the output file in a separate folder. This suggests the following FlowForce Server job steps:

  • Look in the hot folder to see if new a input files has arrived
  • Perform the data mapping on the input file and place the output file in a separate folder
  • Move the input file to a permanent location

The diagram below shows a folder structure we can use for the workflow, with files ready to drop into the hot folder for processing:

Hot folder structure

The hot folder is C:CameraGPShotFolder and the generated .gpx files will be placed in C:CameraGPSoutputFiles. When the data mapping is done, input files will be moved to C:CameraGPScompletedInput. Deploy the Mapping on a ServerMapForce Beta is a component of FlowForce Server that works just like MapForce, and adds a feature to deploy mapping files to a FlowForce Server. Existing data mappings need little or no special preparation for deployment.The only thing we need to think about are the filenames of the input and output files. We will instruct FlowForce Server to provide the input filename as a job parameter as new files arrive in the hot folder for processing. The original mapping assigned a wildcard filename for the input component, which is no longer needed. We will also want FlowForce to specify the location of the output file.We can open the mapping in MapForce Beta, remove the filename from the input component, and add a remove-folder filepath function to the output file so FlowForce Server can set the destination. The screenshot below shows the new filename definitions in the mapping.

Mapping file names

In the MapForce Beta dialog to deploy the mapping, we can choose to immediately open the mapping in a FlowForce Server job definition window in a Web browser to finish defining the job operations.

Deploy mapping dialog

Defining the Job in FlowForce ServerThe screenshot below shows the complete job steps defined in a FlowForce Server job properties window:

FlowForce Server job definition

The job trigger is defined at the bottom of the window. Every 30 seconds, FlowForce Server will check the hot folder. If the contents have changed, FlowForce Server will execute the job steps. Each Execution step could be a deployed MapForce mapping, a system step, or even another FlowForce Server job.The name of each new file entering the hot folder becomes the parameter called {triggerfile} that we use in the mapping step as the input filename, and in the move step as the name of the file to be moved.The Working-directory parameter in the mapping step defines where the output files will be placed.FlowForce Server also includes features to set automatic run and stop times for jobs, user permissions and roles, and Queue settings to define minimum time between job runs and maximum parallel instances of a job.In our scenario, we are likely to receive multiple input files in groups as they are copied from photographers’ memory cards. Multiple parallel runs can greatly improve throughput. As a rule of thumb, you might want to match the number of cores or CPUs in the machine running FlowForce Server.

FlowForce Server job queue settings

Exploring the Job Log

Each time FlowForce Server runs our job, six lines are added to the Log View shown in the illustration at the top of this post. The first and last lines record the start and completion of the job, and each Execution step generated its own start and completion messages. The phrase “completed with status: 0” means the step was successful with no errors.

FlowForce job log

We can click the more links for a detailed report on each Execution step. The screenshot below shows the message for the MapForce mapping Execution step:

FlowForce job detail

Each file dropped into the hot folder generates an individual FlowForce Server job instance, even if multiple files are added as a group. This makes it easy to track any individual input file that generates an error.When we dropped four files into the hot folder, FlowForce Server ran four jobs, and the contents of the output folder looked like this:

Job results folder

Click here to read more about FlowForce Server Beta 3 at the Altova Web site, or visit the FlowForce Server Beta 3 download page to get started automating data transformations in your data center!

Tags: , ,

FlowForce Server Beta 3 is Now Available


Altova FlowForce Server Beta 3

MapForce 2012 users who need to automate data transformations in a centralized environment are enthusiastic about FlowForce Server, the new server product from Altova that automates execution of MapForce data transformations. Now we are pleased to announce the release of FlowForce Server Beta 3, offered as a free public beta test to all licensed users of Altova MapForce 2012 Enterprise Edition and MapForce 2012 Professional Edition.

The beta test period for FlowForce Server Beta 3 is extended until March 31, 2013.

FlowForce Server Beta 3 adds support for remote job requests via an HTTP client and job parameters that can be passed to any step in a job. When used together with the request interface, job parameters empower the HTTP client to specify input values in the job request.

FlowForce Server Beta 3 also permits any job to be called as a step within another job, implements individual job queues that make it possible to control server resources used by jobs, and adds many more refinements and enhancements.

FlowForce Server Beta 3 is available in a 32-bit version and a 64-bit version. The 64-bit version allows the MapForce Beta 3 module to process extremely large data files and provides compatibility with 64-bit database drivers.

Click here to read more about FlowForce Server Beta 3 at the Altova Web site, or visit the FlowForce Server Beta 3 download page to get started automating data transformations in your data center!

Tags: , ,

FlowForce Server Beta 2 is Now Available


FlowForce Server Beta 2 FlowForce Server, the new server product from Altova that automates execution of MapForce data transformations, has gained hundreds of enthusiastic followers since the release of FlowForce Server Beta 1 last December. Now we are pleased to announce the release of FlowForce Server Beta 2, offered as a free public beta test to all licensed users of Altova MapForce 2012 Enterprise Edition and MapForce 2012 Professional Edition.

The beta test period for FlowForce Server Beta 2 is extended until September 1, 2012.

FlowForce Server Beta 2 adds robust access control to segregate jobs and related data files, so departments can work independently without seeing or overwriting each other’s data. Access control functionality includes defined Users and Roles, Privileges, and Credentials, all managed by FlowForce Server Administrators.

FlowForce Server Beta 2 also includes Web interface enhancements to provide more status information about job schedules, active triggers, and more.

FlowForce Server Beta 2 is available in a 32-bit version and a 64-bit version. The new 64-bit version allows the MapForce Beta 2 module to process very large data files and provides compatibility with 64-bit database drivers.

Click here to read more about FlowForce Server Beta 2 at the Altova Web site, or visit the FlowForce Server Beta 2 download page to get a copy of the installers for yourself!

Tags: , ,

Announcing Altova FlowForce® Server Beta 1


Altova FlowForce® Server Beta 1 is an exciting new tool for execution of automated data mappings designed to provide comprehensive management and control over data transformations performed by dedicated high-speed servers, virtual machines, or even regular workstations, depending on the size of the task. To gather user input and accelerate product development, Altova is offering Altova FlowForce Server Beta 1 as a free public beta test to all licensed users of Altova MapForce® 2012 Enterprise Edition and MapForce® 2012 Professional Edition. Users employ Altova MapForce data mappings for two different types of data transformations:

  • One-time data conversions
  • Data transformations that are repeated by date or time, when new source data is available, or based on some other external event

While command-line execution, royalty-free code generation, and the MapForce API can assist with automation of repeated transformations, FlowForce Server Beta 1 provides much greater power and flexibility. FlowForce Server Beta 1 is a server-based tool with a Web interface that makes it much easier to implement, manage, or modify data transformation jobs in a busy data processing environment. FlowForce Server Beta 1 can administer multiple transformation jobs simultaneously, lets users define and adjust a variety of job triggers and actions on the fly, can perform housekeeping tasks like moving output files or cleaning up intermediate work, records detailed logs of all activity, and much more. FlowForce Server Beta 1 consists of four components that work together as illustrated in the diagram below. Altova FlowForce Server Beta 1 block diagram The FlowForce Server continuously checks for trigger conditions, starts and monitors job execution, and writes detailed logs. MapForce Server is an implementation of the MapForce Built-in execution engine that executes mapping packages previously deployed via the MapForce graphical environment. The FlowForce Web Administration Interface is a standalone web application that runs in an internet browser and provides the front-end of FlowForce Server. MapForce Beta is an enhanced version of the Altova MapForce application with an integrated deployment feature to deploy MapForce data mappings to a FlowForce server package. Altova FlowForce Server Beta 1 is available immediately as a free public beta test to all licensed users of Altova MapForce® 2012 Enterprise Edition and MapForce® 2012 Professional Edition. Visit the FlowForce Server Beta 1 page at the Altova Web site for more information!

Tags: , , ,