Taming Bad Input Data with FlowForce Server

Whenever you accept data from an outside source you risk encountering errors. We have blogged about this phenomenon in the past in Expect the Unexpected – Altova MissionKit Solves a Number Format Mystery and in the series of posts on Processing the Groupon API.

Bad data in an input file can cause the data transformation step of a FlowForce Server job to fail. When a FlowForce Server Job fails, further execution steps will not be performed. FlowForce Server is designed this way to prevent an error in one job step from cascading into a series of additional invalid results. Happily, FlowForce Server also includes features to help you recover from errors and keep production flowing.

In this post we will further extend the data mapping and report rendering job described in Customizing a FlowForce Server Job to gracefully handle bad data in an input file.

We started by creating a variation of one of the files with bad data, and we added a folder to the workflow to be the destination for bad input files.

We made the bad input file by copying an existing input file and editing it with DiffDog.

The first numeric column in the input .csv file is a time stamp for hours, minutes, seconds, and thousandths. We simply edited the values on lines 14 and 15 to be outside the 24-hour maximum.

We launched MapForce and assigned the file with bad data as the input for the CameraLogToGPX mapping. When we clicked the Output button to execute the mapping, the following error occurred:

FlowForce Server Job Steps with Error Handling

Next, we defined a new version of the FlowForce Server job to process input inside an Error/Success Handling step. If the data mapping fails, we will move the bad input file and any partially written output .gpx file to the badData folder. If the data mapping is successful, we move on to do the transformation to generate the .html report, then move the input and .gpx files to the completed work folder.

Note the last step of the On-error section. A data mapping error will halt execution of the For-each file loop, so we recursively call the whole job again to finish any unprocessed input files.

If a data mapping error is critical to the enterprise and demands immediate action, we could even add a job step inside the On-error section to send an email message:

Of course the addressee, Subject, Massage Body, and Attachment fields are fully configurable.

If the data mapping step succeeds, FlowForce Server executes the On-success section and continues on with the .html rendering job step.

Running the Job

The job fires based on a time trigger and the FlowForce Server Log records each execution step. In the portion of the log below we can see how the bad data input file is handled. The third line in the sequence indicates the error, then the input file and partially-generated .gpx file are sent to the badData folder.

The last line above shows the server starting work on the next input file in the folder.

When the job completes processing, we see the expected results in the contents of the working folders:

FlowForce Server is available for Windows, Linux, and soon for Mac OS platforms. To get started yourself, click here to download a free trial!