Tuesday, July 25, 2017

Data Mapper - Generate an Executable (.exe) File

This video shows you how to generate an executable (.exe) file to run a data transform to convert a CSV file into an XML document using Liquid Data Mapper 2017.






A free trial of the Liquid Data Mapper is available from:
https://www.liquid-technologies.com/trial-download

Friday, July 21, 2017

Data Mapper - Generate a C# Data Mapping Project

This video shows you how to generate a C# data transform with Visual Studio project files to convert a CSV file into an XML document using Liquid Data Mapper 2017.






A free trial of the Liquid Data Mapper is available from:
https://www.liquid-technologies.com/trial-download

Thursday, July 13, 2017

Data Mapper - Convert CSV to XML

This video shows you how to create a data transform to convert a CSV file into an XML document using Liquid Data Mapper 2017.






A free trial of the Liquid Data Mapper is available from:
https://www.liquid-technologies.com/trial-download

Wednesday, July 12, 2017

Data Mapper - Copying Data Video

This video shows you how to create a simple transform to copy data between a data source and data target using Liquid Data Mapper.






A free trial of the Liquid Data Mapper is available from:
https://www.liquid-technologies.com/trial-download

Tuesday, July 11, 2017

Data Mapper - First Transform Video

This video shows how to create a simple data transform between constant values and an XML document based on an XML Schema (XSD).






A free trial of the Liquid Data Mapper is available from:
https://www.liquid-technologies.com/trial-download

Tuesday, June 27, 2017

XML Standards Schema Library Updates

The following new XML Standards have been added to the XML Schema Library:

Membrane Service Proxy 1.4.1

Membrane Service Proxy is an open source, reverse HTTP proxy and framework written in Java for REST and SOAP APIs, that can be used for: Service Virtualization Backend Security HTTP based Integration API Monitoring OAuth2 SSO.
http://schemas.liquid-technologies.com/MembraneSOA/1.4.1/

FpML 5.9

FpML® (Financial products Markup Language) is the industry-standard protocol for complex financial products. It is based on XML (eXtensible Markup Language), the standard meta-language for describing data shared between applications.
Confirmation : http://schemas.liquid-technologies.com/FpML/5.9/confirmation/
Legal : http://schemas.liquid-technologies.com/FpML/5.9/legal/
Pretrade : http://schemas.liquid-technologies.com/FpML/5.9/pretrade/
Record Keeping : http://schemas.liquid-technologies.com/FpML/5.9/recordkeeping/
Reporting : http://schemas.liquid-technologies.com/FpML/5.9/reporting/
Transparency : http://schemas.liquid-technologies.com/FpML/5.9/transparency/


Please contact us if you would like to suggest an XML Standard which should be added to the XML Schema Library.

Tuesday, June 20, 2017

Liquid Data Mapper - Contexts

The Liquid Data Mapper makes building complex data transformations quick and easy, but before you start there are a few basic concepts you need to learn.

We looked at Contexts in the article Aggregate Functions, but it's worth examining this transform again in a little more detail in order to better understand the concept of contexts. It is a good idea to step through the transform using F11 as you follow the description and also experiment with changing the Sum.Context input and look at the different results produced.


Liquid Data Mapper - Contexts
Liquid Data Mapper - Contexts

The output of the transform is a CSV with two columns ‘Store’ and ‘Total Assets’. When the transform runs it reads the XmlReader.Bookstore items, and creates a CsvWriter.Row from each XmlReader.Bookstore entry returned. So the CsvWriter.Row is in the context of the current XmlReader.Bookstore.

Next the mapper reads XmlReader.Bookstore.Name and creates a CsvWriter.Store item. This action is made within the context of the parent CsvWriter.Row, so the CsvWriter.Store value comes from XmlReader.Bookstore.Name using the current XmlReader.Bookstore.

Next the Sum.Result value is requested. The Sum will produce a Sum.Result for each value read from Sum.Context. The Sum.Context points to XmlReader.Bookstore, which is in context, so the Sum evaluates all the Sum.Value items within the Sum.Context. (i.e. the current XmlReader.Bookstore).

The values come from the Multipy.Result, which is calculated by evaluating Multipy.Value1 and Multipy.Value2. An implied context is created for the parent nodes XmlReader.StockItem and XmlReader.Stock so when XmlReader.price and XmlReader.quantity are read they are both taken from the same XmlReader.StockItem.

We can examine the output window for a trace describing what actions will be performed.

0001 [0000] : WriterTextFileDocument(file:02, data:03)
0002 [0000] :     Constant(..\..\Output\Agregate Functions 01.csv:String)
0003 [0000] :     WriteTextFileRow(file:01, value:04)
0004 [0000] :         ReadChildItems(source:05, nodeType:Bookstore (Node))
0005 [0000] :             XmlRootNode(xmlDocument:06, nodeType:Company (Node))
0006 [0000] :                 XmlFileReader(filename:07)
0007 [0000] :                     Constant(..\..\Data Files\CompanyData.xml:String)
0008 [0000] :             WriterTextFileColumn(file:01, type:Store (String?), value:09)
0009 [0000] :                 ReadChildItems(source:10, nodeType:Name (String))
0010 [0000] :                     ReferenceTo(04)
0011 [0000] :             WriterTextFileColumn(file:01, type:Total Assets (String?), value:12)
0012 [0000] :                 SumFunction(Ctx:13, value:14)
0013 [0013] :                     ReferenceTo(04)
0014 [0013] :                         MultiplyFunction(value 1:15, value 2:19)
0015 [0013] :                             ReadChildItems(source:16, nodeType:price (Double))
0016 [0013] :                                 ReadChildItems(source:17, nodeType:StockItem (Node))
0017 [0013] :                                     ReadChildItems(source:18, nodeType:Stock (Node))
0018 [0013] :                                         ReferenceTo(04)
0019 [0013] :                                 ReadChildItems(source:20, nodeType:quantity (Int32))
0020 [0013] :                                     ReferenceTo(16)

A breakdown of this trace is included in the help.


Tip: Execute the transform by pressing Shift-F5. Debug the transform by pressing F5 to start the debugger, then:
    F11 - Step Into
    F9  - Set break point
    F5 - Run (stops on break points)



A free trial of the Liquid Data Mapper is available from:
https://www.liquid-technologies.com/trial-download

Friday, June 16, 2017

Liquid Data Mapper - Sub Functions

The Liquid Data Mapper makes building complex data transformations quick and easy, but before you start there are a few basic concepts you need to learn.

Custom Sub Functions are a good way to keep your transforms maintainable and easy to read by creating a library of custom components with reusable functionality.



Liquid Data Mapper - Sub Functions
Liquid Data Mapper - Sub Functions

In this example we want to tidy up our customers first name and last name, by capitalizing the first letter. The transform to do this is fairly easy to follow, it takes the first letter and makes it upper case, it takes everything after the first letter and makes it lower case and concatenates the 2 bits back together.

Although its simple there are a few parts to it and we need to use it more than once, so it ends up making our transform look quite complex.

The solution is to create a Sub Function.

Sub Function Sample - CapitalizeFirstLetter

Liquid Data Mapper - Sub Functions
Liquid Data Mapper - Sub Functions
This is a simple sub function that will capitalize the firth letter of the input string and return it as an output string.

The name and type of the sub functions inputs and outputs can be configured. You can also add more inputs and outputs as required.

It is important to name the sub function. This can be done by clicking on the canvas and setting the ‘Name’ Property in the properties window. This is the name that will then be used to find the sub function.

Sub functions can also make use of other sub functions.


Liquid Data Mapper - Sub Functions
Liquid Data Mapper - Sub Functions
We can now use the Sub Function 'CapitalizeFirstLetter' in our original transform to massively simplify the diagram but with identical functionality.


Tip: Execute the transform by pressing Shift-F5. Debug the transform by pressing F5 to start the debugger, then:
    F11 - Step Into
    F9  - Set break point
    F5 - Run (stops on break points)



A free trial of the Liquid Data Mapper is available from:
https://www.liquid-technologies.com/trial-download

Wednesday, June 14, 2017

Liquid Data Mapper - Dynamic Data Sources

The Liquid Data Mapper makes building complex data transformations quick and easy, but before you start there are a few basic concepts you need to learn.

It is possible to load data source documents from a URL that is calculated at runtime. These include XML documents, JSON documents, CSV files, and EDI documents.

This makes it possible to open associated files based on data in other files or to process all the files in a folder.


Liquid Data Mapper - Dynamic Data Sources
Liquid Data Mapper - Dynamic Data Sources

This simple example shows how to open two XML files 'ContactList.part1.xml'  and 'ContactList.part2.xml', and combine their contents. Although the filenames come from 'Constant' component values in this example, they could just as easily be values read from another data source.

ListFiles Component


Liquid Data Mapper - Dynamic Data Sources
Liquid Data Mapper - Dynamic Data Sources

This second example shows how to combine all the files in the folder “..\..\Data Files” that match the pattern “ContactList.part*.xml” (relative paths are resolved relative to the transform).

These settings can be edited by selecting the 'ListFiles 1' component and looking at the properties window.
Liquid Data Mapper - ListFiles Component Properties
Liquid Data Mapper - 'ListFiles' Component Properties

In both of the above examples, the first XML document 'ContactList.part1.xml' contains eight 'ContactDetails' elements and the second XML document 'ContactList.part2.xml' contains twelve 'ContactDetails' elements. When you run the transform, the resulting output contains all twenty 'ContactDetails' elements.

<!--Created by Liquid Data Mapper Libraries (www.liquid-technologies.com)-->
<Contacts>
    <ContactDetails>
        <Name>Aleshia Tomkiewicz</Name>
        <Emails>atomkiewicz@hotmail.com</Emails>
        <PhoneNumbers>01835-703597</PhoneNumbers>
    </ContactDetails>
    <ContactDetails>
        <Name>Evan Zigomalas</Name>
        <Emails>evan.zigomalas@gmail.com</Emails>
        <PhoneNumbers>01937-864715, 01714-737668</PhoneNumbers>
    </ContactDetails>
    <ContactDetails>
        <Name>France Andrade</Name>
        <Emails>france.andrade@hotmail.com</Emails>
        <PhoneNumbers>01347-368222</PhoneNumbers>
    </ContactDetails>
    <ContactDetails>
        <Name>Ulysses Mcwalters</Name>
        <Emails>ulysses@hotmail.com</Emails>
        <PhoneNumbers>01912-771311, 01302-601380</PhoneNumbers>
    </ContactDetails>
    <ContactDetails>
        <Name>Tyisha Veness</Name>
        <Emails>tyisha.veness@hotmail.com</Emails>
        <PhoneNumbers>01547-429341, 01290-367248</PhoneNumbers>
    </ContactDetails>
    <ContactDetails>
        <Name>Eric Rampy</Name>
        <Emails>erampy@rampy.co.uk</Emails>
        <PhoneNumbers>01969-886290, 01545-817375</PhoneNumbers>
    </ContactDetails>
    ...etc...
</Contacts>


Tip: Execute the transform by pressing Shift-F5.Debug the transform by pressing F5 to start the debugger
    F11 - Step Into
    F9  - Set break point
    F5 - Run (stops on break points)



A free trial of the Liquid Data Mapper is available from:
https://www.liquid-technologies.com/trial-download

Monday, June 12, 2017

Liquid Data Mapper - Duplicating Nodes

The Liquid Data Mapper makes building complex data transformations quick and easy, but before you start there are a few basic concepts you need to learn.

By default the data mapper diagram will only allow us to connect one data source node connection point to one data target node connection point. In order to connect a second connection point you can add additional duplicate nodes by right clicking on connection point and selecting 'Duplicate'.


Liquid Data Mapper - Duplicating Nodes
Liquid Data Mapper - Duplicating Nodes
In this example, we are copying all the data from our source XML file from 'XML reader 3' to the output 'XML Writer 3', but we also need to take data from a second data source (in this case the second data set are constant values e.g. ‘Joe Bloggs’, but it could just as easily come from another data source such as XML file, Database etc.).

As you can see in the diagram, 'Xml Writer 3' has a Duplicate 'ContactDetails' node allowing us to connect a second data source.

The resulting XML would merge the data from the two data sources and look as follows:

<!--Created by Liquid Data Mapper Libraries (www.liquid-technologies.com)-->
<Contacts>
    <ContactDetails>
        <Name>Aleshia Tomkiewicz</Name>
        <PhoneNumbers>01835-703597</PhoneNumbers>
    </ContactDetails>
    <ContactDetails>
        <Name>Evan Zigomalas</Name>
        <PhoneNumbers>01937-864715 , 01714-737668</PhoneNumbers>
    </ContactDetails>
    <ContactDetails>
        <Name>France Andrade</Name>
        <PhoneNumbers>01347-368222</PhoneNumbers>
    </ContactDetails>
...etc...
    <ContactDetails>
        <Name>Joe Bloggs</Name>
        <Emails>joe@bloggs.com</Emails>
        <PhoneNumbers>0123456789</PhoneNumbers>
    </ContactDetails>
</Contacts>

Notice the last ContactDetails is constructed from our constant data source.


Tip: Execute the transform by pressing Shift-F5.Debug the transform by pressing F5 to start the debugger
    F11 - Step Into
    F9  - Set break point
    F5 - Run (stops on break points)



A free trial of the Liquid Data Mapper is available from:
https://www.liquid-technologies.com/trial-download

Friday, June 09, 2017

Liquid Data Mapper - Scalar Components

The Liquid Data Mapper makes building complex data transformations quick and easy, but before you start there are a few basic concepts you need to learn.

Scalar Components

A Scalar component is one that takes primitive input values and return a single primitive value. However, the previous article 'Sequences', shows that the value returned from a connection point is always a sequence of values, i.e. a set of zero or more (0-n) values.

Clearly if scalar components only operate on single values but connection points always convey sequences, then we need to be able to convert between them. The problem is resolved by extracting the values from the incoming sequence and feeding them to the underlying scalar function one at a time.

Notation - For these examples, curly brackets are used to describe the contents of a sequence:
    {‘A’} is a sequence containing a single value ‘A’
    {} is an empty sequence
    {‘A’, ’B’} is sequence containing 2 values ‘A’ & ’B’

Liquid Data Mapper - Scalar Components
Liquid Data Mapper - Scalar Components
In this example, the ‘Upper 1’ component is processed as follows:

      FOREACH value1 IN inputSequenceString
             RETURN_VALUE ConvertToUpper (value1)

The sequence returned from the ‘Upper 1’ component is:
    { ConvertToUpper (‘a’), ConvertToUpper (‘b’), ConvertToUpper (‘c’) }

Which evaluates to the sequence:
    { ‘A’, ‘B’, ‘C’ }

<!--Created by Liquid Data Mapper Libraries (www.liquid-technologies.com)-->
<Contacts>
    <ContactDetails>
        <Name>A</Name>
        <Name>B</Name>
        <Name>C</Name>
    </ContactDetails>
</Contacts>


Scalar components with one input behave as you would expect, but scalar components with two or more inputs have a more noticeable effect of this transform.
Liquid Data Mapper - Scalar Components
Liquid Data Mapper - Scalar Components
In this example, the string ‘Left’ component (which takes the first n chars from the input string) has the inputs ‘String’ values and ‘Length’ values and it is evaluated as follows:

FOREACH stringValue in InputSequenceString
      FOREACH lengthValue in InputSequenceLength
             RETURN_VALUE GetStringLeft (stringValue, lengthValue)

So in this case the ‘String’ contains the sequence {‘ABCDEFGHIJ’, ‘qrstuvwxyz’}, and the ‘Length’ input contains the sequence {1, 4}, the resulting sequence is:

    { GetStringLeft (‘ABCDEFGHIJ’, 1), GetStringLeft(‘ABCDEFGHIJ’, 4), GetStringLeft(‘qrstuvwxyz’, 1), GetStringLeft(‘qrstuvwxyz’, 4)}

Which evaluates to
     {‘A’, ‘ABCD’, ‘q’, ‘qrst’}

The result is the Cartesian product of the inputs:

<!--Created by Liquid Data Mapper Libraries (www.liquid-technologies.com)-->
<Contacts>
    <ContactDetails>
        <Name>A</Name>
        <Name>ABCD</Name>
        <Name>q</Name>
        <Name>qrst</Name>
    </ContactDetails>
</Contacts>


Tip: Execute the transform by pressing Shift-F5.Debug the transform by pressing F5 to start the debugger
    F11 - Step Into
    F9  - Set break point
    F5 - Run (stops on break points)



A free trial of the Liquid Data Mapper is available from:
https://www.liquid-technologies.com/trial-download

Thursday, June 08, 2017

Liquid Data Mapper - Sequences

The Liquid Data Mapper makes building complex data transformations quick and easy, but before you start there are a few basic concepts you need to learn.

The value returned from a connection point is always a sequence of values, i.e. a set of zero or more (0-n) values.

Even if the data source only provides a single value such as an attribute value in an XML document, this is dealt with as a sequence that contains a single value. Similarly if a source value is missing, for example an optional element in an XML document, then an empty sequence is returned.

Notation - For these examples, curly brackets are used to describe the contents of a sequence:
    {‘A’} is a sequence containing a single value ‘A’
    {} is an empty sequence
    {‘A’, ’B’} is sequence containing 2 values ‘A’ & ’B’

 
Liquid Data Mapper - Sequence
Liquid Data Mapper - Sequence
In this simple example we have a 'Constant' component value as data input. This provides the simplest possible sequence as it just returns the sequence {‘A’}. So when it is used to create the Name element in the 'XML Writer', we get a single Name element with the value ‘A’:

<!--Created by Liquid Data Mapper Libraries (www.liquid-technologies.com)-->
<Contacts>
    <ContactDetails>
        <Name>A</Name>
    </ContactDetails>
</Contacts>

Liquid Data Mapper - Sequence
Liquid Data Mapper - Sequence

In this example, three 'Constant' components provide input sequences {‘A’}, {‘B’} and {‘C’} and are used as data inputs to the 'Combine' component. The 'Combine' component combines these into a single sequence {‘A’, ‘B’, ‘C’}. So now as three values are being used to create the Name element in the 'XML Writer', we get three Name elements in the XML:

<!--Created by Liquid Data Mapper Libraries (www.liquid-technologies.com)-->
<Contacts>
    <ContactDetails>
        <Name>A</Name>
        <Name>B</Name>
        <Name>C</Name>
    </ContactDetails>
</Contacts>


Tip: Execute the transform by pressing Shift-F5.Debug the transform by pressing F5 to start the debugger
    F11 - Step Into
    F9  - Set break point
    F5 - Run (stops on break points)


A free trial of the Liquid Data Mapper is available from:
https://www.liquid-technologies.com/trial-download

Wednesday, June 07, 2017

Liquid Data Mapper - Aggregate Functions

The Liquid Data Mapper makes building complex data transformations quick and easy, but before you start there are a few basic concepts you need to learn.

Aggregate Functions

You can use the Liquid Data Mapper to calculate aggregate functions such as Sum, Min, Max, Average, Count, etc.

The functions operate within a context, and all input values found within this context are included within the calculation.

Liquid Data Mapper - Aggregate Functions
Liquid Data Mapper - Aggregate Functions
In this example we are using each BookStore as a Context, so the 'Sum' aggregation function will operate on each BookStore element. The value being summed is calculated by multiplying the price by the quantity for each StockItem.

Our sample data contains three bookstores so we will get three sets of results:

Chester,1426.31
London,730.17
Leeds,423.49

Changing Context

If we now change the mapping so XmlReader.Company is connected to the Sum.Context, the 'Sum' aggregation function is now operating on all of the items in all of the Bookstores in the Company, so we only get one total value output against each Bookstore line item.
Liquid Data Mapper - Aggregate Functions
Liquid Data Mapper - Aggregate Functions
Now our sample returns a total result for every Bookstore:

Chester,2579.97
London,2579.97
Leeds,2579.97

This is probably not we want as our output, so we need to change the CSV Writer.Row context from XmlReader.Bookstore to XmlReader.Company. We will also add a Constant value to CSV Writer.Store with a value 'All Stores Total'.

Liquid Data Mapper - Aggregate Functions
Liquid Data Mapper - Aggregate Functions
Now our sample returns a total result for every Bookstore with one value:

Total for all stores,2579.97


Tip: Execute the transform by pressing Shift-F5.Debug the transform by pressing F5 to start the debugger
    F11 - Step Into
    F9  - Set break point
    F5 - Run (stops on break points)


A free trial of the Liquid Data Mapper is available from:
https://www.liquid-technologies.com/trial-download

Tuesday, June 06, 2017

Liquid Data Mapper - Filtering Data

The Liquid Data Mapper makes building complex data transformations quick and easy, but before you start there are a few basic concepts you need to learn.

Filtering Data

Filtering data is much the same as Sorting data, except you need to provide a Boolean value that will be used to determine whether the current node will be accepted or not.

In this example, the data source is an XML file based on the same root element as the destination:
Liquid Data Mapper - Filtering Data
Liquid Data Mapper - Filtering Data

The sample shows XML data being used to create an XML document, when you run the transform you will see the resulting XML document filtered where the 'Name' field is 'Niesha Bruch':

<!--Created by Liquid Data Mapper Libraries (www.liquid-technologies.com)-->
<Contacts>
    <ContactDetails>
        <Name>Niesha Bruch</Name>
        <PhoneNumbers>01874-856950 , 01342-793603</PhoneNumbers>
    </ContactDetails>
</Contacts>

Tip: Execute the transform by pressing Shift-F5.Debug the transform by pressing F5 to start the debugger
    F11 - Step Into
    F9  - Set break point
    F5 - Run (stops on break points)



A free trial of the Liquid Data Mapper is available from:
https://www.liquid-technologies.com/trial-download

Liquid Data Mapper - Sorting Data

The Liquid Data Mapper makes building complex data transformations quick and easy, but before you start there are a few basic concepts you need to learn.

Sorting Data

This example shows how to sort the ContactDetails data elements by name.

The Set based functions have an input Node, the output of the function is determined dynamically by what is connected to this input.

You can add additional 'order by' criteria by right clicking on the Sort Key and adding another sort key above or below it.

In this example, the data source is an XML file based on the same root element as the destination:
Liquid Data Mapper - Sorting Data
Liquid Data Mapper - Sorting Data
The sample shows XML data being used to create an XML document, when you run the transform you will see the resulting XML document sorted by the 'Name' fields:

<!--Created by Liquid Data Mapper Libraries (www.liquid-technologies.com) -->
<Contacts>
    <ContactDetails>
        <Name>Aleshia Tomkiewicz</Name>
        <PhoneNumbers>01835-703597</PhoneNumbers>
    </ContactDetails>
    <ContactDetails>
        <Name>Charisse Spinello</Name>
        <PhoneNumbers>01719-831436</PhoneNumbers>
    </ContactDetails>
    <ContactDetails>
        <Name>Charlesetta Erm</Name>
        <PhoneNumbers>01276-816806 , 01517-624517</PhoneNumbers>
    </ContactDetails>
    <ContactDetails>
        <Name>Corrinne</Name>
        <PhoneNumbers>01625-932209</PhoneNumbers>
    </ContactDetails>
    ... etc ...
</Contacts>


Tip: Execute the transform by pressing Shift-F5.
Debug the transform by pressing F5 to start the debugger
    F11 - Step Into
    F9  - Set break point
    F5 - Run (stops on break points)



A free trial of the Liquid Data Mapper is available from:
https://www.liquid-technologies.com/trial-download

Monday, June 05, 2017

Liquid Data Mapper - Debugging Transforms

The Liquid Data Mapper makes building complex data transformations quick and easy, but before you start there are a few basic concepts you need to learn.

Step by Step Debugger

Running a transform is fine, but when things go wrong it’s useful to be able to step through the transform and see where and why the problem is occurring.

Stepping through transforms is also a great way to learn how the data mapper works:


Liquid Data Mapper - Debugging Transforms
Liquid Data Mapper - Debugging Transforms

Start Debugging by pressing F5 or F11 to start the debugger.

Once the debugger has started keep pressing F11 to move to the next step in the transform. A tooltip window provides a description of the current items value.

As the transforms become more complex it is useful to be able to set breakpoints. You can do this by selecting a connector or connection point and pressing F9 (pressing it a second time will remove it).

When you have a break point set pressing F5 will run until it is hit.
    F5 - Run (stops on break points)
    F9  - Set break point
    F11 - Step Into

When debugging you can explore the current state by looking at the Variables, Watch, and Call Stack windows.


A free trial of the Liquid Data Mapper is available from:
https://www.liquid-technologies.com/trial-download

Liquid Data Mapper - Data Manipulation

The Liquid Data Mapper makes building complex data transformations quick and easy, but before you start there are a few basic concepts you need to learn.

Data Manipulation

In this this example we will show how to read data from a source file and copy it to the output target file, but we will also convert the 'Name' field values to upper case as we copy the values.

To do this we need to drag the 'Upper' component on to the canvas from the 'Component Palette' (see the View menu if it’s not visible) and connect the 'Name' field input and output nodes.

The data source will be an XML file based on the same root element as the destination:

Liquid Data Mapper - Data Manipulation
Liquid Data Mapper - Data Manipulation
The sample shows XML data being used to create an XML document, when you run the transform you will see the resulting XML document with the 'Name' fields converted to upper case:

<!--Created by Liquid Data Mapper Libraries (www.liquid-technologies.com) -->
<Contacts>
    <ContactDetails>
        <Name>ALESHIA TOMKIEWICZ</Name>
        <PhoneNumbers>01835-703597</PhoneNumbers>
    </ContactDetails>
    <ContactDetails>
        <Name>EVAN ZIGOMALAS</Name>
        <PhoneNumbers>01937-864715 , 01714-737668</PhoneNumbers>
    </ContactDetails>
    <ContactDetails>
        <Name>FRANCE ANDRADE</Name>
        <PhoneNumbers>01347-368222</PhoneNumbers>
    </ContactDetails>
    <ContactDetails>
        <Name>ULYSSES MCWALTERS</Name>
        <PhoneNumbers>01912-771311 , 01302-601380</PhoneNumbers>
    </ContactDetails>
     ... etc ...
</Contacts>


Tip: You can execute a transform by pressing Shift-F5.

A free trial of the Liquid Data Mapper is available from:
https://www.liquid-technologies.com/trial-download

Liquid Data Mapper - Copying Data

The Liquid Data Mapper makes building complex data transformations quick and easy, but before you start there are a few basic concepts you need to learn.

Copying Data

In this this example we will show how to read data from a source file and copy it to the output target file.

The data source will be an XML file based on the same root element as the destination:
Liquid Data Mapper - Copying Data
Liquid Data Mapper - Copying Data
 

Copying Data Recursively

When you are working with complex data copying every value can be a lot of work. When the source and destination types are the same you can use the 'Copy Recursively' function.

You can change a connector to 'Copy Recursively' by right clicking on it and all subsequent matching child items will be automatically connected:

Liquid Data Mapper - Copying Data Recursively
Liquid Data Mapper - Copying Data Recursively

Both samples show XML data being used to create an XML document, when you run either transform you will see the resulting XML document:

<!--Created by Liquid Data Mapper Libraries (www.liquid-technologies.com) -->
<Contacts>
    <ContactDetails>
        <Name>Aleshia Tomkiewicz</Name>
        <PhoneNumbers>01835-703597</PhoneNumbers>
    </ContactDetails>
    <ContactDetails>
        <Name>Evan Zigomalas</Name>
        <PhoneNumbers>01937-864715 , 01714-737668</PhoneNumbers>
    </ContactDetails>
    <ContactDetails>
        <Name>France Andrade</Name>
        <PhoneNumbers>01347-368222</PhoneNumbers>
    </ContactDetails>
    <ContactDetails>
        <Name>Ulysses Mcwalters</Name>
        <PhoneNumbers>01912-771311 , 01302-601380</PhoneNumbers>
    </ContactDetails>
    ... etc ...
</Contacts>

Tip: You can execute a transform by pressing Shift-F5.

 A free trial of the Liquid Data Mapper is available from:
https://www.liquid-technologies.com/trial-download

Liquid Data Mapper - First Transform

The Liquid Data Mapper makes building complex data transformations quick and easy, but before you start there are a few basic concepts you need to learn.

All Transforms are Target Driven, So you need a target component before anything will happen.
Data targets are shown in the ‘Component Palette’ (see the View menu if it’s not visible) and can be dragged onto the canvas to create them, upon creation a wizard to help you with the initial configuration is displayed.

The current list of data targets includes:
  • Database (SQL Server, MySQL, Oracle, etc.) Data Mapping Target
  • EDI (EDIFACT and X12) Data Mapping Target
  • JSON Data Mapping Target
  • Text files (Fixed field flat file and CSV) Data Mapping Target
  • Web Services (SOAP WSDL and RESTful WADL) Data Mapping Target
  • XML Data Mapping Target
This simple example shows an XML target based on the XML Schema 'Customers.xsd', populated with data taken from three constant values:
Liquid Data Mapper - First Transform
Liquid Data Mapper - First Transform

The sample shows some data being used to create an XML document, when you run the transform you will see the resulting XML document:

<!--Created by Liquid Data Mapper Libraries (www.liquid-technologies.com) -->
<Contacts>
    <ContactDetails>
        <Name>Joe Bloggs</Name>
        <Emails>joe@bloggs.com</Emails>
        <PhoneNumbers>0123456789</PhoneNumbers>
    </ContactDetails>
</Contacts>

Tip: You can execute a transform by pressing Shift-F5.

A free trial of the Liquid Data Mapper is available from:
https://www.liquid-technologies.com/trial-download

Wednesday, April 19, 2017

New JSON and XML Tools With Liquid Studio Update 1

Press Release: Liquid Studio 2017 - Update 1 provides new features and enhancements for JSON and XML Developers.

LEEDS, UK, April 19, 2017 - Liquid Technologies Limited (https://www.liquid-technologies.com) have announced the release of Liquid Studio 2017 - Update 1, containing new features, enhancements and fixes to  increasing productivity of JSON and XML developers. Update 1 release includes Liquid XML Data Binder support for Visual Studio 2017 projects, graphical JSON Schema Editor Extension for Visual Studio, JSON Schema and XML Schema conversion tools, along with new Data Mapper components.

Liquid Technologies have provided Visual Studio tools for over a decade, and as a 2017 Sim-Ship partner show they are committed to long term support of Microsoft's new development platforms and tools.

"We aim to provide consistent product improvements for our users with new and enhanced features in order to continue to offer the best value XML, JSON and Web Services toolkit available." said Simon Sprott, CTO, "The improvements in Update 1 will further add to the enjoyment of using our tools, increasing productivity and saving developers many hours of time and effort."

New feature highlights include:
For more information of all our products and features, including a free trial download, please see our web site: https://www.liquid-technologies.com.

Liquid Studio 2017 - Update 1 is a free update for all users who purchased Liquid Studio 2017 and all users who have a current support plan.

Thursday, March 23, 2017

What's new in XSD 1.1

The XSD 1.1 standard builds on the XSD 1.0 adding a number of features that were conspicuous by their absence. Solving many of the problems that keep coming up in forums and discussion groups, it also remains compatible with the previous standard, meaning no need to upgrade existing schemas.

Liquid Studio 2017 support XSD 1.1 as part of the graphical XML Schema editor.

Summary of what’s new in XSD 1.1

  • xs:alternative – provides value driven typing, if X=true then the element is of type Y
  • xs:assert element – allows cross field validation via XPath predicate expressions.
  • xs:assertion facet – allows value level validation via XPath predicate expressions.
  • xs:openContent – enables extensibility by allowing additional elements throughout an element
  • defaultAttributeGroup – add a group of attributes to every element
  • defaultOpenContent – make every element extensible x

Compatibility – using XSD 1.0 schemas with an XSD 1.1 validator

The XSD 1.1 validator is compatible with the XSD 1.0 standard meaning you can pass your XSD 1.0 schemas using a XSD 1.1 validator and not notice any difference in the result.

xs:alternative

Provides a mechanism for refining the type of an element based on an XPath expression. In XSD 1.0 this had to be been done explicitly using the xsi:type attribute in the XML document, but now it can be done implicitly via the schema.

It sounds more complex than it is, an example will make it clearer.

xs:alternative Sample
<?xml version="1.0" encoding="utf-8" ?>
<!--Created with Liquid Studio 2017 (https://www.liquid-technologies.com)-->
<xs:schema elementFormDefault="qualified" 
           xmlns:xs="http://www.w3.org/2001/XMLSchema">
    <xs:element name="Example">
        <xs:complexType>
            <xs:sequence>
                <xs:element name="Address" type="AddressType" maxOccurs="unbounded">
                    <xs:alternative test="@country = 'UK'" type="UKAddressType" />
                    <xs:alternative test="@country = 'US'" type="USAddressType" />
                </xs:element>
            </xs:sequence>
        </xs:complexType>
    </xs:element>
    <xs:complexType name="AddressType">
        <xs:sequence>
            <xs:element name="Line1" type="xs:string" />
            <xs:element name="Line2" type="xs:string" />
        </xs:sequence>
        <xs:attribute name="country">
            <xs:simpleType>
                <xs:restriction base="xs:string">
                    <xs:enumeration value="UK" />
                    <xs:enumeration value="US" />
                </xs:restriction>
            </xs:simpleType>
        </xs:attribute>
    </xs:complexType>
    <xs:complexType name="UKAddressType">
        <xs:complexContent>
            <xs:extension base="AddressType">
                <xs:sequence>
                    <xs:element name="County" type="xs:string" />
                    <xs:element name="Postcode" type="xs:string" />
                </xs:sequence>
            </xs:extension>
        </xs:complexContent>
    </xs:complexType>
    <xs:complexType name="USAddressType">
        <xs:complexContent>
            <xs:extension base="AddressType">
                <xs:sequence>
                    <xs:element name="State" type="xs:string" />
                    <xs:element name="ZipCode" type="xs:string" />
                </xs:sequence>
            </xs:extension>
        </xs:complexContent>
    </xs:complexType>
</xs:schema>

enter image description here

When the following XML document is validated, the test in each xs:alternative is evaluated, and the first one to pass is used. The first Address element has a the country set to ‘UK’ this matches the test @country = ‘UK’ in the first xs:alternative, so the validation continues on using the type UKAddressType.

The 3rd address has country = ‘UK’ so the UKAddressType is used, however its does not contain the correct child elements, so validation fails.

<?xml version="1.0" encoding="utf-8"?>
<!-- Created with Liquid Studio 2017 (https://www.liquid-technologies.com) -->
<Example xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="AlternativeExample.xsd">
    <Address country="UK">
        <Line1>string</Line1>
        <Line2>string</Line2>
        <County>string</County>
        <Postcode>string</Postcode>
    </Address>
    <Address country="US">
        <Line1>string</Line1>
        <Line2>string</Line2>
        <State>string</State>
        <ZipCode>string</ZipCode>
    </Address>
    <Address country="UK">
        <Line1>string</Line1>
        <Line2>string</Line2>
        <State>INVALID</State>
        <ZipCode>INVALID</ZipCode>
    </Address>
</Example>

Properties
  • test : The XPath expression which must evaluate to true in order for this alternative definition to be used. It is important to note that the XPath expression is limited to selecting attributes only. i.e. you can only base the expression on attribute values, you can not use it to access child elements.
  • xpathDefaultNamespace : The default namespace used within the expression. If not specified then it is the default namespace of the containing element in the schema.
xs:assert

The new xs:assert provides a mechanism for cross field validation, this makes it possible to implement complex business rules in an XSD schema, previously this kind of validation could only be accomplished in a limited sense with keyref’s, or by other tools (i.e. schematron).

Although this new feature is very powerful, compromises have had to be made between performance, ease of implementation and functionality. In order to simplify implementations, and limit resources use, the XPath expression can only access nodes within the element it is attached to. In the example below, as it is attached to the paragraph element it can only access nodes within the paragraph (table/title), it can NOT access the parent element Example.

The xs:assert has a attribute called test, this contains an XPath expression which is evaluated, if it does not evaluate to true, a validation error is raised.

An xs:assert can be added to the following XSD entities
  • xs:alternative
  • xs:complexType
  • xs:element
Properties
  • test : The XPath expression which must evalute to true for validation to be successfull.
  • xpathDefaultNamespace : The default namespace used within the expression. If not specified then it is the default namespace of the containing element in the schema.
Example xs:assert schema

The following schema describes an assert on the paragraph element. Its basically saying if you have a table element the element before it must be a title.

<?xml version="1.0" encoding="utf-8" ?>
<!--Created with Liquid Studio 2017 (https://www.liquid-technologies.com)-->
<xs:schema elementFormDefault="qualified" 
            xmlns:xs="http://www.w3.org/2001/XMLSchema">
    <xs:element name="Example">
        <xs:complexType>
            <xs:sequence>
                <xs:element name="paragraph" type="paragraphType"
                            maxOccurs="unbounded" />
            </xs:sequence>
        </xs:complexType>
    </xs:element>
    <xs:complexType mixed="true" name="paragraphType">
        <xs:sequence>
            <xs:element name="title" type="xs:string" minOccurs="0" />
            <xs:element name="table" type="xs:string" minOccurs="0" />
        </xs:sequence>
        <xs:assert test="if (table) then table/preceding-sibling::title else true()" />
    </xs:complexType>
</xs:schema>

XSD 1.1 xs:assert sample

Example xs:assert sample XML document

The following sample XML shows 3 paragraphs the first 2 are valid, the last one has a table element, but the element before is not a title (their is no element before it), so validation fails.

<?xml version="1.0" encoding="utf-8"?>
<!-- Created with Liquid Studio 2017 (https://www.liquid-technologies.com) -->
<Example xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="AssertSchema.xsd">
    <paragraph>
        <title>VALID</title>
        <table>VALID</table>
    </paragraph>

    <paragraph>
        <title>VALID</title>
    </paragraph>

    <paragraph>
        <table>NOT OK - Missing title</table>
    </paragraph>
</Example>

xs:assertion facet
Assertions are a new type of facet that can be applied to xs:simpleTypes, they contain an XPath expression which must evaluate to true in order validate. They can only operate on the value of the type (they can’t access other nodes in the XML document).

Properties
  • test : The XPath expression which must evalute to true for validation to be successfull.
  • xpathDefaultNamespace : The default namespace used within the expression. If not specified then it is the default namespace of the containing element in the schema.
$value : Within the XPath ‘test’ expression the built in variable $value is defined which represents the value to be tested.

Sample xs:assertion
In the sample bellow the xs:assertion ensures that only even values are allowed.

<?xml version="1.0" encoding="utf-8" ?>
<!--Created with Liquid Studio 2017 (https://www.liquid-technologies.com)-->
<xs:schema elementFormDefault="qualified" 
           xmlns:xs="http://www.w3.org/2001/XMLSchema">
    <xs:simpleType name="EvenNumbers">
        <xs:restriction base="xs:int">
            <xs:assertion test="$value mod 2 = 0" />
        </xs:restriction>
    </xs:simpleType>
    <xs:element name="Numbers">
        <xs:complexType>
            <xs:sequence>
                <xs:element name="Number" type="EvenNumbers" />
            </xs:sequence>
        </xs:complexType>
    </xs:element>
</xs:schema>
XSD 1.1. xs:assertion sample

xs:openContent
Open Content allows you to design schemas that are extensible in a way that was simply not possible in XSD 1.0. It provides a mechanism that allows additional elements to be either interleaved with existing elements or placed at the end.

xs:openContent Sample
It is not unusual to want to create a schema that contains a set of defined elements, but can also contain anything else. The consuming application would then processes what it understands and ignore anything else. This seemingly simple goal is very awkward to achieve in XSD 1.0, but with the introduction of open content its simple.

<?xml version="1.0" encoding="utf-8" ?>
<!--Created with Liquid Studio 2017 (https://www.liquid-technologies.com)-->
<xs:schema elementFormDefault="qualified" 
           xmlns:xs="http://www.w3.org/2001/XMLSchema">
    <xs:element name="Book">
        <xs:complexType>
            <xs:openContent mode="interleave">
                <xs:any namespace="##any" processContents="lax" />
            </xs:openContent>
            <xs:sequence>
                <xs:element name="Title" type="xs:string" />
                <xs:element name="Author" type="xs:string" />
                <xs:element name="Date" type="xs:gYear" />
                <xs:element name="ISBN" type="xs:string" />
                <xs:element name="Publisher" type="xs:string" />
            </xs:sequence>
        </xs:complexType>
    </xs:element>
</xs:schema>
enter image description here

The resulting XML document now allows the inclusion of any elements before, between and after the ones defined in the XSD. In this example the element ‘NewElement’ is added in the middle.

<?xml version="1.0" encoding="utf-8"?>
<!-- Created with Liquid Studio 2017 (https://www.liquid-technologies.com) -->
<Book xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      xsi:noNamespaceSchemaLocation="OpenContentExample.xsd">
    <Title>string</Title>
    <Author>string</Author>
    <NewElement/>
    <Date>2016</Date>
    <ISBN>string</ISBN>
    <Publisher>string</Publisher>
</Book>

The mode Attribute
  • none : Stops any open content rules being applied to this element (only relivant if a defaultOpenContent element has been applied to the whole schema).
  • Interleave : Allows additional elements to be placed before, between and after the elements defined in the schema.
  • suffix : Allows additional elements to only be placed after the elements defined in the schema.
The xs:openContent element also requires a child xs:any which defines the type of elements that can be added (more about the changes to xs:any later).

defaultAttributeGroup
This is a new attribute that can be attached to the xs:schema. When set it references an xs:attributeGroup which will be added to all the elements defined in this schema. It can be turned off on specific elements my setting the defaultAttributesApply=’false’ attribute on the given element.

xs:defaultOpenContent
This is a root level open content element that is applied to all the elements defined within this schema. It can be turned off on specific elements by adding the an xs:openContent element with the attribute mode=’none’.

Changes to xs:any
The xs:any construct is a little more user freindly in XSD 1.1. its gained a coulple of atttributes to better control what is allowed, and it can be used a little more freely than before.
  • notNamespace - this is a list of all the namesapces that are NOT permitted.
  • notQName - this is a list of all the qualified element names that are NOT permitted.
The use of xs:any is now a little less arduous, in XSD 1.0 the following is invalid.

 <xs:element name="Example">
    <xs:complexType>
        <xs:sequence>
            <xs:any minOccurs="0" namespace="##any" />
            <xs:element name="A" type="xs:string" minOccurs="0" />
        </xs:sequence>
    </xs:complexType>
</xs:element>

This is because the validator can not tell which rule should be applied to an element ‘A’ in the instance document. Both the any and the definition for ‘A’ would be acceptable. In order to prevent this ambiguity the XSD 1.0 rules make this construct invalid.
However, in XSD 1.1 the construct is allowed as prioity is given to formally defined elements over those the would match an xs:any.

Summary
The enhancements made to the XSD standard seem fairly minor but now provide the ability to create truly extensible schemas, and support the implementation of complex business rules. For these reasons alone the XSD 1.1 standard is worth embracing, the other features are just a bonus.