Creating Mendz.ETL-based Joiners
Advanced ETL requirements can involve building data from various sources. Mendz.ETL-based joiners allow developers to define advanced query operations on data sources, generating data into a desired format that can be better consumed by the ETL flow.
Joiners can be called in source adapter or custom "router" implementations. Router.JoinRoute() and Router.JoinSplitRoute() are examples. Joiner implementations can combine data or perform query operations on sets of sources to create a new input format, which can then be transformed and loaded to a target or to a set of targets.
If, for example, you've experienced working on ACORD, a complex XML format that can contain normalized transactional, lookup and relational sections, you may find yourself splitting out its sections into their own files. You can then implement a JoinerBase class to query these files back together and create a flattened (or de-normalized) "view" of the records. The flattened view can then be easily transformed by mappers and the results loaded to targets.
It may also be possible that you want to join sources with other data sources like custom lookup lists, collections, databases, etc. You can create your own parsing programs or you can use LINQ to express and implement these queries.
Mendz.ETL itself also provides IData* interfaces, which can be used to create data level conversions, filters, insertions, joins and splits. Implement IData* to create re-useable data manipulation utilities in your ETL solution. IData* implementations can be used in adapters, mappers, validators and/or joiners.
Get Mendz.ETL and start building your joiners like a pro!
Joiners can be called in source adapter or custom "router" implementations. Router.JoinRoute() and Router.JoinSplitRoute() are examples. Joiner implementations can combine data or perform query operations on sets of sources to create a new input format, which can then be transformed and loaded to a target or to a set of targets.
If, for example, you've experienced working on ACORD, a complex XML format that can contain normalized transactional, lookup and relational sections, you may find yourself splitting out its sections into their own files. You can then implement a JoinerBase class to query these files back together and create a flattened (or de-normalized) "view" of the records. The flattened view can then be easily transformed by mappers and the results loaded to targets.
It may also be possible that you want to join sources with other data sources like custom lookup lists, collections, databases, etc. You can create your own parsing programs or you can use LINQ to express and implement these queries.
Mendz.ETL itself also provides IData* interfaces, which can be used to create data level conversions, filters, insertions, joins and splits. Implement IData* to create re-useable data manipulation utilities in your ETL solution. IData* implementations can be used in adapters, mappers, validators and/or joiners.
Get Mendz.ETL and start building your joiners like a pro!
Comments
Post a Comment