Creating Mendz.ETL-based Target Adapters

Mendz.ETL suggests three basic ingredients to ETL solutions: source adapter, mapper and target adapter. Here are some ideas for your target adapters.

Target adapters load data to a target. In ETL, target adapters perform the "L" or load part. In fact, when derived from TargetAdapterBase, you only need to implement the LoadOutput() method. Basically, target adapters write data to the target. The target can be files, or databases or web services, for example. The implementation should take care of opening access to the target, writing to it and releasing resources when done.

Mendz.Data.Common provides the FullTargetAdapter and the LineTargetAdapter classes ready to use. FullTargetAdapter can be used for small document target that can be written in full through the ETL flow. LineTargetAdapter can be used for document targets that you want to be write line per line in the ETL flow.

For example, FullSourceAdapter can be used to read a small XML file, which can then be passed to an XSLT mapper, the result of which can be passed to a FullTargetAdapter. Since .Net does not support streaming XSLT (yet) anyway, and for as long as the XML file is small, this easily makes sense. In this context, the call to Router.Route(source, mapper, target); is direct to the point: extract the full source, transform via XSLT, and load the full result to a target.

Mendz.ETL itself also provides IData* interfaces, which can be used to create data level conversions, filters, insertions, joins and splits. Implement IData* to create re-useable data manipulation utilities in your ETL solution. IData* implementations can be used in adapters, mappers, validators and/or joiners.

Callers call the target adapter's Load() method. Router.Route(), for example, consumes the target adapter passed to it by calling its Load() method. For target adapters derived from TargetAdapterBase, note that it implements Load() as follows:

1. If set, raises OnTargetAdapterStart.
2. Start producer-consumer pattern where LoadOutput() is the consumer.
3. For each output:
   1. If set, raises OnLoading.
   2. Add output to LoadOutput().
   3. If set, raises OnLoaded.
4. Complete adding to LoadOutput().
5. Wait for LoadOutput() to finish.
6. If set, calls TargetValidator.Validate(TargetSpecification).
7. If set, raises OnTargetAdapterEnd.

An event handler is invoked only if it is set (not null). If none of the events are set, the call is basically a loop to call the LoadOutput() implementation.

Get Mendz.ETL and start building your target adapters like a pro!