Should you buy or build your ETL?
A recent article on Intelligent Enterprise breaks down the buy versus build arguments for ETL software. The article caught my attention because I am generally of the opinion that unless you are suffering from a severe case of NIHS, it would be difficult to argue a case for building BI tools from scratch, given the maturity of the software and the range of tools available.
Features that give an advantage to an off-the-shelf ETL
- A graphical interface to represent the ETL process, SQL code, and any other related objects
- A built in metadata structure
- Production ready functionality - log files, alerts, and other features that ease administration and maintenance.
- Data lineage and dependency - tools to find the source data, or conversely, to find the downstream dependencies (reports, etc.)
- Data cleansing and de-duplication
- Good performance, without too much tweaking of either the ETL engine or the database
Disadvantages of shrink wrapped ETL
- Liscening and maintenance costs - with the rise of open source ETL software, most notably, the Kettle project, this might be mitgated
- Uncertainty - either unrealistically high expectations, or a underestimation of the power of the tool
- Limited compliance - use is restricted to the vendor’s supported hardware, operating systems and databases
- Proprietary scripting language - additional ramp-up time would be required, depending on how compliant the vendor is with established standards
Either way, nothing can replace the planning necessary for an ETL projecet, nor the need for the usual production standards: maintainable code, a change control system, audit reports, and the ability to restart or back out of a load.
Related Posts:
Build vs. buy, CPM edition
Comments
One Response to “Should you buy or build your ETL?”
Leave a Reply
[…] you buy or build your ETL tool, ETL development can be broken down into 10 […]