datapipeline/CHANGELOG.md

# Changelog

Framework `datapipeline` (Gem). Neueste Version oben.

## 2.0.0 – 2026-10-08

- Gem `datapipeline`: `datapipeline.gemspec`, Executable `etl` (exe/etl), Code unter `lib/datapipeline/`
- eigenes Projekt `datapipeline` (vorher Teil von `etl`, dessen Historie die frühere Entwicklung enthält); die
  Anwendungen in `etl` binden das Gem mit `path: '../datapipeline'` ein
- **Breaking:** alle Klassen im Namensraum `DataPipeline`: `DataPipeline::StoragePlugin`, `DataPipeline::CsvStorage`,
  `…::PostgresStorage`, `…::JsonStorage`, `…::SqliteStorage`, `…::ExcelStorage`, `…::RestStorage`, `DataPipeline::Slack`.
  `container :csv` usw. bleibt gleich; eigene Plugins dürfen weiter auf oberster Ebene (`<Name>Storage`) liegen
- `DataPipeline::VERSION`; Versionsverlauf hier statt in `version.rb`
- Postgres-Plugin lädt `pg` über `require_gem` (Hinweis, wenn das Gem fehlt)

## 1.9.0 – 2026-10-08

- rest: call(method, path, json:, params:, retries:) for rpc style apis, Error#status, ssl_verify: false
- json: a single object is read as one row
- dsl: dry_run? for steps with side effects
- application: rechnung_anlegen.rb (pds API: verify or create customer, invoice to a project file)

## 1.8.2 – 2026-10-08

- rest: write_url as lambda for one url per row (e.g. PUT /archiv/<id>)
- REST examples (read with paging/retry/lookup, write in batches, API to API) with a local demo API

## 1.8.1 – 2026-10-08

- sqlite: database opened on first use, create: true and truncate: true; postgres: truncate: true
- log line "start" without trailing blank when no version is set
- examples with data in examples/, documented in DOKUMENTATION.md section 1 and tested

## 1.8.0 – 2026-10-08

- DataPipeline::Layout: field list per record type (type, length, required); csv option layout: writes in field order and rejects unknown fields; validate layout checks required, length, format
- validate warn: true (warning, row kept, stats.warnings) and ignore: fields; warning log capped at 100
- DataPipeline.params reads --name=value / --param key=value from ARGV when not started by bin/etl
- bin/etl passes unknown --name=value options to the script and to params

## 1.7.0 – 2026-10-08

- new plugins: json (array and ndjson), sqlite (:sqlite, :sqlite3), excel (:excel, :xlsx), rest
- gems of a plugin are loaded on first use (StoragePlugin.require_gem with a hint when missing)

## 1.6.0 – 2026-10-08

- plugins loaded on first use (autoload): pg only for postgres, http only for slack
- DataPipeline.register for own plugins, optionally loaded from a path on first use
- defaults are frozen and replaced instead of changed; DataPipeline.new(**settings) per pipeline, reset_settings!
- write can be called several times, only: lambda routes rows to a target
- clear error when read or write is missing

## 1.5.0 – 2026-10-08

- on_error :abort | :skip | :collect (to: storage) for row errors in body steps
- validate step with rules (required, max, min, format, in, type) or block
- run returns stats (read, written, output, rejected, errors, duration), on_finish hooks, logger (default warn on stderr)
- watermark: highest value of a column saved after a successful run (json file)
- dry_run (nothing written, preview of the first rows) and limit
- bin/etl run PIPELINE.rb with --from/--till/--param/--dry-run/--limit and exit codes, Rakefile with arguments through bin/etl

## 1.4.0 – 2026-10-08

- streamed works: rows go one by one from source.read { } through the chain to target.write_row, header/trailer/prepend/append and conditions included
- batch and streamed share one engine for chained steps
- parallel mode: every step gets its own copy of the row
- lookup: load a table once as hash (grouped or unique) instead of one query per row
- postgres: read with block uses a server side cursor (fetch_size), queries on the same connection stay possible while reading

## 1.3.0 – 2026-10-08

- chained mode (chained true): every body step gets the output of the previous step
- new dsl steps map, filter, header, trailer with context (read, written, index); map and filter turn chaining on
- all prepend and append transformations run once, stable order of registration
- condition also as lambda
- without any transformation the source is copied unchanged

## 1.2.1 – 2026-10-08

- transformations returning nil (or nil inside an array) produce no row
- explicit dsl methods version, verbose, streamed; unknown calls raise NoMethodError (method_missing removed, unused setting processing removed)
- container only reports a missing plugin class as unsupported, plugin errors pass through

## 1.2.0 – 2026-10-08

- csv: read works with ruby 3, defaults merged, option columns for fixed column order, streaming read/write_row, string keys like postgres
- postgres: bind parameters (params:), write with table: option, nil as NULL, one transaction, quoted identifiers, optional typed results, close
- storage plugin: common interface with set, query, write_row, close
- pipeline closes source and target after run
- slack notification fixed
- removed unused lib/source.rb and lib/target.rb

## 1.1.2 – 2025-02-06

- split data and output data to be able to insert results to output nut comming from db
- added query to Datapipline to execute subqueries if nessessary
- added prepend, append and normal to transform. we are now be able to order processing

## 1.1.1 – 2025-01-10

- merge options in csv
- verbose and counter

## 1.1.0 – 2024-12-21

- change to plugins, postgres and csv for read and write

## 1.0.3 – 2024-10-18

- added slack as plugin
- added settings to source
- added verbose in processing non stream
- added transformation in blocks
- added transformation condition

## 1.0.2 – 2024-10-15

- handle data streamed

## 1.0.1 – 2024-10-10

- handle missing or wrong parameter