datapipeline 2.0.0: Gem aus dem Projekt etl herausgelöst

2026-10-08 15:30:02 +02:00
commit cce7b03ca9
56 changed files with 5205 additions and 0 deletions
@@ -0,0 +1,123 @@
+# Changelog
+
+Framework `datapipeline` (Gem). Neueste Version oben.
+
+## 2.0.0 – 2026-10-08
+
+- Gem `datapipeline`: `datapipeline.gemspec`, Executable `etl` (exe/etl), Code unter `lib/datapipeline/`
+- eigenes Projekt `datapipeline` (vorher Teil von `etl`, dessen Historie die frühere Entwicklung enthält); die
+  Anwendungen in `etl` binden das Gem mit `path: '../datapipeline'` ein
+- **Breaking:** alle Klassen im Namensraum `DataPipeline`: `DataPipeline::StoragePlugin`, `DataPipeline::CsvStorage`,
+  `…::PostgresStorage`, `…::JsonStorage`, `…::SqliteStorage`, `…::ExcelStorage`, `…::RestStorage`, `DataPipeline::Slack`.
+  `container :csv` usw. bleibt gleich; eigene Plugins dürfen weiter auf oberster Ebene (`<Name>Storage`) liegen
+- `DataPipeline::VERSION`; Versionsverlauf hier statt in `version.rb`
+- Postgres-Plugin lädt `pg` über `require_gem` (Hinweis, wenn das Gem fehlt)
+
+## 1.9.0 – 2026-10-08
+
+- rest: call(method, path, json:, params:, retries:) for rpc style apis, Error#status, ssl_verify: false
+- json: a single object is read as one row
+- dsl: dry_run? for steps with side effects
+- application: rechnung_anlegen.rb (pds API: verify or create customer, invoice to a project file)
+
+## 1.8.2 – 2026-10-08
+
+- rest: write_url as lambda for one url per row (e.g. PUT /archiv/<id>)
+- REST examples (read with paging/retry/lookup, write in batches, API to API) with a local demo API
+
+## 1.8.1 – 2026-10-08
+
+- sqlite: database opened on first use, create: true and truncate: true; postgres: truncate: true
+- log line "start" without trailing blank when no version is set
+- examples with data in examples/, documented in DOKUMENTATION.md section 1 and tested
+
+## 1.8.0 – 2026-10-08
+
+- DataPipeline::Layout: field list per record type (type, length, required); csv option layout: writes in field order and rejects unknown fields; validate layout checks required, length, format
+- validate warn: true (warning, row kept, stats.warnings) and ignore: fields; warning log capped at 100
+- DataPipeline.params reads --name=value / --param key=value from ARGV when not started by bin/etl
+- bin/etl passes unknown --name=value options to the script and to params
+
+## 1.7.0 – 2026-10-08
+
+- new plugins: json (array and ndjson), sqlite (:sqlite, :sqlite3), excel (:excel, :xlsx), rest
+- gems of a plugin are loaded on first use (StoragePlugin.require_gem with a hint when missing)
+
+## 1.6.0 – 2026-10-08
+
+- plugins loaded on first use (autoload): pg only for postgres, http only for slack
+- DataPipeline.register for own plugins, optionally loaded from a path on first use
+- defaults are frozen and replaced instead of changed; DataPipeline.new(**settings) per pipeline, reset_settings!
+- write can be called several times, only: lambda routes rows to a target
+- clear error when read or write is missing
+
+## 1.5.0 – 2026-10-08
+
+- on_error :abort | :skip | :collect (to: storage) for row errors in body steps
+- validate step with rules (required, max, min, format, in, type) or block
+- run returns stats (read, written, output, rejected, errors, duration), on_finish hooks, logger (default warn on stderr)
+- watermark: highest value of a column saved after a successful run (json file)
+- dry_run (nothing written, preview of the first rows) and limit
+- bin/etl run PIPELINE.rb with --from/--till/--param/--dry-run/--limit and exit codes, Rakefile with arguments through bin/etl
+
+## 1.4.0 – 2026-10-08
+
+- streamed works: rows go one by one from source.read { } through the chain to target.write_row, header/trailer/prepend/append and conditions included
+- batch and streamed share one engine for chained steps
+- parallel mode: every step gets its own copy of the row
+- lookup: load a table once as hash (grouped or unique) instead of one query per row
+- postgres: read with block uses a server side cursor (fetch_size), queries on the same connection stay possible while reading
+
+## 1.3.0 – 2026-10-08
+
+- chained mode (chained true): every body step gets the output of the previous step
+- new dsl steps map, filter, header, trailer with context (read, written, index); map and filter turn chaining on
+- all prepend and append transformations run once, stable order of registration
+- condition also as lambda
+- without any transformation the source is copied unchanged
+
+## 1.2.1 – 2026-10-08
+
+- transformations returning nil (or nil inside an array) produce no row
+- explicit dsl methods version, verbose, streamed; unknown calls raise NoMethodError (method_missing removed, unused setting processing removed)
+- container only reports a missing plugin class as unsupported, plugin errors pass through
+
+## 1.2.0 – 2026-10-08
+
+- csv: read works with ruby 3, defaults merged, option columns for fixed column order, streaming read/write_row, string keys like postgres
+- postgres: bind parameters (params:), write with table: option, nil as NULL, one transaction, quoted identifiers, optional typed results, close
+- storage plugin: common interface with set, query, write_row, close
+- pipeline closes source and target after run
+- slack notification fixed
+- removed unused lib/source.rb and lib/target.rb
+
+## 1.1.2 – 2025-02-06
+
+- split data and output data to be able to insert results to output nut comming from db
+- added query to Datapipline to execute subqueries if nessessary
+- added prepend, append and normal to transform. we are now be able to order processing
+
+## 1.1.1 – 2025-01-10
+
+- merge options in csv
+- verbose and counter
+
+## 1.1.0 – 2024-12-21
+
+- change to plugins, postgres and csv for read and write
+
+## 1.0.3 – 2024-10-18
+
+- added slack as plugin
+- added settings to source
+- added verbose in processing non stream
+- added transformation in blocks
+- added transformation condition
+
+## 1.0.2 – 2024-10-15
+
+- handle data streamed
+
+## 1.0.1 – 2024-10-10
+
+- handle missing or wrong parameter