datapipeline/CHANGELOG.md

Changelog

Framework datapipeline (Gem). Neueste Version oben.

2.1.0 – 2026-10-09

• Sperre gegen parallele Läufe: lock 'datei' in der Pipeline, etl --lock=datei für das ganze Skript; ein zweiter Lauf bricht mit DataPipeline::LockError ab, Exit-Code 3
• postgres, sqlite: upsert: %w[schlüssel] (insert … on conflict … do update); sqlite legt mit create: true den Unique-Index an
• postgres: write_row schreibt gesammelt je batch_size Zeilen (Default 1000) in einer Transaktion statt je Zeile einzeln; batch_size: :all = eine Transaktion für den ganzen Stream, nach einem Abbruch zurückgerollt
• csv: encoding: (z. B. 'ISO-8859-1', 'Windows-1252'), replace: für nicht darstellbare Zeichen, bom: true; ein UTF-8-BOM wird beim Lesen übersprungen
• Layout: format: true bringt Werte vor Prüfen und Schreiben ins Format ihres Typs (N mit Nachkommastellen, kaufmännisch gerundet; DAT/TIM aus Date, Time und ISO-Text)
• neues Plugin :fixed (DataPipeline::FixedStorage): Festbreiten-Dateien nach Layout schreiben und lesen
• csv: mit layout: und headers: false werden Dateien mit Satzarten gelesen, Schlüssel aus den Feldnamen
• DSL: Kurzschritte rename, select, cast (:integer, :decimal, :date, :string) und default
• veraltet, entfällt in 3.0: transform, der Parallelmodus und condition: als String; je Prozess einmal WARN deprecated: … im Log
• Verbindungen in config/connections.yml (mit ERB für ENV): connection(:name, **opts) und container(:typ, connection: :name)
• totals 'spalte' (exakte Summen über Quelle und Ergebnis, stats.total) und reconcile { |stats| … }: Abgleich vor dem Schreiben, eine Meldung bricht mit DataPipeline::ReconcileError ab
• Laufhistorie: history 'datei' bzw. etl --history=datei, eine JSON-Zeile je Lauf; DataPipeline.history liest sie; name des Laufs (über etl der Skriptname), Stats#to_h

2.0.0 – 2026-10-08

• Gem datapipeline: datapipeline.gemspec, Executable etl (exe/etl), Code unter lib/datapipeline/
• eigenes Projekt datapipeline (vorher Teil von etl, dessen Historie die frühere Entwicklung enthält); die Anwendungen in etl binden das Gem mit path: '../datapipeline' ein
• Breaking: alle Klassen im Namensraum DataPipeline: DataPipeline::StoragePlugin, DataPipeline::CsvStorage, …::PostgresStorage, …::JsonStorage, …::SqliteStorage, …::ExcelStorage, …::RestStorage, DataPipeline::Slack. container :csv usw. bleibt gleich; eigene Plugins dürfen weiter auf oberster Ebene (<Name>Storage) liegen
• DataPipeline::VERSION; Versionsverlauf hier statt in version.rb
• Postgres-Plugin lädt pg über require_gem (Hinweis, wenn das Gem fehlt)

1.9.0 – 2026-10-08

• rest: call(method, path, json:, params:, retries:) for rpc style apis, Error#status, ssl_verify: false
• json: a single object is read as one row
• dsl: dry_run? for steps with side effects
• application: rechnung_anlegen.rb (pds API: verify or create customer, invoice to a project file)

1.8.2 – 2026-10-08

• rest: write_url as lambda for one url per row (e.g. PUT /archiv/)
• REST examples (read with paging/retry/lookup, write in batches, API to API) with a local demo API

1.8.1 – 2026-10-08

• sqlite: database opened on first use, create: true and truncate: true; postgres: truncate: true
• log line "start" without trailing blank when no version is set
• examples with data in examples/, documented in DOKUMENTATION.md section 1 and tested

1.8.0 – 2026-10-08

• DataPipeline::Layout: field list per record type (type, length, required); csv option layout: writes in field order and rejects unknown fields; validate layout checks required, length, format
• validate warn: true (warning, row kept, stats.warnings) and ignore: fields; warning log capped at 100
• DataPipeline.params reads --name=value / --param key=value from ARGV when not started by bin/etl
• bin/etl passes unknown --name=value options to the script and to params

1.7.0 – 2026-10-08

• new plugins: json (array and ndjson), sqlite (:sqlite, :sqlite3), excel (:excel, :xlsx), rest
• gems of a plugin are loaded on first use (StoragePlugin.require_gem with a hint when missing)

1.6.0 – 2026-10-08

• plugins loaded on first use (autoload): pg only for postgres, http only for slack
• DataPipeline.register for own plugins, optionally loaded from a path on first use
• defaults are frozen and replaced instead of changed; DataPipeline.new(**settings) per pipeline, reset_settings!
• write can be called several times, only: lambda routes rows to a target
• clear error when read or write is missing

1.5.0 – 2026-10-08

• on_error :abort | :skip | :collect (to: storage) for row errors in body steps
• validate step with rules (required, max, min, format, in, type) or block
• run returns stats (read, written, output, rejected, errors, duration), on_finish hooks, logger (default warn on stderr)
• watermark: highest value of a column saved after a successful run (json file)
• dry_run (nothing written, preview of the first rows) and limit
• bin/etl run PIPELINE.rb with --from/--till/--param/--dry-run/--limit and exit codes, Rakefile with arguments through bin/etl

1.4.0 – 2026-10-08

• streamed works: rows go one by one from source.read { } through the chain to target.write_row, header/trailer/prepend/append and conditions included
• batch and streamed share one engine for chained steps
• parallel mode: every step gets its own copy of the row
• lookup: load a table once as hash (grouped or unique) instead of one query per row
• postgres: read with block uses a server side cursor (fetch_size), queries on the same connection stay possible while reading

1.3.0 – 2026-10-08

• chained mode (chained true): every body step gets the output of the previous step
• new dsl steps map, filter, header, trailer with context (read, written, index); map and filter turn chaining on
• all prepend and append transformations run once, stable order of registration
• condition also as lambda
• without any transformation the source is copied unchanged

1.2.1 – 2026-10-08

• transformations returning nil (or nil inside an array) produce no row
• explicit dsl methods version, verbose, streamed; unknown calls raise NoMethodError (method_missing removed, unused setting processing removed)
• container only reports a missing plugin class as unsupported, plugin errors pass through

1.2.0 – 2026-10-08

• csv: read works with ruby 3, defaults merged, option columns for fixed column order, streaming read/write_row, string keys like postgres
• postgres: bind parameters (params:), write with table: option, nil as NULL, one transaction, quoted identifiers, optional typed results, close
• storage plugin: common interface with set, query, write_row, close
• pipeline closes source and target after run
• slack notification fixed
• removed unused lib/source.rb and lib/target.rb

1.1.2 – 2025-02-06

• split data and output data to be able to insert results to output nut comming from db
• added query to Datapipline to execute subqueries if nessessary
• added prepend, append and normal to transform. we are now be able to order processing

1.1.1 – 2025-01-10

• merge options in csv
• verbose and counter

1.1.0 – 2024-12-21

• change to plugins, postgres and csv for read and write

1.0.3 – 2024-10-18

• added slack as plugin
• added settings to source
• added verbose in processing non stream
• added transformation in blocks
• added transformation condition

1.0.2 – 2024-10-15

• handle data streamed

1.0.1 – 2024-10-10

• handle missing or wrong parameter