From 1c489f6994b5cb16c518100af149a6449c7d0858 Mon Sep 17 00:00:00 2001 From: wkh Date: Fri, 30 Oct 2026 09:54:05 +0100 Subject: [PATCH] =?UTF-8?q?Schnellzugriff=20per=20Seek,=20FNV-1a-Hash,=20D?= =?UTF-8?q?ezimalrechnung=20f=C3=BCr=20Summen=20und=20/act?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit search, seek, delete und update.rdb lesen bei -mh, -mi, -mr nur Indexeintrag und Satz. Hashindizes (.h) müssen neu gebaut werden. total, mean, subtotal und die /act-Befehle rechnen mit BigDecimal, gerundet wird halb auf. --- CHANGELOG.md | 5 ++ README.md | 8 ++- demo/testall.old | 2 +- doc/review.md | 10 +-- lib/rdb.rb | 1 + lib/rdb/cmd/act.rb | 55 ++++++++------ lib/rdb/cmd/relational.rb | 4 +- lib/rdb/env.rb | 8 ++- lib/rdb/fast.rb | 146 ++++++++++++++++++++++++++++---------- test/rdb_test.rb | 37 ++++++++++ 10 files changed, 204 insertions(+), 72 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ebe40cb..a0c9cb0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,11 @@ folgen [Semantic Versioning](https://semver.org/lang/de/). Die Version steht in - `gregorian -u` schreibt den Monat ohne führende Null (`6/01/89`), wie im Handbuch. - `mean` schreibt einen ganzzahligen Mittelwert ohne `.00` (`4786`, Handbuch S. 26). +### Verbessert +- Schnellzugriff: `search`, `seek`, `delete` und `update.rdb` lesen bei `-mh`, `-mi` und `-mr` nur noch den Indexeintrag und den einen Satz (Seek auf Byte-Positionen, bei `.i` Halbierung über die Datei), nicht mehr Tabelle und Index vollständig. `Data` liest die Zeilen erst, wenn eine Methode alle braucht. +- Die Hashfunktion ist FNV-1a statt der Bytesumme. Vorher fielen 3000 fünfstellige Nummern in 30 Plätze (größter Eimer 223), jetzt liegt der größte Eimer bei etwa 4. **Hashindizes (`.h`) müssen neu gebaut werden** (`index -mh`). +- `total`, `mean`, `subtotal` und die /act-Befehle rechnen mit `BigDecimal` statt `Float`: `0.1 + 0.2 + 0.7` ergibt `1.0`, Beträge summieren sich auf den Rappen genau, und es wird kaufmännisch gerundet (halb auf). + ### Sicherheit - `report`: Spaltenwerte in `!>` wurden unmaskiert in einen Shell-Aufruf eingesetzt, eine Zelle konnte so Befehle ausführen. Sie werden jetzt mit `Shellwords.escape` eingesetzt. - `screen`: Spaltennamen, die keine gültigen Shell-Variablennamen sind, brechen mit einer Meldung ab, statt ins erzeugte Skript zu gelangen. diff --git a/README.md b/README.md index 9bf6465..49a8c30 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ sind nach dem Handbuch neu geschrieben. mise trust # einmal nach dem Clone: mise.toml freigeben mise install # Ruby 3.4 mise run install # legt die Symlinks row, column, total ... in bin/ an -mise run test # 126 Tests und die Beispiele aus dem Handbuch +mise run test # 131 Tests und die Beispiele aus dem Handbuch ``` Ohne `mise trust` brechen in einem frisch geklonten Ordner alle `mise`-Befehle ab. @@ -243,8 +243,10 @@ er nicht mehr passt (außer mit `-a`). Beide halten die Indizes aktuell. Beispiele aus den Man-Pages sind als Tests enthalten (`test/rdb_test.rb`), die aus dem Lehrbuch in `demo/handbuch/` (`test/handbuch_test.rb`). - Die Indexdateien sind dem Original nachgebaut, aber nicht gleich (Handbuch S. 103–108): - die Hashtabelle `.h` hat 2n statt 2n+1 Plätze, und in `.i` steht der Schlüssel vor - dem Offset statt dahinter. Alte /rdb-Indizes muss man neu bauen. + die Hashtabelle `.h` hat 2n statt 2n+1 Plätze (Hashfunktion FNV-1a), und in `.i` steht der + Schlüssel vor dem Offset statt dahinter. Alte /rdb-Indizes und `.h`-Dateien früherer + Versionen dieses Nachbaus muss man neu bauen. +- Summen, Mittelwerte und /act-Beträge sind Dezimalzahlen (`BigDecimal`), gerundet wird halb auf. - Dateien, die kein UTF-8 sind, liest `rdb` als Latin-1 und schreibt sie als UTF-8. Fehler erscheinen als eine Zeile; mit `RDB_DEBUG=1` kommt der Ruby-Backtrace. Werte mit Tab oder Zeilenumbruch lehnen `append`, `enter`, `replace` und `ve` ab. diff --git a/demo/testall.old b/demo/testall.old index 5cde987..6cbf9b3 100644 --- a/demo/testall.old +++ b/demo/testall.old @@ -659,7 +659,7 @@ Item Amount Cost Value Description $ echo 5 | seek inventory Item 172 204 0 8 $ echo 5 | seek -mh inventory Item -172 204 113 122 +172 204 104 113 ## searchtree: is Sally an ancestor of Jane? (needs a table of Parent, Child) $ searchtree parent Sally Jane Parent Child TRUE diff --git a/doc/review.md b/doc/review.md index fbcc704..76d6a57 100644 --- a/doc/review.md +++ b/doc/review.md @@ -74,14 +74,14 @@ Index ohne Entsprechung in der Suche offen. ## Verbesserungspotential -1. **Indexe sind nicht schnell.** Jede Suche liest Tabelle und Indexdatei +1. **Indexe sind nicht schnell. — Behoben** (Seek statt Volllesen, lazy `Data`). Ursprünglich: Jede Suche liest Tabelle und Indexdatei komplett ([fast.rb:151](../lib/rdb/fast.rb), [fast.rb:171](../lib/rdb/fast.rb), `Data.new` pro Schlüssel in `delete`/`update.rdb`): O(n) pro Schlüssel. Abhilfe: mit `seek`/`pread` direkt an der Fixbreiten-Position lesen. -2. **Schlechte Hash-Funktion** ([fast.rb:68](../lib/rdb/fast.rb)): +2. **Schlechte Hash-Funktion — Behoben** (FNV-1a, größter Eimer 4). Ursprünglich: ([fast.rb:68](../lib/rdb/fast.rb)): `bytes.sum % slots`. Bestätigt: 3000 fünfstellige Nummern landen in 30 Slots, größter Bucket 223. Abhilfe: FNV-1a o. Ä., Primzahl-Slotzahl. -3. **Geld in `Float`** ([act.rb:11](../lib/rdb/cmd/act.rb), `to_f` im /act-Teil +3. **Geld in `Float` — Behoben** (`BigDecimal` in `total`/`mean`/`subtotal` und /act). Ursprünglich: ([act.rb:11](../lib/rdb/cmd/act.rb), `to_f` im /act-Teil und in `total`). Abhilfe: `BigDecimal` oder ganzzahlige Rappen. 4. **awk-Brücke** ([env.rb:119-138](../lib/rdb/env.rb)): ganze Tabelle als String durch `capture3`, kein Streaming; handgeschriebener Parser erkennt Strings und @@ -98,5 +98,5 @@ Index ohne Entsprechung in der Suche offen. ## Empfohlene Reihenfolge -Schwachstellen und Fehler sind erledigt. Weiter mit: Hash-Funktion und -Index-Lesezugriffe (Verbesserung 1 und 2), dann Float-Geld. +Schwachstellen, Fehler und Verbesserung 1–3 sind erledigt. Offen sind 4–8: die +awk-Brücke, globale Methoden auf `Object`, doppelter Code, Testlaufzeit, Repo-Aufräumen. diff --git a/lib/rdb.rb b/lib/rdb.rb index b8209af..ad54bc9 100644 --- a/lib/rdb.rb +++ b/lib/rdb.rb @@ -3,6 +3,7 @@ # A Ruby re-creation of /rdb, the relational database tools Mark Williams # Company sold for COHERENT (~1991). +require 'bigdecimal' require 'open3' require 'date' require 'tmpdir' diff --git a/lib/rdb/cmd/act.rb b/lib/rdb/cmd/act.rb index c6d5389..dc47633 100644 --- a/lib/rdb/cmd/act.rb +++ b/lib/rdb/cmd/act.rb @@ -8,12 +8,23 @@ ACT_COMMANDS = %w[act adjust balance bom calcpay calculate cashflow chartdup close consolidate cstate explode fillform foot getjournal invoice sale tax update.inv w2].freeze +# Amounts are decimal numbers, not floats: sums of cents stay exact. def num(str) - str.to_s.strip.empty? ? 0.0 : str.to_s.strip.to_f + s = str.to_s.strip + s.match?(Rdb::NUM) ? BigDecimal(s.sub(/\.\z/, '')) : BigDecimal(s.to_f.to_s) +end + +# Two decimals, always (money drops them from whole amounts). +def fixed2(val) + val = BigDecimal(val.to_s) unless val.is_a?(BigDecimal) + val.round(2, :half_up).to_s('F').then { |t| t.sub(/\.(\d*)\z/) { ".#{Regexp.last_match(1).ljust(2, '0')}" } } end def money(val) - val == val.to_i ? val.to_i.to_s : format('%.2f', val) + val = BigDecimal(val.to_s) unless val.is_a?(BigDecimal) + return val.to_i.to_s if val == val.to_i + + val.round(2, :half_up).to_s('F').sub(/\.(\d)\z/, '.\10') end def need(file) @@ -73,7 +84,7 @@ Rdb.register('adjust', 'Create adjusted trial balance table', 'adjust') do |_arg m = (meta[h['Account']] ||= {}) h.each { |k, v| m[k] = v if !v.to_s.strip.empty? && !%w[Debit Credit].include?(k) && !m.key?(k) } end - net = Hash.new(0.0) + net = Hash.new(BigDecimal(0)) [tb, ja].each do |t| d = t.index_of('Debit') c = t.index_of('Credit') @@ -135,7 +146,7 @@ Rdb.register('getjournal', 'Copy journals for sales, purchasing, pay systems', ' a = t.index_of('Amount') total = t.rows.sum { |r| num(r[a]) } Rdb::Table.new(%w[Date Amount Account Ref Description], - [[next_date, format('%.2f', total), '', t.rows.first&.fetch(t.head.index('Ref') || 0, ''), "#{args[0].delete_prefix('journal')} from #{args[0]}"]]).write + [[next_date, fixed2(total), '', t.rows.first&.fetch(t.head.index('Ref') || 0, ''), "#{args[0].delete_prefix('journal')} from #{args[0]}"]]).write end Rdb.register('consolidate', 'Combine all subsidiary journals to general journal', 'consolidate nextdate') do |args| @@ -172,7 +183,7 @@ Rdb.register('fillform', 'Fill a tax from with adjusted trial balance data', 'fi lines = need("taxform.#{args[0]}") trial = need('adjustedtrial') ti = trial.index_of('Taxline') - sums = Hash.new(0.0) + sums = Hash.new(BigDecimal(0)) trial.rows.each { |r| sums[r[ti]] += (num(r[trial.index_of('Debit')]) - num(r[trial.index_of('Credit')])).abs } rows = lines.rows.map { |r| [r[0], money(sums[r[0]]), r[1].to_s] } File.write(args[0], Rdb::Table.new(%w[Taxline Amount Description], rows).to_s) @@ -191,7 +202,7 @@ end Rdb.register('tax', 'Compute tax from income and tax table', 'tax income < taxtable') do |args| die 'usage: tax income < taxtable' unless args.size == 1 - income = args[0].to_f + income = num(args[0]) rows = table_in.rows.map { |r| r.map { |c| num(c) } }.sort_by(&:first) row = rows.reverse.find { |r| r[0] <= income } puts format('%7d', row ? (row[1] + (income - row[0]) * row[2] / 100.0).round : 0) @@ -201,7 +212,7 @@ Rdb.register('cashflow', 'Compute balance column of cash table', 'cashflow < cas t = table_in a = t.index_of('Amount') b = t.index_of('Balance') - bal = 0.0 + bal = BigDecimal(0) rows = t.rows.map do |r| r = r.dup if r[a].strip.empty? && !r[b].strip.empty? @@ -239,9 +250,9 @@ end Rdb.register('bom', 'Produce bill-of-materials from parts list', 'bom') do |_args| part = need('part') items = need('saleitem') - qty = Hash.new(0.0) + qty = Hash.new(BigDecimal(0)) items.rows.each { |r| qty[r[items.index_of('Code')]] += num(r[items.index_of('Qty')]) } - need_total = Hash.new(0.0) + need_total = Hash.new(BigDecimal(0)) part.rows.each do |r| code = r[part.index_of('Code')] need_total[r[part.index_of('Subpart')]] += qty[code] * num(r[part.index_of('Count')]) if qty.key?(code) @@ -252,8 +263,8 @@ end Rdb.register('explode', 'Produce table of subparts and their count for a part', 'explode Part Amount [ table ]') do |args| die 'usage: explode Part Amount [table]' if args.size < 2 t = need(args[2] || 'parttable') - totals = Hash.new(0.0) - queue = [[args[0], args[1].to_f]] + totals = Hash.new(BigDecimal(0)) + queue = [[args[0], num(args[1])]] until queue.empty? part, amount = queue.shift totals[part] += amount @@ -298,9 +309,9 @@ Rdb.register('invoice', 'Print invoice for a sale order', 'invoice') do |_args| puts address_block(customers, oh['Cust']) mine = items.rows.select { |r| r[0] == n } items.with(rows: mine).write - puts format('%-40s Gross %.2f', '', num(oh['Gross'])) - puts format('%-40s Tax %.2f', '', num(oh['Tax'])) - puts format('%-40s Total %.2f', '', num(oh['Total'])) + puts format('%-40s Gross %s', '', fixed2(num(oh['Gross']))) + puts format('%-40s Tax %s', '', fixed2(num(oh['Tax']))) + puts format('%-40s Total %s', '', fixed2(num(oh['Total']))) end end @@ -314,15 +325,15 @@ Rdb.register('cstate', 'Produce customer statement', 'cstate') do |_args| chosen = c == 'all' ? orders.rows.map { |r| r[1] }.uniq : [c] chosen.each do |cust| - lines = orders.rows.select { |r| r[1] == cust }.map { |r| [cust, r[2], format('%.2f', -num(r[orders.index_of('Total')])), r[0]] } + - deposits.rows.select { |r| r[0] == cust }.map { |r| [cust, r[1], format('%.2f', num(r[2])), r[3]] } + lines = orders.rows.select { |r| r[1] == cust }.map { |r| [cust, r[2], fixed2(-num(r[orders.index_of('Total')])), r[0]] } + + deposits.rows.select { |r| r[0] == cust }.map { |r| [cust, r[1], fixed2(num(r[2])), r[3]] } balance = lines.sum { |l| num(l[2]) } puts ' Customer Statement', ' ------------------' puts address_block(customers, cust) - puts "Now Due and Payable is your balance of: #{format('%.2f', -balance)}" + puts "Now Due and Payable is your balance of: #{fixed2(-balance)}" puts 'Details of Your Orders and Payments' Rdb::Table.new(%w[Cust Date Amount Ref], lines.sort_by { |l| l[1] }).write - puts format('%-20s %s', '', format('%.2f', balance)) + puts format('%-20s %s', '', fixed2(balance)) end end end @@ -348,7 +359,7 @@ Rdb.register('sale', 'Enter sale order, and item, update customer', 'sale') do | end puts address_block(customers, cust) n = Rdb::Table.open('salesorder').rows.map { |r| r[0].to_i }.max.to_i + 1 - gross = 0.0 + gross = BigDecimal(0) items = [] loop do code = ask('Code (Return when done): ') @@ -359,13 +370,13 @@ Rdb.register('sale', 'Enter sale order, and item, update customer', 'sale') do | q = ask('Qty: ').to_i price = num(p[products.index_of('Price')]) - items << [n.to_s, (items.size + 1).to_s, code, '10', q.to_s, money(price), format('%.2f', price * q), p[products.index_of('Name')]] + items << [n.to_s, (items.size + 1).to_s, code, '10', q.to_s, money(price), fixed2(price * q), p[products.index_of('Name')]] gross += price * q end next if items.empty? - tax = gross * 0.05 - order = [n.to_s, cust, next_date, '0', format('%.2f', gross), format('%.2f', tax), format('%.2f', gross + tax)] + tax = gross * BigDecimal('0.05') + order = [n.to_s, cust, next_date, '0', fixed2(gross), fixed2(tax), fixed2(gross + tax)] puts Rdb::Table.new(%w[Number Cust Date Shipped Gross Tax Total], [order]).to_s if ask('Is this correct? (y, n) ').downcase.start_with?('y') File.write('salesorder', Rdb::Table.open('salesorder').tap { |t| t.rows << order }.to_s) diff --git a/lib/rdb/cmd/relational.rb b/lib/rdb/cmd/relational.rb index 6ce3a59..4730269 100644 --- a/lib/rdb/cmd/relational.rb +++ b/lib/rdb/cmd/relational.rb @@ -209,7 +209,7 @@ STATS.each do |cmd, desc| sel.each do |i| vals = t.rows.map { |r| r[i] } filled = vals.reject { |v| v.strip.empty? } - nums = filled.all? { |v| numeric?(v) } ? vals.map { |v| v.strip.to_f } : nil + nums = filled.all? { |v| numeric?(v) } ? vals.map { |v| v.strip.empty? ? BigDecimal(0) : dec_num(v) } : nil dec = filled.map { |v| decimals(v) }.max.to_i result[i] = case name @@ -256,7 +256,7 @@ Rdb.register('subtotal', 'Output subtotals of columns in a table', vals = group.map { |r| r[i] }.reject { |v| v.strip.empty? } next if vals.empty? || !vals.all? { |v| numeric?(v) } - sub[i] = fmt(vals.sum { |v| v.strip.to_f }, vals.map { |v| decimals(v) }.max) + sub[i] = fmt(vals.sum { |v| dec_num(v) }, vals.map { |v| decimals(v) }.max) end if list out.concat(group) diff --git a/lib/rdb/env.rb b/lib/rdb/env.rb index 55851ee..bfce2ff 100644 --- a/lib/rdb/env.rb +++ b/lib/rdb/env.rb @@ -30,8 +30,14 @@ module Rdb str.strip[/\.(\d+)\z/, 1].to_s.size end + # Decimal arithmetic, so that 0.1 + 0.2 is 0.3 and money adds up to the cent. + def dec_num(str) + BigDecimal(str.strip.sub(/\.\z/, '')) + end + def fmt(num, dec) - dec.zero? ? num.round.to_s : format("%.#{dec}f", num) + num = BigDecimal(num.to_s) unless num.is_a?(BigDecimal) + dec.zero? ? num.round(0, :half_up).to_i.to_s : num.round(dec, :half_up).to_s('F').then { |t| t.sub(/\.(\d*)\z/) { ".#{Regexp.last_match(1).ljust(dec, '0')}" } } end def blank_row?(row) diff --git a/lib/rdb/fast.rb b/lib/rdb/fast.rb index dcdaa82..15eba93 100644 --- a/lib/rdb/fast.rb +++ b/lib/rdb/fast.rb @@ -27,29 +27,39 @@ module Rdb Hit = Struct.new(:row, :xfrom, :xto) - # A table file read as bytes with the offset of every row. + # A table file. The head is read at once; the rows only when a method + # needs all of them (sequential search, -mb, update.rdb). A lookup through + # an index reads just its row, by byte offset. class Data - attr_reader :path, :head, :rows + attr_reader :path, :head def initialize(path) @path = path File.exist?(path) or Rdb.die("#{path}: no such table") - bytes = File.binread(path) - lines = [] - off = 0 - bytes.each_line("\n") do |l| - lines << Row.new(off, off + l.bytesize, l.chomp("\n")) - off += l.bytesize + File.open(path, 'rb') do |f| + first = f.gets.to_s + Rdb.die("#{path}: is a list; fast access needs a table") if first == "\n" + @head = first.chomp("\n").split("\t", -1).map { |h| h.dup.force_encoding(Encoding::UTF_8).strip } + @body = f.pos + second = f.gets + @body = f.pos if second && Table.dash_line?(second.chomp("\n").split("\t", -1)) + end + end + + def rows + @rows ||= begin + bytes = File.binread(path).byteslice(@body..).to_s + off = @body + bytes.each_line("\n").map do |l| + row = Row.new(off, off + l.bytesize, l.chomp("\n")) + off += l.bytesize + row + end end - Rdb.die("#{path}: is a list; fast access needs a table") if bytes.start_with?("\n") - @head = lines.shift&.line.to_s.split("\t", -1).map { |h| h.dup.force_encoding(Encoding::UTF_8).strip } - lines.shift if lines.first && Table.dash_line?(lines.first.cells) - @rows = lines - @by_from = lines.to_h { |r| [r.from, r] } end def live - @live ||= @rows.reject(&:blank?) + @live ||= rows.reject(&:blank?) end def column(name) @@ -57,7 +67,14 @@ module Rdb end def row_at(offset) - @by_from[offset] + return @by_from[offset] if @rows && (@by_from ||= @rows.to_h { |r| [r.from, r] }) + return nil if offset < @body + + File.open(path, 'rb') do |f| + f.seek(offset) + line = f.gets or return nil + Row.new(offset, offset + line.bytesize, line.chomp("\n")) + end end end @@ -80,8 +97,10 @@ module Rdb format("%8d\n", num) end + # FNV-1a over the key without blanks. (A plain sum of the bytes put + # 3000 five-digit numbers into 30 slots.) def hash_slot(slots, key) - key.delete(" \t").bytes.sum % slots + key.b.delete(" \t").each_byte.reduce(0xcbf29ce484222325) { |h, b| ((h ^ b) * 0x100000001b3) & 0xffffffffffffffff } % slots end def live_rows(data) @@ -122,6 +141,56 @@ module Rdb # ---- searching ---------------------------------------------------------- + # One entry of a fixed-width index (.r, .h) without reading the rest. + def fixed_count(path) + [(File.size(path) - FIXED_HEAD.bytesize) / ENTRY_WIDTH, 0].max + end + + def fixed_entry(io, i) + io.seek(FIXED_HEAD.bytesize + ENTRY_WIDTH * i) + io.read(ENTRY_WIDTH).to_i + end + + # Entries of an inverted index (.i) from the first key that is not below + # target, found by bisecting on byte positions. Yields hashes like + # read_inverted does. + def each_inverted_from(path, below) + File.open(path, 'rb') do |f| + f.gets + f.gets + lo = f.pos + hi = f.size + while lo < hi + mid = (lo + hi) / 2 + if mid == lo + at = lo + else + f.seek(mid - 1) + f.gets + at = f.pos + end + if at >= hi + hi = mid + next + end + f.seek(at) + line = f.gets + if below.call(line.split("\t", 2)[0]) + lo = at + line.bytesize + else + hi = at + end + end + f.seek(lo) + loop do + from = f.pos + line = f.gets or break + key, off = line.chomp("\n").split("\t", 2) + yield({ from: from, to: from + line.bytesize, key: key, offset: off.to_i }) + end + end + end + def read_fixed(path) File.binread(path).byteslice(FIXED_HEAD.bytesize..).to_s.lines.map { |l| l.to_i } end @@ -166,36 +235,37 @@ module Rdb end end when 'r' - ents = read_fixed(path + '.r') + ip = path + '.r' n = keyvals[0].to_i - if n.positive? && n <= ents.size - r = data.row_at(ents[n - 1]) and hits << Hit.new(r, FIXED_HEAD.bytesize + ENTRY_WIDTH * (n - 1), FIXED_HEAD.bytesize + ENTRY_WIDTH * n) + if n.positive? && n <= fixed_count(ip) + off = File.open(ip, 'rb') { |f| fixed_entry(f, n - 1) } + r = data.row_at(off) and hits << Hit.new(r, FIXED_HEAD.bytesize + ENTRY_WIDTH * (n - 1), FIXED_HEAD.bytesize + ENTRY_WIDTH * n) end when 'i' - ents = read_inverted(path + '.i') - lo = ents.bsearch_index { |e| (partial ? norm(e[:key], true)[0, keyvals[0].strip.size] : e[:key].strip) >= (partial ? norm(keyvals[0], true) : keyvals[0].strip) } - if lo - ents[lo..].each do |e| - break unless matches?(e[:key], keyvals[0], partial, exact) + target = partial ? norm(keyvals[0], true) : keyvals[0].strip + below = ->(key) { (partial ? norm(key, true)[0, keyvals[0].strip.size] : key.strip) < target } + each_inverted_from(path + '.i', below) do |e| + break unless matches?(e[:key], keyvals[0], partial, exact) - r = data.row_at(e[:offset]) - hits << Hit.new(r, e[:from], e[:to]) if r - break if first - end + r = data.row_at(e[:offset]) + hits << Hit.new(r, e[:from], e[:to]) if r + break if first end when 'h' - ents = read_fixed(path + '.h') - slots = ents.size - i = hash_slot(slots, keyvals[0]) - slots.times do - o = ents[i] - break if o.zero? + ip = path + '.h' + slots = fixed_count(ip) + File.open(ip, 'rb') do |f| + i = hash_slot(slots, keyvals[0]) + slots.times do + o = fixed_entry(f, i) + break if o.zero? - if o.positive? && (r = data.row_at(o)) && matches?(r.cells[idxs[0]], keyvals[0], false, exact) - hits << Hit.new(r, FIXED_HEAD.bytesize + ENTRY_WIDTH * i, FIXED_HEAD.bytesize + ENTRY_WIDTH * (i + 1)) - break if first + if o.positive? && (r = data.row_at(o)) && matches?(r.cells[idxs[0]], keyvals[0], false, exact) + hits << Hit.new(r, FIXED_HEAD.bytesize + ENTRY_WIDTH * i, FIXED_HEAD.bytesize + ENTRY_WIDTH * (i + 1)) + break if first + end + i = (i + 1) % slots end - i = (i + 1) % slots end else live_rows(data).each do |r| diff --git a/test/rdb_test.rb b/test/rdb_test.rb index d9c59b5..18c1a61 100644 --- a/test/rdb_test.rb +++ b/test/rdb_test.rb @@ -924,3 +924,40 @@ class RobustnessTest < RdbCase assert_empty Dir[File.join(@dir, 'wf.tmp*')] end end + +class ExactnessTest < RdbCase + def test_total_adds_decimals_exactly + out = rdb('total', 'A', stdin: "A\n-\n0.1\n0.2\n0.7\n0.3\n") + assert_equal '1.3', data(out).first.first + end + + def test_mean_rounds_half_up + out = rdb('mean', 'A', stdin: "A\n-\n0.01\n0.02\n") + assert_equal '0.02', data(out).first.first + end + + def test_subtotal_adds_decimals_exactly + out = rdb('subtotal', 'K', 'A', stdin: "K\tA\n-\t-\nx\t0.1\nx\t0.2\nx\t0.7\n") + assert_equal '1.0', data(out).first[1] + end + + def test_hash_spreads_numeric_keys + slots = Hash.new(0) + (0...3000).each { |i| slots[Rdb::Fast.hash_slot(6000, format('%05d', i))] += 1 } + assert_operator slots.size, :>, 2000 + assert_operator slots.values.max, :<=, 8 + end + + def test_indexed_search_does_not_read_all_rows + rows = (1..500).map { |i| "#{format('%04d', i)}\tv#{i}\n" }.join + write('big', "K\tV\n-\t-\n#{rows}") + %w[h i r].each { |m| rdb('index', "-m#{m}", 'big', 'K') } + data_obj = Rdb::Fast::Data.new(File.join(@dir, 'big')) + %w[h i].each do |m| + assert_equal ['0321', 'v321'], data(rdb('search', "-m#{m}", 'big', 'K', stdin: "0321\n")).first, "-m#{m}" + end + assert_equal ['0007', 'v7'], data(rdb('search', '-mr', 'big', stdin: "7\n")).first + assert_nil data_obj.instance_variable_get(:@rows) + assert_equal 'v500', data_obj.row_at(data_obj.rows.last.from).cells[1] + end +end