# Product-Need Incremental Calculation

## Problem

`productNeedArray()` recalculates ALL products every call. `calculateBalance` is O(n^2) per product — the bottleneck.

## Why We Cannot Partially Calculate a Product

Product C has 30 rows from parents A, B, X, D, E, F, H, J. Balance is cumulative: row N's balance depends on row N-1. You cannot update "only A's rows" in C — inserting or removing ANY row shifts all subsequent balances. When C is touched, ALL of C must be recalculated.

This means the incremental approach must determine WHICH products need recalculation, then fully recalculate each of them.

## How It Works

### Example BOM tree

```text
A → B → C
X → D → C
E → C
H → C
J → C
```

C is a sub-material of B, D, E, H, J. A is a parent of B. X is a parent of D.

### What each row in `calcProductIdx` contains

Every product's calc array has rows from **3 sources**:

1. **Source rows** — `fromProduct` (stock balance), `fromProduction` (production orders), `fromSales` (sales orders), `fromPurchase` (purchase orders) — built by `addSourceArray`
2. **Cascade rows from changed path** — `fromMaterial`, `from_row` points to parent in the chain. Row B's cascade row in C has `origin_product_id = A` (the root product that started the cascade)
3. **Cascade rows from unchanged paths** — same `fromMaterial` but `origin_product_id` is a product that did NOT change (H, J)

### The `origin_product_id` field

Each row gets `origin_product_id` set:

- Source rows: `origin_product_id = own product_id` (set in `addSourceArray`)
- Cascade rows: `origin_product_id = fromItem.origin_product_id` (propagated through cascade chain in `addLowerLevelNeed`)

This field survives JSON serialization as a normal string — no special handling needed.

### `from_row` lifecycle

- During calc: `from_row` holds a live Lua object reference to the parent row
- After calc: `from_row = nil`, `from_product_id = "P001"`, `from_product_row = 3` (set in `checkCalculation`) — two separate fields, no string parsing
- Before incremental calc: `restoreFromRowRef` uses `from_product_id` + `from_product_row` to restore `from_row` back to object reference

The two-field approach (`from_product_id` + `from_product_row`) was chosen instead of a single `"pid:idx"` string because it avoids string parsing on both Lua and JavaScript sides. The JavaScript receiver converts these to a numeric idx via lookup and deletes the fields.

### Gate: `incremental_product_need` constant

Controlled by `constant.incremental_product_need` in `constant.json`. When `false`, all incremental code is skipped — full recalculation every call.

## Step-by-Step Flow

### Step 1: Find changed products

`collectChangedProductId(changeIdx, grid)` scans all add/modify/delete entries in `serverChangeIdx` in a single loop. For every changed row:

- If the row has `product_id`, the product is added to `changedIdx` directly
- If the row is from `resource-load_group` or `capacity` (has `resource_id` but no `product_id`), the grid's `product_work-estimated` data is scanned for work phases using that `resource_id`, and their `product_id` values are added to `changedIdx`
- If the row is from `order-sales` or `order-purchase` (has `order_id` but no `product_id`), the grid's `order_row-sales` or `order_row-purchase` data is scanned for order rows matching that `order_id`, and their `product_id` values are added to `changedIdx`

No indexes are created — the lookups scan grid data arrays directly. The `serverChangeIdx` is built by `nc-calc-server-load-data.lua` during data loading. Grid data is already updated in-place before product-need runs.

### Step 2: Build tree down

`buildTreeDown(changedIdx)` walks the material tree level by level from each changed product, collecting all sub-materials at all levels.

Example: A changed → B is A's material → C is B's material → `treeArray = {A, B, C}`.

D, E, H, J are NOT in the tree — A's change does not cascade through them.

The walk uses `cache.treeAmountProductIdx` which maps each product to its sub-materials. For testability, `buildTreeDown` accepts an optional `treeIdxOverride` parameter — in production the cache is used, in tests a custom tree can be passed.

### Step 3: Remove stale rows

`removeRow(calcProductIdx, serverChangeIdx, treeArray, changedIdx)` handles two reasons to remove rows:

1. **Deleted source rows** — record exists in `serverChangeIdx[tableName].delete`
2. **Cascade rows from changed path** — `from == fromMaterial` AND `origin_product_id` is in `changedIdx`, only on sub-materials (not changed products themselves)

```text
C's rows before removeRow:
  [source] stock balance         — KEEP
  [source] production order      — KEEP
  [source] sales order           — KEEP
  [cascade, origin=A] from B     — REMOVE (A changed)
  [cascade, origin=X] from D     — KEEP (X didn't change)
  [cascade, origin=H] from H     — KEEP (H didn't change)
  [cascade, origin=J] from J     — KEEP (J didn't change)

C's rows after removeRow: source + X/D + H + J cascade rows
After recalculation:       source + X/D + H + J + NEW cascade from A→B→C
```

If a product's array becomes empty after removal, it is removed from `calcProductIdx`.

Changed products (A) keep ALL their rows — cascade rows from upper products are untouched.

### Step 4: Restore `from_row` references

`restoreFromRowRef(calcProductIdx, treeArray)` — for each affected product, restore `from_row` object reference from `from_product_id` + `from_product_row` fields. This is needed because after JSON serialization, `from_row` was converted to two separate fields. Before the cascade loop runs, the live object reference must be restored so that `addLowerLevelNeed` can follow the chain.

### Step 5: Set `do_calc` flags

All products get `do_calc = 0`, then `do_calc = 1` on `treeArray` products only.

The normal cascade loop (`collectNeedDown`) only processes products with `do_calc = 1`. When `addLowerLevelNeed` inserts cascade rows into a product, it sets `do_calc = 1` on that product — so products that receive new cascade rows automatically get their balance recalculated.

### Step 6: Normal cascade loop runs

Unchanged from full calculation:

1. `collectNeedDown` → only processes products with `do_calc = 1`
2. `calculateBalance(A)` → produces new `need_down` for B
3. `addLowerLevelNeed` inserts rows into B → `do_calc = 1`
4. `calculateBalance(B)` → produces new `need_down` for C
5. `addLowerLevelNeed` inserts rows into C → `do_calc = 1`
6. H, J: `do_calc = 0`, they do NOT produce new `need_down`. Their cascade rows in C are preserved from step 3
7. `calculateBalance(C)` → runs on full array: source rows + preserved H/J cascade + new A→B cascade → correct result

### Step 7: Compare new to old using `change-data`

- `setIncrementalChange` compares recalculated products against cached previous result
- Products that changed → add/modify/delete entries sent to frontend
- Products that didn't change → no entries, silently skipped
- New data merged into cache → cache stays complete for next incremental cycle

### Summary diagram

```mermaid
flowchart TD
    S1["1. collectChangedProductId → changedIdx + changedCount"] --> S2["2. buildTreeDown → treeArray"]
    S2 --> S3["3. removeRow: delete source + cascade from changed path"]
    S3 --> S4["4. restoreFromRowRef"]
    S4 --> S5["5. do_calc=1 on treeArray, do_calc=0 on rest"]
    S5 --> S6["6. Normal cascade loop"]
    S6 --> S7["7. change-data diff → send changes"]
```

## Why This Approach Is Correct

**C gets ALL its rows, not just A's path:**

- Source rows from `addSourceArray` (stock, production orders, sales, purchase) — untouched
- Cascade rows from H→C and J→C — preserved (origin is H/J, not in `changedIdx`)
- Cascade rows from X→D→C — preserved (origin is X, not in `changedIdx`)
- NEW cascade rows from A→B→C — created by cascade loop
- Balance recalculated across ALL rows → correct cumulative result

**X and D are NOT recalculated:**

- X's source data didn't change
- X→D→C cascade rows in C are preserved
- X itself doesn't run `calculateBalance`

**Only the minimum is recalculated:**

- A: source rows updated, balance recalculated
- B: cascade rows from A path removed, new ones inserted from A's cascade, balance recalculated
- C: cascade rows from A path removed, new ones from A→B path, preserved rows from H/J/X/D, balance recalculated
- H, J, X, D, E: completely untouched, zero work

## Change Data Structure

`serverChangeIdx` (passed as `gridIdx.changeIdx`) is a table-first structure with three change types. Each table entry contains `add`, `modify`, `delete` hash maps keyed by record id, plus counts.

For product-need incremental, this structure serves three purposes:

1. **Source for `changedIdx`** — extract `product_id` from each changed row to find which products changed
2. **Source for `removeRow`** — `delete` entries tell which source rows to remove from `calcProductIdx`
3. **Source for threshold check** — total counts determine whether incremental is worth doing

The change data is built by `nc-calc-server-load-data.lua` during data loading — grid data is already updated in-place before product-need runs, so the change data only tracks what changed, not the current values.

### `from_row` serialization format

After `checkCalculation`, cascade rows have `from_row = nil` and two fields instead:

- `from_product_id` — string, the parent product id (e.g. `"P001"`)
- `from_product_row` — number, the row position within that parent's array (e.g. `3`)

This is more robust than a single `"pid:idx"` string because it avoids string parsing on both sides. The JavaScript receiver converts the two fields to a numeric idx via lookup and deletes them.

## Implementation

### Functions in `product-need.lua`

| Function | Purpose |
|---|---|
| `collectChangedProductId(changeIdx, grid)` | Single loop: scan all tables for rows with `product_id`, scan grid data for resource/capacity/order lookups → `changedIdx`, `changedCount` |
| `removeRow(calcProductIdx, changeIdx, treeArray, changedIdx)` | Remove deleted source rows + cascade rows from changed path |
| `restoreFromRowRef(calcProductIdx, treeArray)` | Restore `from_row` from `from_product_id` + `from_product_row` back to object reference |
| `incrementalUpdate(calcProductIdx, serverChangeIdx, grid)` | Calls the functions above + sets `do_calc` + returns `{change_count_idx, tree_product_count}` or nil |

### Call site in `productNeedArray`

Tree cache invalidation runs before `treeAmountProductIdx()` when materials or products changed. Then after sort, `incrementalUpdate` runs:

- Tree cache invalidation: when `product_material-estimated` or `product` has changes, `needCalc.resetWorkProductArray()` and cache cleared
- After `addSourceArray` + `treeAmountProductIdx` + `sortArray`: `incrementalUpdate(calcProductIdx, gridIdx.changeIdx, grid)`
- Result: `info.incremental_change_count_idx` and `info.incremental_tree_product_count` for stat printing

### `buildTreeDown` in `need-calc.lua`

Walks the material tree level by level from `changedIdx` using `cache.treeAmountProductIdx`. Returns ordered array of all affected products. Accepts optional `treeIdxOverride` for tests.

## Performance

| | Full Load | Incremental (10 of 1000 changed) |
|---|---|---|
| addSourceArray | all products | all products (unchanged) |
| removeRow | not needed | only affected products |
| calculateBalance | all products, 500ms | 10 + their sub-materials, ~5ms |
| cascade loop | full tree | only affected sub-tree |
| from_row restore | not needed | one pass over affected products |
| **Total** | **~660ms** | **~10-20ms** |

## Files

### What Does NOT Change

- `addSourceArray()` — still builds full `calcProductIdx` on every call
- `estimateNeed.add()` — still adds rows for all products
- `sortArray` via `runCallback` — still sorts all
- `addActionLimit` via `runCallback` — existing gate
- `checkCalculation` — `from_row` object → `from_product_id` + `from_product_row` fields
- Output diff via `setIncrementalChange()` — accepts idx maps via `iterItems`
- `nc-calc-server-load-data.lua` — already builds `changeIdx` and updates grid data in-place
- `nc-production.js` — flatten + sort + resolve already works

## Correctness Proof

**Claim**: `incrementalUpdate` + normal cascade loop produces identical `calcProductIdx` to full recalculation.

**Proof by showing every product's row array matches after incremental vs full**:

1. **Changed products found**: `collectChangedProductId` scans ALL add/modify/delete entries → every product with changed source data is in `changedIdx`. **Sound**.

2. **Affected sub-tree complete**: `buildTreeDown` walks the material tree level by level from each changed product — first all direct materials, then their materials, and so on until no more sub-materials remain. Every product that could receive a cascade row from any changed product, at any depth, is collected. **Complete**.

3. **Removed rows are stale**:
   - Deleted source rows → gone from source data, must be removed. Correct.
   - Cascade rows from changed path (`from == fromMaterial`, `origin_product_id` in `changedIdx`, on sub-materials only) → these rows were derived from OLD parent balances. Parents will recalculate with new data and produce NEW cascade rows via `addLowerLevelNeed`. Old ones are stale. Correct.
   - Cascade rows from unchanged paths (H→C where H not in `changedIdx`) → preserved. H is not recalculated, its cascade rows remain valid. Correct.

4. **All rows present at recalculation** — for any product P in `treeArray`:
   - Source rows: `addSourceArray` rebuilds from updated grid data (runs BEFORE `incrementalUpdate`). Same as full. ✓
   - Preserved cascade rows from unchanged paths: `removeRow` does not touch them. Same as full. ✓
   - New cascade rows from changed path: inserted by `addLowerLevelNeed` during cascade loop using fresh parent balances. Same as full. ✓
   → P's row array = same rows as full recalculation.

5. **Unchanged products**: `do_calc = 0`, never touched by cascade loop. Same as full. ✓

**Conclusion**: Every product has identical rows → `calculateBalance` produces identical balances. Result equals full recalculation. ∎

## Tables That Affect Product-Need Calculation

### Source tables — handled automatically

`collectChangedProductId` scans ALL tables in `serverChangeIdx` and extracts `product_id` from every row in add/modify/delete. Any table whose rows have a `product_id` field is automatically handled.

| Table | Role | Has `product_id` | Handled |
|---|---|---|---|
| `product` | Source row: stock balance | yes | automatic — `changedIdx` → tree → cascade |
| `production_order` | Source row: production amounts | yes | automatic — `changedIdx` → tree → cascade |
| `order_row-sales` | Source row: sales orders | yes | automatic — `changedIdx` → tree → cascade |
| `order_row-purchase` | Source row: purchase orders | yes | automatic — `changedIdx` → tree → cascade |

### Date calculation — handled automatically

| Table | Role | Has `product_id` | Handled |
|---|---|---|---|
| `product_work-estimated` | `materialNeedDate` reads work phase dates | yes | automatic — product picked up by `collectChangedProductId` |

### Tree structure — needs cache invalidation (handled)

| Table | Role | Has `product_id` | Handled |
|---|---|---|---|
| `product_material-estimated` | `treeAmountProductIdx` built from material data | yes | automatic — product picked up. ALSO needs tree cache invalidation: `needCalc.resetWorkProductArray()` + cache clear. Handled in `productNeedArray` where `incrementalUpdate` is called. |

When `product` table has entries: `workProductArray` (module-level in `need-calc.lua`) may be stale — new products are not in it, deleted products are still in it. Also handled by cache invalidation.

### Date calculation — handled via grid data scan (no `product_id`)

| Table | Role | Has `product_id` | Handled |
|---|---|---|---|
| `resource-load_group` | `materialNeedDate` reads `resourceIdx[resource_id]` for `extra_hours`, `last_phase_extra_hours` | no (has `resource_id`) | grid scan: `product_work-estimated` data → find products using `resource_id` → add to `changedIdx` |
| `capacity` | `materialNeedDate` reads `capacityIdx[resource_id]` for capacity intervals | no (has `resource_id`) | same grid scan as resource |

### Order state — handled via grid data scan (no `product_id`)

| Table | Role | Has `product_id` | Handled |
|---|---|---|---|
| `order-sales` | `addSourceArray` filters by `needStateIdx[linked.order_state]` | no (has `order_id`) | grid scan: `order_row-sales` data → find products with matching `order_id` → add to `changedIdx` |
| `order-purchase` | Same as order-sales | no (has `order_id`) | grid scan: `order_row-purchase` data → same |

### Tables that do not affect product-need

| Table | Why irrelevant |
|---|---|
| `product_work-schedule` | Virtual production orders + work-center only, not used in `materialNeedDate` |
| `product_work-actual` | Work-center only |
| `product_material-actual` | Work-center only |
| `resource_efficiency` | Work-center capacity only |
| `person-employee` | Work-center only |
| `company-customer` | Grid output only |
| `company-supplier` | Grid output only |
| `stock-product` | Not in product-need calc |
| `attachment` | Not in calc |

### Summary: all product-need tables handled

All tables that affect product-need calculation are handled by `collectChangedProductId`. Tables with `product_id` are picked up automatically. Tables without `product_id` (resource, capacity, orders) are handled by scanning grid data within the same loop.

## Threshold: Bail Out When Too Many Changes

When changes are massive, the overhead of lookups, tree rebuilds, `removeRow` and `restoreFromRowRef` exceeds the cost of full recalculation (~660ms).

After all lookups complete in `collectChangedProductId`, if `changedCount` exceeds `maxProductCount` (calculated as product count × `incremental_max_change_percent` from `constant.json`, default 30%), `incrementalUpdate` returns `nil`. Calling code handles `nil` return — skip incremental, set `do_calc = 1` on all products, let full cascade loop run.

Three levels:

| Changes | Action | Why |
|---|---|---|
| 0 | No incremental, return cached | No work needed |
| 1 to threshold | Incremental with lookups | Faster than full |
| > threshold | Full calc | Lookups + tree rebuild + removeRow + restoreFromRowRef cost more than full recalc |

## Updated Summary Diagram

```mermaid
flowchart TD
    S0{"changeIdx exists?"} -->|no| FULL["Full calc"]
    S0 -->|yes| S1
    S1["1. collectChangedProductId(changeIdx)<br/>extract product_id from all tables"] --> S1B{"changedCount > threshold?"}
    S1B -->|yes| FULL
    S1B -->|no| S2
    S2["2. Invalidate tree cache if<br/>materials or products changed<br/>(in productNeedArray, before treeAmountProductIdx)"] --> S3
    S3["3. buildTreeDown(changedIdx) → treeArray"] --> S4
    S4["4. removeRow: delete source rows +<br/>cascade rows from changed path"] --> S5
    S5["5. restoreFromRowRef"] --> S6
    S6["6. do_calc=1 on treeArray,<br/>do_calc=0 on rest"] --> S7
    S7["7. Normal cascade loop"] --> S8
    S8["8. change-data diff"]
```

## Reference

- Data flow: `nc-plugin/plugin/nc-calc-server/documentation/incremental-update-data-flow.md`
- Change tracking: `nc-doc/technical/change-data/incremental-update-work-center.md`
- Work-center incremental: `form/nc/nc-work-center/documentation/dev/work-center-incremental-update.md`
- Work-center needs NO code changes — existing calc runs on updated grid data, `serverChangeIdx` passes through directly
- Parent: `calc-server-plan.md`
