# Product-Need Incremental Calculation

## Problem

`productNeedArray()` recalculates ALL products every call. `calculateBalance` is O(n²) per product — the bottleneck.

## Why We Cannot Partially Calculate a Product

Product C has 30 rows from parents A, B, X, D, E, F. Balance is cumulative: row N's balance depends on row N-1. You cannot update "only A's rows" in C — inserting or removing ANY row shifts all subsequent balances. When C is touched, ALL of C must be recalculated.

This means the incremental approach must determine WHICH products need recalculation, then fully recalculate each of them.

## The Incremental Approach

### Example BOM tree

```text
A → B → C
X → D → C
E → C
```

C is a sub-material of B, D, E. A is a parent of B. X is a parent of D.

### Step 1: Find changed products from `serverChangeIdx`

`serverChangeIdx` has `{product = {add, modify, delete}, production_order = {add, modify, delete}, order_row-sales = ..., order_row-purchase = ...}`. Each row has `product_id`. Collect changed product IDs.

Example: production order for **A** modified → changed set = `{A}`.

### Step 2: Build tree DOWN — find all sub-materials

From changed products, follow the BOM tree DOWN through all levels. These sub-products will be affected by cascade (`need_down` flows from parent to child material).

Example: A changed → tree down → B is A's material → C is B's material. Affected down set = `{A, B, C}`.

D and E are NOT in the down set — A's change does not cascade through them.

### Step 3: Build tree UP — find all upper-level products that feed INTO affected products

From the affected set `{A, B, C}`, find ALL products that have any of these as sub-materials. These upper products may have `need_down` rows flowing INTO our affected products. When we reset affected products, those `need_down` rows must be re-added.

Example: who uses C as material? → H uses C, J uses C. Upper set = `{H, J}`.

X uses D which uses C, but X/D are NOT in the affected set, so X is NOT in the upper set. D is not affected because A's change doesn't cascade through X→D→C — D's source data is unchanged.

### Step 4: Reset affected products to source rows

For each product in affected set `{A, B, C}`:

- Remove ALL rows that were built by cascade (rows with `from = fromMaterial`)
- Keep only source rows (from `addSourceArray`: stock, production orders, sales orders, purchase orders)
- For directly changed products (A), source rows already have updated data from `serverChangeIdx` (incremental load updated the grid data before calculation)

Upper products `{H, J}` are NOT reset — they keep their fully calculated data with all cascade rows intact.

### Step 5: Calculate affected products

- Affected products `{A, B, C}` have `do_calc = 1`
- Upper products `{H, J}` have `do_calc = 0` — they are in `calcProductIdx` but not recalculated. Their existing `need_down` rows flow into affected products during the cascade loop.
- All other products have `do_calc = 0` — not recalculated, not touched

Calculation runs normally:

1. `calculateBalance(A)` → produces `need_down` for B
2. `addLowerLevelNeed` inserts rows into B → `do_calc = 1` on B
3. `calculateBalance(B)` → produces `need_down` for C
4. `addLowerLevelNeed` inserts rows into C → `do_calc = 1` on C
5. Upper products H, J: their existing `need_down` rows are already in the `needDownArray` from the previous calculation pass OR re-collected from their existing data — so their rows flow into C correctly
6. `calculateBalance(C)` → runs on full array: source rows + A's cascade rows + H/J's cascade rows → correct result

### Step 6: Compare new to old using `change-data`

After calculation:

- `setIncrementalChange` compares recalculated products against cached previous result
- Products that changed → add/modify/delete entries sent to frontend
- Products that didn't change (same result as before) → no entries, silently skipped
- New data merged into cache → cache stays complete for next incremental cycle

```mermaid
flowchart TD
    S1["1. serverChangeIdx → changed: A"] --> S2["2. Tree DOWN: A→B→C"]
    S2 --> S3["3. Tree UP: who uses C? → H, J"]
    S3 --> S4["4. Reset A,B,C to source rows"]
    S4 --> S5["5. Calculate: A,B,C do_calc=1, H,J do_calc=0"]
    S5 --> S6["6. change-data diff new vs old → send changes"]
```

## Why This Approach Is Correct

**C gets ALL its rows, not just A's path:**

- Source rows from `addSourceArray` (stock, production orders, sales, purchase)
- Cascade rows from A→B→C (recalculated because A changed)
- Cascade rows from H→C and J→C (from upper products, preserved or re-added)
- Balance recalculated across ALL rows → correct cumulative result

**X and D are NOT recalculated:**

- X's source data didn't change
- X→D→C path is untouched
- C's rows from X→D path may be re-created during cascade, but X itself doesn't run `calculateBalance`

## No Flatten — `calcProductIdx` is idx map

`addSourceArray` builds `{[product_id] = {row1, row2, ...}}`. All functions (`calculateBalance`, `addActionLimit`, `checkCalculation`) iterate per-product via `runCallback`. JS flattens client-side.

`checkCalculation` sets `product_row = i` (ipairs index). `from_row` is stored as `"productId:productRow"` string (e.g. `"P001:3"`). JS resolves it to a flat array index after flatten+sort.

## `first_row`/`last_row` moved to JS

All functions use `i == 1` / `i == #productArray` directly. JS sets `first_row`/`last_row` during flatten for CSS rules.

## Files Changed

| File | Change |
|------|--------|
| `product-need.lua` | `calculateBalance` receives `productArray`, no boundary detection, uses index checks |
| `product-need.lua` | `addActionLimit` receives `productArray`, uses `i == count` |
| `product-need.lua` | `checkCalculation` per-product iteration with inner loops |
| `product-need.lua` | Flatten removed, `need.calcProductIdx = calcProductIdx` (idx map) |
| `product-need.lua` | Incremental: reset affected products, mark `do_calc`, compare via change-data |
| `product-need.lua` | `needColumn.data = {}`, idx map via `resultCache.data` |
| `product-need.lua` | `gridProductCache[orgId]` — stripped product rows (whitelisted fields only), substituted in both return paths |
| `change-data.lua` | `iterItems` helper iterates both arrays and idx maps with nested `pairs`/`ipairs` |
| `change-data.lua` | `lastModifyId` accepts idx maps via `iterItems` |
| `change-data.lua` | `setIncrementalChange` accepts idx maps via `iterItems` + `modify_id == nil` fallback for calculated data |
| `estimate-need.lua` | Removed dead `last_row` break, use `originalProductCount` loop bound |
| `nc-production.js` | Flatten idx map → set `product_row` → sort → assign `idx` → build lookup → resolve `from_row`/`calc_idx` → set `first_row`/`last_row` |
| `nc-production.js` | `updateData()` reads `product_need` from `ret.data` not `ret.grid` |
| `nc-production.js` | `calc_idx` resolved via `gridDataRowLookup` in `updateData` and `rebuildAfterIncremental` |
| `nc-production.js` | `weekly_need` sent in `callServer` parameter, Lua sends only active tab's weekly data |
| `nc-production.js` | All `createIndex`/`createIndexArray` calls on weekly data guarded with `if (gridDataWeekly)` |

## Data Flow: Lua → JS

### Lua sends data via `ret.data`, grid column `.data` is always `{}`

`needColumn.data = {}` — grid column has no data. All product_need data is transported via `resultCache.data`:

```lua
needColumn.data = {}
weeklyNeedColumn.data = {}

resultCache[orgId].data = {
    chart_option = chartOption,
    product_need = calcProductIdx,
    product_need_weekly = weeklyNeedArray,
    product = grid["product"].data
}

-- Return paths substitute product with gridProductCache[orgId]
-- (stripped rows with only whitelisted fields, ~60-80% smaller)
gridProductCache[orgId] = { stripProductRow(row) for each row }
```

### JS reads from `ret.data` on full load, `state.data` on incremental

Both paths end with `gridData` = flat sorted array.

```mermaid
flowchart TD
    subgraph "Full Load"
        FL1["Lua: needColumn.data = {}"]
        FL2["Lua: resultCache.data.product_need = calcProductIdx (idx map)"]
        FL3["nc.setState(state, 'data', ret.data)"]
        FL4["updateData(ret)"]
        FL5["flattenProductNeed(ret.data.product_need)"]
        FL1 --> FL2 --> FL3 --> FL4 --> FL5
    end
    subgraph "Incremental Load"
        IL1["Lua: resultCache.grid = nil"]
        IL2["Lua: setIncrementalChange diffs calcProductIdx"]
        IL3["nc.incrementalUpdate: mergeChangeArr → state.data"]
        IL4["rebuildAfterIncremental()"]
        IL5["flattenProductNeed(state.data.product_need)"]
        IL1 --> IL2 --> IL3 --> IL4 --> IL5
    end
```

### JS: `flattenProductNeed(source)`

Flattens idx map or array → sets `product_row` per product array → sorts by `product_id`+`product_row` → assigns `idx` → builds lookup → resolves `from_row` strings → sets `first_row`/`last_row`.

### JS: `createIndex` calls — `gridData` always set, `gridDataWeekly` guarded

`gridData` is always set by `flattenProductNeed`. `gridDataWeekly` can be `undefined` when no weekly data — calls on it must be inside `if (gridDataWeekly)` guard. No `{}` fallbacks.

## Bugs Found and Fixed

### Bug 1: Delete entries lacked `product_id`

**File**: `nc-calc-server-load-data.lua:307`

Delete entry was `{record_id = id}` — no `product_id`. For `production_order`, `order_row` tables, could not find which product to recalculate.

**Fix**: Include `product_id = item.product_id` in delete entries when available.

### Bug 2: `setIncrementalChange` did not detect disappeared calculated rows

**File**: `change-data.lua:233-273`

`iterItems` processed only items IN the new data. Cached items that DISAPPEARED (e.g. production order deleted → product_need row gone from output) were never detected as deletes. Old rows stayed in cache forever → stale data on frontend.

**Fix**: After `iterItems`, scan cache for items whose `product_id` was recalculated but whose `record_id` is not in the new data → create delete entries.

### product_id change and swap scenarios

Product table uses `product.product_id` as `primaryKeyField`.

**product_id changed (A → B)**: Binary search: old "A" missing → delete. New "B" not in cache → add. B recalculated, A gone. Correct.

**product_id swapped (A ↔ B)**: Both treated as delete + add. Both recalculated. Correct.

## What Does NOT Change

- `addSourceArray()` — still builds full calcProductIdx (in-memory, fast)
- `estimateNeed.add()` — still adds rows for all products
- `sortArray` via `runCallback` — still sorts all
- `addActionLimit` via `runCallback` — existing gate
- Output diff via `setIncrementalChange()` — accepts idx maps via `iterItems`
- CSS rules in `product-need.json` — still read `first_row`/`last_row` (set by JS)

## Performance

| | Full Load | Incremental (10 of 1000 changed) |
|---|---|---|
| calculateBalance | 500ms | 5ms ← THE WIN |
| global sort + renumber | eliminated | eliminated |
| **Total** | **~660ms** | **~160ms** |

## Plan: Remove Old Code, Add New

### DONE — DELETE from `product-need.lua`

- `markChangedProduct` function — deleted
- `resetAndMarkChanged` function — deleted
- Export `productNeed.flagDirtyProduct = flagDirtyProduct` — deleted
- Inline code replaces `changeData.flagDirtyProduct` call (lines 1168-1204): saves `prevBalancePassCountIdx`, resets `do_calc = 0` on all, marks changed products from `serverChangeIdx` with `do_calc = 1`

### DONE — DELETE from `change-data.lua`

- `flagDirtyProduct` function — deleted

### DONE — DELETE from `nc-calc-server-test.lua`

- Test functions for `markChangedProduct` and `resetAndMarkChanged` — deleted

### DONE — ADD to `nc-calc-server-test.lua`

- `test.calcServer.incrementalUpdateReturnsChangeIdx()` — full calc then incremental, verifies `change_idx` structure has `{add_count, modify_count, delete_count}` and per-table entries

### REMAINING — ADD to `need-calc.lua`

Two new functions for tree traversal (used by incremental logic in `product-need.lua`):

- **`buildTreeDown(productIdArray)`** — given a set of product IDs, follow BOM tree DOWN through all levels. Returns set of ALL sub-material product IDs at every level. Uses existing `treeAmountProductIdx` structure.

- **`buildTreeUp(productIdArray)`** — given a set of product IDs, find ALL products that have any of these as sub-materials. Returns set of upper-level product IDs. Uses existing `treeAmountProductIdx` structure (reverse lookup).

### REMAINING — ADD to `product-need.lua`

New inline logic in `productNeedArray()` replacing `prevBalancePassCountIdx` approach:

1. If `serverChangeIdx` exists:
   - Collect changed product IDs from all tables in `serverChangeIdx` (generic `pairs` loop, extract `product_id`)
   - Call `needCalc.buildTreeDown(changedSet)` → affected down set
   - Call `needCalc.buildTreeUp(affectedDownSet)` → upper set
   - Reset affected products to source rows (remove `from = fromMaterial` rows)
   - Set `do_calc = 1` on affected products, `do_calc = 0` on everything else
2. If no `serverChangeIdx` → full calculation (all `do_calc = 1`), no reset

After calculation:

- `changedProductIdx` = all products in affected set that were recalculated
- `setIncrementalChange` compares against cache, sends only actual changes
- Merge new into cache

### REMAINING — ADD to `nc-calc-server-test.lua`

New tests for tree traversal logic:

- Tree down traversal: given product, finds all sub-materials at all levels
- Tree up traversal: given products, finds all parents that use them as materials
- Reset to source rows: cascade rows removed, source rows kept
- Full incremental cycle: change → reset → recalculate → diff
- Upper product need_down preserved: C gets rows from both changed path (A→B→C) and unchanged path (H→C)

## Reference

- Work-center: `form/nc/nc-work-center/documentation/dev/work-center-incremental-update.md`
- Work-center needs NO code changes — existing calc runs on updated grid data, `serverChangeIdx` passes through directly
- Parent: `calc-server-plan.md`
