# Incremental Update Data Flow

## changeIdx Format

All three change types use hash maps keyed by id. Counts at both top and per-table level.

### Wire format (JSON — Lua to browser, zstd compressed)

All change messages are **zstd compressed** before publishing to NATS. The browser's `parseMessageData()` detects zstd magic bytes and decompresses transparently before JSON parsing. When total change count exceeds `max_change_count` (default 500 in `nats-grid-config.lua`), `sendChangeMessage()` splits by table into separate messages — each message has identical format `{message_type, bucket, change_idx, grid_name}` with one table per `change_idx`. The browser receives each message independently and calls `mergeChangeArr()` per message with a callback for async processing via `processChunked`; since `mergeTableChange` processes per-table independently, multiple smaller messages work correctly without any reassembly logic.

```mermaid
flowchart LR
    S["changeIdx"] --> C{"total > max_change_count?"}
    C -->|no| J["toJsonRaw()"]
    C -->|yes| SPL["split by table"]
    SPL --> J
    J --> Z["zstd compress"]
    Z --> P["publish"]
    P --> N["NATS"]
    N --> B["browser parseMessageData()<br/>isZstd() → decompress"]
    B --> A["mergeChangeArr() per message"]
```

```json
{
    "add_count": 5,
    "modify_count": 3,
    "delete_count": 1,
    "product_work": {
        "add_count": 3,
        "modify_count": 1,
        "delete_count": 0,
        "add": {
            "id4": { "record_id": "id4", "field1": "val4", "amount": 100 },
            "id5": { "record_id": "id5", "field1": "val5", "amount": 200 }
        },
        "modify": {
            "id1": { "record_id": "id1", "field1": "updated" }
        },
        "delete": {}
    },
    "person": {
        "add_count": 2,
        "modify_count": 2,
        "delete_count": 1,
        "add": {
            "p3": { "record_id": "p3", "name": "New Person" }
        },
        "modify": {
            "p1": { "record_id": "p1", "name": "Changed" },
            "p2": { "record_id": "p2", "status": "active" }
        },
        "delete": {
            "p4": { "record_id": "p4", "modify_time": "2026-04-22T10:00:00Z" }
        }
    }
}
```

### Rules

- `record_id` is the **only** matching key — primary keys like `production_order_id` or `material_id` can change between rows, so they must NEVER be used for change matching
- `add`/`modify` = hash map `record_id → full row` — O(1) lookup, always includes `modify_time`
- `delete` = hash map `record_id → {record_id, modify_time}` — minimal row with time for conflict resolution
- `row` = full row data, NO extra keys (`change`, `calc` — NONE)
- Counts are set by backend, used by frontend for stats

### What this replaces

| Old | New | Why |
|-----|-----|-----|
| `changedRow` (change-type-first) | `changeIdx` (table-first) | Restructure + rename |
| `change_idx` arrays with `change` key | `changeIdx` hash maps, no extra keys | No arrays, no `change` key |
| `changeData.setIncrementalChange()` cache arrays | `changeIdx` direct tracking | For work-center |
| `field_diff` in modify items | full row in modify hash | Browser replaces entire row |
| `deleted = {[id] = true}` | `delete = {[id] = {record_id, modify_time}}` | Consistent hash map format + time for conflict resolution |

---

## Browser Processing — `mergeTableChange()` Algorithm

### recordIdIdx — O(1) lookups by `record_id`

A `record_id → array index` hash map is built on first call and reused across all subsequent change messages. When the grid data array is replaced (full reload), the reference check detects this and rebuilds the index.

```text
gridRef.recordIdIdx = { "test-00000042": 41, "test-00000015": 14, ... }
gridRef.recordIdIdxRef = dataArray  // reference check for auto-rebuild
```

All three change types use direct O(1) lookups — no scanning, no `pendingCount` needed:

```mermaid
flowchart TD
    B["Build recordIdIdx<br/>{record_id → array index}<br/>if null or data replaced"] --> D
    subgraph D["Process deletes"]
        D1["for id in deleteIdx"] --> D2["idx = recordIdIdx[id]"] --> D3{"idx?"}
        D3 -- "found" --> D4["dataArray[idx].calc.change = 'delete'"]
        D3 -- "null" --> D5["skip"]
    end
    D --> M
    subgraph M["Process modifies"]
        M1["for id in modifyIdx"] --> M2["idx = recordIdIdx[id]"] --> M3{"idx?"}
        M3 -- "found" --> M4["copy fields + detect changed fields"]
        M3 -- "null" --> M5["skip"]
    end
    M --> A
    subgraph A["Process adds"]
        A1["Collect new adds"] --> A2["Sort by idx descending"]
        A2 --> A3["for addRow in sorted"]
        A3 --> A4{"recordIdIdx[id]?"}
        A4 -- "exists" --> A5["dataArray[idx] = addRow<br/>preserve changeArray"]
        A4 -- "null" --> A6["splice at idx-1<br/>or push if beyond end"]
        A6 --> A7["reIndexArray()<br/>sequential idx = i+1"]
    end
    A --> DC["nc.grid.dataChanged()"]
```

### Code — the actual lookups

```javascript
// Build record_id → array index if missing or data array was replaced
let recordIdIdx = gridRef.recordIdIdx
if (recordIdIdx == null || gridRef.recordIdIdxRef !== dataArray) {
    recordIdIdx = {}
    for (let i = 0; i < dataArray.length; i++) {
        const id = dataArray[i].record_id
        if (id != null) recordIdIdx[id] = i
    }
    gridRef.recordIdIdx = recordIdIdx
    gridRef.recordIdIdxRef = dataArray
}

// Process deletes - O(1) per record
for (const id in deleteIdx) {
    const idx = recordIdIdx[id]
    if (idx == null) continue
    dataArray[idx].calc.change = 'delete'
}

// Process modifies - O(1) per record
for (const id in modifyIdx) {
    const idx = recordIdIdx[id]
    if (idx == null) continue
    // Copy new values + detect changed fields
    for (const k in modRow) item[k] = modRow[k]
}

// Process adds - sort descending by idx so inserts don't shift each other
const newAddArray = []
for (const id in addIdx) {
    const addRow = addIdx[id]
    const existingIdx = recordIdIdx[id]
    if (existingIdx != null) {
        dataArray[existingIdx] = addRow  // replace in place
    } else {
        newAddArray.push(addRow)
    }
}
newAddArray.sort((a, b) => (b.idx ?? 0) - (a.idx ?? 0))
for (const addRow of newAddArray) {
    const targetIdx = addRow.idx - 1
    if (targetIdx >= dataArray.length) {
        dataArray.push(addRow)
    } else {
        dataArray.splice(targetIdx, 0, addRow)
    }
}
reIndexArray(dataArray, recordIdIdx)  // sequential idx = i+1
```

### Complexity

| Scenario | Old (single-pass scan) | New (recordIdIdx) |
|----------|------------------------|-------------------|
| 100k rows, 100 changes in 3 messages | 3 × O(100k) scans = O(300k) | O(100k) build + 3 × O(100) lookups |
| 100k rows, 0 changes | O(100k) scan | O(100k) build (once), O(1) after |
| Second call with same data | O(100k) scan again | O(0) — index reused |

---

## NATS Visual Change Tracking with `change_state`

### Why indicators are separate from data

Old approach mutated rows with `__changed`, `__changed_time`, `_viewed_modify_id`. Problems:

1. Grid redraws create new row objects — `_viewed_modify_id` orphaned on old object
2. Boolean `__changed` — no field-level indicators, whole-row only
3. `__changed` pollutes data model — UI state belongs in UI layer

Solution: `state.rec.change_state` — a separate lookup keyed by `record_id`, survives grid redraws.

### Data structure

```javascript
state.rec.change_state = {
    "message": {
        viewedIdx: {},         // record_id → true (user clicked = acknowledged)
        changedFieldIdx: {     // record_id → { field: true }
            "test-00000042": { amount: true, resource_id: true },
            "test-00000015": { work_phase_state: true }
        },
        changeTypeIdx: {}      // record_id → 'add' | 'modify' | 'delete'
    }
}
```

### Lifecycle: detect → display → dismiss

```mermaid
flowchart LR
    subgraph Detect["1. DETECT"]
        A["changeIdx from NATS"] --> B["mergeTableChange()<br/>old vs new field compare"]
        B --> C["changedFieldIdx<br/>changeTypeIdx"]
    end
    subgraph Display["2. DISPLAY"]
        C --> D["Formatter reads<br/>changeState directly"]
        D --> E["nats-row-new/changed/deleted<br/>nats-cell-changed"]
    end
    subgraph Dismiss["3. DISMISS"]
        F["Click change icon<br/>(circle-xmark)"] --> G{"change type?"}
        G -- "delete" --> H["splice row from array<br/>cleanup changeState"]
        G -- "add/modify" --> I["viewedIdx[id] = true"]
        I --> J["nc.grid.redrawRows()"]
        J --> K["Formatter re-runs<br/>icon + classes removed"]
    end
    Detect --> Display --> Dismiss
```

### Data flow sequence

```mermaid
sequenceDiagram
    participant Lua as Lua Backend
    participant NATS as NATS
    participant JS as nc-nats.js
    participant CD as change-data.js
    participant Grid as SlickGrid
    participant User as User

    Note over Lua,User: Initial Load
    Lua->>NATS: sendGrid(bucket, gridData)
    Note over JS: subscribeWithVersioning subscribes to changes (deliver_policy: by_start_sequence) then grid (from seq 1)
    NATS->>JS: Grid messages
    JS->>Grid: state.grid.message.data = rows

    Note over Lua,User: Modify arrives
    Lua->>NATS: changeIdx with modify[id] = row
    NATS->>JS: change_idx
    JS->>CD: callback(parsedMsg) — each message independently
    CD->>CD: compare old vs new fields
    CD->>CD: changedFieldIdx[id] = { amount: true }
    CD->>Grid: data updated in place
    Grid->>Grid: formatter → nats-cell-changed
    Grid->>User: Changed cells red bold

    Note over User,Grid: User clicks change icon (circle-xmark)
    User->>Grid: Click icon in idx cell
    Grid->>JS: changeIconClicked(area, rowIndex)
    JS->>JS: delete → splice row, add/modify → viewedIdx[id] = true
    JS->>Grid: dataChanged() → invalidate()
    Grid->>User: Icon + highlight gone
```

### Built-in change visualization (formatter)

Change visualization is built into the SlickGrid formatter (`nc-grid-slick.js`). Every grid that receives changes via `mergeChangeArr` gets automatic row and cell highlighting — no per-grid column JSON configuration needed.

The formatter reads from `grid.state.rec.change_state[gridName]` — NOT from mutated row data. This means:

- No `row_class_rules` needed in column JSON for change visualization
- No `cell_class_rules` needed for field-level highlighting
- `calc.change` mutation stays for backward compat (work-center grids, NATS "Change" column)

```javascript
// Inside the formatter — runs for every cell render
const changeState = grid.state?.rec?.change_state?.[grid.gridName]
if (changeState && dataContext?.record_id != null) {
    const id = dataContext.record_id
    if (changeState.changeTypeIdx[id]) {
        // adds nats-row-new / nats-row-changed / nats-row-deleted
    }
    if (changeState.changedFieldIdx[id]?.[colDef.variable] && !changeState.viewedIdx[id]) {
        // adds nats-cell-changed
    }
}
```

| Class | When |
|-------|------|
| `nats-row-new` | `changeTypeIdx[id] === 'add'` |
| `nats-row-changed` | `changeTypeIdx[id] === 'modify'` |
| `nats-row-deleted` | `changeTypeIdx[id] === 'delete'` |
| `nats-cell-changed` | `changedFieldIdx[id][field]` exists AND not viewed |

### CSS

```css
.nats-row-new      { color: ForestGreen !important; }
.nats-row-changed  { color: darkblue !important; }
.nats-row-deleted  { color: darkred !important; }
.nats-cell-changed { color: #cb0404 !important; }

.nc-change-icon {
    cursor: pointer;
    vertical-align: middle;
    margin-left: 4px;
    opacity: 0.6;
    transition: opacity 0.15s, transform 0.15s;
}
.nc-change-icon:hover {
    opacity: 1;
    transform: scale(1.3);
}
```

### Change icon (circle-xmark)

The formatter appends a `circle-xmark` SVG icon in the `idx` column for rows with unviewed changes. The icon color matches the change type:

| Change type | Icon color |
|-------------|-----------|
| `add` | `ForestGreen` |
| `modify` | `darkblue` |
| `delete` | `darkred` |

Clicking the icon:

- **delete rows**: removes the row from the array via `splice`
- **add/modify rows**: sets `viewedIdx[id] = true` and redraws, removing the icon and CSS classes

The icon is rendered as inline SVG by `changeIconSvg(color)` from `nc-svg-icon.js`.

### Change message flow (nc-nats.js)

`changeMessage()` receives each NATS change message, sums `*_count` from `change_idx` for stats, then calls `mergeChangeArr` with a callback for async processing:

```javascript
change.mergeChangeArr(state, changeData.change_idx, result => {
    accumulateFlushResult(result)
    updateChangeLabel()
    nc.grid.redrawRows(state, 'message')
})
```

`updateChangeLabel()` shows: `1 message | applied 33/33: 10/10 deleted, 13/13 changed, 14/14 added | grid: 100,004 rows`

### Files

| File | Role |
|------|------|
| `change-data.js` | `mergeTableChange()` — single-pass loop, builds `changedFieldIdx` + `changeTypeIdx` |
| `change-data.js` | Initializes per-table `change_state[tableName]` on first change |
| `nc-nats.js` | `changeMessage()` sums `*_count` for stats, calls `mergeChangeArr` async |
| `nc-nats.js` | `changeIconClicked()` handles icon click: splice deletes, clear viewed for add/modify |
| `nc-nats.js` | `gridRowDoubleClicked()` is no-op (only icon click removes changes) |
| `nc-nats.js` | `startTest()` / `clearGrid()` reset `state.rec.change_state = {}` |
| `nc-grid-slick.js` | Built-in formatter reads `changeState` for row/cell classes + change icon |
| `nc-grid-slick.js` | `ensureUniqueIdx()` prevents duplicate idx before `setItems` (sort crash fix) |
| `nc-grid-slick.js` | `gridDataChanged(createIdx=false)` runs `ensureUniqueIdx` for sort path |
| `nc-grid-style.scss` | CSS classes for visual indicators (`nats-*`) + `.nc-change-icon` hover |
| `nc-svg-icon.js` | `changeIconSvg(color)` generates inline SVG for change icon |
| `nc-grid-common.js` | `ncChangeIconClicked()` emits `grid-change-icon-clicked` event |
| `slick-selection.js` | onClick detects `.nc-change-icon` clicks, calls `ncChangeIconClicked()` |

---

## Database Sync (4D → PostgreSQL → Browser)

### Complete data flow

```mermaid
flowchart TD
    subgraph STEP1["STEP 1: Detect & Build changeIdx"]
        A1["setIncremental(true)<br/>Reset counts"] --> A2["Query DB with<br/>last_modify_id filter"]
        A2 --> A3{"Row in grid?"}
        A3 -- "NOT in grid" --> A4["changeIdx[t].add[id] = row<br/>add_count++"]
        A3 -- "EXISTS (incremental)" --> A5["util.recToRec() update<br/>changeIdx[t].modify[id] = row<br/>modify_count++"]
        A3 -- "Same DB, non-incremental" --> A6["Discard duplicate"]
        A4 --> A7["Binary search: deleted<br/>changeIdx[t].delete[id] = {record_id}<br/>delete_count++"]
        A5 --> A7
        A7 --> A8["gridIdx.changeIdx = getChangeIdx()"]
    end

    subgraph STEP2["STEP 2: Pass changeIdx to Calc"]
        B1["gridIdx.changeIdx"] --> B2A["work-center<br/>param.changeIdx = pass-through"]
        B1 --> B2B["product-need<br/>changeData.buildChangeIdx()<br/>(cache comparison)"]
        B2A --> B3["Update last_modify_id"]
        B2B --> B3
    end

    subgraph STEP3["STEP 3: Apply (change-data.js)"]
        C1["response.change_idx"] --> C2["mergeChangeArr()<br/>single pass, O(1) lookups"]
        C2 --> C3["Grid + last_modify_id updated"]
    end

    STEP1 --> STEP2 --> STEP3
```

### changeIdx structure diagram

```mermaid
graph TD
    CI["changeIdx"] --> AC["add_count: 5"]
    CI --> MC["modify_count: 3"]
    CI --> DC["delete_count: 1"]
    CI --> T1["tableName"]

    T1 --> TAC["add_count: 3"]
    T1 --> TMC["modify_count: 1"]
    T1 --> TDC["delete_count: 0"]
    T1 --> ADD["add<br/>{[id] = row}"]
    T1 --> MOD["modify<br/>{[id] = row}"]
    T1 --> DEL["delete<br/>{[id] = {record_id}}"]

    style CI fill:#e1f5fe
    style T1 fill:#f3e5f5
    style ADD fill:#e8f5e9
    style MOD fill:#fff3e0
    style DEL fill:#ffebee
```

### Two incremental approaches

| Module | Approach | Why |
|--------|----------|-----|
| `work-center.lua` | Direct from load-data | Output rows ARE input rows |
| `product-need.lua` | Cache comparison via `buildChangeIdx()` | Output is CALCULATED (balance, need_down) |

```mermaid
flowchart LR
    subgraph WC["work-center: pass-through"]
        WC1["load-data"] -->|"changeIdx"| WC2["work-center.lua"]
        WC2 --> WC3["param.changeIdx"]
    end
    subgraph PN["product-need: cache diff"]
        PN1["load-data"] -->|"input rows"| PN2["product-need.lua"]
        PN2 -->|"compare<br/>with cache"| PN3["buildChangeIdx()"]
        PN3 --> PN4["param.changeIdx"]
    end
```

### state.grid after mergeChangeArr

```javascript
state.grid.product_work.data = [
    { record_id: "id1", field1: "updated", calc: { change: "modify", changeArray: [] } },
    { record_id: "id2", field1: "val2",    calc: { change: "delete", changeArray: [] } },
    { record_id: "id3", field1: "val3",    calc: { change: "add",    changeArray: [] } }
]
```

- `calc.change` = last operation on this row
- `calc.changeArray` = accumulated history (preserved across replacements)
- Deleted rows stay in array (marked, not removed)

---

## Tests

### Frontend — change-data

```bash
cd nc-frontend && bun vitest run change-data   # 10ms
```

Key cases:

- Add new row → `calc.change = "add"`, spliced at correct position
- Add rows sorted by idx descending to prevent position shift
- `reIndexArray()` assigns sequential idx after all adds
- `ensureUniqueIdx()` in `gridDataChanged` prevents sort crash on duplicate idx
- Modify existing → fields updated in place, `calc.change = "modify"`
- Delete → `calc.change = "delete"`, row NOT removed from array
- Add replacing existing → row replaced in place, `changeArray` preserved
- Mixed changes in single call
- Multiple tables in single call
- Non-existent modify/delete → skipped (returns 0)
- `incrementalUpdate()` returns `true` for incremental, `false` for full
- idx uniqueness tests: adds at positions, same-position adds, sort simulation

### Frontend — nats-grid

```bash
cd nc-frontend && bun vitest run nats-grid   # 6ms
```

Key cases:

- Grid message assembly (multi-chunk, out-of-order)
- Change message send/receive round-trip
- Multi-table changeIdx
- Chat and presence messages
- Message chunking with custom sizes

### Backend — change-data

```bash
cd nc-server && lj test/lib/change-data-test.lua   # <0.01s
```

### Backend — nats-grid

```bash
cd nc-server && lj test/lib/nats-grid-test.lua      # <0.01s
```

---

## File Paths

| Component | Path |
|-----------|------|
| Change data (JS) | `nc-frontend/src/core/change-data.js` |
| Change icon SVG | `nc-frontend/src/core/nc-svg-icon.js` |
| NATS grid (JS) | `nc-frontend/src/core/nats-grid.js` |
| NATS admin (JS) | `nc-backend/preference/form/nc/nc-admin/nc-nats/nc-nats.js` |
| NATS admin JSON | `nc-backend/preference/form/nc/nc-admin/nc-nats/nc-nats.json` |
| Grid common | `nc-frontend/src/components/nc-grid/nc-grid-common.js` |
| Grid slick | `nc-frontend/src/components/nc-grid/nc-grid-slick/nc-grid-slick.js` |
| Grid slick vue | `nc-frontend/src/components/nc-grid/nc-grid-slick/nc-grid-slick.vue` |
| Slick selection | `nc-frontend/src/components/nc-grid/nc-grid-slick/option/slick-selection.js` |
| Grid styles | `nc-frontend/src/components/nc-grid/nc-grid-style.scss` |
| Change data tests | `nc-frontend/test/change-data.test.js` |
| NATS grid tests | `nc-frontend/test/nats-grid.test.js` |
| NATS test (Lua) | `nc-server/lib/message/nats/nc-nats-test.lua` |
| Change data (Lua) | `nc-server/lib/db/change-data.lua` |
| NATS grid (Lua) | `nc-server/lib/message/nats/nats-grid.lua` |
| Load data | `nc-plugin/plugin/nc-calc-server/nc-calc-server-load-data.lua` |
| Calc server | `nc-plugin/plugin/nc-calc-server/nc-calc-server-calc.lua` |
| Work center | `nc-backend/preference/form/nc/nc-work-center/nc-calc-server-work-center.lua` |
| Binary search | `nc-backend/plugin/db-sync/db-sync-binary-search.lua` |
