# Incremental Update Data Flow

## NATS Incremental Update with Visual Change Tracking

### Context

This feature implements incremental data updates for NATS-based grids with visual change tracking. When data arrives via NATS push, users need to see what changed since they last viewed the data.

### User Requirements

1. **Persist highlight until user clicks** - Changes come as push, user may not be at screen
2. **Flash animation then persistent highlight** - Visual feedback when change arrives, then subtle persistence
3. **Click to clear** - User clicks row to acknowledge they've seen the change
4. **Use modify_id** - ISO timestamp string to track what user has viewed

### Design Decisions

| Aspect | Decision | Rationale |
|--------|----------|-----------|
| Storage | `_viewed_modify_id` in each row | Simple, follows existing patterns |
| modify_id | ISO timestamp string | Human readable, works with `dt.currentDateTimeUtc()` |
| Test method | Send same record_id again with new modify_id | Natural incremental update flow |
| Highlight | Flash (1s) + persistent CSS | Catch attention, then remain for review |

### Current State Analysis

#### Existing Infrastructure (Works - Don't Change)

| Component | File | Status |
|-----------|------|--------|
| NATS message handling | `/Volumes/nc/nc-frontend/src/core/nats-grid.js` | ✅ Working |
| Test data generation | `/Volumes/nc/nc-server/lib/message/nats/nats-grid-test.lua` | ✅ Working |
| CSS classes | `nc-grid-style.scss` | ✅ Exists (`nc-grid-row-changed`, `nats-changed`) |
| Row click events | `nc-grid-slick-event.js` | ✅ Wired up |
| change-data.js | `/Volumes/nc/nc-frontend/src/core/change-data.js` | ✅ Has patterns to use |

#### Current Highlight Implementation (Needs Fix)

**File**: `/Volumes/nc/nc-backend/preference/form/nc/nc-admin/nc-nats/nc-nats.js`

**Problem**: Lines 373-389 auto-clear after 1 second. Doesn't persist for user review.

```javascript
// CURRENT (WRONG):
function highlightChangedRows(rowIdxArray) {
    for (const idx of rowIdxArray) {
        const row = state.grid.test_message?.data?.[idx - 1]
        if (row) {
            row.__changed = true
        }
    }
    setTimeout(() => {
        for (const idx of rowIdxArray) {
            const row = state.grid.test_message?.data?.[idx - 1]
            if (row) {
                row.__changed = false  // PROBLEM: Auto-clears!
            }
        }
        nc.grid.refreshCells(state, 'test_message')
    }, 1000)
}
```

### Implementation Plan

#### Phase 1: Backend - Add modify_id to Test Data

**File**: `/Volumes/nc/nc-server/lib/message/nats/nats-grid-test.lua`

**Change**: Add `modify_id` field to generated test rows

```lua
-- Line 58-61: Modify generateTestRow function
function natsTest.generateTestRow(idx, modifyId)
    local stateArray = {'ready', 'setup', 'working', 'done'}
    return {
        idx = idx,
        record_id = string.format("test-%08d", idx),
        modify_id = modifyId or dt.currentDateTimeUtc(),  -- NEW: ISO timestamp
        resource_id = string.format("RES-%d", (idx % 10) + 1),
        product_id = string.format("PROD-%d", (idx % 100) + 1),
        work_phase_state = stateArray[(idx % 4) + 1],
        amount = (idx % 100) + 1,
        message_id = nuidNext(),
        timestamp = dt.currentDateTimeUtc()
    }
end
```

**Also update**: `generateTestData()` to pass modify_id

```lua
-- Line 63-69
function natsTest.generateTestData(rowCount)
    local data = {}
    local modifyId = dt.currentDateTimeUtc()
    for i = 1, rowCount do
        data[#data + 1] = natsTest.generateTestRow(i, modifyId)
    end
    return data
end
```

#### Phase 2: Grid Column - Add modify_id Column

**File**: `/Volumes/nc/nc-backend/preference/form/nc/nc-admin/nc-nats/column/message.json`

**Change**: Add modify_id column after record_id

```json
{
    "name": "Record ID",
    "variable": "record_id",
    "width": 120
},
{
    "name": "Modified",
    "variable": "modify_id",
    "width": 130
},
```

#### Phase 3: Frontend - Use change-data.js Patterns

**File**: `/Volumes/nc/nc-frontend/src/core/nats-grid.js`

**Change**: `receiveGridMessage()` should use `change-data.js` patterns for consistency

**Note**: Current implementation works, just need to document the pattern.

#### Phase 4: Frontend - Persistent Highlighting

**File**: `/Volumes/nc/nc-backend/preference/form/nc/nc-admin/nc-nats/nc-nats.js`

**Changes**:

1. Replace `detectChangedRows()` with modify_id comparison
2. Replace `highlightChangedRows()` with persistent version
3. Implement proper row click handler

```javascript
// NEW: Compare modify_id instead of field values
function detectChangedRows(currentData, newData) {
    const changed = []
    const currentIdx = {}
    for (const row of currentData) {
        if (row.record_id) {
            currentIdx[row.record_id] = row
        }
    }
    for (const row of newData) {
        if (row.record_id && currentIdx[row.record_id]) {
            const existing = currentIdx[row.record_id]
            const viewed = existing._viewed_modify_id
            // Changed if modify_id differs from what user viewed
            if (row.modify_id && row.modify_id !== viewed) {
                row.__changed = true
                changed.push(row.idx)
            }
        }
    }
    return changed
}

// NEW: Persistent highlight with flash animation
function highlightChangedRows(rowIdxArray) {
    for (const idx of rowIdxArray) {
        const row = state.grid.test_message?.data?.[idx - 1]
        if (row) {
            row.__changed = true
            row.__changed_time = Date.now()  // For flash animation
        }
    }
    nc.grid.refreshCells(state, 'test_message')
    // Flash animation: add class briefly, then remove but keep __changed=true
    setTimeout(() => {
        for (const idx of rowIdxArray) {
            const row = state.grid.test_message?.data?.[idx - 1]
            if (row) {
                delete row.__changed_time  // Remove flash marker
            }
        }
        nc.grid.refreshCells(state, 'test_message')
    }, 1000)
}

// NEW: Row click clears highlight and stores viewed modify_id
function gridRowClicked(area, rowIndex, event, selected) {
    if (area !== 'test_message') {
        return
    }
    const row = state.grid.test_message?.data?.[rowIndex - 1]
    if (row && row.__changed) {
        row._viewed_modify_id = row.modify_id  // Store what user viewed
        row.__changed = false
        nc.grid.refreshCells(state, area)
    }
}
```

#### Phase 5: Update init.json

**File**: `/Volumes/nc/nc-backend/preference/form/nc/nc-admin/nc-nats/init.json`

**Change**: No changes needed - column file is already listed.

### Data Flow

```mermaid
sequenceDiagram
    participant Lua as Lua Backend
    participant NATS as NATS Server
    participant JS as JavaScript (nc-nats.js)
    participant Grid as SlickGrid
    participant User as User

    Note over Lua,User: Initial Load
    Lua->>Lua: generateTestRow() with modify_id="2026-02-07T10:00:00Z"
    Lua->>NATS: grid.sendGrid(bucket, gridData, option, grid_name)
    NATS->>JS: Chunk received with grid_name in message
    JS->>JS: Extract grid_name, set tableName = grid_name
    JS->>JS: Set _viewed_modify_id = modify_id
    JS->>Grid: Display rows (no highlight)

    Note over Lua,User: Update Arrives
    Lua->>Lua: Same record_id, new modify_id="2026-02-07T10:05:00Z"
    Lua->>NATS: Send updated record with grid_name
    NATS->>JS: Push received
    JS->>JS: modify_id (10:05) !== _viewed_modify_id (10:00)
    JS->>JS: Set calc.change = "modify", calc.change_time = now
    JS->>Grid: refreshCells(grid_name)
    Grid->>Grid: Apply nc-grid-row-changed + flash animation
    Grid->>User: Show highlighted row

    Note over JS,User: After 1 second
    JS->>JS: Flash ends (calc.change_time > 1s ago)
    JS->>Grid: refreshCells(grid_name)
    Grid->>User: Persistent highlight (no flash)

    Note over User,Grid: User Clicks Row
    User->>Grid: Click row
    Grid->>JS: gridRowClicked(area, rowIndex)
    JS->>JS: Set _viewed_modify_id = "2026-02-07T10:05:00Z"
    JS->>JS: Set __changed = false
    JS->>Grid: refreshCells(area)
    Grid->>User: Remove highlight
```

### Grid Name Parameter (2026-02-07)

**Problem**: Frontend had hardcoded grid names like `test_message`, making the system inflexible.

**Solution**: Add `grid_name` parameter to NATS grid functions.

#### Backend Changes (nats-grid.lua)

```lua
-- OLD: grid.sendGrid(bucket, gridData, option)
-- NEW: grid.sendGrid(bucket, gridData, option, grid_name)

function grid.sendGrid(bucket, gridData, option, grid_name)
    -- ... existing code ...
    local msg = {
        message_type = config.message_type_grid,
        bucket = bucket,
        message_count = totalMessageCount,
        grid_name = grid_name  -- NEW: Include grid name in message
    }
    -- ... rest of function ...
end
```

#### Frontend Changes (nc-nats.js)

```javascript
// Extract grid_name from message
function natsMessage(gridData) {
    const tableName = gridData.grid_name || gridData.data?.[0]?.from_db
    // Use tableName dynamically instead of hardcoded "test_message"
    state.data[tableName] = combinedData
    nc.grid.dataChanged(state.grid[tableName], combinedData)
}
```

#### Test File Updates

```lua
-- nats-grid-test.lua
gridTest.smallSent = grid.sendGrid(bucket, gridTest.smallGridData, gridTest.messageOption, "test_grid")

-- nats-test.lua
local gridData = {nats_name = "test-grid", row_array = {...}}
grid.sendGrid(bucket, gridData, gridTest.messageOption, "test_grid")
```

### CSS Classes (Row State Styling)

All row state styling is in `nc-main.css`:

```css
/* grid row state styling - faint backgrounds */
.nc-grid-row-new {
    background-color: rgba(34, 139, 34, 0.04) !important;
}

.nc-grid-row-changed {
    background-color: rgba(0, 0, 139, 0.008) !important;
}

.nc-grid-row-equal {
    background-color: transparent;
}

.nc-grid-row-deleted {
    background-color: rgba(220, 20, 60, 0.03) !important;
    color: dimgray !important;
}

/* changed cell with slow color animation */
.nc-grid-cell-changed {
    color: #cb0404 !important;
    font-weight: bold;
    transition:
        color 2s ease,
        background-color 2s ease,
        box-shadow 2s ease;
}

.nc-grid-cell-changed:hover {
    transition:
        color 0.3s ease,
        background-color 0.3s ease,
        box-shadow 0.3s ease;
}

/* NATS flash animation */
@keyframes nats-flash {
    0% { background-color: #ff6b6b; }
    100% { background-color: darkblue; }
}
```

**Row state colors:**

- New: very faint green (`rgba(34, 139, 34, 0.04)`)
- Changed: very faint blue (`rgba(0, 0, 139, 0.008)`)
- Deleted: very faint red (`rgba(220, 20, 60, 0.03)`) + dark gray text

**Changed cells:**

- Slow 2-second color transition (persistent visual feedback)
- Fast 0.3-second transition on hover (responsive interaction)

### Files to Modify

| File | Type | Change |
|------|------|--------|
| `nats-test.lua` | Backend | Add `modify_id` field to `generateTestRow()` |
| `message.json` | Config | Add `modify_id` column |
| `nc-nats.js` | Frontend | Replace highlight logic with persistent version |
| `nc-nats.css` | Style | Add flash animation (optional) |

### Testing Steps

1. **Start stress test** - Verify initial data loads with modify_id column
2. **Check column** - Verify ISO timestamps display in "Modified" column
3. **Trigger update** - Modify a row and send again via NATS
4. **Verify flash** - Row flashes red, then settles to dark blue
5. **Verify persistence** - Highlight stays until clicked
6. **Click row** - Click on highlighted row
7. **Verify clear** - Highlight disappears, _viewed_modify_id updated
8. **Send another update** - Same record with new modify_id
9. **Verify re-highlight** - Row highlights again
10. **Wait and test** - Verify works when user is away (changes accumulate)

### Integration with change-data.js

The NATS grid uses a simpler pattern than the full incremental update system:

| Feature | Full System (change-data.js) | NATS Grid |
|---------|------------------------------|-----------|
| Change marker | `calc.change = "modify"` | `__changed = true` |
| View tracking | `state.rec.last_modify_id` | `_viewed_modify_id` per row |
| Comparison | Server-side (Lua) | Client-side (JS) |
| Persistence | Session-based | Session-based |

This simpler approach works for NATS because:

- Data comes as complete rows (not incremental)
- No delete handling needed
- User interaction model is different (live updates vs sync)

---

## Database Sync Overview (4D → PostgreSQL → Browser)

### Overview

The incremental update system detects and propagates changes from external databases (4D) to the local PostgreSQL database, then to the browser UI. It supports three types of changes: **new**, **modify**, and **delete**.

### Complete Data Flow

```text
┌─────────────────────────────────────────────────────────────────────────────┐
│ STEP 1: Detect Changes (nc-calc-server-calc.lua:queryData)                │
├─────────────────────────────────────────────────────────────────────────────┤
│ Location: /Volumes/nc/nc-plugin/plugin/nc-calc-server/nc-calc-server-calc.lua│
│ Function: queryData() (lines 90-372)                                       │
│                                                                             │
│ Process:                                                                    │
│ 1a. Query external DB with last_modify_id filter → returns changed rows    │
│      - Uses queryParam.last_modify_id from param.rec.last_modify_id        │
│      - Only fetches records modified since last sync                       │
│                                                                             │
│ 1b. Build syncRec from gridInfo + param.schema (lines 200-217):            │
│      - tblNameRecType: tableName (e.g., "product_work")                     │
│      - recordType: from connection query                                   │
│      - schema: param.schema                                                │
│      - sourceId: param.schema (external DB)                                │
│      - targetId: "" (local DB - auto-detected)                             │
│      - queryName: query JSON file path                                     │
│      - recordIdFieldLocal: primary key field in local schema               │
│      - recordIdFieldLocalCompare: primary key field in external schema     │
│      - modifyIdFieldLocal: modify_time field in local schema               │
│      - modifyIdFieldLocalCompare: modify_time field in external schema    │
│      - primaryKeyFieldLocal: primary key field                             │
│                                                                             │
│ 1c. Call binarySearch.tableChange(syncRec, option) (line 219):            │
│      Location: /Volumes/nc/nc-backend/plugin/db-sync/db-sync-binary-search.lua│
│      Function: tableChange(syncRec, option) (lines 1065-1103)              │
│                                                                             │
│      Internal process:                                                       │
│      ├─ syncUtil.readPref() → syncPrf (loads db-sync.json preferences)     │
│      ├─ Set sourceCountAll=0, targetCountAll=0 (forces count queries)      │
│      ├─ Set prevSyncModifyId="" (skip modify_id filter for full compare)   │
│      └─ binarySearch.binarySearch(syncRec, syncPrf, binaryStat, bsOption) ││
│         └─ Compares source vs target using binary search                   │
│            ├─ Uses range-bounded queries for efficiency                    │
│            ├─ Processes batches of records                                 │
│            └─ Returns {add={id1,id2}, modify={id3}, delete={id4}}         │
│                                                                             │
│ 1d. Build gridIdx.deleted from changes.delete (lines 221-231):             │
│      - Format: [{table="product_work", record_id="xxx"}, ...]              │
│      - Only populated if #changes.delete > 0                              │
│                                                                             │
│ 1e. Return query results + gridIdx.deleted to caller                      │
└─────────────────────────────────────────────────────────────────────────────┘
                                    ↓
┌─────────────────────────────────────────────────────────────────────────────┐
│ STEP 2: Build Change Array (work-center.lua:workCenter)                   │
├─────────────────────────────────────────────────────────────────────────────┤
│ Location: /Volumes/nc/nc-backend/preference/form/nc/nc-work-center/        │
│          nc-calc-server-work-center.lua                                   │
│ Function: workCenter() (lines 700-850)                                     │
│                                                                             │
│ Process:                                                                    │
│ 2a. Receive data + gridIdx.deleted from queryData()                        │
│                                                                             │
│ 2b. Call changeData.setIncrementalChange() (lines 783-794):                │
│      Location: /Volumes/nc/nc-server/lib/db/change-data.lua                 │
│      Function: setIncrementalChange(data, lastModifyId, deletedRecordArr)  │
│      (lines 37-92)                                                          │
│                                                                             │
│      Returns changeArr with __change markers:                              │
│      ├─ For deleted records:                                               │
│      │   {record_id="deleted_id", __change="delete"}                       │
│      ├─ For new records:                                                   │
│      │   {full_record_data, __change="new"}                               │
│      └─ For modified records:                                             │
│          {full_record_data, __change="change"}                            │
│                                                                             │
│ 2c. Set param.change_arr = changeArr                                      │
│ 2d. Update rec.last_modify_id                                              │
└─────────────────────────────────────────────────────────────────────────────┘
                                    ↓
┌─────────────────────────────────────────────────────────────────────────────┐
│ STEP 3: Apply Changes (change-data.js:incrementalUpdate)                  │
├─────────────────────────────────────────────────────────────────────────────┤
│ Function: incrementalUpdate(state, response, option) (lines 116-164)      │
│                                                                             │
│ Process:                                                                    │
│ 3a. Receive response.change_arr from server                                │
│                                                                             │
│ 3b. Call applyChangeArr(state, changeArr) (lines 34-109):                  │
│                                                                             │
│      For each table in changeArr:                                          │
│      For each change record in table:                                      │
│                                                                             │
│      ├─ "new" case (lines 65-77):                                          │
│      │   - Delete __change field                                           │
│      │   - Add calc.change = "new"                                         │
│      │   - Add calc.change_time = ISO timestamp                           │
│      │   - Push to dataArray (or replace if exists)                       │
│      │                                                                     │
│      ├─ "change" case (lines 78-89):                                       │
│      │   - Delete __change field                                           │
│      │   - Add calc.change = "modify"                                     │
│      │   - Add calc.change_time = ISO timestamp                           │
│      │   - Replace existing record (full row replace)                      │
│      │                                                                     │
│      └─ "delete" case (lines 91-102):                                      │
│          - MARK with calc.change = "delete"                               │
│          - Add calc.change_time = ISO timestamp                           │
│          - DO NOT remove from array!                                       │
│                                                                             │
│ 3c. Update grid data to match state.data (updateGridData)                 │
│ 3d. Update state.rec.last_modify_id                                         │
│ 3e. UI filters/hides rows where calc.change === "delete"                   │
└─────────────────────────────────────────────────────────────────────────────┘
```

### Key Data Structures

#### syncRec (built in nc-calc-server-calc.lua:queryData)

```lua
local syncRec = {
    tblNameRecType = "product_work",           -- Table name
    recordType = "",                            -- From connection
    schema = "4d",                              -- External DB schema
    sourceId = "4d",                            -- External DB ID
    targetId = "",                              -- Local DB (auto-detected)
    queryName = "plugin/nc-calc-server/query/product_work-actual.json",
    recordIdFieldLocal = "pr.record_id",        -- Local field name
    recordIdFieldLocalCompare = "record_id",    -- External field name
    modifyIdFieldLocal = "pr.modify_time",      -- Local modify time
    modifyIdFieldLocalCompare = "modify_time",   -- External modify time
    primaryKeyFieldLocal = "pr.record_id"       -- Primary key
}
```

#### gridIdx.deleted (passed to changeData.setIncrementalChange)

```lua
-- Array of deleted record identifiers
gridIdx.deleted = {
    {table = "product_work", record_id = "xxx"},
    {table = "product_work", record_id = "yyy"},
    ...
}
```

#### changeArr (returned by changeData.setIncrementalChange)

```lua
-- Change array with __change markers
changeArr = {
    product_work = {
        {record_id = "deleted_id", __change = "delete"},
        {record_id = "new_id", field1="val1", ..., __change = "new"},
        {record_id = "mod_id", field1="newval", ..., __change = "change"}
    }
}
```

#### state.data (after applyChangeArr in change-data.js)

```javascript
// Actual data arrays in browser state with calc.change markers
state.data.product_work = [
    {
        record_id: "id1",
        field1: "val1",
        calc: {
            change: "modify",
            change_time: "2026-01-25T12:00:00Z"
        }
    },
    {
        record_id: "id2",
        field1: "val2",
        calc: {
            change: "delete",    // MARKED as deleted
            change_time: "2026-01-25T12:00:00Z"
        }
    },
    {
        record_id: "id3",
        field1: "val3",
        calc: {
            change: "new",
            change_time: "2026-01-25T12:00:00Z"
        }
    }
]
// UI can filter/hide rows where calc.change === "delete"
```

### Binary Search Options

The `binarySearch.binarySearch()` function accepts an `option` parameter to skip heavy checks:

```lua
option = {
    skip_reporting = true    -- Skip syncReport calls
}
```

#### When to use options

- **skip_reporting=true**: Used when statistics/debug output is not needed

### Reused Infrastructure

| Helper | Source | Purpose |
|--------|--------|---------|
| `syncUtil.readPref()` | db-sync-util.lua:130 | Load sync preferences |
| `binarySearch.binarySearch()` | db-sync-binary-search.lua:410 | Main binary search |
| `changeData.setIncrementalChange()` | change-data.lua:37 | Build changeArr |
| `incrementalUpdate()` | change-data.js:116 | Apply changes in browser |

### Frontend Tests

Location: `/Volumes/nc/nc-frontend/test/change-data.test.js`

Tests the JavaScript change handling:

- **applyChangeArr()**: Tests new/modify/delete marking
- **incrementalUpdate()**: Tests response processing

Run with:

```bash
cd /Volumes/nc/nc-frontend
bunx vitest run change-data
```

Key test cases (48 tests):

- New records get `calc.change = "new"` + timestamp
- Modified records get `calc.change = "modify"` + timestamp, **preserves existing calc object**
- Deleted records get `calc.change = "delete"` + timestamp, **NOT removed**
- Mixed changes in single call
- Multiple tables
- Grid mapping (table_name → grid_name)
- Return value: `true` for incremental, `undefined` for full updates

### Backend Tests

Location: `/Volumes/nc/nc-server/test/lib/change-data-test.lua`

Tests the Lua change handling:

- **lastModifyId()**: Tests highest modify_id calculation
- **setIncrementalChange()**: Tests change array creation with deleted records

Run with:

```bash
cd /Volumes/nc/nc-server
lj test/lib/change-data-test.lua
```

Key test cases (21 tests):

- Basic new/modify/delete detection
- Multiple data sources (PRODUCT, PERSON)
- Deleted records merged with new/modified records
- Empty data handling
- Return values: `changeArr`, `changeCount`, `newLastModifyId`

### File Paths

| Component | Full Path |
|-----------|-----------|
| Binary search | `/Volumes/nc/nc-backend/plugin/db-sync/db-sync-binary-search.lua` |
| Calc server | `/Volumes/nc/nc-plugin/plugin/nc-calc-server/nc-calc-server-calc.lua` |
| Work center | `/Volumes/nc/nc-backend/preference/form/nc/nc-work-center/nc-calc-server-work-center.lua` |
| Change data (Lua) | `/Volumes/nc/nc-server/lib/db/change-data.lua` |
| Change data (JS) | `/Volumes/nc/nc-backend/preference/form/nc/lib/change-data.js` |
| Frontend tests | `/Volumes/nc/nc-frontend/test/change-data.test.js` |
| Backend tests | `/Volumes/nc/nc-server/test/lib/change-data-test.lua` |

### Important Implementation Notes

#### Deleted Records Merge Fix (change-data.lua)

**Bug**: When `setIncrementalChange()` processed both deleted records AND new/modified records, the new/modified records would replace the deleted records in `changeArr[tableName]`.

**Fix** (lines 82-92): The function now checks if `changeArr[tableName]` already contains deleted records before assigning new/modified records. If both exist, they are merged by appending new/modified records after deleted records.

```lua
if #tableChangeArr > 0 then
    -- Merge with existing deleted records if any
    if changeArr[tableName] and #changeArr[tableName] > 0 then
        -- Append new/modified records after deleted records
        for _, record in ipairs(tableChangeArr) do
            changeArr[tableName][#changeArr[tableName] + 1] = record
        end
    else
        changeArr[tableName] = tableChangeArr
    end
end
```

#### Calc Object Preservation (change-data.js)

**Feature**: The "change" case preserves existing `calc` object data when marking modifications.

```javascript
// Merge calc objects to preserve existing calc data
const existingCalc = existingRecord.calc || {}
changeRecord.calc = changeRecord.calc || {}
Object.assign(changeRecord.calc, existingCalc)
changeRecord.calc.change = "modify"
```

This ensures that any existing calculated fields in `calc` are not lost when a record is modified.

#### Return Value Semantics (change-data.js)

**Feature**: `incrementalUpdate()` returns boolean to indicate update type, not the change_arr object.

```javascript
const changeArr = response.change_arr
if (!changeArr) {
    // Non-incremental response (full data or grid update)
    // ... process full update ...
    return undefined // Full update, not incremental
}
// ... process incremental ...
return true // Incremental update
```

This makes it easy to check if an update was incremental: `if (incrementalUpdate(state, response) === true)`

---

## Implementation Status (2026-02-07)

### NATS Incremental Update - COMPLETED ✅

All planned features have been implemented:

#### Backend Changes

- ✅ `nats-test.lua`: Added `modify_id` field to test data
- ✅ `nats-test.lua`: Added `from_db` field for change-data.lua integration
- ✅ `nats-test.lua`: Integrated `change-data.lua` for proper incremental updates
- ✅ `nats-test.lua`: Always purges stream before sending (clean state each test)
- ✅ `nats-test.lua`: Added `updateRandomFields()` function for simulating field changes
- ✅ `rec.json`: Added `use_change_data = false` option
- ✅ `hdr.json`: Added "Use Change Data" translation
- ✅ `nc-nats.json`: Added checkbox for `use_change_data` option

#### Frontend Changes

- ✅ `message.json`: Added `modify_id` column to grid
- ✅ `nc-nats.js`: Replaced `detectChangedRows()` with modify_id comparison
- ✅ `nc-nats.js`: Updated `highlightChangedRows()` with persistent highlighting
- ✅ `nc-nats.js`: Implemented `gridRowClicked()` to clear highlights on click
- ✅ `nc-nats.js`: Initialize `_viewed_modify_id` for new rows
- ✅ `nc-nats.json`: Added `v-on:grid-row-clicked` event handler
- ✅ `nc-main.css`: Added flash animation CSS

#### How change-data.lua Integration Works

When `use_change_data = true`:

1. **First test**: Sends initial data with `from_db = "test_message"` and `modify_id`
2. **Second test**:
   - Generates updated rows (same `record_id`, new `modify_id`, random field changes)
   - Generates new rows (new `record_id`, new `modify_id`)
   - Calls `changeData.setIncrementalChange()` to detect changes
   - Backend compares `modify_id` against `last_modify_id` per table
   - Returns `__change` markers: "new", "change", "delete"
3. **Frontend receives**: Full data with change markers, applies visual highlighting

#### Testing

- ✅ Run initial test with `use_change_data = false` - data loads with modify_id
- ✅ Run second test with `use_change_data = true` - uses change-data.lua for detection
- ✅ Run subsequent tests - incremental updates work correctly
- ✅ Click row - highlight clears, _viewed_modify_id updates

#### Files Modified

| File | Status |
|------|--------|
| `/Volumes/nc/nc-server/lib/message/nats/nats-grid-test.lua` | ✅ |
| `/Volumes/nc/nc-backend/preference/form/nc/nc-admin/nc-nats/column/message.json` | ✅ |
| `/Volumes/nc/nc-backend/preference/form/nc/nc-admin/nc-nats/rec.json` | ✅ |
| `/Volumes/nc/nc-backend/preference/form/nc/nc-admin/nc-nats/hdr.json` | ✅ |
| `/Volumes/nc/nc-backend/preference/form/nc/nc-admin/nc-nats/nc-nats.js` | ✅ |
| `/Volumes/nc/nc-backend/preference/form/nc/nc-admin/nc-nats/nc-nats.json` | ✅ |
| `/Volumes/nc/nc-frontend/src/style/nc-main.css` | ✅ |
| `/Volumes/nc/nc-frontend/src/components/nc-grid/nc-grid-style.scss` | ✅ |
| `/Volumes/nc/nc-plugin/plugin/nc-calc-server/documentation/incremental-update-data-flow.md` | ✅ |

### Row State Styling (2026-02-07)

Row state styling moved from `nc-grid-style.scss` to `nc-main.css`:

- New rows: `rgba(34, 139, 34, 0.04)` - very faint green background
- Changed rows: `rgba(0, 0, 139, 0.008)` - very faint blue background
- Deleted rows: `rgba(220, 20, 60, 0.03)` - very faint red background + dark gray text
- Changed cells: slow 2-second color transition for persistent visual feedback
- Changed cells hover: fast 0.3-second transition for responsive interaction

---
