# NATS Grid Messaging System

## Overview

The NATS Grid Messaging System provides a complete solution for transmitting large grid datasets, real-time change notifications, chat messaging, and presence tracking using NATS JetStream technology.

## Features

- **Chunked Grid Transmission**: Send large datasets in chunks (100 rows initial, 1000 rows subsequent)
- **Ordered Consumer**: Guaranteed message delivery order for reliable consumption
- **Change Messages**: Incremental updates using new/change/delete patterns
- **Client-to-Client Chat**: Direct messaging and broadcast support
- **Presence Tracking**: Online/offline/away status with automatic timeout cleanup

## Architecture

### Server-Side (Lua)

```text
lib/message/nats/
├── nats-grid-config.lua   -- Configuration constants
├── nats-grid.lua          -- Core grid messaging module
└── nats.lua               -- NATS connection API
```

### Client-Side (JavaScript)

```text
src/core/
└── nc-nats.js             -- Chunked message handling, chat, presence
```

## Configuration

### File: `nats-grid-config.lua`

```lua
local gridConfig = {
	-- Chunk sizes for grid transmission
	initial_chunk_size = 100,      -- First message: max 100 rows
	subsequent_chunk_size = 1000,  -- Subsequent messages: max 1000 rows

	-- Message types
	message_type_grid = "grid",
	message_type_chat = "chat",
	message_type_presence = "presence",
	message_type_change = "change",

	-- Subject prefixes
	subject_grid_prefix = "grid.",
	subject_chat_prefix = "chat.",
	subject_presence_prefix = "presence.",
	subject_change_prefix = "change.",

	-- Ordered consumer settings
	ordered_consumer_name_prefix = "ord",
	consumer_inactive_threshold = 5 * 60 * 1e9, -- 5 minutes

	-- Stream configuration
	stream_storage = "file",
	stream_num_replicas = 1,
	stream_max_age = 24 * 60 * 60 * 1e9,

	-- Consumer configuration
	consumer_ack_policy = "none",
	consumer_deliver_policy = "all",
	consumer_replay_policy = "instant",
	consumer_flow_control = true,
	consumer_idle_heartbeat = 5 * 1e9,
}
```

## Message Formats

### Grid Message (Chunked)

```json
{
	"message_type": "grid",
	"message_id": "unique-id",
	"bucket": "bucket-name",
	"chunk_index": 0,
	"total_chunks": 3,
	"row_count": 100,
	"total_rows": 250,
	"data": [
		{"record_id": "id1", "field1": "value1"},
		{"record_id": "id2", "field1": "value2"}
	],
	"sequence": 1,
	"timestamp": "2024-01-01T00:00:00Z"
}
```

### Change Message

```json
{
	"message_type": "change",
	"message_id": "unique-id",
	"bucket": "bucket-name",
	"change_arr": {
		"table_name": [
			{
				"record_id": "id1",
				__change": "new",
				"field1": "value1"
			},
			{
				"record_id": "id2",
				__change: "change",
				"field1": "updated"
			},
			{
				"record_id": "id3",
				__change: "delete"
			}
		]
	},
	"sequence": 123,
	"timestamp": "2024-01-01T00:00:00Z"
}
```

### Chat Message

```json
{
	"message_type": "chat",
	"message_id": "unique-id",
	"from_client": "client1",
	"to_client": "client2",
	"message": "Hello!",
	"timestamp": "2024-01-01T00:00:00Z"
}
```

Broadcast chat uses `"to_client": "*"`.

### Presence Message

```json
{
	"message_type": "presence",
	"message_id": "unique-id",
	"client_id": "client1",
	"status": "online",
	"timestamp": "2024-01-01T00:00:00Z"
}
```

Valid statuses: `"online"`, `"offline"`, `"away"`.

## Server-Side Usage (Lua)

### Sending Grid Data

```lua
local grid = require "message.nats.nats-grid"

-- Send grid with default chunking
local ret = grid.sendGridChunked("bucket-name", gridData, {
	initial_chunk_size = 100,
	subsequent_chunk_size = 1000
})

-- Returns:
-- {ok = true, total_rows = 250, total_chunks = 3, time = 0.123}
```

### Sending Change Messages

```lua
local changeArr = {
	table_name = {
		{record_id = "id1", field1 = "new", __change = "new"},
		{record_id = "id2", field1 = "mod", __change = "change"},
		{record_id = "id3", __change = "delete"}
	}
}

local ret = grid.sendChangeMessage("bucket-name", changeArr)
```

### Change Detection

```lua
-- Automatically detect changes between old and new data
local ret = grid.sendGridWithChanges("bucket-name", oldGridData, newGridData)
```

### Chat Messages

```lua
-- Direct message
local ret = grid.sendChatMessage("from-client", "to-client", "Hello!")

-- Broadcast to all
local ret = grid.broadcastChatMessage("from-client", "Broadcast message")
```

### Presence Updates

```lua
local ret = grid.sendPresence("client-id", "online")
local ret = grid.sendPresence("client-id", "away")
local ret = grid.sendPresence("client-id", "offline")
```

### Ordered Consumer

```lua
-- Create ordered consumer for reliable message consumption
local consumer = grid.createOrderedConsumer("bucket-name")

-- Fetch messages
local msgs = consumer:fetch(10)

-- Iterate through messages
local iter = consumer:messages({pull_max_message_count = 100})
local msg = iter:next()
```

## Client-Side Usage (JavaScript)

### Subscribing to Messages

```javascript
import { connectNats, subscribeToGrid } from '/src/core/nc-nats.js'

const nats = await connectNats()

// Subscribe to grid updates
const gridSub = subscribeToGrid(nats, 'bucket-name', (msg) => {
	console.log('Received grid:', msg.total_rows, 'rows')
	// msg.data contains the assembled grid data
})

// Subscribe to change messages
const changeSub = subscribeToChange(nats, 'bucket-name', (msg) => {
	console.log('Changes:', msg.change_arr)
	// Process changes using change-data.js patterns
})

// Subscribe to chat
const chatSub = subscribeToChat(nats, 'client-id', (msg) => {
	console.log('Chat from', msg.from_client, ':', msg.message)
})

// Subscribe to presence
const presenceSub = subscribeToPresence(nats, 'client-id', (msg) => {
	console.log('Client', msg.client_id, 'is', msg.status)
})
```

### Publishing Messages

```javascript
import { publishGridChunked, publishChangeMessage, publishChatMessage, publishPresence } from '/src/core/nc-nats.js'

// Publish grid data
const gridRet = await publishGridChunked(nats, 'bucket-name', gridData, {
	initial_chunk_size: 100,
	subsequent_chunk_size: 1000
})

// Publish changes
const changeRet = await publishChangeMessage(nats, 'bucket-name', changeArr)

// Send chat message
const chatRet = await publishChatMessage(nats, 'from-client', 'to-client', 'Hello!')

// Update presence
const presenceRet = await publishPresence(nats, 'client-id', 'online')
```

### Message Processing

The client automatically:

- **Buffers chunks** as they arrive
- **Assembles complete grids** when all chunks are received
- **Processes change messages** using `change-data.js` patterns
- **Tracks presence** with automatic cleanup of stale entries (10 minutes)
- **Maintains chat history** (limit to 100 messages)

### Reactive State

```javascript
import { messageBuffer, presenceList, chatMessages } from '/src/core/nc-nats.js'

// Watch for assembled grids
watch(messageBuffer, (buffer) => {
	for (const msgId in buffer) {
		const grid = assembleGrid(buffer[msgId])
		// Process complete grid
	}
})

// Watch presence updates
watch(presenceList, (list) => {
	for (const clientId in list) {
		console.log(clientId, list[clientId].status)
	}
})

// Watch chat messages
watch(chatMessages, (msgs) => {
	msgs.forEach(msg => console.log(msg.from_client, msg.message))
})
```

## Testing

### Lua Tests

Run comprehensive NATS grid tests:

```bash
# Run all NATS tests
lj /Volumes/nc/nc-server/test/lib/nats-test.lua

# Run specific grid tests with trace
lj /Volumes/nc/nc-server/test/lib/nats-test.lua -t

# Run dedicated grid tests
lj /Volumes/nc/nc-server/test/lib/nats-grid-test.lua

# Stress test with configurable row count
lj /Volumes/nc/nc-server/test/lib/nats-grid-test.lua stress 100000
```

### JavaScript Tests

Run Vitest tests for client-side grid handling:

```bash
# Run all NATS grid tests
vitest run /Volumes/nc/nc-frontend/test/nats-grid-test.js

# Run with watch mode
vitest watch /Volumes/nc/nc-frontend/test/nats-grid-test.js
```

## Performance

### Chunking Strategy

- **First chunk**: 100 rows (optimized for quick initial display)
- **Subsequent chunks**: 1000 rows (optimized for throughput)

### Examples

| Rows | Initial Chunks | Subsequent Chunks | Total Chunks |
 | 50   | 1 (50 rows)    | 0                 | 1            |
| 250  | 1 (100 rows)   | 2 (150 rows)      | 3            |
| 1500 | 1 (100 rows)   | 2 (1400 rows)     | 3            |
| 5250 | 1 (100 rows)   | 5 (5150 rows)     | 6            |

### Timing

Typical performance on localhost:

- **1000 rows**: ~0.1 seconds
- **10,000 rows**: ~0.5 seconds
- **100,000 rows**: ~5 seconds

## Stream Management

### Stream Naming

Grid streams use the pattern: `GRID_<bucket>`

Example: `GRID_work-center`

### Stream Lifecycle

- Streams are automatically created on first message
- Maximum age: 24 hours (configurable via `stream_max_age`)
- Automatic cleanup of old messages

## Subject Hierarchy

```text
grid.<bucket>         -- Grid data chunks
change.<bucket>       -- Change messages
chat.<client-id>      -- Client-to-client chat
chat.*                -- Broadcast chat
presence.<client-id>  -- Presence updates
```

## Error Handling

### Connection Errors

```lua
local ret = grid.sendGridChunked("bucket", gridData)
if ret.error then
	-- Handle error
	util.printError("Grid send failed: %s", ret.error)
end
```

### Consumer Errors

```lua
local consumer = grid.createOrderedConsumer("bucket")
if consumer.error then
	-- Consumer creation failed
	util.printError("Consumer failed: %s", consumer.error)
end
```

### Client-Side Errors

```javascript
try {
	const nats = await connectNats()
} catch (err) {
	console.error('NATS connection failed:', err)
}
```

## Best Practices

1. **Always use ordered consumers** for reliable message ordering
2. **Enable change detection** for incremental updates instead of full grid transmission
3. **Use presence tracking** to monitor client connectivity
4. **Implement cleanup** for stale message buffers (client-side)
5. **Configure chunk sizes** based on network conditions
6. **Monitor stream sizes** to prevent excessive storage

## Migration from Old API

### From `nats-old-tcp`

**Old Code:**

```lua
local natsTcp = require "message.nats-old-tcp"
```

**New Code:**

```lua
local natsTcp = require "message.nats.nats"
local grid = require "message.nats.nats-grid"
local gridConfig = require "message.nats.nats-grid-config"
```

## Troubleshooting

### Messages Not Arriving

1. Check stream exists: `js:streamInfo("GRID_<bucket>")`
2. Verify consumer is created: `js:consumerInfo("GRID_<bucket>", "consumer-name")`
3. Check network connectivity to NATS server

### Chunk Assembly Issues

1. Verify `message_id` is consistent across chunks
2. Check `chunk_index` and `total_chunks` match
3. Enable logging for chunk reception debugging

### Performance Issues

1. Reduce chunk sizes for slower networks
2. Use change messages instead of full grid transmission
3. Consider stream compression

## References

- [NATS JetStream Documentation](https://docs.nats.io/nats-concepts/jetstream)
- [Lua NATS Client](lib/message/nats/nats.lua)
- [JavaScript NATS Client](https://github.com/nats-io/nats.ws)

## Changelog

### 2024-01-25

- Initial implementation of NATS Grid Messaging System
- Chunked grid transmission (100/1000 row chunks)
- Change message support (new/change/delete)
- Client-to-client chat and presence tracking
- Ordered consumer support
- Comprehensive testing (Lua and Vitest)
