# parse-time-sheet.lua

Parses time sheet markdown files to structured JSON with validation and error reporting.

## Usage

```bash
cd /Volumes/nc/nc-server/tool/fix-code && lj parse-time-sheet.lua       # Parse and print only
cd /Volumes/nc/nc-server/tool/fix-code && lj parse-time-sheet.lua save  # Save json and recreated files to local folder, recreated for diff compare
cd /Volumes/nc/nc-server/tool/fix-code && lj parse-time-sheet.lua test  # Test patterns
```

## Input Format

```markdown
# Pasi hours 2025

- sat 2025-03-15

  - admin: 15:50-16:20
    - Card pay limit in OP
  - devops: 16:20-19:01
    - STUN server to Hetzner
      - <https://community.hetzner.com/tutorials/install-turn-stun-server-on-debian-ubuntu-with-Coturn>
      - needs dns fixes, does not work yet, needs dns setup
    - fix: nc-vscode-extension.js crash, add try-catch
    - fix: remove version number from .vsix
  - Tauri: 19:01-19:19
    - fix: add old MA app src to src-Tauri/src-electron

- sun 2025-03-16

  - research: 15:05-16:35
    - RAG re-rank models
  - Tauri: 16:35-21:42
    - AI: Create a Tauri version 2 app that connects to port 5959
  - nc-server: 21:42-22:45
    - fix: sock:receiveFrom() works, return also addrLen
```

## Statistics Output

After processing, main level statistics are printed:

```text
Events: 5847, Errors: 2, Warnings: 4, Days: 255, in 0.028 seconds

===          Statistics          ===
Person:             pasi
Start date:         2025-03-15
End date:           2025-12-23
Week count:         40.6 weeks
Day range:          284 days
Day count:          255 days
Non working days:   29 days
Total duration:     2050.8 hours
Work days per week: 6.30 days/week
Average per day:    8.04 hours/day
  5 days per week:  10.11 hours/day
Average per week:   50.5 hours/week
```

## Output JSON

The script generates JSON files (`.json`) and recreated markdown (`-recreated.md`).

### Top-Level Fields

| Field | Description |
|-------|-------------|
| `name` | Output JSON file path |
| `file_path` | Input file path |
| `person` | Username from filename |
| `start_date` | Earliest date |
| `end_date` | Latest date |
| `day_range` | Days from start to end (inclusive) |
| `day_count` | Work days |
| `total_duration` | Total hours worked |
| `average_day_duration` | Hours per work day |
| `work_days_per_week` | Work days/week (`day_count/day_range*7`) |
| `line_count` | Input file lines |
| `empty_line_count` | Empty lines in input |
| `event_count` | Total events |
| `error_count` | Error count |
| `warning_count` | Warning count |
| `parse_time` | Timestamp of processing |
| `parsing_time` | Seconds to parse |
| `event` | Array of parsed events |

### Event Fields

All events have these fields:

- `idx`: Event index
- `level`: Hierarchy level (1 = top, 2+ = indented)
- `line_number`: Line number in input
- `original_line`: Original text from input
- `project`: Project name ("day" for date entries, "task" for rows)
- `event_type`: "date", "time", "row", or "other"
- `description`: Description text
- `empty_lines_after`: Empty lines after this event
- `error`: Array of error codes
- `warning`: Array of warning codes

#### Date Event (event_type: "date")

```json
{
  "idx": 2,
  "level": 1,
  "line_number": 3,
  "original_line": "- sat 2025-03-15",
  "project": "day",
  "event_type": "date",
  "date": "2025-03-15",
  "day_name": "sat",
  "description": "- sat 2025-03-15",
  "day_duration": 3.48,
  "duration_array": [0.5, 2.68, 0.3],
  "time_row_count": 3,
  "total_row_count": 11,
  "min_start_time": "2025-03-15 15:50",
  "min_start_time_number": 1742053800,
  "max_end_time": "2025-03-15 19:19",
  "max_end_time_number": 1742066340,
  "start_time": "2025-03-15 00:00:00",
  "start_time_number": 1741996800,
  "end_time": "2025-03-15 23:59:59",
  "end_time_number": 1742083199,
  "empty_lines_after": 1
}
```

#### Time Event (event_type: "time")

```json
{
  "idx": 3,
  "level": 2,
  "line_number": 5,
  "original_line": "  - admin: 15:50-16:20",
  "project": "admin",
  "event_type": "time",
  "date": "2025-03-15",
  "description": "",
  "start_time": "2025-03-15 15:50",
  "start_time_number": 1742053800,
  "end_time": "2025-03-15 16:20",
  "end_time_number": 1742055600,
  "duration": 0.5
}
```

#### Row Event (event_type: "row")

```json
{
  "idx": 4,
  "level": 3,
  "line_number": 6,
  "original_line": "    - Card pay limit in OP",
  "project": "task",
  "event_type": "row",
  "date": "2025-03-15",
  "description": "- Card pay limit in OP"
}
```

## Error Types

### Errors

| Code | Description |
|------|-------------|
| `end_time_before_start_time` | End time is before start time |
| `datetime_out_of_sequence` | Events not in chronological order |
| `event_before_date_start_time` | Event starts before date constraint |
| `event_after_date_end_time` | Event ends after date constraint |
| `dates_smaller_than_previous` | Date is before previous date |
| `invalid_level_1_event_type` | Level 1 must be date event |
| `invalid_date_event_at_sub_level` | Date event at level > 1 |
| `date_before_first_event` | Date before first date event |
| `date_after_one_year` | Date more than 1 year from first |
| `date_after_last_event` | Date after last date event |

### Warnings

| Code | Description |
|------|-------------|
| `time_event_at_sub_level` | Time event at level >= 3 |

## Configuration

| Name | Type | Default | Description |
|------|------|---------|-------------|
| `nextDayUserIdx` | table | `{pasi=true}` | Users allowed next-day morning hours |
| `morningHourCutOff` | number | 8 | Morning hour cutoff (hours before = next day) |
| `maxErrorPrintCount` | number | 120 | Max errors/warnings displayed |
| `saveToFile` | boolean | false | Save JSON and recreated markdown |
| `useLint` | boolean | false | Run linter before parsing |

## Error Display Format

```text
Events: 5847, Errors: 2, Warnings: 4, Days: 255, in 0.028 seconds

Errors (2 total):
1.
# Pasi hours 2025 <-- ! invalid_level_1_event_type
- sat 2025-03-15
2.
  - nc-server: 16:00-20:45 <--
    - fix: delay load calc
    - fix: xmlua, xml2 = ffi.loadDll("xml/xml2")
...
      - fix: xml2 = ffi.loadDll("xml/xml2-16.dll")
      - fix: xml win libs x64
      - fix: xml tests in win
  - devops: 20:25-21:25 <-- ! datetime_out_of_sequence, previous ended: 2025-12-10 20:45, current starts: 2025-12-10 20:25
    - fix: add extension runonsave

Warnings (4 total):
1.
  - Ateno: 17:58-18:46 <--
    - Data fixing, find mail Re: Ateno data
      - Työmääräimet/Toiminnot/Hae avoimet toimitetut
        - rivejä:
        - kesto 17:58-18:07 = 7 min <- datetime_out_of_sequence, previous ended: 2025-05-22 18:46, current starts: 2025-05-22 17:58
```

### Error Markers

- `<-- !` - Error (red)
- `<-` - Warning (yellow)
- `<--` - Previous event reference (green)

### Display Rules

1. Find MOST RECENT DAY entry before error
2. Show that day entry first
3. Show events between day and error
4. If >5 events between: first 3 + "..." + last 3 + error
5. Show next line if same level or child

## Features

### Output Files

- **`.json`**: Structured event data with all fields
- **`-recreated.md`**: Markdown reconstructed from events (for validation)

### Time Validation

- Validates HH:MM, H:MM, HH:MM:SS formats
- Detects time overlaps and sequence errors
- Handles next-day morning hours (configured users)
- Validates against date time constraints

### NO-PARSING Support

Content after `NO-PARSING` marker is ignored (line 700).

## PEG Patterns & Parsing

### WHY patternNotPattern is REQUIRED

**THE FUNDAMENTAL PROBLEM:**

Time sheet files are human-written. Humans write inconsistently. Day entries appear as "- sat 2025-03-15" (space) OR "- sat: 2025-03-15" (colon) OR "- sat:2025-03-15" (no space). Time entries appear as "9:05-16:20", "9:05 - 16:20", or "9:05:30-16:20:45" with seconds.

**WHY SIMPLE PATTERNS FAIL:**

A pattern like `dash * dayNameCapture * dateCapture` expects day name to be immediately followed by valid date. When input is "- sat: 2025-03-15", the parser sees "sat:" - the colon is NOT a valid date character, so dateCapture fails. The parse fails completely.

You would need separate patterns for every delimiter variation: space, colon, multiple spaces, tab, no space. This is impossible - infinite combinations.

**THE NOTPATTERN SOLUTION:**

notPattern means "match anything EXCEPT this thing, keep going until you find it." The pattern `notDatePattern * dateCapture` tells parser: skip over characters that are NOT valid date, then capture date when found.

This single pattern handles ALL delimiter variations. For "- sat:2025-03-15", notDatePattern skips colon and spaces, then dateCapture captures "2025-03-15". Same pattern works for "- sat 2025-03-15". One pattern, all variations handled.

**FOR TIME EXTRACTION:**

To extract start and end times from "15:50-19:01", you need notTimePattern to skip between the two times. The pattern `notTimePattern * timeCapture * notTimePattern * timeCapture` means: capture first time, skip everything NOT time (the dash), capture second time. Works for "15:50-19:01", "15:50 - 19:01", or "15:50 -19:01" because notTimePattern skips all non-time characters between.

**THIS IS WHY IT IS CRITICAL:**

Without notPattern, pattern parsing FAILS on all but perfectly formatted input. With notPattern, one pattern handles all format variations. notPattern is ESSENTIAL for ANY pattern that needs to skip delimiters to find the target pattern to capture.

### What patternNotPattern returns

`peg.patternNotPattern` returns TWO patterns:

1. **pattern** - matches the specific text exactly
2. **notPattern** - matches anything EXCEPT the specific text, stopping BEFORE it

#### Function Signature

```lua
peg.patternNotPattern(pattern, countLeast, countMost, countAntiLeast)
```

**Parameters:**

- `pattern`: The pattern to match (string or pattern)
- `countLeast`: Minimum repetitions of pattern (default 1)
- `countMost`: Maximum repetitions of pattern (default 1)
- `countAntiLeast`: Minimum repetitions of anti-pattern (default 1)

**Returns:**

- pattern: matches 'pattern' with optional repetition counts
- notPattern: matches anything EXCEPT 'pattern'

#### Parameter Examples

1. **Basic usage (default)**: `peg.patternNotPattern(":")`
   - pattern: matches ":" exactly once
   - notPattern: matches anything except ":"

2. **Optional patterns (countLeast = 0)**: `peg.patternNotPattern(" ", 0)`
   - pattern: matches " " zero or more times
   - notPattern: matches anything except " "

3. **Range of repetitions (countLeast, countMost)**: `peg.patternNotPattern(peg.range("09"), 2, 4)`
   - pattern: matches 2-4 digits
   - notPattern: matches anything except digits

4. **Minimum anti-pattern length (countAntiLeast)**: `peg.patternNotPattern(peg.define.separator, 0, nil, 2)`
   - pattern: matches zero or more separators
   - notPattern: matches anything except separators (needs at least 1 char)

### CRITICAL RULES

ONLY peg patterns at TOP level of file allowed!
NO OTHER patterns OR parsing in code allowed!
ONLY dayEntryPattern and taskEntryPattern ARE ALLOWED
dayEntryPattern and taskEntryPattern MUST return ALL needed information with captures

### Pattern Building Rules

1. **ONLY peg.patternNotPattern for character patterns**
2. **Use notPattern to match up to but NOT including text**
3. **Use peg.least() only for repetitions of defined patterns**
4. **ALL patterns at file top** - no inline pattern building
5. **NO string concatenation** - use PEG operators (*, +) only

### PEG Pattern Requirements

- ONLY `peg.patternNotPattern()` for character patterns
- NO inline pattern definitions (no `peg.capture()`, `peg.least()`, etc. in patterns)
- MUST use `dt.timeCapturePattern` and `dt.dateCapturePattern` for time/date
- ALL patterns defined at top, used only in main patterns

### Test-Driven Development Approach

**IMPORTANT**: When adding new patterns or fixing bugs:

1. **First**: Add ALL edge cases and broken examples to the test suite
2. **Second**: Run tests to see them fail
3. **Third**: Fix the code to make all tests pass
4. **Never**: Fix code without corresponding tests

This ensures all edge cases are covered and prevents regressions.

#### Test Categories

The test suite covers:

1. **Non-Day Entries** (project: time-time description):
   - Indented and non-indented entries
   - With and without dashes
   - Time ranges with seconds
   - Description-only entries

2. **Day Entries** (day[:]? date [time-range] [duration]):
   - All day name variations (mon, tue, wed, thu, fri, sat, sun)
   - With and without colons
   - With time constraints (start only, end only, both)
   - With durations (7h20min)

3. **Time Patterns** (standalone time validation):
   - Single digit hours (9, 16, 23)
   - Hour:minute (9:5, 16:20)
   - Hour:minute:second (9:5:30, 16:20:45)
   - Dot separators (9.5, 16.20, 9.15.30)

4. **Invalid Entries**:
   - Invalid time formats
   - Level 1 entries without dates (e.g., "# Pasi hours 2025")

### Day Entry Patterns

Actual format from pasi-hours-2025-03-15.md:

```text
- sat 2025-03-15
- sun 2025-03-16
- mon 2025-03-17
- tue 2025-03-18
```

Parser supports these patterns (not currently used):

```text
- sat: 2025-03-15
- sat:2025-03-15
- pe 2024-08-09 9:5 - 16:2 7h20min
- Mon 2025-01-01 9:00-17:0
- ma 2024-08-12 9:1-17:1 7h5min
-ti 2024-08-13
- to: 2024-08-15 8:0-16:3
- fr:2024-08-16
- la 2024-08-17 -  (start time only)
- su 2024-08-18 -20:0  (end time only)
- pe 2024-08-09 9:15:30 - 16:20:45 7h5min
```

### Non-Day Entry Patterns

Actual format from pasi-hours-2025-03-15.md:

```text
  - admin: 15:50-16:20
    - Card pay limit in OP
  - devops: 16:20-19:01
    - STUN server to Hetzner
    - fix: nc-vscode-extension.js crash, add try-catch
  - Tauri: 19:01-19:19
    - fix: add old MA app src to src-Tauri/src-electron
  - research: 15:05-16:35
    - RAG re-rank models
  - nc-server: 21:42-22:45
    - fix: sock:receiveFrom() works, return also addrLen
  - mail: 11:45-12:12:42
    - (time with seconds)
```

All PEG patterns defined at file top level, using `peg.patternNotPattern()` for character patterns.

```lua
-- Character patterns
local dash = peg.least(peg.set("-* "), 0)
local alpha = peg.range("az") + peg.range("AZ")
local datePattern, notDatePattern = peg.patternNotPattern(dt.datePattern, 1, 1)
local timePattern, notTimePattern = peg.patternNotPattern(dt.timePattern, 1, 1)
local notProjectPattern, projectPattern = peg.patternNotPattern(peg.set(": ,"), 1, 1)
-- Capture patterns
local afterTextCapture = peg.capture(peg.define.anyText)
local dateCapture = peg.capture(datePattern)
local timeCapture = peg.capture(timePattern)
local dayNameCapture = peg.capture(peg.leastMost(alpha, 2, 10))
local projectCapture = peg.capture(projectPattern)
-- Main patterns
local dayEntryPattern = dash * dayNameCapture * notDatePattern * dateCapture * afterTextCapture
local dayTimePattern = notTimePattern * timeCapture * notTimePattern * timeCapture * afterTextCapture
local taskEntryPattern = dash * projectCapture * notProjectCapture * notTimeCapture * timeCapture * notTimeCapture * timeCapture * afterTextCapture
local taskEntryPatternNoInfo = dash * projectCapture * notProjectCapture * timeCapture * notTimeCapture * timeCapture * afterTextCapture
local projectEntryPattern = dash * projectCapture * colon * afterTextCapture
```

### patternNotPattern

Returns TWO patterns: one matching text, one matching anything EXCEPT text. Critical for skipping delimiters.

```lua
-- Example: "sat:2025-03-15" - notDatePattern skips ":" to find date
dash * dayNameCapture * notDatePattern * dateCapture * afterTextCapture
```

### Testing

Run `lj parse-time-sheet.lua test` to validate all pattern matching.

## Notes

- No fallback values - errors reported immediately
- All paths must be absolute
- Short project names (ai, nc, ui) correctly identified
- Multiple spaces between date/time handled correctly
- Date time constraints validated in context
