# XML Processing Technical Manual

## Overview

This manual covers the comprehensive XML processing capabilities including linting, validation against schemas (XSD, DTD, RelaxNG, Schematron), XSLT transformations, parsing, security features, and more.

## Core Libraries

### 1. xml-lint.lua - XML Linting and Basic Processing

Location: `/Volumes/nc/nc-server/lib/xml/xml-lint.lua`

**Primary Functions:**

```lua
-- Main validation and linting
xmlLint.lint(xmlFile, options)                    -- Comprehensive linting with options
xmlLint.checkWellFormed(xmlFile)                  -- Checks if XML is well-formed
xmlLint.validateXSD(xmlFile, xsdFile)             -- Validates XML against XSD schema
xmlLint.validateDTD(xmlFile, dtdFile)             -- Validates XML against DTD
xmlLint.validateRelaxNG(xmlFile, rngFile)         -- Validates against RelaxNG schema
xmlLint.validateSchematron(xmlFile, schFile)      -- Validates against Schematron rules

-- XML processing utilities
xmlLint.parseXML(xmlFile, options)                -- Parse XML document
xmlLint.formatXML(xmlFile, outputFile)            -- Format XML with proper indentation
xmlLint.cleanXML(xmlFile, options)                -- Clean XML (remove blanks, etc.)
xmlLint.processXInclude(xmlFile, outputFile)       -- Process XInclude directives

-- XPath and querying
xmlLint.xpath(xmlFile, query)                     -- Execute XPath query
xmlLint.lintXpath(xmlFile, query)                 -- Lint with XPath
xmlLint.listNode(doc)                             -- List document nodes

-- Information and detection
xmlLint.getXMLEncoding(xmlFile)                   -- Get XML encoding
xmlLint.getXMLVersion(xmlFile)                    -- Get XML version
xmlLint.isHTML(xmlFile)                           -- Detect if file is HTML
xmlLint.autoLint(xmlFile)                         -- Automatic linting
```

**Common Options:**

```lua
options = {
    -- Output control
    no_out = true,              -- Suppress XML output
    quiet = true,               -- Suppress non-error output
    output_file = "out.xml",    -- Output file path

    -- Validation options
    schema = "path/to/xsd",     -- XSD schema for validation
    dtd = "path/to/dtd",       -- DTD for validation
    valid = true,              -- Validate with internal DTD
    relaxng = "path/to/rng",   -- RelaxNG schema
    schematron = "path/to/sch", -- Schematron schema

    -- Processing options
    format = true,             -- Format output
    xpath = "//expression",    -- XPath query to execute
    xinclude = true,           -- Process XInclude
    clean = true,              -- Clean XML

    -- Security options
    no_net = true,             -- Disable network access
    no_xinclude = true,        -- Disable XInclude processing

    -- Advanced options
    huge = true,               -- Allow huge files
    stream = true,             -- Streaming mode
    recover = true,            -- Recover from errors
}
```

**Advanced Options and Features:**

```lua
-- Parsing and loading options
options = {
    -- DTD handling
    load_dtd = true,           -- Load external DTD
    dtd_attribute = true,      -- Add default DTD attributes
    drop_dtd = true,           -- Remove DTD from output
    no_default_dtd = true,     -- Don't use default DTD

    -- Entity processing
    substitute_entity = true,  -- Substitute entities
    convert_cdata = true,      -- Convert CDATA to text

    -- Namespace handling
    clean_namespace = true,    -- Clean up redundant namespaces

    -- Content handling
    remove_blank_node = true,  -- Remove blank text nodes
    disable_dict = true,       -- Disable XML dictionary

    -- Parsing behavior
    recover = true,            -- Recover from errors
    no_error = true,           -- Suppress error messages
    no_warning = true,         -- Suppress warnings
    sax = true,               -- Use SAX parser (streaming)
    xml_1_0_rule = true,       -- Use XML 1.0 rules (pre-5th edition)

    -- Memory and performance
    huge = true,               -- Allow huge files (no limits)
    no_compact_mode = true,    -- Disable compact mode

    -- Security
    max_amplification = 1000,  -- Set amplification limit

    -- HTML support (partial)
    html = true,               -- Parse as HTML (experimental)
}
```

**When to Use Each Feature:**

1. **Validation - Use Before Processing**:

   ```lua
   -- Always validate XML before processing
   local isValid, errors = xmlLint.checkWellFormed("data.xml")
   if not isValid then
       -- Handle errors
   end

   -- Validate against schema if available
   isValid, errors = xmlLint.validateXSD("data.xml", "schema.xsd")
   ```

2. **Error Recovery - For Unreliable Sources**:

   ```lua
   -- Process possibly malformed XML
   local success, errors = xmlLint.lint("data.xml", {
       recover = true,         -- Try to recover from errors
       no_error = false,       -- Still report errors
       format = true          -- Format output for inspection
   })
   ```

3. **Large File Processing - For Memory Efficiency**:

   ```lua
   -- Process huge XML files
   xmlLint.lint("huge.xml", {
       stream = true,          -- Use streaming mode
       huge = true,            -- Remove size limits
       no_out = true           -- Don't load entire file
   })
   ```

4. **Security - For Untrusted XML**:

   ```lua
   -- Secure processing of external XML
   xmlLint.lint("untrusted.xml", {
       no_net = true,          -- Disable network access
       no_xinclude = true,     -- Disable XInclude
       substitute_entity = false, -- Don't substitute external entities
       max_amplification = 10  -- Limit expansion
   })
   ```

5. **Content Cleaning - For Normalization**:

   ```lua
   -- Clean up messy XML
   xmlLint.cleanXML("messy.xml", {
       remove_blanks = true,     -- Remove blank text nodes
       drop_dtd = true,           -- Remove DTD declarations
       clean_namespaces = true,   -- Clean redundant namespaces
       remove_cdata = true       -- Convert CDATA to text
   })
   ```

6. **XPath Queries - For Data Extraction**:

   ```lua
   -- Extract specific data
   local success, errors = xmlLint.lint("data.xml", {
       xpath = "//book[price>50]/title",
       no_out = true
   })
   -- Note: XPath result capture needs implementation
   ```

7. **Debugging - For Development**:

   ```lua
   -- Get detailed document structure
   local doc = xmlLint.parseXML("test.xml")
   xmlLint.listNode(doc)

   -- Check encoding and version
   local encoding = xmlLint.getXMLEncoding("file.xml")
   local version = xmlLint.getXMLVersion("file.xml")
   ```

8. **Auto-detection - For Flexible Input**:

   ```lua
   -- Automatically detect and handle HTML/XML
   local success, errors = xmlLint.autoLint("unknown.html", {
       format = true,
       output_file = "output.xml"
   })
   ```

**Output Handling Examples:**

```lua
-- Format and save XML
xmlLint.formatXML("input.xml", "formatted.xml")

-- Process and output to different format
xmlLint.lint("data.xml", {
    format = true,
    output_file = "output.xml",
    no_out = false
})

-- Get raw XML string (for further processing)
local result, errors = xmlLint.lint("data.xml", {
    no_out = false,
    format = true
})
-- result contains the formatted XML string
```

### 2. xml-schema.lua - Schema Validation

Location: `/Volumes/nc/nc-server/lib/xml/xml-schema.lua`

**Primary Functions:**

```lua
-- Schema validation
xmlSchema.validateXSD(XMLFileName, XSDFileName)  -- XSD validation
xmlSchema.validateDTD(xmlOrFile, dtdFileName, option) -- DTD validation
```

**DTD Options (supports cXML format):**

```lua
option = {
    replace_doctype = true,     -- Replace existing DOCTYPE
    replace_version = true      -- Update XML version to match DTD
}
```

**Features:**

- Automatic DOCTYPE replacement for DTD validation
- Version synchronization between XML and DTD
- cXML document support
- Comprehensive error reporting

**Advanced Features and Use Cases:**

1. **XSD Validation with Full Error Reporting**:

   ```lua
   -- Validate XML against XSD schema
   local isValid, errors = xmlSchema.validateXSD("document.xml", "schema.xsd")
   if not isValid then
       for _, error in ipairs(errors) do
           print(string.format("Error [%d]: %s", error.type, error.message))
       end
   end
   ```

2. **DTD Validation with cXML Support**:

   ```lua
   -- Standard DTD validation
   local isValid, errors = xmlSchema.validateDTD("invoice.xml", "invoice.dtd")

   -- cXML DTD validation with version management
   local isValid, errors = xmlSchema.validateDTD("cxml.xml", "cxml.dtd", {
       replace_doctype = true,    -- Force use of specified DTD
       replace_version = true     -- Update cXML version to match DTD
   })
   ```

3. **Input Flexibility**:

   ```lua
   -- Validate from file path
   xmlSchema.validateXSD("/path/to/file.xml", "/path/to/schema.xsd")

   -- Validate XML content string directly
   local xmlContent = [[<root><item>Test</item></root>]]
   xmlSchema.validateDTD(xmlContent, "schema.dtd")
   ```

4. **Error Handling Patterns**:

   ```lua
   -- Comprehensive error handling
   local function validateWithDetails(xmlFile, schemaFile)
       local isValid, errors = xmlSchema.validateXSD(xmlFile, schemaFile)

       if not isValid then
           print("Validation failed for:", xmlFile)
           print("Schema:", schemaFile)
           print("Errors found:", #errors)

           for i, err in ipairs(errors) do
               print(string.format("%d. [%d] %s", i, err.type, err.message))
           end

           return false
       end

       print("Validation successful")
       return true
   end
   ```

**DTD Validation Details for cXML:**

The `validateDTD` function provides special handling for cXML documents:

- **DOCTYPE Replacement**: Automatically inserts or replaces DOCTYPE declarations
- **Version Synchronization**: Updates cXML version attribute to match DTD requirements
- **Entity Extraction**: Automatically extracts version information from DTD entities
- **Error Context**: Provides clear error messages with source identification

**Implementation Notes:**

- XSD validation uses libxml2's schema validation engine
- DTD validation parses DTD content for version matching
- Both validation methods provide structured error arrays
- Automatic cleanup of XML parser resources after validation

### 3. xml-xslt.lua - XSLT Transformations

Location: `/Volumes/nc/nc-server/lib/xml/xml-xslt.lua`

**Primary Functions:**

```lua
-- Core transformation
xmlXslt.convert(xmlOrFilename, xsltOrFilename, option, outputFile)

-- Helper functions
xmlXslt.getInfo()                           -- Get processor info
xmlXslt.cleanup()                           -- Cleanup resources
xmlXslt.shutdown()                          -- Full shutdown
```

**XSLT Options:**

```lua
options = {
    -- Output control
    output_file = "output.xml",   -- Output file path
    format = "xml",               -- Output format: xml, html, text

    -- Processing options
    cleanup_after_convert = true, -- Auto cleanup after transformation
    parse_after = "declaration", -- Parse after specific marker

    -- Encoding support
    -- Automatically handles ISO-8859-15 to UTF-8 conversion
}
```

**Advanced Features and Use Cases:**

1. **File-based Transformation**:

   ```lua
   -- Transform XML file using XSLT file
   local result, errors = xmlXslt.convert(
       "data.xml",              -- Input XML file
       "transform.xsl",         -- XSLT stylesheet
       {                       -- Options
           output_file = "output.html",
           format = "html"
       }
   )

   if not result then
       print("Transformation failed:", errors)
   end
   ```

2. **String-based Transformation**:

   ```lua
   -- Transform XML content from string
   local xmlString = [[
   <catalog>
       <book id="1">
           <title>XML Guide</title>
           <author>John Doe</author>
           <price>29.99</price>
       </book>
   </catalog>
   ]]

   local xsltString = [[
   <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
       <xsl:output method="text"/>
       <xsl:template match="/">
           <xsl:for-each select="catalog/book">
               Book: <xsl:value-of select="title"/> - $<xsl:value-of select="price"/>
           </xsl:for-each>
       </xsl:template>
   </xsl:stylesheet>
   ]]

   local result, errors = xmlXslt.convert(xmlString, xsltString, {
       format = "text"
   })

   if result then
       print(result)  -- Output: Book: XML Guide - $29.99
   end
   ```

3. **Format Conversion**:

   ```lua
   -- Convert XML to HTML
   xmlXslt.convert("data.xml", "to-html.xsl", {
       format = "html",
       output_file = "report.html"
   })

   -- Convert to plain text
   xmlXslt.convert("data.xml", "to-text.xsl", {
       format = "text",
       output_file = "report.txt"
   })

   -- Convert to XML (with transformation)
   xmlXslt.convert("data.xml", "restructure.xsl", {
       format = "xml",
       output_file = "restructured.xml"
   })
   ```

4. **XSLT Parameters**:

   ```lua
   -- Pass parameters to XSLT (Note: This needs to be implemented in XSLT)
   local result = xmlXslt.convert("data.xml", "report.xsl", {
       parameters = {
           ["report-date"] = os.date("%Y-%m-%d"),
           ["user-id"] = "12345",
           ["show-details"] = "true"
       }
   })
   ```

5. **Encoding Handling**:

   ```lua
   -- Automatic encoding conversion
   -- If XML contains encoding="ISO-8859-15", it will be converted to UTF-8

   local result = xmlXslt.convert("latin9.xml", "transform.xsl", {
       output_file = "utf8-output.html"
   })
   ```

6. **Memory Management**:

   ```lua
   -- Manual cleanup for batch processing
   local files = {"file1.xml", "file2.xml", "file3.xml"}
   local stylesheet = "transform.xsl"

   for _, file in ipairs(files) do
       xmlXslt.convert(file, stylesheet, {
           output_file = file:gsub("%.xml$", ".html"),
           cleanup_after_convert = false  -- Don't cleanup each time
       })
   end

   -- Cleanup once after all transformations
   xmlXslt.cleanup()
   ```

7. **Error Handling**:

   ```lua
   -- Robust error handling
   local function transformWithErrorHandling(xmlFile, xsltFile, outputFile)
       local result, errors = xmlXslt.convert(xmlFile, xsltFile, {
           output_file = outputFile
       })

       if not result then
           print("Transformation failed:")
           if type(errors) == "table" then
               for _, err in ipairs(errors) do
                   print("  " .. err.message)
               end
           else
               print("  " .. tostring(errors))
           end
           return false
       end

       print("Transformation successful: " .. outputFile)
       return true
   end
   ```

8. **Parser Options**:

   ```lua
   -- Custom parsing behavior
   xmlXslt.convert("data.xml", "transform.xsl", {
       parse_after = "<?xml-stylesheet",  -- Parse after specific instruction
       format = "html"
   })
   ```

**XSLT Processor Information:**

```lua
-- Get processor status and capabilities
local info = xmlXslt.getInfo()
print("Initialized:", info.initialized)
print("Library:", info.libxslt_version)
print("Supported formats:", table.concat(info.supported_formats, ", "))
```

**Automatic Output Method Detection**:

The convert function automatically modifies XSLT output method based on the format option:

```lua
-- If XSLT doesn't specify output method, it will be added
xmlXslt.convert(xmlContent, [[
   <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
       <xsl:template match="/">
           <html><body><xsl:value-of select="."/></body></html>
       </xsl:template>
   </xsl:stylesheet>
]], {format = "html"})

-- The above will automatically add: <xsl:output method="html" encoding="UTF-8" indent="yes"/>
```

**Performance Considerations:**

1. **Initialize Once**: The XSLT processor is initialized on first use
2. **Batch Processing**: Reuse the same stylesheet for multiple transformations
3. **Memory Cleanup**: Use `cleanup()` after batch processing, `shutdown()` when done
4. **File vs String**: File-based input is more memory efficient for large documents

**Output Directory Creation**:

```lua
-- Output directories are created automatically if they don't exist
xmlXslt.convert("data.xml", "transform.xsl", {
   output_file = "reports/2024/output.html"  -- Creates reports/2024/ if needed
})
```

**Implementation Notes:**

- Uses libxslt via xmlsec library
- Supports libxml2's recovery mode for parsing errors
- Automatic memory management for transformed documents
- Supports both file paths and string content as input
- Error messages include context (filename or content preview)

### 3.1. XSLT 2.0 and EXSLT Extensions

#### Overview

**XSLT 2.0** is a major upgrade to the XSLT specification that introduces significant new capabilities including XPath 2.0 expressions, user-defined functions, and native support for multiple output documents. However, **libxslt (used by xml-xslt.lua) only supports XSLT 1.0**.

**EXSLT** (Extensions to XSLT) is a community-driven initiative that provides standardized extensions for XSLT 1.0 processors, adding functionality that bridges the gap between XSLT 1.0 and 2.0.

**Key Point**: For XSLT processing in this system, we have XSLT 1.0 with EXSLT extensions, not XSLT 2.0.

#### XSLT 2.0 Features (Not Available in libxslt)

If you had XSLT 2.0 support, you would have:

1. **XPath 2.0 Expressions**:
   - Rich data type system
   - Sequence operations
   - FLWOR expressions (For-Let-Where-Order-Return)

2. **Strong Typing and Schema Awareness**:
   - Type-aware transformations
   - Schema validation integration
   - Type safety at compile time

3. **User-Defined Functions**:
   - Create custom XSLT functions
   - Function libraries and imports

4. **Multiple Output Documents**:
   - Native `<xsl:result-document>` support
   - Generate multiple files from one transformation

5. **Advanced String Processing**:
   - Built-in regular expressions
   - `matches()`, `replace()`, `tokenize()`

6. **Date/Time Functions**:
   - Native date/time support
   - Duration calculations

7. **Grouping Capabilities**:
   - `<xsl:for-each-group>` for data grouping
   - Complex grouping patterns

#### EXSLT Extensions for XSLT 1.0

EXSLT provides these extensions that work with libxslt:

##### Common Module (`http://exslt.org/common`)

```xml
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:exsl="http://exslt.org/common"
    extension-element-prefixes="exsl">

<!-- Node set conversion -->
<xsl:variable name="nodes" select="exsl:node-set($variable)"/>

<!-- Load external document -->
<xsl:variable name="external" select="document('external.xml')"/>
</xsl:stylesheet>
```

##### Math Module (`http://exslt.org/math`)

**Available Functions**:

- `math:abs()` - Absolute value
- `math:acos()` - Arc cosine
- `math:asin()` - Arc sine
- `math:atan()` - Arc tangent
- `math:atan2()` - Arc tangent with two parameters
- `math:cos()` - Cosine
- `math:sin()` - Sine
- `math:tan()` - Tangent
- `math:exp()` - Exponential
- `math:log()` - Natural logarithm
- `math:sqrt()` - Square root
- `math:power()` - Power function
- `math:highest()` - Maximum value in node-set
- `math:lowest()` - Minimum value in node-set
- `math:max()` - Maximum value
- `math:min()` - Minimum value
- `math:constant()` - Mathematical constants (PI, E)

**Example**:

```xml
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:math="http://exslt.org/math"
    extension-element-prefixes="math">

<xsl:template match="prices">
    <statistics>
        <count><xsl:value-of select="count(price)"/></count>
        <sum><xsl:value-of select="sum(price)"/></sum>
        <average><xsl:value-of select="sum(price) div count(price)"/></average>
        <rounded-average>
            <xsl:value-of select="math:round(sum(price) div count(price) * 100) div 100"/>
        </rounded-average>
        <min><xsl:value-of select="math:min(price)"/></min>
        <max><xsl:value-of select="math:max(price)"/></max>
    </statistics>
</xsl:template>

<xsl:template match="coordinates">
    <distance>
        <xsl:value-of select="math:sqrt(x*x + y*y)"/>
    </distance>
    <angle>
        <xsl:value-of select="math:atan2(y, x) * 180 div math:constant('PI')"/>
    </angle>
</xsl:template>
</xsl:stylesheet>
```

##### String Module (`http://exslt.org/strings`)

**Available Functions**:

- `str:tokenize()` - Split string into tokens
- `str:split()` - Split string by delimiter
- `str:replace()` - String replacement with regex
- `str:padding()` - Add padding to string
- `str:encode-uri()` - URI encoding
- `str:decode-uri()` - URI decoding
- `str:concat()` - Concatenate strings
- `str:upper-case()` - Uppercase conversion
- `str:lower-case()` - Lowercase conversion

**Example**:

```xml
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:str="http://exslt.org/strings"
    extension-element-prefixes="str">

<xsl:template match="full-name">
    <proper-name>
        <!-- Capitalize first letter, lowercase the rest -->
        <xsl:value-of select="str:upper-case(substring(., 1, 1))"/>
        <xsl:value-of select="str:lower-case(substring(., 2))"/>
    </proper-name>
</xsl:template>

<xsl:template match="tags">
    <tag-list>
        <xsl:for-each select="str:tokenize(., ',')">
            <tag><xsl:value-of select="str:trim(.)"/></tag>
        </xsl:for-each>
    </tag-list>
</xsl:template>

<xsl:template match="description">
    <word-count>
        <xsl:value-of select="count(str:tokenize(., ' '))"/>
    </word-count>
</xsl:template>
</xsl:stylesheet>
```

##### Date Module (`http://exslt.org/dates-and-times`)

**Available Functions**:

- `date:date-time()` - Current date/time
- `date:add()` - Add duration to date
- `date:difference()` - Difference between dates
- `date:format()` - Format date as string
- `date:parse()` - Parse string to date
- `date:sum()` - Sum of durations
- `date:seconds()` - Convert to seconds

**Example**:

```xml
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:date="http://exslt.org/dates-and-times"
    extension-element-prefixes="date">

<xsl:template match="/">
    <timestamp>
        <iso-format><xsl:value-of select="date:date-time()"/></iso-format>
        <readable-format>
            <xsl:value-of select="date:format(date:date-time(), 'yyyy-MM-dd HH:mm:ss')"/>
        </readable-format>
        <us-format>
            <xsl:value-of select="date:format(date:date-time(), 'MM/dd/yyyy hh:mm a')"/>
        </us-format>
    </timestamp>
</xsl:template>

<xsl:template match="order[ship-date]">
    <shipping-info>
        <order-date><xsl:value-of select="ship-date"/></order-date>
        <estimated-delivery>
            <xsl:value-of select="date:add(ship-date, 'P7D')"/>
        </estimated-delivery>
    </shipping-info>
</xsl:template>
</xsl:stylesheet>
```

##### Sets Module (`http://exslt.org/sets`)

**Available Functions**:

- `set:difference()` - Set difference
- `set:intersection()` - Set intersection
- `set:distinct()` - Remove duplicates
- `set:has-same-node()` - Check if nodes are same
- `set:leading()` - Nodes in first set before second
- `set:trailing()` - Nodes in first set after second

**Example**:

```xml
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:set="http://exslt.org/sets"
    extension-element-prefixes="set">

<xsl:template match="products">
    <unique-products>
        <xsl:for-each select="set:distinct(product/@id)">
            <product id="{.}">
                <xsl:value-of select="../product[@id=current()/@id]/name"/>
            </product>
        </xsl:for-each>
    </unique-products>
</xsl:template>

<xsl:template match="comparison">
    <differences>
        <only-in-a>
            <xsl:copy-of select="set:difference(set:a, set:b)"/>
        </only-in-a>
        <only-in-b>
            <xsl:copy-of select="set:difference(set:b, set:a)"/>
        </only-in-b>
        <common>
            <xsl:copy-of select="set:intersection(set:a, set:b)"/>
        </common>
    </differences>
</xsl:template>
</xsl:stylesheet>
```

#### Implementation with xml-xslt.lua

Currently, xml-xslt.lua uses libxslt which includes EXSLT support. EXSLT functions are automatically available when declared in XSLT stylesheets.

**Checking EXSLT Support**:

```lua
local xmlXslt = require "xml/xml-xslt"

-- Check if EXSLT is available (it should be with libxslt)
print("EXSLT supported:", xmlXslt.isEXSLTSupported())

-- Get processor information
local info = xmlXslt.getInfo()
print("EXSLT modules:", table.concat(info.exslt_modules or {}, ", "))
```

**Using EXSLT in Transformations**:

```lua
local xsltWithExslt = [[
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:math="http://exslt.org/math"
    xmlns:str="http://exslt.org/strings"
    xmlns:date="http://exslt.org/dates-and-times"
    extension-element-prefixes="math str date">

    <xsl:template match="/">
        <report>
            <timestamp>
                <xsl:value-of select="date:format(date:date-time(), 'yyyy-MM-dd')"/>
            </timestamp>
            <calculations>
                <pi-value><xsl:value-of select="math:constant('PI')"/></pi-value>
                <sqrt-2><xsl:value-of select="math:sqrt(2)"/></sqrt-2>
            </calculations>
        </report>
    </xsl:template>
</xsl:stylesheet>
]]

local result, errors = xmlXslt.convert("data.xml", xsltWithExslt, {
    format = "xml"
})
```

#### Migration Considerations

**When to Use EXSLT vs XSLT 2.0**:

1. **Use EXSLT** (Current Implementation):
   - Working with libxslt/xml-xslt.lua
   - Need advanced functions but can stay with XSLT 1.0
   - Maintain compatibility with existing stylesheets

2. **Need XSLT 2.0**:
   - Schema-aware transformations
   - Multiple output documents
   - User-defined functions
   - XPath 2.0 sequences

3. **Alternatives for XSLT 2.0**:
   - **Saxon-C**: C++ version of Saxon processor
   - **Saxon-HE**: Free Java edition
   - External processor integration

**Performance Considerations**:

- EXSLT functions are implemented in C within libxslt
- Generally faster than XSLT workarounds
- Additional memory overhead for complex operations
- Consider the cost vs benefit for each EXSLT module used

### 4. xml-util.lua - Common Utilities

Location: `/Volumes/nc/nc-server/lib/xml/xml-util.lua`

**Primary Functions:**

```lua
-- Error handling and cleanup
xmlUtil.shutdown()                           -- Cleanup XML resources
xmlUtil.getLastError()                       -- Get last XML error
xmlUtil.collectErrorArray(existingArray)     -- Collect errors

-- Content handling
xmlUtil.xmlContent(xmlFileOrContent)         -- Get XML content from file/string
xmlUtil.returnError(option, msg, ...)                -- Create error array

-- Error context (for callbacks)
xmlUtil.errorCtx                             -- Error context pointer
xmlUtil.errorCallback                        -- Error callback function
```

### 5. XML Parsing Libraries

#### expat.lua - High-performance XML parser

```lua
-- Parsing functions
expat.parse(spec_t, callbacks)               -- Parse with callbacks
expat.treeParse(spec_t, known_tags)          -- Parse to tree structure

-- Parser methods
expat.parser.read(read, callbacks)           -- Parse from reader
expat.parser.string(s, callbacks)            -- Parse string
expat.parser.path(file, callbacks)           -- Parse file
```

#### xmllpegparser.lua - LPeg-based parser

```lua
local xmlParser = require "xml/xmllpegparser"

-- Parse to table
local result = xmlParser.parse(xmlString, options)

-- Parse with visitor pattern
local result = xmlParser.parse(xmlString, {
    onStartElement = function(name, attrs) ... end,
    onEndElement = function(name) ... end,
    onText = function(text) ... end,
    onComment = function(comment) ... end
})
```

#### slaxml.lua - Pure Lua XML parser

```lua
local slaxml = require "xml/slaxml"

-- Parse with callbacks
slaxml.parse(xmlString, {
    StartElement = function(name, attrs) ... end,
    EndElement = function(name) ... end,
    Text = function(text) ... end
})
```

### 6. XML Security (xmlsec.lua)

Location: `/Volumes/nc/nc-server/lib/xml/xmlsec.lua`

**Primary Functions:**

```lua
-- Digital signatures
xmlsec.action("sign", inFile, keyFile, password, outFile)     -- Sign XML
xmlsec.action("verify", inFile, keyFile, password, outFile)   -- Verify signature

-- Lifecycle
xmlsec.init()                            -- Initialize XMLSec
xmlsec.shutdown()                        -- Shutdown XMLSec
xmlsec.outText(doc, outFile)            -- Output document
```

### 7. XML Generation (xml_write.lua)

Location: `/Volumes/nc/nc-server/lib/xml/xml_write.lua`

```lua
local xml = require "xml/xml_write"

-- Create XML document
xml.doc(function()
    xml.element("root", {xmlns = "http://example.com"}, function()
        xml.element("item", {id = 1}, function()
            xml.text("Content")
        end)
    end)
end)
```

### 8. High-level XML Helpers (xml.lua)

Location: `/Volumes/nc/nc-server/lib/xml/xml.lua`

```lua
-- Conversion functions
xml.xmlStringToTable(xmlString)          -- Convert string to table
xml.xmlFileToTable(pathToXml)            -- Convert file to table

-- Utility functions
xml.escapeXml(txt)                       -- Escape XML entities
xml.findXmlTagWithAttribute(text, tag, attr)  -- Find tags
xml.captureXmlAttribute(text, attr)       -- Extract attributes
```

## Testing Framework

### Test Files Location: `/Volumes/nc/nc-server/test/xml/`

#### xml-lint-test.lua (777 lines)

**Test Categories:**

- Module loading and initialization
- Well-formedness checking (positive/negative tests)
- Schema validation (XSD, DTD, RelaxNG, Schematron)
- XPath queries and results
- XML formatting and cleaning
- Error handling and edge cases
- Option combinations and integration tests
- Performance tests (large files, streaming)

**Key Test Functions:**

```lua
test.lint.require()                     -- Module loading
test.lint.wellFormed()                  -- Valid XML
test.lint.wellFormedInvalid()           -- Invalid XML
test.lint.validateXSD()                 -- XSD validation
test.lint.validateDTD()                 -- DTD validation
test.lint.xpath()                       -- XPath queries
test.lint.format()                      -- XML formatting
test.lint.options()                     -- Individual option tests
```

#### xml-schema-test.lua (170 lines)

**Test Categories:**

- DTD validation with/without replacement
- XSD validation scenarios
- cXML document processing
- Error handling for invalid inputs
- Version synchronization

**Key Test Functions:**

```lua
test.schema.require()                   -- Module loading
test.schema.dtd()                       -- Basic DTD test
test.schema.dtdNoReplace()              -- DTD without replacement
test.schema.dtdInvalidXml()             -- Invalid XML test
test.schema.xsdInvalid()                -- Invalid XSD test
test.schema.xsdValid()                  -- Valid XSD test
```

#### xml-xslt-test.lua (366 lines)

**Test Categories:**

- Basic XSLT transformations
- Output format variations
- File-based conversions
- Error handling scenarios
- Unicode and large content handling
- Batch processing

**Key Test Functions:**

```lua
test.xslt.require()                     -- Module loading
test.xslt.convert()                     -- Basic conversion
test.xslt.convertFile()                 -- File conversion
test.xslt.convertToHtml()               -- HTML output
test.xslt.convertString()               -- String conversion
test.xslt.invalidXSLT()                 -- Error handling
test.xslt.unicode()                     -- Unicode test
test.xslt.largeContent()                -- Large file test
```

#### xmllpegparser-test.lua (148 lines)

**Test Categories:**

- Parser requirements and initialization
- XML structure parsing
- Entity handling
- DOCTYPE processing
- Custom visitor patterns

**Key Test Functions:**

```lua
test.parser.require()                   -- Module loading
test.parser.basicParse()                -- Basic parsing
test.parser.xmlStructure()              -- Structure verification
test.parser.entities()                  -- Entity handling
test.parser.doctype()                   -- DOCTYPE parsing
```

### Test Data Organization

```text
xml-test-data/
├── Schemas/
│   ├── all_0.xsd              -- Schema with <all> complex type
│   ├── all_attrs.xsd          -- Attribute validation
│   ├── choice_0.xsd           -- Choice elements
│   └── complex_0.xsd          -- Complex types
├── XML Files/
│   ├── all_1.xml              -- Valid XML (a, b, c elements)
│   ├── all_4.xml              -- Invalid XML (empty)
│   ├── xpath_test.xml         -- Complex XML for XPath
│   └── comprehensive_test.xml -- Feature-rich XML
├── DTD/
│   └── test.dtd               -- Basic DTD definitions
├── XSLT/
│   ├── convert.xsl            -- Book catalog to HTML
│   └── transform.xsl          -- General transformation
├── RelaxNG/
│   └── test.rng               -- RelaxNG schema
├── Schematron/
│   └── test.sch               -- Schematron rules
└── Output/
    └── *.html                 -- Generated HTML files
```

## Usage Examples

### Basic XML Validation

```lua
local xmlLint = require "xml/xml-lint"

-- Check well-formedness
local isValid, errors = xmlLint.checkWellFormed("data.xml")
if not isValid then
    for _, err in ipairs(errors) do
        print("Error:", err.message)
    end
end

-- Validate against XSD
local isValid, errors = xmlLint.validateXSD("data.xml", "schema.xsd")
if not isValid then
    -- Handle validation errors
end

-- DTD validation with options
local isValid, errors = xmlSchema.validateDTD(
    "data.xml",
    "schema.dtd",
    {replace_doctype = true, replace_version = true}
)
```

### XPath Queries

```lua
local xmlLint = require "xml/xml-lint"

-- Simple XPath
local result, errors = xmlLint.xpath("data.xml", "//book[price>35]")

-- XPath with namespace
local result, errors = xmlLint.xpath("data.xml", "//ns:book", {
    namespaces = {
        ns = "http://example.com/ns"
    }
})

-- XPath in lint
local isValid, errors, results = xmlLint.lint("data.xml", {
    xpath = "//item[@id='123']",
    no_out = true
})
```

### XSLT Transformations

```lua
local xmlXslt = require "xml/xml-xslt"

-- Transform XML to HTML
local success, errors, result = xmlXslt.convert(
    "data.xml",
    "transform.xsl",
    {
        output_method = "html",
        output_encoding = "UTF-8",
        output_indent = true,
        validate_input = true
    },
    "output.html"
)

-- Transform XML string
local xmlString = "<root><item>Test</item></root>"
local success, errors, htmlOutput = xmlXslt.convert(
    xmlString,
    "transform.xsl",
    {
        output_method = "text",
        parameters = {
            param1 = "value1",
            param2 = "value2"
        }
    }
)
```

### XML to HTML Conversion

#### Overview

XML to HTML conversion is the process of transforming XML data into HTML format for web display. This is primarily accomplished using XSLT (Extensible Stylesheet Language Transformations), which provides a powerful way to convert XML documents into different formats including HTML.

#### Methods of Conversion

1. **XSLT Transformation** (Primary method)
2. **Direct parsing and generation** (For simple cases)
3. **Template-based systems** (Custom solutions)

#### XSLT for HTML Conversion

**Basic XSLT Template for HTML**:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

    <xsl:output method="html"
                encoding="UTF-8"
                indent="yes"
                doctype-public="-//W3C//DTD HTML 4.01//EN"/>

    <!-- Root template -->
    <xsl:template match="/">
        <html>
            <head>
                <title><xsl:value-of select="/document/title"/></title>
                <style>
                    body { font-family: Arial, sans-serif; }
                    table { border-collapse: collapse; width: 100%; }
                    th, td { border: 1px solid #ddd; padding: 8px; }
                    th { background-color: #f2f2f2; }
                </style>
            </head>
            <body>
                <xsl:apply-templates/>
            </body>
        </html>
    </xsl:template>

    <!-- Document title -->
    <xsl:template match="document/title">
        <h1><xsl:value-of select="."/></h1>
    </xsl:template>

    <!-- Table generation -->
    <xsl:template match="data/table">
        <table>
            <thead>
                <tr>
                    <xsl:for-each select="columns/column">
                        <th><xsl:value-of select="@name"/></th>
                    </xsl:for-each>
                </tr>
            </thead>
            <tbody>
                <xsl:apply-templates select="rows/row"/>
            </tbody>
        </table>
    </xsl:template>

    <!-- Table rows -->
    <xsl:template match="row">
        <tr>
            <xsl:for-each select="cell">
                <td><xsl:value-of select="."/></td>
            </xsl:for-each>
        </tr>
    </xsl:template>

    <!-- List generation -->
    <xsl:template match="data/list">
        <ul>
            <xsl:apply-templates select="item"/>
        </ul>
    </xsl:template>

    <xsl:template match="item">
        <li><xsl:value-of select="."/></li>
    </xsl:template>

    <!-- Handle unknown elements -->
    <xsl:template match="*">
        <div class="unknown-element">
            <strong>&lt;<xsl:value-of select="name()"/>&gt;</strong>
            <xsl:apply-templates/>
        </div>
    </xsl:template>
</xsl:stylesheet>
```

**Advanced XSLT Features**:

```xml
<!-- Conditional rendering -->
<xsl:template match="product">
    <div class="product">
        <h3><xsl:value-of select="name"/></h3>
        <xsl:if test="@featured='true'">
            <span class="featured-badge">FEATURED</span>
        </xsl:if>
        <p>Price: $<xsl:value-of select="price"/></p>
        <xsl:choose>
            <xsl:when test="stock &gt; 0">
                <span class="in-stock">In Stock</span>
            </xsl:when>
            <xsl:otherwise>
                <span class="out-of-stock">Out of Stock</span>
            </xsl:otherwise>
        </xsl:choose>
    </div>
</xsl:template>

<!-- Looping and numbering -->
<xsl:template match="chapter">
    <div class="chapter">
        <h2>
            Chapter <xsl:number level="single" format="1"/>:
            <xsl:value-of select="title"/>
        </h2>
        <div class="content">
            <xsl:apply-templates/>
        </div>
    </div>
</xsl:template>

<!-- Sorting -->
<xsl:template match="products">
    <xsl:for-each select="product">
        <xsl:sort select="price" data-type="number" order="ascending"/>
        <div class="product">
            <xsl:value-of select="name"/> - $<xsl:value-of select="price"/>
        </div>
    </xsl:for-each>
</xsl:template>

<!-- Variables and parameters -->
<xsl:variable name="currency" select="'USD'"/>
<xsl:param name="showImages" select="true()"/>

<xsl:template match="price">
    <span class="price">
        <xsl:if test="$showImages and ../image">
            <img src="{../image}" alt="{../name}" class="product-thumb"/>
        </xsl:if>
        <xsl:value-of select="format-number(., '###,##0.00')"/>
        <xsl:value-of select="$currency"/>
    </span>
</xsl:template>
```

#### Using XML to HTML Conversion

**Basic Conversion**:

```lua
local xmlXslt = require "xml/xml-xslt"

-- Convert XML file to HTML
local success, errors, result = xmlXslt.convert(
    "catalog.xml",           -- Input XML file
    "catalog-to-html.xsl",   -- XSLT stylesheet
    {                        -- Options
        output_method = "html",
        output_encoding = "UTF-8",
        output_indent = true,
        validate_input = true
    },
    "catalog.html"           -- Output file
)

if not success then
    print("Conversion failed:")
    for _, err in ipairs(errors) do
        print("  " .. err.message)
    end
end
```

**Dynamic Conversion with Parameters**:

```lua
-- Convert with runtime parameters
local success, errors = xmlXslt.convert(
    "report.xml",
    "report.xsl",
    {
        output_method = "html",
        parameters = {
            ["show-details"] = "true",
            ["report-date"] = os.date("%Y-%m-%d"),
            ["user-id"] = userId
        }
    },
    "report.html"
)
```

**Convert XML String to HTML**:

```lua
-- Convert XML data from memory
local xmlData = [[
<catalog>
    <book id="1">
        <title>Learning XML</title>
        <author>Erik T. Ray</author>
        <price>39.95</price>
    </book>
    <book id="2">
        <title>XML in Action</title>
        <author>William J. Pardi</author>
        <price>49.95</price>
    </book>
</catalog>
]]

local success, errors, htmlOutput = xmlXslt.convert(
    xmlData,
    "catalog-to-html.xsl",
    {
        output_method = "html"
    }
)

if success then
    -- Write HTML to file or serve via web
    local file = io.open("output.html", "w")
    file:write(htmlOutput)
    file:close()
end
```

#### Best Practices for XML to HTML Conversion

1. **Separate Content from Presentation**:
   - Keep data in XML
   - Use CSS for styling
   - Generate clean semantic HTML

2. **Use Semantic HTML Elements**:

   ```xml
   <!-- Good: Use semantic elements -->
   <xsl:template match="article">
       <article>
           <header><xsl:value-of select="title"/></header>
           <section><xsl:apply-templates select="content"/></section>
       </article>
   </xsl:template>

   <!-- Avoid: Generic divs -->
   <xsl:template match="article">
       <div class="article">
           <div class="title"><xsl:value-of select="title"/></div>
           <div class="content"><xsl:apply-templates select="content"/></div>
       </div>
   </xsl:template>
   ```

3. **Handle Edge Cases**:

   ```xml
   <!-- Handle empty elements -->
   <xsl:template match="description[.!='']">
       <p class="description"><xsl:value-of select="."/></p>
   </xsl:template>

   <!-- Provide defaults -->
   <xsl:template match="price">
       <span class="price">
           $<xsl:value-of select="format-number(., '###,##0.00')"/>
       </span>
   </xsl:template>
   <xsl:template match="price[not(.)]">
       <span class="price">Price not available</span>
   </xsl:template>
   ```

4. **Performance Optimization**:

   ```xml
   <!-- Use keys for fast lookups -->
   <xsl:key name="product-by-id" match="product" use="@id"/>

   <xsl:template match="order-item">
       <xsl:variable name="product" select="key('product-by-id', @product-id)"/>
       <div class="item">
           <xsl:value-of select="$product/name"/>
       </div>
   </xsl:template>
   ```

#### Common Use Cases

1. **Report Generation**:
   - Convert XML data to HTML reports
   - Include charts and graphs
   - Support printable versions

2. **Content Management**:
   - XML content storage
   - Multiple output formats
   - Template-based presentation

3. **Data Visualization**:
   - Tabular data to HTML tables
   - Hierarchical data to nested lists
   - RSS/Atom feeds to HTML

4. **Email Generation**:
   - XML email templates
   - Personalized content
   - HTML email output

#### HTML Output Options

```lua
-- Different HTML output configurations
options = {
    -- HTML 4.01 (default)
    output_method = "html",
    output_version = "4.01",

    -- HTML 5 ( XHTML-like syntax)
    output_method = "xhtml",
    output_version = "5.0",

    -- With DOCTYPE
    output_doctype_public = "-//W3C//DTD HTML 4.01//EN",
    output_doctype_system = "http://www.w3.org/TR/html4/strict.dtd",

    -- Standalone HTML
    output_standalone = "yes",

    -- Include XML declaration for XHTML
    omit_xml_declaration = "no"
}
```

### XML Parsing with Different Parsers

```lua
-- Using Expat (fast, streaming)
local expat = require "xml/expat"
local result = expat.treeParse({file = "data.xml"})

-- Using LPeg parser (pattern-based)
local lpegParser = require "xml/xmllpegparser"
local result = lpegParser.parse(xmlString, {
    onStartElement = function(name, attrs)
        print("Start:", name)
    end,
    onText = function(text)
        print("Text:", text)
    end
})

-- Using SLAXML (pure Lua)
local slaxml = require "xml/slaxml"
slaxml.parse(xmlString, {
    StartElement = function(name, attrs)
        -- Handle element start
    end
})
```

### XML Generation

```lua
local xml = require "xml/xml_write"

-- Generate XML document
local xmlContent = xml.doc(function()
    xml.declaration("1.0", "UTF-8")
    xml.element("catalog", {xmlns = "http://example.com"}, function()
        xml.element("book", {id = "1"}, function()
            xml.element("title", {}, "XML Programming")
            xml.element("price", {}, "29.99")
        end)
    end)
end)

-- Write to file
local file = io.open("output.xml", "w")
file:write(xmlContent)
file:close()
```

### XML Security Operations

```lua
local xmlsec = require "xml/xmlsec"

-- Initialize XMLSec
xmlsec.init()

-- Sign XML document
xmlsec.action("sign", "document.xml", "private.key", "password", "signed.xml")

-- Verify signature
local valid = xmlsec.action("verify", "signed.xml", "public.key", nil, nil)

-- Cleanup
xmlsec.shutdown()
```

## Error Handling

### Error Types

```lua
ERROR_TYPE = {
    GENERIC = 0,        -- General parsing errors
    WARNING = 1,        -- Validation warnings
    VALIDATION = 2,     -- Schema validation errors
    SECURITY = 3,       -- Security violations
    NETWORK = 4,        -- Network-related errors
    IO = 5             -- Input/output errors
}
```

### Error Structure

```lua
error = {
    type = error_type_number,
    message = "Error description",
    line = line_number,        -- When available
    column = column_number,    -- When available
    file = "filename.xml"      -- When available
}
```

### Error Handling Pattern

```lua
local function processXML(xmlFile)
    local isValid, errors = xmlLint.checkWellFormed(xmlFile)

    if not isValid then
        print("XML validation failed:")
        for _, err in ipairs(errors) do
            print(string.format("  Type %d: %s", err.type, err.message))
            if err.line then
                print(string.format("  Line: %d, Column: %d", err.line, err.column))
            end
        end
        return false
    end

    -- Continue processing
    return true
end
```

## Performance Considerations

### Optimization Strategies

1. **Choose the right parser:**
   - Expat: Fastest, for large files and streaming
   - LPeg: Good balance of speed and flexibility
   - SLAXML: Pure Lua, no native dependencies

2. **Memory management:**

   ```lua
   -- Enable streaming for large files
   local result = xmlLint.lint("huge.xml", {stream = true})

   -- Cleanup after transformations
   xmlXslt.cleanup()
   xmlUtil.shutdown()
   ```

3. **Batch processing:**

   ```lua
   -- Process multiple files efficiently
   local files = {"file1.xml", "file2.xml", "file3.xml"}
   for _, file in ipairs(files) do
       local result = xmlLint.lint(file, {no_out = true})
   end
   xmlUtil.shutdown()  -- Cleanup once
   ```

4. **Caching:**
   - Cache parsed XSLT stylesheets
   - Reuse validation contexts
   - Store compiled patterns

### Memory Usage

- Call `xmlUtil.shutdown()` after processing to free memory
- Use `cleanup_after_transform = true` in XSLT
- Enable `stream = true` for very large files
- Monitor with `huge = true` for files > 100MB

## Security Considerations

### Secure XML Processing

```lua
-- Disable network access
local result = xmlLint.lint("untrusted.xml", {
    no_net = true,
    no_xinclude = true
})

-- Limit amplification
local result = xmlXslt.convert(xml, xslt, {
    security_amp = 1000,  -- Limit expansion factor
    no_net = true
})

-- Validate all input
local result = xmlXslt.convert(xml, xslt, {
    validate_input = true,
    validate_stylesheet = true
})
```

### Security Features

- Network access control (`no_net`)
- XInclude processing control (`no_xinclude`)
- Amplification limit enforcement
- Input validation requirements
- DTD external entity protection

## Integration with ERP System

### ERP Sync Integration

The ERP sync module uses XML processing for:

1. **Import Validation:**

   ```lua
   -- In erp-sync.lua (lines 118-129)
   local isValid, errors = xmlLint.validateXSD(tempFile, schemaFile)
   if not isValid then
       -- Log errors and reject file
   end
   ```

2. **Export Validation:**

   ```lua
   -- In erp-sync.lua (lines 100-118)
   -- Generate XML
   -- Validate against schema
   -- Clean up temporary files
   ```

3. **cXML Processing:**
   - Special handling for cXML documents
   - DOCTYPE replacement for DTD validation
   - Version synchronization

## Troubleshooting

### Common Issues and Solutions

1. **Module loading errors**

   ```bash
   # Check if libraries are in path
   ls /Volumes/nc/nc-server/lib/xml/

   # Test module loading
   lua -e "require('xml/xml-lint')"
   ```

2. **Validation fails unexpectedly**

   ```lua
   -- Enable verbose output
   local result = xmlLint.lint("file.xml", {quiet = false})

   -- Check error details
   for _, err in ipairs(errors) do
       print("Full error:", json.toJsonRaw(err))
   end
   ```

3. **XSLT transformation errors**

   ```lua
   -- Validate inputs first
   local xmlValid = xmlLint.checkWellFormed(xmlFile)
   local xsltValid = xmlLint.checkWellFormed(xsltFile)

   -- Use validated transformation
   local result = xmlXslt.convert(xml, xslt, {
       validate_input = true,
       validate_stylesheet = true
   })
   ```

4. **Memory leaks**

   ```lua
   -- Ensure proper cleanup
   xmlUtil.shutdown()
   xmlXslt.cleanup()
   xmlsec.shutdown()
   ```

### Debug Commands

```bash
# Run all tests
cd /Volumes/nc/nc-server/test/xml
lua xml-lint-test.lua
lua xml-schema-test.lua
lua xml-xslt-test.lua

# Run specific test
lua xml-lint-test.lua test.lint.validateXSD

# Check file existence
ls -la xml-test-data/

# Validate with external tools
xmllint --schema schema.xsd file.xml
xsltproc transform.xsl file.xml
```

## Dependencies

### Core Dependencies

```lua
-- Required modules
"mffi"              -- FFI bindings
"xmlua"             -- Lua XML library (libxml2 wrapper)
"xmlsec-xslt"       -- XSLT FFI bindings (libxslt)
"fs"                -- File system utilities
"peg"               -- Pattern matching
"util"              -- General utilities
"json"              -- JSON encoding/decoding

-- External libraries
libxml2             -- XML C library (2.9+)
libxslt             -- XSLT C library (1.1+)
libxmlsec1          -- XML Security library
libexpat            -- Expat XML parser
OpenSSL             -- Crypto for XMLSec
```

### Platform-specific Paths

```lua
-- Windows
xmlPath = [[C:\Program Files\libxml2\]]
xsltPath = [[C:\Program Files\libxslt\]]
xmlsecPath = [[C:\Program Files\xmlsec\]]

-- macOS (via Homebrew)
xmlPath = [[/usr/local/opt/libxml2/]]
xsltPath = [[/usr/local/opt/libxslt/]]
xmlsecPath = [[/usr/local/opt/xmlsec/]]

-- Linux
xmlPath = [[/usr/lib/x86_64-linux-gnu/]]
xsltPath = [[/usr/lib/x86_64-linux-gnu/]]
xmlsecPath = [[/usr/lib/x86_64-linux-gnu/]]
```

## API Reference

### Complete Function Index

#### xml-lint.lua

- `lint(xmlFile, options)`
- `checkWellFormed(xmlFile)`
- `validateXSD(xmlFile, xsdFile)`
- `validateDTD(xmlFile, dtdFile)`
- `validateRelaxNG(xmlFile, rngFile)`
- `validateSchematron(xmlFile, schFile)`
- `parseXML(xmlFile, options)`
- `formatXML(xmlFile, outputFile)`
- `cleanXML(xmlFile, options)`
- `processXInclude(xmlFile, outputFile)`
- `xpath(xmlFile, query)`
- `lintXpath(xmlFile, query)`
- `listNode(doc)`
- `getXMLEncoding(xmlFile)`
- `getXMLVersion(xmlFile)`
- `isHTML(xmlFile)`
- `autoLint(xmlFile)`

#### xml-schema.lua

- `validateXSD(XMLFileName, XSDFileName)`
- `validateDTD(xmlOrFile, dtdFileName, option)`

#### xml-xslt.lua

- `convert(xmlOrFilename, xsltOrFilename, option, outputFile)`
- `getInfo()`
- `cleanup()`
- `shutdown()`

#### xml-util.lua

- `shutdown()`
- `getLastError()`
- `collectErrorArray(existingArray)`
- `xmlContent(xmlFileOrContent)`
- `returnError(msg, ...)`

## Version Information

- XML linting: Based on libxml2 2.9+
- XSLT processing: Based on libxslt 1.1+
- XML Security: Based on xmlsec1 1.2+
- Last updated: 2024-12-09
- Documentation version: 2.0

## Migration Guide

### From Previous Versions

1. **Error handling changes:**

   ```lua
   -- Old
   local errors = xmlLint.validateXSD(file, xsd)

   -- New (consistent error array)
   local isValid, errors = xmlLint.validateXSD(file, xsd)
   ```

2. **Module initialization:**

   ```lua
   -- No longer needed
   -- xmlLint.init()
   -- xmlXslt.init()

   -- Automatic on first use
   ```

3. **Options format:**

   ```lua
   -- Old positional arguments
   xmlLint.lint(file, true, false, "schema.xsd")

   -- New named options
   xmlLint.lint(file, {
       no_out = true,
       schema = "schema.xsd"
   })
   ```

### Breaking Changes

- `validateSchematron` temporarily disabled (needs library update)
- Error handling unified across all modules
- Options now use key-value table format
- Network access disabled by default for security

## XML Validation Languages - Comprehensive Reference

### Overview

XML validation is the process of ensuring an XML document conforms to defined rules and constraints. Multiple validation languages exist, each with different strengths and use cases:

### Document Type Definition (DTD)

**Definition**: The original schema language for XML, defined in the XML 1.0 specification

**Syntax**:

```xml
<!-- DTD declarations -->
<!DOCTYPE root [
  <!ELEMENT root (child+)>
  <!ELEMENT child (#PCDATA)>
  <!ATTLIST child id CDATA #REQUIRED>
]>
```

**Key Features**:

- Non-XML syntax
- Element and attribute declarations
- Entity definitions
- Simple content models

**Limitations**:

- No data type support (except basic tokenized types)
- No namespace support
- Limited expressive power

### XML Schema Definition (XSD)

**Definition**: W3C standard for defining XML document structure and data types

**Example**:

```xml
<?xml version="1.0"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="book">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="title" type="xs:string"/>
        <xs:element name="author" type="xs:string"/>
        <xs:element name="price" type="xs:decimal"/>
      </xs:sequence>
      <xs:attribute name="isbn" type="xs:string" use="required"/>
    </xs:complexType>
  </xs:element>
</xs:schema>
```

**Key Features**:

- Rich data type system (built-in and custom types)
- Namespace support
- Type inheritance and extension
- Reusability through named types
- Assertion facets for constraints

**Use Cases**:

- Complex data modeling
- Type-rich XML documents
- Web service contracts
- Data interchange standards

### RelaxNG (Regular Language for XML Next Generation)

**Definition**: ISO/IEC 19757-2 standard for XML schema validation, designed as a simpler alternative to XSD

**Two Syntaxes**:

1. **XML Syntax**:

```xml
<grammar xmlns="http://relaxng.org/ns/structure/1.0">
  <start>
    <element name="book">
      <element name="title"><text/></element>
      <element name="author"><text/></element>
      <optional>
        <element name="price"><data type="decimal"/></element>
      </optional>
    </element>
  </start>
</grammar>
```

1. **Compact Syntax**:

```rng
grammar {
  start = element book {
    element title { text },
    element author { text },
    element price? { xsd:decimal }
  }
}
```

**Pattern Types**:

```rng
# Structural patterns
element name { pattern }      # Defines elements
attribute name { pattern }    # Defines attributes
text                          # Text content
empty                         # Empty content

# Combination patterns
pattern1 | pattern2           # Choice (alternatives)
pattern1 , pattern2           # Group (sequential)
pattern1 & pattern2           # Interleave (unordered)

# Repetition patterns
pattern*                      # Zero or more
pattern+                      # One or more
pattern?                      # Optional (zero or one)

# Data patterns
data xsd:string              # String datatype
value "literal"              # Specific value
list                         # Whitespace-separated tokens
```

**Advanced Features**:

```rng
# Namespaces
default namespace = "http://example.com"
namespace ns = "http://ns.example.com"

# Named patterns
email = xsd:string { pattern = "[^@]+@[^@]+\.[a-z]+" }

# References
start = element root { header, content }
header = element header { title, date }
content = element content { paragraph* }
```

**Key Features**:

- Simpler than XSD
- Excellent namespace support
- Two equivalent syntaxes (XML and compact)
- Pluggable datatype libraries
- Clear semantics

### Schematron

**Definition**: ISO/IEC 19757-3 standard for rule-based XML validation using XPath expressions

**Core Concept**: Uses XPath expressions to assert conditions about XML documents

**Basic Structure**:

```xml
<?xml version="1.0"?>
<schema xmlns="http://purl.oclc.org/dsdl/schematron" queryBinding="xslt2">
  <!-- Namespace declarations -->
  <ns prefix="inv" uri="http://example.com/invoice"/>

  <!-- Pattern for related rules -->
  <pattern name="Invoice Validation Rules">
    <!-- Rule context: where assertions apply -->
    <rule context="inv:invoice">
      <!-- Assertion: must be true -->
      <assert test="inv:total = sum(inv:line-items/inv:line-item/inv:amount)">
        Error: Invoice total must equal sum of line items
      </assert>

      <!-- Report: must be false -->
      <report test="inv:date > current-date()">
        Warning: Invoice date is in the future
      </report>
    </rule>

    <rule context="inv:line-item">
      <assert test="inv:quantity > 0">
        Error: Quantity must be positive
      </assert>
      <assert test="@discount &lt; 1">
        Error: Discount must be less than 100%
      </assert>
    </rule>
  </pattern>

  <!-- Business rules pattern -->
  <pattern name="Business Rules">
    <rule context="inv:invoice[@type='purchase-order']">
      <assert test="inv:approval">
        Error: Purchase orders require approval
      </assert>
    </rule>
  </pattern>
</schema>
```

**Phased Validation**:

```xml
<schema xmlns="http://purl.oclc.org/dsdl/schematron">
  <phase id="basic">
    <active pattern="BasicChecks"/>
  </phase>

  <phase id="advanced">
    <active pattern="BasicChecks"/>
    <active pattern="BusinessRules"/>
  </phase>

  <pattern id="BasicChecks">
    <!-- Basic validation rules -->
  </pattern>

  <pattern id="BusinessRules">
    <!-- Complex business rules -->
  </pattern>
</schema>
```

**XPath Functions in Schematron**:

- Standard XPath 1.0/2.0 functions
- Schematron-specific extensions
- Custom function libraries

**Key Features**:

- Rule-based validation
- XPath expression power
- Business logic validation
- Co-constraint support
- Human-readable error messages
- Phased validation

**Use Cases**:

- Business rule enforcement
- Complex relationship validation
- Data quality checks
- Cross-element constraints

### Validation Terminology

**Well-Formed XML**:

- Follows basic XML syntax rules
- Single root element
- Properly nested tags
- Quoted attributes
- Required XML declaration if present

**Valid XML**:

- Well-formed AND
- Conforms to schema definition

**Namespace**:

- Qualifies element/attribute names with URIs
- Prevents name conflicts
- Syntax: `prefix:local-name`
- Declaration: `xmlns:prefix="uri"`

**Augmented Validation**:

- Enhanced validation beyond basic structure
- Includes:
  - Co-constraints (relationships between elements)
  - Business rules
  - Conditional validation
  - Cross-document validation

### Comparison of Schema Languages

| Feature | DTD | XSD | RelaxNG | Schematron |
|---------|-----|-----|----------|------------|
| **Syntax** | Non-XML | XML | XML/Compact | XML |
| **Data Types** | Limited | Rich | Pluggable | N/A |
| **Namespaces** | No | Yes | Excellent | Full |
| **Learning Curve** | Easy | Hard | Medium | Medium |
| **Expressiveness** | Basic | High | High | Very High |
| **Co-constraints** | No | Limited | No | Excellent |
| **Error Messages** | Basic | Standard | Clear | Customizable |
| **Schema Size** | Small | Large | Medium | Small |
| **Tool Support** | Universal | Excellent | Good | Good |

### Validation Pipeline

Typical multi-layered validation approach:

1. **Well-formedness Check**:

   ```lua
   local isValid, errors = xmlLint.checkWellFormed("document.xml")
   ```

2. **Namespace Resolution**:

   ```lua
   -- Handled automatically during parsing
   local result = xmlLint.lint("document.xml", {nonet = true})
   ```

3. **Structure Validation** (choose one):

   ```lua
   -- XSD validation
   local isValid, errors = xmlLint.validateXSD("doc.xml", "schema.xsd")

   -- DTD validation
   local isValid, errors = xmlLint.validateDTD("doc.xml", "doc.dtd")

   -- RelaxNG validation
   local isValid, errors = xmlLint.validateRelaxNG("doc.xml", "schema.rng")
   ```

4. **Business Rule Validation**:

   ```lua
   -- Schematron validation
   local isValid, errors = xmlLint.validateSchematron("doc.xml", "rules.sch")
   ```

### Choosing the Right Schema Language

**Use DTD when**:

- Legacy XML formats
- Simple structure validation
- Maximum tool compatibility needed

**Use XSD when**:

- Rich data typing required
- Complex type hierarchies
- SOAP/Web services
- Industry standards compliance

**Use RelaxNG when**:

- Schema maintainability is priority
- Excellent namespace support needed
- Simple, clear syntax preferred
- Alternative to XSD complexity

**Use Schematron when**:

- Business rules need validation
- Co-constraints between elements
- Custom error messages required
- Supplementing other schema languages

### Best Practices

1. **Layer Validation**: Use multiple schema languages for comprehensive validation
2. **Clear Error Messages**: Provide helpful validation feedback
3. **Phased Validation**: Validate in stages (syntax → structure → business rules)
4. **Namespace Awareness**: Properly declare and use namespaces
5. **Performance**: Consider validation performance for large documents
6. **Testing**: Include invalid test cases in test suites
7. **Documentation**: Document schema rules and constraints

## When to Use Each Library - Decision Guide

### xml-lint.lua - Use For

**Primary Use Cases:**

- XML validation and well-formedness checking
- Schema validation (XSD, DTD, RelaxNG, Schematron)
- XML cleaning and formatting
- XPath querying and data extraction
- Security-focused XML processing

**Specific Scenarios:**

```lua
-- 1. Input Validation (ALWAYS do this first)
function processUpload(file)
    -- Check if it's valid XML
    local isValid, errors = xmlLint.checkWellFormed(file)
    if not isValid then
        return nil, errors
    end

    -- Validate against business schema
    isValid, errors = xmlLint.validateXSD(file, "business-rules.xsd")
    if not isValid then
        return nil, errors
    end

    -- Safe to process
    return processFile(file)
end

-- 2. Data Cleaning Before Processing
function normalizeXML(inputFile, outputFile)
    xmlLint.cleanXML(inputFile, {
        remove_blanks = true,      -- Remove unnecessary whitespace
        drop_dtd = true,            -- Remove DTD declarations
        clean_namespaces = true,   -- Clean up redundant namespaces
        output_file = outputFile
    })
end

-- 3. Security Scanning
function scanForSecurityIssues(file)
    xmlLint.lint(file, {
        no_net = true,               -- Disable network access
        no_xinclude = true,          -- Disable XInclude
        substitute_entity = false,   -- Don't substitute entities
        max_amplification = 10       -- Limit expansion attacks
    })
end

-- 4. Debugging XML Issues
function debugXML(file)
    -- Get document structure
    local doc = xmlLint.parseXML(file)
    xmlLint.listNode(doc)

    -- Check encoding issues
    print("Encoding:", xmlLint.getXMLEncoding(file))
    print("Version:", xmlLint.getXMLVersion(file))

    -- Find specific elements
    xmlLint.lint(file, {
        xpath = "//error|//warning|//exception",
        no_out = true
    })
end

-- 5. Batch Processing with Different Validations
function validateBatch(files)
    local results = {}
    for _, file in ipairs(files) do
        local result = {
            file = file,
            wellFormed = xmlLint.checkWellFormed(file),
            validXSD = false,
            validDTD = false,
            encoding = xmlLint.getXMLEncoding(file)
        }

        -- Try XSD validation if XSD exists
        local xsdFile = file:gsub("%.xml$", ".xsd")
        if fs.fileExists(xsdFile) then
            result.validXSD = xmlLint.validateXSD(file, xsdFile)
        end

        table.insert(results, result)
    end
    return results
end
```

### xml-schema.lua - Use For

**Primary Use Cases:**

- Focused schema validation (XSD or DTD)
- cXML document validation
- Schema compliance checking
- Version-aware validation

**Specific Scenarios:**

```lua
-- 1. B2B Document Validation (cXML, EDI, etc.)
function validateCXML(cxmlFile)
    return xmlSchema.validateDTD(cxmlFile, "cxml_1.2.dtd", {
        replace_doctype = true,    -- Ensure correct DOCTYPE
        replace_version = true     -- Match DTD version
    })
end

-- 2. Compliance Checking
function checkCompliance(xmlFile, xsdSchema)
    local isValid, errors = xmlSchema.validateXSD(xmlFile, xsdSchema)

    if not isValid then
        -- Log compliance violations
        for _, error in ipairs(errors) do
            logViolation(error.type, error.message)
        end
    end

    return isValid
end

-- 3. Multi-Schema Validation
function validateWithSchemas(xmlFile, schemas)
    for _, schema in ipairs(schemas) do
        local isValid, errors = xmlSchema.validateXSD(xmlFile, schema)
        if not isValid then
            return false, string.format("Failed %s validation: %s",
                schema, errors[1].message)
        end
    end
    return true
end

-- 4. Version Migration Support
function migrateXML(xmlFile, fromDTD, toDTD)
    -- Validate against old version
    local oldValid = xmlSchema.validateDTD(xmlFile, fromDTD)
    if not oldValid then
        return false, "Document doesn't conform to old version"
    end

    -- Update for new version
    return xmlSchema.validateDTD(xmlFile, toDTD, {
        replace_version = true
    })
end
```

### xml-xslt.lua - Use For

**Primary Use Cases:**

- XML to HTML conversion
- Data transformation and restructuring
- Report generation
- Format conversion (XML ↔ HTML ↔ Text)
- Template-based document generation

**Specific Scenarios:**

```lua
-- 1. Report Generation Pipeline
function generateReports(xmlData, reportType)
    local xsltFile = "templates/" .. reportType .. ".xsl"
    local outputFile = "reports/" .. reportType .. "_" ..
                      os.date("%Y%m%d") .. ".html"

    return xmlXslt.convert(xmlData, xsltFile, {
        output_file = outputFile,
        format = "html"
    })
end

-- 2. Data Transformation
function transformForSystem(xmlData, targetSystem)
    local transforms = {
        system1 = "to-system1.xsl",
        system2 = "to-system2.xsl",
        system3 = "to-system3.xsl"
    }

    local xslt = transforms[targetSystem]
    if not xslt then
        return nil, "Unknown target system: " .. targetSystem
    end

    return xmlXslt.convert(xmlData, xslt, {
        format = "xml"
    })
end

-- 3. Multi-Format Output
function publishToAllFormats(xmlFile, baseName)
    local formats = {"html", "text", "pdf-ready-xml"}
    local xsltBase = "publish/"
    local results = {}

    for _, format in ipairs(formats) do
        local success, error = xmlXslt.convert(
            xmlFile,
            xsltBase .. "to-" .. format .. ".xsl",
            {
                output_file = "output/" .. baseName .. "." .. format,
                format = format
            }
        )
        results[format] = success
    end

    return results
end

-- 4. Template Processing with Parameters
function generatePersonalizedReport(xmlData, userId, template)
    return xmlXslt.convert(xmlData, template, {
        parameters = {
            ["user-id"] = userId,
            ["current-date"] = os.date("%Y-%m-%d"),
            ["company"] = "My Company"
        },
        format = "html"
    })
end

-- 5. Batch Document Processing
function processDocumentBatch(xmlFiles, stylesheet)
    -- Initialize once
    local results = {}

    for _, file in ipairs(xmlFiles) do
        local outputFile = file:gsub("%.xml$", ".html")

        local success = xmlXslt.convert(file, stylesheet, {
            output_file = outputFile,
            format = "html",
            cleanup_after_convert = false  -- More efficient for batch
        })

        table.insert(results, {
            input = file,
            output = outputFile,
            success = success
        })
    end

    -- Cleanup once after all transformations
    xmlXslt.cleanup()
    return results
end
```

### Combined Usage Patterns

**1. Complete Document Processing Pipeline:**

```lua
function processIncomingDocument(filePath)
    -- Step 1: Basic validation
    local isValid, errors = xmlLint.checkWellFormed(filePath)
    if not isValid then
        return nil, "Not well-formed: " .. errors[1].message
    end

    -- Step 2: Schema validation
    isValid, errors = xmlSchema.validateXSD(filePath, "schema.xsd")
    if not isValid then
        return nil, "Schema validation failed: " .. errors[1].message
    end

    -- Step 3: Security scan
    xmlLint.lint(filePath, {
        no_net = true,
        max_amplification = 100,
        no_out = true
    })

    -- Step 4: Clean and normalize
    local cleanFile = filePath:gsub("%.xml$", "_clean.xml")
    xmlLint.cleanXML(filePath, {
        output_file = cleanFile,
        remove_blanks = true
    })

    -- Step 5: Transform to target format
    local htmlFile = filePath:gsub("%.xml$", ".html")
    local success = xmlXslt.convert(cleanFile, "to-html.xsl", {
        output_file = htmlFile,
        format = "html"
    })

    -- Step 6: Final validation of output
    if success then
        return htmlFile
    else
        return nil, "Transformation failed"
    end
end
```

**2. Configuration-Driven XML Processing:**

```lua
function processWithConfig(xmlFile, config)
    config = config or {}

    -- Pre-processing
    if config.preprocess then
        xmlLint.lint(xmlFile, config.preprocess)
    end

    -- Validation
    if config.validation then
        for _, schema in ipairs(config.validation.schemas or {}) do
            local valid = xmlSchema.validateXSD(xmlFile, schema)
            if not valid and config.validation.strict then
                return nil, "Validation failed against: " .. schema
            end
        end
    end

    -- Transformation
    if config.transformation then
        local outputs = {}
        for _, transform in ipairs(config.transformation) do
            local output = xmlXslt.convert(xmlFile, transform.stylesheet, {
                format = transform.format or "xml",
                parameters = transform.parameters or {}
            })
            table.insert(outputs, {
                format = transform.format,
                content = output
            })
        end
        return outputs
    end

    return xmlFile  -- Return original if no transformation
end
```

### Performance Guidelines

1. **Use xml-lint for validation**: Fast and lightweight
2. **Batch XSLT transformations**: Reuse stylesheets, cleanup once
3. **Stream large files**: Use xml-lint with stream option
4. **Cache parsed schemas**: Avoid re-parsing XSD/DTD files
5. **Memory management**: Always cleanup after batch operations

### Security Guidelines

1. **Always validate untrusted XML**:

   ```lua
   function processUntrustedXML(xmlString)
       -- Temporarily write to file for validation
       local tempFile = "temp/untrusted.xml"
       fs.writeFile(tempFile, xmlString)

       -- Security-focused validation
       local isValid = xmlLint.checkWellFormed(tempFile) and
                      xmlLint.lint(tempFile, {
                          no_net = true,
                          no_xinclude = true,
                          max_amplification = 10
                      })

       fs.remove(tempFile)
       return isValid
   end
   ```

2. **Disable network access for external XML**:

   ```lua
   xmlLint.lint(externalFile, {
       no_net = true,
       substitute_entity = false
   })
   ```

3. **Limit resource usage**:

   ```lua
   xmlLint.lint(largeFile, {
       max_amplification = 1000,
       huge = false,  -- Set to true only if necessary
       stream = true
   })
   ```
