diff --git a/docs/DOCUMENTATION.md b/docs/DOCUMENTATION.md new file mode 100644 index 0000000..2db8c0c --- /dev/null +++ b/docs/DOCUMENTATION.md @@ -0,0 +1,783 @@ +# Ham Radio Award Portal - Documentation + +## Table of Contents + +1. [Architecture](#architecture) +2. [Components](#components) +3. [Code Structure](#code-structure) +4. [Awards System](#awards-system) + +--- + +## Architecture + +### Overview + +The Ham Radio Award Portal is a full-stack web application designed to help amateur radio operators track their award progress by syncing QSOs (contacts) from ARRL's Logbook of the World (LoTW). + +### Technology Stack + +**Backend:** +- **Runtime**: Bun - Fast JavaScript runtime +- **Framework**: Elysia.js - Lightweight, high-performance web framework +- **Database**: SQLite - Embedded database for data persistence +- **ORM**: Drizzle ORM - Type-safe database queries +- **Authentication**: JWT tokens via `@elysiajs/jwt` +- **Password Hashing**: bcrypt + +**Frontend:** +- **Framework**: SvelteKit - Modern reactive framework +- **Bundler**: Vite - Fast development server and build tool +- **Language**: JavaScript with TypeScript type definitions +- **State Management**: Svelte stores +- **Styling**: CSS modules + +### System Architecture + +``` +┌─────────────────┐ HTTP/REST ┌─────────────────┐ +│ │ ◄──────────────────► │ │ +│ SvelteKit │ │ ElysiaJS │ +│ Frontend │ │ Backend │ +│ │ │ Server │ +└─────────────────┘ └────────┬────────┘ + │ + ▼ + ┌─────────────────┐ + │ SQLite DB │ + │ (Drizzle ORM) │ + └─────────────────┘ + ▲ + │ + ┌────────┴────────┐ + │ ARRL LoTW │ + │ External API │ + └─────────────────┘ +``` + +### Key Design Decisions + +1. **SQLite over PostgreSQL/MySQL**: Simplified deployment, embedded database, excellent for single-user or small-scale deployments +2. **Bun over Node.js**: Faster startup, better performance, native TypeScript support +3. **ElysiaJS over Express**: Better TypeScript support, faster performance, modern API design +4. **SvelteKit over React**: Smaller bundle sizes, better performance, simpler reactivity model + +--- + +## Components + +### Backend Components + +#### 1. Server (`src/backend/index.js`) + +Main entry point that configures and starts the ElysiaJS server. + +**Key Features:** +- CORS configuration for cross-origin requests +- JWT authentication plugin +- Route handlers for API endpoints +- Error handling middleware + +**Routes:** +- `GET /api/health` - Health check endpoint +- `POST /api/auth/register` - User registration +- `POST /api/auth/login` - User login +- `GET /api/auth/me` - Get current user +- `PUT /api/auth/lotw-credentials` - Update LoTW credentials +- `POST /api/lotw/sync` - Sync QSOs from LoTW +- `GET /api/qsos` - Get QSOs with filtering +- `GET /api/qsos/stats` - Get QSO statistics + +#### 2. Database Schema (`src/backend/db/schema/index.js`) + +Defines the database structure using Drizzle ORM schema builder. + +**Tables:** + +- **users**: User accounts, authentication credentials, LoTW credentials +- **qsos**: Amateur radio contacts in ADIF format with LoTW confirmation data +- **awards**: Award definitions with JSON rule configurations +- **award_progress**: Cached award progress for each user + +#### 3. Services + +**Auth Service** (`src/backend/services/auth.service.js`) +- User registration and login +- Password hashing with bcrypt +- JWT token generation and validation +- User profile management + +**LoTW Service** (`src/backend/services/lotw.service.js`) +- Synchronization with ARRL's Logbook of the World +- ADIF format parsing +- Long-polling for report generation +- QSO deduplication +- Band/mode normalization +- Error handling and retry logic + +#### 4. Configuration (`src/backend/config/`) + +- **database.js**: Database connection and client initialization +- **constants.js**: Application constants (JWT expiration, etc.) + +### Frontend Components + +#### 1. Pages (SvelteKit Routes) + +- **`/`**: Dashboard with welcome message +- **`/auth/login`**: User login form +- **`/auth/register`**: User registration form +- **`/qsos`**: QSO logbook with filtering and LoTW sync +- **`/settings`**: LoTW credentials management + +#### 2. Libraries + +**API Client** (`src/frontend/src/lib/api.js`) +- Centralized API communication +- Automatic JWT token injection +- Response/error handling +- Typed API methods + +**Stores** (`src/frontend/src/lib/stores.js`) +- Authentication state management +- Persistent login with localStorage +- Reactive user data + +**Types** (`src/frontend/src/lib/types/`) +- TypeScript type definitions +- Award system types +- API response types + +--- + +## Code Structure + +### Directory Layout + +``` +award/ +├── award-definitions/ # Award rule definitions (JSON) +│ ├── dxcc.json # DXCC award configurations +│ ├── dxcc-cw.json # DXCC CW-specific award +│ ├── was.json # WAS award configurations +│ ├── vucc-sat.json # VUCC Satellite award +│ └── sat-rs44.json # Satellite RS-44 award +│ +├── drizzle/ # Database migrations +│ └── 0000_init.sql # Initial schema +│ +├── docs/ # Documentation +│ └── DOCUMENTATION.md # This file +│ +├── src/ +│ ├── backend/ # Backend server code +│ │ ├── config/ +│ │ │ ├── constants.js +│ │ │ └── database.js +│ │ ├── db/ +│ │ │ └── schema/ +│ │ │ └── index.js # Drizzle schema definitions +│ │ ├── services/ +│ │ │ ├── auth.service.js +│ │ │ └── lotw.service.js +│ │ └── index.js # Main server entry point +│ │ +│ ├── frontend/ # Frontend application +│ │ ├── src/ +│ │ │ ├── lib/ +│ │ │ │ ├── api.js # API client +│ │ │ │ ├── stores.js # Svelte stores +│ │ │ │ └── types/ # TypeScript definitions +│ │ │ └── routes/ # SvelteKit pages +│ │ │ ├── +page.svelte +│ │ │ ├── auth/ +│ │ │ ├── qsos/ +│ │ │ └── settings/ +│ │ ├── static/ # Static assets +│ │ └── vite.config.js +│ │ +│ └── shared/ # Shared TypeScript types +│ └── index.ts +│ +├── award.db # SQLite database file +├── drizzle.config.ts # Drizzle configuration +├── package.json # Dependencies and scripts +├── CLAUDE.md # Claude AI instructions +└── README.md # Project overview +``` + +### File Organization Principles + +1. **Separation of Concerns**: Clear separation between backend and frontend +2. **Shared Types**: Common types in `src/shared/` to ensure consistency +3. **Service Layer**: Business logic isolated in service files +4. **Configuration**: Config files co-located in dedicated directories +5. **Database-First**: Schema drives the application structure + +### Database Schema Details + +#### Users Table +```javascript +{ + id: integer (primary key, auto-increment) + email: text (unique, not null) + passwordHash: text (not null) + callsign: text (not null) + lotwUsername: text (nullable) + lotwPassword: text (nullable, encrypted) + createdAt: timestamp + updatedAt: timestamp +} +``` + +#### QSOs Table +```javascript +{ + id: integer (primary key, auto-increment) + userId: integer (foreign key → users.id) + callsign: text + qsoDate: text (ADIF format: YYYYMMDD) + timeOn: text (HHMMSS) + band: text (160m, 80m, 40m, 30m, 20m, 17m, 15m, 12m, 10m, 6m, 2m, 70cm) + mode: text (CW, SSB, FM, AM, FT8, FT4, PSK31, etc.) + freq: integer (Hz) + freqRx: integer (Hz, satellite only) + entity: text (DXCC entity name) + entityId: integer (DXCC entity number) + grid: text (Maidenhead grid square) + gridSource: text (LOTW, USER, CALC) + continent: text (NA, SA, EU, AF, AS, OC, AN) + cqZone: integer + ituZone: integer + state: text (US state, CA province, etc.) + county: text + satName: text (satellite name) + satMode: text (satellite mode) + lotwQslRdate: text (LoTW confirmation date) + lotwQslRstatus: text ('Y', 'N', '?') + lotwSyncedAt: timestamp + createdAt: timestamp +} +``` + +#### Awards Table +```javascript +{ + id: text (primary key) + name: text + description: text + definition: text (JSON) + isActive: boolean + createdAt: timestamp +} +``` + +#### Award Progress Table +```javascript +{ + id: integer (primary key, auto-increment) + userId: integer (foreign key → users.id) + awardId: text (foreign key → awards.id) + workedCount: integer + confirmedCount: integer + totalRequired: integer + workedEntities: text (JSON array) + confirmedEntities: text (JSON array) + lastCalculatedAt: timestamp + lastQsoSyncAt: timestamp + updatedAt: timestamp +} +``` + +--- + +## Awards System + +### Overview + +The awards system is designed to be flexible and extensible. Awards are defined as JSON configuration files that specify rules for calculating progress based on QSOs. + +### Award Definition Structure + +Each award is defined with the following structure: + +```json +{ + "id": "unique-identifier", + "name": "Award Display Name", + "description": "Award description", + "category": "award-category", + "rules": { + "type": "entity", + "entityType": "dxcc|state|grid|...", + "target": 100, + "filters": { + "operator": "AND|OR", + "filters": [...] + } + } +} +``` + +### Entity Types + +| Entity Type | Description | Example Field | +|-------------|-------------|---------------| +| `dxcc` | DXCC entities (countries) | `entityId` | +| `state` | US States, Canadian provinces | `state` | +| `grid` | Maidenhead grid squares | `grid` | + +### Filter Operators + +| Operator | Description | +|----------|-------------| +| `eq` | Equal to | +| `ne` | Not equal to | +| `in` | In list | +| `nin` | Not in list | +| `contains` | Contains substring | +| `AND` | All filters must match | +| `OR` | At least one filter must match | + +### Award Categories + +- **`dxcc`**: DXCC (Countries/Districts) awards +- **`was`**: Worked All States awards +- **`vucc`**: VHF/UHF Century Club awards +- **`satellite`**: Satellite-specific awards + +--- + +## Award Examples + +### 1. DXCC Mixed Mode + +**File**: `award-definitions/dxcc.json` + +```json +{ + "id": "dxcc-mixed", + "name": "DXCC Mixed Mode", + "description": "Confirm 100 DXCC entities on any band/mode", + "category": "dxcc", + "rules": { + "type": "entity", + "entityType": "dxcc", + "target": 100 + } +} +``` + +**Explanation:** +- Counts unique DXCC entities (`entityId` field) +- No filtering - all QSOs count toward progress +- Target: 100 confirmed entities +- Both worked (any QSO) and confirmed (LoTW QSL) are tracked + +**Progress Calculation:** +```javascript +worked = COUNT(DISTINCT entityId WHERE lotwQslRstatus IS NOT NULL) +confirmed = COUNT(DISTINCT entityId WHERE lotwQslRstatus = 'Y') +``` + +--- + +### 2. DXCC CW (CW-specific) + +**File**: `award-definitions/dxcc-cw.json` + +```json +{ + "id": "dxcc-cw", + "name": "DXCC CW", + "description": "Confirm 100 DXCC entities using CW mode only", + "category": "dxcc", + "rules": { + "type": "entity", + "entityType": "dxcc", + "target": 100, + "filters": { + "operator": "AND", + "filters": [ + { + "field": "mode", + "operator": "eq", + "value": "CW" + } + ] + } + } +} +``` + +**Explanation:** +- Counts unique DXCC entities +- Only QSOs with `mode = 'CW'` are counted +- Target: 100 confirmed entities +- Example: A CW QSO with Germany (entityId=230) counts, but SSB does not + +**Progress Calculation:** +```javascript +worked = COUNT(DISTINCT entityId + WHERE mode = 'CW' AND lotwQslRstatus IS NOT NULL) +confirmed = COUNT(DISTINCT entityId + WHERE mode = 'CW' AND lotwQslRstatus = 'Y') +``` + +--- + +### 3. WAS Mixed Mode + +**File**: `award-definitions/was.json` + +```json +{ + "id": "was-mixed", + "name": "WAS Mixed Mode", + "description": "Confirm all 50 US states", + "category": "was", + "rules": { + "type": "entity", + "entityType": "state", + "target": 50, + "filters": { + "operator": "AND", + "filters": [ + { + "field": "entity", + "operator": "eq", + "value": "United States" + } + ] + } + } +} +``` + +**Explanation:** +- Counts unique US states +- Only QSOs where `entity = 'United States'` +- Target: 50 confirmed states +- Both worked and confirmed are tracked separately + +**Progress Calculation:** +```javascript +worked = COUNT(DISTINCT state + WHERE entity = 'United States' AND lotwQslRstatus IS NOT NULL) +confirmed = COUNT(DISTINCT state + WHERE entity = 'United States' AND lotwQslRstatus = 'Y') +``` + +**Example States:** +- `"CA"` - California +- `"TX"` - Texas +- `"NY"` - New York +- (All 50 US states) + +--- + +### 4. VUCC Satellite + +**File**: `award-definitions/vucc-sat.json` + +```json +{ + "id": "vucc-satellite", + "name": "VUCC Satellite", + "description": "Confirm 100 unique grid squares via satellite", + "category": "vucc", + "rules": { + "type": "entity", + "entityType": "grid", + "target": 100, + "filters": { + "operator": "AND", + "filters": [ + { + "field": "satellite", + "operator": "eq", + "value": true + } + ] + } + } +} +``` + +**Explanation:** +- Counts unique grid squares (4-character: FN31, CM97, etc.) +- Only satellite QSOs (where `satName` is not null) +- Target: 100 confirmed grids +- Grid squares are counted independently of band/mode + +**Progress Calculation:** +```javascript +worked = COUNT(DISTINCT SUBSTRING(grid, 1, 4) + WHERE satName IS NOT NULL AND lotwQslRstatus IS NOT NULL) +confirmed = COUNT(DISTINCT SUBSTRING(grid, 1, 4) + WHERE satName IS NOT NULL AND lotwQslRstatus = 'Y') +``` + +**Example Grids:** +- `"FN31pr"` → counts as `"FN31"` +- `"CM97"` → counts as `"CM97"` + +--- + +### 5. Satellite RS-44 (Specific Satellite) + +**File**: `award-definitions/sat-rs44.json` + +```json +{ + "id": "sat-rs44", + "name": "RS-44 Satellite Award", + "description": "Make 100 QSOs via RS-44 satellite", + "category": "satellite", + "rules": { + "type": "entity", + "entityType": "callsign", + "target": 100, + "filters": { + "operator": "AND", + "filters": [ + { + "field": "satName", + "operator": "eq", + "value": "RS-44" + } + ] + } + } +} +``` + +**Explanation:** +- Counts unique callsigns worked via RS-44 satellite +- Only QSOs where `satName = 'RS-44'` +- Target: 100 unique callsigns +- Each station (callsign) counts once regardless of multiple QSOs + +**Progress Calculation:** +```javascript +worked = COUNT(DISTINCT callsign + WHERE satName = 'RS-44' AND lotwQslRstatus IS NOT NULL) +confirmed = COUNT(DISTINCT callsign + WHERE satName = 'RS-44' AND lotwQslRstatus = 'Y') +``` + +--- + +### Advanced Filter Examples + +#### Multiple Conditions (AND) + +Count only 20m CW QSOs: + +```json +{ + "filters": { + "operator": "AND", + "filters": [ + { "field": "band", "operator": "eq", "value": "20m" }, + { "field": "mode", "operator": "eq", "value": "CW" } + ] + } +} +``` + +#### Multiple Options (OR) + +Count any phone mode QSOs: + +```json +{ + "filters": { + "operator": "OR", + "filters": [ + { "field": "mode", "operator": "eq", "value": "SSB" }, + { "field": "mode", "operator": "eq", "value": "FM" }, + { "field": "mode", "operator": "eq", "value": "AM" } + ] + } +} +``` + +#### Band Ranges + +Count HF QSOs (all HF bands): + +```json +{ + "filters": { + "operator": "OR", + "filters": [ + { "field": "band", "operator": "eq", "value": "160m" }, + { "field": "band", "operator": "eq", "value": "80m" }, + { "field": "band", "operator": "eq", "value": "60m" }, + { "field": "band", "operator": "eq", "value": "40m" }, + { "field": "band", "operator": "eq", "value": "30m" }, + { "field": "band", "operator": "eq", "value": "20m" }, + { "field": "band", "operator": "eq", "value": "17m" }, + { "field": "band", "operator": "eq", "value": "15m" }, + { "field": "band", "operator": "eq", "value": "12m" }, + { "field": "band", "operator": "eq", "value": "10m" } + ] + } +} +``` + +#### Grid Square by Continent + +Count VUCC grids in Europe only: + +```json +{ + "filters": { + "operator": "AND", + "filters": [ + { "field": "continent", "operator": "eq", "value": "EU" }, + { + "filters": [ + { "field": "band", "operator": "eq", "value": "2m" }, + { "field": "band", "operator": "eq", "value": "70cm" } + ], + "operator": "OR" + } + ] + } +} +``` + +--- + +## Creating Custom Awards + +To create a new award: + +1. **Create a new JSON file** in `award-definitions/` +2. **Define the award structure** following the schema above +3. **Add to database** (via API or database migration) + +### Example: IOTA Award + +```json +{ + "id": "iota-mixed", + "name": "IOTA Mixed Mode", + "description": "Confirm 100 Islands on the Air", + "category": "iota", + "rules": { + "type": "entity", + "entityType": "iota", + "target": 100, + "filters": { + "operator": "AND", + "filters": [ + { "field": "iotaNumber", "operator": "ne", "value": null } + ] + } + } +} +``` + +**Note**: This would require adding an `iotaNumber` field to the QSO table. + +--- + +## Award Progress API + +### Get User Award Progress + +``` +GET /api/awards/progress +``` + +**Response:** +```json +{ + "userId": 1, + "awards": [ + { + "id": "dxcc-mixed", + "name": "DXCC Mixed Mode", + "worked": 87, + "confirmed": 73, + "totalRequired": 100, + "percentage": 73 + } + ] +} +``` + +### Get Detailed Award Breakdown + +``` +GET /api/awards/progress/:awardId +``` + +**Response:** +```json +{ + "awardId": "dxcc-mixed", + "workedEntities": [ + { "entity": "Germany", "entityId": 230, "worked": true, "confirmed": true }, + { "entity": "Japan", "entityId": 339, "worked": true, "confirmed": false } + ], + "confirmedEntities": [ + { "entity": "Germany", "entityId": 230 } + ] +} +``` + +--- + +## Future Enhancements + +The award system is designed for extensibility. Planned enhancements: + +1. **Additional Entity Types**: + - IOTA (Islands on the Air) + - CQ Zones + - ITU Zones + - Counties + +2. **Advanced Filters**: + - Date ranges + - Regular expressions + - Custom field queries + +3. **Award Endorsements**: + - Band-specific endorsements + - Mode-specific endorsements + - Combined endorsements + +4. **Award Tiers**: + - Bronze, Silver, Gold levels + - Progressive achievements + +5. **User-Defined Awards**: + - Allow users to create custom awards + - Share award definitions with community + +--- + +## Contributing + +When adding new awards or modifying the award system: + +1. Follow the JSON schema for award definitions +2. Add tests for new filter types +3. Update this documentation with examples +4. Ensure database schema supports required fields +5. Test progress calculations thoroughly + +--- + +## Resources + +- [ARRL LoTW](https://lotw.arrl.org/) +- [ADIF Specification](https://adif.org/) +- [DXCC List](https://www.arrl.org/dxcc) +- [VUCC Program](https://www.arrl.org/vucc) +- [WAS Award](https://www.arrl.org/was) diff --git a/src/backend/index.js b/src/backend/index.js index 245d699..57a960f 100644 --- a/src/backend/index.js +++ b/src/backend/index.js @@ -8,6 +8,11 @@ import { getUserById, updateLoTWCredentials, } from './services/auth.service.js'; +import { + syncQSOs, + getUserQSOs, + getQSOStats, +} from './services/lotw.service.js'; /** * Main backend application @@ -179,7 +184,17 @@ const app = new Elysia() } try { - await updateLoTWCredentials(user.id, body.lotwUsername, body.lotwPassword); + // Get current user data to preserve password if not provided + const userData = await getUserById(user.id); + if (!userData) { + set.status = 404; + return { success: false, error: 'User not found' }; + } + + // If password is empty, keep existing password + const lotwPassword = body.lotwPassword || userData.lotwPassword; + + await updateLoTWCredentials(user.id, body.lotwUsername, lotwPassword); return { success: true, @@ -196,11 +211,106 @@ const app = new Elysia() { body: t.Object({ lotwUsername: t.String(), - lotwPassword: t.String(), + lotwPassword: t.Optional(t.String()), }), } ) + /** + * POST /api/lotw/sync + * Sync QSOs from LoTW (requires authentication) + */ + .post('/api/lotw/sync', async ({ user, set }) => { + if (!user) { + set.status = 401; + return { success: false, error: 'Unauthorized' }; + } + + try { + // Get user's LoTW credentials from database + const userData = await getUserById(user.id); + if (!userData || !userData.lotwUsername || !userData.lotwPassword) { + set.status = 400; + return { + success: false, + error: 'LoTW credentials not configured. Please add them in Settings.', + }; + } + + // Decrypt password (for now, assuming it's stored as-is. TODO: implement encryption) + const lotwPassword = userData.lotwPassword; + + // Sync QSOs from LoTW + const result = await syncQSOs(user.id, userData.lotwUsername, lotwPassword); + + return result; + } catch (error) { + set.status = 500; + return { + success: false, + error: `LoTW sync failed: ${error.message}`, + }; + } + }) + + /** + * GET /api/qsos + * Get user's QSOs (requires authentication) + */ + .get('/api/qsos', async ({ user, query, set }) => { + if (!user) { + set.status = 401; + return { success: false, error: 'Unauthorized' }; + } + + try { + const filters = {}; + if (query.band) filters.band = query.band; + if (query.mode) filters.mode = query.mode; + if (query.confirmed) filters.confirmed = query.confirmed === 'true'; + + const qsos = await getUserQSOs(user.id, filters); + + return { + success: true, + qsos, + count: qsos.length, + }; + } catch (error) { + set.status = 500; + return { + success: false, + error: 'Failed to fetch QSOs', + }; + } + }) + + /** + * GET /api/qsos/stats + * Get QSO statistics (requires authentication) + */ + .get('/api/qsos/stats', async ({ user, set }) => { + if (!user) { + set.status = 401; + return { success: false, error: 'Unauthorized' }; + } + + try { + const stats = await getQSOStats(user.id); + + return { + success: true, + stats, + }; + } catch (error) { + set.status = 500; + return { + success: false, + error: 'Failed to fetch statistics', + }; + } + }) + // Health check endpoint .get('/api/health', () => ({ status: 'ok', diff --git a/src/backend/services/lotw.service.js b/src/backend/services/lotw.service.js new file mode 100644 index 0000000..27b0e93 --- /dev/null +++ b/src/backend/services/lotw.service.js @@ -0,0 +1,531 @@ +import { db } from '../config/database.js'; +import { qsos } from '../db/schema/index.js'; + +/** + * LoTW (Logbook of the World) Service + * Fetches QSOs from ARRL's LoTW system + */ + +// Configuration for long-polling +const POLLING_CONFIG = { + maxRetries: 30, // Maximum number of retry attempts + retryDelay: 10000, // Delay between retries in ms (10 seconds) + requestTimeout: 60000, // Timeout for individual requests in ms (1 minute) + maxTotalTime: 600000, // Maximum total time to wait in ms (10 minutes) +}; + +/** + * Check if LoTW response indicates the report is still being prepared + * @param {string} responseData - The response text from LoTW + * @returns {boolean} True if report is still pending + */ +function isReportPending(responseData) { + const trimmed = responseData.trim().toLowerCase(); + + // LoTW returns various messages when report is not ready: + // - Empty responses + // - "Report is being prepared" or similar messages + // - HTML error pages + // - Very short responses that aren't valid ADIF + + // Check for empty or very short responses + if (trimmed.length < 100) { + return true; + } + + // Check for HTML responses (error pages) + if (trimmed.includes('') || trimmed.includes('')) { + return true; + } + + // Check for common "not ready" messages + const pendingMessages = [ + 'report is being prepared', + 'your report is being generated', + 'please try again', + 'report queue', + 'not yet available', + 'temporarily unavailable', + ]; + + for (const msg of pendingMessages) { + if (trimmed.includes(msg)) { + return true; + } + } + + // Check if it looks like valid ADIF data (should start with } + */ +function sleep(ms) { + return new Promise(resolve => setTimeout(resolve, ms)); +} + +/** + * Fetch QSOs from LoTW with long-polling support + * @param {string} lotwUsername - LoTW username + * @param {string} lotwPassword - LoTW password + * @param {Date} sinceDate - Only fetch QSOs since this date + * @returns {Promise} Array of QSO objects + */ +export async function fetchQSOsFromLoTW(lotwUsername, lotwPassword, sinceDate = null) { + // LoTW report URL + const url = 'https://lotw.arrl.org/lotwuser/lotwreport.adi'; + + // Build query parameters - qso_query=1 is REQUIRED to get QSO records! + const params = new URLSearchParams({ + login: lotwUsername, + password: lotwPassword, + qso_query: '1', // REQUIRED: Without this, no QSO records are returned + qso_qsl: 'yes', // Only get QSOs with QSLs (confirmed) + qso_qsldetail: 'yes', // Include QSL details (station location) + qso_mydetail: 'yes', // Include my station details + qso_withown: 'yes', // Include own callsign + }); + + // Add date filter - ALWAYS send qso_qslsince to get all QSOs + // LoTW default behavior only returns QSOs since last download, so we must + // explicitly send a date to get QSOs since that date + const dateStr = sinceDate + ? sinceDate.toISOString().split('T')[0] + : '2026-01-01'; // Default: Get QSOs since 2026-01-01 + params.append('qso_qslsince', dateStr); + + console.error('Date filter:', dateStr, sinceDate ? '(User-specified date)' : '(Default: since 2026-01-01)'); + + const fullUrl = `${url}?${params.toString()}`; + + console.error('Fetching from LoTW:', fullUrl.replace(/password=[^&]+/, 'password=***')); + + const startTime = Date.now(); + + // Long-polling loop + for (let attempt = 0; attempt < POLLING_CONFIG.maxRetries; attempt++) { + try { + // Check if we've exceeded max total time + const elapsed = Date.now() - startTime; + if (elapsed > POLLING_CONFIG.maxTotalTime) { + throw new Error(`LoTW sync timeout: exceeded maximum wait time of ${POLLING_CONFIG.maxTotalTime / 1000} seconds`); + } + + if (attempt > 0) { + console.error(`Retry attempt ${attempt + 1}/${POLLING_CONFIG.maxRetries} (elapsed: ${Math.round(elapsed / 1000)}s)`); + } + + // Make request with timeout + const controller = new AbortController(); + const timeoutId = setTimeout(() => controller.abort(), POLLING_CONFIG.requestTimeout); + + const response = await fetch(fullUrl, { signal: controller.signal }); + + clearTimeout(timeoutId); + + // Handle HTTP errors + if (!response.ok) { + if (response.status === 503) { + // Service unavailable - might be temporary, retry + console.error('LoTW returned 503 (Service Unavailable), waiting before retry...'); + await sleep(POLLING_CONFIG.retryDelay); + continue; + } else if (response.status === 401) { + throw new Error('Invalid LoTW credentials. Please check your username and password in Settings.'); + } else if (response.status === 404) { + throw new Error('LoTW service not found (404). The LoTW API URL may have changed.'); + } else { + // Other errors - log but retry + console.error(`LoTW returned ${response.status} ${response.statusText}, waiting before retry...`); + await sleep(POLLING_CONFIG.retryDelay); + continue; + } + } + + // Get response text + const adifData = await response.text(); + console.error(`Response length: ${adifData.length} bytes`); + + // Check if report is still pending + if (isReportPending(adifData)) { + console.error('LoTW report is still being prepared, waiting...', adifData.substring(0, 100)); + + // Wait before retrying + await sleep(POLLING_CONFIG.retryDelay); + continue; + } + + // We have valid data! + console.error('LoTW report ready, parsing ADIF data...'); + console.error('ADIF preview:', adifData.substring(0, 200)); + + // Parse ADIF format + const qsos = parseADIF(adifData); + console.error(`Successfully parsed ${qsos.length} QSOs from LoTW`); + + return qsos; + + } catch (error) { + const elapsed = Date.now() - startTime; + + if (error.name === 'AbortError') { + console.error(`Request timeout on attempt ${attempt + 1}, retrying...`); + await sleep(POLLING_CONFIG.retryDelay); + continue; + } + + // Re-throw credential/auth errors immediately + if (error.message.includes('credentials') || error.message.includes('401') || error.message.includes('404')) { + throw error; + } + + // For other errors, log and retry if we haven't exhausted retries + if (attempt < POLLING_CONFIG.maxRetries - 1) { + console.error(`Error on attempt ${attempt + 1}: ${error.message}`); + console.error('Retrying...'); + await sleep(POLLING_CONFIG.retryDelay); + continue; + } else { + throw error; + } + } + } + + // If we get here, we exhausted all retries + const totalTime = Math.round((Date.now() - startTime) / 1000); + throw new Error(`LoTW sync failed: Report not ready after ${POLLING_CONFIG.maxRetries} attempts (${totalTime}s). LoTW may be experiencing high load. Please try again later.`); +} + +/** + * Parse ADIF (Amateur Data Interchange Format) data + * @param {string} adifData - Raw ADIF text data + * @returns {Array} Array of QSOs (each QSO is an array of field objects) + */ +function parseADIF(adifData) { + const qsos = []; + const records = adifData.split(''); + + console.error(`Total records after splitting by : ${records.length}`); + + for (let i = 0; i < records.length; i++) { + const record = records[i]; + + // Skip empty records or records that are just header info + if (!record.trim()) continue; + if (record.trim().startsWith('<') && !record.includes('value + // Important: The 'type' part is optional, and field names can contain underscores + // We use the length parameter to extract exactly that many characters + const regex = /<([A-Z_]+):(\d+)(?::[A-Z]+)?>([\s\S])/gi; + let match; + + while ((match = regex.exec(record)) !== null) { + const [fullMatch, fieldName, lengthStr, firstChar] = match; + const length = parseInt(lengthStr, 10); + + // Extract exactly 'length' characters starting from the position after the '>' + const valueStart = match.index + fullMatch.length - 1; // -1 because firstChar is already captured + const value = record.substring(valueStart, valueStart + length); + + qso[fieldName.toLowerCase()] = value.trim(); + + // Move regex lastIndex to the end of the value so we can find the next tag + regex.lastIndex = valueStart + length; + } + + // Only add if we have actual QSO data (has CALL or call field) + if (Object.keys(qso).length > 0 && (qso.call || qso.call)) { + qsos.push(qso); + + // Log first few QSOs for debugging + if (qsos.length <= 3) { + console.error(`Parsed QSO #${qsos.length}: ${qso.call} on ${qso.qso_date} ${qso.band} ${qso.mode}`); + } + } + } + + console.error(`Total QSOs parsed: ${qsos.length}`); + return qsos; +} + +/** + * Convert ADIF QSO to database format + * @param {Object} adifQSO - QSO object from ADIF parser + * @param {number} userId - User ID + * @returns {Object} Database-ready QSO object + */ +export function convertQSODatabaseFormat(adifQSO, userId) { + return { + userId, + callsign: adifQSO.call || '', + qsoDate: adifQSO.qso_date || '', + timeOn: adifQSO.time_on || adifQSO.time_off || '000000', + band: normalizeBand(adifQSO.band), + mode: normalizeMode(adifQSO.mode), + freq: adifQSO.freq ? parseInt(adifQSO.freq) : null, + freqRx: adifQSO.freq_rx ? parseInt(adifQSO.freq_rx) : null, + entity: adifQSO.country || adifQSO.dxcc_country || '', + entityId: adifQSO.dxcc || null, + grid: adifQSO.gridsquare || '', + continent: adifQSO.continent || '', + cqZone: adifQSO.cq_zone ? parseInt(adifQSO.cq_zone) : null, + ituZone: adifQSO.itu_zone ? parseInt(adifQSO.itu_zone) : null, + state: adifQSO.state || adifQSO.us_state || '', + satName: adifQSO.sat_name || '', + satMode: adifQSO.sat_mode || '', + lotwQslRdate: adifQSO.qslrdate || '', + lotwQslRstatus: adifQSO.qsl_rcvd || 'N', + lotwSyncedAt: new Date(), + }; +} + +/** + * Normalize band name + * @param {string} band - Band from ADIF + * @returns {string|null} Normalized band + */ +function normalizeBand(band) { + if (!band) return null; + + const bandMap = { + '160m': '160m', + '80m': '80m', + '60m': '60m', + '40m': '40m', + '30m': '30m', + '20m': '20m', + '17m': '17m', + '15m': '15m', + '12m': '12m', + '10m': '10m', + '6m': '6m', + '4m': '4m', + '2m': '2m', + '1.25m': '1.25m', + '70cm': '70cm', + '33cm': '33cm', + '23cm': '23cm', + '13cm': '13cm', + '9cm': '9cm', + '6cm': '6cm', + '3cm': '3cm', + '1.2cm': '1.2cm', + 'mm': 'mm', + }; + + return bandMap[band.toLowerCase()] || band; +} + +/** + * Normalize mode name + * @param {string} mode - Mode from ADIF + * @returns {string} Normalized mode + */ +function normalizeMode(mode) { + if (!mode) return ''; + + const modeMap = { + 'cw': 'CW', + 'ssb': 'SSB', + 'am': 'AM', + 'fm': 'FM', + 'rtty': 'RTTY', + 'psk31': 'PSK31', + 'psk63': 'PSK63', + 'ft8': 'FT8', + 'ft4': 'FT4', + 'jt65': 'JT65', + 'jt9': 'JT9', + 'js8': 'JS8', + 'mfsk': 'MFSK', + ' Olivia': 'OLIVIA', + }; + + const normalized = modeMap[mode.toLowerCase()]; + return normalized || mode.toUpperCase(); +} + +/** + * Sync QSOs from LoTW to database + * @param {number} userId - User ID + * @param {string} lotwUsername - LoTW username + * @param {string} lotwPassword - LoTW password + * @param {Date} sinceDate - Only sync QSOs since this date + * @returns {Promise} Sync result with counts + */ +export async function syncQSOs(userId, lotwUsername, lotwPassword, sinceDate = null) { + try { + // Fetch QSOs from LoTW + const adifQSOs = await fetchQSOsFromLoTW(lotwUsername, lotwPassword, sinceDate); + + if (!adifQSOs || adifQSOs.length === 0) { + return { + success: true, + total: 0, + added: 0, + updated: 0, + message: 'No QSOs found in LoTW', + }; + } + + let addedCount = 0; + let updatedCount = 0; + const errors = []; + + // Process each QSO + for (const qsoData of adifQSOs) { + try { + console.error('Raw ADIF QSO data:', JSON.stringify(qsoData)); + const dbQSO = convertQSODatabaseFormat(qsoData, userId); + console.error('Converted QSO:', JSON.stringify(dbQSO)); + console.error('Processing QSO:', dbQSO.callsign, dbQSO.qsoDate, dbQSO.band, dbQSO.mode); + + // Check if QSO already exists (by callsign, date, time, band, mode) + const { eq } = await import('drizzle-orm'); + const existing = await db + .select() + .from(qsos) + .where(eq(qsos.userId, userId)) + .where(eq(qsos.callsign, dbQSO.callsign)) + .where(eq(qsos.qsoDate, dbQSO.qsoDate)) + .where(eq(qsos.band, dbQSO.band)) + .where(eq(qsos.mode, dbQSO.mode)) + .limit(1); + + console.error('Existing QSOs found:', existing.length); + if (existing.length > 0) { + console.error('Existing QSO:', JSON.stringify(existing[0])); + } + + if (existing.length > 0) { + // Update existing QSO + console.error('Updating existing QSO'); + await db + .update(qsos) + .set({ + lotwQslRdate: dbQSO.lotwQslRdate, + lotwQslRstatus: dbQSO.lotwQslRstatus, + lotwSyncedAt: dbQSO.lotwSyncedAt, + }) + .where(eq(qsos.id, existing[0].id)); + updatedCount++; + } else { + // Insert new QSO + console.error('Inserting new QSO with data:', JSON.stringify(dbQSO)); + try { + const result = await db.insert(qsos).values(dbQSO); + console.error('Insert result:', result); + addedCount++; + } catch (insertError) { + console.error('Insert failed:', insertError.message); + console.error('Insert error details:', insertError); + throw insertError; + } + } + } catch (error) { + console.error('ERROR processing QSO:', error); + errors.push({ + qso: qsoData, + error: error.message, + }); + } + } + + return { + success: true, + total: adifQSOs.length, + added: addedCount, + updated: updatedCount, + errors: errors.length > 0 ? errors : undefined, + }; + } catch (error) { + throw new Error(`LoTW sync failed: ${error.message}`); + } +} + +/** + * Get QSOs for a user + * @param {number} userId - User ID + * @param {Object} filters - Query filters + * @returns {Promise} Array of QSOs + */ +export async function getUserQSOs(userId, filters = {}) { + const { eq, and } = await import('drizzle-orm'); + + console.error('getUserQSOs called with userId:', userId, 'filters:', filters); + + // Build where conditions + const conditions = [eq(qsos.userId, userId)]; + + if (filters.band) { + conditions.push(eq(qsos.band, filters.band)); + } + + if (filters.mode) { + conditions.push(eq(qsos.mode, filters.mode)); + } + + if (filters.confirmed) { + conditions.push(eq(qsos.lotwQslRstatus, 'Y')); + } + + // Use and() to combine all conditions + const results = await db.select().from(qsos).where(and(...conditions)); + + console.error('getUserQSOs returning', results.length, 'QSOs'); + + // Order by date descending, then time + return results.sort((a, b) => { + const dateCompare = b.qsoDate.localeCompare(a.qsoDate); + if (dateCompare !== 0) return dateCompare; + return b.timeOn.localeCompare(a.timeOn); + }); +} + +/** + * Get QSO statistics for a user + * @param {number} userId - User ID + * @returns {Promise} Statistics object + */ +export async function getQSOStats(userId) { + const { eq } = await import('drizzle-orm'); + const allQSOs = await db.select().from(qsos).where(eq(qsos.userId, userId)); + + console.error('getQSOStats called with userId:', userId, 'found', allQSOs.length, 'QSOs in database'); + + const confirmed = allQSOs.filter((q) => q.lotwQslRstatus === 'Y'); + + // Count unique entities + const uniqueEntities = new Set(); + const uniqueBands = new Set(); + const uniqueModes = new Set(); + + allQSOs.forEach((q) => { + if (q.entity) uniqueEntities.add(q.entity); + if (q.band) uniqueBands.add(q.band); + if (q.mode) uniqueModes.add(q.mode); + }); + + const stats = { + total: allQSOs.length, + confirmed: confirmed.length, + uniqueEntities: uniqueEntities.size, + uniqueBands: uniqueBands.size, + uniqueModes: uniqueModes.size, + }; + + console.error('getQSOStats returning:', stats); + + return stats; +} diff --git a/src/frontend/src/lib/api.js b/src/frontend/src/lib/api.js index bb41eec..d873c8b 100644 --- a/src/frontend/src/lib/api.js +++ b/src/frontend/src/lib/api.js @@ -127,6 +127,12 @@ export const qsosAPI = { return apiRequest(`/qsos?${params}`); }, + /** + * Get QSO statistics + * @returns {Promise} QSO statistics + */ + getStats: () => apiRequest('/qsos/stats'), + /** * Sync QSOs from LoTW * @returns {Promise} Sync result diff --git a/src/frontend/src/routes/qsos/+page.svelte b/src/frontend/src/routes/qsos/+page.svelte new file mode 100644 index 0000000..9510606 --- /dev/null +++ b/src/frontend/src/routes/qsos/+page.svelte @@ -0,0 +1,462 @@ + + + + QSO Log - Ham Radio Awards + + +
+
+

QSO Log

+ +
+ + {#if syncResult} +
+ {#if syncResult.success} +

Sync Complete!

+

Total: {syncResult.total}, Added: {syncResult.added}, Updated: {syncResult.updated}

+ {#if syncResult.errors && syncResult.errors.length > 0} +

Some QSOs had errors

+ {/if} + {:else} +

Sync Failed

+

{syncResult.error}

+ {/if} + +
+ {/if} + + {#if stats} +
+
+
{stats.total}
+
Total QSOs
+
+
+
{stats.confirmed}
+
Confirmed
+
+
+
{stats.uniqueEntities}
+
DXCC Entities
+
+
+
{stats.uniqueBands}
+
Bands
+
+
+ {/if} + +
+

Filters

+
+ + + + + + + +
+
+ + {#if loading} +
Loading QSOs...
+ {:else if error} +
Error: {error}
+ {:else if qsos.length === 0} +
+

No QSOs found. Sync from LoTW to get started!

+
+ {:else} +
+ + + + + + + + + + + + + + + {#each qsos as qso} + + + + + + + + + + + {/each} + +
DateTimeCallsignBandModeEntityGridStatus
{formatDate(qso.qsoDate)}{formatTime(qso.timeOn)}{qso.callsign}{qso.band || '-'}{qso.mode || '-'}{qso.entity || '-'}{qso.grid || '-'} + {#if qso.lotwQslRstatus === 'Y'} + Confirmed + {:else} + Pending + {/if} +
+
+

Showing {qsos.length} QSOs

+ {/if} +
+ + diff --git a/src/frontend/src/routes/settings/+page.svelte b/src/frontend/src/routes/settings/+page.svelte new file mode 100644 index 0000000..085a36e --- /dev/null +++ b/src/frontend/src/routes/settings/+page.svelte @@ -0,0 +1,335 @@ + + + + Settings - Ham Radio Awards + + +
+
+

Settings

+ +
+ + + +
+

LoTW Credentials

+

+ Configure your ARRL Logbook of the World (LoTW) credentials to sync your QSOs. + Your credentials are stored securely and used only to fetch your confirmed QSOs. +

+ + {#if hasCredentials} +
+ Credentials configured - You can update them below if needed. +
+ {/if} + +
+ {#if error} +
{error}
+ {/if} + + {#if success} +
+ LoTW credentials saved successfully! +
+ {/if} + +
+ + +
+ +
+ + +

+ Leave blank to keep existing password +

+
+ + +
+ +
+

About LoTW

+

+ LoTW (Logbook of the World) is ARRL's system for confirming amateur radio contacts. + Once configured, you can sync your confirmed QSOs to track award progress. +

+

+ Don't have a LoTW account?{' '} + + Sign up at LoTWARRL.org + +

+
+
+
+ +