Joerg
dd3beef9af
- Award detail page now shows QSO counts per (entity, band, mode) slot - Click count to open modal with all QSOs for that slot - Click QSO in list to view full details - Add mode filter: "Mixed Mode" aggregates by band, specific modes show (band, mode) columns - Backend groups by slot and collects all confirmed QSOs in qsos array - Frontend displays clickable count links (removed blue bubbles) Backend changes: - calculateDOKAwardProgress(): groups by (DOK, band, mode), collects qsos array - calculatePointsAwardProgress(): updated for all count modes with qsos array - getAwardEntityBreakdown(): groups by (entity, band, mode) slots Frontend changes: - Add mode filter dropdown with "Mixed Mode" default - Update grouping logic to handle mixed mode vs specific mode - Replace count badges with simple clickable links - Add QSO list modal showing all QSOs per slot - Add Mode column to QSO list (useful in mixed mode) Co-Authored-By: Claude <noreply@anthropic.com>
807 lines
31 KiB
Markdown
807 lines
31 KiB
Markdown
|
|
Default to using Bun instead of Node.js.
|
|
|
|
- Use `bun <file>` instead of `node <file>` or `ts-node <file>`
|
|
- Use `bun test` instead of `jest` or `vitest`
|
|
- Use `bun build <file.html|file.ts|file.css>` instead of `webpack` or `esbuild`
|
|
- Use `bun install` instead of `npm install` or `yarn install` or `pnpm install`
|
|
- Use `bun run <script>` instead of `npm run <script>` or `yarn run <script>` or `pnpm run <script>`
|
|
- Use `bunx <package> <command>` instead of `npx <package> <command>`
|
|
- Bun automatically loads .env, so don't use dotenv.
|
|
|
|
## APIs
|
|
|
|
- `Bun.serve()` supports WebSockets, HTTPS, and routes. Don't use `express`.
|
|
- `bun:sqlite` for SQLite. Don't use `better-sqlite3`.
|
|
- `Bun.redis` for Redis. Don't use `ioredis`.
|
|
- `Bun.sql` for Postgres. Don't use `pg` or `postgres.js`.
|
|
- `WebSocket` is built-in. Don't use `ws`.
|
|
- Prefer `Bun.file` over `node:fs`'s readFile/writeFile
|
|
- Bun.$`ls` instead of execa.
|
|
|
|
## Logging
|
|
|
|
The application uses a custom logger that outputs to both files and console.
|
|
|
|
### Backend Logging
|
|
|
|
Backend logs are written to `logs/backend.log`:
|
|
- **Log levels**: `debug` (0), `info` (1), `warn` (2), `error` (3)
|
|
- **Default**: `debug` in development, `info` in production
|
|
- **Override**: Set `LOG_LEVEL` environment variable (e.g., `LOG_LEVEL=debug`)
|
|
- **Output format**: `[timestamp] LEVEL: message` with JSON data
|
|
- **Console**: Also outputs to console in development mode
|
|
- **File**: Always writes to `logs/backend.log`
|
|
|
|
### Frontend Logging
|
|
|
|
Frontend logs are sent to the backend and written to `logs/frontend.log`:
|
|
- **Logger**: `src/frontend/src/lib/logger.js`
|
|
- **Endpoint**: `POST /api/logs`
|
|
- **Batching**: Batches logs (up to 10 entries or 5 seconds) for performance
|
|
- **User context**: Automatically includes userId and user-agent
|
|
- **Levels**: Same as backend (debug, info, warn, error)
|
|
|
|
**Usage in frontend**:
|
|
```javascript
|
|
import { logger } from '$lib/logger';
|
|
|
|
logger.info('User action', { action: 'click', element: 'button' });
|
|
logger.error('API error', { error: err.message });
|
|
logger.warn('Deprecated feature used');
|
|
logger.debug('Component state', { state: componentState });
|
|
```
|
|
|
|
**Important**: The logger uses the nullish coalescing operator (`??`) to handle log levels. This ensures that `debug` (level 0) is not treated as falsy.
|
|
|
|
Example `.env` file:
|
|
```
|
|
NODE_ENV=development
|
|
LOG_LEVEL=debug
|
|
```
|
|
|
|
**Log Files**:
|
|
- `logs/backend.log` - Backend server logs
|
|
- `logs/frontend.log` - Frontend client logs
|
|
- Logs are excluded from git via `.gitignore`
|
|
|
|
## Testing
|
|
|
|
Use `bun test` to run tests.
|
|
|
|
```ts#index.test.ts
|
|
import { test, expect } from "bun:test";
|
|
|
|
test("hello world", () => {
|
|
expect(1).toBe(1);
|
|
});
|
|
```
|
|
|
|
## Frontend
|
|
|
|
Use HTML imports with `Bun.serve()`. Don't use `vite`. HTML imports fully support React, CSS, Tailwind.
|
|
|
|
Server:
|
|
|
|
```ts#index.ts
|
|
import index from "./index.html"
|
|
|
|
Bun.serve({
|
|
routes: {
|
|
"/": index,
|
|
"/api/users/:id": {
|
|
GET: (req) => {
|
|
return new Response(JSON.stringify({ id: req.params.id }));
|
|
},
|
|
},
|
|
},
|
|
// optional websocket support
|
|
websocket: {
|
|
open: (ws) => {
|
|
ws.send("Hello, world!");
|
|
},
|
|
message: (ws, message) => {
|
|
ws.send(message);
|
|
},
|
|
close: (ws) => {
|
|
// handle close
|
|
}
|
|
},
|
|
development: {
|
|
hmr: true,
|
|
console: true,
|
|
}
|
|
})
|
|
```
|
|
|
|
HTML files can import .tsx, .jsx or .js files directly and Bun's bundler will transpile & bundle automatically. `<link>` tags can point to stylesheets and Bun's CSS bundler will bundle.
|
|
|
|
```html#index.html
|
|
<html>
|
|
<body>
|
|
<h1>Hello, world!</h1>
|
|
<script type="module" src="./frontend.tsx"></script>
|
|
</body>
|
|
</html>
|
|
```
|
|
|
|
With the following `frontend.tsx`:
|
|
|
|
```tsx#frontend.tsx
|
|
import React from "react";
|
|
import { createRoot } from "react-dom/client";
|
|
|
|
// import .css files directly and it works
|
|
import './index.css';
|
|
|
|
const root = createRoot(document.body);
|
|
|
|
export default function Frontend() {
|
|
return <h1>Hello, world!</h1>;
|
|
}
|
|
|
|
root.render(<Frontend />);
|
|
```
|
|
|
|
Then, run index.ts
|
|
|
|
```sh
|
|
bun --hot ./index.ts
|
|
```
|
|
|
|
For more information, read the Bun API docs in `node_modules/bun-types/docs/**.mdx`.
|
|
|
|
## Project: Quickawards by DJ7NT
|
|
|
|
Quickawards is a amateur radio award tracking application that calculates progress toward various awards based on QSO (contact) data.
|
|
|
|
### Award System Architecture
|
|
|
|
The award system is JSON-driven and located in `award-definitions/` directory. Each award has:
|
|
- `id`: Unique identifier (e.g., "dld", "dxcc")
|
|
- `name`: Display name
|
|
- `description`: Short description
|
|
- `caption`: Detailed explanation
|
|
- `category`: Award category ("dxcc", "darc", etc.)
|
|
- `rules`: Award calculation logic
|
|
|
|
### Award Rule Types
|
|
|
|
1. **`entity`**: Count unique entities (DXCC countries, states, grid squares)
|
|
- `entityType`: What to count ("dxcc", "state", "grid", "callsign")
|
|
- `target`: Number required for award
|
|
- `filters`: Optional filters (band, mode, etc.)
|
|
- `displayField`: Optional field to display
|
|
|
|
2. **`dok`**: Count unique DOK (DARC Ortsverband Kennung) combinations
|
|
- `target`: Number required
|
|
- `confirmationType`: "dcl" (DARC Community Logbook)
|
|
- `filters`: Optional filters (band, mode, etc.) for award variants
|
|
- Counts unique (DOK, band, mode) combinations
|
|
- Only DCL-confirmed QSOs count
|
|
- Example variants: DLD 80m, DLD CW, DLD 80m CW
|
|
|
|
3. **`points`**: Point-based awards
|
|
- `stations`: Array of {callsign, points}
|
|
- `target`: Points required
|
|
- `countMode`: "perStation", "perBandMode", or "perQso"
|
|
|
|
4. **`filtered`**: Filtered version of another award
|
|
- `baseRule`: The base entity rule
|
|
- `filters`: Additional filters to apply
|
|
|
|
5. **`counter`**: Count QSOs or callsigns
|
|
|
|
### Key Files
|
|
|
|
**Backend Award Service**: `src/backend/services/awards.service.js`
|
|
- `getAllAwards()`: Returns all available award definitions
|
|
- `calculateAwardProgress(userId, award, options)`: Main calculation function
|
|
- `calculateDOKAwardProgress(userId, award, options)`: DOK-specific calculation
|
|
- `calculatePointsAwardProgress(userId, award, options)`: Point-based calculation
|
|
- `getAwardEntityBreakdown(userId, awardId)`: Detailed entity breakdown
|
|
- `getAwardProgressDetails(userId, awardId)`: Progress with details
|
|
|
|
**Database Schema**: `src/backend/db/schema/index.js`
|
|
- QSO fields include: `darcDok`, `dclQslRstatus`, `dclQslRdate`
|
|
- DOK fields support DLD award tracking
|
|
- DCL confirmation fields separate from LoTW
|
|
|
|
**Award Definitions**: `award-definitions/*.json`
|
|
- Add new awards by creating JSON definition files
|
|
- Add filename to `loadAwardDefinitions()` file list in awards.service.js
|
|
|
|
**ADIF Parser**: `src/backend/utils/adif-parser.js`
|
|
- `parseADIF(adifData)`: Parse ADIF format into QSO records
|
|
- Handles case-insensitive `<EOR>` delimiters (supports `<EOR>`, `<eor>`, `<Eor>`)
|
|
- Uses `matchAll()` for reliable field parsing
|
|
- Skips header records automatically
|
|
- `parseDCLResponse(response)`: Parse DCL's JSON response format `{ "adif": "..." }`
|
|
- `normalizeBand(band)`: Standardize band names (80m, 40m, etc.)
|
|
- `normalizeMode(mode)`: Standardize mode names (CW, FT8, SSB, etc.)
|
|
- Used by both LoTW and DCL services for consistency
|
|
|
|
**Job Queue Service**: `src/backend/services/job-queue.service.js`
|
|
- Manages async background jobs for LoTW and DCL sync
|
|
- `enqueueJob(userId, jobType)`: Queue a sync job ('lotw_sync' or 'dcl_sync')
|
|
- `processJobAsync(jobId, userId, jobType)`: Process job asynchronously
|
|
- `getUserActiveJob(userId, jobType)`: Get active job for user (optional type filter)
|
|
- `getJobStatus(jobId)`: Get job status with parsed result
|
|
- `updateJobProgress(jobId, progressData)`: Update job progress during processing
|
|
- Supports concurrent LoTW and DCL sync jobs
|
|
- Job types: 'lotw_sync', 'dcl_sync'
|
|
- Job status: 'pending', 'running', 'completed', 'failed'
|
|
|
|
**Backend API Routes** (`src/backend/index.js`):
|
|
- `POST /api/lotw/sync`: Queue LoTW sync job
|
|
- `POST /api/dcl/sync`: Queue DCL sync job
|
|
- `GET /api/jobs/:jobId`: Get job status
|
|
- `GET /api/jobs/active`: Get active job for current user
|
|
- `GET /*`: Serves static files from `src/frontend/build/` with SPA fallback
|
|
|
|
**SPA Routing**: The backend serves the SvelteKit frontend build from `src/frontend/build/`.
|
|
- Paths with file extensions (`.js`, `.css`, etc.) are served as static files
|
|
- Paths without extensions (e.g., `/qsos`, `/awards`) are served `index.html` for client-side routing
|
|
- Common missing files like `/favicon.ico` return 404 immediately
|
|
- If frontend build is missing entirely, returns a user-friendly 503 HTML page
|
|
- Prevents ugly Bun error pages when accessing client-side routes via curl or non-JS clients
|
|
|
|
**DCL Service**: `src/backend/services/dcl.service.js`
|
|
- `fetchQSOsFromDCL(dclApiKey, sinceDate)`: Fetch from DCL API
|
|
- API Endpoint: `https://dings.dcl.darc.de/api/adiexport`
|
|
- Request: POST with JSON body `{ key, limit: 50000, qsl_since, qso_since, cnf_only }`
|
|
- `cnf_only: null` - Fetch ALL QSOs (confirmed + unconfirmed)
|
|
- `cnf_only: true` - Fetch only confirmed QSOs (dcl_qsl_rcvd='Y')
|
|
- `qso_since: DATE` - QSOs since this date (YYYYMMDD format)
|
|
- `qsl_since: DATE` - QSL confirmations since this date (YYYYMMDD format)
|
|
- `parseDCLJSONResponse(jsonResponse)`: Parse example/test payloads
|
|
- `syncQSOs(userId, dclApiKey, sinceDate, jobId)`: Sync QSOs to database
|
|
- `getLastDCLQSLDate(userId)`: Get last QSL date for incremental sync
|
|
- `getLastDCLQSODate(userId)`: Get last QSO date for incremental sync
|
|
- Debug logging (when `LOG_LEVEL=debug`) shows API params with redacted key (first/last 4 chars)
|
|
- Fully implemented and functional
|
|
- **Note**: DCL API is a custom prototype by DARC; contact DARC for API specification details
|
|
|
|
### DLD Award Implementation (COMPLETED)
|
|
|
|
The DLD (Deutschland Diplom) award was recently implemented:
|
|
|
|
**Definition**: `award-definitions/dld.json`
|
|
```json
|
|
{
|
|
"id": "dld",
|
|
"name": "DLD",
|
|
"description": "Deutschland Diplom - Confirm 100 unique DOKs on different bands/modes",
|
|
"caption": "Contact and confirm stations with 100 unique DOKs (DARC Ortsverband Kennung) on different band/mode combinations.",
|
|
"category": "darc",
|
|
"rules": {
|
|
"type": "dok",
|
|
"target": 100,
|
|
"confirmationType": "dcl",
|
|
"displayField": "darcDok"
|
|
}
|
|
}
|
|
```
|
|
|
|
**Implementation Details**:
|
|
- Function: `calculateDOKAwardProgress()` in `src/backend/services/awards.service.js` (lines 173-268)
|
|
- Counts unique (DOK, band, mode) combinations
|
|
- Only DCL-confirmed QSOs count (`dclQslRstatus === 'Y'`)
|
|
- Each unique DOK on each unique band/mode counts separately
|
|
- Returns worked, confirmed counts and entity breakdowns
|
|
|
|
**Database Fields Used**:
|
|
- `darcDok`: DOK identifier (e.g., "F03", "P30", "G20")
|
|
- `band`: Band (e.g., "80m", "40m", "20m")
|
|
- `mode`: Mode (e.g., "CW", "SSB", "FT8")
|
|
- `dclQslRstatus`: DCL confirmation status ('Y' = confirmed)
|
|
- `dclQslRdate`: DCL confirmation date
|
|
|
|
**Documentation**: See `docs/DOCUMENTATION.md` for complete documentation including DLD award example.
|
|
|
|
**Frontend**: `src/frontend/src/routes/qsos/+page.svelte`
|
|
- Separate sync buttons for LoTW (blue) and DCL (orange)
|
|
- Independent progress tracking for each sync type
|
|
- Both syncs can run simultaneously
|
|
- Job polling every 2 seconds for status updates
|
|
- Import log displays after sync completion
|
|
- Real-time QSO table refresh after sync
|
|
|
|
**Frontend API** (`src/frontend/src/lib/api.js`):
|
|
- `qsosAPI.syncFromLoTW()`: Trigger LoTW sync
|
|
- `qsosAPI.syncFromDCL()`: Trigger DCL sync
|
|
- `jobsAPI.getStatus(jobId)`: Poll job status
|
|
- `jobsAPI.getActive()`: Get active job on page load
|
|
|
|
### Adding New Awards
|
|
|
|
To add a new award:
|
|
|
|
1. Create JSON definition in `award-definitions/`
|
|
2. Add filename to `loadAwardDefinitions()` in `src/backend/services/awards.service.js`
|
|
3. If new rule type needed, add calculation function
|
|
4. Add type handling in `calculateAwardProgress()` switch statement
|
|
5. Add type handling in `getAwardEntityBreakdown()` if needed
|
|
6. Update documentation in `docs/DOCUMENTATION.md`
|
|
7. Test with sample QSO data
|
|
|
|
### Creating DLD Award Variants
|
|
|
|
The DOK award type supports filters to create award variants. Examples:
|
|
|
|
**DLD on 80m** (`dld-80m.json`):
|
|
```json
|
|
{
|
|
"id": "dld-80m",
|
|
"name": "DLD 80m",
|
|
"description": "Confirm 100 unique DOKs on 80m",
|
|
"caption": "Contact 100 different DOKs on the 80m band.",
|
|
"category": "darc",
|
|
"rules": {
|
|
"type": "dok",
|
|
"target": 100,
|
|
"confirmationType": "dcl",
|
|
"displayField": "darcDok",
|
|
"filters": {
|
|
"operator": "AND",
|
|
"filters": [
|
|
{ "field": "band", "operator": "eq", "value": "80m" }
|
|
]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
**DLD in CW mode** (`dld-cw.json`):
|
|
```json
|
|
{
|
|
"rules": {
|
|
"type": "dok",
|
|
"target": 100,
|
|
"confirmationType": "dcl",
|
|
"filters": {
|
|
"operator": "AND",
|
|
"filters": [
|
|
{ "field": "mode", "operator": "eq", "value": "CW" }
|
|
]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
**DLD on 80m using CW** (combined filters, `dld-80m-cw.json`):
|
|
```json
|
|
{
|
|
"rules": {
|
|
"type": "dok",
|
|
"target": 100,
|
|
"confirmationType": "dcl",
|
|
"filters": {
|
|
"operator": "AND",
|
|
"filters": [
|
|
{ "field": "band", "operator": "eq", "value": "80m" },
|
|
{ "field": "mode", "operator": "eq", "value": "CW" }
|
|
]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
**Available filter operators**:
|
|
- `eq`: equals
|
|
- `ne`: not equals
|
|
- `in`: in array
|
|
- `nin`: not in array
|
|
- `contains`: contains substring
|
|
|
|
**Available filter fields**: Any QSO field (band, mode, callsign, grid, state, satName, etc.)
|
|
|
|
### Confirmation Systems
|
|
|
|
- **LoTW (Logbook of The World)**: ARRL's confirmation system
|
|
- Service: `src/backend/services/lotw.service.js`
|
|
- API: `https://lotw.arrl.org/lotwuser/lotwreport.adi`
|
|
- Fields: `lotwQslRstatus`, `lotwQslRdate`
|
|
- Used for DXCC, WAS, VUCC, most awards
|
|
- ADIF format with `<EOR>` delimiters
|
|
- Supports incremental sync by `qso_qslsince` parameter (format: YYYY-MM-DD)
|
|
|
|
- **DCL (DARC Community Logbook)**: DARC's confirmation system
|
|
- Service: `src/backend/services/dcl.service.js`
|
|
- API: `https://dings.dcl.darc.de/api/adiexport`
|
|
- Fields: `dclQslRstatus`, `dclQslRdate`
|
|
- DOK fields: `darcDok` (partner's DOK), `myDarcDok` (user's DOK)
|
|
- Required for DLD award
|
|
- German amateur radio specific
|
|
- Request format: POST JSON `{ key, limit, qsl_since, qso_since, cnf_only }`
|
|
- `cnf_only: null` - Fetch all QSOs (confirmed + unconfirmed)
|
|
- `cnf_only: true` - Fetch only confirmed QSOs
|
|
- `qso_since` - QSOs since this date (YYYYMMDD)
|
|
- `qsl_since` - QSL confirmations since this date (YYYYMMDD)
|
|
- Response format: JSON with ADIF string in `adif` field
|
|
- Syncs ALL QSOs (both confirmed and unconfirmed)
|
|
- Unconfirmed QSOs stored but don't count toward awards
|
|
- Updates QSOs only if confirmation data has changed
|
|
|
|
### ADIF Format
|
|
|
|
Both LoTW and DCL return data in ADIF (Amateur Data Interchange Format):
|
|
- Field format: `<FIELD_NAME:length>value`
|
|
- Record delimiter: `<EOR>` (end of record, case-insensitive)
|
|
- Header ends with: `<EOH>` (end of header)
|
|
- Example: `<CALL:5>DK0MU<BAND:3>80m<QSO_DATE:8>20250621<EOR>`
|
|
- **Important**: Parser handles case-insensitive `<EOR>`, `<eor>`, `<Eor>` tags
|
|
|
|
**DCL-specific fields**:
|
|
- `DCL_QSL_RCVD`: DCL confirmation status (Y/N/?)
|
|
- `DCL_QSLRDATE`: DCL confirmation date (YYYYMMDD)
|
|
- `DARC_DOK`: QSO partner's DOK
|
|
- `MY_DARC_DOK`: User's own DOK
|
|
- `STATION_CALLSIGN`: User's callsign
|
|
|
|
### Recent Commits
|
|
|
|
- `aeeb75c`: feat: add QSO count display to filter section
|
|
- Shows count of QSOs matching current filters next to "Filters" heading
|
|
- Displays "Showing X filtered QSOs" when filters are active
|
|
- Displays "Showing X total QSOs" when no filters applied
|
|
- Dynamically updates when filters change
|
|
- `bee02d1`: fix: count QSOs confirmed by either LoTW or DCL in stats
|
|
- QSO stats were only counting LoTW-confirmed QSOs (`lotwQslRstatus === 'Y'`)
|
|
- QSOs confirmed only by DCL were excluded from "confirmed" count
|
|
- Fixed by changing filter to: `q.lotwQslRstatus === 'Y' || q.dclQslRstatus === 'Y'`
|
|
- Now correctly shows all QSOs confirmed by at least one system
|
|
- `233888c`: fix: make ADIF parser case-insensitive for EOR delimiter
|
|
- **Critical bug**: LoTW uses lowercase `<eor>` tags, parser was splitting on uppercase `<EOR>`
|
|
- Caused 242K+ QSOs to be parsed as 1 giant record with fields overwriting each other
|
|
- Changed to case-insensitive regex: `new RegExp('<eor>', 'gi')`
|
|
- Replaced `regex.exec()` while loop with `matchAll()` for-of iteration
|
|
- Now correctly imports all QSOs from large LoTW reports
|
|
- `645f786`: fix: add missing timeOn field to LoTW duplicate detection
|
|
- LoTW sync was missing `timeOn` in duplicate detection query
|
|
- Multiple QSOs with same callsign/date/band/mode but different times were treated as duplicates
|
|
- Now matches DCL sync logic: `userId, callsign, qsoDate, timeOn, band, mode`
|
|
- `7f77c3a`: feat: add filter support for DOK awards
|
|
- DOK award type now supports filtering by band, mode, and other QSO fields
|
|
- Allows creating award variants like DLD 80m, DLD CW, DLD 80m CW
|
|
- Uses existing filter system with eq, ne, in, nin, contains operators
|
|
- Example awards created: dld-80m, dld-40m, dld-cw, dld-80m-cw
|
|
- `9e73704`: docs: update CLAUDE.md with DLD award variants documentation
|
|
- `7201446`: fix: return proper HTML for SPA routes instead of Bun error page
|
|
- When accessing client-side routes (like /qsos) via curl or non-JS clients,
|
|
the server attempted to open them as static files, causing Bun to throw
|
|
an unhandled ENOENT error that showed an ugly error page
|
|
- Now checks if a path has a file extension before attempting to serve it
|
|
- Paths without extensions are immediately served index.html for SPA routing
|
|
- Also improves the 503 error page with user-friendly HTML when frontend build is missing
|
|
- `223461f`: fix: enable debug logging and improve DCL sync observability
|
|
- `27d2ef1`: fix: preserve DOK data when DCL doesn't send values
|
|
- DCL sync only updates DOK/grid fields when DCL provides non-empty values
|
|
- Prevents accidentally clearing DOK data from manual entry or other sources
|
|
- Preserves existing DOK when DCL syncs QSO without DOK information
|
|
- `e09ab94`: feat: skip QSOs with unchanged confirmation data
|
|
- LoTW/DCL sync only updates QSOs if confirmation data has changed
|
|
- Tracks added, updated, and skipped QSO counts
|
|
- LoTW: Checks if lotwQslRstatus or lotwQslRdate changed
|
|
- DCL: Checks if dclQslRstatus, dclQslRdate, darcDok, myDarcDok, or grid changed
|
|
- `3592dbb`: feat: add import log showing synced QSOs
|
|
- Backend returns addedQSOs and updatedQSOs arrays in sync result
|
|
- Frontend displays import log with callsign, date, band, mode for each QSO
|
|
- Separate sections for "New QSOs" and "Updated QSOs"
|
|
- Sync summary shows total, added, updated, skipped counts
|
|
- `8a1a580`: feat: implement DCL ADIF parser and service integration
|
|
- Add shared ADIF parser utility (src/backend/utils/adif-parser.js)
|
|
- Implement DCL service with API integration
|
|
- Refactor LoTW service to use shared parser
|
|
- Tested with example DCL payload (6 QSOs parsed successfully)
|
|
- `c982dcd`: feat: implement DLD (Deutschland Diplom) award
|
|
- `322ccaf`: docs: add DLD (Deutschland Diplom) award documentation
|
|
|
|
### Sync Behavior
|
|
|
|
**Import Log**: After each sync, displays a table showing:
|
|
- New QSOs: Callsign, Date, Band, Mode
|
|
- Updated QSOs: Callsign, Date, Band, Mode (only if data changed)
|
|
- Skipped QSOs: Counted but not shown (data unchanged)
|
|
|
|
**Duplicate Handling**:
|
|
- QSOs matched by: userId, callsign, qsoDate, timeOn, band, mode
|
|
- If confirmation data unchanged: Skipped (not updated)
|
|
- If confirmation data changed: Updated with new values
|
|
- Prevents unnecessary database writes and shows accurate import counts
|
|
|
|
**DOK Update Behavior**:
|
|
- If QSO imported via LoTW (no DOK) and later DCL confirms with DOK: DOK is added ✓
|
|
- If QSO already has DOK and DCL sends different DOK: DOK is updated ✓
|
|
- If QSO has DOK and DCL syncs without DOK (empty): Existing DOK is preserved ✓
|
|
- LoTW never sends DOK data; only DCL provides DOK fields
|
|
|
|
**Important**: DCL sync only updates DOK/grid fields when DCL provides non-empty values. This prevents accidentally clearing DOK data that was manually entered or imported from other sources.
|
|
|
|
### DCL Sync Strategy
|
|
|
|
**Current Behavior**: DCL syncs ALL QSOs (confirmed + unconfirmed)
|
|
|
|
The application syncs both confirmed and unconfirmed QSOs from DCL:
|
|
- **Confirmed QSOs**: `dclQslRstatus = 'Y'` - Count toward awards
|
|
- **Unconfirmed QSOs**: `dclQslRstatus = 'N'` - Stored but don't count toward awards
|
|
|
|
**Purpose of syncing unconfirmed QSOs**:
|
|
- Users can see who they've worked (via "Not Confirmed" filter)
|
|
- Track QSOs awaiting confirmation
|
|
- QSOs can get confirmed later and will be updated on next sync
|
|
|
|
**Award Calculation**: Always uses confirmed QSOs only (e.g., `dclQslRstatus === 'Y'` for DLD award)
|
|
|
|
### DCL Incremental Sync Strategy
|
|
|
|
**Challenge**: Need to fetch both new QSOs AND confirmation updates to old QSOs
|
|
|
|
**Example Scenario**:
|
|
1. Full sync on 2026-01-20 → Last QSO date: 2026-01-20
|
|
2. User works 3 new QSOs on 2026-01-25 (unconfirmed)
|
|
3. Old QSO from 2026-01-10 gets confirmed on 2026-01-26
|
|
4. Next sync needs both: new QSOs (2026-01-25) AND confirmation update (2026-01-10)
|
|
|
|
**Solution**: Use both `qso_since` and `qsl_since` parameters with OR logic
|
|
|
|
```javascript
|
|
// Proposed sync logic (requires OR logic from DCL API)
|
|
const lastQSODate = await getLastDCLQSODate(userId); // Track QSO dates
|
|
const lastQSLDate = await getLastDCLQSLDate(userId); // Track QSL dates
|
|
|
|
const requestBody = {
|
|
key: dclApiKey,
|
|
limit: 50000,
|
|
qso_since: lastQSODate, // Get new QSOs since last contact
|
|
qsl_since: lastQSLDate, // Get QSL confirmations since last sync
|
|
cnf_only: null, // Fetch all QSOs
|
|
};
|
|
```
|
|
|
|
**Required API Behavior (OR Logic)**:
|
|
- Return QSOs where `(qso_date >= qso_since) OR (qsl_date >= qsl_since)`
|
|
- This ensures we get both new QSOs and confirmation updates
|
|
|
|
**Current DCL API Status**:
|
|
- Unknown if current API uses AND or OR logic for combined filters
|
|
- **Action Needed**: Request OR logic implementation from DARC
|
|
- Test current behavior to confirm API response pattern
|
|
|
|
**Why OR Logic is Needed**:
|
|
- With AND logic: Old QSOs getting confirmed are missed (qso_date too old)
|
|
- With OR logic: All updates captured efficiently in one API call
|
|
|
|
### QSO Page Filters
|
|
|
|
The QSO page (`src/frontend/src/routes/qsos/+page.svelte`) includes advanced filtering capabilities:
|
|
|
|
**Available Filters**:
|
|
- **Search Box**: Full-text search across callsign, entity (DXCC country), and grid square fields
|
|
- Press Enter to apply search
|
|
- Case-insensitive partial matching
|
|
- **Band Filter**: Dropdown to filter by amateur band (160m, 80m, 60m, 40m, 30m, 20m, 17m, 15m, 12m, 10m, 6m, 2m, 70cm)
|
|
- **Mode Filter**: Dropdown to filter by mode (CW, SSB, AM, FM, RTTY, PSK31, FT8, FT4, JT65, JT9)
|
|
- **Confirmation Type Filter**: Filter by confirmation status
|
|
- "All QSOs": Shows all QSOs (no filter)
|
|
- "LoTW Only": Shows QSOs confirmed by LoTW but NOT DCL
|
|
- "DCL Only": Shows QSOs confirmed by DCL but NOT LoTW
|
|
- "Both Confirmed": Shows QSOs confirmed by BOTH LoTW AND DCL
|
|
- "Not Confirmed": Shows QSOs confirmed by NEITHER LoTW nor DCL
|
|
- **Clear Button**: Resets all filters and reloads all QSOs
|
|
|
|
**Backend Implementation** (`src/backend/services/lotw.service.js`):
|
|
- `getUserQSOs(userId, filters, options)`: Main filtering function
|
|
- Supports pagination with `page` and `limit` options
|
|
- Filter logic uses Drizzle ORM query builders for safe SQL generation
|
|
- Debug logging when `LOG_LEVEL=debug` shows applied filters
|
|
|
|
**Frontend API** (`src/frontend/src/lib/api.js`):
|
|
- `qsosAPI.getAll(filters)`: Fetch QSOs with optional filters
|
|
- Filters passed as query parameters: `?band=20m&mode=CW&confirmationType=lotw&search=DL`
|
|
|
|
**QSO Count Display**:
|
|
- Shows count of QSOs matching current filters next to "Filters" heading
|
|
- **With filters active**: "Showing **X** filtered QSOs"
|
|
- **No filters**: "Showing **X** total QSOs"
|
|
- Dynamically updates when filters are applied or cleared
|
|
- Uses `pagination.totalCount` from backend API response
|
|
|
|
### DXCC Entity Priority Logic
|
|
|
|
When syncing QSOs from multiple confirmation sources, the system follows a priority order for DXCC entity data:
|
|
|
|
**Priority Order**: LoTW > DCL
|
|
|
|
**Implementation** (`src/backend/services/dcl.service.js`):
|
|
```javascript
|
|
// DXCC priority: LoTW > DCL
|
|
// Only update entity fields from DCL if:
|
|
// 1. QSO is NOT LoTW confirmed, AND
|
|
// 2. DCL actually sent entity data, AND
|
|
// 3. Current entity is missing
|
|
const hasLoTWConfirmation = existingQSO.lotwQslRstatus === 'Y';
|
|
const hasDCLData = dbQSO.entity || dbQSO.entityId;
|
|
const missingEntity = !existingQSO.entity || existingQSO.entity === '';
|
|
|
|
if (!hasLoTWConfirmation && hasDCLData && missingEntity) {
|
|
// Fill in entity data from DCL (only if DCL provides it)
|
|
updateData.entity = dbQSO.entity;
|
|
updateData.entityId = dbQSO.entityId;
|
|
// ... other entity fields
|
|
}
|
|
```
|
|
|
|
**Rules**:
|
|
1. **LoTW-confirmed QSOs**: Always use LoTW's DXCC data (most reliable)
|
|
2. **DCL-only QSOs**: Use DCL's DXCC data IF available in ADIF payload
|
|
3. **Empty entity fields**: If DCL doesn't send DXCC data, entity remains empty
|
|
4. **Never overwrite**: Once LoTW confirms with entity data, DCL sync won't change it
|
|
|
|
**Important Note**: DCL API currently doesn't send DXCC/entity fields in their ADIF export. This is a limitation of the DCL API, not the application. If DCL adds these fields in the future, the system will automatically use them for DCL-only QSOs.
|
|
|
|
### Recent Development Work (January 2025)
|
|
|
|
**QSO Page Enhancements**:
|
|
- Added confirmation type filter with exclusive logic (LoTW Only, DCL Only, Both Confirmed, Not Confirmed)
|
|
- Added search box for filtering by callsign, entity, or grid square
|
|
- Renamed "All Confirmation" to "All QSOs" for clarity
|
|
- Fixed filter logic to properly handle exclusive confirmation types
|
|
|
|
**Bug Fixes**:
|
|
- Fixed confirmation filter showing wrong QSOs (e.g., "LoTW Only" was also showing DCL QSOs)
|
|
- Implemented proper SQL conditions for exclusive filters using separate condition pushes
|
|
- Added debug logging to track filter application
|
|
|
|
**DXCC Entity Handling**:
|
|
- Clarified that DCL API doesn't send DXCC fields (current limitation)
|
|
- Implemented priority logic: LoTW entity data takes precedence over DCL
|
|
- System ready to auto-use DCL DXCC data if they add it in future API updates
|
|
|
|
### Critical LoTW Sync Behavior (LEARNED THE HARD WAY)
|
|
|
|
**⚠️ IMPORTANT: LoTW sync MUST only import confirmed QSOs**
|
|
|
|
After attempting to implement "QSO Delta" sync (all QSOs, confirmed + unconfirmed), we discovered:
|
|
|
|
**The Problem:**
|
|
LoTW ADIF export with `qso_qsl=no` (all QSOs mode) only includes:
|
|
- `CALL` (callsign)
|
|
- `QSL_RCVD` (confirmation status: Y/N)
|
|
|
|
**Missing Fields for Unconfirmed QSOs:**
|
|
- `DXCC` (entity ID) ← **CRITICAL for awards!**
|
|
- `COUNTRY` (entity name)
|
|
- `CONTINENT`
|
|
- `CQ_ZONE`
|
|
- `ITU_ZONE`
|
|
|
|
**Result:** Unconfirmed QSOs have `entityId: null` and `entity: ""`, breaking award calculations.
|
|
|
|
**Current Implementation (CORRECT):**
|
|
```javascript
|
|
// lotw.service.js - fetchQSOsFromLoTW()
|
|
const params = new URLSearchParams({
|
|
login: lotwUsername,
|
|
password: loTWPassword,
|
|
qso_query: '1',
|
|
qso_qsl: 'yes', // ONLY confirmed QSOs
|
|
qso_qslsince: dateStr, // Incremental sync
|
|
});
|
|
```
|
|
|
|
**Why This Matters:**
|
|
- Awards require `entityId` to count entities
|
|
- Without `entityId`, QSOs can't be counted toward DXCC, WAS, etc.
|
|
- Users can still see "worked" stations in QSO list, but awards only count confirmed
|
|
- DCL sync can import all QSOs because it provides entity data via callsign lookup
|
|
|
|
**Attempted Solution (REVERTED):**
|
|
- Tried implementing callsign prefix lookup to populate missing `entityId`
|
|
- Created `src/backend/utils/callsign-lookup.js` with basic prefix mappings
|
|
- Complexity: 1000+ DXCC entities, many special event callsigns, portable designators
|
|
- Decision: Too complex, reverted (commit 310b154)
|
|
|
|
**Takeaway:** LoTW confirmed QSOs have reliable DXCC data. Don't try to workaround this fundamental limitation.
|
|
|
|
### QSO Confirmation Filters
|
|
|
|
Added "Confirmed by at least 1 service" filter to QSO view (commit 688b0fc):
|
|
|
|
**Filter Options:**
|
|
- "All QSOs" - No filter
|
|
- "Confirmed by at least 1 service" (NEW) - LoTW OR DCL confirmed
|
|
- "LoTW Only" - Confirmed by LoTW but NOT DCL
|
|
- "DCL Only" - Confirmed by DCL but NOT LoTW
|
|
- "Both Confirmed" - Confirmed by BOTH LoTW AND DCL
|
|
- "Not Confirmed" - Confirmed by NEITHER
|
|
|
|
**SQL Logic:**
|
|
```sql
|
|
-- "Confirmed by at least 1 service"
|
|
WHERE lotwQslRstatus = 'Y' OR dclQslRstatus = 'Y'
|
|
|
|
-- "LoTW Only"
|
|
WHERE lotwQslRstatus = 'Y' AND (dclQslRstatus IS NULL OR dclQslRstatus != 'Y')
|
|
|
|
-- "DCL Only"
|
|
WHERE dclQslRstatus = 'Y' AND (lotwQslRstatus IS NULL OR lotwQslRstatus != 'Y')
|
|
|
|
-- "Both Confirmed"
|
|
WHERE lotwQslRstatus = 'Y' AND dclQslRstatus = 'Y'
|
|
|
|
-- "Not Confirmed"
|
|
WHERE (lotwQslRstatus IS NULL OR lotwQslRstatus != 'Y')
|
|
AND (dclQslRstatus IS NULL OR dclQslRstatus != 'Y')
|
|
```
|
|
|
|
### Recent Development Work (January 2025)
|
|
|
|
**Sync Type Support (ATTEMPTED & REVERTED):**
|
|
- Commit 5b78935: Added LoTW sync type support (QSL/QSO delta/full)
|
|
- Commit 310b154: Reverted - LoTW doesn't provide entity data for unconfirmed QSOs
|
|
- **Lesson:** Keep it simple - only sync confirmed QSOs from LoTW
|
|
|
|
**Dashboard Enhancements:**
|
|
- Added sync job history display with real-time polling (every 2 seconds)
|
|
- Shows job progress, status, and import logs
|
|
- Cancel button for stale/failed jobs with rollback capability
|
|
- Tracks all QSO changes in `qso_changes` table for rollback
|
|
|
|
**Rollback System:**
|
|
- `cancelJob(jobId, userId)` - Cancels and rolls back sync jobs
|
|
- Tracks added QSOs (deletes them on rollback)
|
|
- Tracks updated QSOs (restores previous state)
|
|
- Only allows canceling failed jobs or stale running jobs (>1 hour)
|
|
- Server-side validation prevents unauthorized cancellations
|
|
|
|
### Award Detail View (January 2025)
|
|
|
|
**Overview**: The award detail page (`src/frontend/src/routes/awards/[id]/+page.svelte`) displays award progress in a pivot table format with entities as rows and band/mode combinations as columns.
|
|
|
|
**Key Features**:
|
|
- **QSO Count per Slot**: Each table cell shows the count of confirmed QSOs for that (entity, band, mode) combination
|
|
- **Drill-Down**: Click a count to open a modal showing all QSOs for that slot
|
|
- **QSO Detail**: Click any QSO in the list to view full QSO details
|
|
- **Mode Filter**: Filter by specific mode or view "Mixed Mode" (aggregates all modes by band)
|
|
|
|
**Backend Changes** (`src/backend/services/awards.service.js`):
|
|
- `calculateDOKAwardProgress()`: Groups by (DOK, band, mode) slots, collects all confirmed QSOs in `qsos` array
|
|
- `calculatePointsAwardProgress()`: Updated for all count modes (perBandMode, perStation, perQso) with `qsos` array
|
|
- `getAwardEntityBreakdown()`: Groups by (entity, band, mode) slots for entity awards
|
|
|
|
**Response Structure**:
|
|
```javascript
|
|
{
|
|
entity: "F03",
|
|
band: "80m",
|
|
mode: "CW",
|
|
worked: true,
|
|
confirmed: true,
|
|
qsos: [
|
|
{ qsoId: 123, callsign: "DK0MU", mode: "CW", qsoDate: "20250115", timeOn: "123456", confirmed: true },
|
|
{ qsoId: 456, callsign: "DL1ABC", mode: "CW", qsoDate: "20250120", timeOn: "234500", confirmed: true }
|
|
]
|
|
}
|
|
```
|
|
|
|
**Mode Filter**:
|
|
- **Mixed Mode (default)**: Shows bands as columns, aggregates all modes
|
|
- Example: Columns are "80m", "40m", "20m"
|
|
- Clicking a count shows all QSOs for that band across all modes
|
|
- **Specific Mode**: Shows (band, mode) combinations as columns
|
|
- Example: Columns are "80m CW", "80m SSB", "40m CW"
|
|
- Filters to only show QSOs with that mode
|
|
|
|
**Frontend Components**:
|
|
- **Mode Filter Dropdown**: Located between summary cards and table
|
|
- Dynamically populated with available modes from the data
|
|
- Clear button appears when specific mode is selected
|
|
- **Count Badges**: Blue clickable links showing QSO count (removed bubbles, kept links)
|
|
- **QSO List Modal**: Shows all QSOs for selected slot with columns: Callsign, Date, Time, Mode
|
|
- **QSO Detail Modal**: Full QSO information (existing feature)
|
|
|
|
**Files Modified**:
|
|
- `src/backend/services/awards.service.js` - Backend grouping and QSO collection
|
|
- `src/frontend/src/routes/awards/[id]/+page.svelte` - Frontend display and interaction
|