feat: add email-server-worker with clean template architecture

Add new email-server-worker project implementing a self-scheduling background worker pattern with HTTP monitoring. Removed all business-specific code from copied source, creating a clean, reusable template.

Key features:
- Self-scheduling worker loop with configurable interval
- Graceful shutdown support (Docker-compatible)
- Prometheus metrics collection
- Health check endpoints (/healthcheck, /metrics, /ping)
- Example worker template for easy customization
- Comprehensive architecture documentation in CLAUDE.md

The worker is now ready for email server implementation with no external dependencies on Evolution/MSSQL/ElasticSearch.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Knee Cola
2025-12-30 09:03:58 +01:00
parent 7a526c5a85
commit 25c2f09eef
30 changed files with 1374 additions and 17 deletions

133
CLAUDE.md
View File

@@ -8,6 +8,7 @@ This is a multi-project repository containing:
- **web-app/**: Next.js 14 utility bills tracking application
- **docker-stack/**: Docker Compose configurations and deployment scripts
- **housekeeping/**: Database backup and maintenance scripts
- **email-server-worker/**: Background worker service with HTTP health monitoring
Each project is self-contained with its own dependencies.
@@ -29,6 +30,13 @@ All commands should be run from within the respective project directory.
- `./db-dump--standalone.sh` - Run standalone database dump
- See housekeeping/README.md for more details
**Email Server Worker** (`cd email-server-worker`):
- `npm install` - Install dependencies
- `npm run start` - Start development server with nodemon
- `npm run build` - Build TypeScript to JavaScript
- `npm run test` - Run tests with Jest in watch mode
- `npm run run-server` - Run built server from ./build directory
## Deployment Commands
**Building Docker Image** (`cd web-app`):
@@ -91,4 +99,127 @@ export const actionName = withUser(async (user: AuthenticatedUser, ...args) => {
### Testing & Code Quality
- ESLint with Next.js and Prettier configurations
- No specific test framework configured - check with user before assuming testing approach
- No specific test framework configured - check with user before assuming testing approach
## Email Server Worker Architecture
The email-server-worker is a TypeScript-based background worker service that combines periodic task execution with HTTP health monitoring and metrics collection.
### Tech Stack
- **Runtime**: Node.js with TypeScript
- **Framework**: Express for HTTP endpoints
- **Metrics**: Prometheus (prom-client) with custom PRTG adapter
- **Testing**: Jest with TypeScript support
### Core Architecture: Worker Pattern
The service implements a **self-contained worker pattern** that runs periodic background tasks while exposing HTTP endpoints for monitoring.
**Entry Point** (`email-server-worker/src/entry.ts:1`):
- Creates Express HTTP server with graceful shutdown support (stoppable)
- Starts the worker via `startSyncWorker()` from `email-server-worker/src/workRunner.ts:134`
- Handles SIGTERM/SIGINT for graceful shutdown (Docker-compatible)
- Calls `disposeSyncWorker()` on shutdown to allow pending work to complete
**Work Runner** (`email-server-worker/src/workRunner.ts:1`):
The work runner implements a self-scheduling loop with the following characteristics:
- **Self-Scheduling Loop**: After completing work, schedules next execution via `setTimeout(workRunner, PULL_INTERVAL)` at `email-server-worker/src/workRunner.ts:113`
- **Graceful Shutdown**: Tracks pending work via Promise, allows in-flight operations to complete before shutdown
- **Status Tracking**: Exports `workerRunnerInfo` with `status` and `lastWorkTime` for health monitoring
- **Error Isolation**: Worker errors don't crash the process - caught, logged, and execution continues
- **Metrics Integration**: Automatic Prometheus metrics collection (duration, success/failure counters)
- **Single Work Instance**: Ensures only one work cycle runs at a time via `pendingWork` Promise
Work Runner States (WorkerRunnerStatus enum):
- `init` - Initial state before first run
- `beginWork` - Work cycle started
- `workDone` - Work completed successfully
- `disposed` - Worker stopped, no longer scheduling
- Other states track Prometheus stats updates
**Worker Implementation Pattern**:
Workers must export a `doWork` function with signature:
```typescript
export const doWork = async () => {
// Perform periodic work here
// Throw errors to increment failedRequestCounter
// Return normally to increment successfulRequestCounter
};
```
The work runner imports and calls this function at `email-server-worker/src/workRunner.ts:88`.
### Key Files & Responsibilities
**Core Worker Files**:
- `email-server-worker/src/entry.ts` - HTTP server setup, signal handling, worker lifecycle management
- `email-server-worker/src/workRunner.ts` - Self-scheduling loop, graceful shutdown, metrics integration
- `email-server-worker/src/app.ts` - Express app configuration, route registration
- `email-server-worker/src/lib/logger.ts` - Debug logger factory (uses 'debug' package)
**HTTP Routes** (`email-server-worker/src/routes/`):
- `healthcheckRouter.ts` - Health check endpoint (checks worker status via `workerRunnerInfo`)
- `metricsRouter.ts` - Prometheus metrics endpoint
- `prtgMetricsRouter.ts` - PRTG-compatible metrics adapter
- `pingRouter.ts` - Simple ping/pong endpoint
- `errorRouter.ts` - Structured error handler for expected errors
- `finalErrorRouter.ts` - Catch-all error handler for unexpected errors
**Infrastructure**:
- `email-server-worker/src/lib/metricsCounters.ts` - Prometheus counter/histogram definitions
- `email-server-worker/src/lib/initTools.ts` - Utility functions (coalesce, etc.)
- `email-server-worker/src/lib/serializeError.ts` - Error serialization for logging
- `email-server-worker/src/lib/Prometheus2Prtg.ts` - Converts Prometheus metrics to PRTG XML format
### Environment Variables
**Required**:
- `PULL_INTERVAL` - Milliseconds between work cycles (default: "10000")
**Optional**:
- `PORT` - HTTP server port (default: "3000")
- `PROMETHEUS_APP_LABEL` - App label for Prometheus metrics (default: "evo-open-table-sync-svc")
- `PROMETHEUS_HISTOGRAM_BUCKETS` - Histogram bucket sizes (default: "0.1, 0.5, 1, 5, 10")
- `DEBUG` - Debug namespaces for console logging (e.g., "server:server")
- `ENV` - Environment mode: "dev", "jest" (affects logging behavior)
### Creating a New Worker
To implement a new worker task:
1. **Create worker file** (e.g., `email-server-worker/src/myWorker.ts`):
```typescript
export const doWork = async () => {
// Implement your periodic task here
logger.info("Work Title", "Work completed successfully");
// Throw errors to mark as failed:
// throw new Error("Something went wrong");
};
```
2. **Update `workRunner.ts`** import at line 6:
```typescript
import { doWork } from "./myWorker";
```
3. **Add environment variables** to `email-server-worker/src/types/environment.d.ts` as needed
4. **Update `package.json` metadata** if the service purpose changes (name, description)
### Docker Deployment
- Uses `stoppable` library for graceful shutdown (10-second timeout before force-close)
- Health check endpoint at `/healthcheck` verifies worker is running and not stalled
- Prometheus metrics at `/metrics` for monitoring
- PRTG-compatible metrics at `/prtg` for legacy monitoring systems
- Graceful shutdown ensures work in progress completes before container stops
### Testing
- **Framework**: Jest with esbuild-jest for TypeScript
- **Test Location**: `email-server-worker/tests/`
- **Mocks**: Common mocks in `email-server-worker/tests/__mocks__/` (prom-client)
- **Test Pattern**: Co-located with source in `tests/` mirroring `src/` structure
- **Run Tests**: `npm run test` (watch mode)

View File

@@ -0,0 +1,7 @@
tests
.git
coverage
node_modules
jest.config.ts
service-tester.sh
build-image.sh

2
email-server-worker/.gitignore vendored Normal file
View File

@@ -0,0 +1,2 @@
node_modules
build

View File

@@ -0,0 +1,75 @@
#-------------------------------------------------------------
# Build command: docker build . -t pr-d-registry.ngit.hr/ngit/evo-open-table-sync-svc:1.0.0
#-------------------------------------------------------------
#--------------------------------------------
# Stage: building TypeScript
#--------------------------------------------
FROM node:18 as build-stage
ENV WORKDIR=/app
WORKDIR /app
# kopiram SSH key & known_hosts koji su potrebni za `npm i`
# zato što `package.json` pri instalacija NGIT paketa koristi
# SSH autentikaciju
COPY _docker_assets/.ssh /root/.ssh
RUN chmod 700 /root/.ssh/*
COPY ./package*.json ./
# instaliram pakete
RUN npm i && npm cache clean --force
COPY ./tsconfig.json ./
COPY ./src ./src
RUN npm run build
#--------------------------------------------
# Stage: instaliram produkcijski node_modules
#--------------------------------------------
FROM node:18 as package-stage
WORKDIR /app
COPY ./package*.json ./
# instaliram SAMO produkcijske
RUN npm i --only=production && npm cache clean --force
#--------------------------------------------
# Stage: priprema finalnog image-a
#--------------------------------------------
FROM gcr.io/distroless/nodejs:18 as assembly-stage
WORKDIR /app
ARG PORT="3000"
ENV PORT=${PORT}
# prometheus config
ARG PROMETHEUS_APP_LABEL
ENV PROMETHEUS_APP_LABEL=${PROMETHEUS_APP_LABEL}
ARG PROMETHEUS_HISTOGRAM_BUCKETS
ENV PROMETHEUS_HISTOGRAM_BUCKETS=${PROMETHEUS_HISTOGRAM_BUCKETS}
# (optional) logiranje na stdout (moguće opcije: "server:server", "server:metrics", "server:healthcheck" )
ARG DEBUG
ENV DEBUG=${DEBUG}
# kopiram node-modules
COPY --from=package-stage /app/package*.json ./
COPY --from=package-stage /app/node_modules ./node_modules
# kopiram buildane datoteke
COPY --from=build-stage /app/build/ ./server
# server vrtim pod ograničenim "nobody" korisnikom
USER nobody:nobody
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s \
CMD ["/nodejs/bin/node", "./server/healthcheck.js"]
# pokrećem server
CMD ["./server/entry.js"]

View File

@@ -1,27 +1,115 @@
# Email Server Worker
This workspace contains the email server worker service for the Evidencija Režija tenant notification system.
Background worker service with HTTP health monitoring and metrics collection.
## Purpose
## Overview
This service manages email operations by:
- Polling MongoDB for email status changes
- Detecting unverified tenant emails (EmailStatus.Unverified)
- Sending verification emails to tenants
- Updating email status to VerificationPending
- Sending scheduled notifications (rent due, utility bills)
This is a TypeScript-based background worker service that combines periodic task execution with HTTP health monitoring and metrics collection. It implements a self-scheduling worker pattern with graceful shutdown support.
## Architecture
## Features
This is a standalone background worker service that:
- Runs independently from the Next.js web-app
- Communicates via the shared MongoDB database
- Integrates with email service provider (e.g., Mailgun, SendGrid)
- **Periodic Task Execution**: Self-scheduling worker loop with configurable interval
- **Graceful Shutdown**: Ensures in-flight work completes before shutdown (Docker-compatible)
- **Health Monitoring**: HTTP health check endpoint to verify worker status
- **Metrics Collection**: Prometheus metrics with PRTG adapter
- **Error Isolation**: Worker errors don't crash the process
## Setup
## Getting Started
TBD
### Installation
```bash
npm install
```
### Development
```bash
npm run start # Start with nodemon (auto-reload)
```
### Build & Run
```bash
npm run build # Compile TypeScript
npm run run-server # Run compiled version
```
### Testing
```bash
npm run test # Run Jest in watch mode
```
## Environment Variables
TBD
### Required
- `PULL_INTERVAL` - Milliseconds between work cycles (default: `"10000"`)
### Optional
- `PORT` - HTTP server port (default: `"3000"`)
- `PROMETHEUS_APP_LABEL` - App label for Prometheus metrics (default: `"email-server-worker"`)
- `PROMETHEUS_HISTOGRAM_BUCKETS` - Histogram bucket sizes (default: `"0.1, 0.5, 1, 5, 10"`)
- `DEBUG` - Debug namespaces for console logging (e.g., `"server:server"`)
- `ENV` - Environment mode: `"dev"`, `"jest"` (affects logging)
## HTTP Endpoints
- `GET /healthcheck` - Health check endpoint (verifies worker is running)
- `GET /metrics` - Prometheus metrics
- `GET /prtg` - PRTG-compatible metrics (XML format)
- `GET /ping` - Simple ping/pong endpoint
## Creating a Worker
See `src/exampleWorker.ts` for the worker template. The worker must export a `doWork` function:
```typescript
export const doWork = async () => {
// Your periodic task logic here
logger.info("Task Completed", "Work done successfully");
// Throw errors to mark as failed:
// throw new Error("Something went wrong");
};
```
Update `src/workRunner.ts` line 6 to import your worker:
```typescript
import { doWork } from "./yourWorker";
```
## Architecture
- **entry.ts** - HTTP server setup, signal handling, worker lifecycle
- **workRunner.ts** - Self-scheduling loop, metrics, graceful shutdown
- **app.ts** - Express app configuration, routes
- **src/lib/** - Shared utilities (logger, metrics, etc.)
- **src/routes/** - HTTP route handlers
## Deployment
The service uses the `stoppable` library for graceful shutdown with a 10-second timeout before force-closing connections. Docker containers will receive SIGTERM signals and shut down gracefully.
## Logging
The service supports two logging mechanisms:
1. **Console Logging**: Uses the `debug` package, controlled by `DEBUG` env variable
## Metrics
Prometheus metrics are automatically collected:
- `request_operations_total` - Total work cycles executed
- `request_operations_ok` - Successful work cycles
- `request_operations_failed` - Failed work cycles
- `request_duration_seconds` - Duration histogram of work cycles
## Documentation
See `CLAUDE.md` in the repository root for complete architecture documentation and guidance.

View File

@@ -0,0 +1,30 @@
#!/bin/bash
if [ "$1" == "" ] ; then
printf "\nDocker image version not set - please specify the version to build"
printf "\n\nSyntax:\n\n build-image.sh 1.0.0\n\n"
exit 1
fi
read -p "BUILD: Push new image to registry [y/n]? " -n 1 -r
echo # (optional) move to a new line
PUSH_IMAGE_TO_REPO="$REPLY"
printf "\nBUILD START ...\n\n"
REGISTRY_URL=registry.ngit.hr
IMAGE_NAME=evo-open-table-sync-svc
IMAGE_VERSION=$1
IMAGE_TAG=$REGISTRY_URL/ngit/$IMAGE_NAME:$IMAGE_VERSION
docker build . -t $IMAGE_TAG
# if [[ "$PUSH_IMAGE_TO_REPO" =~ ^[Yy]$ ]]
# then
# printf "\nPushing image ...\n\n"
# docker push $IMAGE_TAG
# fi
printf "\nBUILD DONE!\n\n"

View File

@@ -0,0 +1,39 @@
/** @type {import('jest/dist/types').InitialOptionsTsJest} */
import type { Config } from 'jest/build/index';
const config:Config = {
// preset: 'ts-jest',
transform: {
'^.+\\.tsx?$': [
'esbuild-jest', {
sourcemap:true, // bez ovog VS code umjesto originala prikazuje transpilirane datoteke
target:'es2020' // ovo je nužno kako bi BigInt funkcionirao
}]
},
maxWorkers: 4,
testEnvironment: 'node',
// The root directory that Jest should scan for tests and modules within
rootDir: "./",
// A list of paths to directories that Jest should use to search for files in
roots: [
"<rootDir>/tests",
],
// The glob patterns Jest uses to detect test files
testMatch: [
"**/?(*.)+(spec).[tj]s?(x)",
],
// Automatically clear mock calls and instances between every test
clearMocks: true,
// An array of regexp pattern strings that are matched against all source file paths, matched files will skip transformation
transformIgnorePatterns: ["/node_modules/"],
// An array of regexp pattern strings that are matched against all test paths, matched tests are skipped
testPathIgnorePatterns: ["/node_modules/", "/build/"],
// Indicates whether each individual test should be reported during the run
verbose: true,
setupFiles: [
'dotenv/config', // učitaj varijable iz .env i učini ih dostupne testiranom software-u
]
};
module.exports = config;

View File

@@ -0,0 +1,49 @@
{
"name": "email-server-worker",
"version": "0.1.0",
"description": "Background worker service with HTTP health monitoring and metrics collection",
"main": "entry.ts",
"scripts": {
"start": "nodemon ./src/entry.ts",
"run-server": "DEBUG=* node --enable-source-maps ./build/entry.js",
"build": "ttsc --project ./",
"test": "ENV=jest jest --watch"
},
"author": "Nikola",
"license": "ISC",
"dependencies": {
"debug": "^2.6.9",
"express": "^4.18.2",
"http-errors": "^1.7.2",
"node-fetch": "^2.6.7",
"prom-client": "^14.0.1",
"stoppable": "^1.1.0"
},
"devDependencies": {
"@babel/preset-typescript": "^7.18.6",
"@types/debug": "^4.1.7",
"@types/express": "^4.17.13",
"@types/http-errors": "^1.8.1",
"@types/jest": "^29.2.5",
"@types/node": "^16.10.2",
"@types/node-fetch": "^2.6.2",
"@types/stoppable": "^1.1.1",
"@types/supertest": "^2.0.11",
"dotenv": "^16.0.3",
"esbuild": "^0.16.14",
"esbuild-jest": "^0.5.0",
"jest": "^29.3.1",
"nodemon": "^2.0.13",
"supertest": "^6.3.3",
"ts-node": "^10.9.1",
"tsconfig-paths": "^4.1.2",
"ttypescript": "^1.5.15",
"typescript": "^4.9.4",
"typescript-transform-paths": "^3.4.4"
},
"babel": {
"presets": [
"@babel/preset-typescript"
]
}
}

View File

@@ -0,0 +1,13 @@
#!/bin/bash
if [ "$1" == "" ] ; then
printf "\nNisi zadao verziju Docker image-a koji treba pokrenuti"
printf "\n\nSintaksa:\n\n run-image.sh 1.0.0\n\n"
exit 1
fi
IMAGE_TAG=pr-d-registry.ngit.hr/ngit/evo-open-table-sync-svc:$1
docker run -p 3000:3000 \
--env DEBUG=* \
$IMAGE_TAG

View File

@@ -0,0 +1,34 @@
import express from 'express';
import createError from 'http-errors';
import { errorRouter } from './routes/errorRouter';
import { finalErrorRouter } from './routes/finalErrorRouter';
import { metricsRouter } from './routes/metricsRouter';
import { pingRouter } from './routes/pingRouter';
import { healthcheckRouter } from './routes/healthcheckRouter';
import { SupportedRoutes } from './types/enums/SupportedRoutes';
const app = express();
// u slučaju kada se server vrti iza proxy-a
// ovaj flag će natjerati Express da informacije poput
// IP adrese klijenta, protokola uzima iz X-Forward-*
// HTTP header polja, koja postavlja proxy
app.set('trust proxy', true);
// prometheus sa ove rute dohvaća zadnje važeću statistiku
app.use(SupportedRoutes.metricsPath, metricsRouter);
app.use(SupportedRoutes.ping, pingRouter);
app.use(SupportedRoutes.healthcheck, healthcheckRouter);
// default handler
app.use((req, res, next) => next(createError(404)));
// error handler za sve predviđene greške
app.use(errorRouter);
// error router za nepredviđene greške
app.use(finalErrorRouter);
export default app;

122
email-server-worker/src/entry.ts Executable file
View File

@@ -0,0 +1,122 @@
import app from './app';
import http from 'http';
import stoppable from 'stoppable';
import { createLogger } from './lib/logger';
import { disposeSyncWorker, startSyncWorker } from './workRunner';
const logger = createLogger("server:server");
/**
* Normalize a port into a number, string, or false.
*/
const normalizePort = (val:string):string|number|boolean => {
const port = parseInt(val, 10);
if (isNaN(port)) {
// named pipe
return val;
}
if (port >= 0) {
// port number
return port;
}
return false;
};
/**
* Event listener for HTTP server "error" event.
*/
const onError = (error:any):void => {
if (error.syscall !== 'listen') {
throw error;
}
const bind = typeof port === 'string'
? 'Pipe ' + port
: 'Port ' + port;
// handle specific listen errors with friendly messages
switch (error.code) {
case 'EACCES':
console.error(bind + ' requires elevated privileges');
process.exit(1);
break;
case 'EADDRINUSE':
console.error(bind + ' is already in use');
process.exit(1);
break;
default:
throw error;
}
};
/**
* Event listener for HTTP server "listening" event.
*/
const onListening = ():void => {
const addr = server.address();
const bind = typeof addr === 'string'
? 'pipe ' + addr
: 'port ' + addr?.port;
logger(`⚡️[server]: Server is running at ${bind}`);
};
/**
* Get port from environment and store in Express.
*/
const port:number|string|boolean = normalizePort(process.env.PORT || '3000');
/**
* How long should stoppable wait before it starts force-closing connections
* @description wait max 10 seconds - needs to be shorter than `healthcheck.timeout` (=15sec)
*/
const FORCE_STOP_TIMEOUT = 10000;
/**
* Create HTTP server.
*/
const server = stoppable( http.createServer(app), FORCE_STOP_TIMEOUT );
// Listen on provided port, on all network interfaces.
server.listen(port);
server.on('error', onError);
server.on('listening', onListening);
/**
* Starting sync worker process
*/
startSyncWorker();
// quit on ctrl-c when running docker in terminal
// (signal neće biti registriran ako je server pokrenuti via `npm` ili `nodemon` - mora biti pokrenuti izravno via Node)
process.on('SIGINT', () => {
logger('Got SIGINT (aka ctrl-c in docker). Graceful shutdown ', new Date().toISOString());
shutdown();
});
// quit properly on docker stop
// (signal neće biti registriran ako je server pokrenuti via `npm` ili `nodemon` - mora biti pokrenuti izravno via Node)
process.on('SIGTERM', () => {
logger('Got SIGTERM (docker container stop). Graceful shutdown ', new Date().toISOString());
shutdown();
});
// shut down server
const shutdown = async () => {
await disposeSyncWorker();
// NOTE: server.close is for express based apps
// If using hapi, use `server.stop`
server.close((err) => {
if (err) {
console.error(err);
process.exitCode = 1;
} else {
logger('Exiting server process...');
}
process.exit();
});
};

View File

@@ -0,0 +1,33 @@
/**
* Example worker implementation
*
* This is a placeholder worker that demonstrates the worker pattern.
* Replace this with your actual worker implementation.
*
* The worker is called periodically by workRunner.ts based on PULL_INTERVAL.
*
* @throws Error to increment failedRequestCounter in Prometheus
* @returns Promise that resolves when work is complete (increments successfulRequestCounter)
*/
export const doWork = async () => {
const startTime = Date.now();
// TODO: Implement your periodic worker logic here
// Examples:
// - Fetch data from external API
// - Process queued tasks from database
// - Send scheduled emails
// - Clean up expired records
// - Sync data between systems
const workDuration = Date.now() - startTime;
// Log success (only in non-test environments)
if (process.env.ENV !== "jest") {
const logMessage = `Example worker completed in ${workDuration}ms`;
console.log(logMessage);
}
// Note: Throw errors to mark work as failed:
// throw new Error("Something went wrong");
};

View File

@@ -0,0 +1,29 @@
import { createLogger } from "./lib/logger";
import http, { IncomingMessage } from "http";
const logger = createLogger("server:healthcheck");
const options = {
host: "localhost",
port: "3000",
timeout: 2000,
path: '/healthcheck/'
};
const request = http.request(options, (res:IncomingMessage) => {
logger(`Healthcheck: STATUS ${res.statusCode}`);
if (res.statusCode == 200) {
process.exit(0);
} else {
process.exit(1);
}
});
request.on("error", function (err:any) {
logger("Healthcheck: ERROR");
process.exit(1);
});
request.end();

View File

@@ -0,0 +1,8 @@
/**
* Za neinicijaliziranu env varijablu vraća default vrijednost
* @param value vrijednost env varijable
* @param defaultValue default vrijednost
* @returns
*/
export const coalesce = (value:string|undefined, defaultValue:string):string => value===undefined ? defaultValue : (value==="" ? defaultValue : value);

View File

@@ -0,0 +1,21 @@
import debug from 'debug';
/**
* Logs to console / stdout
* @param namespace
* @returns instance of Debug
*/
export const createLogger = (namespace:string):debug.Debugger => {
const dbg = debug(namespace);
const rx = /nodemon/gi;
if(rx.test(process.env?.npm_lifecycle_script ?? "")) {
// When started via nodemon:
// forcing the use of console insted of stdout
// -> nodemon doesn't work with stdout
dbg.log = console.log.bind(console);
}
return(dbg);
};

View File

@@ -0,0 +1,50 @@
import { Counter, Histogram, register } from 'prom-client';
import { coalesce } from './initTools';
/** Histogram Buckets */
const PROMETHEUS_HISTOGRAM_BUCKETS = coalesce(process.env.PROMETHEUS_HISTOGRAM_BUCKETS, "0.1, 0.5, 1, 5, 10");
/** Labela kojom želimo da bude označena metrika prikupljena na ovom web servisu */
const PROMETHEUS_APP_LABEL = coalesce(process.env.PROMETHEUS_APP_LABEL, 'email-server-worker');
// na "app" labele ćemo razdvajanje rezultata u Grafani
register.setDefaultLabels({ app: PROMETHEUS_APP_LABEL });
/**
* Broji koliko je ukupno zahtjeva zaprimljeno za obradu
*/
export const totalRequestCounter = new Counter({
name: "request_operations_total",
help: "ukupan broj zaprimljenih zahtjeva",
/** countere razdvajamo po vrsti zahtjeva */
labelNames: ['path'],
});
/**
* Broji zahtjeve koji su uspješno obrađeni
*/
export const successfulRequestCounter = new Counter({
name: "request_operations_ok",
help: "broj zahtjeva koji su uspješno obrađeni",
/** countere razdvajamo po vrsti zahtjeva */
labelNames: ['path'],
});
/**
* Broji zahtjeve kod čije obrade je došlo do greške
*/
export const failedRequestCounter = new Counter({
name: "request_operations_failed",
help: "broj zahtjeva kod čije obrade je došlo do greške",
/** countere razdvajamo po vrsti zahtjeva i rezultatu izvođenja */
labelNames: ["path", "status"],
});
/** Histogram mjeri koliko traje obrada pristiglog zahtjeva */
export const requestDurationHistogram = new Histogram({
name: "request_duration_seconds",
help: "Trajanje request-a u sekundama",
/** countere razdvajamo po vrsti zahtjeva i rezultatu izvođenja */
labelNames: ["path", "status"],
buckets: PROMETHEUS_HISTOGRAM_BUCKETS?.split(',').map((el) => parseFloat(el))
});

View File

@@ -0,0 +1,19 @@
/**
* This function serializes an error object into a string that can be logged
* @param ex error object
* @returns string
* @description SQL Server may generate more than one error for one request so you can access preceding errors with `err.precedingErrors`, while the `ex` itself is a generic error without any useful information
*/
export const serializeError = (ex:Error | Error & { precedingErrors?:Error[] }):string => {
const { name, message, stack, precedingErrors } = (ex as Error & { precedingErrors?:Error[] });
// SQL Server may generate more than one error for one request so you can access preceding errors with `ex.precedingErrors`,
// while the `ex` itself is a generic error without any useful information
if(precedingErrors) {
return(serializeError(precedingErrors[0]));
}
return `${name}:${message}`;
}

View File

@@ -0,0 +1,81 @@
import { ErrorRequestHandler, Request, Response } from "express";
import createHttpError, { HttpError } from "http-errors";
import { createLogger } from '../lib/logger';
import { NgitLocals } from "../types/NgitLocals";
import { failedRequestCounter } from "../lib/metricsCounters";
import { SupportedRoutes } from "../types/enums/SupportedRoutes";
const consoleLog = createLogger("server:server");
/**
* Router koji se zadnji poziva, a koji sastavlja odgovor u slučaju greške
* @param err
* @param req
* @param res
* @param next
*/
export const errorRouter:ErrorRequestHandler = async (err:HttpError, req, res, next) => {
const requestPath = req.path as SupportedRoutes;
// kako je ovaj error handler dosta složen, moguće je da negdje baci grešku
// > zato je zamotan u try-catch
// > na taj način osiguravam da neće srušiti cijeli proces
try {
let { name:errorLogName, message:errorLogText } = err;
let responseBody:string = "";
switch(err.status) {
case 400:
responseBody = 'bad request';
break;
case 401:
responseBody = 'unauthorized';
break;
case 403:
responseBody = 'forbidden';
break;
case 404:
consoleLog(`page not found ${req.method} ${requestPath}`)
responseBody = 'page not found';
errorLogText = `page ${requestPath} not found`;
break;
case 500:
responseBody = "internal server error";
errorLogText = err.message;
break;
default:
responseBody = err.name;
errorLogText = `err.status=${err.status};err.name=${err.name};err.message=${err.message}`;
}
consoleLog(`${errorLogName}:${errorLogText}`);
// `headersSent` će biti TRUE ako je router kod kojeg se dogodila greška već poslao header-e
// > ako ih probam ponovo postaviti, to će baciti grešku ... a to ovdje mogu izbjeći
if(!res.headersSent) {
res.status(err.status);
res.setHeader('Content-Type', "text/html");
res.end(responseBody);
} else {
// AKO nije pozvan `end` - pozovi ga i završi obradu zahtjeva
// ... u suprotnom će konekcija ostati otvorena do timeout-a
if(!res.writableEnded) {
res.end();
}
}
} catch(ex:any) {
// ovu grešku će obraditi `finalErrorRouter`
next(createHttpError(500, ex));
}
// ne mogu dopustiti da prometheus client sruši server
try {
failedRequestCounter.inc({ path: requestPath, status: err.status });
(res.locals as NgitLocals).stopPrometheusTimer({ path: req.path, status: err.status });
} catch(ex:any) {
console.error(ex);
}
};

View File

@@ -0,0 +1,34 @@
import { ErrorRequestHandler, Request, Response } from "express";
import { HttpError } from "http-errors";
import { createLogger } from '../lib/logger';
import { NgitLocals } from "../types/NgitLocals";
const consoleLog = createLogger("server:server");
/**
* Router koji se izvršava u slučaju grube greške koja nije obrađena nigdje prije
* @param err error objekt
* @param req express request
* @param res express response
* @param next
*/
export const finalErrorRouter:ErrorRequestHandler = async (err:HttpError, req, res, next) => {
const errorLogText:string = JSON.stringify({ message:err.message, name:err.name, stack:err.stack });
consoleLog(`Server Error ${err.status}\n${errorLogText}`);
// `headersSent` će biti TRUE ako je router kod kojeg se dogodila greška već poslao header-e
// > ako ih probam ponovo postaviti, to će baciti grešku i u ovom slučaju SRUŠITI SERVER - to ne smijemo dopustiti
if(!res.headersSent) {
res.status(err.status);
res.setHeader('Content-Type', "text/html");
res.end(`unhandled server error`);
} else {
// AKO nije pozvan `end` - pozovi ga i završi obradu zahtjeva
// ... u suprotnom će konekcija ostati otvorena do timeout-a
if(!res.writableEnded) {
res.end();
}
}
};

View File

@@ -0,0 +1,35 @@
import { RequestHandler, Router } from "express";
import { workerRunnerInfo } from "../workRunner";
import { coalesce } from "../lib/initTools";
const PULL_INTERVAL = parseInt(coalesce(process.env.PULL_INTERVAL, "10000"));
/** Maximum time between two worker jobs */
const MAX_WORKER_LATENCY = PULL_INTERVAL * 2.5;
/**
* Router koji se izvršava u slučaju grube greške koja nije obrađena nigdje prije
* @param req express request
* @param res express response
* @param next
*/
export const healthcheckRouter:RequestHandler = async (req, res, next) => {
const workerLatency = Date.now() - workerRunnerInfo.lastWorkTime;
if(workerLatency > MAX_WORKER_LATENCY) {
const msg = `No work done in ${workerLatency}ms. Last worker status = "${workerRunnerInfo.status}"`;
console.warn(msg)
res.status(500);
res.setHeader('Content-Type', 'text/plain');
res.end(msg);
} else {
res.status(200);
res.setHeader('Content-Type', 'text/plain');
res.end('OK');
}
};
export const pingRouter = Router();
pingRouter.get('/', healthcheckRouter);

View File

@@ -0,0 +1,19 @@
import { Router, NextFunction, Request, Response } from "express";
import createError from 'http-errors';
import { register } from 'prom-client';
import { createLogger } from '../lib/logger';
const logger = createLogger("server:metrics");
export const metricsRouter = Router();
metricsRouter.get('/', async (req:Request, res:Response, next:NextFunction) => {
// ne mogu dopustiti da prometheus client sruši server
try {
logger(`⚡️[server]: GET /metrics`);
res.set('Content-Type', register.contentType);
res.end(await register.metrics());
} catch(ex:any) {
next(createError(500, (ex as Error).message));
}
});

View File

@@ -0,0 +1,16 @@
import { RequestHandler, Router } from "express";
/**
* Router koji se izvršava u slučaju grube greške koja nije obrađena nigdje prije
* @param req express request
* @param res express response
* @param next
*/
export const pingRequestHandler:RequestHandler = async (req, res, next) => {
res.status(200);
res.setHeader('Content-Type', 'text/plain');
res.end('PONG');
};
export const pingRouter = Router();
pingRouter.get('/', pingRequestHandler);

View File

@@ -0,0 +1,7 @@
import { LabelValues } from "prom-client";
/** data assignet to `express.response.locals` */
export type NgitLocals = {
/** Prometheus client timer */
stopPrometheusTimer: (labels?: LabelValues<"path"|"status">) => number,
};

View File

@@ -0,0 +1,5 @@
export enum SupportedRoutes {
metricsPath='/metrics',
ping='/ping',
healthcheck='/healthcheck',
}

View File

@@ -0,0 +1,29 @@
declare global {
namespace NodeJS {
interface ProcessEnv {
/**
* (optional) environment u kojem se proces vrti
* @default undefined
* */
ENV?:"dev"|"jest"
/**
* (optional) App label to be used in Prometheus (Grafana)
* @default "email-server-worker"
* */
PROMETHEUS_APP_LABEL?: string
/**
* (optional) Prometheus histogram bucket sizes (grafana)
* @default "0.1, 0.5, 1, 5, 10"
* */
PROMETHEUS_HISTOGRAM_BUCKETS?: string
/**
* (required) Pull interval in milliseconds - how often should worker cycle run
* @default "10000"
* */
PULL_INTERVAL:string
}
}
}
export {}

View File

@@ -0,0 +1,165 @@
import { failedRequestCounter, requestDurationHistogram, successfulRequestCounter, totalRequestCounter } from "./lib/metricsCounters";
import { coalesce } from "./lib/initTools";
import { createLogger } from "./lib/logger";
import { serializeError } from "./lib/serializeError";
import { doWork } from "./exampleWorker";
/** time between two pull operations */
const PULL_INTERVAL = parseInt(coalesce(process.env.PULL_INTERVAL, "10000"));
const consoleLog = createLogger("server:server");
/** Writes entry to log */
const logWrite = (logTitle:string, logMessage:string) => {
consoleLog(`${logTitle}: ${logMessage}}`);
}
/** Writes error to log */
const logError = (ex: any) =>
logWrite(serializeError(ex), "error");
/**
* zastavica za zaustavljanje sinhronizacije
*/
let disposed:boolean = false;
/** is worker started - prevents multiple starts */
let workerStarted:boolean = false;
/** Promise which is resolved once the pending work in progress is completed */
let pendingWork:Promise<void>|undefined;
/** Worker re-run timeout */
let pendingTimeout:NodeJS.Timeout|undefined;
/** Enumeracija pojedinih statusa obrade jednog work-a */
export enum WorkerRunnerStatus {
init="init",
disposed="disposed",
beginWork="beginWork",
updatedStats1="updatedStats1",
updatedStats2="updatedStats2",
stoppedStatTimer="stoppedStatTimer",
workDone="workDone",
newIntervalScheduled="newIntervalScheduled",
currentWorkResolved="currentWorkResolved",
}
/** Info o statusu workera */
export type WorkerRunnerInfo = {
/** zadnje izvršena readnja */
status: WorkerRunnerStatus,
/** vrijeme kada je worker zadnji puta pokrenut */
lastWorkTime: number,
}
/** Info o statusu workera, koji koristi healthcheck kako bi vidio da li stvar funkcionira */
export const workerRunnerInfo:WorkerRunnerInfo = {
status: WorkerRunnerStatus.init,
lastWorkTime: Date.now()
}
export const workRunner = async () => {
pendingTimeout = undefined;
workerRunnerInfo.lastWorkTime = Date.now();
workerRunnerInfo.status = WorkerRunnerStatus.beginWork;
// AKO je modul zaustavljen
// -> nemoj se pokrenuti
if(disposed) {
workerRunnerInfo.status = WorkerRunnerStatus.disposed;
return;
}
// kreiram Promise koji omogućuje da dispose zna
// pričekati da worker završi sa poslom (ako je u tom trenutku aktivan)
pendingWork = new Promise(async (resolve) => {
try {
totalRequestCounter.inc();
const stopPrometheusTimer = requestDurationHistogram.startTimer();
workerRunnerInfo.status = WorkerRunnerStatus.updatedStats1;
try {
// ne dopuštam da stvar sruši worker
await doWork();
workerRunnerInfo.status = WorkerRunnerStatus.workDone;
// ažuriram statistiku
successfulRequestCounter.inc();
workerRunnerInfo.status = WorkerRunnerStatus.updatedStats2;
} catch(ex:any) {
// ažuriram statistiku
failedRequestCounter.inc();
logError(ex);
}
stopPrometheusTimer();
workerRunnerInfo.status = WorkerRunnerStatus.stoppedStatTimer;
} catch(ex:any) {
logError(ex);
}
// nemoj pokrenuti timer ako je worker u međuvremenu disposed
if(!disposed) {
// pull again after timeout
pendingTimeout = setTimeout(workRunner, PULL_INTERVAL);
workerRunnerInfo.status = WorkerRunnerStatus.newIntervalScheduled;
} else {
logWrite("Info", "... exiting worker loop");
}
resolve();
workerRunnerInfo.status = WorkerRunnerStatus.currentWorkResolved;
pendingWork = undefined;
});
// this is an async function which must return a promise
// > so return the promise which will be resolved once the work is done
return(pendingWork);
};
/**
* Starts the worker
*/
export const startSyncWorker = () => {
if(!workerStarted && !disposed) {
workerStarted = true;
workRunner();
logWrite("Info", "Worker Started");
};
}
/**
* Stops and disposes the worker
*/
export const disposeSyncWorker = async () => {
logWrite("Info", "Disposing worker ...");
disposed = true;
// preventing timer from trigger another work cycle
if(pendingTimeout) {
clearTimeout(pendingTimeout);
}
// IF no work is currently in progress
// > return a resolved promise
if(!pendingWork) {
return(Promise.resolve());
}
await pendingWork;
logWrite("Info", "Worker disposed!");
}
/** Ovo se koristi samo za Unit Testing */
export const reset_dispose = () => {
disposed = false;
}

View File

@@ -0,0 +1,29 @@
import { LabelValues } from "prom-client";
export class Counter {
public inc() {
}
}
export class Histogram<T extends string> {
startTimer(labels?: LabelValues<T>): (labels?: LabelValues<T>) => void {
return((labels?: LabelValues<T>) => { });
}
}
class Register {
public setDefaultLabels(labels: Object) {
}
public metrics(): Promise<string> {
return(Promise.resolve(""));
}
public get contentType() {
return("");
}
}
export const register = new Register();

View File

@@ -0,0 +1,33 @@
import { Request, Response, NextFunction } from 'express';
import { NgitLocals } from "../../src/types/NgitLocals";
interface IMockHttpContext {
reqPath?:string
headersSent?:boolean
writableEnded?:boolean
method?:string
}
export const mockHttpContext = ({reqPath="/", headersSent=false, writableEnded=false, method="GET"}:IMockHttpContext|undefined = {}) => {
const req = {
path:reqPath,
method,
url:`https://localhost${reqPath}`,
params: {},
} as unknown as Request;
const res = {
end: jest.fn(),
status: jest.fn(),
setHeader: jest.fn(),
locals: {
stopPrometheusTimer: jest.fn(),
} as unknown as NgitLocals,
headersSent,
writableEnded,
} as unknown as Response;
const next:NextFunction = jest.fn();
return({req,res,next})
}

View File

@@ -0,0 +1,118 @@
import { errorRouter } from '../../src/routes/errorRouter';
import createError from "http-errors";
import { mockHttpContext } from "../helpers/mockHttpContext";
describe("errorRouter", () => {
beforeEach(() => {
mockWrite.mockClear();
});
test("u slučaju greške 404 mora vratiti string poruku 'page not found'", async () => {
const err = createError(404)
const {req,res,next} = mockHttpContext();
await errorRouter(err, req, res, next);
expect(res.status).toHaveBeenCalledWith(404);
expect(res.setHeader).toHaveBeenCalledWith('Content-Type', "text/html");
expect(res.end).toHaveBeenCalledWith("page not found");
});
test("u slučaju greške 404 mora logirati request, response i tekst greške", async () => {
const err = createError(404)
const reqPath = "/neki-path/";
const {req,res,next} = mockHttpContext({ reqPath });
await errorRouter(err, req, res, next);
expect(res.locals.logger.info).toHaveBeenCalledWith("response", "page not found");
expect(res.locals.logger.error).toHaveBeenCalledWith(err.name, "page "+req.path+" not found");
});
test("ako su header-i već poslani, ne smiju biti poslani još jednom", async () => {
const err = createError(404)
const {req,res,next} = mockHttpContext({ headersSent:true, writableEnded:true });
await errorRouter(err, req, res, next);
expect(res.status).not.toHaveBeenCalled();
expect(res.setHeader).not.toHaveBeenCalled();
expect(res.end).not.toHaveBeenCalled();
});
test("ako NIJE već pozvana [end] metoda, treba je pozvati", async () => {
const err = createError(404)
const {req,res,next} = mockHttpContext({ headersSent:true, writableEnded:false });
await errorRouter(err, req, res, next);
expect(res.status).not.toHaveBeenCalled();
expect(res.setHeader).not.toHaveBeenCalled();
expect(res.end).toHaveBeenCalled();
});
test("mora zaustaviti Prometheus Timer", async () => {
const err = createError(404)
const {req,res,next} = mockHttpContext({ headersSent:true, writableEnded:false });
await errorRouter(err, req, res, next);
expect(res.locals.stopPrometheusTimer).toHaveBeenCalled();
});
test("u slučaju greške 500 mora vratiti string poruku 'internal server error'", async () => {
const err = createError(500)
const {req,res,next} = mockHttpContext();
await errorRouter(err, req, res, next);
expect(res.status).toHaveBeenCalledWith(500);
expect(res.setHeader).toHaveBeenCalledWith('Content-Type', "text/html");
expect(res.end).toHaveBeenCalledWith("internal server error");
});
test("u slučaju greške 400 mora vratiti string poruku 'bad request' i logirati grešku", async () => {
const errorMessage = "mock error text 1";
const err = createError(400, errorMessage);
const {req,res,next} = mockHttpContext();
await errorRouter(err, req, res, next);
expect(res.status).toHaveBeenCalledWith(400);
expect(res.setHeader).toHaveBeenCalledWith('Content-Type', "text/html");
expect(res.end).toHaveBeenCalledWith("bad request");
expect(res.locals.logger.errorwrite).toHaveBeenCalledWith(err.name, errorMessage);
});
test("u slučaju greške 401 mora vratiti string poruku 'unauthorized' i logirati grešku", async () => {
const errorMessage = "mock error text 2";
const err = createError(401, errorMessage)
const {req,res,next} = mockHttpContext();
await errorRouter(err, req, res, next);
expect(res.status).toHaveBeenCalledWith(401);
expect(res.setHeader).toHaveBeenCalledWith('Content-Type', "text/html");
expect(res.end).toHaveBeenCalledWith("unauthorized");
expect(res.locals.logger.error).toHaveBeenCalledWith(err.name, errorMessage);
});
test("u slučaju greške 403 mora vratiti string poruku 'forbidden' i logirati grešku", async () => {
const errorMessage = "mock error text 3";
const err = createError(403, errorMessage);
const {req,res,next} = mockHttpContext();
await errorRouter(err, req, res, next);
expect(res.status).toHaveBeenCalledWith(403);
expect(res.setHeader).toHaveBeenCalledWith('Content-Type', "text/html");
expect(res.end).toHaveBeenCalledWith("forbidden");
expect(res.locals.logger.error).toHaveBeenCalledWith(err.name, errorMessage);
});
});

View File

@@ -0,0 +1,36 @@
{
"compilerOptions": {
"target": "es2020", // https://github.com/microsoft/TypeScript/wiki/Node-Target-Mapping
"module": "commonjs",
"esModuleInterop": true, // solves the problem regarding the default importa vs importa *
"strict": true,
"sourceMap": true, // please do create source maps
"skipLibCheck": true, // don't verify typescript of 3rd party modules
"rootDir": "src", // root directory under which source files are located - it's subtree will be mirrored in "outDir"
"outDir": "build", // where the build files should be stored
// "baseUrl" ---- se NE SMIJE koristiti
// POJAŠNJENJE: ako zadamo "baseUrl" Intellisense će početi kod autocompletion-a (Ctrl+Space)
// umjesto relativnih insertirati apsolutni path do modula,
// a takav path nije dobar za build niti debugging
// "baseUrl": "./", // set a base directory to resolve non-absolute module names - This must be specified if "paths" is used
"paths": {
},
"plugins": [
{
// Slijedeće je namijenjeno BUILD projekta
// POJAŠNJENJE: build tadi `ttypescript`
// koji ne zna interpretirati što je podešeno pod "path"
// > to za njega rješava "typescript-transform-paths"
"transform": "typescript-transform-paths"
}
]
},
"include": ["src/**/*"], // location of files which need to be compiled
// Slijedeće je namijenjeno DEBUGGING servera u VS Code-u
// POJAŠNJENJE: kod debugginga modul se pokreće pomoću `ts-node`,
// koji ne zna sam interpretirati što je podešeno pod "paths"
// > to za njega rješava "tsconfig-paths/register"
"ts-node": {
"require": ["tsconfig-paths/register"]
},
}