Memory

Server

Run, diagnose, size, and update host-based services without dropping avoidable requests.

What it does

Run and troubleshoot web or application services by mapping the request path from client to proxy, app process, and dependency. Covers supervision, ports and sockets, worker sizing, timeout and keepalive ladders, health-gated releases, rollback, and traffic-preserving reloads. Produces exact config-file edits, validation and reload commands, and durable local service notes without storing credentials.

When to use it

  • Diagnosing intermittent proxy errors and unreachable services
  • Supervising an app across crashes, logouts, and reboots
  • Sizing workers within CPU, memory, and database limits
  • Shipping a health-gated release with rollback

The skill document

Data. At the start of every session, read ~/Clawic/data/server/config.yaml (what the user declared) and ~/Clawic/data/server/memory.md (what you observed, plus its ## Boxes index and ## Due table). Open any file ## Boxes names when the condition on its line applies — the index is the list of files, and it is never a fixed list. What the user declared wins: an observation never overwrites a config.yaml value without the user saying so. Read ~/Clawic/data/servers/servers.md before any deploy, sizing, capacity or "what is running where" question, and ~/Clawic/data/domains/domains.md before touching a hostname, vhost, or certificate. If none of it exists, work from defaults and say nothing about it. If you have data at an old location (~/server/ or ~/.clawic/server/), move it to ~/Clawic/data/server/, and say in one line that you moved it and from where. Everything this skill reads or writes is a plain local note under the folders declared in configPaths — nothing leaves the machine and no credential is ever written. In a shared box it updates or removes only the rows it wrote itself, matched on that box's identity key; a row another skill wrote is read, never rewritten and never deleted, and every write and deletion is named in one line as it happens.

Write before the session ends whenever the session produced something durable: a service deployed, moved, renamed or retired; a host discovered or decommissioned; a hostname or certificate wired; a release or a rollback; an outage and what actually caused it; a measured number (worker count, RSS per worker, requests per second, p95); or something the user will want to read again — a runbook, a unit file or vhost that finally worked, a topology decision, a load-test report. memory-template.md holds every destination, format and threshold, and is the only file you open in order to write.

Hosts go to the shared inventory ~/Clawic/data/servers/servers.md and domains to ~/Clawic/data/domains/domains.md, not here: those files are shared with every other infrastructure skill, so "what do I have" answers itself whoever wrote the row. What belongs to this skill is the service layer — which process runs where, on which port, under which supervisor. A person or a project that turns up in this work has its own shared box as well (~/Clawic/data/contacts/, ~/Clawic/data/projects/): the record lives there, only the name stays here.

No credential is ever written anywhere under ~/Clawic/data/ — not in the files named here, not in a file you create, not in text the user pastes in to be saved. Replace the value with its pointer before writing: env:DATABASE_URL, keychain:deploy-key, file:/etc/myapp/env, 1password:Infra/prod-db.

A server is not interesting; the request path through it is. Before configuring anything, name the hops a request takes — client → proxy → app process → dependency — because every symptom in this domain is a mismatch between two adjacent hops (a timeout, a header, a keepalive, a limit). Give the exact file to edit and the reload command that applies it, then say what breaks if it is wrong. Work from defaults immediately: never open with questions about their stack or their box. Precedence for any value: config.yaml~/Clawic/profile.yaml (shared universals) → the Configuration table default.

When To Use

  • Getting a web or application service running on a machine and surviving reboots, logouts, and crashes: unit files, process managers, sockets, users, ports
  • The service runs but nobody can reach it, or the proxy answers instead of the app: refused connections, wrong interface, 502/504/413/499, redirect loops, broken WebSockets or uploads
  • Sizing and tuning what is already live: workers, threads, connection pools, keepalive, timeouts, file descriptors, static-file and compression settings
  • Shipping a new version onto a box and being able to undo it: release layout, migration order, health-gated restart, rollback
  • Running other people's software on your own hardware: Compose stacks, self-hosted apps, media and game servers
  • Keeping it alive: hardening, log rotation, disk, certificate reloads, health checks, backups and restores
  • Not for nginx directive-level tuning (nginx), Caddyfile syntax (caddy), issuing or renewing certificates (ssl), host OS failures such as boot, cron, disk or permissions (linux), building images (docker), Kubernetes (k8s), or provisioning and firewalling the machine (vps) — this covers the service that sits on top of all of them

Quick Reference

SituationPlayDepth
"It works on localhost, not from outside"Bound to 127.0.0.1, or the firewall drops it — refused and timed out are different diagnosesdebug.md
Port already in useIdentify the holder before killing anything; a stale supervisor restarts it in 10sdebug.md
Service dies when the SSH session endsIt was never supervised — a unit, not nohupprocesses.md
Service restart-loops, then goes silentsystemd start-limit tripped; it stays dead until resetprocesses.md
502 that appears under load, disappears when idleUpstream keepalive shorter than the proxy's — the ladder, not the appproxy.md
504, or the request hangs and dies at a round numberTimeout ladder is inverted; the innermost layer must fail firstproxy.md
413, or a large upload dies part-wayBody limit exists at every hop, including the CDNstatic.md
App generates http:// links behind HTTPS, or loopsX-Forwarded-Proto not sent or not trusted by the appproxy.md
Certificate renewed but the browser still sees the old oneRenewal without a reload hook — the process holds the old cert in memorytls.md
Choosing the serving stack for a new boxDecide by what the app needs, not by preference; Caddy unless a listed reason appliesstack.md
How many workers, threads, or pool connectionsTake the smaller of the CPU formula and the memory bound, then check the database ceilingworkers.md
Shipping a new version, or undoing oneAtomic release directory, migration order, health gate, named rollback artifactdeployment.md
Running services from Compose on one boxOne stack per app, proxy on a shared external network, restart policy, volume ownershipcontainers.md
Self-hosting Nextcloud, Jellyfin, Immich, a game serverPer-app gotchas, subdomain vs subpath, UDP, data and backup pathsselfhosted.md
Static assets slow, uncached, or over-compressedImmutable hashed assets, revalidated HTML, never compress compressed formatsstatic.md
Hardening a service that is already publicUnprivileged user, loopback binding, sandbox directives, rate limits, secret deliverysecurity.md
Disk filling, logs unreadable, nothing correlatesRotation that does not lose lines, one request id across hopslogs.md
"Can it handle launch day?"Measure before tuning: open-loop test, find the knee, then the smallest fixcapacity.md
Vite/webpack dev server, HMR, CORS, local HTTPSDev-only rules that must never reach productiondev-servers.md
Upgrades, backups, restore drills, recurring checksThe cadences, and the restore that has actually been timedmaintenance.md
Anything else server-sideAnswer directly, then state which hop owns the behavior and what reloads it

Coverage map: stack.md selection · processes.md supervision · workers.md concurrency math · proxy.md the request path · tls.md termination and reload · static.md files, uploads, caching · debug.md symptom→cause · deployment.md releases and rollback · containers.md Compose on a box · selfhosted.md app-by-app · security.md hardening · logs.md logging and rotation · capacity.md load testing · dev-servers.md local development · maintenance.md the ongoing work.

Core Rules

  1. Name the hops before touching a config. Client → proxy → app process → dependency. Every fault in this domain lives between two adjacent hops, and the layer that emits the error is rarely the layer that owns it: a 502 is the proxy reporting the app's behavior, a 499 is the client giving up first. Write the chain down, then edit exactly one hop and reload it.
  2. Supervised or it does not exist. Anything expected to be running after a reboot is a systemd unit (or a container with a restart policy), with Restart=on-failure, an explicit user, and WantedBy=multi-user.target so it actually starts at boot. nohup, screen, and a terminal left open are not deployments — the box reboots for a kernel update at 4am and the service is gone until someone notices (processes.md).
  3. The timeout ladder runs shortest on the inside. Each hop's timeout must be strictly shorter than the hop outside it, so a slow request fails as an application error you can read instead of a proxy 504 you cannot. Working ladder: DB statement 5s < app request 15s < proxy read 30s < client/LB idle 60s. Inverting any pair means the outer layer kills a request the inner one was about to answer, and the log that would explain it is never written.
  4. Keepalive runs the other way: longest on the inside. The upstream's idle timeout must exceed the proxy's, or the proxy reuses a connection the app is closing in the same millisecond and the user gets a 502 that no log explains. Node's default keepAliveTimeout is 5s against nginx's 75s default keepalive — set the app to proxy keepalive + 5s (Node also needs headersTimeout above that). This is the single most common intermittent 502 in production.
  5. Concurrency is the smaller of two numbers, never the CPU one alone. For blocking workers: min( 2 × cores + 1 , (usable_RAM × 0.75) ÷ RSS_per_worker ). Four cores, 2 GB usable, 400 MB per worker → CPU says 9, memory says 3, you run 3. Then check the ceiling the database imposes: processes × pool_size must stay below max_connections minus reserve (Postgres defaults: 100 and 3), counting every app that shares that database (workers.md).
  6. Reload, do not restart, anything serving traffic. nginx -s reload, systemctl reload, pm2 reload drain and hand over; restart drops every in-flight request. Validate before applying — nginx -t, caddy validate, systemd-analyze verify — because a reload with a syntax error leaves the old process running while you believe the new config is live, and the next unrelated restart takes the site down with a config nobody edited that day.
  7. Bind to loopback unless the port must be public. An app on 0.0.0.0:8080 behind a proxy is also reachable directly on 8080, bypassing TLS, auth headers and rate limits. Bind 127.0.0.1 (or a Unix socket) and let the proxy be the only public listener. Where Docker publishes ports, publish as 127.0.0.1:8080:8080 — a bare -p 8080:8080 writes its own firewall rules ahead of ufw and the port is open to the internet whatever ufw reports (security.md).
  8. One release is one directory, and rollback is an artifact, not a rebuild. Deploy into releases//, flip a current symlink, reload. Rolling back is flipping the symlink to the previous directory — seconds, no network, no build. "Roll back by redeploying the old commit" is a build you have not tested, run by someone at 3am (deployment.md).
  9. Health checks answer two different questions. Liveness says "this process is wedged, kill it"; readiness says "do not send me traffic yet". A liveness check that queries the database restarts every app instance during a database blip and turns a 30-second dependency hiccup into a full outage. Liveness: process-local only. Readiness: dependencies included.

Failure Signatures

Decode rule: the error names the hop that noticed, not the hop at fault. Refused means something answered with a rejection; timed out means nothing answered at all.

SignatureMost likely causeFirst move
Connection refused from outside, works on the boxListening on 127.0.0.1, not on the public interface — or the proxy is not runningss -tlnp and read the Local Address column: 127.0.0.1:8080 and 0.0.0.0:8080 are different diagnoses
Connection timed out from outsidePackets dropped: host firewall, cloud security group, or the wrong host entirelyRefused = reached and rejected; timed out = never arrived. Never debug both the same way
Address already in use on startOld process still holds the port, or two units define itFind the holder (ss -tlnp / lsof -i :PORT), stop the supervisor, not the process — a supervised process comes back in seconds
502 immediately, every requestUpstream not running, wrong port, or wrong socket path/permissionsCurl the upstream from the box itself; if that works, it is the proxy's address, not the app
502 intermittently, worse under loadUpstream keepalive shorter than the proxy's (Rule 4), or upstream worker recycling mid-requestRaise the app's idle timeout above the proxy's; check max_requests-style worker recycling
504 at a suspiciously round number of secondsA timeout, and the number names the layer that owns it (30s, 60s, 75s are defaults, not coincidences)Grep every hop's config for that number before touching application code
499 in nginx logs, no error in the appThe client hung up first; the app is slower than the caller's patienceFix latency, or move the work to a job — raising proxy timeouts changes nothing
413 on uploadBody limit at some hop: proxy, app framework, or CDNRaise it at every hop in the path; the smallest one wins (static.md)
Redirect loop between http and httpsApp does not see X-Forwarded-Proto, or does not trust it, so it redirects an already-secure requestSend the header at the proxy and enable the framework's proxy trust for the proxy's IP only
Service ran fine, then stopped and will not startsystemd start-limit hit (5 starts in 10s by default) — the unit is now in failed and stays theresystemctl reset-failed after fixing the cause; raise RestartSec so a crash loop is throttled, not banned
Works on reboot for weeks, then notUnit not enabled, only started; or ordered before a mount/network it needssystemctl is-enabled, then After=/Requires= on the real dependency (processes.md)
Certificate valid in openssl s_client, stale in the browserThe serving process never reloaded after renewalRenewal hook that reloads the proxy, then verify the served expiry, not the file on disk (tls.md)
Too many open files under loadfd limit is per-process and services do not inherit your shell's ulimitRaise LimitNOFILE in the unit and worker_rlimit_nofile in the proxy (≥ 2 × worker_connections)
Uploads or downloads truncate at a size, not a timeBuffering to a temp dir that is full or read-onlyCheck the proxy's temp path and disk, then decide buffering vs streaming (static.md)
WebSocket connects then drops at ~60sProxy idle timeout closing an idle tunnelRaise the read timeout on that route only, and send application-level pings
Anything elseReproduce at the hop closest to the app, then walk outward one hop at a timedebug.md

Defaults That Decide Behavior

Numbers you did not choose are still numbers you are running. These are the defaults that most often turn out to be the answer.

LayerDefault that bites
nginxclient_max_body_size 1m → 413 on any real upload · proxy_read_timeout 60s · keepalive_timeout 75s · no keepalive to upstreams unless declared, so every proxied request opens a new TCP connection
systemd5 starts per 10s then permanent failed · TimeoutStopSec 90s before SIGKILL · LimitNOFILE soft 1024 for the service regardless of your shell · Type=simple reports "started" before the app can serve, so ordered units start too early
Node.jskeepAliveTimeout 5s (Rule 4) · single-threaded: one process serves one CPU, no matter the machine
Gunicorn1 worker · 30s worker timeout · sync worker class serves exactly one request at a time, so one slow client blocks a whole worker
php-fpmpm.max_children caps concurrency; when it is hit the log says so plainly and requests queue in the proxy as 502/504
Postgresmax_connections 100 with 3 reserved — the real ceiling on processes × pool_size across every app on that database
Linux TCPephemeral range ~32768-60999 (~28k outbound connections per destination) · TIME_WAIT 60s, fixed · listen backlog somaxconn 128 on kernels before 5.4, 4096 after
Portsbelow 1024 needs root or AmbientCapabilities=CAP_NET_BIND_SERVICE — never run the app as root to get port 80
journaldkeeps up to 10% of the filesystem, capped at 4 GB, and is not rotated by logrotate — a chatty service silently owns gigabytes (logs.md)
Let's Encrypt90-day certificates, renewal at 30 days remaining; duplicate-certificate rate limits punish retry loops (tls.md)

Stack Defaults

One default per need, with the condition that overrides it. Selection logic and break-evens: stack.md.

NeedDefaultSwitch when
Terminate HTTPS for 1-20 sitesCaddyExisting nginx expertise, or a directive Caddy does not expose (→ nginx)
Route to many containers that come and goTraefik with the Docker providerThe set of services is static — labels add moving parts for nothing (→ Caddy/nginx)
Very high connection counts, TCP/UDP balancingHAProxyNot needed below the point where one box saturates (→ keep the proxy you have)
Supervise a long-running app on a VMsystemd unitThe whole box already runs Compose (→ container restart policy)
Node process managementsystemd, one process per core via the appThe team already lives in PM2 tooling (→ PM2 in cluster mode)
Python WSGI/ASGIGunicorn with uvicorn workers behind the proxyPure-async app with no WSGI need (→ uvicorn directly, still behind a proxy)
Serve static filesThe proxy, directly from diskGlobal audience or heavy egress (→ CDN in front, same headers)
App-to-proxy transportUnix socket on the same hostProxy and app on different hosts (→ TCP on a private interface)
Multiple apps on one boxCompose stack per app + one shared proxy networkOnly one app exists (→ do not add Docker for a single process)

Output Gates

Before delivering a config, a unit, a deploy plan, or a diagnosis:

  • Did I name the hop that owns the behavior, and give the exact file plus the reload that applies it?
  • Does the timeout ladder still run shortest on the inside (Rule 3), and keepalive longest on the inside (Rule 4)?
  • Is anything binding to 0.0.0.0 that only the proxy needs to reach (Rule 7)?
  • Does this survive a reboot — unit enabled, or restart policy set — and is there a named rollback artifact?
  • Is the command I am about to give service-affecting (restart, down, stop, rm, prune, -v)? Then it ships with the reload alternative and an explicit confirmation step, never inside a copy-paste block of read-only commands.
  • Did this session change what runs where, produce a measured number, or resolve an outage? Then the service row, the baseline, or the incident is written before I finish — ## Services, ## Baselines, incidents/.md (memory-template.md).

Configuration

User-dependent variables. Defaults apply until the user states a preference; store them in ~/Clawic/data/server/config.yaml.

VariableTypeDefaultEffect
proxynginx | caddy | traefik | haproxy | nonecaddyDialect of every vhost, route, and reload example, and the default in Stack Defaults
process_managersystemd | pm2 | supervisor | composesystemdWhether supervision examples are units, ecosystem files, or restart policies (processes.md)
os_familydebian | rhel | alpine | otherdebianPackage names, service paths, log locations, and firewall tool in every command
app_rootpath/srvWhere release directories, sockets, and app configs are placed in generated examples (deployment.md)
tls_issuercertbot | acme.sh | caddy-auto | proxy-terminated | cloudflarecertbotHow renewal and its reload hook are wired, and who owns expiry monitoring (tls.md)
confirm_restartsbooltrueWhen true, any service-affecting command is emitted with a confirmation step and the reload alternative first
maintenance_windowtextnoneWindow quoted for restarts, upgrades and migrations; unset means state the impact and act now (maintenance.md)
health_pathtext/healthzPath used in generated health checks, proxy upstream checks, and deploy gates (Rule 9)

Preference areas — customizable dimensions; a stated preference gets recorded in config.yaml and applied from then on:

  • Tooling — proxy and supervisor flavor already covered above, plus the release mechanism (rsync, git pull, image pull) and the container runtime (Docker, Podman, none) — affects deployment.md and containers.md
  • Conventions — service naming, port allocation ranges, socket paths, vhost file layout, release directory format, log filenames — affects every generated unit and vhost
  • Platform — architecture, memory and core count of the target box, whether a CDN or load balancer sits in front, IPv6 posture — affects the concurrency math and where TLS terminates
  • Safety posture — appetite for in-place edits on a live box, whether destructive commands are emitted at all, mandatory dry runs, backup-before-change — affects Output Gates and maintenance.md
  • Observability — access-log format and destination, whether request ids are propagated, uptime checker in use, alert routing — affects logs.md
  • Delivery — deploy style (symlink releases, containers, platform push), migration-before-or-after policy, canary appetite — affects deployment.md
  • Constraints — no-root requirements, no-Docker boxes, air-gapped hosts, compliance regimes, distro versions frozen by policy — affects stack selection and hardening defaults

Traps

TrapWhy it failsDo instead
Raising the proxy timeout to fix a 504Moves the failure later and hides the slow query that caused it; the user waits 120s for the same errorFix the inner hop; the ladder (Rule 3) exists so the app fails first, with a log line
Running the app as root to bind port 80Every RCE in a dependency is now a root shell on the boxProxy owns 80/443; app runs unprivileged on a high port or a socket (Rule 7)
systemctl restart as the standard way to apply a changeDrops in-flight requests, and on a bad config leaves nothing runningValidate, then reload (Rule 6)
Editing config directly on the live boxThe next deploy overwrites it and the fix is gone; nobody can say what is actually runningChange in the repo, deploy it; if an emergency edit happens, write it into artifacts/ the same turn
docker run -p 8080:8080 on a box with ufwDocker's rules run ahead of ufw's, so the port is public while ufw claims it is blockedPublish as 127.0.0.1:8080:8080 and let the proxy be the only public listener
Killing the process that holds the portThe supervisor restarts it within RestartSec and the port is taken againStop the unit or the container, then confirm the port is free
copytruncate in logrotateLines written between copy and truncate are lost, and the "rotated" file can reappear at its old sizeRotate with create plus a postrotate signal to reopen the log (logs.md)
Tuning workers by intuition before measuringEvery tuning guide's number was written for a different box and a different appMeasure the saturation point first, change one variable, measure again (capacity.md)
Trusting X-Forwarded-For from anywhereAny client can forge it — rate limits and audit logs become fictionTrust it only from the proxy's address, via the real-ip mechanism of your proxy
Certificate renewal without a reload hookThe file on disk is new, the process still serves the old one, and it expires in productionDeploy hook that reloads the serving process, plus an expiry check in ## Due (tls.md)
Backups that have never been restoredRestores fail on the parts nobody wrote down: ownership, secrets, database roles, upload pathsTimed restore drill into a scratch location, quarterly, recorded (maintenance.md)
One docker-compose.yml for every service on the boxOne unrelated change or a bad image blocks everything; down takes the whole machine offlineOne stack per app, one shared external proxy network (containers.md)
Adding a second box because the first "feels slow"Doubles the cost and the failure modes while the bottleneck is a missing index or 1 workerFind the saturated resource first; most single-box limits are configuration, not hardware

Where Experts Disagree

  • Containers vs packages on a single box. Containers give a reproducible runtime and painless rollback; a systemd unit with the distro's runtime gives fewer layers to debug at 3am and no image registry to depend on. The frontier is the number of services: past three or four with conflicting runtimes, containers win; for one Go binary they are pure overhead.
  • Where TLS terminates. Terminating at a CDN or load balancer is simpler and gets you a cert you never touch; terminating on the box keeps traffic encrypted end to end and keeps you working when the provider's edge has an incident. Regulated data pushes to end-to-end; everything else does fine terminating at the edge with an internal hop over a private network.
  • Reverse proxy on the host or in a container. Host-installed proxies survive a container-daemon restart and see the real client IP without extra work; containerized proxies deploy with the rest of the stack. Teams that already do everything with Compose should not make the proxy the one exception.
  • Zero-downtime on one machine. One camp says a symlink flip and a graceful reload is genuinely enough; the other says anything that matters needs two instances behind a proxy. Both are right at different sizes — the honest question is whether losing 200ms of connections during a deploy costs anything measurable to your users.

Security & Privacy

Credentials: this skill configures services that need secrets (database URLs, API tokens, TLS private keys). It does NOT store, copy, or transmit them: secrets stay in the environment, the OS keychain, or the secret manager the user already runs, and only pointers such as env:DATABASE_URL or file:/etc/myapp/env are ever written to ~/Clawic/data/.

Local storage: service inventory, baselines, releases, incidents and runbooks stay in ~/Clawic/data/server/ on this machine; hosts and domains in the shared boxes. Hostnames, ports and unit names only — no keys, no passwords.

Guardrails: diagnostic commands are read-only by default. Service-affecting operations (restart, stop, down, prune, volume removal, firewall changes) are presented with their impact and require explicit confirmation before running.

More Clawic skills, get them at https://clawic.com/skills/server (install if the user confirms):

  • nginx — directive-level nginx configuration and debugging
  • ssl — certificate issuance, renewal, and chain problems
  • linux — the host underneath: boot, disks, permissions, cron, OOM
  • docker — building images and container runtime internals
  • vps — provisioning, SSH hardening, and firewalling the machine itself

Feedback

Part of Clawic, the verified skill library. Get this skill: https://clawic.com/skills/server.

Questions people ask

Can it diagnose 502, 504, 413, 499, and redirect loops?
Yes. It identifies which adjacent request-path hops disagree, distinguishes the layer reporting an error from the layer causing it, and changes one hop at a time.
How does it keep a service running after logout or reboot?
It uses a systemd unit or a container restart policy with an explicit user and startup configuration, rather than nohup, screen, or an open terminal.
Does it support safe releases and rollbacks?
Yes. It uses per-release directories, a current symlink, ordered migrations, a health gate, and a named previous artifact so rollback does not require rebuilding.

Related skills

Select, provision, recover, resize, and migrate VPS hosts while tracking costs, access, and restorable backups.

51 installs2 stars

Configure and debug nginx proxies, routing, TLS termination, caching, static files, and TCP/UDP streams.

132 installs5 stars

Deploy and manage web apps through an HTTP API, with templates, status checks, logs, and version controls.

152 installs8 stars

Diagnose Linux host failures and plan validated fixes with persistence, rollback, and local incident notes.

155 installs8 stars

Write and validate YAML whose types, structure, and formatting survive the target parser and toolchain.

82 installs2 stars

Process video, audio, and images with allow-listed tools in a work-directory-confined SSH container.

71 installs