> For the complete documentation index, see [llms.txt](https://docs.hivel.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.hivel.ai/self-managed-hivel-deployment/on-premises-generic/hivel-on-premises-deployment-guide-generic/troubleshooting.md). # Troubleshooting Each issue below includes symptom, likely causes, diagnostic steps, resolution, verification, and information to collect before contacting Hivel support. ### Services Not Starting **Symptom:** One or more containers fail to start or exit immediately **Likely causes:** * Missing Docker network * A dependent service (Redis, PostgreSQL) not yet healthy * Invalid value in `.env` **Diagnostic commands:** ``` docker logs docker ps -a docker network ls | grep microservices_network ./deploy.sh --status ``` **Resolution:** 1. Create the missing network if absent: `docker network create microservices_network` 2. Fix any invalid `.env` value that `docker logs` points to 3. Re-run: `./deploy.sh --service ` **Verify the fix:** ```bash docker ps ./deploy.sh --health ``` **Success:** Service shows as `Up` and health check returns healthy **Collect for Hivel support:** Full `docker logs ` output, `docker ps -a`, and `.env` file (with all passwords redacted) ### Database Connection Issues **Symptom:** Services cannot connect to PostgreSQL; psql fails with "connection refused" or authentication error **Likely causes:** * Wrong credentials in `.env` * PostgreSQL not accepting remote connections * Firewall blocking port 5432 **Diagnostic commands:** ``` psql -h $DB_HOST -U $DB_USER -d $DB_NAME cat config/.env | grep DB_ ``` **Resolution:** 1. Confirm `pg_hba.conf` allows access from the host 2. Confirm firewall allows port 5432 from the host 3. Verify `DB_USER` and `DB_PASSWORD` in `.env` match PostgreSQL **Verify the fix:** ```bash psql -h $DB_HOST -U $DB_USER -d $DB_NAME ``` **Success:** psql prompt opens with no error **Collect for Hivel support:** Exact psql error text, and confirmation of which credential phase ([Environment Configuration](https://docs.hivel.ai/self-managed-hivel-deployment/on-premises/environment-configuration)) is in `.env` (Phase 1 = superuser, Phase 2 = application user) ### Registry Pull Failed **Symptom:** Docker pull from ECR fails; "authentication failed" or "invalid credentials" error **Likely causes:** * Expired or incorrect AWS credentials * Credentials not re-authenticated after expiry **Diagnostic commands:** ``` aws ecr get-login-password --region ap-south-1 | \ docker login --username AWS --password-stdin 730335373269.dkr.ecr.ap-south-1.amazonaws.com aws sts get-caller-identity ``` **Resolution:** 1. Re-run `aws configure` with the credentials Hivel provided 2. Re-authenticate to ECR using the command above **Verify the fix:** ```bash docker login ``` **Success:** Output shows `Login Succeeded` **Collect for Hivel support:** Exact error from `docker login`, and output of `aws sts get-caller-identity` (account ID only - redact sensitive data) ### `./deploy.sh` Not Found **Symptom:** "command not found" or "no such file" when running `./deploy.sh` **Likely causes:** * Not running from the correct directory * Installer package never downloaded or unzipped * `deploy.sh` exists but is not executable **Diagnostic commands:** {% code expandable="true" %} ``` cd /opt/hivel-onprem ls -la deploy.sh pwd ``` {% endcode %} **Resolution:** 1. Navigate to `/opt/hivel-onprem`: `cd /opt/hivel-onprem` 2. If `deploy.sh` is missing entirely, re-download and re-extract the installer package (see [Project Setup](https://docs.hivel.ai/self-managed-hivel-deployment/on-premises/project-setup)) 3. If `deploy.sh` exists but is not executable, run: `chmod +x deploy.sh` **Verify the fix:** ```bash ./deploy.sh --help ``` **Success:** Command reference prints with no "not found" error **Collect for Hivel support:** Output of `pwd` and `ls -la` in the installer directory, and confirmation that `hivel-onprem` was successfully downloaded and unzipped ### Migration Failed **Symptom:** Flyway migration exits with non-zero code; schema not initialized **Likely causes:** * `.env` contains application-user credentials instead of superuser credentials * `insightly` database does not exist **Diagnostic commands:** ``` docker logs flyway-migration psql -h $DB_HOST -U $DB_USER -d postgres -c "\l" | grep insightly cat config/.env | grep DB_ ``` **Resolution:** 1. Confirm superuser (not application-user) credentials are in `.env` for migration (see [Environment Configuration](https://docs.hivel.ai/self-managed-hivel-deployment/on-premises/environment-configuration) Phase 1) 2. Confirm the `insightly` database exists (see [Database Setup](https://docs.hivel.ai/self-managed-hivel-deployment/on-premises/database-setup-self-managed-postgresql)) 3. Re-run migration: `./deploy.sh --service flyway-migration` **Verify the fix:** ```bash docker inspect flyway-migration --format='{{.State.ExitCode}}' ``` **Success:** Exit code is `0` **Collect for Hivel support:** Full `docker logs flyway-migration` output ### Wrong Credentials After Migration **Symptom:** Services fail to authenticate after migration completes; connection errors appear in service logs **Likely cause:** `.env` was never switched from superuser to application-user credentials after migration succeeded **Diagnostic command:** ``` cat config/.env | grep DB_USER # should show ``` **Resolution:** 1. Update `.env` to Phase 2 credentials (see [Environment Configuration](https://docs.hivel.ai/self-managed-hivel-deployment/on-premises/environment-configuration)) 2. Restart services: `./deploy.sh --restart` **Verify the fix:** ```bash ./deploy.sh --health ``` **Success:** All services report healthy **Collect for Hivel support:** Service logs showing the original authentication failure ### Port Already in Use **Symptom:** "Address already in use" or "bind failed" when starting services on port 80 or 443 **Diagnostic command:** ``` sudo lsof -i :80 sudo kill -9 ``` **Resolution:** 1. Identify the process occupying the port (PID from `lsof` output) 2. Stop the process: `sudo kill -9 ` 3. Redeploy: `./deploy.sh --vm` ### Health Checks Failing **Symptom:** Services show as `Up` but health checks fail; services are unstable or slow to start **Likely causes:** * Service needs more startup time * Host is resource-constrained **Diagnostic commands (allow 2–3 minutes on first start, then check):** ``` ./deploy.sh --logs docker stats free -h ``` **Resolution:** 1. Allow 2–3 minutes for initial startup 2. If resource-constrained, confirm host meets [Server Infrastructure Setup](https://docs.hivel.ai/self-managed-hivel-deployment/on-premises/server-infrastructure-setup) sizing 3. Inspect logs for specific errors: `./deploy.sh --logs ` ### Log rotation not working **Symptom:** Log files not rotating; `/var/log/hivel/` directory is missing or empty; cron job not present **Diagnostic commands:** ``` sudo crontab -l ls -la /var/log/hivel/ sudo ./scripts/docker-log-rotate.sh chmod +x scripts/docker-log-rotate.sh ``` **Resolution:** Re-run the setup script: bash ```bash sudo ./scripts/setup-log-rotation.sh ``` **Success:** Cron job appears in `crontab -l` output; log directory contains subdirectories per service **Collect for Hivel support:** Output of `sudo crontab -l` and `ls -la /var/log/hivel/`
## Need More Help? If none of these steps resolve your issue: 1. Collect the diagnostics listed for your issue 2. Redact all sensitive data (passwords, tokens, private IPs) 3. Contact with your findings {% columns %} {% column %} Back to File Locations Quick Reference {% endcolumn %} {% column %} {% endcolumn %} {% column %} Backup and Restore {% endcolumn %} {% endcolumns %}