> For the complete documentation index, see [llms.txt](https://docs.hivel.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.hivel.ai/archive/generic/hivel-on-premises-deployment-guide-generic/11.-troubleshooting.md).

# 11. Troubleshooting

On-Prem Deployment Guide (Generic)

Each entry below follows the same format: symptom, likely causes, diagnostic commands, resolution, how to confirm the fix, and what to collect if you still need to escalate to Hivel support.

### Services Not Starting

**Symptom:** one or more containers fail to start, or docker ps shows a service missing/exited unexpectedly.

**Likely causes:** the shared Docker network doesn't exist yet, a dependent service (e.g. Redis, PostgreSQL) isn't healthy yet, or a bad value in .env.

**Diagnostic commands:**

```
# Check container logs
docker logs <service-name>

# Check Docker status
docker ps -a

# Verify network exists
docker network ls | grep microservices_network

# Check service dependencies
./deploy.sh --status
```

**Resolution:** create the missing network with docker network create microservices\_network if absent; fix whatever .env value the logs point to; re-run `./deploy.sh --service <name>`.

**How to verify the fix:** docker ps shows the service as Up; `./deploy.sh --health <name>` reports healthy.

**What to collect for Hivel support:** full docker logs \<service-name> output, `docker ps -a`, and your .env with all passwords redacted.

### Database Connection Issues

**Symptom:** services fail to start, or psql cannot connect, with a connection or authentication error.

**Likely causes:** wrong credentials in .env, the database's network rules don't allow access from VM1/VM2, or the database isn't running.

**Diagnostic commands:**

```
# Test connection from host
psql -h $DB_HOST -U $DB_USER -d $DB_NAME

# Check .env file
cat config/.env | grep DB_

# Verify the database's network rules allow access from your servers
# Check the database is reachable from VM1/VM2
```

**Resolution:** confirm the database's firewall/security group allows both VM1's and VM2's addresses (Database Setup, Step 2); confirm DB\_USER/DB\_PASSWORD in .env match what's actually set on the database.

**How to verify the fix:** the psql command above returns a prompt with no error.

**What to collect for Hivel support:** the exact psql error text, and confirmation of which credential phase (root vs. application user - see Environment Configuration) is currently in .env.

### ECR Pull Failed

**Symptom:** docker login to Hivel's ECR fails, or ./deploy.sh reports an image pull error.

**Likely causes:** AWS credentials expired, mistyped, or not the credential set Hivel provided for ECR access.

**Diagnostic commands:**

```
# Re-authenticate with ECR
aws ecr get-login-password --region ap-south-1 | \
  docker login --username AWS --password-stdin \
  730335373269.dkr.ecr.ap-south-1.amazonaws.com

# Verify AWS credentials
aws sts get-caller-identity

# Check ECR permissions
aws ecr describe-repositories --region ap-south-1
```

**Resolution:** re-run aws configure with the credentials Hivel provided; re-authenticate to ECR using the command above.

**How to verify the fix:** docker login returns Login Succeeded.

**What to collect for Hivel support:** the exact error text from docker login, and the output of aws sts get-caller-identity (account ID only - redact anything sensitive).

### ./deploy.sh not found

**Likely causes:** not in the right path (deploy.sh only exists inside the extracted installer directory), or the installer package was never fully downloaded/unzipped, or deploy.sh exists but isn't executable.

**Diagnostic commands:**

{% code expandable="true" %}

```
cd /opt/docker-onprem-installer
ls -la deploy.sh
pwd
```

{% endcode %}

**Resolution:** cd into /opt/docker-onprem-installer before running the command - it must be run from inside that directory, not from /opt or elsewhere. If deploy.sh is missing entirely, re-download and re-extract the installer package (see [6. Environment Configuration](/archive/generic/hivel-on-premises-deployment-guide-generic/6.-environment-configuration.md)). If it's present but not executable, run `chmod +x deploy.sh`.

**Verify the fix:** `./deploy.sh --help` runs and prints the command reference with no "not found" error.

**Collect for Hivel support:** the output of pwd and ls -la in the installer directory, and confirmation that the docker-onprem-installer package was successfully downloaded and unzipped.

### Migration Failed

**Symptom:** flyway-migration exits with a non-zero code, or logs show errors instead of "Successfully applied X migration(s)".

**Likely causes:** .env has application-user credentials instead of root/master credentials, or the "insightly" database doesn't exist yet.

**Diagnostic commands:**

```
# Check migration logs
docker logs flyway-migration

# Verify database exists
psql -h $DB_HOST -U $DB_USER -d postgres -c "\l" | grep insightly

# Check .env configuration (should have root credentials)
cat config/.env | grep DB_

# Verify you're using root/master credentials for migration
# If using a secrets manager, retrieve credentials, e.g.:
aws secretsmanager get-secret-value --secret-id <your-secret-name> --query SecretString --output text

# Test connection with root credentials
psql -h $DB_HOST -U $DB_USER -d $DB_NAME
```

**Resolution:** confirm root/master (not application-user) credentials are in .env for migration (Environment Configuration, Phase 1); confirm the "insightly" database exists (Database Setup, Step 3).

**How to verify the fix:** docker inspect `flyway-migration --format='{{.State.ExitCode}}'` returns 0.

**What to collect for Hivel support:** the full docker logs flyway-migration output.

**Warning:** Do not proceed until migration is successful.

### Wrong Credentials After Migration

**Symptom:** services fail to connect to the database after migration completes, even though migration itself succeeded.

**Likely causes:** .env was never switched from root/master credentials to the application user (\<user>) after migration.

**Diagnostic commands:**

```
# Verify .env has application user credentials (not root)
cat config/.env | grep DB_USER
# Should show: DB_USER=<user>
```

**Resolution:**

```
# If still using root credentials, update to application user
nano config/.env
# Change:
# DB_USER=<user>
# DB_PASSWORD='<password>'  # or your custom password

# Restart services
./deploy.sh --restart
```

**How to verify the fix:** affected services report healthy via `./deploy.sh --health`.

**What to collect for Hivel support:** the service logs showing the original authentication failure.

### Port Already in Use

**Symptom:** a service fails to start with a "port already allocated" or "address already in use" error.

**Likely causes:** another process on the host is already bound to the port a Hivel service needs.

**Diagnostic commands:**

```
# Find process using port
sudo lsof -i :80
sudo lsof -i :3000
```

**Resolution:**

```
# Kill process if needed
sudo kill -9 <PID>
```

**How to verify the fix:** re-run the deployment command; the service starts without a port error.

**What to collect for Hivel support:** the output of sudo lsof -i :\<port> and the exact error message from the failed deployment.

### Health Checks Failing

**Symptom:** `./deploy.sh --health` reports a service as unhealthy even though it appears to be running.

**Likely causes:** the service needs more time to start (especially on first deployment), or the host is resource-constrained.

**Diagnostic commands:**

```
# Check service logs
./deploy.sh --logs <service-name>

# Give services more time to start (especially first time)
# Wait 2-3 minutes after deployment

# Check resource constraints
docker stats
free -h
```

**Resolution:** wait 2–3 minutes after deployment before re-checking; if the host is resource-constrained, confirm it meets the sizing in VM / Server Requirements; otherwise investigate the specific error the logs show.

**How to verify the fix:** `./deploy.sh --health <service-name>` reports healthy.

**What to collect for Hivel support:** the full `./deploy.sh --logs <service-name>` output and the output of docker stats / free -h at the time of the failure.

### Log Rotation Not Working

**Symptom:** logs aren't rotating daily, or /var/log/hivel/ isn't being populated.

**Likely causes:** the cron job wasn't installed, or the rotation script lost its executable permission.

**Diagnostic commands:**

```
# Check cron job
sudo crontab -l

# Check log directory
ls -la /var/log/hivel/

# Run rotation manually
sudo ./scripts/docker-log-rotate.sh

# Check script permissions
chmod +x scripts/docker-log-rotate.sh
```

**Resolution:** if the cron job is missing, re-run sudo ./scripts/setup-log-rotation.sh (Log Rotation Setup); if the script lacks execute permission, run the chmod +x command above.

**How to verify the fix:** sudo crontab -l lists the rotation job, and sudo ./scripts/docker-log-rotate.sh runs without a permissions error.

**What to collect for Hivel support:** the output of sudo crontab -l and ls -la /var/log/hivel/.

**Next step:** if none of the above resolves your issue, go to Support and Summary for the full escalation checklist.

\ <br>

<br>

{% columns %}
{% column %} <a href="/archive/generic/hivel-on-premises-deployment-guide-generic/10.-verification-and-access.md" class="button primary" data-icon="backward">Verification and Access</a>
{% endcolumn %}

{% column %}

{% endcolumn %}

{% column %} <a href="/archive/generic/hivel-on-premises-deployment-guide-generic/12.-backup-and-restore.md" class="button primary" data-icon="forward">Backup and Restore</a>
{% endcolumn %}
{% endcolumns %}

<br>