> For the complete documentation index, see [llms.txt](https://docs.hivel.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.hivel.ai/self-managed-hivel-deployment/virtual-private-cloud/aws/hivel-on-premises-deployment-guide-aws/troubleshooting.md).

# Troubleshooting

On-prem Deployment Guide (AWS)

Use this guide to diagnose and resolve common deployment issues. Each issue follows this format: symptom, likely causes, diagnostic commands, resolution steps, and how to verify the fix.

### Services Not Starting

**Symptom:** One or more containers listed by `docker ps -a` are not in an Up state, or `./deploy.sh --status` shows a service as down.

**Likely causes:** The shared Docker network doesn't exist yet, a dependent service (Redis or RDS) isn't healthy, or a bad .env value.

**Diagnostics:**

```
# Check container logs
docker logs <service-name>

# Check Docker status
docker ps -a

# Verify network exists
docker network ls | grep microservices_network

# Check service dependencies
./deploy.sh --status
```

**Resolution:**

1. If the network is missing: `docker network create microservices_network`
2. Check logs for .env errors and fix them
3. Redeploy: `./deploy.sh --service <name>` (Method 1) or `docker-compose up -d` (Method 2)

**Verify the fix:** `docker ps` shows the service as Up; `./deploy.sh --health <name>` returns healthy.

**For support:** Provide full `docker logs <service-name>` output, `docker ps -a`, and your .env (passwords redacted).

### Database Connection Issues

**Symptom:** `psql` fails to connect, or service logs show a database connection error.

**Likely causes:** Wrong credentials in .env, RDS security group doesn't allow VM access, or wrong credential phase (root vs. application user) is in use.

**Diagnostics:**

```
# Test connection from host
psql -h $DB_HOST -U $DB_USER -d $DB_NAME

# Check .env file
cat config/.env | grep DB_

# Verify RDS security group allows EC2 access
# Check RDS is accessible from EC2
```

**Resolution:**

1. Confirm RDS security group allows inbound access: verify inbound rules on port 5432 for your VM security group (RDS Database Setup → Step 2)
2. Confirm correct credential phase in .env: root credentials for migration, application-user credentials for services
3. Retry the psql command

**Verify the fix:** The `psql` command returns a prompt with no error.

**For support:** Provide the exact psql error text and which credential phase is in .env.

### ECR Pull Failed

**Symptom:** `docker login` or `docker pull`/`docker-compose pull` fails with an authentication or authorization error.

**Likely causes:** AWS credentials weren't configured or have expired, or (Option B) the registry URL in docker-compose.yml/deploy.sh doesn't match your ECR.

**Diagnostics:**

```
# Re-authenticate with ECR
aws ecr get-login-password --region ap-south-1 | \
  docker login --username AWS --password-stdin \
  730335373269.dkr.ecr.ap-south-1.amazonaws.com

# Verify AWS credentials
aws sts get-caller-identity

# Check ECR permissions
aws ecr describe-repositories --region ap-south-1
```

**Resolution:**

1. Re-authenticate with ECR:
2. If credentials expired, run: `aws configure` with correct credentials
3. If using Option B, confirm all ECR references in your configuration match your own ECR registry

**Verify the fix:** `docker login` returns "Login Succeeded".

**For support:** Provide the exact error from `docker login` and the account ID output from `aws sts get-caller-identity` (passwords/keys redacted).

### ./deploy.sh Not Found

**Likely causes:** Not in the right directory, installer package wasn't fully downloaded/unzipped, or deploy.sh exists but isn't executable.

**Diagnostics:**

{% code expandable="true" %}

```
cd /opt/hivel-onprem
ls -la deploy.sh
pwd
```

{% endcode %}

**Resolution:**

1. Change to the installer directory: `cd /opt/hivel-onprem` (all commands must run from here)
2. If deploy.sh is missing, re-download and re-extract the installer (Project Setup)
3. If present but not executable: `chmod +x deploy.sh`

**Verify the fix:** `./deploy.sh --help` runs and prints the command reference with no "not found" error.

**For support:** Provide output of `pwd` and `ls -la` in the installer directory, and confirm the hivel-onprem package was successfully downloaded and unzipped.

### Migration Failed

**Symptom:** `flyway-migration` exits with a non-zero code, or logs show an error instead of "Successfully applied X migration(s)".

**Likely causes:** .env has application-user credentials instead of root/master credentials, or the insightly database doesn't exist yet.

**Diagnostics:**

```
# Check migration logs
docker logs flyway-migration

# Verify database exists
psql -h $DB_HOST -U $DB_USER -d postgres -c "\l" | grep insightly

# Check .env configuration (should have root credentials)
cat config/.env | grep DB_

# Verify you're using root/master credentials for migration
# If RDS uses Secrets Manager, retrieve credentials:
aws secretsmanager get-secret-value --secret-id <your-secret-name> --query SecretString --output text

# Test connection with root credentials
psql -h $DB_HOST -U $DB_USER -d $DB_NAME

```

**Resolution:**

1. Confirm root/master credentials (not application-user) are in .env (Environment Configuration, Phase 1)
2. Confirm the database exists (RDS Database Setup → Step 3)
3. Re-run: `./deploy.sh --service flyway-migration`

**Verify the fix:** `docker inspect flyway-migration --format='{{.State.ExitCode}}'` returns 0.

**For support:** Provide full `docker logs flyway-migration` output.

### Wrong Credentials After Migration

**Symptom:** Services fail to connect to the database immediately after a successful migration.

**Likely cause:** .env was never switched from root/master credentials to application-user credentials.

**Diagnostics:**

```
cat config/.env | grep DB_USER
```

**Resolution:**

1. Update .env to Phase 2 credentials (application-user, not root) from Environment Configuration
2. Restart services: `./deploy.sh --restart`

**Verify the fix:** Affected services report healthy in `./deploy.sh --health`.

**For support:** Provide service logs showing the original authentication failure.

### Port Already in Use

**Symptom:** A service fails to start with "address already in use" or "port is already allocated" error.

**Likely cause:** Another process on the VM is already bound to the port a Hivel service needs.

**Diagnostics:**

```
# Find process using port
sudo lsof -i :80
sudo lsof -i :3000

```

**Resolution:**

1. Stop or reassign the process already using the port: `sudo kill -9 <PID>`
2. Redeploy: `./deploy.sh --service <name>`

**Verify the fix:** The service starts and shows Up in `docker ps`.

**For support:** Provide the output of `sudo lsof -i :<port>` before you resolved it, if the cause wasn't obvious.

### Health Checks Failing

**Symptom:** `./deploy.sh --health` reports a service as unhealthy even though it's running.

**Likely causes:** The service needs more startup time (especially on first deploy), or the VM is resource-constrained.

**Diagnostics:**

```
# Check service logs
./deploy.sh --logs <service-name>

# Give services more time to start (especially first time)
# Wait 2-3 minutes after deployment

# Check resource constraints
docker stats
free -h
```

**Resolution:**

1. If newly deployed, wait 2–3 minutes for services to stabilize
2. If resource-constrained, verify your VM meets EC2 Instance Requirements
3. Inspect service logs for specific errors and address them directly

**Verify the fix:** `./deploy.sh --health <name>` reports healthy.

**For support:** Provide output of `./deploy.sh --logs <service-name>` and `docker stats` at time of failure.

### Log Rotation Not Working

**Symptom:** Logs aren't appearing under `/var/log/hivel/`, or old logs aren't being cleaned up.

**Likely causes:** The setup script wasn't run, the cron job didn't install, or the rotation script lost execute permission.

**Diagnostics:**

```
# Check cron job
sudo crontab -l

# Check log directory
ls -la /var/log/hivel/

# Run rotation manually
sudo ./scripts/docker-log-rotate.sh

# Check script permissions
chmod +x scripts/docker-log-rotate.sh

```

**Resolution:**

1. If cron job is missing: `sudo ./scripts/setup-log-rotation.sh`
2. If permissions were lost: `chmod +x scripts/docker-log-rotate.sh`
3. Optionally test manually: `sudo ./scripts/docker-log-rotate.sh`

**Verify the fix:** `sudo crontab -l` lists the rotation job; `ls -la /var/log/hivel/` shows per-service directories.

**For support:** Provide output of `sudo crontab -l` and the result of running `sudo ./scripts/docker-log-rotate.sh` manually.

#### Next Steps

If debugging is complete, continue to [Backup and Restore](https://docs.hivel.ai/self-managed-hivel-deployment/virtual-private-cloud/aws/hivel-on-premises-deployment-guide-aws/backup-and-restore) to set up ongoing operations.

<br>

{% columns %}
{% column %} <a href="/self-managed-hivel-deployment/virtual-private-cloud/aws/hivel-on-premises-deployment-guide-aws/application-setup-guide-user-sign-up-and-integration.md" class="button primary" data-icon="backward">Back to Application Setup Guide</a>
{% endcolumn %}

{% column %}

{% endcolumn %}

{% column %} <a href="/self-managed-hivel-deployment/virtual-private-cloud/aws/hivel-on-premises-deployment-guide-aws/backup-and-restore.md" class="button primary" data-icon="forward">Backup and Restore</a>
{% endcolumn %}
{% endcolumns %}