Troubleshooting

On-prem Deployment Guide (AWS)

Use this guide to diagnose and resolve common deployment issues. Each issue follows this format: symptom, likely causes, diagnostic commands, resolution steps, and how to verify the fix.

Services Not Starting

Symptom: One or more containers listed by docker ps -a are not in an Up state, or ./deploy.sh --status shows a service as down.

Likely causes: The shared Docker network doesn't exist yet, a dependent service (Redis or RDS) isn't healthy, or a bad .env value.

Diagnostics:

# Check container logs
docker logs <service-name>

# Check Docker status
docker ps -a

# Verify network exists
docker network ls | grep microservices_network

# Check service dependencies
./deploy.sh --status

Resolution:

1. If the network is missing: docker network create microservices_network

2. Check logs for .env errors and fix them

3. Redeploy: ./deploy.sh --service <name> (Method 1) or docker-compose up -d (Method 2)

Verify the fix: docker ps shows the service as Up; ./deploy.sh --health <name> returns healthy.

For support: Provide full docker logs <service-name> output, docker ps -a, and your .env (passwords redacted).

Database Connection Issues

Symptom: psql fails to connect, or service logs show a database connection error.

Likely causes: Wrong credentials in .env, RDS security group doesn't allow VM access, or wrong credential phase (root vs. application user) is in use.

Diagnostics:

# Test connection from host
psql -h $DB_HOST -U $DB_USER -d $DB_NAME

# Check .env file
cat config/.env | grep DB_

# Verify RDS security group allows EC2 access
# Check RDS is accessible from EC2

Resolution:

1. Confirm RDS security group allows inbound access: verify inbound rules on port 5432 for your VM security group (RDS Database Setup → Step 2)

2. Confirm correct credential phase in .env: root credentials for migration, application-user credentials for services

3. Retry the psql command

Verify the fix: The psql command returns a prompt with no error.

For support: Provide the exact psql error text and which credential phase is in .env.

ECR Pull Failed

Symptom: docker login or docker pull/docker-compose pull fails with an authentication or authorization error.

Likely causes: AWS credentials weren't configured or have expired, or (Option B) the registry URL in docker-compose.yml/deploy.sh doesn't match your ECR.

Diagnostics:

# Re-authenticate with ECR
aws ecr get-login-password --region ap-south-1 | \
  docker login --username AWS --password-stdin \
  730335373269.dkr.ecr.ap-south-1.amazonaws.com

# Verify AWS credentials
aws sts get-caller-identity

# Check ECR permissions
aws ecr describe-repositories --region ap-south-1

Resolution:

1. Re-authenticate with ECR:

2. If credentials expired, run: aws configure with correct credentials

3. If using Option B, confirm all ECR references in your configuration match your own ECR registry

Verify the fix: docker login returns "Login Succeeded".

For support: Provide the exact error from docker login and the account ID output from aws sts get-caller-identity (passwords/keys redacted).

./deploy.sh Not Found

Likely causes: Not in the right directory, installer package wasn't fully downloaded/unzipped, or deploy.sh exists but isn't executable.

Diagnostics:

cd /opt/hivel-onprem
ls -la deploy.sh
pwd

Resolution:

1. Change to the installer directory: cd /opt/hivel-onprem (all commands must run from here)

2. If deploy.sh is missing, re-download and re-extract the installer (Project Setup)

3. If present but not executable: chmod +x deploy.sh

Verify the fix: ./deploy.sh --help runs and prints the command reference with no "not found" error.

For support: Provide output of pwd and ls -la in the installer directory, and confirm the hivel-onprem package was successfully downloaded and unzipped.

Migration Failed

Symptom: flyway-migration exits with a non-zero code, or logs show an error instead of "Successfully applied X migration(s)".

Likely causes: .env has application-user credentials instead of root/master credentials, or the insightly database doesn't exist yet.

Diagnostics:

# Check migration logs
docker logs flyway-migration

# Verify database exists
psql -h $DB_HOST -U $DB_USER -d postgres -c "\l" | grep insightly

# Check .env configuration (should have root credentials)
cat config/.env | grep DB_

# Verify you're using root/master credentials for migration
# If RDS uses Secrets Manager, retrieve credentials:
aws secretsmanager get-secret-value --secret-id <your-secret-name> --query SecretString --output text

# Test connection with root credentials
psql -h $DB_HOST -U $DB_USER -d $DB_NAME

Resolution:

1. Confirm root/master credentials (not application-user) are in .env (Environment Configuration, Phase 1)

2. Confirm the database exists (RDS Database Setup → Step 3)

3. Re-run: ./deploy.sh --service flyway-migration

Verify the fix: docker inspect flyway-migration --format='{{.State.ExitCode}}' returns 0.

For support: Provide full docker logs flyway-migration output.

Wrong Credentials After Migration

Symptom: Services fail to connect to the database immediately after a successful migration.

Likely cause: .env was never switched from root/master credentials to application-user credentials.

Diagnostics:

cat config/.env | grep DB_USER

Resolution:

1. Update .env to Phase 2 credentials (application-user, not root) from Environment Configuration

2. Restart services: ./deploy.sh --restart

Verify the fix: Affected services report healthy in ./deploy.sh --health.

For support: Provide service logs showing the original authentication failure.

Port Already in Use

Symptom: A service fails to start with "address already in use" or "port is already allocated" error.

Likely cause: Another process on the VM is already bound to the port a Hivel service needs.

Diagnostics:

# Find process using port
sudo lsof -i :80
sudo lsof -i :3000

Resolution:

1. Stop or reassign the process already using the port: sudo kill -9 <PID>

2. Redeploy: ./deploy.sh --service <name>

Verify the fix: The service starts and shows Up in docker ps.

For support: Provide the output of sudo lsof -i :<port> before you resolved it, if the cause wasn't obvious.

Health Checks Failing

Symptom: ./deploy.sh --health reports a service as unhealthy even though it's running.

Likely causes: The service needs more startup time (especially on first deploy), or the VM is resource-constrained.

Diagnostics:

# Check service logs
./deploy.sh --logs <service-name>

# Give services more time to start (especially first time)
# Wait 2-3 minutes after deployment

# Check resource constraints
docker stats
free -h

Resolution:

1. If newly deployed, wait 2–3 minutes for services to stabilize

2. If resource-constrained, verify your VM meets EC2 Instance Requirements

3. Inspect service logs for specific errors and address them directly

Verify the fix: ./deploy.sh --health <name> reports healthy.

For support: Provide output of ./deploy.sh --logs <service-name> and docker stats at time of failure.

Log Rotation Not Working

Symptom: Logs aren't appearing under /var/log/hivel/, or old logs aren't being cleaned up.

Likely causes: The setup script wasn't run, the cron job didn't install, or the rotation script lost execute permission.

Diagnostics:

# Check cron job
sudo crontab -l

# Check log directory
ls -la /var/log/hivel/

# Run rotation manually
sudo ./scripts/docker-log-rotate.sh

# Check script permissions
chmod +x scripts/docker-log-rotate.sh

Resolution:

1. If cron job is missing: sudo ./scripts/setup-log-rotation.sh

2. If permissions were lost: chmod +x scripts/docker-log-rotate.sh

3. Optionally test manually: sudo ./scripts/docker-log-rotate.sh

Verify the fix: sudo crontab -l lists the rotation job; ls -la /var/log/hivel/ shows per-service directories.

For support: Provide output of sudo crontab -l and the result of running sudo ./scripts/docker-log-rotate.sh manually.

Next Steps

If debugging is complete, continue to Backup and Restore to set up ongoing operations.

Back to Application Setup Guide

Backup and Restore