Troubleshooting

On-Prem Deployment Guide (Azure)

Each entry below follows the same format: symptom, likely causes, diagnostic commands, resolution, how to confirm the fix, and what to collect if you still need to escalate to Hivel support.

Services Not Starting

Likely causes: Missing Docker network, a dependent service (e.g. Redis, database) not yet healthy, or a bad .env value.

Diagnostic commands:

# Check container logs
docker logs <service-name>
 
# Check Docker status
docker ps -a
 
# Verify network exists
docker network ls | grep microservices_network
 
# Check service dependencies
./deploy.sh --status

Resolution steps: create the missing network with docker network create microservices_network if absent; fix the .env value that docker logs points to; re-run ./deploy.sh --service <name>.

How to verify the fix: docker ps shows the service as Up; ./deploy.sh --health <name> returns healthy.

What to collect for Hivel support: full docker logs <service-name> output, docker ps -a, and your .env with all passwords redacted.

Database Connection Issues

Likely causes: wrong credentials in .env, the Azure PostgreSQL firewall/networking rules don't include the VM's IP, or sslmode=require was omitted.

Diagnostic commands:

# Test connection from host
psql "host=$DB_HOST port=$DB_PORT dbname=$DB_NAME user=$DB_USER password=$DB_PASSWORD sslmode=require"
 
# Check .env file
cat config/.env | grep DB_
 
# Verify the PostgreSQL server's networking rules allow VM access
# Check the database is accessible from the VM

Resolution steps: verify the PostgreSQL server's Networking blade (Azure Portal) has a firewall rule for the VM's IP (see Database Setup); confirm DB_USER/DB_PASSWORD in .env match the credential phase you intend (admin for migration, application user for services).

How to verify the fix: the psql command above returns a prompt with no error.

What to collect for Hivel support: the exact psql error text, and confirmation of which credential phase is currently in .env.

ECR Pull Failed

Likely causes: expired or incorrect AWS credentials, or credentials weren't re-authenticated after expiry.

Diagnostic commands:

# Re-authenticate with ECR
aws ecr get-login-password --region ap-south-1 | \
  docker login --username AWS --password-stdin \
  730335373269.dkr.ecr.ap-south-1.amazonaws.com
 
# Verify AWS credentials
aws sts get-caller-identity
 
# Check ECR permissions
aws ecr describe-repositories --region ap-south-1

Resolution steps: re-run aws configure with the credentials Hivel provided; re-authenticate to ECR using the command above.

How to verify the fix: docker login returns Login Succeeded.

What to collect for Hivel support: the exact error from docker login, and the output of aws sts get-caller-identity (account ID only - redact anything sensitive).

./deploy.sh not found

Likely causes: not in the right path (deploy.sh only exists inside the extracted installer directory), or the installer package was never fully downloaded/unzipped, or deploy.sh exists but isn't executable.

Diagnostic commands:

cd /opt/hivel-onprem
ls -la deploy.sh
pwd

Resolution: cd into /opt/hivel-onprem before running the command - it must be run from inside that directory, not from /opt or elsewhere. If deploy.sh is missing entirely, re-download and re-extract the installer package (see Project Setup). If it's present but not executable, run chmod +x deploy.sh.

Verify the fix: ./deploy.sh --help runs and prints the command reference with no "not found" error.

Collect for Hivel support: the output of pwd and ls -la in the installer directory, and confirmation that the hivel-onprem package was successfully downloaded and unzipped.

Migration Failed

Likely causes: .env has application-user credentials instead of admin/root credentials, or the insightly database doesn't exist yet.

Diagnostic commands:

# Check migration logs
docker logs flyway-migration
 
# Verify database exists
psql "host=$DB_HOST port=$DB_PORT dbname=postgres user=$DB_USER password=$DB_PASSWORD sslmode=require" -c "\l" | grep insightly
 
# Check .env configuration (should have admin credentials)
cat config/.env | grep DB_
 
# Verify you're using admin/root credentials for migration
# If credentials are stored in Key Vault, retrieve them:
az keyvault secret show --vault-name <your-vault-name> --name <secret-name> --query value -o tsv
 
# Test connection with admin credentials
psql "host=$DB_HOST port=$DB_PORT dbname=$DB_NAME user=$DB_USER password=$DB_PASSWORD sslmode=require"

Resolution steps: confirm admin/root (not application-user) credentials are in .env for migration (Database Migration, Phase 1); confirm the insightly database exists (Database Setup).

How to verify the fix: docker inspect flyway-migration --format='{{.State.ExitCode}}' returns 0.

What to collect for Hivel support: full docker logs flyway-migration output.

Wrong Credentials After Migration

Likely causes: .env was never switched from admin to application-user credentials after migration succeeded.

If services fail to connect after migration:

Diagnostic commands:

# Verify .env has application user credentials (not admin)
cat config/.env | grep DB_USER
# Should show: DB_USER= <user>
 
# If still using admin credentials, update to application user
nano config/.env
# Change:
# DB_USER=<user>
# DB_PASSWORD='<password>'  # or your custom password
 
# Restart services
./deploy.sh --restart

How to verify the fix: affected services report healthy in ./deploy.sh --health .

What to collect for Hivel support: the service logs showing the original authentication failure.

Port Already in Use

Likely causes: another process on the VM is already bound to a port Hivel needs (commonly 80 or 3000).

Diagnostic commands:

# Find process using port
sudo lsof -i :80
sudo lsof -i :3000
 
# Kill process if needed
sudo kill -9 <PID>

How to verify the fix: re-run ./deploy.sh --status and confirm the affected service starts.

What to collect for Hivel support: output of sudo lsof -i :<port> before you kill the conflicting process, if available.

Health Checks Failing

Likely causes: the service needs more startup time (especially on first deploy), or the VM is resource-constrained.

Diagnostic commands:

# Check service logs
./deploy.sh --logs <service-name>
 
# Give services more time to start (especially first time)
# Wait 2-3 minutes after deployment
 
# Check resource constraints
docker stats
free -h

Resolution steps: wait 2–3 minutes after first deployment before treating a health check failure as real; if resource-constrained, confirm the VM meets the sizing in Prerequisites and Virtual Machine Requirements; otherwise inspect the specific error in the logs.

How to verify the fix: ./deploy.sh --health <service-name> returns healthy.

What to collect for Hivel support: ./deploy.sh --logs <service-name> output and docker stats snapshot.

Log Rotation Not Working

Likely causes: the setup script wasn't run, or the rotation script lost its execute permission.

Diagnostic commands:

# Check cron job
sudo crontab -l
 
# Check log directory
ls -la /var/log/hivel/
 
# Run rotation manually
sudo ./scripts/docker-log-rotate.sh
 
# Check script permissions
chmod +x scripts/docker-log-rotate.sh

Resolution steps: re-run sudo ./scripts/setup-log-rotation.sh if the cron job is missing; re-apply execute permission if it was lost.

How to verify the fix: sudo crontab -l lists the rotation job, and ls -la /var/log/hivel/ shows per-service directories.

What to collect for Hivel support: output of sudo crontab -l and ls -la /var/log/hivel/.

Next step: if resolved, continue to Backup and Restore.

Verification and Access

Backup and Restore