Common Issues

Solutions for frequently encountered problems.

Startup Issues

Service Fails to Start

Symptom: Application exits immediately after startup.

Possible Causes:

1. Nacos not reachable

   Error: Failed to connect to Nacos server

   Solution: Ensure Nacos is running on port 8848.

   curl http://localhost:8848/nacos/v1/console/health/liveness

2. Database connection failed

   Error: Communications link failure

   Solution: Verify MySQL is running and credentials are correct.

   mysql -h <mysql-host> -u <mysql-user> -p RecordPlatform

3. Port already in use

   Error: Address already in use: 8000

   Solution: Kill the existing process or use a different port.

   lsof -i :8000
   kill -9 <PID>

Dubbo Service Registration Failed

Symptom: Provider services not visible in Nacos console.

Solution:

1. Check Nacos address in configuration
2. Verify network connectivity to Nacos
3. Check Dubbo protocol port is not blocked

Authentication Issues

401 Unauthorized

Symptom: All API requests return 401.

Possible Causes:

1. JWT expired

   • Tokens expire after the configured TTL
   • Solution: Re-authenticate to get a new token

2. Invalid JWT_KEY

   • If JWT_KEY changed after token issuance
   • Solution: Clear client tokens and re-login

3. Missing Authorization header

   • Request lacks Authorization: Bearer <token>
   • Solution: Include the header in all authenticated requests

403 Forbidden

Symptom: User authenticated but access denied.

Possible Causes:

1. Insufficient permissions

   • User lacks required role/permission
   • Solution: Check user's role assignments

2. Resource ownership

   • Trying to access another user's resources
   • Solution: Verify resource ownership or admin privileges

Storage Issues

File Upload Fails

Symptom: Upload returns error or times out.

Possible Causes:

1. S3 node offline

   # Check node status
   curl http://localhost:8092/actuator/health

   Solution: Restart the offline node or wait for failover.

2. Insufficient replicas

   • Healthy nodes cannot satisfy effective quorum or degraded-write minimum replicas
   • Solution: Ensure online nodes satisfy storage.replication.quorum and are not below storage.degraded-write.min-replicas.

3. Bucket not exists

   Error: The specified bucket does not exist

   Solution: Create the bucket or check bucket name configuration.

File Download Fails

Symptom: Download returns 404 or corrupted data.

Possible Causes:

1. File not fully replicated

   • Saga transaction incomplete
   • Solution: Check saga status in database.

2. Shard corruption

   • One or more shards are corrupted
   • Solution: Trigger consistency repair via admin endpoint.

3. Node containing shard is offline

   • Solution: Wait for node recovery or trigger rebalance.

Saga Transaction Stuck

Symptom: Upload shows as pending indefinitely.

Solution:

1. Check file_saga table for status
2. Review saga step failures in file_saga_step
3. Trigger manual compensation if needed:

   UPDATE file_saga SET status = 'COMPENSATING'
   WHERE saga_id = '<id>' AND status = 'RUNNING';

Blockchain Issues

Connection Timeout

Symptom: Blockchain operations timeout after 30s.

Possible Causes:

1. FISCO node unreachable

   # Test connectivity
   telnet 127.0.0.1 20200

   Solution: Verify node is running and network allows connection.

2. Certificate mismatch

   • SDK certificates don't match node certificates
   • Solution: Regenerate and deploy matching certificates.

3. Circuit breaker open

   • Too many failures triggered circuit breaker
   • Solution: Wait for half-open state or restart service.

Contract Call Failed

Symptom: Smart contract operations return errors.

Possible Causes:

1. Contract not deployed

   Error: Contract address is null

   Solution: Deploy contracts and update addresses in config.

2. Insufficient gas

   • Transaction exceeds gas limit
   • Solution: Increase gas limit in configuration.

3. Permission denied

   • Caller not authorized for the contract
   • Solution: Check contract ACL configuration.

Performance Issues

Slow API Response

Symptom: API requests take >5 seconds.

Diagnostic Steps:

1. Check database queries

   # Enable slow query log
   SET GLOBAL slow_query_log = 'ON';
   SET GLOBAL long_query_time = 1;

2. Check connection pools

   • Druid monitor: /record-platform/druid/
   • Look for connection wait times

3. Check S3 node latency

   • Review s3_node_load_score metrics
   • Consider adding more nodes

High Memory Usage

Symptom: JVM heap usage >80%.

Solutions:

1. Increase heap size

   JAVA_OPTS="-Xms4g -Xmx8g"

2. Check for memory leaks

   • Enable heap dumps on OOM
   • Analyze with Eclipse MAT or VisualVM

3. Reduce concurrent uploads

   • Limit multipart.max-file-size
   • Reduce chunk buffer sizes

Database Connection Pool Exhausted

Symptom: "Cannot acquire connection from pool" errors.

Solutions:

1. Increase pool size

   spring:
     datasource:
       druid:
         max-active: 100

2. Fix connection leaks

   • Check for unclosed connections in code
   • Enable connection leak detection in Druid

3. Optimize slow queries

   • Add missing indexes
   • Review query execution plans

Redis Issues

Cache Miss Rate High

Symptom: Database queries for cached data.

Solutions:

1. Check Redis connectivity

   redis-cli -h <redis-host> ping

2. Verify TTL settings

   • Ensure cache isn't expiring too quickly

3. Check memory eviction

   • If Redis is evicting keys due to memory pressure
   • Increase maxmemory or reduce cached data

Redis Connection Timeout

Symptom: Intermittent Redis timeouts.

Solutions:

1. Check network latency

   redis-cli -h <redis-host> --latency

2. Increase connection timeout

   spring:
     redis:
       timeout: 5000

3. Use connection pooling

   • Configure Lettuce pool settings

RabbitMQ Issues

Messages Not Being Consumed

Symptom: Queue depth keeps growing.

Solutions:

1. Check consumer is running

   rabbitmqctl list_consumers

2. Check for consumer errors

   • Review application logs for listener exceptions

3. Check queue bindings

   rabbitmqctl list_bindings

Dead Letter Queue Growing

Symptom: DLQ has many messages.

Solutions:

1. Review message processing errors

   • Check application logs for root cause

2. Reprocess DLQ messages

   • Move messages back to main queue after fixing issue

3. Adjust retry settings

   spring:
     rabbitmq:
       listener:
         simple:
           retry:
             max-attempts: 5

BSN Besu Nonce Coordination

Signer Writer Lock Prevents Startup

Symptom: platform-fisco fails with current chainId/signer is already owned by another writer (the runtime log may be localized).

Resolution:

1. Treat this as a safety gate. Do not point the new process at a different or empty BSN_BESU_NONCE_STATE_DIRECTORY to bypass it.
2. Locate every process/container that has the same signer secret. Fence and stop the old active writer first.
3. Verify the replacement uses the same durable state directory and a filesystem with reliable file-lock and atomic-move semantics.
4. Start exactly one replacement. The empty .lock file may remain after shutdown; ownership is the live OS file lock, so deleting that file is neither required nor a safe fencing mechanism.

Same-signer active-active across independent hosts is unsupported. Use a cold standby, distinct signers, or a separately reviewed distributed lease/nonce coordinator.

Unresolved Broadcast Blocks New Writes

Symptom: a send timed out, disconnected, returned already known/a nonce conflict, or returned an invalid transaction hash; subsequent writes fail with unresolved broadcast; manual reconciliation is required.

This is fail-closed behavior. The durable .state file records the schema, chain ID, signer, lastNonce, nextNonce, outcome, local transaction hash, optional remote hash, and update time. It never records the private key or signed raw transaction.

Reconciliation:

1. Stop new writes for this signer and fence every possible writer. Keep the state file as incident evidence; never delete or edit it while a process is running.
2. Read lastNonce and localTransactionHash from the state file. Against the same BSN provider endpoint, query the transaction by that exact hash and query both eth_getTransactionCount(signer, "pending") and "latest".
3. If the exact transaction exists, keep the state file and wait for the provider's pending nonce to advance beyond lastNonce. Transaction visibility confirms that lastNonce must not be reused, but it does not bypass the coordinator's PENDING gate.
4. Only after the provider's pending nonce is greater than lastNonce should you restart the single writer. The coordinator will then observe the advanced PENDING value and continue at max(nodePending, durableNextNonce). If the transaction is absent and PENDING has not advanced, that observation alone does not prove the request was never accepted. Wait for provider reconciliation or obtain explicit provider evidence that the transaction was definitively rejected. Do not submit a different payload with lastNonce.
5. Only after conclusive non-acceptance evidence, with every old writer fenced and the service stopped, archive the complete .state file with the incident record and move it out of the configured directory. Restart the single writer so it rebuilds from the node's stable PENDING nonce. If acceptance remains uncertain, preserve the state and keep writes blocked.

A contract revert or failed receipt still consumes the nonce and never justifies rolling the state back. Rollback of this code must also preserve the state directory until all recorded transactions are reconciled.

Quota Issues

Upload Fails with QUOTA_EXCEEDED

Symptom: Upload rejected with error code 50013 (QUOTA_EXCEEDED).

Cause: The tenant/user has exceeded their storage quota and enforcement mode is ENFORCE.

Solution:

1. Check current quota status via GET /api/v1/files/quota
2. If in SHADOW mode, uploads are allowed but logged — verify quota.enforcement-mode in config
3. To increase quota limits, contact the admin or adjust quota_policy table