Troubleshooting¶

Common API Errors¶

HTTP Code	Error	Cause	Resolution
400	Invalid GCS URI	Malformed `gs://` path	Verify format: `gs://bucket/path/file.mp3`
400	Download failed	File not accessible in GCS	Check bucket permissions and file existence
401	Missing API key	`X-API-Key` header not provided	Add `X-API-Key: <key>` header to request
401	Invalid API key	Key not in the allowed set	Verify key with Beatdapp; check for extra whitespace
403	Forbidden	Cloud Run IAM rejection	Add `Authorization: Bearer $(gcloud auth print-identity-token)` header
422	Inference failed	Model error processing the audio	Check that the audio file is valid (see Audio Requirements below)
429	Rate limited	Too many requests	Wait for the number of seconds in the `Retry-After` response header
503	Service busy	All GPU slots occupied	Retry after a few seconds

Deployment Issues¶

Image Not Found¶

Verify the image exists in Artifact Registry:

gcloud artifacts docker images list \
  ${REGION}-docker.pkg.dev/${GCP_PROJECT}/aimd-inference \
  --include-tags

GPU Quota Error¶

Deploy with --max-instances 1 and accept no zonal redundancy when prompted.

Container Startup Timeout¶

The service loads five neural network models at startup. This can take 30–60 seconds. Ensure your Cloud Run startup probe allows enough time. The default configuration should handle this automatically.

Secret Not Found¶

If you previously used Secret Manager and switched to API_KEYS environment variable, add --clear-secrets to your deploy command to remove stale references:

gcloud run services update aimd-inference-mvp \
  --region us-central1 \
  --clear-secrets

Debugging¶

Check Cloud Run Logs¶

gcloud logging read \
  "resource.type=cloud_run_revision AND resource.labels.service_name=aimd-inference-mvp" \
  --limit 50 \
  --project ${GCP_PROJECT}

Verify Model Loading¶

Use the health endpoint to confirm all models are loaded:

curl ${SERVICE_URL}/health \
  -H "Authorization: Bearer $(gcloud auth print-identity-token)"

Expected: models_loaded: 5, xgb_loaded: true, gpu_available: true.

Check Rate Limiter State¶

Use the /status endpoint to see current rate limiter state:

curl ${SERVICE_URL}/status \
  -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
  -H "X-API-Key: ${API_KEY}"

Request Tracing¶

Pass a custom X-Request-ID header to trace requests through logs:

curl -X POST ${SERVICE_URL}/predict \
  -H "Authorization: Bearer $(gcloud auth print-identity-token)" \
  -H "Content-Type: application/json" \
  -H "X-API-Key: ${API_KEY}" \
  -H "X-Request-ID: my-trace-id-123" \
  -d '{"gcs_uri": "gs://your-bucket/audio.mp3"}'

The same ID appears in the X-Request-ID response header and in the service logs.

Audio File Requirements¶

Supported formats: MP3, WAV, FLAC, M4A, OGG (any format supported by FFmpeg/librosa)
Location: Files must be accessible via a GCS URI (gs://...)
Minimum duration: 30 seconds recommended. Shorter files are zero-padded, which may affect prediction quality.
Maximum duration: No hard limit. Processing time scales linearly with the number of 30-second snippets.
Sample rate: Any sample rate is accepted; audio is resampled internally to 16 kHz.