# Provider Testing Checklist

Run these tests when configuring a new AI provider or after changing credentials.

---

## Prerequisites

- [ ] `APP_ENV=local` is set (the test script refuses to run otherwise)
- [ ] The provider API key is set in `.env` as `AI_API_KEY`
- [ ] The provider base URL is accessible from the server (no firewall blocks)
- [ ] `composer install` has been run (autoloader available)

---

## Quick Test

```bash
php tests/provider-test.php all
```

This runs all preset configurations and reports pass/fail per provider.

---

## Single Provider Test

```bash
php tests/provider-test.php deepseek    # DeepSeek V4
php tests/provider-test.php openai      # OpenAI
php tests/provider-test.php generic     # Custom / unknown provider
```

---

## Manual Provider Configuration Test

For a provider not in the presets, edit `.env` directly and run the basic test:

```bash
php tests/run.php
```

Then verify the AI integration end-to-end by uploading a test document through the web UI.

---

## DeepSeek V4

| # | Test | Steps | Expected |
|---|------|-------|----------|
| D1 | Set `AI_PROVIDER=deepseek` | Add to `.env` | Config validates |
| D2 | Set `AI_BASE_URL=https://api.deepseek.com` | Add to `.env` | Config validates |
| D3 | Set `AI_MODEL=deepseek-v4-flash` | Add to `.env` | Config validates |
| D4 | Basic request | Upload a small PDF with text | Questions or suggestions appear |
| D5 | JSON parsing | AI returns JSON inside ``` fences | Fences stripped, JSON parsed correctly |
| D6 | No `response_format` sent | Check API logs or use a proxy | `response_format` NOT in the request payload |
| D7 | Invalid API key | Set `AI_API_KEY=sk-bad-key` | "AI analysis is not configured correctly" or API error |
| D8 | Wrong model | Set `AI_MODEL=nonexistent-model` | Provider error → user-friendly message |
| D9 | Rate limit | Send many rapid requests | "temporarily busy. Please wait" or API error |
| D10 | Timeout | Set short timeout or large document | "Could not reach the AI service" |

### DeepSeek V4 Pro

| # | Test | Expected |
|---|------|----------|
| D11 | Set `AI_MODEL=deepseek-v4-pro` | Works identically to flash (larger context, same API) |

---

## OpenAI

| # | Test | Steps | Expected |
|---|------|-------|----------|
| O1 | Set `AI_PROVIDER=openai` | Add to `.env` | Config validates |
| O2 | Set `AI_BASE_URL=https://api.openai.com/v1` | Add to `.env` | Config validates |
| O3 | Set `AI_MODEL=gpt-4o` | Add to `.env` | Config validates |
| O4 | `response_format: json_object` sent | Check payload | Present in request |
| O5 | Basic request | Upload a small PDF | Questions/suggestions appear |
| O6 | Content filter | Upload sensitive/banned content | AI returns `content_filter` finish reason → user-friendly message |

---

## Custom / Generic Provider

> Use `AI_PROVIDER=custom` with any OpenAI-compatible endpoint.

| # | Test | Steps | Expected |
|---|------|-------|----------|
| C1 | Set `AI_BASE_URL=https://your-provider.com/v1` | Add to `.env` | Config validates |
| C2 | Set `AI_MODEL=your-model-name` | Add to `.env` | Config validates |
| C3 | Basic request | Upload a test document | Results appear |
| C4 | Invalid API key | Use a bad key | Graceful error, no secrets exposed |
| C5 | Non-200 response | Simulate or wait for provider issue | User-friendly error |
| C6 | Non-JSON response | If provider returns text | "could not be parsed" message |
| C7 | Empty response | If provider returns empty body | "returned an empty response" |

### Generic provider — JSON mode check

If the provider supports `response_format: json_object`:

| # | Test | Expected |
|---|------|----------|
| C8 | Provider not in `JSON_MODE_PROVIDERS` | `json_object` NOT sent. JSON instruction prepended to user prompt. |
| C9 | Provider IS in `JSON_MODE_PROVIDERS` | `json_object` IS sent. No JSON instruction appended. |

---

## Environment Validation

| # | Test | Steps | Expected |
|---|------|-------|----------|
| E1 | Missing `AI_API_KEY` | Remove from `.env` | "AI analysis is not configured correctly" |
| E2 | Missing `AI_BASE_URL` | Remove from `.env` | "AI analysis is not configured correctly" |
| E3 | Missing `AI_MODEL` | Remove from `.env` | "AI analysis is not configured correctly" |
| E4 | Non-HTTPS URL (production) | Set `AI_BASE_URL=http://...` and `APP_ENV=production` | Rejected |
| E5 | Localhost URL (production) | Set `AI_BASE_URL=https://localhost/v1` and `APP_ENV=production` | Rejected unless `AI_ALLOW_LOCAL_URL=true` |
| E6 | Private IP URL (production) | Set `AI_BASE_URL=https://192.168.1.1/v1` and `APP_ENV=production` | Rejected unless `AI_ALLOW_LOCAL_URL=true` |
| E7 | Unsafe provider string | Set `AI_PROVIDER=malicious<script>` | Rejected by regex validator |

---

## Non-Functional Checks

| # | Check | Status |
|---|-------|--------|
| N1 | API key never appears in page source, error logs, or network responses | ⬜ |
| N2 | Provider name and model logged on successful validation | ⬜ |
| N3 | `tests/provider-test.php` removed or access-restricted before production | ⬜ |
| N4 | `APP_ENV` is not `local` in production | ⬜ |
| N5 | All provider-specific code paths exercised by automated tests | ⬜ |

---

## Sign-Off

| Date | Tester | Provider(s) tested | Result |
|------|--------|--------------------|--------|
|      |        |                    |        |
|      |        |                    |        |