Skip to content

Commit

Permalink
docs(pipe/auto-pay): udpate readme
Browse files Browse the repository at this point in the history
  • Loading branch information
benjaminshafii committed Jan 6, 2025
1 parent 3e12279 commit 9f6e19a
Show file tree
Hide file tree
Showing 3 changed files with 81 additions and 13 deletions.
87 changes: 77 additions & 10 deletions pipes/auto-pay/README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
# Auto-Pay Pipe

Automatically trigger bank transfers based on screen activity. The pipe monitors your screen for payment-related information and can initiate transfers through the Wise API.


Right now it simplifies the preparation and requires human-in-the-loop to confirm the transfer.

> ⚠️ **Early Development Notice**: This pipe is in active development and subject to breaking changes. The API, UI, and functionality may change significantly as we improve the system.
## Features

- Real-time screen monitoring to detect payment information
Expand All @@ -17,14 +20,63 @@ Automatically trigger bank transfers based on screen activity. The pipe monitors

## How it Works

1. **Screen Analysis**: Uses screenpipe's OCR to analyze your screen activity in real-time
2. **Payment Detection**: Identifies payment-related information from various sources (invoices, emails, etc.)
3. **Data Extraction**: Extracts structured payment information using AI models
4. **Transfer Creation**: Automatically creates and executes transfers through Wise:
- Creates quotes for currency conversion
- Manages recipient accounts
- Validates transfer requirements
- Initiates and funds transfers
The pipe follows a three-stage process:

1. **Browse & Monitor**
- Continuously monitors your screen activity using screenpipe's OCR
- Captures text from various sources (browsers, PDFs, emails)
- Stores data locally for privacy and quick access

2. **Detect & Extract**
- AI models analyze captured text to identify payment information
- Extracts structured data like amounts, recipients, and bank details
- Assigns confidence scores to detected payments
- Presents findings for your review

3. **Prepare & Execute**
- Validates extracted information against Wise requirements
- Creates necessary recipient records and quotes
- Handles currency conversion if needed
- Initiates transfer after your confirmation

## Setup & Testing

### Prerequisites
- OpenAI API key (for payment detection)
- Wise API credentials:
- API Key
- Profile ID

### Testing Flow

1. **Start with Sandbox**
```json
// pipe.json settings
{
"wiseApiKey": "your-sandbox-api-key",
"wiseProfileId": "your-profile-id",
"enableProduction": false // Keep false for testing
}
```

2. **Test Payment Detection**
- Open an invoice or payment details in your browser/PDF viewer
- Click "Start Detection" in the Auto-Pay interface
- Review detected payment information
- Confidence scores help evaluate detection accuracy

3. **Test Transfer Creation**
- Use Wise Sandbox credentials first
- Transfers will be created in test mode
- Monitor the process in Wise Sandbox dashboard
- Check transfer statuses and notifications

### Moving to Production

1. Update `pipe.json` with production credentials
2. Set `enableProduction` to `true`
3. Thoroughly test with small amounts first
4. Monitor transfer execution closely

## Privacy & Security

Expand All @@ -41,4 +93,19 @@ Automatically trigger bank transfers based on screen activity. The pipe monitors
- Handling international transfers
- Managing contractor payments

## Known Limitations

- Currently optimized for English language content
- Best results with clearly structured payment information
- May require manual correction for complex invoices
- Transfer speeds depend on Wise processing times

## Future Improvements

- Multi-language support
- Enhanced OCR accuracy
- Batch payment processing
- Custom validation rules
- Integration with accounting software

<img width="1312" alt="Screenshot 2024-12-21 at 4 39 29 PM" src="https://github.com/user-attachments/assets/2e395762-198f-43e6-9e5a-2974b8e71fcf" />
2 changes: 1 addition & 1 deletion pipes/auto-pay/src/agents/payment-detector-agent.ts
Original file line number Diff line number Diff line change
Expand Up @@ -378,7 +378,7 @@ export function usePaymentDetector(recognizedItemId: string) {
setIsProcessing(false);
abortControllerRef.current = null;
}
}, [recognizedItemId]);
}, [recognizedItemId, settings?.openaiApiKey]);

// Cleanup on unmount
useEffect(() => {
Expand Down
5 changes: 3 additions & 2 deletions pipes/auto-pay/src/agents/payment-preparer-agent.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ import { useCallback, useState, useRef, useEffect } from 'react';
import { z } from 'zod';
import type { PaymentInfo } from '@/types/wise';
import { useSettings } from '@/hooks/use-settings';
import { useSettingsStore } from '@/lib/settings';

// Zod schemas for Wise transfer data
const transferDetailsSchema = z.object({
Expand Down Expand Up @@ -91,8 +92,8 @@ export async function runPaymentPreparer(
try {
// Clear any existing steps for this item
useAgentStepsStore.getState().clearSteps(recognizedItemId);
const apiKey = useSettings().settings?.openaiApiKey;
const openai = createOpenAI({ apiKey });
const apiKey = useSettingsStore.getState().openaiApiKey;
const openai = createOpenAI({ apiKey: apiKey || undefined });

// Check if already aborted
if (signal?.aborted) {
Expand Down

0 comments on commit 9f6e19a

Please sign in to comment.