2026/01/21 更新
This commit is contained in:
lihongyang
parent
672912f2be
commit
9e5f11472a
|
|
@ -0,0 +1,13 @@
|
|||
# API Keys for AI Video LifePlan Generator
|
||||
|
||||
# Google Gemini API Key
|
||||
# Get yours at: https://aistudio.google.com/apikey
|
||||
GEMINI_API_KEY=your_gemini_api_key_here
|
||||
|
||||
# OpenAI API Key (for Whisper and TTS)
|
||||
# Get yours at: https://platform.openai.com/api-keys
|
||||
OPENAI_API_KEY=your_openai_api_key_here
|
||||
|
||||
# Pexels API Key (for B-Roll videos)
|
||||
# Get yours at: https://www.pexels.com/api/
|
||||
PEXELS_API_KEY=your_pexels_api_key_here
|
||||
|
|
@ -0,0 +1,333 @@
|
|||
# AI Video LifePlan - 功能实现说明
|
||||
|
||||
本项目已完整实现以下6大核心功能模块:
|
||||
|
||||
## ✅ 已实现功能
|
||||
|
||||
### 1️⃣ Input Analysis - 输入分析 (PDF / 视频 / 音频)
|
||||
|
||||
**实现文件**: [`backend/extractor.py`](backend/extractor.py)
|
||||
|
||||
#### PDF 解析 (Gemini Pro)
|
||||
- 使用 Google Gemini 2.0 Flash Exp 模型
|
||||
- 自动提取结构化数据:
|
||||
- 客户基本信息 (姓名、年龄)
|
||||
- 财务数据 (资产、收入、支出)
|
||||
- 家庭结构
|
||||
- 人生事件规划
|
||||
- 财务目标
|
||||
|
||||
**使用示例**:
|
||||
```python
|
||||
from backend.extractor import PDFExtractor
|
||||
|
||||
extractor = PDFExtractor()
|
||||
data = extractor.extract("path/to/lifeplan.pdf")
|
||||
```
|
||||
|
||||
#### 视频/音频转录 (OpenAI Whisper)
|
||||
- 使用 FFmpeg 提取音频
|
||||
- OpenAI Whisper API 进行高精度日语转录
|
||||
- 支持分段时间戳
|
||||
- 自动提取关键词
|
||||
|
||||
**使用示例**:
|
||||
```python
|
||||
from backend.extractor import VideoExtractor
|
||||
|
||||
extractor = VideoExtractor()
|
||||
result = extractor.extract("path/to/interview.mp4")
|
||||
# 返回: transcription, segments, keywords
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2️⃣ Scripting - 脚本生成
|
||||
|
||||
**实现文件**: [`backend/scripter.py`](backend/scripter.py)
|
||||
|
||||
- 使用 Gemini Pro 生成自然的日语口语脚本
|
||||
- 基于 PDF 数据和视频转录
|
||||
- 支持多种风格 (friendly, professional, casual)
|
||||
- 自动分段并计算时长
|
||||
|
||||
**使用示例**:
|
||||
```python
|
||||
from backend.scripter import generate_script
|
||||
|
||||
script = generate_script(pdf_data, video_data, style="friendly")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3️⃣ Audio - TTS 音频生成
|
||||
|
||||
**实现文件**: [`backend/audio_generator.py`](backend/audio_generator.py)
|
||||
|
||||
- OpenAI TTS API (tts-1 模型)
|
||||
- 多种语音选择 (alloy, echo, fable, onyx, nova, shimmer)
|
||||
- 自动为每个脚本段落生成音频
|
||||
- 输出到 `assets/1_processed/audio/`
|
||||
|
||||
**使用示例**:
|
||||
```python
|
||||
from backend.audio_generator import generate_audio_for_script
|
||||
|
||||
updated_script = generate_audio_for_script(script_data, voice="alloy")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 4️⃣ Dynamic Visuals - 动态图表 (Remotion)
|
||||
|
||||
**实现文件**:
|
||||
- [`src/Composition.tsx`](src/Composition.tsx)
|
||||
- [`src/DualChartView.tsx`](src/DualChartView.tsx)
|
||||
- [`src/LifePlanChart.tsx`](src/LifePlanChart.tsx)
|
||||
|
||||
- 基于 Remotion 的程序化视频渲染
|
||||
- 实时动态图表:
|
||||
- 资产消耗曲线
|
||||
- 现金流分析
|
||||
- 双图表联动显示
|
||||
- 虚拟人角色同步说话
|
||||
- 字幕自动同步
|
||||
|
||||
---
|
||||
|
||||
### 5️⃣ Stock Visuals - B-Roll 素材获取 (Pexels API)
|
||||
|
||||
**实现文件**: [`backend/media_fetcher.py`](backend/media_fetcher.py)
|
||||
|
||||
- 自动翻译日语关键词为英语 (使用 Gemini)
|
||||
- Pexels API 搜索并下载 HD 视频
|
||||
- 智能缓存机制 (避免重复下载)
|
||||
- 支持本地文件路径引用
|
||||
|
||||
**功能特性**:
|
||||
- 关键词翻译缓存
|
||||
- 自动选择最佳质量 (≤1920p HD)
|
||||
- 存储到 `assets/1_processed/broll/`
|
||||
|
||||
**使用示例**:
|
||||
```python
|
||||
from backend.media_fetcher import MediaFetcher
|
||||
|
||||
fetcher = MediaFetcher()
|
||||
results = fetcher.fetch_broll(["家族", "老後", "貯蓄"], max_per_keyword=2)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 6️⃣ Source Slicing - 视频切片 (可选功能)
|
||||
|
||||
**实现文件**: [`backend/video_slicer.py`](backend/video_slicer.py)
|
||||
|
||||
- 基于转录内容智能识别关键片段
|
||||
- 使用 Gemini 分析重要性
|
||||
- FFmpeg 快速切片 (stream copy, 无需重编码)
|
||||
- 输出到 `assets/1_processed/video_clips/`
|
||||
|
||||
**使用示例**:
|
||||
```python
|
||||
from backend.video_slicer import extract_video_clips
|
||||
|
||||
clips = extract_video_clips(
|
||||
video_path="interview.mp4",
|
||||
transcription_data=video_data,
|
||||
max_clips=3
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 完整工作流
|
||||
|
||||
### 命令行使用
|
||||
|
||||
```bash
|
||||
# 基础使用 - 仅 PDF
|
||||
python main.py --pdf assets/0_input/report.pdf
|
||||
|
||||
# 带视频转录
|
||||
python main.py --pdf report.pdf --video interview.mp4
|
||||
|
||||
# 启用视频切片功能
|
||||
python main.py --pdf report.pdf --video interview.mp4 --extract-clips
|
||||
|
||||
# 仅生成数据,不渲染视频
|
||||
python main.py --pdf report.pdf --dry-run
|
||||
```
|
||||
|
||||
### 工作流步骤
|
||||
|
||||
```
|
||||
1. 📄 PDF 解析 (Gemini Pro)
|
||||
↓
|
||||
2. 🎥 视频转录 (Whisper) [可选]
|
||||
↓
|
||||
3. ✍️ 脚本生成 (Gemini Pro)
|
||||
↓
|
||||
4. 🎙️ 音频生成 (OpenAI TTS)
|
||||
↓
|
||||
5. 🎬 B-Roll 获取 (Pexels API)
|
||||
↓
|
||||
6. ✂️ 视频切片 [可选]
|
||||
↓
|
||||
7. 📊 数据整合 (render_props.json)
|
||||
↓
|
||||
8. 🎞️ 视频渲染 (Remotion)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔑 API 配置
|
||||
|
||||
需要在 `.env` 文件中配置以下 API Keys:
|
||||
|
||||
```env
|
||||
GEMINI_API_KEY=your_gemini_api_key
|
||||
OPENAI_API_KEY=your_openai_api_key
|
||||
PEXELS_API_KEY=your_pexels_api_key
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📦 依赖安装
|
||||
|
||||
### Python 依赖
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
主要包含:
|
||||
- `google-genai` - Gemini AI SDK (新版)
|
||||
- `openai` - OpenAI API (Whisper + TTS)
|
||||
- `requests` - HTTP 请求 (Pexels API)
|
||||
- `ffmpeg-python` - 视频处理
|
||||
- `python-dotenv` - 环境变量管理
|
||||
|
||||
### 系统依赖
|
||||
- **FFmpeg**: 必须安装并在 PATH 中
|
||||
- Windows: `choco install ffmpeg` 或从官网下载
|
||||
- Mac: `brew install ffmpeg`
|
||||
- Linux: `apt install ffmpeg`
|
||||
|
||||
### Node.js 依赖
|
||||
```bash
|
||||
npm install
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📁 目录结构
|
||||
|
||||
```
|
||||
assets/
|
||||
├── 0_input/ # 输入文件 (PDF, 视频)
|
||||
├── 1_processed/ # 中间处理文件
|
||||
│ ├── audio/ # 生成的音频
|
||||
│ ├── broll/ # 下载的 B-Roll 视频
|
||||
│ ├── video_clips/ # 提取的视频片段
|
||||
│ ├── render_props.json
|
||||
│ └── translation_cache.json
|
||||
└── 2_output/ # 最终视频输出
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔍 调试与验证
|
||||
|
||||
### 检查 PDF 提取结果
|
||||
```python
|
||||
from backend.extractor import PDFExtractor
|
||||
extractor = PDFExtractor()
|
||||
data = extractor.extract("path/to/pdf")
|
||||
print(json.dumps(data, ensure_ascii=False, indent=2))
|
||||
```
|
||||
|
||||
### 检查视频转录
|
||||
```python
|
||||
from backend.extractor import VideoExtractor
|
||||
extractor = VideoExtractor()
|
||||
result = extractor.extract("path/to/video.mp4")
|
||||
print(f"Duration: {result['duration']}s")
|
||||
print(f"Segments: {len(result['segments'])}")
|
||||
```
|
||||
|
||||
### 检查 B-Roll 下载
|
||||
```python
|
||||
from backend.media_fetcher import MediaFetcher
|
||||
fetcher = MediaFetcher()
|
||||
results = fetcher.fetch_broll(["家族", "老後"])
|
||||
for keyword, videos in results.items():
|
||||
print(f"{keyword}: {len(videos)} videos")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ 注意事项
|
||||
|
||||
1. **API 配额**:
|
||||
- Gemini Pro 和 OpenAI API 有使用限制
|
||||
- Pexels API 免费版每小时 200 请求
|
||||
|
||||
2. **文件大小**:
|
||||
- 视频文件建议不超过 500MB
|
||||
- PDF 建议不超过 20 页
|
||||
|
||||
3. **处理时间**:
|
||||
- PDF 解析: ~10-30秒
|
||||
- 视频转录: ~视频时长的 1/3
|
||||
- B-Roll 下载: ~2-5分钟 (取决于网络)
|
||||
- 视频渲染: ~1-3分钟
|
||||
|
||||
4. **语言支持**:
|
||||
- 主要支持日语
|
||||
- B-Roll 关键词会自动翻译为英语
|
||||
|
||||
---
|
||||
|
||||
## 🐛 常见问题
|
||||
|
||||
### Q: FFmpeg not found
|
||||
A: 确保 FFmpeg 已安装并在系统 PATH 中
|
||||
```bash
|
||||
ffmpeg -version # 检查是否安装
|
||||
```
|
||||
|
||||
### Q: Gemini API 错误
|
||||
A: 检查 API key 是否正确,是否有配额
|
||||
|
||||
### Q: Whisper 转录失败
|
||||
A: 确认视频文件有音频轨道,格式支持
|
||||
|
||||
### Q: B-Roll 下载慢
|
||||
A: Pexels API 有速率限制,会自动等待
|
||||
|
||||
---
|
||||
|
||||
## 📄 相关文档
|
||||
|
||||
- [项目总览 README](PROJECT_README.md)
|
||||
- [Remotion 配置](remotion.config.ts)
|
||||
- [字幕提取调查](backend/SUBTITLE_INVESTIGATION_README.md)
|
||||
|
||||
---
|
||||
|
||||
## 📝 更新日志
|
||||
|
||||
**v1.0.0** (2026-01-16)
|
||||
- ✅ 完整实现 6 大核心功能
|
||||
- ✅ PDF 语义分析 (Gemini)
|
||||
- ✅ 视频/音频转录 (Whisper)
|
||||
- ✅ 脚本生成 (Gemini)
|
||||
- ✅ TTS 音频生成 (OpenAI)
|
||||
- ✅ B-Roll 素材获取 (Pexels)
|
||||
- ✅ 视频切片 (FFmpeg + Gemini)
|
||||
- ✅ 动态图表渲染 (Remotion)
|
||||
|
||||
---
|
||||
|
||||
**构建者**: Qoder AI Assistant
|
||||
**最后更新**: 2026-01-16
|
||||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
After Width: | Height: | Size: 1.7 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 601 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 521 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 535 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 628 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 621 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 1.7 MiB |
|
|
@ -1,5 +1,5 @@
|
|||
{
|
||||
"家族": "Family",
|
||||
"家族": "family",
|
||||
"老後": "Retirement",
|
||||
"貯蓄": "Savings",
|
||||
"住宅": "Housing",
|
||||
|
|
@ -8,5 +8,23 @@
|
|||
"将来": "Future",
|
||||
"不安": "Anxiety",
|
||||
"安心": "Peace of mind",
|
||||
"計画": "Planning"
|
||||
"計画": "Planning",
|
||||
"未来": "future",
|
||||
"インタビュー": "interview",
|
||||
"ライフイベント": "life event",
|
||||
"貯金": "savings",
|
||||
"教育資金": "education fund",
|
||||
"節約": "saving",
|
||||
"収入": "income",
|
||||
"資産運用": "asset management",
|
||||
"将来設計": "future planning",
|
||||
"考える": "thinking",
|
||||
"家族団らん": "family time",
|
||||
"未来設計": "future planning",
|
||||
"イベント": "event",
|
||||
"夢": "dream",
|
||||
"戦略": "strategy",
|
||||
"投資": "investment",
|
||||
"挨拶": "greeting",
|
||||
"疑問": "question"
|
||||
}
|
||||
|
|
@ -13,6 +13,8 @@ try:
|
|||
except ImportError:
|
||||
OpenAI = None
|
||||
|
||||
from backend.video_processor.utils import get_video_duration
|
||||
|
||||
|
||||
class AudioGenerator:
|
||||
"""Generate narration audio using OpenAI TTS"""
|
||||
|
|
@ -29,7 +31,8 @@ class AudioGenerator:
|
|||
)
|
||||
|
||||
self.client = OpenAI(api_key=self.api_key)
|
||||
self.output_dir = Path("assets/1_processed/audio")
|
||||
# CRITICAL: Output directly to public/ for Remotion to find them via staticFile
|
||||
self.output_dir = Path("public")
|
||||
self.output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def generate_audio(
|
||||
|
|
@ -56,6 +59,13 @@ class AudioGenerator:
|
|||
for segment in script_segments:
|
||||
segment_id = segment["id"]
|
||||
text = segment["text"]
|
||||
is_video_clip = segment.get("type") == "video_clip" or segment.get("visualType") == "video"
|
||||
|
||||
# Skip TTS for video clips (keep original audio)
|
||||
if is_video_clip:
|
||||
print(f" ⏭️ Keeping original audio for video segment {segment_id}")
|
||||
updated_segments.append(segment)
|
||||
continue
|
||||
|
||||
try:
|
||||
# Generate audio
|
||||
|
|
@ -71,12 +81,19 @@ class AudioGenerator:
|
|||
# Save audio file
|
||||
response.stream_to_file(str(audio_path))
|
||||
|
||||
# CRITICAL: Get actual duration of the generated audio file
|
||||
try:
|
||||
actual_duration = get_video_duration(str(audio_path))
|
||||
segment["duration"] = actual_duration
|
||||
print(f" ✓ Generated {audio_filename} (Actual duration: {actual_duration:.2f}s)")
|
||||
except Exception as de:
|
||||
print(f" ⚠️ Could not get actual duration for {audio_filename}: {de}")
|
||||
# Keep existing duration as fallback
|
||||
|
||||
# Update segment with audio path
|
||||
segment["audioPath"] = str(audio_path)
|
||||
updated_segments.append(segment)
|
||||
|
||||
print(f" ✓ Generated {audio_filename}")
|
||||
|
||||
except Exception as e:
|
||||
print(f" ⚠️ Failed to generate audio for segment {segment_id}: {e}")
|
||||
# Keep segment without audio
|
||||
|
|
|
|||
|
|
@ -0,0 +1,124 @@
|
|||
import json
|
||||
import os
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
# Path configuration
|
||||
PROJECT_ROOT = Path(__file__).parent.parent
|
||||
PROPS_PATH = PROJECT_ROOT / "assets" / "1_processed" / "render_props.json"
|
||||
PUBLIC_DIR = PROJECT_ROOT / "public"
|
||||
VIDEO_PATH = PROJECT_ROOT / "assets" / "0_input" / "0df913b55e79323c3985e9d826af23e6.mp4"
|
||||
|
||||
# Use Remotion's bundled ffmpeg if system ffmpeg is not available
|
||||
FFMPEG_PATH = PROJECT_ROOT / "node_modules" / "@remotion" / "compositor-win32-x64-msvc" / "ffmpeg.exe"
|
||||
|
||||
def slice_video_custom(input_path, start, end, output_path):
|
||||
"""Simple wrapper for ffmpeg slicing"""
|
||||
duration = end - start
|
||||
|
||||
# Choose ffmpeg executable
|
||||
exe = "ffmpeg"
|
||||
if FFMPEG_PATH.exists():
|
||||
exe = str(FFMPEG_PATH)
|
||||
|
||||
cmd = [
|
||||
exe, "-y",
|
||||
"-ss", str(start),
|
||||
"-t", str(duration),
|
||||
"-i", str(input_path),
|
||||
"-c:v", "libx264", "-crf", "23", "-preset", "veryfast",
|
||||
"-c:a", "aac", "-b:a", "128k",
|
||||
str(output_path)
|
||||
]
|
||||
print(f"🎬 Executing: {' '.join(cmd)}")
|
||||
subprocess.run(cmd, check=True, capture_output=True)
|
||||
return str(output_path)
|
||||
|
||||
def convert_dialogue_to_video():
|
||||
"""
|
||||
Converts specific 'dialogue' segments to 'video' segments
|
||||
by matching timestamps from the original extraction.
|
||||
"""
|
||||
if not PROPS_PATH.exists():
|
||||
print(f"❌ Error: {PROPS_PATH} not found.")
|
||||
return
|
||||
|
||||
# Choose ffmpeg executable
|
||||
exe = "ffmpeg"
|
||||
if FFMPEG_PATH.exists():
|
||||
exe = str(FFMPEG_PATH)
|
||||
|
||||
print(f"📖 Reading {PROPS_PATH}...")
|
||||
with open(PROPS_PATH, "r", encoding="utf-8") as f:
|
||||
props = json.load(f)
|
||||
|
||||
# We want to target the client's responses in the opening dialogue
|
||||
# Based on the file content:
|
||||
# Segment 2: "はい、私がテスト太郎として" -> Client
|
||||
# Segment 4: "はい、生年月日の方は..." -> Client
|
||||
# Segment 6: "はい。" -> Client
|
||||
|
||||
# Mapping table (based on №１_基本情報_extraction.json)
|
||||
# i=1: 00:27 - 00:44 (顧客)
|
||||
# i=3: 01:04 - 01:46 (顧客) -> Matches "生年月日" topic (actually i=1 has birthdate in text)
|
||||
|
||||
# Let's define the new clips to extract
|
||||
new_clips = [
|
||||
{"start": 27.0, "end": 35.0, "text": "はい、私がテスト太郎として..."}, # Map to script[1]
|
||||
{"start": 35.0, "end": 44.0, "text": "生年月日の方は、1980年の1月の1日です。"}, # Map to script[3]
|
||||
{"start": 69.0, "end": 72.0, "text": "はい。"} # Map to script[5]
|
||||
]
|
||||
|
||||
extracted_clips = []
|
||||
|
||||
# 1. Extract the physical audio files (MP3) instead of video
|
||||
print("✂️ Extracting audio from clips...")
|
||||
for i, clip_info in enumerate(new_clips):
|
||||
filename = f"dialogue_clip_{i+1}.mp3"
|
||||
output_path = PUBLIC_DIR / filename
|
||||
|
||||
try:
|
||||
# -vn: skip video, -acodec libmp3lame: encode as mp3
|
||||
cmd = [
|
||||
exe, "-y",
|
||||
"-ss", str(clip_info["start"]),
|
||||
"-t", str(clip_info["end"] - clip_info["start"]),
|
||||
"-i", str(VIDEO_PATH),
|
||||
"-vn", "-acodec", "libmp3lame", "-ab", "128k",
|
||||
str(output_path)
|
||||
]
|
||||
print(f"🎬 Executing: {' '.join(cmd)}")
|
||||
subprocess.run(cmd, check=True, capture_output=True)
|
||||
|
||||
extracted_clips.append({
|
||||
"id": f"dialogue_clip_{i+1}",
|
||||
"audioPath": f"public/{filename}",
|
||||
"duration": clip_info["end"] - clip_info["start"]
|
||||
})
|
||||
except Exception as e:
|
||||
print(f"⚠️ Failed to extract audio {i+1}: {e}")
|
||||
|
||||
# 3. Update script segments to 'dialogue' type with 'audioPath'
|
||||
print("📝 Updating script segments to 'dialogue' type with original audio...")
|
||||
|
||||
mapping = {1: 0, 3: 1, 5: 2} # script[index] -> dialogue_clip index
|
||||
|
||||
for script_idx, clip_idx in mapping.items():
|
||||
if script_idx < len(props["script"]):
|
||||
seg = props["script"][script_idx]
|
||||
clip = extracted_clips[clip_idx]
|
||||
|
||||
seg["visualType"] = "dialogue"
|
||||
seg["visualAssetIndex"] = None
|
||||
seg["duration"] = clip["duration"]
|
||||
seg["audioPath"] = clip["audioPath"]
|
||||
print(f" ✅ Updated segment {seg['id']} with original audio {clip['audioPath']}")
|
||||
|
||||
# 4. Save
|
||||
with open(PROPS_PATH, "w", encoding="utf-8") as f:
|
||||
json.dump(props, f, ensure_ascii=False, indent=2)
|
||||
|
||||
print(f"✨ Successfully updated {PROPS_PATH}. Please run Remotion to preview.")
|
||||
|
||||
if __name__ == "__main__":
|
||||
convert_dialogue_to_video()
|
||||
|
|
@ -8,8 +8,25 @@ This module handles:
|
|||
"""
|
||||
|
||||
import os
|
||||
import json
|
||||
import time
|
||||
from typing import Dict, Any, List, Optional
|
||||
from pathlib import Path
|
||||
from dotenv import load_dotenv
|
||||
|
||||
# Load environment variables
|
||||
load_dotenv()
|
||||
|
||||
try:
|
||||
from google import genai
|
||||
from google.genai import types
|
||||
except ImportError:
|
||||
genai = None
|
||||
|
||||
try:
|
||||
from openai import OpenAI
|
||||
except ImportError:
|
||||
OpenAI = None
|
||||
|
||||
|
||||
class PDFExtractor:
|
||||
|
|
@ -20,6 +37,12 @@ class PDFExtractor:
|
|||
if not self.api_key:
|
||||
raise ValueError("GEMINI_API_KEY not found in environment")
|
||||
|
||||
if genai is None:
|
||||
raise ImportError("google-genai not installed. Run: pip install google-genai")
|
||||
|
||||
self.client = genai.Client(api_key=self.api_key)
|
||||
self.model = 'gemini-2.0-flash-exp'
|
||||
|
||||
def extract(self, pdf_path: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Extract structured data from PDF
|
||||
|
|
@ -34,12 +57,106 @@ class PDFExtractor:
|
|||
- life_events: Major events (retirement, education, housing)
|
||||
- insurance_data: Current policies and recommendations
|
||||
"""
|
||||
# TODO: Implement Gemini Pro PDF parsing
|
||||
# 1. Upload PDF to Gemini API
|
||||
# 2. Use prompt engineering to extract structured JSON
|
||||
# 3. Validate against schema
|
||||
print(f" 📄 Uploading PDF to Gemini: {Path(pdf_path).name}...")
|
||||
|
||||
raise NotImplementedError("PDF extraction not yet implemented")
|
||||
# Upload PDF to Gemini
|
||||
with open(pdf_path, 'rb') as f:
|
||||
upload_config = types.UploadFileConfig(mime_type='application/pdf')
|
||||
pdf_file = self.client.files.upload(file=f, config=upload_config)
|
||||
|
||||
# Wait for processing
|
||||
while pdf_file.state == "PROCESSING":
|
||||
print(" ⏳ Processing PDF...")
|
||||
time.sleep(2)
|
||||
pdf_file = self.client.files.get(name=pdf_file.name)
|
||||
|
||||
if pdf_file.state == "FAILED":
|
||||
raise RuntimeError(f"PDF processing failed: {pdf_file.error}")
|
||||
|
||||
print(" ✓ PDF uploaded successfully")
|
||||
|
||||
# Extraction prompt
|
||||
prompt = """
|
||||
あなたはライフプラン報告書の専門家です。以下のPDFからライフプラン情報を抽出してください。
|
||||
|
||||
以下のJSON形式で出力してください:
|
||||
|
||||
```json
|
||||
{
|
||||
"client_name": "クライアント名",
|
||||
"current_age": 現在の年齢(数値),
|
||||
"retirement_age": 退職予定年齢(数値),
|
||||
"life_expectancy": 想定寿命(数値),
|
||||
"current_assets": 現在の資産総額(万円単位の数値),
|
||||
"annual_income": 年収(万円単位の数値),
|
||||
"annual_expenses": 年間支出(万円単位の数値),
|
||||
"family_structure": {
|
||||
"spouse": 配偶者の有無(boolean),
|
||||
"children": 子供の人数(数値)
|
||||
},
|
||||
"life_events": [
|
||||
{
|
||||
"year": イベント発生年,
|
||||
"age": その時の年齢,
|
||||
"event_type": "retirement|education|housing|marriage|other",
|
||||
"description": "イベントの説明",
|
||||
"cost": かかる費用(万円、任意)
|
||||
}
|
||||
],
|
||||
"financial_goals": [
|
||||
{
|
||||
"goal": "目標の説明",
|
||||
"target_amount": 目標金額(万円),
|
||||
"target_year": 達成目標年
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
JSONのみを出力し、その他の説明は不要です。
|
||||
PDFから情報が取得できない場合は、妥当な推定値を使用してください。
|
||||
"""
|
||||
|
||||
print(" 🤖 Analyzing PDF with Gemini...")
|
||||
|
||||
response = self.client.models.generate_content(
|
||||
model=self.model,
|
||||
contents=[pdf_file, prompt]
|
||||
)
|
||||
|
||||
# Parse response
|
||||
response_text = response.text.strip()
|
||||
|
||||
# Extract JSON from markdown code blocks if present
|
||||
if "```json" in response_text:
|
||||
json_start = response_text.find("```json") + 7
|
||||
json_end = response_text.find("```", json_start)
|
||||
response_text = response_text[json_start:json_end].strip()
|
||||
elif "```" in response_text:
|
||||
json_start = response_text.find("```") + 3
|
||||
json_end = response_text.find("```", json_start)
|
||||
response_text = response_text[json_start:json_end].strip()
|
||||
|
||||
try:
|
||||
extracted_data = json.loads(response_text)
|
||||
print(f" ✓ Extracted data for: {extracted_data.get('client_name', 'Unknown')}")
|
||||
return extracted_data
|
||||
except json.JSONDecodeError as e:
|
||||
print(f" ⚠️ Failed to parse JSON response: {e}")
|
||||
print(f" Raw response: {response_text[:200]}...")
|
||||
# Return fallback data
|
||||
return {
|
||||
"client_name": "山田太郎",
|
||||
"current_age": 35,
|
||||
"retirement_age": 65,
|
||||
"life_expectancy": 90,
|
||||
"current_assets": 5000,
|
||||
"annual_income": 600,
|
||||
"annual_expenses": 450,
|
||||
"family_structure": {"spouse": True, "children": 2},
|
||||
"life_events": [],
|
||||
"financial_goals": []
|
||||
}
|
||||
|
||||
|
||||
class VideoExtractor:
|
||||
|
|
@ -50,6 +167,11 @@ class VideoExtractor:
|
|||
if not self.api_key:
|
||||
raise ValueError("OPENAI_API_KEY not found in environment")
|
||||
|
||||
if OpenAI is None:
|
||||
raise ImportError("openai not installed. Run: pip install openai")
|
||||
|
||||
self.client = OpenAI(api_key=self.api_key)
|
||||
|
||||
def extract(self, video_path: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Extract audio transcription and meaningful timestamps
|
||||
|
|
@ -60,15 +182,68 @@ class VideoExtractor:
|
|||
Returns:
|
||||
Dictionary containing:
|
||||
- transcription: Full text transcription
|
||||
- timestamps: List of {time, text, keywords} objects
|
||||
- segments: Recommended video segments for B-Roll
|
||||
- segments: List of {start, end, text} segments with timestamps
|
||||
- keywords: Extracted keywords for B-Roll search
|
||||
"""
|
||||
# TODO: Implement Whisper transcription
|
||||
# 1. Extract audio from video using FFmpeg
|
||||
# 2. Run Whisper API
|
||||
# 3. Parse timestamps and keywords
|
||||
from backend.video_processor.utils import extract_audio
|
||||
|
||||
raise NotImplementedError("Video extraction not yet implemented")
|
||||
print(f" 🎥 Extracting audio from video: {Path(video_path).name}...")
|
||||
|
||||
# Extract audio using FFmpeg
|
||||
try:
|
||||
audio_path = extract_audio(video_path)
|
||||
print(f" ✓ Audio extracted to: {Path(audio_path).name}")
|
||||
except Exception as e:
|
||||
print(f" ❌ Audio extraction failed: {e}")
|
||||
raise
|
||||
|
||||
print(" 🎙️ Transcribing with Whisper...")
|
||||
|
||||
# Transcribe with Whisper
|
||||
try:
|
||||
with open(audio_path, 'rb') as audio_file:
|
||||
transcript = self.client.audio.transcriptions.create(
|
||||
model="whisper-1",
|
||||
file=audio_file,
|
||||
language="ja",
|
||||
response_format="verbose_json",
|
||||
timestamp_granularities=["segment"]
|
||||
)
|
||||
|
||||
print(f" ✓ Transcription complete ({len(transcript.segments)} segments)")
|
||||
|
||||
# Extract keywords using simple heuristics
|
||||
# In production, you'd use Gemini to extract semantic keywords
|
||||
keywords = self._extract_keywords(transcript.text)
|
||||
|
||||
return {
|
||||
"transcription": transcript.text,
|
||||
"duration": transcript.duration if hasattr(transcript, 'duration') else 0,
|
||||
"segments": [
|
||||
{
|
||||
"start": seg.start if hasattr(seg, 'start') else 0,
|
||||
"end": seg.end if hasattr(seg, 'end') else 0,
|
||||
"text": seg.text.strip() if hasattr(seg, 'text') else ""
|
||||
}
|
||||
for seg in transcript.segments
|
||||
],
|
||||
"keywords": keywords
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
print(f" ❌ Whisper transcription failed: {e}")
|
||||
raise
|
||||
|
||||
def _extract_keywords(self, text: str) -> List[str]:
|
||||
"""Extract keywords from transcription text (simple heuristic)"""
|
||||
# Common life plan keywords in Japanese
|
||||
common_keywords = [
|
||||
"家族", "老後", "退職", "貯蓄", "資産", "住宅",
|
||||
"教育", "保険", "年金", "投資", "将来", "計画"
|
||||
]
|
||||
|
||||
found_keywords = [kw for kw in common_keywords if kw in text]
|
||||
return found_keywords[:5] # Limit to top 5
|
||||
|
||||
|
||||
def extract_pdf_data(pdf_path: str) -> Dict[str, Any]:
|
||||
|
|
|
|||
|
|
@ -0,0 +1,37 @@
|
|||
import json
|
||||
from pathlib import Path
|
||||
|
||||
def fix_script_ids():
|
||||
"""
|
||||
Specifically re-indexes all script segments in render_props.json
|
||||
and updates audioPath to match the new ID.
|
||||
"""
|
||||
props_path = Path("assets/1_processed/render_props.json")
|
||||
if not props_path.exists():
|
||||
print("❌ Error: render_props.json not found.")
|
||||
return
|
||||
|
||||
print(f"📖 Reading {props_path}...")
|
||||
with open(props_path, "r", encoding="utf-8") as f:
|
||||
data = json.load(f)
|
||||
|
||||
print("🔢 Re-indexing and updating audio paths...")
|
||||
for i, segment in enumerate(data["script"]):
|
||||
new_id = i + 1
|
||||
segment["id"] = new_id
|
||||
|
||||
# Only update audioPath if it's not a video clip (which uses original audio)
|
||||
if segment.get("visualType") != "video":
|
||||
segment["audioPath"] = f"public/segment_{new_id:03d}.mp3"
|
||||
else:
|
||||
# Video clips don't have audioPath in the same way (they use the video asset)
|
||||
if "audioPath" in segment:
|
||||
del segment["audioPath"]
|
||||
|
||||
with open(props_path, "w", encoding="utf-8") as f:
|
||||
json.dump(data, f, ensure_ascii=False, indent=2)
|
||||
|
||||
print(f"✅ IDs and audio paths fixed in {props_path}.")
|
||||
|
||||
if __name__ == "__main__":
|
||||
fix_script_ids()
|
||||
|
|
@ -46,7 +46,7 @@ class KeywordTranslator:
|
|||
"""Translate Japanese keywords to English using Gemini"""
|
||||
|
||||
def __init__(self, cache_file: str = "assets/1_processed/translation_cache.json"):
|
||||
self.model = 'gemini-3-pro-preview' # Latest model for translation
|
||||
self.model = 'gemini-2.0-flash-exp' # Latest model for translation
|
||||
self.cache_file = Path(cache_file)
|
||||
self.cache = self._load_cache()
|
||||
console.print("[green]✓[/green] Initialized Gemini translator with cache")
|
||||
|
|
|
|||
|
|
@ -8,38 +8,201 @@ This module fetches B-Roll footage based on keywords extracted from:
|
|||
"""
|
||||
|
||||
import os
|
||||
import json
|
||||
import requests
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any, List, Optional
|
||||
|
||||
try:
|
||||
from google import genai
|
||||
from google.genai import types
|
||||
except ImportError:
|
||||
genai = None
|
||||
|
||||
|
||||
class MediaFetcher:
|
||||
"""Fetch B-Roll videos from Pexels API"""
|
||||
|
||||
def __init__(self, api_key: Optional[str] = None):
|
||||
def __init__(self, api_key: Optional[str] = None, gemini_key: Optional[str] = None):
|
||||
self.api_key = api_key or os.getenv("PEXELS_API_KEY")
|
||||
if not self.api_key:
|
||||
raise ValueError("PEXELS_API_KEY not found in environment")
|
||||
|
||||
self.gemini_key = gemini_key or os.getenv("GEMINI_API_KEY")
|
||||
self.base_url = "https://api.pexels.com/videos/search"
|
||||
self.output_dir = Path("public/broll")
|
||||
self.output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
def fetch_broll(self, keywords: List[str], max_per_keyword: int = 3) -> Dict[str, List[str]]:
|
||||
# Translation cache
|
||||
self.cache_file = Path("assets/1_processed/translation_cache.json")
|
||||
self.translation_cache = self._load_cache()
|
||||
|
||||
# Initialize Gemini for translation
|
||||
if self.gemini_key and genai:
|
||||
self.gemini_client = genai.Client(api_key=self.gemini_key)
|
||||
else:
|
||||
self.gemini_client = None
|
||||
|
||||
def _load_cache(self) -> Dict[str, str]:
|
||||
"""Load translation cache from file"""
|
||||
if self.cache_file.exists():
|
||||
with open(self.cache_file, 'r', encoding='utf-8') as f:
|
||||
return json.load(f)
|
||||
return {}
|
||||
|
||||
def _save_cache(self):
|
||||
"""Save translation cache to file"""
|
||||
self.cache_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(self.cache_file, 'w', encoding='utf-8') as f:
|
||||
json.dump(self.translation_cache, f, ensure_ascii=False, indent=2)
|
||||
|
||||
def _translate_to_english(self, japanese_keywords: List[str]) -> Dict[str, str]:
|
||||
"""Translate Japanese keywords to English using Gemini"""
|
||||
translations = {}
|
||||
keywords_to_translate = [kw for kw in japanese_keywords if kw not in self.translation_cache]
|
||||
|
||||
if not keywords_to_translate:
|
||||
return {kw: self.translation_cache[kw] for kw in japanese_keywords}
|
||||
|
||||
if self.gemini_client:
|
||||
prompt = f"""
|
||||
Translate the following Japanese keywords to English (single words or short phrases).
|
||||
Return ONLY a JSON object mapping Japanese to English.
|
||||
|
||||
Keywords: {', '.join(keywords_to_translate)}
|
||||
|
||||
Example format:
|
||||
{{
|
||||
"家族": "family",
|
||||
"老後": "retirement"
|
||||
}}
|
||||
"""
|
||||
try:
|
||||
response = self.gemini_client.models.generate_content(
|
||||
model='gemini-2.0-flash-exp',
|
||||
contents=prompt
|
||||
)
|
||||
|
||||
response_text = response.text.strip()
|
||||
if "```json" in response_text:
|
||||
json_start = response_text.find("```json") + 7
|
||||
json_end = response_text.find("```", json_start)
|
||||
response_text = response_text[json_start:json_end].strip()
|
||||
elif "```" in response_text:
|
||||
json_start = response_text.find("```") + 3
|
||||
json_end = response_text.find("```", json_start)
|
||||
response_text = response_text[json_start:json_end].strip()
|
||||
|
||||
new_translations = json.loads(response_text)
|
||||
self.translation_cache.update(new_translations)
|
||||
self._save_cache()
|
||||
|
||||
except Exception as e:
|
||||
print(f" ⚠️ Translation failed: {e}, using fallback...")
|
||||
# Fallback translations
|
||||
fallback = {
|
||||
"家族": "family", "老後": "retirement", "退職": "retirement",
|
||||
"貯蓄": "savings", "資産": "finance", "住宅": "housing",
|
||||
"教育": "education", "保険": "insurance", "年金": "pension",
|
||||
"投資": "investment", "将来": "future", "計画": "planning"
|
||||
}
|
||||
for kw in keywords_to_translate:
|
||||
self.translation_cache[kw] = fallback.get(kw, kw)
|
||||
else:
|
||||
# No Gemini, use fallback
|
||||
fallback = {
|
||||
"家族": "family", "老後": "retirement", "退職": "retirement",
|
||||
"貯蓄": "savings", "資産": "finance", "住宅": "housing",
|
||||
"教育": "education", "保険": "insurance", "年金": "pension",
|
||||
"投資": "investment", "将来": "future", "計画": "planning"
|
||||
}
|
||||
for kw in keywords_to_translate:
|
||||
self.translation_cache[kw] = fallback.get(kw, kw)
|
||||
|
||||
return {kw: self.translation_cache.get(kw, kw) for kw in japanese_keywords}
|
||||
|
||||
def fetch_broll(self, keywords: List[str], max_per_keyword: int = 2) -> Dict[str, List[Dict[str, Any]]]:
|
||||
"""
|
||||
Fetch B-Roll videos based on keywords
|
||||
|
||||
Args:
|
||||
keywords: List of Japanese keywords (will be translated to English)
|
||||
keywords: List of Japanese keywords
|
||||
max_per_keyword: Maximum videos to fetch per keyword
|
||||
|
||||
Returns:
|
||||
Dictionary mapping keywords to video URLs
|
||||
Dictionary mapping Japanese keywords to list of video info
|
||||
"""
|
||||
# TODO: Implement Pexels API integration
|
||||
# 1. Translate Japanese keywords to English (using Gemini)
|
||||
# 2. Search Pexels for each keyword
|
||||
# 3. Download videos to assets/1_processed/broll/
|
||||
# 4. Return local file paths
|
||||
print(f" 🎬 Fetching B-Roll for keywords: {', '.join(keywords)}")
|
||||
|
||||
raise NotImplementedError("B-Roll fetching not yet implemented")
|
||||
# Translate keywords
|
||||
translations = self._translate_to_english(keywords)
|
||||
print(f" 🌎 Translations: {translations}")
|
||||
|
||||
results = {}
|
||||
|
||||
for jp_keyword, en_keyword in translations.items():
|
||||
print(f" 🔍 Searching Pexels for '{en_keyword}'...")
|
||||
|
||||
# Search Pexels
|
||||
headers = {"Authorization": self.api_key}
|
||||
params = {
|
||||
"query": en_keyword,
|
||||
"per_page": max_per_keyword,
|
||||
"orientation": "landscape"
|
||||
}
|
||||
|
||||
try:
|
||||
response = requests.get(self.base_url, headers=headers, params=params)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
videos = []
|
||||
for video in data.get("videos", [])[:max_per_keyword]:
|
||||
# Download HD video file
|
||||
video_files = video.get("video_files", [])
|
||||
hd_file = None
|
||||
|
||||
# Find HD version
|
||||
for vf in video_files:
|
||||
if vf.get("quality") == "hd" and vf.get("width", 0) <= 1920:
|
||||
hd_file = vf
|
||||
break
|
||||
|
||||
if not hd_file and video_files:
|
||||
hd_file = video_files[0] # Fallback to first available
|
||||
|
||||
if hd_file:
|
||||
# Download video
|
||||
video_url = hd_file["link"]
|
||||
filename = f"{en_keyword.title()}_{video['id']}.mp4"
|
||||
filepath = self.output_dir / filename
|
||||
|
||||
if not filepath.exists():
|
||||
print(f" ⬇️ Downloading {filename}...")
|
||||
video_response = requests.get(video_url, stream=True)
|
||||
with open(filepath, 'wb') as f:
|
||||
for chunk in video_response.iter_content(chunk_size=8192):
|
||||
f.write(chunk)
|
||||
else:
|
||||
print(f" ✓ Already cached: {filename}")
|
||||
|
||||
videos.append({
|
||||
"filepath": str(filepath),
|
||||
"duration": video.get("duration", 0),
|
||||
"photographer": video.get("user", {}).get("name", "Unknown")
|
||||
})
|
||||
|
||||
results[jp_keyword] = videos
|
||||
print(f" ✓ Found {len(videos)} videos for '{jp_keyword}'")
|
||||
|
||||
time.sleep(0.5) # Rate limiting
|
||||
|
||||
except requests.exceptions.RequestException as e:
|
||||
print(f" ⚠️ Pexels API error for '{en_keyword}': {e}")
|
||||
results[jp_keyword] = []
|
||||
|
||||
return results
|
||||
|
||||
def _translate_keyword(self, japanese_keyword: str) -> str:
|
||||
"""Translate Japanese keyword to English using Gemini"""
|
||||
|
|
@ -63,9 +226,18 @@ def fetch_broll(script: Dict[str, Any]) -> Dict[str, Any]:
|
|||
"""Convenience function for fetching B-Roll assets"""
|
||||
# Extract keywords from script
|
||||
keywords = []
|
||||
for segment in script.get("segments", []):
|
||||
# TODO: Extract keywords from segment text
|
||||
pass
|
||||
|
||||
# Try to extract from video_data if available
|
||||
if "video_data" in script and "keywords" in script["video_data"]:
|
||||
keywords.extend(script["video_data"]["keywords"])
|
||||
|
||||
# Add default keywords if none found
|
||||
if not keywords:
|
||||
keywords = ["家族", "老後", "貯蓄"]
|
||||
|
||||
try:
|
||||
fetcher = MediaFetcher()
|
||||
return fetcher.fetch_broll(keywords)
|
||||
return fetcher.fetch_broll(keywords, max_per_keyword=2)
|
||||
except Exception as e:
|
||||
print(f" ⚠️ B-Roll fetching failed: {e}")
|
||||
return {}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,260 @@
|
|||
"""
|
||||
PowerPoint presentation extractor
|
||||
|
||||
This module handles:
|
||||
- Extracting slides from PPTX files
|
||||
- Converting slides to images
|
||||
- Extracting text content from slides
|
||||
- Matching slides to script segments
|
||||
"""
|
||||
|
||||
import os
|
||||
from typing import Dict, Any, List, Optional
|
||||
from pathlib import Path
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
|
||||
try:
|
||||
from pptx import Presentation
|
||||
except ImportError:
|
||||
Presentation = None
|
||||
|
||||
try:
|
||||
from PIL import Image
|
||||
import io
|
||||
except ImportError:
|
||||
Image = None
|
||||
|
||||
|
||||
class PPTExtractor:
|
||||
"""Extract slides and content from PowerPoint presentations"""
|
||||
|
||||
def __init__(self):
|
||||
if Presentation is None:
|
||||
raise ImportError("python-pptx not installed. Run: pip install python-pptx")
|
||||
if Image is None:
|
||||
raise ImportError("Pillow not installed. Run: pip install Pillow")
|
||||
|
||||
def extract(self, ppt_path: str, output_dir: Optional[str] = None) -> Dict[str, Any]:
|
||||
"""
|
||||
Extract slides from PowerPoint presentation
|
||||
|
||||
Args:
|
||||
ppt_path: Path to PPTX file
|
||||
output_dir: Directory to save slide images (default: assets/1_processed/slides)
|
||||
|
||||
Returns:
|
||||
Dictionary containing slide information:
|
||||
{
|
||||
"total_slides": int,
|
||||
"slides": [
|
||||
{
|
||||
"index": 0,
|
||||
"title": str,
|
||||
"text": str,
|
||||
"image_path": str,
|
||||
"notes": str
|
||||
},
|
||||
...
|
||||
]
|
||||
}
|
||||
"""
|
||||
ppt_path = Path(ppt_path)
|
||||
if not ppt_path.exists():
|
||||
raise FileNotFoundError(f"PPT file not found: {ppt_path}")
|
||||
|
||||
# Set output directory
|
||||
if output_dir is None:
|
||||
# CRITICAL: Output directly to public/ for Remotion to find them via staticFile
|
||||
output_dir = Path("public")
|
||||
else:
|
||||
output_dir = Path(output_dir)
|
||||
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
print(f" 📊 Loading PowerPoint: {ppt_path.name}")
|
||||
prs = Presentation(str(ppt_path))
|
||||
|
||||
slides_data = []
|
||||
for idx, slide in enumerate(prs.slides):
|
||||
slide_info = self._extract_slide_content(slide, idx)
|
||||
|
||||
# Export slide as image using LibreOffice/PowerPoint
|
||||
image_path = output_dir / f"slide_{idx + 1:03d}.png"
|
||||
slide_info["image_path"] = str(image_path)
|
||||
|
||||
slides_data.append(slide_info)
|
||||
print(f" ✓ Extracted slide {idx + 1}: {slide_info['title'][:50]}...")
|
||||
|
||||
result = {
|
||||
"total_slides": len(slides_data),
|
||||
"slides": slides_data,
|
||||
"source_file": str(ppt_path)
|
||||
}
|
||||
|
||||
print(f" ✓ Extracted {len(slides_data)} slides from PowerPoint")
|
||||
|
||||
# Export slides as images
|
||||
self._export_slides_as_images(str(ppt_path), output_dir)
|
||||
|
||||
return result
|
||||
|
||||
def _extract_slide_content(self, slide, index: int) -> Dict[str, Any]:
|
||||
"""Extract text content from a single slide"""
|
||||
slide_info = {
|
||||
"index": index,
|
||||
"title": "",
|
||||
"text": "",
|
||||
"notes": ""
|
||||
}
|
||||
|
||||
# Extract title (usually the first text box)
|
||||
shapes_text = []
|
||||
for shape in slide.shapes:
|
||||
if hasattr(shape, "text") and shape.text:
|
||||
text = shape.text.strip()
|
||||
if text:
|
||||
if not slide_info["title"]:
|
||||
slide_info["title"] = text
|
||||
else:
|
||||
shapes_text.append(text)
|
||||
|
||||
# Combine all text
|
||||
slide_info["text"] = "\n".join(shapes_text)
|
||||
|
||||
# Extract notes
|
||||
if slide.has_notes_slide:
|
||||
notes_slide = slide.notes_slide
|
||||
if notes_slide.notes_text_frame:
|
||||
slide_info["notes"] = notes_slide.notes_text_frame.text.strip()
|
||||
|
||||
return slide_info
|
||||
|
||||
def _export_slides_as_images(self, ppt_path: str, output_dir: Path):
|
||||
"""
|
||||
Export slides as PNG images
|
||||
Tries multiple methods in order:
|
||||
1. Windows PowerPoint COM (if on Windows)
|
||||
2. LibreOffice (cross-platform)
|
||||
3. Placeholder images (fallback)
|
||||
"""
|
||||
import sys
|
||||
import subprocess
|
||||
|
||||
# Method 1: Try Windows PowerPoint COM
|
||||
if sys.platform == 'win32':
|
||||
try:
|
||||
print(" 🖼️ Converting slides to images with PowerPoint...")
|
||||
success = self._export_with_powerpoint_com(ppt_path, output_dir)
|
||||
if success:
|
||||
print(" ✓ Slides exported as images")
|
||||
return
|
||||
except Exception as e:
|
||||
print(f" ⚠️ PowerPoint COM failed: {e}")
|
||||
|
||||
# Method 2: Try LibreOffice
|
||||
try:
|
||||
print(" 🖼️ Converting slides to images with LibreOffice...")
|
||||
subprocess.run([
|
||||
"soffice",
|
||||
"--headless",
|
||||
"--convert-to", "png",
|
||||
"--outdir", str(output_dir),
|
||||
ppt_path
|
||||
], check=True, capture_output=True, timeout=60)
|
||||
print(" ✓ Slides exported as images")
|
||||
return
|
||||
except (subprocess.CalledProcessError, FileNotFoundError, subprocess.TimeoutExpired):
|
||||
pass
|
||||
|
||||
# Method 3: Generate placeholders
|
||||
print(" ⚠️ No conversion tool available, generating placeholder images...")
|
||||
self._generate_placeholder_images(output_dir)
|
||||
|
||||
def _export_with_powerpoint_com(self, ppt_path: str, output_dir: Path) -> bool:
|
||||
"""
|
||||
Export slides using Windows PowerPoint COM interface
|
||||
Returns True if successful, False otherwise
|
||||
"""
|
||||
try:
|
||||
import win32com.client
|
||||
import pythoncom
|
||||
except ImportError:
|
||||
print(" ⚠️ pywin32 not installed. Run: pip install pywin32")
|
||||
return False
|
||||
|
||||
try:
|
||||
pythoncom.CoInitialize()
|
||||
powerpoint = win32com.client.Dispatch("PowerPoint.Application")
|
||||
powerpoint.Visible = 1
|
||||
|
||||
# Ensure output directory exists
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Open presentation (use absolute path)
|
||||
prs = powerpoint.Presentations.Open(str(Path(ppt_path).absolute()), WithWindow=False)
|
||||
|
||||
# Export each slide as PNG
|
||||
for i, slide in enumerate(prs.Slides):
|
||||
output_file = output_dir / f"slide_{i + 1:03d}.png"
|
||||
# Use absolute path for export
|
||||
slide.Export(str(output_file.absolute()), "PNG", 1920, 1080)
|
||||
|
||||
# Close presentation
|
||||
prs.Close()
|
||||
powerpoint.Quit()
|
||||
pythoncom.CoUninitialize()
|
||||
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f" ⚠️ PowerPoint COM error: {e}")
|
||||
try:
|
||||
powerpoint.Quit()
|
||||
pythoncom.CoUninitialize()
|
||||
except:
|
||||
pass
|
||||
return False
|
||||
|
||||
def _generate_placeholder_images(self, output_dir: Path):
|
||||
"""Generate placeholder slide images"""
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
|
||||
# Create placeholder images for slides
|
||||
slide_files = list(output_dir.glob("slide_*.png"))
|
||||
if not slide_files:
|
||||
# Generate at least one placeholder
|
||||
img = Image.new('RGB', (1920, 1080), color='white')
|
||||
draw = ImageDraw.Draw(img)
|
||||
|
||||
# Draw simple text
|
||||
text = "PowerPoint Slide"
|
||||
draw.text((960, 540), text, fill='black', anchor="mm")
|
||||
|
||||
img.save(output_dir / "slide_001.png")
|
||||
print(" ✓ Generated placeholder slide image")
|
||||
|
||||
|
||||
def extract_ppt_data(ppt_path: str) -> Dict[str, Any]:
|
||||
"""Convenience function for PPT extraction"""
|
||||
extractor = PPTExtractor()
|
||||
return extractor.extract(ppt_path)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Test extraction
|
||||
import sys
|
||||
|
||||
if len(sys.argv) < 2:
|
||||
print("Usage: python ppt_extractor.py <path_to_pptx>")
|
||||
sys.exit(1)
|
||||
|
||||
ppt_path = sys.argv[1]
|
||||
result = extract_ppt_data(ppt_path)
|
||||
|
||||
print(f"\n📊 Extracted {result['total_slides']} slides")
|
||||
for slide in result['slides']:
|
||||
print(f"\nSlide {slide['index'] + 1}:")
|
||||
print(f" Title: {slide['title']}")
|
||||
print(f" Text length: {len(slide['text'])} chars")
|
||||
print(f" Image: {slide['image_path']}")
|
||||
|
|
@ -0,0 +1,57 @@
|
|||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Add project root to sys.path to allow importing backend modules
|
||||
project_root = Path(__file__).parent.parent
|
||||
sys.path.append(str(project_root))
|
||||
|
||||
from backend.audio_generator import generate_audio_for_script
|
||||
|
||||
def rebuild_audio_from_props():
|
||||
"""
|
||||
Reads assets/1_processed/render_props.json, generates new audio files
|
||||
for the narration text, and updates the durations and paths.
|
||||
"""
|
||||
props_path = project_root / "assets" / "1_processed" / "render_props.json"
|
||||
|
||||
if not props_path.exists():
|
||||
print(f"❌ Error: {props_path} not found.")
|
||||
return
|
||||
|
||||
print(f"📖 Reading script from {props_path}...")
|
||||
with open(props_path, "r", encoding="utf-8") as f:
|
||||
props = json.load(f)
|
||||
|
||||
# Prepare script data for the generator
|
||||
# RE-INDEX segments first
|
||||
print("🔢 Re-indexing script segments and fixing paths...")
|
||||
for i, segment in enumerate(props["script"]):
|
||||
new_id = i + 1
|
||||
segment["id"] = new_id
|
||||
if segment.get("visualType") != "video":
|
||||
segment["audioPath"] = f"public/segment_{new_id:03d}.mp3"
|
||||
elif "audioPath" in segment:
|
||||
del segment["audioPath"]
|
||||
|
||||
script_data = {"segments": props["script"]}
|
||||
|
||||
# Generate new audio files (outputs to public/ folder)
|
||||
print("🎙️ Generating new audio files...")
|
||||
updated_script_data = generate_audio_for_script(script_data)
|
||||
|
||||
# Update the script in our props object
|
||||
props["script"] = updated_script_data["segments"]
|
||||
|
||||
# Recalculate total duration if needed (Remotion uses segment durations)
|
||||
total_duration = sum(seg.get("duration", 0) for seg in props["script"])
|
||||
print(f"⏱️ New total duration: {total_duration:.2f} seconds")
|
||||
|
||||
# Save updated render_props.json
|
||||
with open(props_path, "w", encoding="utf-8") as f:
|
||||
json.dump(props, f, ensure_ascii=False, indent=2)
|
||||
|
||||
print(f"✅ Successfully updated {props_path} with new audio and durations.")
|
||||
|
||||
if __name__ == "__main__":
|
||||
rebuild_audio_from_props()
|
||||
|
|
@ -9,10 +9,12 @@ This module generates natural Japanese narration scripts based on:
|
|||
|
||||
import os
|
||||
import json
|
||||
import re
|
||||
from typing import Dict, Any, List, Optional
|
||||
|
||||
try:
|
||||
import google.generativeai as genai
|
||||
from google import genai
|
||||
from google.genai import types
|
||||
except ImportError:
|
||||
genai = None
|
||||
|
||||
|
|
@ -27,12 +29,12 @@ class ScriptGenerator:
|
|||
|
||||
if genai is None:
|
||||
raise ImportError(
|
||||
"google-generativeai not installed. "
|
||||
"Run: pip install google-generativeai"
|
||||
"google-genai not installed. "
|
||||
"Run: pip install google-genai"
|
||||
)
|
||||
|
||||
genai.configure(api_key=self.api_key)
|
||||
self.model = genai.GenerativeModel('gemini-1.5-pro')
|
||||
self.client = genai.Client(api_key=self.api_key)
|
||||
self.model = 'gemini-2.0-flash-exp'
|
||||
|
||||
def generate(
|
||||
self,
|
||||
|
|
@ -40,27 +42,27 @@ class ScriptGenerator:
|
|||
video_data: Optional[Dict[str, Any]] = None,
|
||||
style: str = "friendly"
|
||||
) -> Dict[str, Any]:
|
||||
"""Generate narration script
|
||||
"""Generate narration script"""
|
||||
print(f" Generating script with Gemini Pro for client: {pdf_data.get('client_name', 'Unknown')}...")
|
||||
|
||||
Args:
|
||||
pdf_data: Structured data from PDF extraction
|
||||
video_data: Optional video transcription data
|
||||
style: Narration style ('friendly', 'professional', 'casual')
|
||||
|
||||
Returns:
|
||||
Dictionary containing:
|
||||
- segments: List of narration segments
|
||||
- total_duration: Estimated total duration in seconds
|
||||
- sync_points: Recommended chart/video sync points
|
||||
"""
|
||||
print(" Generating script with Gemini Pro...")
|
||||
# Merge Video Data into PDF Data for prompt building
|
||||
merged_data = {**pdf_data}
|
||||
if video_data:
|
||||
merged_data["transcription"] = video_data.get("transcription", "")
|
||||
merged_data["clips"] = video_data.get("clips", [])
|
||||
|
||||
# Build the prompt
|
||||
prompt = self._build_prompt(pdf_data, style)
|
||||
prompt = self._build_prompt(merged_data, style)
|
||||
|
||||
# DEBUG: Print the first part of the prompt
|
||||
# print(f"--- DEBUG PROMPT START ---\n{prompt[:500]}\n--- DEBUG PROMPT END ---")
|
||||
|
||||
try:
|
||||
# Call Gemini API
|
||||
response = self.model.generate_content(prompt)
|
||||
response = self.client.models.generate_content(
|
||||
model=self.model,
|
||||
contents=prompt
|
||||
)
|
||||
|
||||
# Parse the response
|
||||
script_data = self._parse_response(response.text)
|
||||
|
|
@ -76,15 +78,60 @@ class ScriptGenerator:
|
|||
def _build_prompt(self, pdf_data: Dict, style: str) -> str:
|
||||
"""Build the prompt for Gemini Pro"""
|
||||
client_name = pdf_data.get("client_name", "お客様")
|
||||
current_age = pdf_data.get("current_age", 35)
|
||||
retirement_age = pdf_data.get("retirement_age", 65)
|
||||
current_assets = pdf_data.get("current_assets", 5000)
|
||||
annual_income = pdf_data.get("annual_income", 600)
|
||||
annual_expenses = pdf_data.get("annual_expenses", 450)
|
||||
current_age = pdf_data.get("current_age", 47)
|
||||
retirement_age = pdf_data.get("retirement_age", 60)
|
||||
current_assets = pdf_data.get("current_assets", 907)
|
||||
annual_income = pdf_data.get("annual_income", 703)
|
||||
annual_expenses = pdf_data.get("annual_expenses", 288)
|
||||
life_events = pdf_data.get("life_events", [])
|
||||
|
||||
# Transcription and Clips
|
||||
transcription = pdf_data.get("transcription", "")
|
||||
clips = pdf_data.get("clips", [])
|
||||
|
||||
# Include PPT data if available
|
||||
ppt_info = ""
|
||||
presentation = pdf_data.get("presentation", {})
|
||||
if presentation and "slides" in presentation:
|
||||
ppt_info = "\n【PPTスライド内容】\n"
|
||||
for slide in presentation["slides"]:
|
||||
ppt_info += f"- スライド {slide['index']}: タイトル: {slide['title']}\n"
|
||||
if slide.get("text"):
|
||||
ppt_info += f" 内容: {slide['text'][:400]}\n"
|
||||
if slide.get("notes"):
|
||||
ppt_info += f" ノート: {slide['notes'][:400]}\n"
|
||||
|
||||
# Life events info
|
||||
events_info = ""
|
||||
if life_events:
|
||||
events_info = "\n【ライフイベント予定】\n"
|
||||
for ev in life_events:
|
||||
events_info += f"- {ev.get('year')}年 ({ev.get('age')}歳): {ev.get('description')} (費用: {ev.get('cost', 0)}万円)\n"
|
||||
|
||||
# Video Transcription and Clips info
|
||||
video_info = ""
|
||||
if transcription:
|
||||
video_info = f"\n【インタビュー動画の内容】\n{transcription[:2000]}\n"
|
||||
|
||||
if clips:
|
||||
video_info += "\n【利用可能なビデオクリップ】\n"
|
||||
for i, clip in enumerate(clips):
|
||||
video_info += f"- クリップ {i}: 「{clip.get('text', '')[:100]}...」\n"
|
||||
|
||||
prompt = f"""
|
||||
あなたはファイナンシャルプランナーのナレーション原稿作成アシスタントです。
|
||||
以下の顧客情報に基づいて、3〜5分の解説動画用のナレーション原稿を作成してください。
|
||||
あなたはプロのファイナンシャルプランナーであり、親しみやすい動画解説者です。
|
||||
以下の顧客情報、ライフイベント、PPTスライド、そしてインタビュー動画の転記内容に基づいて、解説動画用の完結なナレーション原稿を作成してください。
|
||||
|
||||
【厳格な指示】
|
||||
1. 抽象的な表現（「〇〇」や「△△」）や、メタ発言（「スライドの内容がありません」など）は絶対に出力しないでください。
|
||||
2. インタビュー内容やPDFの数値を最大限に活用し、事実に基づいた具体的な解説を行ってください。
|
||||
3. 各セグメントの视覚素材（`visualType`）は、以下から適切に選択してください：
|
||||
- `ppt`: PPTスライドを表示する場合。`visualAssetIndex` を指定してください（基本はこちらを優先）。
|
||||
- `chart`: 資産推移や収支の動的なグラフを表示する場合。シミュレーションの結果を具体的に説明するシーンで使用します。
|
||||
- `video`: 本人のインタビュー映像を表示する場合。本人の肉声や想いを伝えるシーンで使用します。`visualAssetIndex` にクリップの番号を指定してください。
|
||||
- `dialogue`: 文字による対話形式（インタビューの再現）を表示する場合。動画の冒頭でインタビューのやり取りをテキストで再現する際に使用します。`speaker` 属性に話者名を指定してください。
|
||||
4. ナレーションの中に「スライド〇〇をご覧ください」や「インタビューでもおっしゃっていた通り...」といった言葉を入れ、視聴者の視線を誘導してください。
|
||||
5. 真人インタビュー（`video` タイプ）を使用する場合、ナレーションテキスト（`text` 属性）はそのクリップで本人が話している内容と一致させてください。
|
||||
|
||||
【顧客情報】
|
||||
- お名前: {client_name}様
|
||||
|
|
@ -93,60 +140,95 @@ class ScriptGenerator:
|
|||
- 現在の資産: {current_assets}万円
|
||||
- 年収: {annual_income}万円
|
||||
- 年間支出: {annual_expenses}万円
|
||||
{events_info}
|
||||
{ppt_info}
|
||||
{video_info}
|
||||
|
||||
【原稿作成の指示】
|
||||
1. 合計3〜5分（180〜300秒）の原稿を作成してください
|
||||
2. 以下のセグメントに分割してください：
|
||||
- オープニング（挨拶と概要説明）
|
||||
- 現在の状況説明
|
||||
- 将来の資産推移の解説
|
||||
- 退職後の生活資金について
|
||||
- リスクと対策の提案
|
||||
- クロージング（まとめと安心感）
|
||||
【原稿構成のフロー】
|
||||
動画全体で少なくとも 12〜18 のセグメントを作成し、以下の流れで構成してください：
|
||||
1. 文字対話オープニング（visualType: dialogue）：インタビュー動画から、ライフプラン相談のきっかけや想いがわかる重要なやり取りを 3〜5 セグメントほど抽出して配置してください。
|
||||
2. 動画全体の導入（Slide 0）：インパクトのある挨拶。
|
||||
3. 自己紹介と現状（Slide 1, 2...）：家計の現状をスライドに沿って。
|
||||
3. ライフイベント（Slide 3, 4...）：具体的なイベント内容の解説。
|
||||
4. インタビュー挿入（Video）：本人の想いや不安、决意を语るシーン。
|
||||
5. シミュレーション詳細（Chart）：動的なグラフ（資産推移・収支明細）を使い、何年後に資産がどうなるかを具体的に説明。
|
||||
6. 対策とアドバイス（Slide 5, 6...）：改善策や提案をスライドで。
|
||||
7. クロージング（Slide 7...）：まとめと個別相談の案内。
|
||||
|
||||
3. 各セグメントには以下を含めてください：
|
||||
- id: セグメント番号（1から開始）
|
||||
- text: ナレーション文（自然な日本語、敬語）
|
||||
- duration: 推定時間（秒）
|
||||
- type: シーンタイプ（"narration", "chart_intro", "chart_highlight", "transition"）
|
||||
- chartSync: チャート表示の指示（該当する場合のみ）
|
||||
【素材使用の鉄則 - 必ず守ること】
|
||||
- **PPTスライドの全消化**: 提供されたスライド（Slide 0 から最後のリファレンスまで）を、内容に合わせて**必ず全て最低1回は表示**してください。スライドを飛ばさないでください。
|
||||
- **ハイブリッド構成**: 「PPTで事実を伝え、Chartでデータを動かし、Videoで本人の声を伝える」というリズムを意識してください。
|
||||
|
||||
4. トーンは{style}（親しみやすく、でも専門的）にしてください
|
||||
|
||||
5. 出力はJSON形式で、以下の構造で返してください：
|
||||
【出力形式】
|
||||
純粋なJSON形式のみで返してください。コードブロックは不要です。
|
||||
{{
|
||||
"segments": [
|
||||
{{
|
||||
"id": 1,
|
||||
"text": "ナレーション文",
|
||||
"duration": 3.5,
|
||||
"type": "narration",
|
||||
"chartSync": {{
|
||||
"chartId": "assetDepletion",
|
||||
"highlightYear": 2025
|
||||
}}
|
||||
"text": "読み上げる文章",
|
||||
"insight": "画面に表示する短い要約・洞察（20文字程度）",
|
||||
"targetYear": 2025,
|
||||
"duration": 秒数,
|
||||
"type": "narration" | "dialogue" | "chart_intro" | "chart_highlight" | "video_clip",
|
||||
"visualType": "ppt" | "video" | "chart" | "dialogue",
|
||||
"visualAssetIndex": 0,
|
||||
"speaker": "話者名",
|
||||
"brollKeyword": "キーワード"
|
||||
}}
|
||||
]
|
||||
}}
|
||||
|
||||
必ずJSON形式のみを返してください。説明文は不要です。
|
||||
"""
|
||||
|
||||
return prompt
|
||||
|
||||
def _parse_response(self, response_text: str) -> Dict[str, Any]:
|
||||
"""Parse Gemini's JSON response"""
|
||||
# Remove markdown code blocks if present
|
||||
# Remove markdown code blocks and any extra text
|
||||
text = response_text.strip()
|
||||
if text.startswith("```json"):
|
||||
text = text[7:]
|
||||
if text.startswith("```"):
|
||||
text = text[3:]
|
||||
if text.endswith("```"):
|
||||
text = text[:-3]
|
||||
text = text.strip()
|
||||
|
||||
# Try to find JSON block
|
||||
if "{" in text and "}" in text:
|
||||
start = text.find("{")
|
||||
end = text.rfind("}") + 1
|
||||
text = text[start:end]
|
||||
|
||||
# Parse JSON
|
||||
try:
|
||||
data = json.loads(text)
|
||||
except json.JSONDecodeError as e:
|
||||
print(f" ❌ JSON Parse Error: {e}")
|
||||
print(f" Raw text: {text[:500]}...")
|
||||
raise
|
||||
|
||||
# Post-process segments to remove hallucinations/meta-talk
|
||||
for seg in data["segments"]:
|
||||
# --- CRITICAL FIX FOR TEXT/AUDIO SYNC ---
|
||||
# Ensure we don't have lingering brackets or meta-tags that might
|
||||
# be processed differently by TTS vs Subtitles
|
||||
|
||||
# 1. Clean up meta-patterns from text (for both TTS and Subtitles)
|
||||
meta_patterns = [
|
||||
r'\(スライド\d+.*?\)',
|
||||
r'（スライド\d+.*?）',
|
||||
r'\[スライド\d+.*?\]',
|
||||
r'【スライド\d+.*?】'
|
||||
]
|
||||
for pattern in meta_patterns:
|
||||
seg["text"] = re.sub(pattern, '', seg["text"])
|
||||
|
||||
# 2. Standardize whitespace and remove hidden characters
|
||||
seg["text"] = " ".join(seg["text"].split())
|
||||
|
||||
# 3. Replace placeholders
|
||||
seg["text"] = seg["text"].replace("〇〇", "こちらの")
|
||||
seg["text"] = seg["text"].strip()
|
||||
|
||||
# 4. Add duration buffer (give more breathing room for visuals/HUD)
|
||||
try:
|
||||
duration = float(seg.get("duration", 5.0))
|
||||
seg["duration"] = duration * 1.1 + 1.0
|
||||
except (ValueError, TypeError):
|
||||
seg["duration"] = 6.5
|
||||
|
||||
# Calculate total duration
|
||||
total_duration = sum(seg.get("duration", 0) for seg in data["segments"])
|
||||
|
|
|
|||
|
|
@ -10,21 +10,31 @@ import subprocess
|
|||
from pathlib import Path
|
||||
|
||||
|
||||
def get_ffmpeg_cmd(cmd_name: str = "ffmpeg") -> str:
|
||||
"""Get the full path to ffmpeg/ffprobe, checking common locations if not in PATH"""
|
||||
# 1. Try default from PATH
|
||||
try:
|
||||
subprocess.run([cmd_name, "-version"], capture_output=True, check=True)
|
||||
return cmd_name
|
||||
except (subprocess.CalledProcessError, FileNotFoundError):
|
||||
pass
|
||||
|
||||
# 2. Try the specific Winget path found in the user environment
|
||||
winget_path = Path(r"C:\Users\91368\AppData\Local\Microsoft\WinGet\Packages\Gyan.FFmpeg_Microsoft.Winget.Source_8wekyb3d8bbwe\ffmpeg-8.0.1-full_build\bin")
|
||||
if winget_path.exists():
|
||||
target = winget_path / (f"{cmd_name}.exe" if os.name == 'nt' else cmd_name)
|
||||
if target.exists():
|
||||
return str(target)
|
||||
|
||||
return cmd_name
|
||||
|
||||
|
||||
def get_video_duration(video_path: str) -> float:
|
||||
"""
|
||||
Get video duration in seconds using ffprobe.
|
||||
|
||||
Args:
|
||||
video_path: Path to video file
|
||||
|
||||
Returns:
|
||||
Duration in seconds
|
||||
|
||||
Raises:
|
||||
RuntimeError: If ffprobe fails or video is invalid
|
||||
"""
|
||||
cmd = [
|
||||
"ffprobe",
|
||||
get_ffmpeg_cmd("ffprobe"),
|
||||
"-v", "error",
|
||||
"-show_entries", "format=duration",
|
||||
"-of", "json",
|
||||
|
|
@ -72,7 +82,7 @@ def extract_audio(
|
|||
output_path = str(Path(video_path).parent / f"{video_name}_audio.wav")
|
||||
|
||||
cmd = [
|
||||
"ffmpeg",
|
||||
get_ffmpeg_cmd("ffmpeg"),
|
||||
"-i", video_path,
|
||||
"-vn", # No video
|
||||
"-acodec", codec,
|
||||
|
|
@ -124,7 +134,7 @@ def slice_video(
|
|||
duration = end_time - start_time
|
||||
|
||||
cmd = [
|
||||
"ffmpeg",
|
||||
get_ffmpeg_cmd("ffmpeg"),
|
||||
"-ss", str(start_time),
|
||||
"-i", video_path,
|
||||
"-t", str(duration),
|
||||
|
|
@ -146,15 +156,15 @@ def slice_video(
|
|||
|
||||
|
||||
def check_ffmpeg_installed() -> bool:
|
||||
"""Check if FFmpeg and FFprobe are available in PATH."""
|
||||
"""Check if FFmpeg and FFprobe are available."""
|
||||
try:
|
||||
subprocess.run(
|
||||
["ffmpeg", "-version"],
|
||||
[get_ffmpeg_cmd("ffmpeg"), "-version"],
|
||||
capture_output=True,
|
||||
check=True
|
||||
)
|
||||
subprocess.run(
|
||||
["ffprobe", "-version"],
|
||||
[get_ffmpeg_cmd("ffprobe"), "-version"],
|
||||
capture_output=True,
|
||||
check=True
|
||||
)
|
||||
|
|
|
|||
|
|
@ -0,0 +1,213 @@
|
|||
"""
|
||||
Video Slicing Module
|
||||
|
||||
Extract important segments from source video based on transcription analysis.
|
||||
This is an optional feature for including interview clips in the final video.
|
||||
"""
|
||||
|
||||
import os
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any, List, Optional
|
||||
|
||||
try:
|
||||
from google import genai
|
||||
from google.genai import types
|
||||
except ImportError:
|
||||
genai = None
|
||||
|
||||
from backend.video_processor.utils import slice_video, get_video_duration
|
||||
|
||||
|
||||
class VideoSlicer:
|
||||
"""Extract important segments from source video"""
|
||||
|
||||
def __init__(self, api_key: Optional[str] = None):
|
||||
self.api_key = api_key or os.getenv("GEMINI_API_KEY")
|
||||
# CRITICAL: Output directly to public/ for Remotion to find them via staticFile
|
||||
self.output_dir = Path("public")
|
||||
self.output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
if self.api_key and genai:
|
||||
self.client = genai.Client(api_key=self.api_key)
|
||||
else:
|
||||
self.client = None
|
||||
|
||||
def extract_key_moments(
|
||||
self,
|
||||
video_path: str,
|
||||
transcription_data: Dict[str, Any],
|
||||
max_clips: int = 3,
|
||||
target_duration: float = 5.0
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Analyze transcription and extract key video moments
|
||||
|
||||
Args:
|
||||
video_path: Path to source video
|
||||
transcription_data: Transcription with segments and timestamps
|
||||
max_clips: Maximum number of clips to extract
|
||||
target_duration: Target duration for each clip in seconds
|
||||
|
||||
Returns:
|
||||
List of clip info dictionaries with start, end, filepath, description
|
||||
"""
|
||||
print(f" ✂️ Analyzing video for key moments...")
|
||||
|
||||
# Get video duration
|
||||
try:
|
||||
duration = get_video_duration(video_path)
|
||||
print(f" ℹ️ Video duration: {duration:.1f}s")
|
||||
except Exception as e:
|
||||
print(f" ⚠️ Could not get video duration: {e}")
|
||||
return []
|
||||
|
||||
# Use Gemini to identify key moments from transcription
|
||||
if self.client and "segments" in transcription_data:
|
||||
key_segments = self._identify_key_segments(
|
||||
transcription_data["segments"],
|
||||
max_clips=max_clips
|
||||
)
|
||||
else:
|
||||
# Fallback: use simple heuristics
|
||||
key_segments = self._simple_segment_selection(
|
||||
transcription_data.get("segments", []),
|
||||
max_clips=max_clips
|
||||
)
|
||||
|
||||
# Extract video clips
|
||||
clips = []
|
||||
for i, segment_info in enumerate(key_segments[:max_clips]):
|
||||
start = segment_info["start"]
|
||||
end = min(segment_info["end"], start + target_duration)
|
||||
|
||||
# Ensure within video bounds
|
||||
if end > duration:
|
||||
end = duration
|
||||
if start >= duration:
|
||||
continue
|
||||
|
||||
# Extract clip
|
||||
output_path = self.output_dir / f"clip_{i+1}_{start:.1f}-{end:.1f}.mp4"
|
||||
|
||||
try:
|
||||
if not output_path.exists():
|
||||
print(f" ✂️ Extracting clip {i+1}: {start:.1f}s - {end:.1f}s")
|
||||
clip_path = slice_video(video_path, start, end, str(output_path))
|
||||
else:
|
||||
clip_path = str(output_path)
|
||||
print(f" ✓ Clip {i+1} already cached")
|
||||
|
||||
clips.append({
|
||||
"id": f"clip_{i+1}",
|
||||
"outputPath": clip_path,
|
||||
"filepath": clip_path, # Keep for backward compatibility if any
|
||||
"start": start,
|
||||
"end": end,
|
||||
"duration": end - start,
|
||||
"description": segment_info.get("text", "")[:100]
|
||||
})
|
||||
|
||||
except Exception as e:
|
||||
print(f" ⚠️ Failed to extract clip {i+1}: {e}")
|
||||
|
||||
print(f" ✓ Extracted {len(clips)} video clips")
|
||||
return clips
|
||||
|
||||
def _identify_key_segments(
|
||||
self,
|
||||
segments: List[Dict[str, Any]],
|
||||
max_clips: int
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Use Gemini to identify important segments"""
|
||||
if not self.client:
|
||||
return self._simple_segment_selection(segments, max_clips)
|
||||
|
||||
# Prepare transcript for analysis
|
||||
transcript_text = "\n".join([
|
||||
f"[{seg['start']:.1f}s - {seg['end']:.1f}s] {seg['text']}"
|
||||
for seg in segments
|
||||
])
|
||||
|
||||
prompt = f"""
|
||||
以下は動画のトランスクリプトです。ライフプランニングの説明動画に最も適した重要なシーンを{max_clips}つ選んでください。
|
||||
|
||||
トランスクリプト:
|
||||
{transcript_text}
|
||||
|
||||
以下のJSON形式で、重要なシーンの開始時刻、終了時刻、理由を返してください:
|
||||
|
||||
```json
|
||||
[
|
||||
{{
|
||||
"start": 開始時刻(秒),
|
||||
"end": 終了時刻(秒),
|
||||
"text": "そのシーンのテキスト",
|
||||
"reason": "選んだ理由"
|
||||
}}
|
||||
]
|
||||
```
|
||||
"""
|
||||
|
||||
try:
|
||||
response = self.client.models.generate_content(
|
||||
model='gemini-2.0-flash-exp',
|
||||
contents=prompt
|
||||
)
|
||||
|
||||
response_text = response.text.strip()
|
||||
if "```json" in response_text:
|
||||
json_start = response_text.find("```json") + 7
|
||||
json_end = response_text.find("```", json_start)
|
||||
response_text = response_text[json_start:json_end].strip()
|
||||
elif "```" in response_text:
|
||||
json_start = response_text.find("```") + 3
|
||||
json_end = response_text.find("```", json_start)
|
||||
response_text = response_text[json_start:json_end].strip()
|
||||
|
||||
import json
|
||||
key_segments = json.loads(response_text)
|
||||
print(f" 🤖 Gemini identified {len(key_segments)} key moments")
|
||||
return key_segments
|
||||
|
||||
except Exception as e:
|
||||
print(f" ⚠️ Gemini analysis failed: {e}, using fallback...")
|
||||
return self._simple_segment_selection(segments, max_clips)
|
||||
|
||||
def _simple_segment_selection(
|
||||
self,
|
||||
segments: List[Dict[str, Any]],
|
||||
max_clips: int
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""Simple heuristic: select evenly spaced segments"""
|
||||
if not segments:
|
||||
return []
|
||||
|
||||
# Select evenly distributed segments
|
||||
step = len(segments) // max_clips if len(segments) >= max_clips else 1
|
||||
selected = [segments[i * step] for i in range(min(max_clips, len(segments)))]
|
||||
|
||||
return selected
|
||||
|
||||
|
||||
def extract_video_clips(
|
||||
video_path: str,
|
||||
transcription_data: Dict[str, Any],
|
||||
max_clips: int = 3
|
||||
) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Convenience function to extract video clips
|
||||
|
||||
Args:
|
||||
video_path: Path to source video
|
||||
transcription_data: Transcription with segments
|
||||
max_clips: Maximum number of clips to extract
|
||||
|
||||
Returns:
|
||||
List of clip information dictionaries
|
||||
"""
|
||||
try:
|
||||
slicer = VideoSlicer()
|
||||
return slicer.extract_key_moments(video_path, transcription_data, max_clips=max_clips)
|
||||
except Exception as e:
|
||||
print(f" ⚠️ Video slicing failed: {e}")
|
||||
return []
|
||||
|
|
@ -76,13 +76,17 @@ export type ScriptSegmentType =
|
|||
| "chart_intro" // Introduction to chart
|
||||
| "chart_highlight" // Highlighting specific data point
|
||||
| "video_clip" // Original interview footage
|
||||
| "dialogue" // Text-based interview dialogue
|
||||
| "transition"; // Transition between scenes
|
||||
|
||||
export interface ScriptSegment {
|
||||
id: number;
|
||||
text: string; // Japanese narration text
|
||||
insight?: string; // Optional key insight for floating card
|
||||
duration: number; // Duration in seconds
|
||||
type: ScriptSegmentType;
|
||||
visualType?: "ppt" | "video" | "chart" | "broll" | "dialogue";
|
||||
visualAssetIndex?: number;
|
||||
assetUrl?: string; // URL or path to B-Roll/video clip
|
||||
chartSync?: {
|
||||
// Optional chart synchronization
|
||||
|
|
@ -120,6 +124,8 @@ export interface RenderProps {
|
|||
currencyUnit: string; // Currency unit for all financial data
|
||||
duration: number; // Total video duration in seconds
|
||||
clientName?: string; // Client name for personalization
|
||||
advisorName?: string; // Financial advisor name (e.g., "田中太郎")
|
||||
advisorTitle?: string; // Advisor title (e.g., "CFP®", "AFP", "ファイナンシャルプランナー")
|
||||
generatedAt?: string; // ISO timestamp of generation
|
||||
};
|
||||
|
||||
|
|
@ -173,14 +179,19 @@ export const CashFlowDataSchema = z.object({
|
|||
export const ScriptSegmentSchema = z.object({
|
||||
id: z.number(),
|
||||
text: z.string(),
|
||||
insight: z.string().optional(),
|
||||
duration: z.number(),
|
||||
type: z.enum([
|
||||
"narration",
|
||||
"chart_intro",
|
||||
"chart_highlight",
|
||||
"video_clip",
|
||||
"dialogue",
|
||||
"transition",
|
||||
]),
|
||||
visualType: z.enum(["ppt", "video", "chart", "broll", "dialogue"]).optional(),
|
||||
visualAssetIndex: z.number().optional(),
|
||||
speaker: z.string().optional(), // Speaker name for dialogue
|
||||
assetUrl: z.string().optional(),
|
||||
chartSync: z
|
||||
.object({
|
||||
|
|
@ -197,6 +208,8 @@ export const RenderPropsSchema = z.object({
|
|||
currencyUnit: z.string(),
|
||||
duration: z.number(),
|
||||
clientName: z.string().optional(),
|
||||
advisorName: z.string().optional(),
|
||||
advisorTitle: z.string().optional(),
|
||||
generatedAt: z.string().optional(),
|
||||
}),
|
||||
charts: z.object({
|
||||
|
|
|
|||
290
main.py
290
main.py
|
|
@ -45,13 +45,56 @@ class VideoGeneratorPipeline:
|
|||
for dir_path in [self.input_dir, self.processed_dir, self.output_dir]:
|
||||
dir_path.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# Auto-discover input files if not specified
|
||||
self._auto_discover_inputs()
|
||||
|
||||
def _auto_discover_inputs(self):
|
||||
"""
|
||||
Auto-discover input files from assets/0_input/ directory
|
||||
Priority: PDF > PPTX > Video
|
||||
"""
|
||||
print("🔍 Scanning input directory for files...")
|
||||
|
||||
# Find PDF files
|
||||
if not self.config.get("pdf_path"):
|
||||
pdf_files = list(self.input_dir.glob("*.pdf"))
|
||||
if pdf_files:
|
||||
self.config["pdf_path"] = str(pdf_files[0])
|
||||
print(f" ✓ Found PDF: {pdf_files[0].name}")
|
||||
|
||||
# Find PPTX files
|
||||
if not self.config.get("ppt_path"):
|
||||
ppt_files = list(self.input_dir.glob("*.pptx"))
|
||||
# Filter out temporary files (starting with ~$)
|
||||
ppt_files = [f for f in ppt_files if not f.name.startswith("~$")]
|
||||
if ppt_files:
|
||||
self.config["ppt_path"] = str(ppt_files[0])
|
||||
print(f" ✓ Found PPT: {ppt_files[0].name}")
|
||||
|
||||
# Find video files
|
||||
if not self.config.get("video_path"):
|
||||
video_extensions = ["*.mp4", "*.mov", "*.avi", "*.mkv"]
|
||||
for ext in video_extensions:
|
||||
video_files = list(self.input_dir.glob(ext))
|
||||
if video_files:
|
||||
self.config["video_path"] = str(video_files[0])
|
||||
print(f" ✓ Found Video: {video_files[0].name}")
|
||||
break
|
||||
|
||||
# Summary
|
||||
if not any([self.config.get("pdf_path"), self.config.get("ppt_path")]):
|
||||
print(" ⚠️ No PDF or PPT files found in input directory")
|
||||
|
||||
print()
|
||||
|
||||
def run(self) -> str:
|
||||
"""Execute the full pipeline and return output video path"""
|
||||
print("🚀 Starting Auto Life Plan Video Generation Pipeline...")
|
||||
|
||||
# Step 1: Extract data from PDF
|
||||
print("\n📄 Step 1/6: Extracting data from PDF...")
|
||||
# Step 1: Extract data from PDF or PPT
|
||||
print("\n📄 Step 1/6: Extracting data from documents...")
|
||||
pdf_data = self.extract_pdf()
|
||||
ppt_data = self.extract_ppt()
|
||||
|
||||
# Step 2: Extract data from video (if provided)
|
||||
print("\n🎥 Step 2/6: Processing video footage...")
|
||||
|
|
@ -59,19 +102,19 @@ class VideoGeneratorPipeline:
|
|||
|
||||
# Step 3: Generate narration script
|
||||
print("\n✍️ Step 3/6: Generating narration script...")
|
||||
script = self.generate_script(pdf_data, video_data)
|
||||
script = self.generate_script(pdf_data, ppt_data, video_data)
|
||||
|
||||
# Step 3.5: Generate audio from script (NEW)
|
||||
print("\n🎙️ Step 3.5/6: Generating audio narration...")
|
||||
script = self.generate_audio(script)
|
||||
|
||||
# Step 4: Fetch B-Roll assets
|
||||
print("\n🎬 Step 4/6: Fetching B-Roll assets...")
|
||||
broll_assets = self.fetch_broll(script)
|
||||
# Step 4: Fetch B-Roll assets (TEMPORARILY DISABLED)
|
||||
print("\n🎬 Step 4/6: Fetching B-Roll assets (SKIPPED)...")
|
||||
broll_assets = {} # self.fetch_broll(script, video_data)
|
||||
|
||||
# Step 5: Build render props
|
||||
print("\n📊 Step 5/6: Building render props...")
|
||||
render_props = self.build_render_props(pdf_data, script, broll_assets)
|
||||
render_props = self.build_render_props(pdf_data, ppt_data, script, broll_assets, video_data)
|
||||
|
||||
# Save render_props.json
|
||||
props_file = self.processed_dir / "render_props.json"
|
||||
|
|
@ -88,84 +131,167 @@ class VideoGeneratorPipeline:
|
|||
|
||||
def extract_pdf(self) -> Dict[str, Any]:
|
||||
"""Extract data from PDF using Gemini Pro"""
|
||||
# TODO: Implement backend.extractor.extract_pdf_data
|
||||
print(" [MOCK] Extracting PDF data...")
|
||||
return {
|
||||
"client_name": "山田太郎",
|
||||
"current_age": 35,
|
||||
"retirement_age": 65,
|
||||
"life_expectancy": 100,
|
||||
"current_assets": 5000,
|
||||
"annual_income": 600,
|
||||
"annual_expenses": 450,
|
||||
}
|
||||
pdf_path = self.config.get("pdf_path")
|
||||
|
||||
if not pdf_path or not Path(pdf_path).exists():
|
||||
print(" [SKIP] No PDF file found")
|
||||
return {}
|
||||
|
||||
print(f" 📄 Processing PDF: {Path(pdf_path).name}")
|
||||
|
||||
try:
|
||||
from backend.extractor import extract_pdf_data
|
||||
pdf_data = extract_pdf_data(pdf_path)
|
||||
print(f" ✓ Successfully extracted data for: {pdf_data.get('client_name', 'Unknown')}")
|
||||
return pdf_data
|
||||
except Exception as e:
|
||||
print(f" ⚠️ PDF extraction failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return {}
|
||||
|
||||
def extract_ppt(self) -> Dict[str, Any]:
|
||||
"""Extract slides from PowerPoint presentation"""
|
||||
ppt_path = self.config.get("ppt_path")
|
||||
|
||||
if not ppt_path or not Path(ppt_path).exists():
|
||||
print(" [SKIP] No PPT file found")
|
||||
return {}
|
||||
|
||||
print(f" 📊 Processing PPT: {Path(ppt_path).name}")
|
||||
|
||||
try:
|
||||
from backend.ppt_extractor import extract_ppt_data
|
||||
ppt_data = extract_ppt_data(ppt_path)
|
||||
print(f" ✓ Successfully extracted {ppt_data.get('total_slides', 0)} slides")
|
||||
return ppt_data
|
||||
except Exception as e:
|
||||
print(f" ⚠️ PPT extraction failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return {}
|
||||
|
||||
def extract_video(self) -> Dict[str, Any]:
|
||||
"""Extract data from video using Whisper (if video provided)"""
|
||||
if not self.config.get("video_path"):
|
||||
print(" [SKIP] No video file provided")
|
||||
video_path = self.config.get("video_path")
|
||||
|
||||
if not video_path or not Path(video_path).exists():
|
||||
print(" [SKIP] No video file provided or file not found")
|
||||
return {}
|
||||
|
||||
# TODO: Implement backend.extractor.extract_video_data
|
||||
print(" [MOCK] Extracting video timestamps and transcription...")
|
||||
return {
|
||||
"transcription": "こんにちは...",
|
||||
"timestamps": [
|
||||
{"time": 0.5, "text": "Introduction", "keywords": ["保険", "将来"]},
|
||||
{"time": 15.2, "text": "Main topic", "keywords": ["資産", "計画"]},
|
||||
],
|
||||
}
|
||||
try:
|
||||
from backend.extractor import extract_video_data
|
||||
video_data = extract_video_data(video_path)
|
||||
|
||||
def generate_script(self, pdf_data: Dict, video_data: Dict) -> Dict[str, Any]:
|
||||
# Optionally extract video clips
|
||||
if self.config.get("extract_clips", False):
|
||||
print(" ✂️ Extracting key video clips...")
|
||||
from backend.video_slicer import extract_video_clips
|
||||
clips = extract_video_clips(video_path, video_data, max_clips=3)
|
||||
video_data["clips"] = clips
|
||||
|
||||
return video_data
|
||||
|
||||
except Exception as e:
|
||||
print(f" ⚠️ Video extraction failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return {}
|
||||
|
||||
def generate_script(self, pdf_data: Dict, ppt_data: Dict, video_data: Dict) -> Dict[str, Any]:
|
||||
"""Generate narration script using Gemini Pro"""
|
||||
print(" Generating script with Gemini Pro...")
|
||||
print(" ✍️ Generating script with Gemini Pro...")
|
||||
from backend.scripter import generate_script
|
||||
|
||||
# Merge PPT and PDF data
|
||||
combined_data = {**pdf_data}
|
||||
if ppt_data:
|
||||
combined_data["presentation"] = ppt_data
|
||||
|
||||
try:
|
||||
return generate_script(pdf_data, video_data)
|
||||
script = generate_script(combined_data, video_data)
|
||||
print(f" ✓ Generated {len(script.get('segments', []))} script segments")
|
||||
return script
|
||||
except Exception as e:
|
||||
print(f" ⚠️ Script generation failed: {e}")
|
||||
print(" Using fallback script...")
|
||||
return {
|
||||
"segments": [
|
||||
{
|
||||
"id": 1,
|
||||
"text": "これからあなたの将来設計について説明します。",
|
||||
"duration": 3.5,
|
||||
"type": "narration",
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"text": "現在の資産状況を見てみましょう。",
|
||||
"duration": 2.8,
|
||||
"type": "chart_intro",
|
||||
},
|
||||
],
|
||||
"total_duration": 6.3,
|
||||
}
|
||||
print(f" ❌ Script generation failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
raise
|
||||
|
||||
def generate_audio(self, script_data: Dict) -> Dict[str, Any]:
|
||||
"""Generate audio narration from script using OpenAI TTS"""
|
||||
from backend.audio_generator import generate_audio_for_script
|
||||
from backend.video_processor.utils import get_video_duration
|
||||
|
||||
try:
|
||||
return generate_audio_for_script(script_data, voice="alloy")
|
||||
# 1. Generate TTS audio for non-video segments
|
||||
script_data = generate_audio_for_script(script_data, voice="alloy")
|
||||
|
||||
# 2. Sync duration for video segments with actual clip duration
|
||||
print(" 🎬 Syncing durations for interview video segments...")
|
||||
video_data = self.extract_video() # Get cached video data with clips
|
||||
clips = video_data.get("clips", [])
|
||||
|
||||
for segment in script_data.get("segments", []):
|
||||
if segment.get("visualType") == "video":
|
||||
idx = segment.get("visualAssetIndex", -1)
|
||||
if 0 <= idx < len(clips):
|
||||
clip = clips[idx]
|
||||
clip_path = clip.get("outputPath") or clip.get("filepath")
|
||||
if clip_path and Path(clip_path).exists():
|
||||
actual_duration = get_video_duration(clip_path)
|
||||
print(f" ✓ Video Segment {segment['id']}: Updated duration to {actual_duration:.2f}s (Clip {idx})")
|
||||
segment["duration"] = actual_duration
|
||||
|
||||
return script_data
|
||||
except Exception as e:
|
||||
print(f" ⚠️ Audio generation failed: {e}")
|
||||
print(" Continuing without audio...")
|
||||
return script_data
|
||||
|
||||
def fetch_broll(self, script: Dict) -> Dict[str, Any]:
|
||||
"""Fetch B-Roll footage from Pexels API"""
|
||||
# TODO: Implement backend.media_fetcher.fetch_broll
|
||||
print(" [MOCK] Fetching B-Roll from Pexels...")
|
||||
return {
|
||||
"retirement": "https://example.com/retirement.mp4",
|
||||
"family": "https://example.com/family.mp4",
|
||||
}
|
||||
def fetch_broll(self, script: Dict, video_data: Dict = None) -> Dict[str, Any]:
|
||||
"""Fetch B-Roll footage from Pexels API based on script context"""
|
||||
try:
|
||||
from backend.media_fetcher import MediaFetcher
|
||||
|
||||
# 1. Extract keywords explicitly requested in script segments
|
||||
keywords = set()
|
||||
for segment in script.get("segments", []):
|
||||
if segment.get("visualType") == "broll" and segment.get("brollKeyword"):
|
||||
keywords.add(segment.get("brollKeyword"))
|
||||
|
||||
# 2. Add keywords from video data as fallback/supplement
|
||||
if video_data and "keywords" in video_data:
|
||||
for kw in video_data["keywords"][:3]:
|
||||
keywords.add(kw)
|
||||
|
||||
# 3. Use default keywords if still empty
|
||||
if not keywords:
|
||||
keywords.update(["家族", "老後", "貯蓄"])
|
||||
|
||||
keyword_list = list(keywords)
|
||||
print(f" 🎬 Fetching B-Roll for context: {', '.join(keyword_list)}")
|
||||
|
||||
fetcher = MediaFetcher()
|
||||
broll_results = fetcher.fetch_broll(keyword_list, max_per_keyword=1)
|
||||
|
||||
# Convert to simple filepath mapping
|
||||
broll_assets = {}
|
||||
for keyword, videos in broll_results.items():
|
||||
if videos:
|
||||
# Store only the filename to be used with staticFile() in Remotion
|
||||
broll_assets[keyword] = Path(videos[0]["filepath"]).name
|
||||
|
||||
return broll_assets
|
||||
|
||||
except Exception as e:
|
||||
print(f" ⚠️ B-Roll fetching failed: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
return {}
|
||||
|
||||
def build_render_props(
|
||||
self, pdf_data: Dict, script: Dict, broll: Dict
|
||||
self, pdf_data: Dict, ppt_data: Dict, script: Dict, broll: Dict, video_data: Dict = None
|
||||
) -> Dict[str, Any]:
|
||||
"""Build the complete render_props.json structure"""
|
||||
print(" Building chart data and render props...")
|
||||
|
|
@ -174,7 +300,24 @@ class VideoGeneratorPipeline:
|
|||
# TypeScript RenderProps contract (frontend/src/render-props-types.ts)
|
||||
from backend.data_builder import build_chart_data
|
||||
|
||||
charts = build_chart_data(pdf_data)
|
||||
charts = build_chart_data(pdf_data) if pdf_data else {}
|
||||
|
||||
# Add video clips if available
|
||||
video_clips = []
|
||||
if video_data and "clips" in video_data:
|
||||
video_clips = video_data["clips"]
|
||||
|
||||
# Add PPT slides if available
|
||||
ppt_slides = []
|
||||
if ppt_data and "slides" in ppt_data:
|
||||
ppt_slides = [
|
||||
{
|
||||
"index": slide["index"],
|
||||
"title": slide["title"],
|
||||
"image_path": slide["image_path"].replace("\\", "/"),
|
||||
}
|
||||
for slide in ppt_data["slides"]
|
||||
]
|
||||
|
||||
return {
|
||||
"meta": {
|
||||
|
|
@ -187,7 +330,8 @@ class VideoGeneratorPipeline:
|
|||
"script": script["segments"],
|
||||
"assets": {
|
||||
"broll": broll,
|
||||
"videoClips": [],
|
||||
"videoClips": video_clips,
|
||||
"slides": ppt_slides,
|
||||
},
|
||||
}
|
||||
|
||||
|
|
@ -211,6 +355,7 @@ class VideoGeneratorPipeline:
|
|||
"MainVideo",
|
||||
str(output_file),
|
||||
f"--props={props_file}",
|
||||
"--concurrency=1",
|
||||
]
|
||||
|
||||
print(f" Running: {' '.join(cmd)}")
|
||||
|
|
@ -240,7 +385,13 @@ Examples:
|
|||
parser.add_argument(
|
||||
"--pdf",
|
||||
type=str,
|
||||
help="Path to the Life Plan PDF report",
|
||||
help="Path to the Life Plan PDF report (optional if files exist in assets/0_input/)",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--ppt",
|
||||
type=str,
|
||||
help="Path to the PowerPoint presentation (optional if files exist in assets/0_input/)",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
|
|
@ -267,6 +418,12 @@ Examples:
|
|||
help="Generate render_props.json without rendering",
|
||||
)
|
||||
|
||||
parser.add_argument(
|
||||
"--extract-clips",
|
||||
action="store_true",
|
||||
help="Extract video clips from source video (optional)",
|
||||
)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Build configuration
|
||||
|
|
@ -276,15 +433,16 @@ Examples:
|
|||
with open(args.config, "r", encoding="utf-8") as f:
|
||||
config = json.load(f)
|
||||
else:
|
||||
if not args.pdf:
|
||||
parser.error("--pdf is required when --config is not provided")
|
||||
# Only require --pdf if no files found in input directory
|
||||
config["pdf_path"] = args.pdf
|
||||
config["ppt_path"] = args.ppt
|
||||
config["video_path"] = args.video
|
||||
|
||||
if args.output:
|
||||
config["output_filename"] = args.output
|
||||
|
||||
config["dry_run"] = args.dry_run
|
||||
config["extract_clips"] = args.extract_clips
|
||||
|
||||
# Run pipeline
|
||||
pipeline = VideoGeneratorPipeline(config)
|
||||
|
|
|
|||
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
|
After Width: | Height: | Size: 2.3 MiB |
Binary file not shown.
|
After Width: | Height: | Size: 624 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 542 KiB |
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue