Kapitel 13/Tutorial.md aktualisiert
This commit is contained in:
@@ -410,120 +410,182 @@ In diesem Schritt erstellen wir den eigentlichen Workflow in **n8n**. Er sorgt d
|
|||||||
|
|
||||||
7. **False-Pfad – VODs herunterladen und hochladen**
|
7. **False-Pfad – VODs herunterladen und hochladen**
|
||||||
|
|
||||||
> [!NOTE]
|
> [!NOTE]
|
||||||
> Dieser Pfad läuft, wenn die State-Datei nicht existiert oder leer ist. Es werden alle VODs heruntergeladen und in Nextcloud hochgeladen.
|
> Dieser Pfad läuft, wenn die State-Datei nicht existiert oder leer ist. Es werden alle VODs heruntergeladen und in Nextcloud hochgeladen.
|
||||||
|
|
||||||
**A. Vorbereitung (einmalig) – rclone an Nextcloud anbinden**
|
**A. Vorbereitung (einmalig) – rclone an Nextcloud anbinden**
|
||||||
**Ort:** Clipper-LXC Shell
|
**Ort:** Clipper-LXC Shell
|
||||||
```bash
|
```bash
|
||||||
apt update && apt install -y rclone
|
apt update && apt install -y rclone
|
||||||
su - clipper
|
su - clipper
|
||||||
rclone config create nc webdav \
|
rclone config create nc webdav \
|
||||||
url=https://DEINE_DOMAIN/remote.php/dav/files/DEIN_BENUTZERNAME/ \
|
url=https://DEINE_DOMAIN/remote.php/dav/files/DEIN_BENUTZERNAME/ \
|
||||||
vendor=nextcloud \
|
vendor=nextcloud \
|
||||||
user=DEIN_BENUTZERNAME \
|
user=DEIN_BENUTZERNAME \
|
||||||
pass=$(rclone obscure 'DEIN_APP_PASSWORT')
|
pass=$(rclone obscure 'DEIN_APP_PASSWORT')
|
||||||
rclone ls nc: --config /home/clipper/.config/rclone/rclone.conf
|
rclone ls nc: --config /home/clipper/.config/rclone/rclone.conf
|
||||||
|
```
|
||||||
|
> [!NOTE]
|
||||||
|
> Uploads laufen im Clipper mit rclone. Das ist robuster als Upload-Nodes in n8n, vermeidet Fehler und räumt lokale Dateien auf.
|
||||||
|
|
||||||
|
**B. Download/Upload Skript erstellen**
|
||||||
|
**Ort:** Clipper-LXC Shell
|
||||||
|
```bash
|
||||||
|
nano <clipper-ordner>/bin/clipper-vod-get
|
||||||
|
```
|
||||||
|
Inhalt:
|
||||||
|
```bash
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
set -euo pipefail
|
||||||
|
. /etc/clipper/clipper.env
|
||||||
|
|
||||||
|
ID="${1:?need VOD id}"
|
||||||
|
URL="${2:-https://www.twitch.tv/videos/${ID}}"
|
||||||
|
|
||||||
|
TMP="${CLIPPER_TMP}"
|
||||||
|
OUT_BASE="${CLIPPER_OUT}/${ID}"
|
||||||
|
LOGDIR="${CLIPPER_LOG}/${ID}"
|
||||||
|
CONF="/home/clipper/.config/rclone/rclone.conf"
|
||||||
|
|
||||||
|
DST="${CLIPPER_NC_REMOTE}/VODs/${ID}/"
|
||||||
|
|
||||||
|
OUT="$TMP/${ID}.%(ext)s"
|
||||||
|
FILE="$TMP/${ID}.mp4"
|
||||||
|
TEMP="$TMP/${ID}.temp.mp4"
|
||||||
|
PART="$TMP/${ID}.mp4.part"
|
||||||
|
LOCK="$TMP/${ID}.lock"
|
||||||
|
|
||||||
|
mkdir -p "$TMP" "$LOGDIR"
|
||||||
|
LOG="$LOGDIR/download.log"
|
||||||
|
log(){ echo "[$(date '+%F %T')] $*"; }
|
||||||
|
exec > >(tee -a "$LOG") 2>&1
|
||||||
|
|
||||||
|
log "=== Start VOD $ID ==="
|
||||||
|
log "URL: $URL"
|
||||||
|
log "DST: $DST"
|
||||||
|
|
||||||
|
if rclone lsf "$DST" --config "$CONF" >/dev/null 2>&1; then
|
||||||
|
log "SKIP: $ID bereits in Nextcloud"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -e "$LOCK" ]]; then
|
||||||
|
log "LOCK: $ID wird bereits verarbeitet"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
trap 'rm -f "$LOCK"' EXIT
|
||||||
|
: > "$LOCK"
|
||||||
|
|
||||||
|
# Resume
|
||||||
|
if [[ -s "$TEMP" && ! -s "$FILE" ]]; then
|
||||||
|
log "RESUME: $TEMP -> $FILE"
|
||||||
|
mv -f "$TEMP" "$FILE"
|
||||||
|
fi
|
||||||
|
|
||||||
|
if [[ -s "$FILE" ]]; then
|
||||||
|
log "MOVE (resume): $FILE → $DST"
|
||||||
|
rclone move "$FILE" "$DST" --config "$CONF" --create-empty-src-dirs -v
|
||||||
|
rm -f "$PART" "$TEMP" || true
|
||||||
|
log "CLEANUP: $TMP"
|
||||||
|
rm -rf "${TMP:?}/"*
|
||||||
|
log "=== Done VOD $ID (resume path) ==="
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Download + Remux
|
||||||
|
yt-dlp -q --no-progress --retries 20 --fragment-retries 50 --retry-sleep 5 \
|
||||||
|
--socket-timeout 30 --hls-prefer-ffmpeg --remux-video mp4 -o "$OUT" "$URL"
|
||||||
|
|
||||||
|
[[ -s "$FILE" ]] || { [[ -s "$TEMP" ]] && mv -f "$TEMP" "$FILE"; }
|
||||||
|
|
||||||
|
if [[ ! -s "$FILE" ]]; then
|
||||||
|
log "ERROR: Download fehlgeschlagen ($FILE fehlt/leer)"
|
||||||
|
exit 2
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "MOVE: $FILE → $DST"
|
||||||
|
rclone move "$FILE" "$DST" --config "$CONF" --create-empty-src-dirs -v
|
||||||
|
rm -f "$PART" "$TEMP" || true
|
||||||
|
|
||||||
|
log "CLEANUP: $TMP"
|
||||||
|
rm -rf "${TMP:?}/"*
|
||||||
|
|
||||||
|
log "=== Done VOD $ID ==="
|
||||||
|
```
|
||||||
|
Rechte setzen:
|
||||||
|
```bash
|
||||||
|
chmod 755 <clipper-ordner>/bin/clipper-vod-get
|
||||||
|
chown clipper:clipper <clipper-ordner>/bin/clipper-vod-get
|
||||||
|
```
|
||||||
|
> [!NOTE]
|
||||||
|
> Pro VOD entsteht ein Logfile in `<clipper-ordner>/logs/<ID>.log`. Du kannst es live mit `tail -f <clipper-ordner>/logs/<ID>.log` verfolgen.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
Mit diesem Aufbau ist der **False-Pfad** fertig: Wenn die State-Datei fehlt oder leer ist, werden alle VODs verarbeitet und in Nextcloud hochgeladen. Der True-Pfad folgt im nächsten Abschnitt.
|
||||||
|
|
||||||
|
**C. n8n‑Verkabelung (Überblick)**
|
||||||
|
Ort: *n8n Weboberfläche*
|
||||||
|
|
||||||
|
**Verkabelung (Kurzüberblick):**
|
||||||
|
1) HTTP Request →
|
||||||
|
2) Split Out: → 3) Merge →
|
||||||
|
4) Split In Batches →
|
||||||
|
5) SSH Node 1 (State-Datei schreiben) →
|
||||||
|
6) SSH Node 2 (Download & Upload)
|
||||||
|
---
|
||||||
|
|
||||||
|
** Node-Einstellungen (1:1 in n8n eintragen)**
|
||||||
|
**1) HTTP Request – Get Videos** -
|
||||||
|
- **Node-Typ:** HTTP Request -
|
||||||
|
- **Methode:** GET -
|
||||||
|
- **URL:** `https://api.twitch.tv/helix/videos?user_id=<DEINE_TWITCH_USER_ID>&type=archive&first=20`
|
||||||
|
- **Authentifizierung:** OAuth2 (Credential: *Twitch API*)
|
||||||
|
- **Header:** Client-Id: <DEINE_CLIENT_ID>
|
||||||
|
- **Response Format:** JSON **
|
||||||
|
|
||||||
|
2) Item Lists – Split Out**
|
||||||
|
- **Node-Typ:** Split Out
|
||||||
|
- **Field to Split Out:** data **
|
||||||
|
|
||||||
|
3) Merge – Combine**
|
||||||
|
- **Node-Typ:** Merge
|
||||||
|
- **Mode:** Combine
|
||||||
|
- **Combine Mode:** All Possible Combinations
|
||||||
|
- **Eingang 1:** If False Ausgang
|
||||||
|
- **Eingang 2:** Item Lists: Split Out
|
||||||
|
|
||||||
|
**4) Split In Batches**
|
||||||
|
- **Node-Typ:** Split In Batches
|
||||||
|
- **Batch Size:** 1
|
||||||
|
|
||||||
|
**5) SSH Node 1 – State-Datei schreiben**
|
||||||
|
- **Node-Typ:** SSH
|
||||||
|
- **Credentials:** *SSH Clipper* (User = clipper)
|
||||||
|
- **Operation:** Execute Command
|
||||||
|
- **Command is an Expression:** **ON**
|
||||||
|
- **Command:**
|
||||||
|
``` bash
|
||||||
|
{{`set -euo pipefail; STATE="/srv/clipper/state/vod_seen.list"; mkdir -p "$(dirname "$STATE")"; if [ -s "$STATE" ]; then printf "%s\n" "${$json.data.id}" >> "$STATE"; else printf "%s\n" "${$json.data.id}" > "$STATE"; fi`}}
|
||||||
```
|
```
|
||||||
> [!NOTE]
|
|
||||||
> Uploads laufen im Clipper mit rclone. Das ist robuster als Upload-Nodes in n8n, vermeidet Fehler und räumt lokale Dateien auf.
|
|
||||||
|
|
||||||
**B. Download/Upload Skript erstellen**
|
**6) SSH Node 2 – Download & Upload (Skript)**
|
||||||
**Ort:** Clipper-LXC Shell
|
- **Node-Typ:** SSH
|
||||||
|
- **Credentials:** *SSH Clipper* (User = clipper)
|
||||||
|
- **Operation:** Execute Command
|
||||||
|
- **Command is an Expression:** **ON**
|
||||||
|
- **Command:**
|
||||||
```bash
|
```bash
|
||||||
nano <clipper-ordner>/bin/clipper-vod-get
|
<clipper-ordner>/bin/clipper-vod-get "{{$('Merge').item.json.data.id}}" "{{ $json.url || ('https://www.twitch.tv/videos/' + $('Merge').item.json.data.id) }}"
|
||||||
```
|
```
|
||||||
Inhalt:
|
|
||||||
```bash
|
|
||||||
#!/usr/bin/env bash
|
|
||||||
set -euo pipefail
|
|
||||||
. /etc/clipper/clipper.env
|
|
||||||
|
|
||||||
ID="${1:?need VOD id}"
|
|
||||||
URL="${2:-https://www.twitch.tv/videos/${ID}}"
|
|
||||||
|
|
||||||
TMP="${CLIPPER_TMP}"
|
|
||||||
OUT_BASE="${CLIPPER_OUT}/${ID}"
|
|
||||||
LOGDIR="${CLIPPER_LOG}/${ID}"
|
|
||||||
CONF="/home/clipper/.config/rclone/rclone.conf"
|
|
||||||
|
|
||||||
DST="${CLIPPER_NC_REMOTE}/VODs/${ID}/"
|
|
||||||
|
|
||||||
OUT="$TMP/${ID}.%(ext)s"
|
|
||||||
FILE="$TMP/${ID}.mp4"
|
|
||||||
TEMP="$TMP/${ID}.temp.mp4"
|
|
||||||
PART="$TMP/${ID}.mp4.part"
|
|
||||||
LOCK="$TMP/${ID}.lock"
|
|
||||||
|
|
||||||
mkdir -p "$TMP" "$LOGDIR"
|
|
||||||
LOG="$LOGDIR/download.log"
|
|
||||||
log(){ echo "[$(date '+%F %T')] $*"; }
|
|
||||||
exec > >(tee -a "$LOG") 2>&1
|
|
||||||
|
|
||||||
log "=== Start VOD $ID ==="
|
|
||||||
log "URL: $URL"
|
|
||||||
log "DST: $DST"
|
|
||||||
|
|
||||||
if rclone lsf "$DST" --config "$CONF" >/dev/null 2>&1; then
|
|
||||||
log "SKIP: $ID bereits in Nextcloud"
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -e "$LOCK" ]]; then
|
|
||||||
log "LOCK: $ID wird bereits verarbeitet"
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
trap 'rm -f "$LOCK"' EXIT
|
|
||||||
: > "$LOCK"
|
|
||||||
|
|
||||||
# Resume
|
|
||||||
if [[ -s "$TEMP" && ! -s "$FILE" ]]; then
|
|
||||||
log "RESUME: $TEMP -> $FILE"
|
|
||||||
mv -f "$TEMP" "$FILE"
|
|
||||||
fi
|
|
||||||
|
|
||||||
if [[ -s "$FILE" ]]; then
|
|
||||||
log "MOVE (resume): $FILE → $DST"
|
|
||||||
rclone move "$FILE" "$DST" --config "$CONF" --create-empty-src-dirs -v
|
|
||||||
rm -f "$PART" "$TEMP" || true
|
|
||||||
log "CLEANUP: $TMP"
|
|
||||||
rm -rf "${TMP:?}/"*
|
|
||||||
log "=== Done VOD $ID (resume path) ==="
|
|
||||||
exit 0
|
|
||||||
fi
|
|
||||||
|
|
||||||
# Download + Remux
|
|
||||||
yt-dlp -q --no-progress --retries 20 --fragment-retries 50 --retry-sleep 5 \
|
|
||||||
--socket-timeout 30 --hls-prefer-ffmpeg --remux-video mp4 -o "$OUT" "$URL"
|
|
||||||
|
|
||||||
[[ -s "$FILE" ]] || { [[ -s "$TEMP" ]] && mv -f "$TEMP" "$FILE"; }
|
|
||||||
|
|
||||||
if [[ ! -s "$FILE" ]]; then
|
|
||||||
log "ERROR: Download fehlgeschlagen ($FILE fehlt/leer)"
|
|
||||||
exit 2
|
|
||||||
fi
|
|
||||||
|
|
||||||
log "MOVE: $FILE → $DST"
|
|
||||||
rclone move "$FILE" "$DST" --config "$CONF" --create-empty-src-dirs -v
|
|
||||||
rm -f "$PART" "$TEMP" || true
|
|
||||||
|
|
||||||
log "CLEANUP: $TMP"
|
|
||||||
rm -rf "${TMP:?}/"*
|
|
||||||
|
|
||||||
log "=== Done VOD $ID ==="
|
|
||||||
```
|
|
||||||
Rechte setzen:
|
|
||||||
```bash
|
|
||||||
chmod 755 <clipper-ordner>/bin/clipper-vod-get
|
|
||||||
chown clipper:clipper <clipper-ordner>/bin/clipper-vod-get
|
|
||||||
```
|
|
||||||
> [!NOTE]
|
|
||||||
> Pro VOD entsteht ein Logfile in `<clipper-ordner>/logs/<ID>.log`. Du kannst es live mit `tail -f <clipper-ordner>/logs/<ID>.log` verfolgen.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
Mit diesem Aufbau ist der **False-Pfad** fertig: Wenn die State-Datei fehlt oder leer ist, werden alle VODs verarbeitet und in Nextcloud hochgeladen. Der True-Pfad folgt im nächsten Abschnitt.
|
|
||||||
|
|
||||||
|
---
|
||||||
|
**Ergebnis**
|
||||||
|
- n8n steuert den gesamten Prozess, Upload erfolgt zuverlässig im Clipper via rclone.
|
||||||
|
- Pro VOD entsteht **eine** MP4 in Nextcloud: <gewünschter Ordner>/VODs/<ID>/.
|
||||||
|
- Pro VOD gibt es ein eigenes Logfileverzeichnis unter <clipper-ordner>/logs/<ID>/download.log.
|
||||||
|
- Lokaler Speicher bleibt frei (automatisches Löschen nach Upload).
|
||||||
|
- Logs der Schritte findest du im SSH‑Node‑Output und persistent im Logfile.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user