diff --git a/progress/TASK-aws-upload-problem.md b/progress/TASK-aws-upload-problem.md new file mode 100644 index 0000000..a622c53 --- /dev/null +++ b/progress/TASK-aws-upload-problem.md @@ -0,0 +1,277 @@ +# PLAN: Debug AWS file-transfer disconnects + +## Objective + +Transfer the notification-wakeup API release archive to the test VM, identify +what terminates SSH transfers, and verify the resulting file's SHA-256 hash. + +Status: investigation pending. The commands below are proposed diagnostic steps; +they have not been executed as part of writing this plan. + +## Target and test controls + +Use the same destination, identity, and archive for every comparison: + +- Host: `test-notify-api.timesafari.app` +- User: `ubuntutest` +- Identity: `~/.ssh/key3.pem` +- Local archive: `~/dl/notify-wakeup-api-amd-0.2.1.tar.bz2` +- Expected size for that archive: `240799305` bytes + +On the Mac, set these variables in the terminal used for testing. Change the +version only if testing a different release, and keep the archive unchanged +throughout the comparisons. + +```bash +NOTIFY_WAKEUP_API_VERSION=0.2.1 +artifact="notify-wakeup-api-amd-${NOTIFY_WAKEUP_API_VERSION:?Set NOTIFY_WAKEUP_API_VERSION}.tar.bz2" +source_file="$HOME/dl/$artifact" +remote_target="ubuntutest@test-notify-api.timesafari.app" +ssh_key="$HOME/.ssh/key3.pem" + +wc -c < "$source_file" +shasum -a 256 "$source_file" +``` + +For each attempt, record the command, network/VPN state, elapsed time, bytes +reported, exit status, and matching server logs. Change one variable at a time. +Use `date -u` on both machines to correlate log timestamps. + +## Diagnostic sequence + +### 1. Observe the server while reproducing the failure + +Open a separate Mac terminal and start an SSH session to the VM: + +```bash +ssh -i ~/.ssh/key3.pem ubuntutest@test-notify-api.timesafari.app +``` + +Inside that session, follow the authentication log: + +```bash +sudo tail -F /var/log/auth.log +``` + +If that file is unavailable, use the SSH service journal: + +```bash +sudo journalctl -f -u ssh +``` + +Leave the observer session open while running the transfer tests from the Mac. +Record whether this session survives each failure. After a failure, inspect +kernel messages and resource availability on the VM: + +```bash +sudo journalctl -k --since "5 minutes ago" --no-pager +df -h "$HOME" +df -i "$HOME" +free -h +uptime +``` + +Look for SSH errors, process crashes, out-of-memory kills, storage errors, and +reboots at the failure time. A full filesystem or exhausted inodes warrants +investigation, but does not by itself explain a TCP reset. If the VM becomes +unreachable, inspect its EC2 status checks and console output through AWS. + +### 2. Test bulk SSH traffic without SCP or destination-file writes + +Run on the Mac, using the variables from the setup block: + +```bash +time ssh -T -v -i "$ssh_key" "$remote_target" 'wc -c' < "$source_file" +transfer_status=$? +printf 'SSH exit status: %s\n' "$transfer_status" +``` + +The remote process counts and discards the incoming bytes. It does not save the +archive. Success means the printed count matches the local `wc -c` result and +SSH exits with status zero; for release 0.2.1 the count is `240799305`. + +| Result | Interpretation and next action | +| --- | --- | +| Connection resets | SCP, SFTP, and destination-file writes are unnecessary to trigger the failure. Investigate SSH, server health, and the network path. | +| Repeated success | Investigate the file-transfer process and destination storage. Repeat the failing SCP command while observing server logs. | +| Observer SSH session also disconnects | Check for a broader network interruption, VM restart, or service failure. | +| Observer SSH session survives | Investigate behavior specific to the transfer connection or its workload. | + +### 3. Compare an independent network + +Repeat the exact byte-count command through a phone hotspot or another +independent network. If a VPN is active, separately compare with it disconnected +where practical. Verify that the test network's public IP is permitted by the +VM's existing SSH access rules; an access-denied or connection-timeout result +does not test bulk-transfer behavior. + +Consistent failure on one network and success on another is strong evidence of +a path-dependent issue. Candidate components include the router, VPN, firewall, +ISP path, and local network filtering. The comparison does not identify which +component is responsible. + +### 4. Isolate packet-priority handling + +On the failing network, repeat the byte-count test with only `IPQoS=none` added: + +```bash +time ssh -T -v -o IPQoS=none -i "$ssh_key" \ + "$remote_target" 'wc -c' < "$source_file" +transfer_status=$? +printf 'SSH exit status: %s\n' "$transfer_status" +``` + +`IPQoS=none` uses the operating system's default packet-priority marking. +OpenSSH 10.1 changed QoS handling, making a controlled comparison relevant to +the OpenSSH 10.2 client. A repeatable improvement warrants examination of packet +marking and network handling; it does not identify a defective component. +See the [SSH option documentation](https://man.openbsd.org/ssh_config#IPQoS) +and [OpenSSH 10.1 release notes](https://www.openssh.org/txt/release-10.1). + +### 5. Follow the evidence + +- If failures follow the network, compare VPN/router/filtering configurations + and, if needed, capture TCP traffic at both endpoints to investigate resets + and retransmissions. A reset observed only at the client can indicate an + intermediary; account for capture coverage before drawing conclusions. +- If server logs identify a crash, resource shortage, or SSH error, investigate + that specific failure before changing algorithms or timeout settings. +- If bulk SSH repeatedly succeeds, reproduce SCP with verbose logging and + inspect the destination filesystem and transfer process: + + ```bash + scp -O -v -i "$ssh_key" "$source_file" "${remote_target}:" + ``` + +- Compare elapsed time and transferred bytes across failures. Similar duration + suggests investigating session limits; a repeatable byte threshold suggests + a data-volume-dependent trigger. Neither pattern alone establishes a cause. +- If the results remain ambiguous, collect `ssh -vvv` output for one controlled + failure and correlate it with the server logs. Inspect effective client + settings with `ssh -G -i "$ssh_key" "$remote_target"` before making persistent + SSH configuration changes. + +## Delivery workaround: private S3 and HTTPS + +Use this route when delivery is needed before the SSH fault is resolved. It +changes the transport path to HTTPS. It requires an existing private bucket, +local AWS credentials authorized to upload and read the object, and outbound +HTTPS access from the VM. + +On the Mac, replace both placeholders: + +```bash +bucket="YOUR-PRIVATE-BUCKET" +region="YOUR-BUCKET-REGION" + +pkgx aws s3 cp "$source_file" \ + "s3://$bucket/transfers/$artifact" --region "$region" +``` + +After the upload succeeds, generate a download URL: + +```bash +pkgx aws s3 presign "s3://$bucket/transfers/$artifact" \ + --region "$region" --expires-in 3600 +``` + +On the VM, paste the URL inside single quotes and use the matching release +filename. Run this from the home directory. The `.download` suffix keeps this +download separate from any partial SCP copy. + +```bash +cd "$HOME" +curl --fail --location --continue-at - \ + --output notify-wakeup-api-amd-0.2.1.tar.bz2.download \ + 'PASTE-PRESIGNED-URL-HERE' +``` + +If interrupted, rerun the same command to resume. If the URL expires, generate +another URL for the same unchanged object and continue the download. The bucket +remains private; anyone holding the URL can read the object until it expires. +Temporary AWS credentials can shorten the URL's effective lifetime. Do not put +the URL or credentials into this plan or committed diagnostic logs. + +If the VM has AWS CLI and an IAM role authorized to read the object, it can +instead download with `aws s3 cp`, using its role credentials. + +See [AWS S3 upload documentation](https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html), +[presigned URL documentation](https://docs.aws.amazon.com/cli/latest/reference/s3/presign.html), +and [curl resume documentation](https://curl.se/docs/manpage.html#-C). + +## Integrity and completion + +Compare the complete SHA-256 strings from the Mac and the VM. For the S3 route: + +```bash +# Mac, using the setup variables: +shasum -a 256 "$source_file" + +# VM: +sha256sum "$HOME/notify-wakeup-api-amd-0.2.1.tar.bz2.download" +``` + +For SCP or rsync, hash the destination archive without the `.download` suffix. +Only use the archive for deployment once the transfer succeeds and the hashes +match. For the S3 route, rename the verified `.download` file to the intended +archive filename as part of deployment. + +- [ ] Capture a failed attempt with matching client and server logs. +- [ ] Run the bulk SSH byte-count test. +- [ ] Compare an independent network and, where applicable, VPN state. +- [ ] Compare `IPQoS=none` while holding other variables constant. +- [ ] Deliver the archive and confirm matching SHA-256 hashes. +- [ ] Identify and document the cause, or explicitly record remaining uncertainty. +- [ ] Verify any proposed fix with repeated complete transfers over the affected path. + +Successful S3 delivery completes the delivery task; it does not establish that +the SSH fault is fixed. + +## Investigation history and evidence + +The reported legacy SCP attempt used OpenSSH `10.2p1` with LibreSSL `3.3.6` on +the Mac. The server advertised `OpenSSH_7.6p1 Ubuntu-4ubuntu0.5`. The connection +used IPv4 address `52.86.179.15` and negotiated `aes128-gcm@openssh.com` with no +compression. The address is an observation from that attempt, not a requirement +to bypass DNS for subsequent tests. + +Relevant log excerpt: + +```text +Authenticated to test-notify-api.timesafari.app ([52.86.179.15]:22) using "publickey". +debug1: Sending command: scp -v -t . +Sending file modes: C0600 240799305 notify-wakeup-api-amd-0.2.1.tar.bz2 +Sink: C0600 240799305 notify-wakeup-api-amd-0.2.1.tar.bz2 +Read from remote host test-notify-api.timesafari.app: Connection reset by peer +client_loop: send disconnect: Broken pipe +lost connection +``` + +Confidence is high that authentication and remote SCP startup succeeded and +that the connection reset at the beginning of the transfer. Confidence in a +specific root cause is unknown. The progress display showed zero percent; that +does not prove that zero payload bytes crossed the network. The reset message +does not distinguish the VM from an intermediary as the source of the reset. + +The earlier `no pubkey loaded` messages did not prevent successful private-key +authentication. The IPv4 connection makes an IPv4-only retry uninformative for +this particular attempt. The legacy SCP failure means an SFTP-only explanation +cannot account for all reported failures. OpenSSH documents the `-O` protocol +selection in its [SCP manual](https://man.openbsd.org/scp). + +The user also reported rsync advancing approximately 1–5% per run before SSH +closed, with cipher overrides, `Compression=no`, `IPQoS=none`, and +`--bwlimit=5000`. Several settings changed together, so improvement cannot be +attributed to a cipher. `IPQoS=none` did not eliminate those reported failures. +The original examples used different hostnames and identity filenames, which +is why the diagnostic sequence fixes both values. + +Rsync's `-P` includes `--partial`; adding another `--partial` is redundant. +Retaining partial files can reduce retransmission on subsequent attempts but +does not repair the connection. See the +[rsync manual](https://download.samba.org/pub/rsync/rsync.1). + +SSH keepalives can detect an unresponsive connection and help some idle-path +problems. They do not reconnect a reset session; the failure during active +transfer provides no evidence that an idle timeout is the cause. See +[ServerAliveInterval](https://man.openbsd.org/ssh_config#ServerAliveInterval). diff --git a/progress/geo-video-workflow.design-map.json b/progress/geo-video-workflow.design-map.json new file mode 100644 index 0000000..28487b0 --- /dev/null +++ b/progress/geo-video-workflow.design-map.json @@ -0,0 +1,110 @@ +{ + "title": "My GEO videos", + "dimensions": [ + { + "id": "flow", + "name": "Video flow" + } + ], + "nodes": [ + { + "id": "nwsv3ea", + "label": "Recorded", + "type": "data", + "status": "proposed", + "notes": "", + "at": { + "flow": { + "x": 140, + "y": 160 + } + } + }, + { + "id": "nwdwbeu", + "label": "A new day!", + "type": "system", + "status": "proposed", + "notes": "", + "at": { + "flow": { + "x": -91, + "y": 157 + } + } + }, + { + "id": "nlv04nm", + "label": "Uploaded to TikTok & captioned", + "type": "data", + "status": "proposed", + "notes": "", + "at": { + "flow": { + "x": 358, + "y": 158 + } + } + }, + { + "id": "nfezk8x", + "label": "Uploaded to Facebook", + "type": "data", + "status": "proposed", + "notes": "", + "at": { + "flow": { + "x": 590, + "y": 127 + } + } + }, + { + "id": "n6bicpt", + "label": "Uploaded to Instagram", + "type": "data", + "status": "proposed", + "notes": "", + "at": { + "flow": { + "x": 591, + "y": 219 + } + } + } + ], + "links": [ + { + "id": "lg4qm0r", + "dim": "flow", + "from": "nwdwbeu", + "to": "nwsv3ea", + "label": "", + "notes": "" + }, + { + "id": "luj5dlj", + "dim": "flow", + "from": "nwsv3ea", + "to": "nlv04nm", + "label": "", + "notes": "" + }, + { + "id": "lfyqqyp", + "dim": "flow", + "from": "nlv04nm", + "to": "nfezk8x", + "label": "", + "notes": "" + }, + { + "id": "l9jyvae", + "dim": "flow", + "from": "nlv04nm", + "to": "n6bicpt", + "label": "", + "notes": "" + } + ] +} diff --git a/project.task.yaml b/project.task.yaml index 447de87..fa6be08 100644 --- a/project.task.yaml +++ b/project.task.yaml @@ -1,3 +1,8 @@ +# +# Overall tasks for this effort +# +# For the format in here, see taskyaml.org +# tasks: diff --git a/projects.md b/projects.md index 0247f45..272807d 100644 --- a/projects.md +++ b/projects.md @@ -4,7 +4,6 @@ ## Documentation & Planning Repos - `endorser-docs/` - Technical documentation -- `README-time-safari/` - Time Safari specific docs - `lives-of-gifts/` - General project content - `uport-demo/` - Endorser web pages - `gifting-org-content/` - Marketing/educational content