trent_larson/README-first
4
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

add design-map for GEO videos, add AWS-upload problem details, and other tiny tweaks

Browse Source
This commit is contained in:
trentlarson
2026-10-19 11:16:38 -06:00
parent 2cec1ae570
commit cb5aec31a0
4 changed files with 392 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
progress/TASK-aws-upload-problem.md
+277
View File
@@ -0,0 +1,277 @@
# PLAN: Debug AWS file-transfer disconnects
## Objective
Transfer the notification-wakeup API release archive to the test VM, identify
what terminates SSH transfers, and verify the resulting file's SHA-256 hash.
Status: investigation pending. The commands below are proposed diagnostic steps;
they have not been executed as part of writing this plan.
## Target and test controls
Use the same destination, identity, and archive for every comparison:
- Host: `test-notify-api.timesafari.app`
- User: `ubuntutest`
- Identity: `~/.ssh/key3.pem`
- Local archive: `~/dl/notify-wakeup-api-amd-0.2.1.tar.bz2`
- Expected size for that archive: `240799305` bytes
On the Mac, set these variables in the terminal used for testing. Change the
version only if testing a different release, and keep the archive unchanged
throughout the comparisons.
```bash
NOTIFY_WAKEUP_API_VERSION=0.2.1
artifact="notify-wakeup-api-amd-${NOTIFY_WAKEUP_API_VERSION:?Set NOTIFY_WAKEUP_API_VERSION}.tar.bz2"
source_file="$HOME/dl/$artifact"
remote_target="ubuntutest@test-notify-api.timesafari.app"
ssh_key="$HOME/.ssh/key3.pem"
wc -c < "$source_file"
shasum -a 256 "$source_file"
```
For each attempt, record the command, network/VPN state, elapsed time, bytes
reported, exit status, and matching server logs. Change one variable at a time.
Use `date -u` on both machines to correlate log timestamps.
## Diagnostic sequence
### 1. Observe the server while reproducing the failure
Open a separate Mac terminal and start an SSH session to the VM:
```bash
ssh -i ~/.ssh/key3.pem ubuntutest@test-notify-api.timesafari.app
```
Inside that session, follow the authentication log:
```bash
sudo tail -F /var/log/auth.log
```
If that file is unavailable, use the SSH service journal:
```bash
sudo journalctl -f -u ssh
```
Leave the observer session open while running the transfer tests from the Mac.
Record whether this session survives each failure. After a failure, inspect
kernel messages and resource availability on the VM:
```bash
sudo journalctl -k --since "5 minutes ago" --no-pager
df -h "$HOME"
df -i "$HOME"
free -h
uptime
```
Look for SSH errors, process crashes, out-of-memory kills, storage errors, and
reboots at the failure time. A full filesystem or exhausted inodes warrants
investigation, but does not by itself explain a TCP reset. If the VM becomes
unreachable, inspect its EC2 status checks and console output through AWS.
### 2. Test bulk SSH traffic without SCP or destination-file writes
Run on the Mac, using the variables from the setup block:
```bash
time ssh -T -v -i "$ssh_key" "$remote_target" 'wc -c' < "$source_file"
transfer_status=$?
printf 'SSH exit status: %s\n' "$transfer_status"
```
The remote process counts and discards the incoming bytes. It does not save the
archive. Success means the printed count matches the local `wc -c` result and
SSH exits with status zero; for release 0.2.1 the count is `240799305`.
| Result | Interpretation and next action |
| --- | --- |
| Connection resets | SCP, SFTP, and destination-file writes are unnecessary to trigger the failure. Investigate SSH, server health, and the network path. |
| Repeated success | Investigate the file-transfer process and destination storage. Repeat the failing SCP command while observing server logs. |
| Observer SSH session also disconnects | Check for a broader network interruption, VM restart, or service failure. |
| Observer SSH session survives | Investigate behavior specific to the transfer connection or its workload. |
### 3. Compare an independent network
Repeat the exact byte-count command through a phone hotspot or another
independent network. If a VPN is active, separately compare with it disconnected
where practical. Verify that the test network's public IP is permitted by the
VM's existing SSH access rules; an access-denied or connection-timeout result
does not test bulk-transfer behavior.
Consistent failure on one network and success on another is strong evidence of
a path-dependent issue. Candidate components include the router, VPN, firewall,
ISP path, and local network filtering. The comparison does not identify which
component is responsible.
### 4. Isolate packet-priority handling
On the failing network, repeat the byte-count test with only `IPQoS=none` added:
```bash
time ssh -T -v -o IPQoS=none -i "$ssh_key" \
"$remote_target" 'wc -c' < "$source_file"
transfer_status=$?
printf 'SSH exit status: %s\n' "$transfer_status"
```
`IPQoS=none` uses the operating system's default packet-priority marking.
OpenSSH 10.1 changed QoS handling, making a controlled comparison relevant to
the OpenSSH 10.2 client. A repeatable improvement warrants examination of packet
marking and network handling; it does not identify a defective component.
See the [SSH option documentation](https://man.openbsd.org/ssh_config#IPQoS)
and [OpenSSH 10.1 release notes](https://www.openssh.org/txt/release-10.1).
### 5. Follow the evidence
- If failures follow the network, compare VPN/router/filtering configurations
and, if needed, capture TCP traffic at both endpoints to investigate resets
and retransmissions. A reset observed only at the client can indicate an
intermediary; account for capture coverage before drawing conclusions.
- If server logs identify a crash, resource shortage, or SSH error, investigate
that specific failure before changing algorithms or timeout settings.
- If bulk SSH repeatedly succeeds, reproduce SCP with verbose logging and
inspect the destination filesystem and transfer process:
```bash
scp -O -v -i "$ssh_key" "$source_file" "${remote_target}:"
```
- Compare elapsed time and transferred bytes across failures. Similar duration
suggests investigating session limits; a repeatable byte threshold suggests
a data-volume-dependent trigger. Neither pattern alone establishes a cause.
- If the results remain ambiguous, collect `ssh -vvv` output for one controlled
failure and correlate it with the server logs. Inspect effective client
settings with `ssh -G -i "$ssh_key" "$remote_target"` before making persistent
SSH configuration changes.
## Delivery workaround: private S3 and HTTPS
Use this route when delivery is needed before the SSH fault is resolved. It
changes the transport path to HTTPS. It requires an existing private bucket,
local AWS credentials authorized to upload and read the object, and outbound
HTTPS access from the VM.
On the Mac, replace both placeholders:
```bash
bucket="YOUR-PRIVATE-BUCKET"
region="YOUR-BUCKET-REGION"
pkgx aws s3 cp "$source_file" \
"s3://$bucket/transfers/$artifact" --region "$region"
```
After the upload succeeds, generate a download URL:
```bash
pkgx aws s3 presign "s3://$bucket/transfers/$artifact" \
--region "$region" --expires-in 3600
```
On the VM, paste the URL inside single quotes and use the matching release
filename. Run this from the home directory. The `.download` suffix keeps this
download separate from any partial SCP copy.
```bash
cd "$HOME"
curl --fail --location --continue-at - \
--output notify-wakeup-api-amd-0.2.1.tar.bz2.download \
'PASTE-PRESIGNED-URL-HERE'
```
If interrupted, rerun the same command to resume. If the URL expires, generate
another URL for the same unchanged object and continue the download. The bucket
remains private; anyone holding the URL can read the object until it expires.
Temporary AWS credentials can shorten the URL's effective lifetime. Do not put
the URL or credentials into this plan or committed diagnostic logs.
If the VM has AWS CLI and an IAM role authorized to read the object, it can
instead download with `aws s3 cp`, using its role credentials.
See [AWS S3 upload documentation](https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html),
[presigned URL documentation](https://docs.aws.amazon.com/cli/latest/reference/s3/presign.html),
and [curl resume documentation](https://curl.se/docs/manpage.html#-C).
## Integrity and completion
Compare the complete SHA-256 strings from the Mac and the VM. For the S3 route:
```bash
# Mac, using the setup variables:
shasum -a 256 "$source_file"
# VM:
sha256sum "$HOME/notify-wakeup-api-amd-0.2.1.tar.bz2.download"
```
For SCP or rsync, hash the destination archive without the `.download` suffix.
Only use the archive for deployment once the transfer succeeds and the hashes
match. For the S3 route, rename the verified `.download` file to the intended
archive filename as part of deployment.
- [ ] Capture a failed attempt with matching client and server logs.
- [ ] Run the bulk SSH byte-count test.
- [ ] Compare an independent network and, where applicable, VPN state.
- [ ] Compare `IPQoS=none` while holding other variables constant.
- [ ] Deliver the archive and confirm matching SHA-256 hashes.
- [ ] Identify and document the cause, or explicitly record remaining uncertainty.
- [ ] Verify any proposed fix with repeated complete transfers over the affected path.
Successful S3 delivery completes the delivery task; it does not establish that
the SSH fault is fixed.
## Investigation history and evidence
The reported legacy SCP attempt used OpenSSH `10.2p1` with LibreSSL `3.3.6` on
the Mac. The server advertised `OpenSSH_7.6p1 Ubuntu-4ubuntu0.5`. The connection
used IPv4 address `52.86.179.15` and negotiated `aes128-gcm@openssh.com` with no
compression. The address is an observation from that attempt, not a requirement
to bypass DNS for subsequent tests.
Relevant log excerpt:
```text
Authenticated to test-notify-api.timesafari.app ([52.86.179.15]:22) using "publickey".
debug1: Sending command: scp -v -t .
Sending file modes: C0600 240799305 notify-wakeup-api-amd-0.2.1.tar.bz2
Sink: C0600 240799305 notify-wakeup-api-amd-0.2.1.tar.bz2
Read from remote host test-notify-api.timesafari.app: Connection reset by peer
client_loop: send disconnect: Broken pipe
lost connection
```
Confidence is high that authentication and remote SCP startup succeeded and
that the connection reset at the beginning of the transfer. Confidence in a
specific root cause is unknown. The progress display showed zero percent; that
does not prove that zero payload bytes crossed the network. The reset message
does not distinguish the VM from an intermediary as the source of the reset.
The earlier `no pubkey loaded` messages did not prevent successful private-key
authentication. The IPv4 connection makes an IPv4-only retry uninformative for
this particular attempt. The legacy SCP failure means an SFTP-only explanation
cannot account for all reported failures. OpenSSH documents the `-O` protocol
selection in its [SCP manual](https://man.openbsd.org/scp).
The user also reported rsync advancing approximately 1–5% per run before SSH
closed, with cipher overrides, `Compression=no`, `IPQoS=none`, and
`--bwlimit=5000`. Several settings changed together, so improvement cannot be
attributed to a cipher. `IPQoS=none` did not eliminate those reported failures.
The original examples used different hostnames and identity filenames, which
is why the diagnostic sequence fixes both values.
Rsync's `-P` includes `--partial`; adding another `--partial` is redundant.
Retaining partial files can reduce retransmission on subsequent attempts but
does not repair the connection. See the
[rsync manual](https://download.samba.org/pub/rsync/rsync.1).
SSH keepalives can detect an unresponsive connection and help some idle-path
problems. They do not reconnect a reset session; the failure during active
transfer provides no evidence that an idle timeout is the cause. See
[ServerAliveInterval](https://man.openbsd.org/ssh_config#ServerAliveInterval).