README-first/progress/TASK-aws-upload-problem.md

PLAN: Debug AWS file-transfer disconnects

Objective

Transfer the notification-wakeup API release archive to the test VM, identify what terminates SSH transfers, and verify the resulting file's SHA-256 hash.

Status: investigation pending. The commands below are proposed diagnostic steps; they have not been executed as part of writing this plan.

Target and test controls

Use the same destination, identity, and archive for every comparison:

• Host: test-notify-api.timesafari.app
• User: ubuntutest
• Identity: ~/.ssh/key3.pem
• Local archive: ~/dl/notify-wakeup-api-amd-0.2.1.tar.bz2
• Expected size for that archive: 240799305 bytes

On the Mac, set these variables in the terminal used for testing. Change the version only if testing a different release, and keep the archive unchanged throughout the comparisons.

NOTIFY_WAKEUP_API_VERSION=0.2.1
artifact="notify-wakeup-api-amd-${NOTIFY_WAKEUP_API_VERSION:?Set NOTIFY_WAKEUP_API_VERSION}.tar.bz2"
source_file="$HOME/dl/$artifact"
remote_target="ubuntutest@test-notify-api.timesafari.app"
ssh_key="$HOME/.ssh/key3.pem"

wc -c < "$source_file"
shasum -a 256 "$source_file"

For each attempt, record the command, network/VPN state, elapsed time, bytes reported, exit status, and matching server logs. Change one variable at a time. Use date -u on both machines to correlate log timestamps.

Diagnostic sequence

1. Observe the server while reproducing the failure

Open a separate Mac terminal and start an SSH session to the VM:

ssh -i ~/.ssh/key3.pem ubuntutest@test-notify-api.timesafari.app

Inside that session, follow the authentication log:

sudo tail -F /var/log/auth.log

If that file is unavailable, use the SSH service journal:

sudo journalctl -f -u ssh

Leave the observer session open while running the transfer tests from the Mac. Record whether this session survives each failure. After a failure, inspect kernel messages and resource availability on the VM:

sudo journalctl -k --since "5 minutes ago" --no-pager
df -h "$HOME"
df -i "$HOME"
free -h
uptime

Look for SSH errors, process crashes, out-of-memory kills, storage errors, and reboots at the failure time. A full filesystem or exhausted inodes warrants investigation, but does not by itself explain a TCP reset. If the VM becomes unreachable, inspect its EC2 status checks and console output through AWS.

2. Test bulk SSH traffic without SCP or destination-file writes

Run on the Mac, using the variables from the setup block:

time ssh -T -v -i "$ssh_key" "$remote_target" 'wc -c' < "$source_file"
transfer_status=$?
printf 'SSH exit status: %s\n' "$transfer_status"

The remote process counts and discards the incoming bytes. It does not save the archive. Success means the printed count matches the local wc -c result and SSH exits with status zero; for release 0.2.1 the count is 240799305.

Result	Interpretation and next action
Connection resets	SCP, SFTP, and destination-file writes are unnecessary to trigger the failure. Investigate SSH, server health, and the network path.
Repeated success	Investigate the file-transfer process and destination storage. Repeat the failing SCP command while observing server logs.
Observer SSH session also disconnects	Check for a broader network interruption, VM restart, or service failure.
Observer SSH session survives	Investigate behavior specific to the transfer connection or its workload.

3. Compare an independent network

Repeat the exact byte-count command through a phone hotspot or another independent network. If a VPN is active, separately compare with it disconnected where practical. Verify that the test network's public IP is permitted by the VM's existing SSH access rules; an access-denied or connection-timeout result does not test bulk-transfer behavior.

Consistent failure on one network and success on another is strong evidence of a path-dependent issue. Candidate components include the router, VPN, firewall, ISP path, and local network filtering. The comparison does not identify which component is responsible.

4. Isolate packet-priority handling

On the failing network, repeat the byte-count test with only IPQoS=none added:

time ssh -T -v -o IPQoS=none -i "$ssh_key" \
  "$remote_target" 'wc -c' < "$source_file"
transfer_status=$?
printf 'SSH exit status: %s\n' "$transfer_status"

IPQoS=none uses the operating system's default packet-priority marking. OpenSSH 10.1 changed QoS handling, making a controlled comparison relevant to the OpenSSH 10.2 client. A repeatable improvement warrants examination of packet marking and network handling; it does not identify a defective component. See the SSH option documentation and OpenSSH 10.1 release notes.

5. Follow the evidence

• If failures follow the network, compare VPN/router/filtering configurations and, if needed, capture TCP traffic at both endpoints to investigate resets and retransmissions. A reset observed only at the client can indicate an intermediary; account for capture coverage before drawing conclusions.

• If server logs identify a crash, resource shortage, or SSH error, investigate that specific failure before changing algorithms or timeout settings.

• If bulk SSH repeatedly succeeds, reproduce SCP with verbose logging and inspect the destination filesystem and transfer process:

  scp -O -v -i "$ssh_key" "$source_file" "${remote_target}:"
  

• Compare elapsed time and transferred bytes across failures. Similar duration suggests investigating session limits; a repeatable byte threshold suggests a data-volume-dependent trigger. Neither pattern alone establishes a cause.

• If the results remain ambiguous, collect ssh -vvv output for one controlled failure and correlate it with the server logs. Inspect effective client settings with ssh -G -i "$ssh_key" "$remote_target" before making persistent SSH configuration changes.

Delivery workaround: private S3 and HTTPS

Use this route when delivery is needed before the SSH fault is resolved. It changes the transport path to HTTPS. It requires an existing private bucket, local AWS credentials authorized to upload and read the object, and outbound HTTPS access from the VM.

On the Mac, replace both placeholders:

bucket="YOUR-PRIVATE-BUCKET"
region="YOUR-BUCKET-REGION"

pkgx aws s3 cp "$source_file" \
  "s3://$bucket/transfers/$artifact" --region "$region"

After the upload succeeds, generate a download URL:

pkgx aws s3 presign "s3://$bucket/transfers/$artifact" \
  --region "$region" --expires-in 3600

On the VM, paste the URL inside single quotes and use the matching release filename. Run this from the home directory. The .download suffix keeps this download separate from any partial SCP copy.

cd "$HOME"
curl --fail --location --continue-at - \
  --output notify-wakeup-api-amd-0.2.1.tar.bz2.download \
  'PASTE-PRESIGNED-URL-HERE'

If interrupted, rerun the same command to resume. If the URL expires, generate another URL for the same unchanged object and continue the download. The bucket remains private; anyone holding the URL can read the object until it expires. Temporary AWS credentials can shorten the URL's effective lifetime. Do not put the URL or credentials into this plan or committed diagnostic logs.

If the VM has AWS CLI and an IAM role authorized to read the object, it can instead download with aws s3 cp, using its role credentials.

See AWS S3 upload documentation, presigned URL documentation, and curl resume documentation.

Integrity and completion

Compare the complete SHA-256 strings from the Mac and the VM. For the S3 route:

# Mac, using the setup variables:
shasum -a 256 "$source_file"

# VM:
sha256sum "$HOME/notify-wakeup-api-amd-0.2.1.tar.bz2.download"

For SCP or rsync, hash the destination archive without the .download suffix. Only use the archive for deployment once the transfer succeeds and the hashes match. For the S3 route, rename the verified .download file to the intended archive filename as part of deployment.

• Capture a failed attempt with matching client and server logs.
• Run the bulk SSH byte-count test.
• Compare an independent network and, where applicable, VPN state.
• Compare IPQoS=none while holding other variables constant.
• Deliver the archive and confirm matching SHA-256 hashes.
• Identify and document the cause, or explicitly record remaining uncertainty.
• Verify any proposed fix with repeated complete transfers over the affected path.

Successful S3 delivery completes the delivery task; it does not establish that the SSH fault is fixed.

Investigation history and evidence

The reported legacy SCP attempt used OpenSSH 10.2p1 with LibreSSL 3.3.6 on the Mac. The server advertised OpenSSH_7.6p1 Ubuntu-4ubuntu0.5. The connection used IPv4 address 52.86.179.15 and negotiated aes128-gcm@openssh.com with no compression. The address is an observation from that attempt, not a requirement to bypass DNS for subsequent tests.

Relevant log excerpt:

Authenticated to test-notify-api.timesafari.app ([52.86.179.15]:22) using "publickey".
debug1: Sending command: scp -v -t .
Sending file modes: C0600 240799305 notify-wakeup-api-amd-0.2.1.tar.bz2
Sink: C0600 240799305 notify-wakeup-api-amd-0.2.1.tar.bz2
Read from remote host test-notify-api.timesafari.app: Connection reset by peer
client_loop: send disconnect: Broken pipe
lost connection

Confidence is high that authentication and remote SCP startup succeeded and that the connection reset at the beginning of the transfer. Confidence in a specific root cause is unknown. The progress display showed zero percent; that does not prove that zero payload bytes crossed the network. The reset message does not distinguish the VM from an intermediary as the source of the reset.

The earlier no pubkey loaded messages did not prevent successful private-key authentication. The IPv4 connection makes an IPv4-only retry uninformative for this particular attempt. The legacy SCP failure means an SFTP-only explanation cannot account for all reported failures. OpenSSH documents the -O protocol selection in its SCP manual.

The user also reported rsync advancing approximately 1–5% per run before SSH closed, with cipher overrides, Compression=no, IPQoS=none, and --bwlimit=5000. Several settings changed together, so improvement cannot be attributed to a cipher. IPQoS=none did not eliminate those reported failures. The original examples used different hostnames and identity filenames, which is why the diagnostic sequence fixes both values.

Rsync's -P includes --partial; adding another --partial is redundant. Retaining partial files can reduce retransmission on subsequent attempts but does not repair the connection. See the rsync manual.

SSH keepalives can detect an unresponsive connection and help some idle-path problems. They do not reconnect a reset session; the failure during active transfer provides no evidence that an idle timeout is the cause. See ServerAliveInterval.