2026-10-04 02:00:01 [INFO] Log file: /data/log/aws_bkup-git-repo_archive_delete.1844155.log
2026-10-04 02:00:01 [INFO] ******** LIVE MODE ********
2026-10-04 02:00:01 [INFO] Starting backup pruning for plan 'bkup-git-repo'
2026-10-04 02:00:01 [INFO] Found credentials in shared credentials file: ~/.aws/credentials
2026-10-04 02:00:02 [INFO] Found backup plan ID 51cd3ce2-aec9-4162-a0d0-1c0f8dad4ea5 for 'bkup-git-repo'.
2026-10-04 02:00:02 [INFO] Vaults used by this plan: backup-daily
2026-10-04 02:00:02 [INFO] Retrieving recovery points from vault 'backup-daily'...
2026-10-04 02:00:02 [INFO]   Found 7 recovery points in backup-daily
2026-10-04 02:00:02 [INFO] Total recovery points found: 7
2026-10-04 02:00:02 [INFO] Monthly archive months: 2026-10, 2026-09, 2026-08, 2026-07, 2026-06, 2026-05, 2026-04, 2026-03, 2026-02, 2026-01, 2025-12, 2025-11
2026-10-04 02:00:02 [INFO] Weekly retention Mondays: 2026-09-14, 2026-09-21, 2026-09-28
2026-10-04 02:00:02 [INFO] ==================================================
2026-10-04 02:00:02 [INFO] KEEP: 6 recovery points
2026-10-04 02:00:02 [INFO] KEEP: arn:aws:ec2:us-west-2::image/ami-0c2f50a2b224703f8 | 2026-09-14 05:00:00+00:00 | EC2
2026-10-04 02:00:02 [INFO] KEEP: arn:aws:ec2:us-west-2::image/ami-0c503743096a94751 | 2026-09-21 05:00:00+00:00 | EC2
2026-10-04 02:00:02 [INFO] KEEP: arn:aws:ec2:us-west-2::image/ami-0f235896b5fde5a05 | 2026-09-28 05:00:00+00:00 | EC2
2026-10-04 02:00:02 [INFO] KEEP: arn:aws:ec2:us-west-2::image/ami-08371301d1ed851ae | 2026-10-02 05:00:00+00:00 | EC2
2026-10-04 02:00:02 [INFO] KEEP: arn:aws:ec2:us-west-2::image/ami-022fc277cb286831f | 2026-10-03 05:00:00+00:00 | EC2
2026-10-04 02:00:02 [INFO] KEEP: arn:aws:ec2:us-west-2::image/ami-0988a6ca63765296e | 2026-10-04 05:00:00+00:00 | EC2
2026-10-04 02:00:02 [INFO] ==================================================
2026-10-04 02:00:02 [INFO] ARCHIVE: 1 recovery points
2026-10-04 02:00:02 [INFO] ARCHIVE: arn:aws:ec2:us-west-2::image/ami-01e2f1cff42b24b99 | AMI=ami-01e2f1cff42b24b99 | 2182.00 GB | METHOD=S3-DEEP-ARCHIVE | 2026-10-01 05:00:00+00:00
2026-10-04 02:00:02 [INFO] ==================================================
2026-10-04 02:00:02 [INFO] SKIP: 0 recovery points
2026-10-04 02:00:02 [INFO] ==================================================
2026-10-04 02:00:02 [INFO] DELETE: 0 recovery points
2026-10-04 02:00:02 [INFO] Waiting for global AMI archive lock...
2026-10-04 07:12:13 [INFO] AMI archive lock acquired.
2026-10-04 07:12:13 [INFO] Starting S3 Deep Archive workflow for ami-01e2f1cff42b24b99
2026-10-04 07:12:13 [INFO] AMI ami-01e2f1cff42b24b99 is 2182.00 GB <= 5000 GB. Using S3-DEEP-ARCHIVE.
2026-10-04 07:12:13 [INFO] ==================================================
2026-10-04 07:12:13 [INFO] ARCHIVING AMI: ami-01e2f1cff42b24b99
2026-10-04 07:12:13 [INFO] Creating S3 AMI store task for ami-01e2f1cff42b24b99...
2026-10-04 07:12:49 [INFO] AWS store task created: s3://backup-old-amis/ami-01e2f1cff42b24b99.bin
2026-10-04 07:12:49 [INFO] Waiting for CreateStoreImageTask for ami-01e2f1cff42b24b99...
2026-10-04 07:12:50 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=0% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:13:50 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=2% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:14:50 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=5% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:15:50 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=7% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:16:50 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=9% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:17:50 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=11% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:18:50 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=12% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:19:50 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=14% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:20:51 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=16% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:21:51 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=18% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:22:51 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=20% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:23:51 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=22% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:24:51 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=24% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:25:51 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=26% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:26:51 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=28% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:27:51 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=30% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:28:52 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=32% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:29:52 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=34% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:30:52 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=36% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:31:52 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=38% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:32:52 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=40% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:33:52 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=41% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:34:52 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=43% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:35:52 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=45% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:36:52 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=47% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:37:53 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=49% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:38:53 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=51% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:39:53 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=53% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:40:53 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=56% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:41:53 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=59% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:42:53 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=63% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:43:53 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=68% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:44:54 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=72% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:45:54 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=77% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:46:54 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=82% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:47:54 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=86% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:48:54 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=89% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:49:54 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=90% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:50:54 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=92% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:51:54 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=94% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:52:54 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=95% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:53:55 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=96% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:54:55 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=97% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:55:55 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=97% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:56:55 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=98% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:57:55 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=98% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:58:55 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 07:59:55 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:00:55 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:01:55 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:02:56 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:03:56 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:04:56 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:05:56 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:06:56 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:07:56 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:08:56 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:09:56 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:10:57 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:11:57 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:12:57 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:13:57 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:14:57 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:15:57 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:16:57 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:17:57 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:18:58 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:19:58 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:20:58 [INFO] AMI ami-01e2f1cff42b24b99: state=InProgress progress=99% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:21:58 [INFO] AMI ami-01e2f1cff42b24b99: state=Completed progress=100% object=ami-01e2f1cff42b24b99.bin
2026-10-04 08:21:58 [INFO] Verifying temporary S3 object...
2026-10-04 08:21:58 [INFO] Temporary S3 object exists: 438305851116 bytes
2026-10-04 08:21:58 [INFO] Final archive key: s3://backup-old-amis/git_ami-01e2f1cff42b24b99.bin
2026-10-04 08:21:58 [INFO] Copying archive to Deep Archive using AWS CLI...
2026-10-04 08:21:58 [INFO] Running AWS CLI S3 copy: s3://backup-old-amis/ami-01e2f1cff42b24b99.bin -> s3://backup-old-amis/git_ami-01e2f1cff42b24b99.bin
2026-10-04 08:32:49 [INFO] Verifying Deep Archive object...
2026-10-04 08:32:49 [INFO] Final archive: size=438305851116 bytes, storage_class=DEEP_ARCHIVE
2026-10-04 08:32:49 [INFO] Deep Archive verification PASSED for ami-01e2f1cff42b24b99
2026-10-04 08:32:49 [INFO] Deleting temporary Standard S3 object: ami-01e2f1cff42b24b99.bin
2026-10-04 08:32:49 [ERROR] ARCHIVE FAILED for ami-01e2f1cff42b24b99. Recovery point WILL NOT be deleted.
Traceback (most recent call last):
  File "/home/prod/gqr/extra/env/prune_aws_backups_by_plan.py", line 1483, in main
    final_key = archive_ami_to_deep_archive(
  File "/home/prod/gqr/extra/env/prune_aws_backups_by_plan.py", line 1033, in archive_ami_to_deep_archive
    s3_client.delete_object(
  File "/home/prod/miniconda3/envs/env1/lib/python3.9/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/prod/miniconda3/envs/env1/lib/python3.9/site-packages/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the DeleteObject operation: User: arn:aws:iam::353199608714:root is not authorized to perform: s3:DeleteObject on resource: "arn:aws:s3:::backup-old-amis/ami-01e2f1cff42b24b99.bin" with an explicit deny in a resource-based policy
2026-10-04 08:32:49 [INFO] AMI archive lock released.
2026-10-04 08:32:49 [INFO] ==================================================
2026-10-04 08:32:49 [INFO] JOB COMPLETE for backup plan 'bkup-git-repo'.
2026-10-04 08:32:49 [INFO] ==================================================