packer构建AWS AMI镜像时报错“scp stderr (length 38): bash: /usr/bin/scp: Permission denied“

现象

在执行 packer build 命令构建自定义AMI时发现provisioner "file"和provisioner "shell"中的脚本文件都无法上传到packer启动的ec2实例中,而且都报错non-zero exit status: 126, Process exited with status 126也们没有其他啥明显的报错信息,被弄得三脸懵逼.

排查过程

开启debug日志

启用搜索引擎大法终于找到可用教程来收集更详细的debug日志, 如下

# 日志路径
export PACKER_LOG_PATH="./packer.log"
# 日志级别,1最详细
export PACKER_LOG=1
# 启用packer debug模式一步步执行方便调试
packer build -debug .

而后发现了更详细的报错信息bash: /usr/bin/scp: Permission denied

2023/04/25 09:52:31 packer-plugin-amazon_v1.2.4_x5.0_linux_amd64 plugin: 2023/04/25 09:52:31 [DEBUG] Starting remote scp process:  scp -vt /tmp
2023/04/25 09:52:31 packer-plugin-amazon_v1.2.4_x5.0_linux_amd64 plugin: 2023/04/25 09:52:31 [DEBUG] Started SCP session, beginning transfers...
2023/04/25 09:52:31 packer-plugin-amazon_v1.2.4_x5.0_linux_amd64 plugin: 2023/04/25 09:52:31 [DEBUG] scp: Uploading script.sh: perms=C0777 size=1092
2023/04/25 09:52:31 packer-plugin-amazon_v1.2.4_x5.0_linux_amd64 plugin: 2023/04/25 09:52:31 [DEBUG] SCP session complete, closing stdin pipe.
2023/04/25 09:52:31 packer-plugin-amazon_v1.2.4_x5.0_linux_amd64 plugin: 2023/04/25 09:52:31 [DEBUG] Waiting for SSH session to complete.
2023/04/25 09:52:31 packer-plugin-amazon_v1.2.4_x5.0_linux_amd64 plugin: 2023/04/25 09:52:31 [DEBUG] scp stderr (length 38): bash: /usr/bin/scp: Permission denied
2023/04/25 09:52:31 packer-plugin-amazon_v1.2.4_x5.0_linux_amd64 plugin: 2023/04/25 09:52:31 [DEBUG] non-zero exit status: 126, Process exited with status 126
2023/04/25 09:52:31 packer-plugin-amazon_v1.2.4_x5.0_linux_amd64 plugin: 2023/04/25 09:52:31 [DEBUG] scp output:
2023/04/25 09:52:31 ui error: ^[[1;31m==> amazon-ebs.autogenerated_1: Upload failed: Process exited with status 126^[[0m
2023/04/25 09:52:31 packer-provisioner-file plugin: closing
2023/04/25 09:52:31 closing

从日志里面就能看出是/usr/bin/scp权限不够导致的, 那么问题来了, 这里/usr/bin/scp是在本地环境还是在packer启动的ec2实例中呢? 根据我不断踩坑发现,虽然我本地/usr/bin/scp虽然也有问题但是罪魁祸首是在ec2实例中的/usr/bin/scp没有权限. 简直了

登录ec2实例测试

要验证scp权限就需要登录到ec2实例中去. packer在构建过程中会打印出ec2实例的ip并在当前目录生成ec2_autogenerated_1.pem用于ssh登录, 日志如下

=> amazon-ebs.autogenerated_1: Pausing after run of step 'StepRunSourceInstance'. Press enter to continue. 
==> amazon-ebs.autogenerated_1: Pausing after run of step 'StepGetPassword'. Press enter to continue. 
==> amazon-ebs.autogenerated_1: Pausing after run of step 'StepCreateSSMTunnel'. Press enter to continue. 
==> amazon-ebs.autogenerated_1: Using SSH communicator to connect: 10.168.77.245
==> amazon-ebs.autogenerated_1: Waiting for SSH to become available...
==> amazon-ebs.autogenerated_1: Connected to SSH!

我们只需要带上自己的用户名例如centos就能执行 ssh -i ec2_autogenerated_1.pem [email protected] 登录,注意用户名要替换成自己的. 登录ec2实例之后发现scp果然存在权限问题,如下:

ssh -i ec2_autogenerated_1.pem [email protected]
[centos@ip tmp]$ scp
-bash: /usr/bin/scp: Permission denied

# 修改权限后scp能正常执行了
[centos@ip tmp]$ sudo chmod 755 /usr/bin/scp
[centos@ip tmp]$ scp
usage: scp [-12346BCpqrv] [-c cipher] [-F ssh_config] [-i identity_file]
           [-l limit] [-o ssh_option] [-P port] [-S program]
           [[user@]host1:]file1 ... [[user@]host2:]file2

解决方法

知道问题后就好办了, 无非是想办法加回scp的权限, 早上无意间看到前辈用到的一种方式就是通过provisioner "shell-local"模块来修改scp的权限如下

....
variable "ssh_additional_args" {
    
    
  type    = string
  default = "-o IdentitiesOnly=yes"
}
build {
    
    
  sources = [
    "source.amazon-ebs.autogenerated_1"
  ]

  #This shell must be kept to avoid "bash: /usr/bin/scp: Permission denied" error
  provisioner "shell-local" {
    
    
    execute_command  = ["/bin/sh", "-c", "echo Hello| {
    
    {.Vars}} {
    
    {.Script}}"]
    environment_vars = ["HELLO_USER=packeruser", "UUID=${build.PackerRunUUID}"]
    inline = ["echo the Packer run uuid is $UUID and ${build.Host}",
      "echo '${build.SSHPrivateKey}' > /tmp/packer-session.pem",
      "chmod 600 /tmp/packer-session.pem",
      "ssh -i /tmp/packer-session.pem ${build.User}@${build.Host} -o \"StrictHostKeyChecking no\" ${var.ssh_additional_args} 'sudo chmod 755 /usr/bin/scp'"]
  }

  provisioner "shell" {
    
    
    script       = "script.sh"
    pause_before = "10s"
    timeout      = "10s"
  }
}

给一个完整的例子

packer {
    
    
  required_plugins {
    
    
    amazon = {
    
    
      version = ">= 1.2.1"
      source  = "github.com/hashicorp/amazon"
    }
  }
}

variable "aws_region" {
    
    
  type    = string
  default = "eu-central-1"
}

variable "aws_regions" {
    
    
  type    = list(string)
  default = ["eu-central-1"]
}


variable "ssh_additional_args" {
    
    
  type    = string
  default = "-o IdentitiesOnly=yes"
}

variable "run_tags" {
    
    
  type = map(string)
  description = "Tags"
  default = {
    
    }
}

variable "vpc_id" {
    
    
  type = string
  description = "(optional) describe your variable"
  default = ""
}

variable "subnet_id" {
    
    
  type = string
  description = "(optional) describe your variable"
  default = ""
}

variable "ami_tags" {
    
    
  type = map(string)
  description = "AMI Tags"
  default = {
    
    }
}

locals {
    
     timestamp = regex_replace(timestamp(), "[- TZ:]", "") }

locals {
    
    
  aws_ami_name = "centos7-${local.timestamp}"
}

data "amazon-ami" "latest-centos7" {
    
    
  filters = {
    
    
    virtualization-type = "hvm"
    name             = "CentOS7-x64-*"
    root-device-type = "ebs"
  }
  #  Account
  owners      = ["xxx"]
  most_recent = true
  region      = var.aws_region
}

source "amazon-ebs" "autogenerated_1" {
    
    
  ami_name      = local.aws_ami_name
  instance_type = "m4.large"
  region        = var.aws_region
  ami_regions   = var.aws_regions
  source_ami    = data.amazon-ami.latest-centos7.id
  ssh_username  = "centos"
  vpc_id             = var.vpc_id
  subnet_id          = var.subnet_id
  tags = var.ami_tags
  run_tags = var.run_tags
  skip_region_validation = true
  ssh_agent_auth              = false
  associate_public_ip_address = false
  ssh_interface               = "private_ip"
  ssh_timeout                 = "5m"
}

build {
    
    
  sources = [
    "source.amazon-ebs.autogenerated_1"
  ]

  #This shell must be kept to avoid "bash: /usr/bin/scp: Permission denied" error
  provisioner "shell-local" {
    
    
    execute_command  = ["/bin/sh", "-c", "echo Hello| {
    
    {.Vars}} {
    
    {.Script}}"]
    environment_vars = ["HELLO_USER=packeruser", "UUID=${build.PackerRunUUID}"]
    inline = ["echo the Packer run uuid is $UUID and ${build.Host}",
      "echo '${build.SSHPrivateKey}' > /tmp/packer-session.pem",
      "chmod 600 /tmp/packer-session.pem",
      "ssh -i /tmp/packer-session.pem ${build.User}@${build.Host} -o \"StrictHostKeyChecking no\" ${var.ssh_additional_args} 'sudo chmod 755 /usr/bin/scp'"]
  }

  provisioner "shell" {
    
    
    script       = "script.sh"
    pause_before = "10s"
    timeout      = "10s"
  }
}

题外话, 用sftp替代scp传输文件

在搜索资料分析packer历史问题的过程中无意间发现有人实现了支持sftp的新功能, 原作者给出了json格式的例子, 更多详细信息请参考Add sftp file transfer support #2504

{
    
    
{
    
    
    "builders": [{
    
    
        "type": "amazon-ebs",
        "ami_name": "packer-test {
    
    {timestamp}}",
        "instance_type": "t2.micro",
        "region": "us-east-1",
        "ssh_username": "ec2-user",
	"ssh_file_transfer_method": "sftp",
        "source_ami": "ami-8da458e6",
        "tags": {
    
    
            "packer-test": "true"
        }
    }],

    "provisioners": [{
    
    
        "type": "file",
        "source": "file.txt",
        "destination": "/tmp/file.txt"
    }, {
    
    
        "type": "shell",
        "inline": ["cat /tmp/file.txt"]
    }]
}

参考

1. command for debugging and log packer build in windows
2. Add sftp file transfer support #2504