ds-controller是一个命令工具,可以用于创建、释放和扩缩容E-MapReduce(简称EMR)集群,也可以用于管控EMR相关的VPC、vSwitch和安全组等,从而管理EMR集群。ds-controller有命令行模式和服务模式两种运行模式。
前提条件
已创建DataScience集群,详情信息请参见创建集群。
背景信息
注意 释放集群节点时,请严格按照命令操作,否则会影响集群的稳定性,请谨慎操作。
安装ds-controller
命令行模式
服务模式
支持如下运行模式:
- 普通运行
./ds-controller
- Docker模式运行
docker pull registry.cn-huhehaote.aliyuncs.com/pai-recommend2/ds-controller:0.0.1 #没有配置文件 docker run -it -d -p 8080:8080 registry-vpc.cn-huhehaote.aliyuncs.com/pai-recommend2/ds-controller:0.0.1 #挂载本地配置文件 docker run -it -d -p 8080:8080 -v /root/.ds/:/go/app/conf registry-vpc.cn-huhehaote.aliyuncs.com/pai-recommend2/ds-controller:0.0.1
服务模式使用示例如下:
- EMR管控
- 创建EMR的DataScience集群。
curl -v -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/createclusterv2" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "EmrVer": "EMR-3.31.0", "ZoneId": "cn-huhehaote-a", "ClusterName": "<dataScience-test>", "ClusterType": "DATA_SCIENCE", "RegionId": "cn-huhehaote", "Master_NodeCount": "1", "Master_InstanceType": "ecs.g5.4xlarge", "Master_DiskType": "CLOUD_EFFICIENCY", "Master_DiskCapacity": "80", "Master_DiskCount": "1", "Master_SysDiskType": "CLOUD_ESSD", "Master_SysDiskCapacity": "120", "Core_NodeCount": "2", "Core_InstanceType": "ecs.g5.4xlarge", "Core_DiskType": "CLOUD_EFFICIENCY", "Core_DiskCapacity": "120", "Core_DiskCount": "4", "Core_SysDiskType": "CLOUD_ESSD", "Core_SysDiskCapacity": "120", "Task_NodeCount": "1", "Task_InstanceType": "ecs.g5.4xlarge", "Task_DiskType": "CLOUD_EFFICIENCY", "Task_DiskCapacity": "120", "Task_DiskCount": "4", "Task_SysDiskType": "CLOUD_ESSD", "Task_SysDiskCapacity": "120", "VSwitchId": "vsw-hp3antge3fe8p3e2s****", "SecurityGroupId": "sg-hp3hk9cues1zuyzn****", "SecurityGroupName": "emr-test", "ChargeType": "PostPaid", "VpcId":"vpc-hp3wsdsed3enmxdvi****", "NetType": "vpc", "MasterPwd": "EMRtest1234!", "IsOpenPublicIp": true, "SshEnable": true, "IoOptimized": true, "HighAvailabilityEnable": false }'
- 获取集群的基本信息。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/describeclusterbasicinfo" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "ClusterId": "C-CB14657FAEE2****" }'
- 获取集群信息。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/listclusterhostgroup" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "ClusterId": "C-CB14657FAEE2****" }'
- 获取空闲状态的集群。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/listclusters" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "ClusterType": "DATA_SCIENCE", "Status": "IDLE" }'
- 获取EMR版本信息。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/listemrmainversion" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "PageNumber": 1, "PageSize": 10 }'
- 修改集群名称。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/modifyclustername" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "ClusterId": "C-CB14657FAEE2****", "Name": "<dataScience-test-new>" }'
- 释放集群。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/releasecluster" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "ClusterId": "C-CB14657FAEE2****" }'
- 扩容集群,例如扩容TASK节点。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/resizeclusterv2" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "ClusterId": "C-CB14657FAEE2****", "HostGroupType": "TASK", "NodeCount": "1", "InstanceType": "ecs.g5.4xlarge", "DiskType": "CLOUD_EFFICIENCY", "DiskCapacity": "80", "DiskCount": "1", "SysDiskType": "CLOUD_ESSD", "SysDiskCapacity": "120", "VSwitchId": "vsw-hp3antge3fe8p3e2s****", "ChargeType": "PostPaid" }'
- 按照机器组ID释放节点,例如TASK组。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/releaseclusterhostgroup" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "ClusterId": "C-CB14657FAEE2****", "HostGroupId": "G-08DBEB063DD4****" }'
- 按照机器组类型释放节点,例如TASK组。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/emr/releaseclusterhostgroupbytype" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "ClusterId": "C-CB14657FAEE2****", "HostGroupType": "TASK" }'
- 创建EMR的DataScience集群。
- ECS管控
- 申请公网IP。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/allocatepublicipaddress" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "InstanceId": "i-hp3h472yhby69kwh****" }'
- 增加安全组规则,例如添加22端口。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/authorizesecuritygroup" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "SecurityGroupId": "sg-hp3hk9cues1zuyzn****", "IpProtocol": "TCP", "PortRange": "22/22", "SourceCidrIp": "0.0.0.0/0" }'
- 创建ECS实例。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/createinstance" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "InstanceType": "ecs.g6.large", "ImageId": "ubuntu_18_04_64_20G_alibase_20190624.vhd", "SecurityGroupId": "sg-hp3hk9cues1zuyzn****", "InstanceName": "<emr_ds_insname>", "InternetChargeType": "PayByTraffic", "Password": "EMRtest1234!", "ZoneId": "cn-huhehaote-a", "SystemDiskSize": 80, "SystemDiskCategory": "cloud_efficiency", "VSwitchId": "vsw-hp3antge3fe8p3e2s****", "InstanceChargeType": "PostPaid", "InternetMaxBandwidthIn": 1, "InternetMaxBandwidthOut": 1 }'
本文示例中的<emr_ds_insname>为您ECS实例的名称。
- 创建安全组。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/createsecuritygroup" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "SecurityGroupName": "<emr_ds_groupname>", "SecurityGroupType": "normal", "VpcId": "vpc-hp3wsdsed3enmxdvi****" }'
本文示例中的<emr_ds_groupname>为您安全组的名称。
- 删除ECS实例。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/deleteinstance" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "InstanceId": "i-hp3h472yhby69kwh****", "Force": true }'
- 查询您在当前地域下的资源列表。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/describeavailableresource" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "DestinationResource": "InstanceType" }'
- 查看镜像资源。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/describeimages" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "PageNumber": 1, "PageSize": 100 }'
- 查询ECS实例的信息。
- 查询一个ECS实例的信息。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/describeecsinstances" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "InstanceId": "[\"i-hp3h472yhby69kwh****\"]" }'
- 查询多个ECS实例的信息。多个实例间使用英文逗号(,)隔开。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/describeecsinstances" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "InstanceId": "[\"i-hp3h472yhby69kwh****\"],...,[\"i-hp2h472yhby69kwh****\"]" }'
- 查询一个ECS实例的信息。
- 查询可用地域。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/describeregions" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote" }'
- 查询您在当前地域下能创建的ECS资源配额,包括能创建的安全组数量、弹性网卡数量、按量付费vCPU核数、抢占式实例vCPU核数、按量付费云盘总容量配额、专用宿主机数量、地域网络类型以及账号是否已完成实名认证。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/describeaccountattributes" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote" }'
- 查询阿里云提供的可用区列表并返回少量库存信息。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/describeregions" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote" }'
- 修改ECS实例的带宽配置。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/modifyinstancenetworkspec" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "InternetMaxBandwidthOut": 6, "InternetMaxBandwidthIn": 6, "NetworkChargeType": "PayByTraffic", "InstanceId": "i-hp3h472yhby69kwh****" }'
- 重启ECS实例。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/rebootinstance" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "InstanceId": "i-hp3h472yhby69kwh****" }'
- 启动ECS实例。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/startinstance" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "InstanceId": "i-hp3h472yhby69kwh****" }'
- 停止ECS实例。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/stopinstance" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "InstanceId": "i-hp3h472yhby69kwh****" }'
- 当您在一个ECS实例中运行如下命令时,可以获取本实例的meta信息,例如主机名、网络类型、mac地址、region、zone、镜像ID等信息, 常用于识别当前环境。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/ecs/metadata" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "Path": "/mac" }'
- 申请公网IP。
- VPC管控
- 创建一个VPC。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/vpc/createvpc" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "CidrBlock": "192.168.0.0/16", "VpcName": "vpc-hp3wsdsed3enmxdvi****", "EnableIpv6": false }'
- 创建一个虚拟交换机。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/vpc/createvsw" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "ZoneId": "cn-huhehaote-a", "CidrBlock": "192.168.0.0/18", "VSwitchName": "<emr_ds_vswname>", "VpcId": "vpc-hp3wsdsed3enmxdvi****" }'
本文示例中的<emr_ds_vswname>为您交换机的名称。
- 删除一个VPC。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/vpc/deletevpc" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "VpcId": "vpc-hp3wsdsed3enmxdvi****" }'
- 删除一个虚拟交换机。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/vpc/deletevsw" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote", "VSwitchId": "vsw-hp3antge3fe8p3e2s****" }'
- 查看VPC信息。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/vpc/describevpcs" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote" }'
- 查询已创建的交换机信息。
curl -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/vpc/describevsws" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "RegionId": "cn-huhehaote" }'
- 创建一个VPC。
- SMS短信服务
使用SendSms发送短信。
curl -v -H"Content-Type:application/json" "http://127.0.0.1:8080/v1/sms/sendsms" -d '{ "AKId": "<yourAccessKeyId>", "AKSecret": "<yourAccessKeySecret>", "PhoneNumbers": "186212xxxxx,186213xxxxx", "SignName":"Datascience", "TemplateCode":"SMS_22062xxxx", "TemplateParam":"clusterid:C-CB14657FAEE2****,regionid:cn-huhehaote" }'