Pulumi Blog: Virtual machines

Top 5 Things an Azure Developer Needs to Know: VMs

Sophia Parafina — Wed, 04 Aug 2021 00:00:00 +0000

So you want to be an Azure dev and all-around infrastructure wizard? Let’s start with the basics, virtual machines! In the previous article, the common use case for virtual machines is migrating applications from dedicated hardware. You want full control of the machine to install required software for the application or configure storage and networking.

Azure provides many ways to create and configure virtual machines ranging from the Azure CLI to the Azure Portal. In this article, we’ll first create a virtual machine using the portal to understand the requirements and process; then, we’ll do it with code using Pulumi’s Azure Native provider, which is built directly from the Azure Resource Manager API.

Creating a VM with the Azure Portal

When creating virtual machines, you should consider the operating system, storage requirements, and VM size with regard to CPU to memory ratio. Azure offers many sizes for virtual machines depending on your workload. If you’re migrating a traditional n-tier application to the cloud, it’s best to start with virtual machines that match your current configuration.

This article uses a simple example to deploy a static web application on a virtual machine to illustrate the process. To get started, sign in to the Azure Portal.

Step 1:

Under Azure Services, select Virtual machines

Select Create > Virtual Machine

Step 2: Create a Resource Group

A Resource Group is a logical collection of virtual machines, storage, virtual networks, web apps, and databases. Users will commonly group an application’s related resources. We’ll create myResourceGroup for our application.

Step 3: Configure the VM

Next, we’ll provide details for the VM by giving it a name, myVM, setting the Region and Availability options to No infrastructure redundancy required. We’ll use an Ubuntu image and set the VM size Standard_DS1_v2.

Step 4: Configure the Administrator Account

Virtual machines are Infrastructure as a Service, which means you will have to perform provisioning, updates, and maintenance. You’ll need an administrator account to perform these tasks. To keep this example simple, we’ll use Password as the Authentication type, but in practice, you should use Public Key Infrastructure (PKI) because it is more secure than a password that can be brute-forced.

Step 5: Configure inbound ports

You have to log into your virtual machine as well as allowing inbound requests to the static website. Select both HTTP (80) and SSH (22) then click on Review + create. Note that you can also add storage by clicking on Disks, but we’ll use the defaults for this example.

If your configuration is valid, you’ll see the following screen. Click Create to instantiate the vm.

Step 6: Enjoy your success

You can go and check out your VM by clicking on Go to resource.

Well, that seems simple and easy, but what if you had to deploy several virtual machines for the application that includes a web server, a worker to process requests, a message queuing system, and a database? What if your current system has multiple web servers and backend workers behind a load balancer? You’ll have to create, deploy, and maintain each VM because this is an IaaS.

There’s a better way using infrastructure as code and cloud engineering methodologies to create an automated process for creating, provisioning, and maintaining virtual machines.

Cloud Engineering to the Rescue!

Creating and provisioning one VM isn’t difficult, but migrating a traditional n-tier application often requires multiple virtual machines including those needed for horizontal scaling. This is where infrastructure as code excels. By using cloud engineering, we can automate build, deploy, and manage process for any infrastructure or application.

Let’s look at an example of deploying a web server using a virtual machine. The complete example is available in either Python or Typescript. We will examine the Python code.

We’re using the Azure Native provider, which implements the best possible support for the Azure platform in Pulumi. It exposes the entire API surface of Azure to developers and operators. The native provider supports over 1,000 resource types, with more added as they are released by Azure.

The script starts with importing the Pulumi Azure Native SDK modules and retrieving configuration parameters such as the username and password required to log into the server for provisioning. It also creates a resource group.

import base64
from pulumi import Config, Output, export
import pulumi_azure_native.compute as compute
import pulumi_azure_native.network as network
import pulumi_azure_native.resources as resources

config = Config()
username = config.require("username")
password = config.require("password")

resource_group = resources.ResourceGroup("server")

The next part creates the networking resources, including a dynamically allocated public IP address for the server.

net = network.VirtualNetwork(
 "server-network",
 resource_group_name=resource_group.name,
 address_space=network.AddressSpaceArgs(
 address_prefixes=["10.0.0.0/16"],
 ),
 subnets=[network.SubnetArgs(
 name="default",
 address_prefix="10.0.1.0/24",
 )])

public_ip = network.PublicIPAddress(
 "server-ip",
 resource_group_name=resource_group.name,
 public_ip_allocation_method=network.IPAllocationMethod.DYNAMIC)

network_iface = network.NetworkInterface(
 "server-nic",
 resource_group_name=resource_group.name,
 ip_configurations=[network.NetworkInterfaceIPConfigurationArgs(
 name="webserveripcfg",
 subnet=network.SubnetArgs(id=net.subnets[0].id),
 private_ip_allocation_method=network.IPAllocationMethod.DYNAMIC,
 public_ip_address=network.PublicIPAddressArgs(id=public_ip.id),
 )])

And here’s our web server and index page using Python’s simpleHTTPServer.

init_script = """#!/bin/bash
echo "Hello, World!" > index.html
nohup python -m SimpleHTTPServer 80 &"""

Finally, here’s how we create the virtual machine. We add the VM to the resource group and associate the network_profile to network_iface that we defined previously. We set the vm_size, set up the os_profile with the hostname and username and password. In the example, we create a storage profile and use Ubuntu for the image profile. The script ends by exporting the public IP address of the webserver.

vm = compute.VirtualMachine(
 "server-vm",
 resource_group_name=resource_group.name,
 network_profile=compute.NetworkProfileArgs(
 network_interfaces=[
 compute.NetworkInterfaceReferenceArgs(id=network_iface.id),
 ],
 ),
 hardware_profile=compute.HardwareProfileArgs(
 vm_size=compute.VirtualMachineSizeTypes.STANDARD_A0,
 ),
 os_profile=compute.OSProfileArgs(
 computer_name="hostname",
 admin_username=username,
 admin_password=password,
 custom_data=base64.b64encode(init_script.encode("ascii")).decode("ascii"),
 linux_configuration=compute.LinuxConfigurationArgs(
 disable_password_authentication=False,
 ),
 ),
 storage_profile=compute.StorageProfileArgs(
 os_disk=compute.OSDiskArgs(
 create_option=compute.DiskCreateOptionTypes.FROM_IMAGE,
 name="myosdisk1",
 ),
 image_reference=compute.ImageReferenceArgs(
 publisher="canonical",
 offer="UbuntuServer",
 sku="16.04-LTS",
 version="latest",
 ),
 ))

combined_output = Output.all(vm.id, public_ip.name, resource_group.name)
public_ip_addr = combined_output.apply(
 lambda lst: network.get_public_ip_address(
 public_ip_address_name=lst[1],
 resource_group_name=lst[2]))
export("public_ip", public_ip_addr.ip_address)

On the face of it, this seems more complex than using the Azure Portal. The critical thing to remember is that this process is reproducible, so if you require additional virtual machines, you can add a loop to create the necessary number of VMs.

But what about provisioning? Because this is code, you can write a provisioning script using Pulumi’s dynamic provider that connects to the webserver, copies files, and executes commands. Here’s a code snippet of a dynamic provider that copies files to your VM using scp.

# CopyFileProvider implements the resource lifecycle for the CopyFile resource type below.
class CopyFileProvider(ProvisionerProvider):
 def on_create(self, inputs: Any) -> Any:
 ssh = connect(inputs['conn'])
 scp = ssh.open_sftp()
 try:
 scp.put(inputs['src'], inputs['dest'])
 finally:
 scp.close()
 ssh.close()
 return inputs

# CopyFile is a provisioner step that can copy a file over an SSH connection.
class CopyFile(dynamic.Resource):
 def __init__(self, name: str, conn: pulumi.Input[ConnectionArgs],
 src: str, dest: str, opts: Optional[pulumi.ResourceOptions] = None):
 self.conn = conn
 """conn contains information on how to connect to the destination, in addition to dependency information."""
 self.src = src
 """
 src is the source of the file or directory to copy. It can be specified as relative to the current
 working directory or as an absolute path. This cannot be specified if content is set.
 """
 self.dest = dest
 """dest is required and specifies the absolute path on the target where the file will be copied to."""

 super().__init__(
 CopyFileProvider(),
 name,
 {
 'dep': conn,
 'conn': conn,
 'src': src,
 'dest': dest,
 'fileHash': sha256sum(src),
 },
 opts,
 )

Summary

Although it’s relatively simple to create virtual machines using the Azure Portal or the Azure CLI, provisioning and maintaining them manually can be complex and error-prone. If we use code to build, deploy, and manage VMs and applications, we can automate the process making updates and maintenance repeatable processes and reduce errors. We can tear down existing VMs and replace them with updated and provisioned versions using infrastructure as code. This is a lot of information if you’re starting your Azure cloud journey, so try something fun and deploy Minecraft on Azure!

Top 5 Things an Azure Developer Needs to Know: Introduction

Sophia Parafina — Mon, 02 Aug 2021 00:00:00 +0000

The Azure cloud platform includes over 200 products and cloud services. Wherever you are in your Microsoft cloud engineering journey, you should be familiar with these top 5 cloud tasks that are essential building blocks commonly used to deploy applications and infrastructure to the Azure cloud.

In this series of articles, we’ll go in-depth on virtual machines, Azure Functions, static websites, building an Azure Kubernetes Service cluster and deploying applications on AKS, and DevOps with Azure App Service.

Azure - a comprehensive cloud platform

With Azure, you can develop, test, deploy and manage your applications in an integrated cloud computing environment. With the Azure Portal, you can manage the cloud infrastructure for your applications. Azure gives you the choice of a portal for managing services or managing them programmatically using service APIs and templates.

With over 200 services, knowing which one to use can be confusing. Deciding where to host your application is the first step. This series of articles covers the most commonly used Azure services such as virtual machines, serverless functions, static websites, Kubernetes, and Azure App Service. In addition, we’ll cover how to deploy applications using Azure DevOps.

In this article, we’ll match use cases to cloud services to help you navigate Azure.

Azure Virtual Machines

A common cloud migration path is to use virtual machines if you have existing infrastructure or on-premise applications. It is especially true if you want complete control over the infrastructure and application environment.

Azure provides Infrastructure as a Service (IaaS) for deploying your applications on either Linux or Windows VMs. IaaS includes all necessary support, such as Azure Virtual Network and storage. As with any IaaS solution, you are responsible for the installation of software, configuration, and maintenance of your application and any maintenance the VM might require, such as operating system patches.

If your application doesn’t fit into a Platform as a Service (PaaS) model, consider using virtual machines to provide the fine-grained control required for your deployment.

Serverless

At the opposite end of cloud infrastructure is serverless computing. Serverless eliminates the need to manage infrastructure and lets you focus on writing and deploying code. Azure automatically provisions, scales, and manages the infrastructure. You can write code that runs in response to events or on a schedule.

Azure Functions lets you focus on the application’s business logic, and the code executes when called by a webhook, an HTTP or REST request, a schedule, or an event. Serverless is particularly appealing because you can use the development language of your choice, ranging from PHP, Python, Node.js, or .NET languages. Azure functions scale as needed, and consumption-based pricing means that you’re charged only when the code runs.

If you don’t need a completely hosted project and only pay for what you use, choose Azure Functions triggered by web-based events, a schedule, or other Azure services.

Static websites

Static websites are quickly becoming the defacto way to publish content. Built from libraries and frameworks that don’t use server side rendering like React, Angular, or Vue, they include HTML, CSS, image assets, and Javascript in the application. Instead of serving assets from a server or servers scaled horizontally, the static assets are distributed through a Content Delivery Network (CDN). This reduces latency, resulting in faster websites and increased customer satisfaction.

Azure provides two ways to deploy static websites. Azure lets you deploy a static web application with Azure App Service and the recently released Azure Static Web Apps. Azure App Service deploys websites by uploading the website to Azure storage configured to serve web content. Azure App Service automatically builds and deploys full stack web apps to Azure from a code repository, which follows the Jamstack model for developing and deploying websites.

We’ll look at both methods and discuss the pros and cons of both methods.

Kubernetes

Kubernetes is the leading container orchestration service. Azure Kubernetes Service (AKS) is a hosted service that can deploy, scale, and manage containerized applications in a cluster. AKS fulfills many different use cases, for example:

If you have an existing application, think n-tier applications, you can containerize the components and “lift and shift” your application to run in Kubernetes.
You can deploy microservices on AKS to add horizontal scaling, load balancing, secrets management, and self-healing.
Data scientists are training machine learning models on AKS using TensorFlow and Kubeflow.
You can even use AKS as part of a build pipeline by pairing with a CI, such as Jenkins, to build containers for your application.

AKS supports many use cases by providing provisioning, scaling, and resource upgrades without requiring expertise in container orchestration. With AKS, the Kubernetes master nodes are managed by Azure, and you are only charged for the worker nodes in your cluster.

We’ll do two separate deep dives into deploying AKS clusters and deploying applications in AKS in this series.

Deploy AKS

Azure provides many ways to deploy an AKS cluster, including:

When you deploy an AKS Cluster, the worker nodes and Kubernetes master are configured with a basic configuration. You can additionally configure networking, storage, monitoring, and policies. We’ll cover configuring AKS in this series.

Deploy Applications

Deploying applications on AKS is equally flexible, and you can deploy using:

kubectl and YAML manifests
Helm, an open-source Kubernetes packaging tool
Azure Pipelines Infrastructure as Code

We’ll examine these options and do a deep dive on application deployment in a follow-up article.

DevOps integration

Before Continuous integration (CI) and Continuous Deployment (CD) became prevalent, applications were manually built, compiled, and maintained in a large codebase. CI/CD automation lets you build quickly, test, and deploy services. Both application fixes and new features are shipped quickly to your customers, and you can rapidly respond to changing customer demands. For example, if demand increases, you can quickly add resources to your application or add a serverless function that increases and decreases resources as needed.

If you’re deploying applications on Azure Functions, virtual machines, AKS, or provisioning infrastructure, the build process should be automated. Azure offers a range of services such as Azure Container Registry, AKS, and Azure Cosmos DB that can integrate with popular CI/CD pipelines like Azure DevOps, Jenkins, Github, and TeamCity.

If you’re interested in using Azure DevOps to automate infrastructure and application delivery, stay tuned!

Let’s Get Started!

So here are the top 5 things essential to know as an Azure developer:

configuring and deploying virtual machines
developing and deploying Azure Functions
deploying static websites on Azure
configuring Azure Kubernetes Service and deploying containerized applications
using Azure DevOps to automate the building and deployment of infrastructure and applications

First up and next in this series of articles is deploying Azure virtual machines with an application.

Reduce Cloud Costs with EC2 ARM Instances

Sophia Parafina — Tue, 26 Jan 2021 00:00:00 +0000

Whether you’re migrating to the cloud or have existing infrastructure, cloud spend can be a significant barrier to your success. Too small of a budget could prevent your organization from meeting your performance metrics. You can use different strategies to reduce cloud spend, such as using Spot Instances, which cost less than On-Demand Instances or scaling your infrastructure based on peak usage times.

With the addition of Graviton2 based EC2 Instances, AWS offers an on-demand alternative for decreasing cloud spend. Both Amazon and independent testing demonstrated that the general-purpose M6g instance delivered up to a 40% gain of price/performance compared to Intel m5.large instances. In addition to the M6g general-purpose instance, AWS offers instances general-purpose burstable (T4g), compute-optimized (C6g), and memory-optimized (R6g) EC2 instances.

Deploying Infrastructure with ARM

Building infrastructure with ARM instances is straightforward. You can reuse code and substitute ARM instances and containers for Intel-based components. Check out this example that deploys a VPC with EC2 container instances. To get started, you can clone the repository with the Pulumi CLI.

$ pulumi new https://github.com/pulumi/examples/tree/master/aws-py-ecs-instances-autoapi/py-ecs-instance

This command will walk you through creating a new Pulumi project.

Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.

project name: (aws-py-ecs-instances-autoapi)
project description: (A container running in AWS ECS Container Instance, using Python infrastructure as code)
Created project 'aws-py-ecs-instances-autoapi'

Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
stack name: (dev)
Created stack 'dev'

Creating virtual environment...
Finished creating virtual environment
Updating pip, setuptools, and wheel in virtual environment...
…
Successfully installed arpeggio-1.10.1 attrs-20.3.0 dill-0.3.3 grpcio-1.35.0 parver-0.3.1 protobuf-3.14.0 pulumi-2.18.1 pulumi-aws-3.24.0 pyyaml-5.4.1 semver-2.13.0 six-1.15.0
Finished installing dependencies
Your new project is ready to go! ✨

To perform an initial deployment, run 'pulumi up'

Because this example is written in Python, the CLI will automatically configure your virtual environment and install the Python packages listed in requirements.txt. You can now edit main.py to use the M6g instance and an ARM version of the nginx container. Dockerhub has many ARM images for popular applications, as well as for base images. We’ll use the arm64v8/nginx image in this example.

Edit this parts of main.py to use an ARM AMI

# Find an "ECS optimized" AMI to use for the EC2 container instances.
ecs_instance_ami = aws.get_ami(
 most_recent="true",
 owners=["amazon"],
 filters=[
 {
 "name": "name",
 "values": ["amzn2-ami-ecs-hvm-*-arm64-*"]
 }
 ]
)

and this part to specify the instance type:

# create launch configuration
launch_config = aws.ec2.LaunchConfiguration(
 "launch-config",
 image_id=ecs_instance_ami.id,
 instance_type="m6g.medium",
 iam_instance_profile=ecs_instance_profile.name, # needed to give instance authority to join the ECS cluster.
 user_data=user_data, # Required so instance knows to connect to the cluster created below.
)

Finally, replace the image with the ARM image from Dockerhub:

# Task definition for creating our containers.
task_def = aws.ecs.TaskDefinition(
 "my-app",
 family="ec2-task-definition",
 cpu="256",
 memory="512",
 network_mode="awsvpc",
 requires_compatibilities=["EC2"],
 execution_role_arn=task_execution_role.arn, # Needed so it has permission to launch tasks on the cluster.
 container_definitions=json.dumps([{
 "name": "my-app",
 "image": "arm64v8/nginx", # a simple nginx example
 "portMappings": [{
 "containerPort": 80,
 "hostPort": 80,
 "protocol": "tcp"
 }]
 }]),
 opts=pulumi.ResourceOptions(depends_on=[cluster])
)

Save your changes, and we’re ready to go to the next step.

Before spinning up our example, we need to set the AWS region we want to use and the autoscalingGroupSize to limit the number of instances.

$ pulumi config set aws:region us-west-2
$ pulumi config set cfg:autoscalingGroupSize 1

Once those are configured, run pulumi up to bring up your infrastructure. You’ll see similar output, including the URL for your deployment.

View Live: https://app.pulumi.com/spara/aws-py-ecs-instances-autoapi/dev/updates/1

 Type Name Status
 + pulumi:pulumi:Stack aws-py-ecs-instances-autoapi-dev created
 + ├─ aws:iam:Role task-execution-role created
 + ├─ aws:ec2:SecurityGroup nginx-sg created
 + ├─ aws:iam:Role ecs-instance-role created
 + ├─ aws:lb:TargetGroup app-tg created
 + ├─ aws:iam:RolePolicyAttachment task-excution-policy-attach created
 + ├─ aws:iam:RolePolicyAttachment ecs-instance-policy-attach created
 + ├─ aws:iam:InstanceProfile ecs-iam-instance-profile created
 + ├─ aws:ec2:LaunchConfiguration launch-config created
 + ├─ aws:lb:LoadBalancer load-balancer created
 + ├─ aws:autoscaling:Group auto-scaling created
 + ├─ aws:ecs:CapacityProvider capacity-provider created
 + ├─ aws:ecs:Cluster cluster created
 + ├─ aws:ecs:TaskDefinition my-app created
 + ├─ aws:lb:Listener web created
 + └─ aws:ecs:Service my-task-runner created

Outputs:
 NOTE: "You may have to wait a minute for AWS to spin up the service. So if the URL throws a 503 error, try again after a bit."
 app_url: "http://load-balancer-7aedaa4-1557043631.us-west-2.elb.amazonaws.com"

Resources:
 + 16 created

Duration: 2m30s

What’s Next?

Before jumping to the next thing, let’s review what we’ve done. Using infrastructure as code enables changing your infrastructure as new technology becomes available. You can experiment rapidly to see if this configuration can meet your performance metrics and remain within your cloud spend budget. Blue/green deployment is straightforward because you are just swapping components on a known and proven infrastructure. These are just some of the advantages of using infrastructure as code.

So what is the next step? In a future article, well demonstrate how to build your own ARM-based container images using Docker’s buildx CLI plugin. Until then, you can learn more about Pulumi and infrastructure as code with our Getting Started tutorials.

Deploying Minecraft on Azure

Sophia Parafina — Wed, 04 Nov 2020 00:00:00 +0000

This article demonstrates how to deploy and provision a virtual machine in Azure using the Pulumi Azure-Native provider. While there are numerous examples of using the Azure console, the Azure CLI, or ARM templates to deploy and provision virtual machines, we’ll use Python to implement a repeatable deployment.

Build it

There are several parts to the process. First, we need to create the network resources to make the virtual machine accessible to Minecraft clients. Second, the virtual machine needs to be created and configured with network resources, storage for server files and logs, and a way to administer it remotely. Finally, the virtual machine needs to be provisioned with the server and all the dependencies that it requires, such as Java. Let’s go step by step through the process.

Resource Group

Resource groups hold all the resources that you use for building infrastructure on Azure. They can contain all the resources for your Azure infrastructure or only the resources that you want to manage together. Resource groups let you deploy, update, and delete resources as a group.

Declaring a resource group with Pulumi is straightforward and requires a name and the location of the region.

resource_group = resources.ResourceGroup("server-rg",
 resource_group_name="minecraft",
 location=location)

Networking resources

Next, we want to create all the resources needed to deploy a virtual machine as a server and make it accessible to clients on the Internet, which, in our case, is a Minecraft client.

We’ll create a virtual network for the server to communicate with other resources and to the Internet. The virtual network needs the resource group name, the location, and a name. We set a private IP address range and Azure will assign an IP address if a resource is deployed in that address space.

net = network.VirtualNetwork(
 "server-network",
 resource_group_name=resource_group.name,
 location=location,
 virtual_network_name="server-network",
 address_space=network.AddressSpaceArgs(
 address_prefixes=["10.0.0.0/16"],
 ),
 subnets=[network.SubnetArgs(
 name="default",
 address_prefix="10.0.0.0/24",
 )]
)

To make the virtual machine accessible on the Internet, we create a public IP. The public IP address is dedicated to a resource unless they are unassigned by you. They are resources in their own right and can be associated with other resources such as firewalls, load balancers, and virtual machines.

public_ip = network.PublicIPAddress(
 "server-ip",
 resource_group_name=resource_group.name,
 location=location,
 public_ip_address_name="server-ip",
 public_ip_allocation_method="Dynamic"
)

Finally, we need a resource to tie the virtual network to the virtual machine. A network interface lets a virtual machine communicate with other resources in the resource group and the Internet. In the code below we specify the subnet we created in the virtual network and set the public IP to the resource we previously created.

network_iface = network.NetworkInterface(
 "server-nic",
 resource_group_name=resource_group.name,
 location=resource_group.location,
 network_interface_name="server-nic",
 ip_configurations=[network.NetworkInterfaceIPConfigurationArgs(
 name="webserveripcfg",
 subnet=network.SubnetArgs(id=net.subnets[0].id),
 private_ip_allocation_method="Dynamic",
 public_ip_address=network.PublicIPAddressArgs(id=public_ip.id),
 )]
)

Now that we have the networking resources available let’s create the virtual machine.

Virtual machine

There are several parts to creating a virtual machine, so we’ll go through the code piece by piece. In the previous step, we created a network interface resource. We’ll use that to set up the network profile for the virtual machine.

network_profile=compute.NetworkProfileArgs(
 network_interfaces=[
 compute.NetworkInterfaceReferenceArgs(id=network_iface.id),
 ],
 ),

Next, we set the hardware profile that sets the virtual machine size, determining the number of cores, amount of memory, and storage size.

 hardware_profile=compute.HardwareProfileArgs(
 vm_size="Standard_A2",
 ),

We need to set up access to the virtual machine. The code below sets the user name and optionally the password if needed. We disable password authentication and opt to use OpenSSH keypairs to authenticate logins to the virtual machine.

os_profile=compute.OSProfileArgs(
 computer_name="hostname",
 admin_username=admin_username,
 #admin_password=admin_password,
 linux_configuration=compute.LinuxConfigurationArgs(
 disable_password_authentication=True,
 ssh= {
 'publicKeys': [{
 'keyData': public_key,
 'path': ssh_path,
 }],
 },
 ),
 ),

To complete the virtual machine, we add storage via an OS disk. Note that OS disks are ephemeral and they will not retain data if the virtual machine is terminated.

storage_profile=compute.StorageProfileArgs(
 os_disk=compute.OSDiskArgs(
 create_option="FromImage",
 name="myosdisk1",
 caching= "ReadWrite",
 disk_size_gb= 100,
 ),
 image_reference=compute.ImageReferenceArgs(
 publisher="canonical",
 offer="UbuntuServer",
 sku="18.04-LTS",
 version="latest",
 ),
 ),

Provisioning the server

Once the virtual machine has been created, we need to provision it with the Minecraft server. The install.sh script installs the prerequisite software, the Minecraft server, and creates the service for starting and stopping the server. We could use scp to copy the install script to the virtual machine from our local machine and then ssh into a terminal to run the script. The alternative is to use a Pulumi dynamic resource provider to provision the virtual machine. The provisioners.py script creates a dynamic provider that enable using scp to transfer files and send commands via ssh. Creating dynamic providers are beyond the scope of this article, but more information is available on the Pulumi docs.

We provision our Minecraft server by creating the conn connection object defined in provisioners.py.

conn = provisioners.ConnectionArgs(
 host= public_ip_addr.ip_address,
 username=admin_username,
 private_key=private_key,
 private_key_passphrase=private_key_passphrase,
)

We pass in the conn object to the CopyFile method which copies install.sh to the virtual machine via scp.

cp_config = provisioners.CopyFile('config',
 conn=conn,
 src='install.sh',
 dest='install.sh',
 opts=pulumi.ResourceOptions(depends_on=[server]),
)

To provision the virtual machine, we call the RemoteExec method to make install.sh executable and run the script as root. The script installs all the prerequisites including Java, downloads the Minecraft server jar and configures it, and creates a service to run the Minecraft server.

install = provisioners.RemoteExec('install',
 conn=conn,
 commands=['sudo chmod 755 install.sh && sudo ./install.sh'],
 opts=pulumi.ResourceOptions(depends_on=[cp_config]),
)

Once the virtual machine is created and provisioned, we publish the virtual machines public IP address, and we’re ready to join the server!

export("Minecraft Server IP Address", public_ip_addr.ip_address)

Next Steps

This article demonstrates the process to create a virtual machine and provision it with a Minecraft server. However, there’s always room for improvement, such as adding Azure Disk Storage to provide durable storage, add a configuration file to customize the server, and automatically updating the server jar file when a new version is released. All these options are available to you when you build your infrastructure with code.

You can download the code from the Pulumi examples repository on Github.

See these resources to learn more about deploying infrastructure on Azure.

Architecture as Code: Virtual Machines

Sophia Parafina — Tue, 31 Mar 2020 00:00:00 +0000

In a previous article, we presented an overview of four infrastructure patterns for deploying modern applications. The article reviewed virtual machines, serverless, Kubernetes, and microservices. In this post, we’ll examine virtual machines in-depth.

The Minimum Virtual Machine

Whether you choose to build a virtual machine using VMWare or from a cloud service provider such as AWS, Azure, or Google Cloud Platform, they require three resources. First, the instance itself must be configured to meet application requirements. Second, network access for maintaining the instance and for applications. Finally, storage for application data.

Let’s start with the EC2 provisioner example, which is available on Github. Make an empty directory and clone the example in the directory using pulumi new.

$ mkdir virtual_machine && cd virtual_machine
$ pulumi new https://github.com/pulumi/examples/tree/master/aws-ts-ec2-provisioners

Follow the instructions in the README to create an OpenSSH keypair and set them as secrets, we’ll need them to log into and provision the VM. Let’s take a look at the code. The first part configures the VM with the OpenSSH keys that we stored as secrets in the Pulumi configuration file (Pulumi.dev.yaml).

The following part defines a security group that allows us to ssh via port 22 and receive requests on port 80.

// Create a new security group that permits SSH and web access.
const secgrp = new aws.ec2.SecurityGroup("secgrp", {
 description: "Foo",
 ingress: [
 { protocol: "tcp", fromPort: 22, toPort: 22, cidrBlocks: ["0.0.0.0/0"] },
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
 ],
});

Note that the security group only allows ingress, if we want to update the VM or add software via a package manager such as yum, we need to define egress.

// Create a new security group that permits SSH and http/https access.
const secgrp = new aws.ec2.SecurityGroup("secgrp", {
 description: "Foo",
 ingress: [
 { protocol: "tcp", fromPort: 22, toPort: 22, cidrBlocks: ["0.0.0.0/0"] },
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
 { protocol: "tcp", fromPort: 443, toPort: 443, cidrBlocks: ["0.0.0.0/0"] },
 ],
 egress: [
 { protocol: "tcp", fromPort: 80, toPort: 80, cidrBlocks: ["0.0.0.0/0"] },
 { protocol: "tcp", fromPort: 443, toPort: 443, cidrBlocks: ["0.0.0.0/0"] },
 ],
});

The code block below selects the AWS AMI and creates an EC2 instance with the OpenSSH keypair and security group with ingress and egress rules configured.

// Get the AMI.
const amiId = aws.getAmi({
 owners: ["amazon"],
 mostRecent: true,
 filters: [{
 name: "name",
 values: ["amzn2-ami-hvm-2.0.????????-x86_64-gp2"],
 }],
}, { async: true }).then(ami => ami.id);

// Create an EC2 server that we'll then provision stuff onto.
const size = "t2.micro";
if (!keyName) {
 const key = new aws.ec2.KeyPair("key", { publicKey });
 keyName = key.keyName;
}

const server = new aws.ec2.Instance("server", {
 instanceType: size,
 ami: amiId,
 keyName: keyName,
 vpcSecurityGroupIds: [ secgrp.id ],

});

Next, let’s add storage to our virtual machine using AWS Elastic Block Storage and attach it to our EC2 instance. Add the following code after the virtual machine is created. The code creates a 10 GB volume and attaches it to our virtual machine via the instanceId property.

// Add EBS storage.
const ebsvolume = new aws.ebs.Volume("vm", {
 availabilityZone: "us-west-2a",
 size: 10,
 tags: {
 Name: "vm_volume",
 },
});

// Attach storage to the virtual machine.
const ebsAtt = new aws.ec2.VolumeAttachment("ebsAtt", {
 deviceName: "/dev/sdh",
 instanceId: server.id,
 volumeId: ebsvolume.id,
});

Now we have a virtual machine that can receive HTTP and HTTPS requests and allows outbound network traffic so that we can update and install software via the package manager.

Provisioning with Dynamic Providers

Provisioning a VM when it’s created by running user-supplied scripts is relatively straightforward. For example, AWS EC2 instances have a userData parameter that allows you to specify an inline script that runs when the instance starts. But what if you need to copy and execute scripts on the virtual machine without replacing the instance?

This where you can use Pulumi’s dynamic provider to provision existing instances. Dynamic providers enable creating custom resource types, such as provisioners, within the source code of your Pulumi program. They let you execute arbitrary code during the deployment process. Dynamic provisioners run during resource provisioning, and enable adding custom logic to a deployment workflow during the create, read, update, or delete steps of a Pulumi program.

Let’s look at the code which creates a dynamic provisioner. In the code below, if Pulumi doesn’t find the resource, it creates the resource. However, if you run pulumi up again and the resource is already running, Pulumi uses the diff method to replace the resource.

// Provisioner lets a custom action run the first time a resource has been created. It takes as input
// a dependent property. Anytime its value changes, the resource is replaced and will re-run its logic.
export class Provisioner<T, U> extends pulumi.dynamic.Resource {
 dep: pulumi.Output<T>;
 result: pulumi.Output<U>;
 constructor(name: string, props: ProvisionerProperties<T, U>, opts?: pulumi.CustomResourceOptions) {
 const provider: pulumi.dynamic.ResourceProvider = {
 diff: async (id: pulumi.ID, olds: State<T, U>, news: State<T, U>) => {
 const replace = JSON.stringify(olds.dep) !== JSON.stringify(news.dep);
 return {
 changes: replace,
 replaces: replace ? ["dep"] : undefined,
 deleteBeforeReplace: true,
 };
 },
 create: async (inputs: State<T, U>) => {
 const result = await props.onCreate(inputs.dep);
 if (result !== undefined) {
 inputs.result = result;
 }
 return { id: uuid.v4(), outs: inputs };
 },
 };
 super(provider, name, { dep: props.dep, result: null }, opts);
 }
}

export interface ProvisionerProperties<T, U> {
 dep: pulumi.Input<T>;
 onCreate: (dep: pulumi.Unwrap<T>) => Promise<pulumi.Unwrap<U>>;
}

interface State<T, U> {
 dep: pulumi.Unwrap<T>;
 result: pulumi.Unwrap<U>;
}

This example creates two interfaces using the provisioner. The first is a copyFile function that lets us copy the configuration files to the virtual machine. The second interface is the runCommand function, which runs commands on the virtual machine. Underlying both functions is the ‘connToSsh2’ function that opens the ssh connection to copy files via scp and to execute commands. When you run pulumi up the example writes a small configuration file on the virtual machine then runs ‘cat’ to print the configuration file to stdout.

Installing PostgreSQL

Let’s modify the example to install and configure PostgreSQL. First, create a directory to hold the installation script and configuration files.

$ mkdir ./postgres

Add the following files to the postgres directory. The postgres.conf file is a bare-bones configuration file the allows PostgreSQL to allow connections from any IP address on port 5432. The pg_hba.conf file permits authentication for connections with a password.

# postgres.conf

# -----------------------------
# PostgreSQL configuration file
# -----------------------------
#

#------------------------------------------------------------------------------
# CONNECTIONS AND AUTHENTICATION
#------------------------------------------------------------------------------

# - Connection Settings -

listen_addresses = '*' # what IP address(es) to listen on;
 # comma-separated list of addresses;
 # defaults to 'localhost'; use '*' for all
 # (change requires restart)
port = 5432 # (change requires restart)

# pg_hba.conf

# TYPE DATABASE USER ADDRESS METHOD

# "local" is for Unix domain socket connections only
local all all trust
# IPv4 local connections:
host all all 127.0.0.1/32 trust
# IPv6 local connections:
host all all ::1/128 trust

Finally, the postgres_install.sh is a bash script that installs Postgres with the yum package manager. It initializes the database, replaces the configuration scripts, and starts the database.

#!/bin/bash

# This script installs and configures PostgreSQL
sudo yum -y install postgresql postgresql-server postgresql-devel postgresql-contrib postgresql-docs
sudo postgresql-setup initdb

# Copy configuration files
sudo cp -f ./pg_hba.conf /var/lib/pgsql/data/pg_hba.conf
sudo cp -f ./postgresql.conf /var/lib/pgsql/data/postgresql.conf

# Start postgres
sudo service postgresql start

To provision the EC2 instance we’ll use the provisioner interface to copy the ./postgres directory to the instance. Next, we’ll make postgres_install.sh executable then run the script to install postgres and start it up.

// Copy a config file to our server.
const cpConf = new provisioners.CopyFile("postgres_conf", {
 conn,
 src: "./postgres/",
 dest: "./",
}, { dependsOn: server });

// Execute a basic command on our server.
const chmodInstall = new provisioners.RemoteExec("chmod-install", {
 conn,
 command: "chmod 755 postgres_install.sh ",
}, { dependsOn: cpConf });

// Execute a basic command on our server.
const installPgsql = new provisioners.RemoteExec("pg_install", {
 conn,
 command: "./postgres_install.sh ",
}, { dependsOn: chmodInstall });

export const publicIp = server.publicIp;
export const publicHostName = server.publicDns;

Note that the dependsOn parameter determines the sequence of deployment steps. This ensures that each part is available to the provisioner at that step. To deploy PostgreSQL on your virtual machine, run pulumi up. You can check the install by logging into the virtual machine and use psql to log into the database and list the tables.

$ ssh -i rsa ec2-user@$(pulumi stack output publicHostName)

# In the virtual machine
$ sudo su - postgres
-bash-4.2$ psql -U postgres
psql (9.2.24)
Type "help" for help.

postgres=# \dS
 pg_catalog | pg_aggregate | table | postgres
 pg_catalog | pg_am | table | postgres
 pg_catalog | pg_amop | table | postgres
 pg_catalog | pg_amproc | table | postgres
 pg_catalog | pg_attrdef | table | postgres
 pg_catalog | pg_attribute | table | postgres
 pg_catalog | pg_auth_members | table | postgres
 pg_catalog | pg_authid | table | postgres
...

postgres=# \q
-bash-4.2$ logout

Conclusion

We can deploy and configure virtual machines programmatically using scripts and a CLI. However, updating existing instances can be onerous because it would require logging into each instance to update them. With Pulumi, we can use a dynamic provider to create a provisioner that can provision and configure an existing virtual machine. In this post, we walked through how to create a virtual machine with storage, starting from an example on Github. We then installed PostgreSQL using a provisioner that copied configuration files, and an installation script then ran the script, all without creating a new virtual machine.

We have more virtual machine examples for other cloud providers. Check out:

In our next installment, we’ll go into detail about deploying applications on serverless infrastructure.