Posts Tagged data-and-analytics

When you manage dozens of data-loading pipelines, copying and pasting IaC configurations between them is a recipe for mishap. IAM policies can drift, naming conventions diverge, and every new source is a new opportunity to make a mistake — not to mention compound the problem of duplication. In this post, we’ll show you how you can identify and encapsulate common patterns into composable components and walk through the production lessons we’ve learned running 25+ pipelines for over three years.

Read more →

Before Platybot, our #analytics Slack channel was a support queue. Every day, people from every team would ask questions: “Which customers use feature X?”, “What’s our ARR by plan type?”, “Do we have a report for template usage?” Our two-person data team was a bottleneck.

Read more →

Pulumi Cloud Resource Search AI assist functionality is now generally available to all organizations! In addition we have shipped some improvements to the feature to make it easier to use and more discoverable: a toggle on the search bar, suggested queries and an “I’m Feeling Lucky” button to generate a random query for you.

Read more →

We’ve seen incredible acceleration of cloud adoption over the past 5 years. Pulumi’s flagship open source IaC solution gives engineers great tools to scale up their cloud infrastructure using the same programming languages and tools they already know and love. As a result, thousands of companies of every size and scale have adopted Pulumi as a lynchpin of their cloud infrastructure strategy.

Today we’re excited to announce Pulumi Insights, the next major productivity enhancement for infrastructure as code. Pulumi Insights provides intelligence, search, and analytics over any infrastructure, in any cloud across your organization, leveraging the latest advances in generative AI and Large Language Models (LLMs). Whether you have an AWS VPC, a Kubernetes CRD, or a DataDog alarm definition, Pulumi Insights enables you to intelligently find and interact with all of your resources from within the Pulumi Cloud.

Read more →

Data science has advanced because tools like Jupyter Notebook hide complexity by running high level code for the specific problem they are trying to solve. Increasing the level of abstraction lets a data scientist be more productive by reducing the effort to try multiple approaches to near zero, which encourages experimentation and better results.

Data scientists typically work locally, but they often store data for analyses and models in the cloud. There are clear advantages to using cloud resources for these tasks:

• Data scientists generally don’t want to manage their storage and databases.
• They need to be able to store large data sets cheaply.
• They need large capacity swings available on-demand.

SDKs like AWS’ Python library, boto3, can create resources, but they still require domain expertise to manage and properly architect a solution. The Pulumi Automation API improves on raw SDKs by providing high-level abstractions for creating and managing cloud services, letting data scientists concentrate on analyses and models without being well-versed in cloud APIs.

Read more →

Whether it’s an IoT installation, a website, or a mobile app, modern software systems generate a trove of usage and performance data. While it can be daunting to collect and manage, surfacing data empowers the business to make informed product investments. In this article, we’ll explore the following:

1. An overview of the traditional Redshift analytics stack on AWS, the use cases it excels at, and where it falls apart.
2. An alternative architecture utilizing serverless and streaming.
3. How to implement this architecture as code in a reusable library with Pulumi.
4. How to automate the development loop when writing Pulumi libraries.

If you’d like to follow along, you can clone and run the reference implementation. If you’re new to Pulumi, you can follow this guide to get started.

Read more →

In this post, we will work through an example that shows how to use Pulumi to create Jupyter Notebooks on Kubernetes. Having worked on Kubernetes since 2015, a couple of critical benefits jump out that may resonate with you as well:

• You write everything in code - TypeScript in our example here.
• You need not initialize Tiller or Helm to work with existing Helm charts like nginx-ingress-controller that we use here.
• The security patterns in Helm and Tiller are no longer concerns, rather you get to focus on the RBAC of the actual service which is Jupyter-notebook in this example.
• You accomplish more with less YAML and iteratively work towards your use cases.

Read more →