Static lib linking in Go program

Unknown — Tue, 26 Aug 2025 00:00:00 +0000

I have a Go program for various utilities. Some of the features talk to hardware: sensors</code> and battery statistics</code>. For funsies, I ported this program into Rust because I want to get a feel of the language outside the data/ml ecosystem. To my surprise, I found that Rust's implementation of these features have better hardware coverage. Specifically, in Go it couldn't get sensors info on some hardware, and battery cycle count isn't available in Go at all.

I like Rust, but I find Go has better TUI ecosystem. As to not shoot my future-self in the foot, utilizing FFI might be more ideal.

How it works</a> </h2> Basically you have to use Rust to build a static library, then import it into Go. Finally, bundle this static lib into a Go binary. Note that you have to expose the Rust functions for C, and you also need to provide a C header file. Regarding static vs dynamic library: you probably have an easier time linking a dynamic binary, but it's more headaches if you want to distribute a plain binary. This is because for dynamic library, the library itself has to be present somewhere on your system so the Go binary can load external library at runtime. Since a binary is OS/arch dependent, so is static library. This is where the fun starts. Good news is you can use cross</a> to cross-compile Rust crates, but this doesn't apply to actually linking the static lib into Go binary. CGO import</a> </h2> In Go, you need to set CGO import: /* #cgo linux LDFLAGS: ./lib/libsystem.a #cgo darwin LDFLAGS: ./lib/libsystem.a -framework IOKit #cgo windows LDFLAGS: ./lib/libsystem.a #include "../../lib/system.h" #include <stdlib.h> */ import "C"</code></pre> Notice that you probably need different configurations for each OS. Notice that for darwin</code>, framework</code> is required since it's MacOS specific. I got the error during build and it pointed me to which framework to specify for LDFLAGS</code>. Go build configurations</a> </h2> You also need to set the right combination for C Compiler</code> and LDFLAGS</code>, in which it is as follows: OS</th> Arch</th> CC</th> LDFLAGS</th></tr></thead> linux</td> amd64</td> </td> -linkmode 'external' -extldflags '-static'</code></td></tr> linux</td> arm64</td> aarch64-linux-gnu-gcc</td> -linkmode 'external' -extldflags '-static'</code></td></tr> darwin</td> amd64</td> </td> -linkmode 'external'</code></td></tr> darwin</td> arm64</td> </td> -linkmode 'external'</code></td></tr> windows</td> amd64</td> x86_64-w64-mingw32-gcc</td> -linkmode 'external' -extldflags '-static -lntdll -lsetupapi'</code></td></tr> </tbody></table> Notice the lntdll</code> flag for windows</code>, during initial binary build I didn't add this and it throws an error, pointing me to add this flag for compilation. Note that linux can cross-compile for windows, but darwin needs a MacOS machine. Also it might throw an error about recommending musl</code>. Because gcc-aarch64-linux-gnu</code> and gcc-mingw-w64-x86-64</code> depend on glibc</code>. To fix this error, you have to use a musl-based C Compiler instead. But this entirely depends on whether your static library talks to glibc</code> or not, you don't have much say in this. CI/CD</a> </h2> You should utilize a matrix strategy to fan out the build jobs, and don't forget to install the C Compiler and Rust target for the correct target OS/arch combination. Relevant snippets: - name: Install cross-compilers run: | if [ "${{ matrix.os }}" == "linux" ]; then sudo apt-get install gcc-aarch64-linux-gnu elif [ "${{ matrix.os }}" == "windows" ]; then sudo apt-get install gcc-mingw-w64-x86-64 fi</code></pre>- name: Add Rust targets run: | if [ "${{ matrix.os }}" == "darwin" ]; then rustup target add x86_64-apple-darwin rustup target add aarch64-apple-darwin elif [ "${{ matrix.os }}" == "windows" ]; then rustup target add x86_64-pc-windows-gnu fi</code></pre> AWS cost optimization tactics Unknown — Mon, 04 Aug 2025 00:00:00 +0000 Imagine working on a product and you finally released it to the public. Things are going great for you, then your budget got out of control and you have to tame the spending. Below are a few tactics you can utilize to curb the costs. Cronjobs -> ECS Tasks on Fargate</a> </h2> Cronjobs mean you need to have an EC2 (vm) running all day and night. Even if your cronjobs only get triggered for an hour late at night, you still need to setup a vm start/stop schedule (read: more setup effort), in which you have to make sure that it starts before your cron schedules, and shuts down after your cronjobs run successfully. It's a lot of time logistics involved, and if the time window the EC2 stays up is too tight some of your jobs might not run. ECS tasks is a way for you to fire up container-based jobs, it only charges you for the amount of time the tasks take to complete. You don't need to provision a vm, because ECS on fargate is serverless. You can even make it cheaper by using fargate spot! (But don't forget to setup retries because sometimes a spot job could fail due to unavailable spot capacity). ECS on Fargate -> ECS on EC2</a> </h2> If your workloads are very predictable - i.e. predictable spikes, you might want to consider using EC2-backed ECS. This is because in lieu of using fargate for compute backend (which has a higher price-point per compute-second), you can provision EC2 yourself as ECS backend. You can make it even cheaper by buying reserved instance. For Not-Always-On Workloads</a> </h2> You have a few approaches here. You can use ECS on fargate and setup a schedule for workload activation/deactivation. Let's say if your workload needs to be active during work hours, but is idle at night, this would work perfectly. Alternatively, if it's a very small workload - let's say half a dozen functions, it could be converted into Lambdas, which has an added benefit that it can scale to zero. This means it only charges you when it receives incoming requests. Bonus: What's up with using EC2</a> </h2> If you want to get something up and running real quick, EC2 is a fine solution. It's what everyone is familiar with, same steps as when you were to work on your local workstation. However, at certain point in time, you would eventually run out of disk space - and you have to set up a disk monitoring and cleanup job. Additionally, you have to perform OS upgrades as well, and things might go wrong, but you'll never know until you perform the upgrade. Yes, this sounds like Russian roulette. And the worst of all is there's no clean way for you to read your workloads' logs without SSH-ing into the vm. Or you have to forward your logs to an external service, but then you have to set it up, which is more work. Terraform project structure Unknown — Tue, 22 Jul 2025 00:00:00 +0000 Recently I gave a terraform talk, and people ask how to get started with terraform. The simple answer is to try to get your hands dirty with it, but at certain point you have to learn how to "manage" the ever-growing terraform project. Single-project structure</a> </h2> In the beginning, you can probably make do with a single terraform workspace containing all resources. A single terraform project can look like this: . ├── backend.tf ├── main.tf ├── outputs.tf ├── providers.tf ├── terraform.tf └── variables.tf</code></pre> backend.tf</code> This is where you store your terraform state. terraform { backend "gcs" { bucket = "xxx" prefix = "terraform/xxx" } }</code></pre> main.tf</code> Your terraform resources should be defined here. You can split it into multiple files for each resource/service group. resource "google_cloud_run_v2_job" "this" { // define your resource here }</code></pre> outputs.tf</code> Anything you want to export stays here. output "foo" { value = "xxx" }</code></pre> providers.tf</code> This is where you define required terraform providers. You probably have to supply required values to authenticate to services. provider "google" { project = var.project_id region = var.region } provider "sops" {}</code></pre> terraform.tf</code> This file is equivalent to your project's dependencies - what your project requires to successfully create and modify resources. terraform { required_providers { github = { source = "integrations/github" version = "6.2.1" } google = { source = "hashicorp/google" version = "5.23.0" } random = { source = "hashicorp/random" version = "3.5.1" } sops = { source = "carlpett/sops" version = "1.0.0" } } }</code></pre> variables.tf</code> Anything specific to your setup and can be changed should be defined as variables. variable "project_id" { type = string default = "xxx" } variable "region" { description = "The region of resources" type = string default = "asia-southeast1" }</code></pre> Sometimes the variables contain secrets, in which case you might want to create a separate terraform.tfvars</code> and specify your values there. Don't check this file into git. terraform.tfvars</code> should look like this: foo = "bar"</code></pre>Multi-project structure</a> </h2> Over time your terraform project can contain more resources - taking longer and longer to terraform plan</code> and terraform apply</code>. If you start experiencing this, it is probably time to split your current project into multiple projects. There are plenty of resources online for this, but I would suggest you to practice it before splitting your production project, as refactoring a terraform project is a very delicate process. Over the years I have experimented with a lot of terraform project structure, since I had to maintain all infrastructure in an organization, spanning all products they have. Following structure has been serving me well: . └── environments ├── core │ ├── gcp │ │ ├── logging │ │ └── networking │ └── github ├── dev │ ├── aws │ │ ├── productA │ │ └── productB │ └── gcp │ ├── productC │ └── productD ├── non-prod │ ├── aws │ │ ├── playgroundA │ │ └── playgroundB │ └── gcp │ ├── sandboxA │ └── sandboxB └── prod ├── aws │ ├── productA │ └── productB └── gcp ├── productC └── productD</code></pre> Essentially, you separate the root level by environments. Under each environment, separate each infrastructure/service vendor. Finally, group terraform resources by product/project. You'll notice that for core infrastructure it has its own core</code> environment, because it is shared between almost all environments (for instance, shared logging sink for all environments). The true cost of deploying on EC2 Unknown — Fri, 18 Jul 2025 00:00:00 +0000 Naturally, when you are working on a project, be it python, go, rust, you would write code and make sure it works as expected. For deployments, packaging your project into a docker image and deploy it as a container would guarantee that it would work the same regardless of which machine it's running on. Initial deployment</a> </h2> It would be understandable for people to think "I need a VM to deploy my stuff," since it uses the same setup as their development workflow. It would go like this: Spin up a VM</li> Install docker</li> Clone source repository</li> Run docker build</code> command</li> Run docker run</code> or docker compose up</code></li> </ol> For simplicity, let's assume there is no databases involved. At this point an app would run normally, however you wouldn't be able to access your service, because you haven't set up ingress rules. To do that, you would need to fiddle with vpc and security groups attached to your ec2. If done correctly, you should be able to access your service with http://${EC2_PUBLIC_IP}:${SERVICE_PORT}</code>. Note that, if you stop your ec2 instance and restart, your ec2 would get a new IP address on reboot, which means you have to change the IP you use to access your service. If you want to map your service to a domain, this would pose an issue because the IP would keep changing, rendering your A record unresolvable. To fix this, you need to reserve an ip (elastic ip on aws) and assign it to your ec2. SSL strikes back</a> </h2> Keen readers would realize that it's http</code>, which is not ideal because there would be no encryption when you are interacting with this service. To solve this, you need to put a reverse proxy in front of your service. You can achieve this in two ways: Add apache2, nginx or caddy in front of your service. This can run as another container on ec2, or as systemd.</li> Create AWS Load Balancer and use ec2 as backend. You also have to generate a certificate via AWS (or you can import it) and attach to your load balancer as well.</li> </ol> You probably don't need a dedicated load balancer for a small project, but at certain scale it would be helpful when you have a lot of incoming requests, because an LB can distribute loads (assuming you point it to a fleet of ec2 serving up the same service), in turn allowing you to scale out your service. What about CI/CD</a> </h2> Manual setup would be for every code changes, you ssh into your ec2, do a git pull</code> and docker build</code> and docker compose down && docker compose up</code>. If you think this is very tedious, that's because it is. You have a few possible approaches to achieve ci/cd: Set up a cronjob to periodically pull your repo and update running image -> has downtime Alternatively, you can replace a cronjob by using a ci/cd system to ssh into ec2 and run the same commands</li> </ul> </li> Use ci/cd system to build and push docker image to aws ecr, then set up a cronjob to periodically update running image -> you need to setup aws iam so your ec2 can pull from a private ecr repo, and this still involves downtime</li> </ol> Both solutions still results in a downtime, in which if this is not acceptable for your application, keep reading... Containers as a service</a> </h2> At this point you can probably tell that deploying to ec2 is simple, unless it involves ci/cd. But since you are using containers, aws has ecs where you can spin up a deployment from a supplied ecr image. Alternatively, you can also use eks, but if you only have a single service, eks is an overkill. For ci/cd, it's as simple as build and push image to ecr, update your ecs deployment manifest and update it - all can be done via cli commands, as long as you supply the correct aws iam credentials. You still need to create a load balancer, because an ecs deployment update works on a rolling basis - a new deployment spins up and if it doesn't fail, only then the load balancer would switch traffic to this new deployment, then tear down the previous revision. This translates to no downtime. Using EC2 but with a twist</a> </h2> The good thing about aws ecs is that you can use ec2 as backend as well, in some cases it might be cheaper than the default backend (fargate - which is serverless). Essentially you add containers orchestration on top of ec2. While deploying on ec2 is simple at a glance, it introduces a lot of issues, namely unavoidable downtime and complex ci/cd setup. By using containers-as-a-service, ci/cd can be easier to achieve, and rolling update would be introduced to avoid service downtime during upgrades. Dataframe processing benchmarks (ภาษาไทย) Unknown — Tue, 08 Jul 2025 00:00:00 +0000 English</a></a> </h4> ช่วงหลังๆ มานี้ duckdb</code> กับ polars</code> มาแรงขึ้นเรื่อยๆ พ่วงมากับการที่ parquet</code> เป็นที่นิยมมากขึ้นในการใช้เก็บข้อมูล เลยว่าเป็นฤกษ์ดีในการมาดู ว่า data format กับ framework จะมีผลกับความไวในการปั่นข้อมูลขนาดไหน Processing 1 Million Rows</a> </h2> สำหรับการ transform ข้อมูลแบบง่ายๆ (colNew = colA/colB) ค่อนข้างชัดว่า ถ้าอ่านเป็น parquet จะใช้เวลาน้อยสุดในการปั่น ถ้าเป็น ndjson หรือที่เรียกกันอีกชื่อว่า single-line json ส่วนมากจะโผล่มาสองแบบ: แบบ plaintext กับแบบที่เอาไปห่อไว้ใน gzip อีกที ตอนที่เขียนบทความนี้ polars ยังไม่สามารถอ่าน gzipped ndjson ตรงๆ ได้ แต่ duckdb ทำได้ ซึ่งก็ค่อนข้างเห็นได้ชัดว่าถ้าให้ duckdb ไปอ่าน gzipped ndjson จะใช้เวลานานกว่าถ้าเทียบกับการอ่าน plaintext ndjson แบบปกติ แต่ก็ไม่แปลกใจที่ pandas จะใช้เวลานานสุดถ้าต้องอ่าน ndjson มวยอ่านข้อมูล 1 ล้านแถว ถ้าเทียบกันยังไง polars ก็ไวกว่า duckdb แต่ถ้าเป็น parquet ก็จะสูสีกันหน่อย สำหรับการใช้ python แบบไม่ผ่าน framework ปั่นข้อมูล ก็ถือว่าใช้เวลานานสุด เป็น for-loop ธรรมดาที่ต่อจาก readlines</code> stream อีกที จริงๆ ไม่ค่อยแนะนำให้ใช้ for-loop แต่ก็เคยเห็นใน production อยู่ อยากแนะนำให้เปลี่ยนไปใช้ dataframe framework ตัวไหนก็ได้ที่สะดวก เพราะว่ามันจะเป็นระเบิดเวลา จะมีโอกาสพังเยอะ จนถึงตอนนั้น pipelines พังไม่ค่อยน่ากลัวเท่า data types พังแล้วต้องไปไล่ซ่อม แต่ก็น่าสนใจตรงที่ ถ้าจับ pandas มาอ่าน parquet ก็ลดเวลาปั่นไปได้เยอะ ก็คือถ้าจำเป็นต้องใช้ pandas แล้วเลี่ยงไม่ได้ การอ่านไฟล์ต้นทางเป็น parquet จะช่วยได้ในเรื่องของความเร็วในการทำงาน แต่ถึงจุดนึงย้ายไปใช้ polars/duckdb/spark จะยั่งยืนกว่าในระยะยาว ในส่วนของการทำ window+groupby หนักๆ สำหรับ 1 ล้านแถว ความเร็วไม่ค่อยต่างกันมาก แต่ polars จะไวกว่านิดหน่อย เอาล่ะ ได้เวลาเร่งเครื่อง... Processing 50 Million Rows</a> </h2> ณ จุดนี้ pandas อ่าน ndjson 50 ล้านแถวไม่รอด แต่น่าสนใจตรงที่ polars ใช้เวลานานกว่าถ้าเทียบกับ duckdb ในการอ่านไฟล์จากทั้ง ndjson และ parquet ที่เด่นจริงๆ คือ การที่ไฟล์ต้นทางเป็น parquet ทำให้ duckdb ใช้เวลาไวขึ้น 6x ถ้าเทียบกับอ่านจาก ndjson สำหรับ polars เอง การอ่านไฟล์เข้ามาจาก parquet ก็ทำให้ปั่นไวขึ้น 3x เทียบกัน ndjson แต่ยังไม่จบ เพราะว่า ถ้าไปมวยต่อไป คือ window+groupby: polars ไวกว่า duckdb คนเขาบอกว่า duckdb ไวกว่า polars ก็จริง แต่ขั้วตรงข้ามที่บอกว่า polars ไวกว่า duckdb ก็จริง ถ้าเทียบกันกับจำนวนแถวที่ต้องปั่นและท่าละเลงข้อมูล ถ้าเป็นอะไรง่ายๆ duckdb จะไวกว่า polars ก็คือถ้ามี 50 ล้านแถว duckdb จะไวกว่า polars สองเท่า แต่ถ้าละเลงหนักๆ แบบมาทั้ง window + groupby จะเป็น polars ไวกว่า duckdb Dataframe processing benchmarks Unknown — Tue, 08 Jul 2025 00:00:00 +0000 ภาษาไทย</a></a> </h4> Given the rise of duckdb</code> and polars</code> in recent years, in addition to parquet</code> format being more widely adopted for data storage format, it might be a good time to see how source data format affects data processing speed - in addition to the framework choice. Processing 1 Million Rows</a> </h2> For a simple transformation (colNew = colA/colB), it is obvious that when source format is parquet, it takes the least time to process the data. With ndjson, also known as single-line json, it usually comes into two variants: uncompressed and compressed. At the time of writing this article, polars can't read gzipped ndjson directly, but duckdb can. It's evident that by processing gzipped ndjson, it takes a lot more time compared to uncompressed ndjson for duckdb. Unsurprisingly, pandas takes significantly longer than duckdb and polars when it comes to reading ndjson. However, it should be of note that given 1 million rows, polars is faster than duckdb when it comes to parsing ndjson, but for parquet, it's almost a tie. As for pure python for comparison - which took longest to execute, it's running a simple for-loop for a readlines</code> stream. I wouldn't recommend using a for-loop for processing data, but I've seen a few in production - in which I would recommend you to migrate to polars or whichever dataframe framework you prefer as soon as possible, because it's a ticking time bomb for all things that could go wrong - and failing pipelines would be the least of your worries. Hint: the revenge of data types. Interestingly enough, reading parquet via pandas results in significantly faster processing speed, so if you really need to use pandas, working with parquet can improve execution performance - but at certain point migrating off to polars/duckdb/spark might be better in the long run. As for a window+groupby aggregation, at 1 million rows, it's almost a tie, but polars takes slightly less time. Now, to crank it up a notch... Processing 50 Million Rows</a> </h2> At this point, pandas struggles to parse 50m rows of ndjson, but interestingly, polars takes longer to run compared to duckdb for both ndjson and parquet. Also take note that given 50 million rows, having parquet as source in duckdb results in 6x faster execution time compared to ndjson. As for polars, parquet can give a 3x performance boost in relation to ndjson as source. But not so fast, because if we are talking about window+groupby aggregation: polars takes less time than duckdb. The claim that duckdb is faster than polars is true. But another conflicting claim that polars is faster than duckdb is also true. Given the amount of data to process and performed operations, for a simple transformation, duckdb is faster than polars - at 50 million rows this translates to duckdb being 2x faster than polars. But if we are talking about heavy-duty aggregations, polars is faster than duckdb. Simplify SSH (and HTTP & Postgres) access management with Warpgate Unknown — Sat, 28 Jun 2025 00:00:00 +0000 Background</a> </h2> If you have to work with virtual machines (VMs), you know the drill: disable password authentication and only allow private key authentication. If you want to sleep better at night, make the VMs only accessible through a private network to reduce the attack surface. What about at scale? What if you are not the only person who has to access those VMs, because you are working in a team. Then things get interesting, because allowing developers direct access to VMs means a lot of security implications, and it's not uncommon for developers in a hurry to share their private key with a team member - and we all know that the key wouldn't be rotated anytime in the near future. How it's been done</a> </h2> In practice, this mostly translates to a bastion host, acting as a bridge between developer's machines and target machines. First you use the bastion host to forward the SSH port, then you use it to access the target machine. This would likely involve the bastion host admin to provision a unique private key for each developer, add its public keys to the bastion host. But this is only the bastion host we are talking about, then there are those target machines where ideally the same public keys should be added to authorized keys as well. You see where this is going. If someone leaves the org, yup, better make sure that you revoke their public key from all machines. One thing to keep in mind is you'll have to add some guardrails to disable shell access, since some developers, in a hurry, against their better judgment, might add a few stowaway cronjobs critical to production to the bastion host 🙃. The hard part that we all dread, but they need to be done</a> </h3> Did I mention audit logs? Have fun combing through the logs. And if you find provisioning private keys tedious, I'm with you on that. Then you have to come up with clever tricks to make sure only certain developer's group has to access to particular VMs, while disable access to other VMs. RBAC strikes back. Then I found Warpgate</a> </h2> I'll admit at first it was tricky to assess what warpgate was trying to solve, but upon reading their documentation</a> it paints a clearer picture - to use RBAC to control access to targets. This comprises three layers: User Identity - can be native to warpgate or via SSO</li> Targets - can be SSH, HTTP, MySQL or Postgres</li> </ol> I find this is very neat since I don't need separate systems for managing non-SSH access. </blockquote> RBAC, which users/groups can access which targets, and which authentication mechanism to use</li> </ol> You can use password, private key or web-based approval workflow. They can be specified separately per each target type as well. </blockquote> The setup works like this: warpgate generates its own private/public key on initialization. You only have to add its public key to target machines, this means for all your VMs fleet, only one public key is required to be added in order for warpgate to be able to access it. When you SSH into your target machine, your host would be warpgate's acme</code> and username is $WARPGATE_USERNAME:$WARPGATE_TARGET_NAME</code>. You can supply a password or private key, or use the web-based approval flow for authentication, depending on how you set it up, and you'll be forwarded to the target machine. It is similar for HTTP target, but the cool thing is you can bind it to a domain as well, which makes it great for internal apps. As for databases, it is similar to ssh targets, existing clients should work without requiring extra configurations. And per ops tradition, warpgate has a terraform provider, and you should definitely use terraform to manage RBAC, because you'll breeze through the audit. But let's not forget one of those times when you need to grant a temporary access, warpgate has you covered as well. The best part is that warpgate is free, as in beer 🍺. Using Nix for CI, but with a twist Unknown — Fri, 20 Jun 2025 00:00:00 +0000 CI (continuous integration) is kind of tricky in a sense that each CI vendor has its own implementations, often glued together via yaml configurations. If you have a CI to build a binary, unless you use bash to 1) install dependencies and b) build the artifacts, most likely you would end up with different implementation for each CI platform. The problem with using plain bash script is that it's a lot of glue code (setting PATH variable for one), not to mention having to make sure that dependencies are pinned to a specific version (because things are going to break with upgrades, I can assure you that). This essentially means that your CI pipelines would be very well optimized if you use vendor-specific building blocks (for installing dependencies, building binaries with cache, etc), but it's not portable. Bash scripts are portable but very brittle in a sense that it's harder to pin dependencies. But Nix exists, so why not. Nix is a build system, but you can use it to install packages. Take gomod2nix</a> for example, basically it stays within your golang project so you can use nix to build a golang binary. It's very pleasant to use, but that's a lot of boilerplate code, and it's still tricky to figure out how to do cross-compilation build. But then flox</a> exists. Think of virtual environment, but behind the scenes it's nix installing tools and packages. This means you can offload dependencies installation to flox/nix, and continue using bash for the build process. In .flox/env/manifest.toml</code>, you would specify packages like this: [install] go = { pkg-path = "go", version = "^1.23.3" } air.pkg-path = "air"</code></pre> And once you activate the flox shell, you can continue doing your own thing. In CI, this means that as long as you can install flox/nix, and set it to utilize cache, you are pretty much good to go. Although you probably have to use vendor-specific configurations for uploading build artifacts, but it's a small price to pay for not having to reimplement everything for each CI system. Hello Garage, goodbye MinIO Unknown — Fri, 13 Jun 2025 00:00:00 +0000 Given the trend where open-source software license changes to be less permissive, or move features behind a paywall, existing users might not be quite as happy about this predicament. Case in point: MinIO Removes Web UI Features from Community Version, Pushes Users to Paid Plans</a>. But this post is not about that, but rather more on why I'm migrating to garage. Recently I've been using minio as a dvc backend (think of git LFS but using blob storage as backend) and it accumulated a lot of small files. Understandably, minio performance tanked because it doesn't handle small files well. Also minio deployment seems to use a little too much resources - it consumes 218 MB of memory. This much memory usage would be expected of a web application, but not for something like minio. Also see this minio issue</a>. To get started with garage, follow this quickstart</a>. Adjust the toml config as necessary. Note that aws s3 api performs a checksum, and recent aws sdk broke garage's default settings. See this issue</a>. In essence, to get the most recent of garage (v1.1.0) working with aws cli and sdk, following configurations should be made: aws config</a> </h4> set region to the value you specified in garage.toml</code></li> </ul> environment variables</a> </h4> For preventing invalid checksum errors. AWS_REQUEST_CHECKSUM_CALCULATION=when_required AWS_RESPONSE_CHECKSUM_VALIDATION=when_required</code></pre>Resources consumption</a> </h2> I didn't expect this much difference, but it's very welcome. Service</th> CPU</th> Memory</th></tr></thead> MinIO</td> 3</td> 218</td></tr> Garage</td> 1</td> 5</td></tr> </tbody></table> Because garage is written in rust, score for Ferris the Crab! Cloudflare's terraform v5 provider migration strategy Unknown — Tue, 10 Jun 2025 00:00:00 +0000 Back in February 2025, Cloudflare announced that its terraform v5 provider is GA</a>. However, this release contains a lot of breaking changes, but I understand why it had to be this way - because it's less work if you generate a terraform provider via OpenAPI specs. The v4 to v5 provider upgrade guide is provided</a>. However, it's not for the faint of heart, and it contains a lot of hacks. Moreover, some resources are renamed, which makes it even more tricky to perform a provider upgrade in-place, seeing it would fail the tfstate validation (trust me I've tried). For a very small project, after a lot of trial and error I finally figured out that you have to remove all resources where the resource name changes in v5, then re-import them later. However, I have a large terraform project, and this solution doesn't scale well. The good thing is that terraform state is in JSON format, so a little python-fu to extract resource names and ids should work, then initialize a new terraform project and slowly re-import everything back. You can copy the existing project, comment out all resource blocks and backend definition, then re-import cloudflare resources in chunks, and terraform plan</code> periodically for sanity check. Python Utils Script</a> </h2> This is a little hacky, but it's a one-time kinda thing. You can modify this for other providers as well, but you'll have to adjust the blocks for resource name and ids extraction. import json with open("terraform.tfstate") as f: state = json.load(f) for resource in state["resources"]: resource_name = "" resource_id = "" if resource["mode"] == "managed": is_module = resource.get("module") if is_module: resource_name = ( resource.get("module") + "." + resource["type"] + "." + resource["name"] ) else: resource_name = resource["type"] + "." + resource["name"] resource_type = resource["type"] for instance in resource["instances"]: if resource_type == "cloudflare_record": instance_id = instance["attributes"]["id"] instance_zone_id = instance["attributes"].get("zone_id") resource_id = f"{instance_zone_id}/{instance_id}" elif resource_type == "cloudflare_api_token": resource_id = instance["attributes"]["id"] elif resource_type == "cloudflare_page_rule": instance_id = instance["attributes"]["id"] instance_zone_id = instance["attributes"].get("zone_id") resource_id = f"{instance_zone_id}/{instance_id}" elif resource_type == "cloudflare_r2_bucket": account = instance["attributes"]["account_id"] id = instance["attributes"].get("id") location = instance["attributes"].get("location") resource_id = f"{account}/{id}/{location}" elif resource_type == "cloudflare_pages_domain": account = instance["attributes"]["account_id"] project_name = instance["attributes"].get("project_name") domain = instance["attributes"].get("domain") resource_id = f"{account}/{project_name}/{domain}" elif resource_type == "cloudflare_pages_project": account = instance["attributes"]["account_id"] project_name = instance["attributes"].get("id") resource_id = f"{account}/{project_name}" if instance.get("index_key"): resource_name_loop = resource_name + '["' + instance["index_key"] + '"]' print(f"tf import '{resource_name_loop}' {resource_id}") else: print(f"tf import '{resource_name}' {resource_id}") print("----")</code></pre> Model inference in ONNX Unknown — Tue, 03 Jun 2025 00:00:00 +0000 Everyone agrees that python is great for data stuff, be it data engineering or machine learning. Although rust is gaining more traction for data-related applications, python is here to stay. Those who venture into machine learning are at least familiar with sklearn - a popular framework for machine learning. A common use case is you feed it a dataframe (or number arrays) and it will return a mathematical model for you. The issue is that packaging it to make it production-ready is not trivial, and that you are tied to using python for your inference code. While I agree that python is great for data stuff, it really doesn't excel at production where you need reliability and sensible error handling (I should know... I write go, rust python and I really don't want my worst enemy to deal with debugging python in production). But thankfully there's a way out of this - serialize a sklearn model into a portable format (onnx) and serve it in rust (I tried serving onnx with go, it was very painful - so rust it is). You can serialize pytorch models into onnx as well, so it's not only limited to classic models. Raw Numbers</a> </h2> mean_s</th> median_s</th> min_s</th> max_s</th> language</th></tr></thead> 0.133962</td> 0.134343</td> 0.127497</td> 0.142944</td> onnx-python</td></tr> 0.021048</td> 0.020614</td> 0.019894</td> 0.024474</td> onnx-rust</td></tr> 0.666279</td> 0.662034</td> 0.654630</td> 0.691337</td> scikitlearn-python</td></tr> </tbody></table> Model Inference Performance</a> </h2> Native sklearn model is slowest, this is expected. By serializing the model into onnx, python serving performance is significantly better. But it's even faster if you serve onnx via rust. This means onnx-rust is 552%</code> faster than onnx-python!!! Is it 🐍 or 🦥, I'll let you decide. Updated 2025-08-15 I'm preparing for an upcoming workshop, so here's more benchmark results: Serving scikit-learn model via fastapi in python, the inference takes 0.0145s</code></li> Serving onnx model via axum in rust, the inference takes 0.0120s</code></li> </ul> This means onnx-rust is 0.1718s</code> faster than scikitlearn-python. Which translates to 17.18%</code> faster.` Cost Analysis</a> </h2> Given a ride-hailing app example: 100k users per month -> 15m requests per month for cost calculation endpoint. Serving the scikit-learn model in python via google cloud run would cost: 8.069 USD</code>. But if we use onnx via rust which is faster, it would cost 6.678 USD</code>. Which is roughly around 17%</code> in terms of price difference, this is because cloud run price, given fixed compute specs, is a function of time. For a larger model, which would mean higher latency, it would translate to more cost savings. My code search setup throughout the years Unknown — Sat, 31 May 2025 00:00:00 +0000 Initial Setup</a> </h2> Around 2022-23, I was browsing awesome self-hosted</code> and came across Sourcegraph</code>, a popular code search tool. I don't plan to pay, and they allowed self-hosting, so it was perfect for me at the time. Then you still can't search code in GitHub (which we used for work), so being able to index ever-growing repos was very important. My productivity skyrocketed and our teams were happy because they don't have to hunt down that specific line in god-knows-which-repo-and-which-file. This went on for two years, then the license changes happened and we can't use self-hosted Sourcegraph with more than a handful of private repos anymore. Livegrep to the Rescue</a> </h2> I then discovered livegrep</code>, a very lightweight code search solution. Previously with Sourcegraph, to make things easier, they bundled all services into a single container image, which is a good thing because it's very convenient. But with livegrep, you need three components: an indexer, backend and frontend. A docker compose is provided so you don't have to stitch everything together yourself. But livegrep is very well optimized, to the point that my current running instance against 200+ repos only consume 99 MB of memory for backend, and 21 MB for frontend, which is a rare sight in 2025. The way livegrep works is that initially you'd run an indexer (it has direct integration with GitHub), which would clone the repos and generate the search index. Then you start a backend that picks up the said index, and frontend talks to backend via REST api. On subsequent indexing, after an indexer job completes, you just have to restart the backend and frontend, respectively. New Kid on the Block</a> </h2> Recently there's Sourcebot</code>, it uses the same innards as Sourcegraph but with different frontend implementation. It's slower than livegrep (very noticeable) and it doesn't work quite great when indexing 200+ repos. Which means.... Some DIY is Needed</a> </h2> livegrep is perfect for everything except that it doesn't automatically clone your private repos by default. If private repos belong to an organization, there is no issues, but if they belong to a user, then livegrep can clone it, but you have to specify those repos manually. This is because the GitHub api livegrep uses to fetch your repos list does not return user's private repos (it's in the documentation). To circumvent this, I wrote a small golang utility to fetch repos list (either belonging to orgs or users), then clone them. I also use Forgejo hosted on my home server as well, which means the default livegrep indexer doesn't support - going DIY means I can extend the repos cloning workflow to non-GitHub providers. And the good thing is the code indexing workflow can be run separately in livegrep, so I just have to look up the code indexing config for livegrep, and add a step in my golang utility to generate it, then send it to livegrep indexer to generate the index and voila! It's really satisfying when you can write code to solve that issue that's been bothering you. WASM FFI performance benchmark Unknown — Tue, 15 Apr 2025 00:00:00 +0000 WASM is known for being very portable because the only thing it asks for is a browser runtime. This means it doesn't matter if you have machines on linux/amd64, windows/amd64, darwin/aarch64, WASM can run on all these platforms without required modifications. WASM can be used directly, or as an FFI through various languages/SDKs. One common pattern is when you have a multi-platform apps, the core logic would be compiled into WASM, which is then embedded within app implementations (which can be in Swift for iOS, Kotlin for Android, Electron for web/desktop). I was curious about the overhead from running WASM through various languages, this is the results. Setup</a> </h2> Given Rust-to-WASM has the best performance</a>, the WASM binary used as FFI would be written in Rust, which then compiled into WASM. The code is pretty simple: add random numbers to a fix-sized array, then sum the array, and divide it with the array length. use rand::Rng; fn main() { let mut vector: Vec<f64> = Vec::new(); let loops: i32 = 999999; for i in 0..loops { let i_float64 = i as f64; let random_float = rand::rng().random_range(0.0..999999.0); vector.push(random_float * i_float64); } println!("Vector length: {:?}", vector.len()); let vector_sum: f64 = vector.iter().sum(); println!( "Vector sum divided by total loops: {:?}", vector_sum / loops as f64 ); }</code></pre> Then execute the WASM binary via wasmtime (native), go, python and rust's wasmtime sdk. Raw Numbers</a> </h2> mean_s</th> median_s</th> min_s</th> max_s</th> language</th></tr></thead> 0.065217</td> 0.064873</td> 0.060479</td> 0.084518</td> go</td></tr> 0.049194</td> 0.048882</td> 0.046070</td> 0.055017</td> native</td></tr> 0.108358</td> 0.107370</td> 0.102075</td> 0.144279</td> python</td></tr> 0.065958</td> 0.065392</td> 0.060728</td> 0.077792</td> rust</td></tr> </tbody></table> We will revisit this later. Benchmark Results</a> </h2> Statistics</a> </h3> Python takes the longest to execute WASM binary, which is expected since Python is an interpreted language. Running WASM binary directly is also fastest, this makes sense because there's no languages/SDKs overhead. Although it's very surprising to see that Go and Rust have almost similar performance, but max</code> value for Go, which is slightly higher than Rust's makes sense due to its garbage collector. Execution Time Throughout Iterations</a> </h3> This coincides with above statistics, and you can clearly see that Go and Rust are having almost identical performance. Conclusion</a> </h2> Python has the most overhead for running WASM as FFI. Native WASM execution is fastest since there is no SDK overhead. Go and Rust are slightly slower than Native (1.62%</code> difference), but Go and Rust itself are having a 0.8%</code> gap (0.000519</code> second difference). WASM deployment benchmark Unknown — Mon, 10 Mar 2025 00:00:00 +0000 Correction 2025-06-16: there was a calculation error for artifacts size difference, it has been fixed. Previously, we've seen how WASM performs</a>. But what about delivery methods? This means runtime overhead as well, because executing a raw binary vs container means startup time is involved. Generally WASM as a binary can be executed in two ways: as a binary or from a container. But it wouldn't be fair to not compare it with native binary (linux/amd64, for example). Experiment Design</a> </h2> Write a simple rust code to add numbers to an array, and sum the array.</li> Build linux/amd64 binary</li> WASM binary</li> </ul> </li> Create a container image linux/amd64</li> wasi/wasm</li> </ul> </li> Compare binary vs container size</li> binary vs container execution time (including startup overhead)</li> </ul> </li> </ol> Wasmedge</a> is used as a container runtime. Results</a> </h2> Below is the raw data. arch</th> binary_size_kb</th> image_size_kb</th> execution_time_binary_ms</th> execution_time_container_ms</th></tr></thead> linux/amd64</td> 477</td> 8810</td> 13</td> 338.5</td></tr> wasm</td> 100</td> 150</td> 47.5</td> 3531</td></tr> </tbody></table> Artifacts Size</a> </h3> We can see that both WASM binary and container image take up less space compared to native architecture. WASM artifacts, both binary and container image, barely take up any space (100kb for binary, 150kb for container image). This results in nearly 377% for binary size difference, and 5773% for container size difference. That's really a lot of space you can save in container registry. Execution Time</a> </h3> Note: linux/amd64 native vs container execution time (excluding container startup overhead) have negligible difference. This mean we can somewhat attribute the Execution Time (Container)</code> axis as container startup overhead. Binary execution time would be faster, but compound it with container startup time, there's more overhead to run a WASM container. Conclusion</a> </h2> WASM is great if you want platform-agnostic deployment. The benefits are that the artifacts size is smaller and it can run on any platform, but at the cost of longer execution time. Migrate from Docker Desktop to Podman on Darwin Unknown — Sun, 02 Mar 2025 00:00:00 +0000 Lately Docker Desktop on Mac behaves weirdly, namely its erratic memory consumption. Not willing to babysit the memory usage, I looked into alternatives, and found a few contenders: Colima, Podman and Orbstack. Being an open-source aficionado, naturally I gravitate to Colima and Podman. Although my spidey sense tells me Podman is more polished. Gave it a go and it works beautifully, so here's my setup to make it backward compatible with tools utilizing docker command and socket. Installing Podman</a> </h2> brew install --cask podman-desktop brew install podman brew install podman-compose</code></pre>Shell Config</a> </h2> I use Fish, but it can be applied to other shells. switch (uname) case Darwin # backward compatibility with docker command function docker podman $argv end # make it work with tools relying on docker socket function lazydocker DOCKER_HOST="unix://$(podman machine inspect --format '{{.ConnectionInfo.PodmanSocket.Path}}')" "$HOME/.nix-profile/bin/lazydocker" end end</code></pre> Cross-platform keyring-like setup (also works without desktop environment) Unknown — Thu, 13 Feb 2025 00:00:00 +0000 Sometimes, running an application requires you to supply a password, or something along these lines. If you have a terminal-based workflow, sometimes it can require an authentication token. These configurations would need to be supplied either at runtime or stored somewhere on your system. We can agree that typing the creds each time isn't fun. But putting those secrets in a file on your filesystem? Attackers love this, so maybe don't make them too happy! Conventional wisdom is you should store the credentials in a system's keyring. This means you need a desktop environment (headless linux won't work with this). And that if you were to migrate to a new machine, you have to set it up again. Hmmmm..... I ran into this exact predicament, and I'm happy to share you my workaround to this. It's very simple - using SOPs</a> to encrypt secrets and store it somewhere on your system. Then create a shell function to fetch this secret key from your encrypted file. Usage</a> </h2> Setup SOPs: https://github.com/getsops/sops#usage</a>. I use age</code>.</li> Create a secrets file: sops -i secrets.sops.yaml</code></li> Write your secret keys and values</li> Move your secrets.sops.yaml</code> to your shell config's path. I use fish</code> so it's at $HOME/.config/fish/secrets/</code>.</li> Create a fish function to fetch a secret via key</li> </ol> function get_fish_secret ~/.nix-profile/bin/sops -d ~/.config/fish/secrets/secrets.sops.yaml | ~/.nix-profile/bin/yq .$argv end</code></pre> In your shell alias/functions/config, access secrets via $(get_fish_secret MINIO_ENDPOINT)</code></li> </ol> This setup also means you can store this with your dotfiles, since it's already encrypted. Here's to KISS, and don't let anyone tell you otherwise. My journey for fully switching from Windows over to Linux Unknown — Sat, 25 Jan 2025 00:00:00 +0000 Note: This is not a post to bash on Windows. It's more of what I need for work and tinkering stuff are better done on Linux Background</a> </h2> My first OS was Windows. My first-ever automation script was written in Batch script, and I learned tons from tinkering with the systems with it. Then I explored Linux during my teens, for around five years I dual-booted Windows and Ubuntu side by side. Love using both, not willing to lose the other kind of thing. In college, my laptop was outdated at that point, battery doesn't hold much charge. Apple happened to have discounts for students, so I got a Macbook. During this time I started learning Python, but I still use Windows for media management and gaming. I really noticed the difference when programming in Unix-based systems, things are much easier and less painful. Then I started working, I'm an avid gamer so I purchased a new PC mostly for gaming and media management (tagging music, books, etc). Due to my work setup (single monitor, Macbook for work), it means if I want to play games on Windows, I have to manually switch the monitor input channel and detach USB hub for peripherals attached to a Macbook to the PC rig. Over time with more work responsibilities, I barely use Windows. This went on for around 5 years. The Turning Point</a> </h2> Since I take on management responsibilities, it means my day starts earlier than I was an individual contributor. Which means I somewhat control my schedule, but it depends on when I have meetings throughout the day. I drink tea/coffee in the morning, cook every other day during lunch. Previously back when I was an IC, I can not touch a computer before 1PM, but now I have to stay glued to the computer to attend meetings and respond to bajillion messages and emails. I live in a house, due to the layout, my desk and the kitchen island is around 10-meter apart. The problem? I have to move my Macbook and dock it to my desk. Every single day. It gets old real fast. So I had a bright idea: partition off a 500 GB SSD for gaming on my PC and use it to install Ubuntu on it. One Month Later</a> </h2> I really love this setup, then I found out Proton is a thing and gaming on Linux is actually ok. Wheels are turning in my head and it clicked: I don't need to use Windows for gaming. To test this idea, I installed Witcher 3 on Ubuntu and gave it a go. Next thing I know is I'm 1/3 finished with the game. This was really fun. Only problem: The rest of the storage on this PC is tied to Windows. Oops. Migration</a> </h2> It would be great if I can move other workloads I still require using Windows fully to Linux. The said workload is music tagging, which luckily you can use puddletag</code> in lieu of MP3Tag</code> on Windows. But I still want foobar2000</code>-like music player on Linux, which isn't available. Then I found out someone packaged it as a snap</code>. And hey it works! To make sure I really don't need to use Windows, I migrated everything I need to use Windows for (gaming - done, music tagging via puddletag - done, foobar2000 - done) and wait for a few weeks. D-day, I backup data on NTFS to an external drive, remove all partitions, and partition it as follows: Drive</th> Usage</th> Mountpath</th></tr></thead> SSD 256 GB</td> boot</td> /boot/efi</td></tr> </td> home</td> /home</td></tr> SSD 500 GB</td> gaming</td> /mnt/ssd</td></tr> HDD 1 TB</td> data</td> /mnt/hdd</td></tr> </tbody></table> Fought a little with Archlinux, because it hates NVIDIA GPU apparently, so had to reflash Ubuntu. I'm using Nix</code> for packages and configurations, in total it took me 1 day to backup + install OS, and the other one for applying the configurations. Closing</a> </h2> Overall I'm very happy with this setup, and turns out I can apply better graphics settings for games compared to when I was on Windows. Information gathering infrastructure Unknown — Mon, 30 Dec 2024 00:00:00 +0000 flowchart LR Feed --> Miniflux %% feeds subgraph Homelab Miniflux --> Wallabag end Wallabag -->KOReader %% newsletters subgraph SaaS Newsletters --> Email end subgraph Automation Email --> EmailToEpubScript end EmailToEpubScript --> Syncthing %% device subgraph Device KOReader --> Ereader Syncthing --> Ereader end </pre> Once upon a time, I was asked by a recruiter regarding how I keep up with news and industry trends. I described something along this diagram, but apparently it went over his head. I'm an avid reader, but reading on a computer or tablet screen is not fun for prolonged period of time. I've been using an ereader since 2012, and I love how easy it is on my eyes. Back then things were simple, almost every site provided RSS. Then Google Reader got shut down, and people moved onto newsletters model, although some sites still provide RSS. Since I prefer to read stuff on my ereader, the flow is like this: For feeds</a> </h2> Subscribe them via Miniflux, an RSS feed aggregator. It's hosted on my homelab.</li> For articles I want to read later, I send them to Wallabag via Miniflux integration. Walalbag is a Pocket-like app, self-hostable.</li> On my ereader, I have KOReader installed. It has a Wallabag plugin to download and sync articles.</li> </ol> For newsletters</a> </h2> Use an email address for newsletter subscription to subscribe newsletters.</li> Use a Go program to fetch emails and convert them to epub</li> Put epub files in a Syncthing folder. Syncthing is a file-sync application, this should be on both your workstation and ereader</li> On ereader, sync a folder that you use via Syncthing to obtain epub files.</li> </ol> Native implementation vs WASM for Go, Python and Rust benchmark Unknown — Wed, 04 Dec 2024 00:00:00 +0000 In 2024 you would hear the term wasm</code> pop up now and then. It is not exactly new, but it's gaining more traction due to wider ecosystem in recent years. Wasm stands for web assembly, essentially it's a format that allows you to execute binaries in browser runtimes. Imagine creating a website, and you need to implement a specific function. The problem is that it is much easier to implement this function in Go, but your website is written in typescript. This happens to be a static site, so a backend shouldn't be necessary. You can either: Implement this feature as a separate api, and have your website call it - but this would add more overhead to your setup</li> Write this function in Go, and compile it into wasm and call it in your frontend code</li> </ol> There are bleeding edge use cases for wasm as well, namely utilizing CDN to serve wasm binaries in lieu of maintaining multi-region infrastructure. The future is interesting. Nowadays, many languages support compilation into wasm. Since I'm familiar with Go, Python and Rust I'm giving this a go. Experiment Setup</a> </h2> A simple multiply function implemented natively in three languages, which then is compiled into wasm binary with following setup: Go</a> </h3> GOOS=wasip1 GOARCH=wasm go build -o main.wasm</code></pre>Python</a> </h3> uv tool install --python 3.11 py2wasm py2wasm main.py -o main.wasm</code></pre>Rust</a> </h3> rustup target add wasm32-wasip1 cargo build --target wasm32-wasip1 --release</code></pre> Wasm binary is executed via wasmtime</code>. Benchmark Results</a> </h2> Note: wasm binary execution has a cold start. This means on the first run, it would be significantly slower than subsequent runs. Statistics</a> </h3> At a glance, it is obvious that Python's wasm takes significantly longer than its native implementation. Go's wasm implementation is also slower. Rust's wasm performance has the least gap with native implementation. Meaning: Rust to wasm does not result in drastic performance difference. Zooming in, we can see that there is still quite a gap between Rust and Rust's wasm, but it's very miniscule. Execution Time Throughout Iterations</a> </h3> For all run iterations, Python plateaus. Go's wasm also stood out. Zooming in, we can clearly see that Rust's wasm is very close to native Go and Rust implementation. In Go and Python, I actually feel it's longer to run the benchmark, but I didn't notice Rust's wasm taking longer than native Rust at all. Conclusion</a> </h2> In terms of performance difference (median) for wasm in Go, Python and Rust: Language</th> Wasm Performance Difference (Second)</th> Wasm Performance Difference (Percent)</th></tr></thead> Go</td> 0.017</td> 1317%</td></tr> Python</td> 0.02</td> 23%</td></tr> Rust</td> 0.003</td> 466%</td></tr> </tbody></table> In terms of actual execution time, Python has the most wasm overhead (at 0.02 second), Go is coming in second at 0.017 second. Rust meanwhile has the least wasm overhead at 0.003 second. But in terms of percent difference from native implementation, Go has a whopping 1317% longer execution time in wasm. Python while having the most wasm runtime difference, it does not increase much from native implementation (at 23%). Rust still has 466% wasm overhead, but in terms of raw numbers it's only 0.003 second. Should you use wasm? It depends on your use case. But if you are one of the lucky few who employ Rust developers, why not give it a go? The wasm performance difference won't likely impact your business. Unless you are Cloudflare or Discord. Various notes on career-related stuff Unknown — Sat, 19 Oct 2024 00:00:00 +0000 Compensation</a> </h2> Sometimes you need to take a pay cut in order to learn new skills. But it will pay off in the end. On the other hand, if you are paid above the market rate, when you have to move it's probably harder because most orgs won't offer the same compensation.</li> </ul> </li> </ul> Skills</a> </h2> PM / PO / EM / Sales should understand technical stuff, as in they are aware of pros and cons of solutions being used. Bonus if they used to be an IC.</li> "Do you need data structures and algorithms for $x" often comes up in community. Short answer is you don't need it until you do. But most people never reach the point where they need to utilize DSA. Not because it's obsolete, rather more of most people can get by with poorly-optimized code.</li> </ul> Career ladder</a> </h2> No matter how well you can perform in your current role, if you don't have the right amount of soft skills and big-picture thinking, you can't advance further.</li> Most ICs who's outgrowing their senior/technical lead roles tend to jump straight into considering that they are staff/principal engineer. This is almost always not the case, because to be a staff/principal, you still need soft skills and business acumen. You are influencing without authority after all.</li> Most engineers are stuck at mid level, and that's to be expected. And even less reach senior level. Less and less for technical lead and engineering manager. Some people are still mid level after 10 years in the industry, and you have to let them be. Not everyone has what it takes to advance.</li> You would also find that some people can operate at a higher level, but it utilizes a lot of cognitive load for them, to the point they prefer to step down a level just to not lose their sanity.</li> </ul> </li> If you are doing your best and are the best performed member of your team - if you are not getting promoted, chances are your manager doesn't want to lose you, because keeping you exactly where you are means your manager doesn't have to find someone to replace you. But it can also be that you need to catch up on your soft skills. Talk to your manager to assess your skills gap.</li> </ul> </li> Having "senior" in the title doesn't make someone actually operating at a senior level. It's just a title. The same goes for "manager" as well. A lot of senior engineers by title are actually mid levels. And a lot of managers only took the job because they want more pay but don't actually enjoy managerial duties.</li> </ul> </li> </ul> Roles</a> </h2> Being a manager isn't about attending meetings all day. Rather it's unblocking people and making sure people get the support they need. But if they do this job well, you wouldn't notice them at all. But if they do it poorly, you will notice their existence.</li> </ul> </li> Data roles can be business or tech oriented, or somewhere in the middle. No point in fighting whether a DS/DE/MLE role should be technical or business-led.</li> Some roles are not entry-level. For instance, data engineering, data science, devops. But the caveat is that if you are joining an established team, it's somewhat doable for entry-level.</li> Some job descriptions for data engineering can include anything from development, ml deployment, data pipelines, visualization, etc. Some ops roles can expect anything from dev, llm experiment and deployment. This is a symptom of hiring managers not really understand roles collaboration. Because realistically no one can juggle everything at the same time.</li> </ul> </li> Your actual responsibilities and title may not align. It's up to you to present your title in resume.</li> After technical lead, one possible path is picking up management, PM and sales roles. If you have to manage but can't understand engineers, it's not going to work.</li> If you have to keep the timeline in check and follow-up with people from multiple teams as a PM, not understanding the concerns from each team is not going to make your job as a PM easier.</li> Pitching products and services to clients while not fully understand your offerings' capabilities can cause more headaches for the engineering team, because if you promised what can't actually be delivered, the engineering team wouldn't want to work with you.</li> </ul> </li> </ul> Job change</a> </h2> If you don't learn something new at work for a while, maybe it's time to move.</li> Sometimes it's not you, it's them. If you want to change job because you don't like someone at work, you probably find it's not effective because such people can exist anywhere. Question is how are you going to handle them. Sometimes things are outside your control, and at certain point there's no reason to fight against the current.</li> </ul> Career change</a> </h2> Data folks want to move to SWE. Vice versa is true. Both citing more pay on the other side.</li> Side projects do help, but what actually happens is people copy-pasting them left and right. So unless your resume stands out, no one would look at your side projects. Even then, your side projects should be original idea.</li> Some people want to switch career for better pay, there's nothing wrong with it. But keep in mind the possibility that you might not be cut out for it.</li> It can be hard to accept that you can't switch over to a career you aimed for. The longer this keeps going, the greater cognitive dissonance. Be kind to yourself, because there are probably other things out there that you are good at.</li> </ul> Calling C from Go, Python and Rust benchmark Unknown — Sat, 12 Oct 2024 00:00:00 +0000 There's a consensus that generally, c is very fast, and python is very slow. But if we are talking about go and rust, you would find that rust is slightly faster than go. So from fastest to slowest: c, rust, go, python. But what if you have go, python and rust calling c function? There would be more overhead, but how much? Calling C from Go, Python and Rust</a> </h2> Stats</a> </h3> Turns out python has the most overhead. But with go and rust, overheads are more or less negligible, with rust having the least overhead. Execution Time</a> </h3> You can see that python execution time plateaued, but go and rust are within the same range as c. You would also notice that go and c have random execution time peaks, but rust is the most stable in terms of execution time. Additionally, calling the same c code, using rust results in the most constant memory usage footprint. The Same Function But Implemented Natively</a> </h2> It's the same pattern if native functions are used instead of c foreign function interface. Calling C vs Native Function</a> </h2> But if we really have to compare CFFI and native function implementation, pure rust</code> is faster than rust calling c</code> Side-by-side Comparison for Go and Rust</a> </h3> Comparing pure go</code> (in blue) and go calling c</code>, pure go execution time is faster. With pure rust</code> (in rust) and rust calling c</code> it's still native implementation faster than CFFI, although with rust, the overhead is more or less negligible. But if we really have to compare c</code> and pure rust</code> (in rust), c is still faster, but the difference here is still negligible. Bonus</a> </h2> You can also call rust from python via pyo3. In which, for the same function implementation, it is waaaaay faster than CFFI! TLDR</a> </h2> Calling c from go and rust doesn't have much overhead. But if you are calling c from python, that's a very different story.</li> If we are talking about pure implementations, native rust is comparable to c.</li> For boosting python performance, utilizing pyo3 via rust is also a good option, since it brings the performance closer to go and rust.</li> </ul> LLM serving latency benchmark Unknown — Wed, 09 Oct 2024 00:00:00 +0000 After the rise of ChatGPT in 2023, these days most people are familiar with the concept of LLM - Large Language Model. If you are LLM users, things are looking bright because there are gazillion options for you to choose from. But if you are on the other side - creating and deploying LLMs, there are certain things you need to think about. For one, your implementation team would have languages/frameworks they specialize in, so that's already a constraint you have to work with when designing a solution. Thankfully there are myriad of options to choose from to deploy LLMs, (see here</a>). Even easier if you don't have to do fine-tuning, because you can serve up the model straightaway. I picked following frameworks to test LLM serving performance: Language</th> Framework</th></tr></thead> C++</td> Llamafile</td></tr> Go</td> Ollama</td></tr> Python</td> HuggingFace</td></tr> Rust</td> mistral.rs</td></tr> </tbody></table> For experiment details, see here</a>. Model used: gemma-2-2b-it</code> Let's dive in! Baseline Stats</a> </h2> From the chart, HuggingFace via Python takes longest to perform inferences. Llamafile via C++ and mistral.rs via Rust have similar performance, but Ollama via Go has slightly more latency. Ollama is written in Go, which probably explains higher latency due to garbage collection which is bundled in Go binaries. Zooming in Llamafile and mistral.rs, it's clear that mistral.rs performs better. This is most likely from the fact that Rust has better optimizations compared to C++. Latency per Iteration</a> </h2> All tests have 10 iterations. From the scatter plot, you can see that sometimes, Llamafile / Ollama / mistral.rs can have very similar performance, but often Ollama takes longer than Llamafile / mistral.rs, which is in-line with the baseline stats from above section. Python still takes the longest, and does not seem to get lower than 14 seconds on each run. Zooming in Llama and mistral.rs, it is clear that mistral.rs has lower latency than Llamafile. Only once Llamafile performs better than mistral.rs. Takeaway</a> </h2> Development time and finding the right people to work with you to deploy LLM, Rust-based frameworks yield the best performance. AWS IAM credentials best practices Unknown — Sat, 05 Oct 2024 00:00:00 +0000 It's hard to escape AWS, seeing how prevalent it is in global internet infrastructure. Chances are, most websites you visit are hosted on AWS. As a software engineer, you probably encounter AWS at certain point in your career, and while getting AWS IAM credentials to work locally during development (via aws-cli</code>) would suffice, sometimes in production land, you might need some adjustments. Having worked with AWS extensively, here are what I found to be useful to keep in mind while working with AWS IAM credentials. Disable default</code> profile</a> </h2> When you first run aws configure</code> it will ask you for access key id</code> and secret access key</code>. You can fill them, but you might accidentally delete or modify resources you didn't intend to. And rectifying the state can be time-consuming. Generally, I find it better to explicitly specify a specific AWS profile whenever I'm interacting with AWS. This is so there is an extra guardrails layer. Additionally, you probably need to use different AWS profiles for each project / service, might as well make it a habit. Define a separate read-only</code> and read-write</code> profile</a> </h2> Let's say I'm working on a service where users can upload files to S3. During development, I would need to use creds with read-write</code> permissions, because I'm going to be doing a lot of tests, and until I finish the development, there would be a lot of manual uploads/deletes. However, once things are deployed to production, I would spend more time investigating errors regarding this upload features, this would translate to "verify that $image exists on S3", and this only requires read-only</code> permission. While it's possible to use creds with read-write</code> permissions, doing so on a prod bucket is like driving without wearing a seat belt. It works, but there is no extra layer of safety to prevent accidental oopsies, but it's not so fun recovering the said images. Grant permissions to groups, not users</a> </h2> This is generic for all RBAC situations, because it's not fun to synchronize permissions for all users in a group. Avoid supplying AWS IAM credentials in production</a> </h2> This applies to other cloud providers as well - generally cloud computes has an IAM role / service account attached, which means you don't have to explicitly inject IAM credentials to make it able to authenticate to AWS. And your security department would rejoice, because they don't have to deal with yet another "quarterly secrets rotation." When (not) to use Terraform Unknown — Sat, 05 Oct 2024 00:00:00 +0000 If we are talking about IaC, Terraform would be on the list. It made IaC popular and help a lot of companies maintain infrastructure at scale. Especially when you have multiple sets of infrastructures to maintain, Terraform can help you reduce the setup time tremendously via using Terraform Modules. Think of this like a function / class in programming languages. I've been cranking out a lot of Terraform, a lot of trials and errors along the way. Picking other people's brains by reading a lot of blog posts, trawl community forums to see how other people use Terraform and what are their challenges. Below are summarizations of what I've experienced, in addition to what the general sentiments are. Terraform is great for foundational resources</a> </h2> Think of foundational resources as the first layer you have to build before you can deploy your apps. This would include databases, VPCs, virtual machines, storage volumes. These are rarely changed, or if you want to, downtimes can occur. Out of all of these, it's probably safest to only modify database configs and virtual machine specs, since it results in the least downtime. For app deployment, YMMV</a> </h2> Although Terraform can also be used to deploy apps, it is not a good idea to have [foundational resources] in the same workspace as [app deployment]. This is because after initial app deployment, subsequent deployments should be done via CI/CD, this essentially means [something else that's not Terraform is updating the app deployment manifest]1</a>. This translates to [your Terraform state will constantly be out-of-sync]. This especially is a problem if your Terraform config uses latest</code> for image tag. If your devs just rolled back a deployment to a previous SHA, if you apply this Terraform workspace, chances are you are going to override the deployment rollback (and this would make the devs confused as heck as to why their deployment is suddenly on the latest version after they just did a rollback). But it's never a no, because I still find it practical to initialize an app deployment in Terraform, but it's in a separate workspace on a per-app basis. This means it won't mess with other infra resources outside of app deployments. ^{1 Technically you can use Terraform in CI/CD for app deployment, but that's another can of worms. </div>}If you can, use it to set up / provision IAM users access</a> </h2> I have a friend who works a lot with auditors, and he told me that most orgs fail auditing because they don't have a process for IAM access provisioning. This means you can't track who has access to what, or when the permissions had been granted to particular users/groups. It is very tedious, and it should be in a separate Terraform workspace. But if you can achieve this, you would have a single source-of-truth for RBAC. Hello World API Performance Benchmark (Go, Node, Python, Rust) Unknown — Fri, 20 Sep 2024 00:00:00 +0000 Correction 2024-09-21: after using multi-stage build, Node image size dropped from 1.1GB to 130MB. The first programming language I achieved proficiency was Python, so for the longest time I've been using it to do most stuff. Last year I picked up Go, and I had a blast with it. This month I picked up Rust for data/ml works, and so far I was very impressed. It got me thinking - it's been said multiple times is Python is slower than compiled languages, and Node is very easy to use but uses a lot of memory. Since I can code in multiple languages, why not do a simple API benchmark? So here's the results. Docker image size</a> </h2> It's a given that when you compile your code into a binary, it would be smaller than adding the whole runtime and dependencies into an image. Language</th> Image Size (MB)</th></tr></thead> Go</td> 13.4</td></tr> Node</td> 130</td></tr> Python</td> 146</td></tr> Rust</td> 8.63</td></tr> </tbody></table> Resources utilization on startup</a> </h2> Once we deploy the APIs, this is how much resources they consume on startup. Language</th> CPU Utilization</th> Memory Utilization</th></tr></thead> Go</td> 1</td> 1</td></tr> Node</td> 1</td> 56</td></tr> Python</td> 2</td> 35</td></tr> Rust</td> 1</td> 0*</td></tr> </tbody></table> *This is a round-down, which means it utilizes really low amount of RAM. API Performance</a> </h2> Total Requests and Response Time</a> </h3> Given concurrent 5000 VUs</code> for 120s</code>, here is total requests you can make in a given duration, with response time. Language</th> Total Requests</th> AVG RT</th> MIN RT</th> MEDIAN RT</th> MAX RT</th> P95 RT</th></tr></thead> Go</td> 898,131</td> 335.77ms</td> 1.78ms</td> 239.55ms</td> 33.38s</td> 914.37ms</td></tr> Node*</td> 383,091</td> 823.61ms</td> 2.29ms</td> 393.98ms</td> 33.75s</td> 1.41s</td></tr> Python*</td> 225,929</td> 1.42s</td> 3.15ms</td> 861.22ms</td> 35.37s</td> 1.98s</td></tr> Rust</td> 901,038</td> 408.57ms</td> 2ms</td> 274.13ms</td> 40.77s</td> 901.35ms</td></tr> </tbody></table> *Container restarted during peak load For total requests, we can see that Go and Rust have very similar performance, while being the most performant of all. And Node is faster than Python, but still significantly slower than Go/Rust. But if we are talking about response time, it is very apparent that python is the least performant of all. Go and Rust is a tie. Peak Resources utilization</a> </h3> Turns out Node is the most resource hungry, and Rust uses slightly more CPU than Go. Language</th> CPU (Mi)</th> Memory (MB)</th></tr></thead> Go</td> 410</td> 52.48</td></tr> Node</td> 1000</td> 203.17</td></tr> Python</td> 1000</td> 67.01</td></tr> Rust</td> 492</td> 45.49</td></tr> </tbody></table> Zooming in on Go and Rust, we can see that Go uses slightly less CPU but more memory than Rust. Conclusion</a> </h2> Python and Node take up the most image size.</li> Go and Rust consume the least amount of resources and have very similar performance. While the differences are almost negligible, Rust can serve more requests.</li> Node is faster than Python, but utilizes more resources.</li> </ul> Simplify self-hosting backups to S3 with docker Unknown — Sat, 07 Sep 2024 00:00:00 +0000 These days there are multiple ways to deploy a workload, be it cloud-based or bare-metal. For cloud, depending on whether you are using PaaS or IaaS, backup options can vary. Why do we need to backup? Because your workloads can contain a state, this can be stored as local files, inside a database, or as other assets outside the application itself. Take a database for example, ideally you would need a daily backup so you can revert a database to a state before its corruption without losing as much data. Some workloads might store uploaded images, for simplicity let's say they are being written to disk. For a backup, it can be as simple as running a database dump command, or compress a data directory and save it somewhere. Following the rule of thumb: you should have off-site backups. In cloud, a managed database would have daily backups as a built-in feature, offloading users from setting up the backup operation themselves. But in a bare-metal setup (which also applies for self-hosting), you need to somehow store the backup artifacts in, say, AWS S3. And there's a lot of commands involved: current_date=$(date +%Y-%m-%d) backup_bucket="${BUCKET_NAME:-backup}" backup_prefix="s3://$backup_bucket/$current_date" filename="$SERVICE_NAME-sqldump-$current_date.bin" PGPASSWORD="$POSTGRES_PASSWORD" pg_dump -Fc -c -U "$POSTGRES_USERNAME" --host "$POSTGRES_HOSTNAME" >"$filename" aws s3 cp "${aws_args[@]}" "$filename" "$backup_prefix/$filename"</code></pre> Creating a bash script can work, but this means you have to install required binaries or else the script won't be able to execute. In which after this you can parameterize the inputs. The final step would involve setting up a cronjob, which is unavoidable, but doing the path logistics is not fun. Additionally, testing a crontab is very clunky, it would involve editing the crontabs to trigger in a minute from now. And when it's triggered, you can't see logs in real-time. But crontabs still execute commands, why not package this into a docker image, so a backup job would essentially be a docker run</code> command? This way it's easier to debug, all dependencies are accounted for, and you can use it anywhere without setting up the environment (you still need docker, but you are unlikely to mess up a docker installation). So when tying it all together, a Dockerfile would contain following binaries (so we can backup postgres or a path, and push it to s3): FROM nixos/nix:latest RUN nix-channel --update RUN nix-env -iA nixpkgs.bash && \ nix-env -iA nixpkgs.gnutar && \ nix-env -iA nixpkgs.gzip && \ nix-env -iA nixpkgs.curl && \ nix-env -iA nixpkgs.postgresql_16 && \ nix-env -iA nixpkgs.awscli2 # set entrypoint WORKDIR /opt/backup COPY entrypoint.sh . RUN chmod +x entrypoint.sh ENTRYPOINT ["bash", "entrypoint.sh"]</code></pre> Notice the entrypoint. Essentially it would take in environment variables, and run the backup commands, then push it to s3. After a successful upload, send a notification. #!/bin/bash # ----------------- VARS ----------------- # #MODE= # `ARCHIVE`, `DB_POSTGRES` #SERVICE_NAME= #BACKUP_PATH= #BUCKET_NAME= #BACKUP_PATH_EXCLUDE= # optional #POSTGRES_USERNAME= #POSTGRES_PASSWORD= #POSTGRES_HOSTNAME= #S3_ENDPOINT= # optional # either of this #NTFY_TOPIC_URL #DISCORD_WEBHOOK_URL # set filename current_date=$(date +%Y-%m-%d) backup_bucket="${BUCKET_NAME:-backup}" backup_prefix="s3://$backup_bucket/$current_date" # backup if [ "$MODE" = "ARCHIVE" ]; then filename="$SERVICE_NAME-$current_date.tar.gz" # ref: https://stackoverflow.com/a/42985721 tar_args=() if [ -v BACKUP_PATH_EXCLUDE ]; then tar_args+=(--exclude "$BACKUP_PATH_EXCLUDE") fi tar_args+=( -czf "$filename" "$BACKUP_PATH" ) tar "${tar_args[@]}" elif [ "$MODE" = "DB_POSTGRES" ]; then filename="$SERVICE_NAME-sqldump-$current_date.bin" PGPASSWORD="$POSTGRES_PASSWORD" pg_dump -Fc -c -U "$POSTGRES_USERNAME" --host "$POSTGRES_HOSTNAME" >"$filename" else echo "$MODE is not supported" fi # upload aws_args=() if [ -v S3_ENDPOINT ]; then aws_args+=(--endpoint-url "$S3_ENDPOINT") fi aws s3 cp "${aws_args[@]}" "$filename" "$backup_prefix/$filename" # notify notify_message="Successfully backup $SERVICE_NAME" if [ -v NTFY_TOPIC_URL ]; then curl -d "$notify_message" "$NTFY_TOPIC_URL" elif [ -v DISCORD_WEBHOOK_URL ]; then curl -i \ -H "Accept: application/json" \ -H "Content-Type:application/json" \ -X POST --data "{\"content\": \"$notify_message\"}" \ "$DISCORD_WEBHOOK_URL" else echo "Cannot send a notification since no notification backend has been configured." fi</code></pre> Personally I use Kubernetes for self-hosting, but I have a few services on a small VPS using docker. This setup means I can use the same method to backup docker or kubernetes workloads. Streamlit load test performance Unknown — Sat, 07 Sep 2024 00:00:00 +0000 Streamlit is well-loved by many people, especially among data folks due to the fact that it does not require prior web programming knowledge to get started. Popular use cases for streamlit can be anything from a quick machine learning application poc or internal dashboards. But what if you want to create a production deployment? Would streamlit still be a viable option? Experiment setup</a> </h2> For a production deployment, let's assume there are 500 concurrent users. For a single-page streamlit with a chat box taking 3 seconds to return a single-paragraph message: With following configuration for the load test: config: phases: - duration: 100 arrivalRate: 50 maxVusers: 500</code></pre>Benchmark result</a> </h2> -------------------------------- Summary report @ 12:50:35(+0700) -------------------------------- browser.http_requests: ................................................... 5926 browser.page.TTFB.http://localhost:8501/: min: ................................................................... 3 max: ................................................................... 19934.3 mean: .................................................................. 6975.6 median: ................................................................ 7260.8 p95: ................................................................... 16486.1 p99: ................................................................... 18963.6 browser.page.codes.200: .................................................. 5950 errors.page.goto: Timeout 30000ms exceeded.: ............................. 1486 errors.page.goto: net::ERR_CONNECTION_RESET at http://localhost:8501/: ... 9 vusers.completed: ........................................................ 421 vusers.created: .......................................................... 1916 vusers.created_by_name.0: ................................................ 1916 vusers.failed: ........................................................... 1495 vusers.session_length: min: ................................................................... 1874.4 max: ................................................................... 30379.4 mean: .................................................................. 23027.7 median: ................................................................ 25598.5 p95: ................................................................... 30040.3 p99: ................................................................... 30040.3 vusers.skipped: .......................................................... 3084</code></pre> Notice the median</code> value of 25598.5</code>, this is roughly around 25 seconds</code> to render the above streamlit page. Takeaway</a> </h2> Moral of the story: streamlit is great for a quick poc or low traffic apps. Anything more than this and you should consider using a dedicated web framework to create the frontend. Also, yes, CPU and memory usage also shot up. Reasons why you shouldn't use programming languages for IaC Unknown — Mon, 05 Aug 2024 00:00:00 +0000 When it comes to IaC (infrastructure as code), most people might have heard of HashiCorp's Terraform (it uses HCL as DSL. Interestingly enough, Terraform also has its own CDK to translate programming languages into HCL), Pulumi or AWS CDK. The latter two support programming languages as DSL. Mostly there are two camps: People who swear by HCL and think you shouldn't use programming languages for IaC</li> People who don't see why you need to pick up a new language in order to use IaC, so they prefer using a programming language they already are familiar with instead</li> </ul> Both camps are not wrong, they are both valid. However, I want to share my take on why you should use HCL for IaC. Pros for using programming languages</a> </h2> The obvious perk is that you don't need to pick up a new language, which means you can get started right away to enjoy the benefits of IaC. Cons</a> </h2> However, when you are looking for resources or templates, you might find it harder because there are many different implementations to achieve the same thing, due to the nature of programming languages where it allows for flexibility. To create a single VM inside a VPC, you can run into many different examples, depending on how people define the IaC. Additionally, if you are using less popular languages (golang, for instance) you might have fewer references than people who use javascript/typescript or python. As for IaC codebase maintenance, you would have to find people who: Can use your programming language of choice for IaC</li> Understands devops and infrastructure</li> </ol> Which would significantly reduce your candidate pools, whereas if you use HCL (a DSL, which means it's less complex than a programming language, and by definition, has less learning curve), you would not be limited by the fact that your candidate has to know a specific programming language. The same argument can be made that you still have to find people who know HCL, but being a DSL, it takes less time to get familiar with it. (Some notes on this: some people find it really hard to pick up a new language/DSL/whathaveyou. So it's understandable if they prefer to use a familiar programming language for IaC, and there's nothing wrong with that.) GCP's service account credentials can be a security risk. Here's how to mitigate them. Unknown — Sun, 14 Jul 2024 00:00:00 +0000 If you look online, many sources would tell you that you should use service account to authenticate for GCP services. While this is true, it's not for all the cases. For local development, you should use Application Default Credentials</a> </h2> Imagine working in a team, and you have to work with Cloud Run, so you request your infra team for a service account. This looks good, but then your teammates also have to work with this service. They happen to be in a hurry, so you share your service account to your teammates. Now this can be a problem, because now there are multiple users who have access to this service account. It would be very tricky to trawl through the audit logs and identify which developer interact with cloud run, because the system only sees a single identity. This is why for a local development workflow, application default credentials should be used to authenticate to GCP. Essentially you perform a login action via the gcloud command line and set ADC, it would then leave a credential file in your local filesystem. Other Google/GCP SDKs can then pick this up and use it to authenticate to GCP services. This also adds a benefit that no credentials have to be generated, meaning there's no cleanup overhead. For cross-perimeter authentication, only required personnel should be able to access the credentials</a> </h2> But if we are talking a deployment, it's possible that you need to have $serviceA talks to $serviceB. In GCP, you can assign service account to the compute directly, as this method doesn't require generating a credentials. But there's also cases where you have to access GCP services from outside of GCP. For example, using Fivetran to read data from BigQuery. Unfortunately this method requires creating a credential tied to a service account. This can become a risk if you don't limit permission for who can create / view credentials. Following the principle of least privilege can in turn mitigate unauthorized actors from accessing service account credentials. Thoughts on summarization service system design Unknown — Sun, 09 Jun 2024 00:00:00 +0000 For a summarization task, there should be an input, in which it's reduced to a handful of paragraphs. This input is in text format. You don't necessarily start from a text format though, since the source content can be audio or video files. But this means at the end, the source input has to be converted into a text format, and this involves a transcription</code> task. Transcription means taking an audio, then convert it to text. Luckily these days there are APIs you can use to achieve this. Depending on each API provider, but it's safe to assume most would support WAVE</code> or FLAC</code> encoding. Audio formats</a> </h2> Audio formats can be categorized into two main groups: lossless and lossy. Lossless means all information are preserved, and can be converted between any lossless format without losing data. For example, going from WAVE -> FLAC</code> should allow you to convert the resulting FLAC -> WAVE</code> without losing quality. However, when you convert MP3 -> WAVE</code>, this is a lossy to lossless format, which means the resulting WAVE</code> file would contain as much information as the MP3 file. Why does audio format matter</a> </h2> If you read GCP's Cloud Speech-to-Text docs</a>, it would tell you that using FLAC</code> is better if you are conscious about storage. This is good and all, but the caveats (not mentioned in the docs) is that, if you convert a lossy format (which is very common in videos) to FLAC, it would take a lot longer than converting it to WAVE. This is because WAVE is an uncompressed</code> format, which means there's less conversion overhead, seeing FLAC utilizes a heavy compression. We are talking about 48 hours</code> conversion time vs a few minutes</code>. Compute is cheap, but storage is cheaper. What about calling patterns</a> </h2> Both transcription and summarization can take quite a while (longer than a few minutes). It does not make sense to throw the entire logic into a function and have your api return it as-is, since it would leave a connection open, and if your workloads are interrupted, all progress are lost. Which is why an event-based approach is a better fit for this scenario. Initial api call would trigger an event, and serverless functions (one for each transcription and summarization) run these tasks, when completed a status is updated into the database. Frontend can poll the backend every few seconds to fetch the status of these long-running operations. What if you are lazy and just want to only create a single backend?</a> </h2> Sure, it can work. As per anything in this universe: if you try hard enough, it will somehow work. But good luck explaining to your users why it takes forever to see a response on the web UI. Faster spark workloads with Comet Unknown — Sun, 07 Apr 2024 00:00:00 +0000 For big data processing, spark is still king. Over the years, many improvements have been made to improve spark performance. Databricks themselves created photon</a>, a spark engine that can accelerate spark queries, but this is proprietary to Databricks. Other alternatives do exist (see here</a> for more details), but they are not trivial to setup. But if you use Apache Arrow DataFusion Comet</a>, surprisingly it does not take much time at all to setup. Comet stands on arrow, a data format growing in popularity. For a start, you would need to compile a jar (you need rust and maven), and good news is comet also supports spark 3.4 (as of 2024-04-07</a>). Then it's the usual: add a jar to spark jars path, and configure spark session to use comet: spark = ( SparkSession.builder.config("spark.executor.memory", "8g") .config("spark.driver.memory", "8g") # comet configs .config("spark.sql.extensions", "org.apache.comet.CometSparkSessionExtensions") .config("spark.comet.enabled", "true") .config("spark.comet.exec.enabled", "true") .config("spark.comet.exec.all.enabled", "true") .getOrCreate() )</code></pre>Result</a> </h2> For a simple group by query, using comet results in 30% faster execution time on average. At most, it can be up to 64% faster. Remarks</a> </h2> However, if you look at comet architecture diagram: This means that for certain operations, comet fallbacks to JVM for execution. If I update the benchmark query to perform window and join, the performance gain from comet is negligible. Comet is still in its early stages, but if your workloads is query-only (ie no window or join - transformations) comet should significantly reduce your workloads runtime. Slim down Python Docker image size with Poetry and Pip Unknown — Sun, 07 Apr 2024 00:00:00 +0000 Python package management is not straightforward, seeing default package manager (pip) does not behave like node's npm, in a sense that it doesn't track dependencies versions. This is why you should use poetry</code> to manage python packages, since it creates a lock file, so you can be sure that on every re-install, the versions would be the same. However, this poses a challenge when you want to create a docker image with poetry, because you need to do an extra pip install poetry</code> (unless you bake this into your base python image). Additionally, turns out using poetry to install packages results in larger docker image size. Dockerfiles</a> </h2> Below are dockerfiles I use to compare between using poetry</code> and pip</code>: Dockerfile.poetry</a> </h3> FROM python:3.12-slim WORKDIR /app # hadolint ignore=DL3013 RUN pip install --no-cache-dir poetry COPY pyproject.toml . COPY poetry.lock . RUN poetry install --only main --no-root --no-directory</code></pre>Dockerfile.pip</a> </h3> FROM python:3.12-slim WORKDIR /app COPY requirements.txt . RUN pip install -r requirements.txt --no-cache-dir</code></pre>Installed packages</a> </h2> requests = "^2.31.0" polars = "^0.20.18" fastapi = "^0.110.1" pydantic = "^2.6.4" python-dotenv = "^1.0.1" langchain = "^0.1.14" psycopg2-binary = "^2.9.9"</code></pre>Result</a> </h2> And the resulting image size is: benchmark_poetry latest 23d3105ad0dd 11 seconds ago 520MB benchmark_pip latest b7932a02a8d1 12 hours ago 388MB</code></pre> As you can see, using poetry makes the image 132 MB</code> larger. Let's say you deploy 12 times per month, that's extra 1584 MB</code>. While I agree that these days storage is cheap, reducing images size here and there won't hurt 😎. Dataframe write performance to Postgres Unknown — Sun, 17 Mar 2024 00:00:00 +0000 Previously, I talked about dataframe performance</a>, but this doesn't include writing data to destination part. At a large scale, big data means you need to use spark for data processing (unless you prefer SQL, in which this post is irrelevant). But not many orgs need big data, so small data frameworks should work, since they are easier to setup and use compared to spark. Initially I wanted to include pandas as well, but sadly it performs significantly worse than polars, so only spark and polars remain on the benchmark. Benchmark results</a> </h2> Repo is here</a>. The results: Polars is faster than spark when it comes to writing data to postgres. This is in-line with spark vs polars performance, since polars puts everything in-memory, whereas spark can read from disk. Additionally, spark is a distributed framework, so there is an overhead compared to polars. Essentially, if your data is small enough (but larger than pandas can deal with) you can use polars. But if your data is very large, consider using spark, so you don't run into scaling issues later on. Addendum</a> </h2> Also note that polars can only utilize sqlalchemy</a> or adbc</a> to write to databases. For polars, using adbc is faster, since the dataframe doesn't require de-serialization into pandas, but adbc only supports sqlite and postgres. Sqlalchemy can write to columnar databases, but due to pandas limitations, it's not quite intuitive to split data into chunks then do incremental inserts to a columnar database. My suggestion would be: if you notice performance degradation for your analytics workload, consider migrating to spark and using columnar database for a warehouse instead. How to connect to Cloud SQL from Cloud Run (no, you don't need a VPC) Unknown — Sat, 10 Feb 2024 00:00:00 +0000 A minimal application architecture would compose of a database, and an application backend. Serverless database is still in its infancy, but thankfully container-based runtime is very much alive and doing well. On GCP, a serverless container-based runtime do exist, known as Cloud Run. Standard database access pattern</a> </h2> Per standard security practices, you should not expose your database to public, this means you should use a proxy/tunnel or private network to reach your database. From above diagram, the Backend instance can access the database directly, since they are within the same VPC. However, for your local machine, you would make a connection to a tunnel, which then forwards your request to the database. This adds another layer of security because access to the tunnel is restricted, meaning not everyone can access your database, even when they know the IP / username / password. Cloud Run and and Cloud SQL</a> </h2> With Cloud Run and Cloud SQL, things are a bit different, but for a good reason. Both Cloud Run and Cloud SQL are in GCP-managed VPC, meaning, they can talk to each other directly. This means, if you are deploying your workloads in a VPC and you need to connect to Cloud SQL, you need to set up VPC peering so your workloads can reach Cloud SQL via private IP. Cloud Run configuration</a> </h2> Good news is you only need to set two parameters to reach Cloud SQL from Cloud Run. In Terraform, this would be equivalent to: resource "google_cloud_run_service" "this" { name = "cloud-run-cloud-sql-template" location = var.region project = var.project_id template { metadata { annotations = { "autoscaling.knative.dev/maxScale" = "100" "run.googleapis.com/client-name" = "cloud-console" "run.googleapis.com/cloudsql-instances" = google_sql_database_instance.this.connection_name "run.googleapis.com/execution-environment" = "gen2" "run.googleapis.com/startup-cpu-boost" = "true" } } spec { container_concurrency = 80 service_account_name = "xxxxxxxxxxxx-compute@developer.gserviceaccount.com" timeout_seconds = 300 containers { image = "asia-southeast1-docker.pkg.dev/xxxxxxxx/cloud-run-cloud-sql-template/cloud-run-cloud-sql-template:xxxxx" env { name = "POSTGRES_HOSTNAME" value = "/cloudsql/${google_sql_database_instance.this.connection_name}" } env { name = "POSTGRES_USERNAME" value = google_sql_user.this.name } env { name = "POSTGRES_PASSWORD" value = random_password.this.result } env { name = "POSTGRES_DBNAME" value = "postgres" } env { name = "POSTGRES_PORT" value = "5432" } } } } }</code></pre> Essentially, you need to set run.googleapis.com/cloudsql-instances</code> under annotations</code> to be your $CloudSQLConnectionName</code>, and in environment variable, use /cloudsql/${google_sql_database_instance.this.connection_name}</code> for $POSTGRES_HOSTNAME</code>. Why do some tutorials tell me to set up Serverless VPC Access</a> </h2> While setting up Serverless VPC Access would also make Cloud Run able to reach Cloud SQL, this is like taking a detour, because the only difference between the non-serverless-vpc-access approach is the ability to use private IP</code> to reach Cloud SQL. By definition, Serverless VPC Access allows serverless services (Cloud Run, Cloud Functions) to reach resources in a VPC using private IP. The network representation would look like this: Summary</a> </h2> You don't need to set up networking configurations to get Cloud Run to reach Cloud SQL. Just supply two configurations and it should work. What is platform engineering? Unknown — Sun, 21 Jan 2024 00:00:00 +0000 Back in 2017-2018, everyone wanted to be a data scientist. Then reality hits, that they need a data engineer for a successful machine learning project. Things didn't end there, since they also need a machine learning engineer to create production-ready code. Some people think you only need an MLE and suddenly your ML project would become a reality, sadly the reality begs to differ, because you also need to find someone to deploy and scale it, enter DevOps engineer (who understands ML, this is very important). So what's next? What if you are working on a dozen ML projects, and you find that you have to create everything from scratch, every time. Would things be better if you can somehow only need to customize the relevant bits, and reuse templates for the rest? I'm happy to tell you that there is a role for this, and it is platform engineering. This applies to other kinds of software development as well, I just use data works as an example. In a nutshell, a platform engineer is someone who understands how to implement end-to-end projects, and is familiar with how teams/roles collaborate, so they can offer shortest path (read: abstractions) to delivery. This is very important, because you have to speak everyone's language, so you can "sell" on why your solutions benefit them, and how they can use it to achieve faster time-to-market. Failing this, you can end up with abstractions no one wants to work with, because it doesn't actually solve their problems. As for abstractions, think of writing functions for reusable parts in your codebase. Just that this time it's at a project level. You could say it's a larger scale of abstractions. Definitely more moving parts and details to consider, which makes this role very challenging! Although, if you look up platform engineering in 2024, many people would tell you it's about Internal Developer Platform (IDP). It is part of a toolbox, but not necessarily what you need for platform engineering. At a certain scale, IDP would be beneficial, but it would be a very long time before you reach that point. Jokingly, I say platform engineering is a glue job. We understand and can function in many different roles, but we don't like it as much as guiding people how to collaborate and work together! Collaboration model for data science projects Unknown — Sat, 20 Jan 2024 00:00:00 +0000 Many data science teams are struggling with implementing end-to-end machine learning projects. It's a very common phenomenon, so if you are experiencing this, you are not alone. Having worked in every stage of data science project lifecycle, in addition to normal web services deployments, this is what I think how we should collaborate. Collaboration model between teams</a> </h2> Note: The diagram does not signify order of communication. Rather, it states the communication pathways between teams. Clients have problems. Business Analysts gather initial requirements.</li> Project managers plan tasks and timelines, to be discussed with Tech Lead on both Web Development and Data Team</li> Tech Lead on both teams discuss how their services should be integrated, then loop in Platform Engineer / Solutions Architect on the deployment model, in which DevOps Engineer would be the one implementing CI/CD / deployment.</li> Once each Team Lead break down the tasks, the team would start developing services.</li> Specifically for Data Team, once the ML model is working, it would be sent over to Machine Learning Engineer to optimize and refactor the code, so it's ready for production.</li> After each team has finished development, start integration tests.</li> Once everything is ready, deploy to prod and test again.</li> </ol> Why involve Web Development team</a> </h2> For web services, there are many frameworks and languages to choose from, ranging from PHP, .NET, NodeJS, etc. But for data works, python is the de-facto language of choice, which means the languages used for web services and data works are different. To which, the solution is to expose data services in REST API / GRPC format, so web services can interact with. This essentially means both teams should communicate on api endpoints and communications between web and data services. Specifically, how data api would be triggered, and what data to send over, in addition to expected return response from data api. This is an over-simplification, which means some details are omitted. Should data scientists deploy models to production? Unknown — Sat, 30 Dec 2023 00:00:00 +0000 Over the years I've heard stories of data teams struggling with deploying machine learning models to production. Clearly there is a pattern here. This article is my reflection on the matter. So what's the problem?</a> </h2> Data scientists, by definition, create mathematical models from data so some unknowns can become known. This is colloquially known as "prediction." For example, if you have sales data from last year, you can use it to forecast sales performance of next year. The fun starts when data scientists have to "deploy" models to production, because their expertise do not always extend to what are required to deploy models to production. Below is a diagram of machine learning project workflow, abridged to only include steps from data team's perspective, excluding web development. The difference between data science, machine learning engineering and mlops</a> </h2> There are multiple definitions for data scientist, machine learning engineer and mlops, but a common theme for each is that: Role</th> Area of Focus</th></tr></thead> Data Scientist</td> Create mathematical models</td></tr> Machine Learning Engineer</td> Make sure mathematical models can be deployed</td></tr> MLOps</td> Deploy mathematical models and make sure the system is up and operational</td></tr> </tbody></table> As you can see, machine learning engineering is more closely aligned with software engineering, which makes sense why data scientists are struggling with model deployments, because it's a territory they are unfamiliar with. In the same vein, mlops is a subtype of devops, which utilizes different skills set than machine learning engineering. mlops main focus is to make sure the whole pipelines are running smoothly and that the systems are operational. Why a separate mlops role</a> </h2> Refer to software engineering vs devops, it's the same debate. The gist is that developing a system and monitoring it are different, plus there are many things to consider when deploying a workload, especially things that don't happen in development workflow. For one, you can't deploy an app the same way as ML api endpoint. The resources requirements are different, and packaging ML system is trickier than an app, seeing some requires GPU during inference, etc. Also monitoring is different, since you have to factor in data and model drift as well. In addition, [the inference api] / [the backend calling an inference api] have to log input/output so the team can do back testing and adjust the models post-initial deployment, seeing the nature of training / users input data do affect the model prediction. Do people who can do all these exist?</a> </h2> They do, but they are not your average engineers. If you do find them, congratulations! Setting up Postgres locally, what could go wrong? Unknown — Sat, 23 Dec 2023 00:00:00 +0000 There are multiple reasons why someone wants to set up a postgres locally. Either for learning SQL or as an application's backend. Over the years I see people struggle with using postgres locally, so here are common use cases and possible issues, with solutions for each. For Learning SQL</a> </h2> SQL is very common for analysts to use for accessing data from a database, because the data size outgrows Excel. However, SQL is a query language, not a database engine. This essentially means if you want to get familiar with SQL, there are other simpler alternatives, such as SQLite or DuckDB (which can load data from local files directly without doing an explicit data import). Plus, you don't need authentication to use either of them! This would save people a lot of headaches from fixing connection errors, which can be anything from wrong username/password, wrong host, host is not reachable, etc. As Application Backend</a> </h2> Postgres setup</a> </h3> There are many ways to set up postgres locally. You can use an official installer to install postgres natively. This is very convenient, but you can only have one version of postgres installed at the same time. However, given the version of postgres you use for your application is very important, I would recommend people to set up postgres via docker, since this allows you to choose any postgres version you want for your project, and you can reset the state of postgres easily, since you can mount the data to a host path. Networking issues</a> </h3> If you are using postgres via docker, and you are developing an application locally, don't forget to expose port 5432</code> when you start a postgres container, otherwise your application won't be able to discover postgres. You'll know it's working if you can discover this postgres via localhost</code>. But, if you are running your application as a docker container: Postgres container</th> Postgres hostname as seen from application</th></tr></thead> Expose port to 5432</code></td> host.docker.internal</code></td></tr> Don't expose port 5432</code> & run application and postgres via docker compose</code></td> $postgresServiceName</code></td></tr> </tbody></table> Reason being from a docker container's perspective, it doesn't share localhost</code> with your host system, which means localhost</code> as seen on your machine, and the one seen in docker are completely different! As for referencing a compose service name as database host, that's a docker networking feature, where each services in a single compose file can discover each other via a compose service name. Closing</a> </h2> Mostly this is networking issues, and not being familiar with docker networking model can trip most people up, hopefully this article is useful to you. Please let me know if you would like me to expand on some other use cases. Things to watch out for GCP SSL with Cloudflare DNS Unknown — Mon, 18 Dec 2023 00:00:00 +0000 For our production workload, we deploy the workloads on Kubernetes, in which an ingress resource is created per each deployment. Resources in ingress are GCP Load Balancer and SSL Certificate. As for DNS, we use Cloudflare since it enables CDN without extra configurations on our part. A few months after the deployment went live initially, we were informed that the website couldn't be accessed. Turns out GCP couldn't renew the SSL Certificate (error FAILED_NOT_VISIBLE</code>.) Looking at GCP docs, turns out if the DNS couldn't be resolved to the Load Balancer IP, it couldn't provision/renew a certificate. The fix? Disable proxy on Cloudflare records and wait until GCP successfully renews the certs. Serverless real-time machine learning inference with AWS Unknown — Tue, 28 Nov 2023 00:00:00 +0000 For a machine learning project, usually it is divided into two main categories: research and production. For research ML project, the model would be created and used locally on a researcher's machine. For a production ML project, a deployment would be involved. Usual pattern is to create a service to load a model, accept input, then return a prediction. Production ML is also divided into two main patterns: batch or real-time. For batch inference, a job would be triggered on an interval to pre-calculate predictions, then store somewhere. As for real-time inference, it is more tricky, since this involves web application architecture (at least the data and application tier). Let's assume there is a website where you can identify animals by uploading a photo. The model can expect a mandatory coordinates point to narrow down the possible list of animals. Ideally there would also be a database containing a list of animals in given geographical area. This service isn't used all the time, but during certain periods there are traffic spikes. Together, during an inference it would look like this: ML service reads input image</li> Extract coordinates from input image</li> Obtain relevant features from database</li> Obtain prediction from model</li> Return prediction</li> </ol> There are many ways to set up this ML inference architecture, but for less operational overheads, serverless offerings can be utilized. On AWS the architecture would look like this: AWS Lambda for ML service, since it automatically scales based on amount of incoming requests.</li> AWS Aurora Serverless for storing data used during feature engineering step.</li> AWS ElastiCache Serverless for caching features, to reduce lookup time during feature engineering.</li> </ul> Given the nature of sporadic service usage, but might have traffic spikes, this setup allows scaling, and you don't have to pay for compute when services are idle during off-peak hours. To elaborate, when there are a lot of concurrent requests, Lambda would scale on its own. But the bottleneck would be the database, since it doesn't always scale orthogonally to Lambda. However, every time an ML service fetches data from database, it doesn't necessarily looking for new data. In most cases, it would look up the same data because users tend to submit images taken from certain geographical areas, which can be their hometown, current location, or tourist spots. An in-memory cache such as Redis (in AWS it's ElastiCache) can be used to reduce database loads by caching frequently-accessed data. Previously the full serverless setup isn't possible because AWS didn't offer serverless in-memory database, but now it does</a>! You can also combine this with AWS ECS on Fargate for an actual application tier, which means you can fully focus on your application and not the operational overhead. And storage can be S3, in which AWS manages everything for you. Print logs here and there and send it to CloudWatch, and now you have observability (ECS and Lambda automatically log metrics!) The possibilities! Some problems can be solved with workflows Unknown — Fri, 24 Nov 2023 00:00:00 +0000 When we face with engineering problems, it's too easy to fall into the trap thinking it should be solved with a technical solution. Seasoned engineers think differently, because they realize that most of the time, it's "people" or "workflow" problems. Let me provide a few examples. Management wants analysts to use Jupyter notebook to reduce time required to create a routine report</a> </h2> Background: Most analysts are comfortable using Microsoft Excel to work with data, some can also use SQL, but it's rare for analysts to be familiar with Python.</li> Jupyter notebook is an interactive development interface for data works, since users can execute a chunk of code at a time, and render data without requiring re-running the full code.</li> </ul> Problem: every month a two-person analyst team would spend two days stitching up multiple CSV files (can be up to 60) via VLookup for a monthly report. This is because analysts have to look up information for each record, in which they use a template query and manually execute 50 queries with changed parameters. Initial solution: management wants analysts to use Jupyter notebook to reduce time spent on monthly report. Issues with proposed solution: analysts are comfortable with SQL, but have no experience with programming languages. Using Jupyter notebook can help reduce operations time, but this would require a huge effort for analysts to learn Python, and the only use case for them would be to run this report monthly. Solution: optimize SQL query to reduce operations time, this means analysts don't have to learn Python. At most, an engineer would have to come up with a helper script to generate SQL from initial input, and analysts only have to run the generated SQL and get a single file, instead of 50 files. Engineer wants product owner to upload image assets to blob storage directly to reduce operations overhead</a> </h2> Problem: a service requires image assets, which is provided by the business / marketing team. A product owner would receive new images and have to inform engineers to update respective images in a CDN bucket. Initial solution: Create a cloud IAM user with upload-only permission to allow a product owner to upload images to CDN bucket directly. Issues with proposed solution: Creating a cloud IAM account increases attack surface, and the product owner could mistakenly upload assets to the wrong path. Given the upload-only permission, they can't delete images in the wrong path, this could lead to a confusion down the line as to which images are actually used. In addition, the product owner doesn't work with cloud services on a daily basis, this would require some onboarding before the product owner is comfortable enough with cloud web console. Solution: this is a tech product, which means the product owner is familiar with some enginering tools. Upon further inquiry, turns out this product owner is familiar with git. This means we can set up a git repository to have a content of CDN bucket, the product owner just have to update the state of git repo to reflect desired state, then make a PR. After a push to master, a CI/CD would kick off to sync the git repo with CDN bucket. Using Apache Iceberg to reduce data lake operations overhead Unknown — Wed, 15 Nov 2023 00:00:00 +0000 Updated 2024-02-21 - update iceberg catalog ref url Every business generates data, some very little, some do generate ginormous amount of data. If you are familiar with the basic web application architecture, there are data, application and web tier. But it doesn't end there, because the data generated has to be analyzed for reports. A lot of organizations have analysts working on production database directly. This works fine and well, until the data they are working with is very large to the point that a single query can take half a day to process! So what do people do? This is when data engineers set up pipelines to move data from $sourceDataLocation to $destinationDataLocation. Sounds simple enough, but the data doesn't only move once! Because most often than not, data from $sourceDataLocation is not in a usable format by analysts. Some transformations have to be made, columns have to be added or joined with other related data, etc. This process is known as Extract, Transform, Load (ETL, but these days you should aim for ELTLTLTLTLTLTLTLTL.) The difference between doing a Load before a Transform is that, if your transformation tasks failed, you don't have to re-ingest the source data! At small data volume this would add more overhead, but if we are talking about 100GB then this process adds more resilience to the pipeline architecture. Since this is big data territory, the general consensus is to use the data lake model, where data are stored in a blob storage, since this means cheaper cost (compared to storing data in a database, either relational or columnar), and more throughput for writing and accessing the data. Spark is used for data processing, since it is a distributed dataframe framework. As with the nature of businesses, some changes would be made to the web application along the way, either due to more features added, or a direction shift. This means over time, there would be new data columns added, some renamed to better fit the purpose of a business. However, this does pose a few issues, namely accessing the data is not as convenient (because it involves a user specifying actual data path in blob storage), and if you want to rename or add columns, you would have to manually go through existing data and update it in the data lake manually. For orgs that have petabyte-scale of data, a single schema change can mean a few weeks of backfilling pipelines. Non-intuitive data access pattern and the need for schema evolution are very painful experiences for data engineers, and they eat up a lot of engineering time to make sure data lake operations are running smoothly. I discovered Apache Iceberg last year, was recently tinkered with it, and it solves a lot of major data lake operation headaches in following ways (there are many more, so check out their official docs!</a>) Area</th> Spark</th> Iceberg</th></tr></thead> Parquet file size</td> Have to manually optimize so that each resulting parquet would be around 128MB for optimum performance.</td> Automatically taken care of.</td></tr> Schema evolution</td> Have to re-process existing data so all files would have the same schema, otherwise it would break a .read() operation.</td> Can execute SQL DDL without requiring full table rewrite.</td></tr> Versioning</td> Have to utilize blob storage versioning, but to actually revert back to a previous state still requires a lot of configuration.</td> Can freely time-travel to the desire revision, since all writes to Iceberg are stored. This mean a table overwrite doesn't actually delete existing data, it is an append underneath, just that the presentation layer only show the latest data state.</td></tr> Partitioned table</td> Have to create a partition-key column. After writing a table, partition keys can't be updated, unless you perform a full table rewrite.</td> Can synthesize a partition column on-the-fly (ex. use timestamp column as partition key, but set it as date), with support for partition key modifications after initial table write.</td></tr> Add new table partition</td> Have to configure spark so it doesn't delete existing partitions if they are not present in your current dataframe.</td> Iceberg supports partition key in the DDL, and on table write you can specify whether it is an overwrite or overwrite on partition.</td></tr> </tbody></table> Also technically, Iceberg doesn't require a stateful component if you utilize hadoop catalog (think of a manifest of all table revisions), because the manifests are stored together with the data as normal json files. However, this means concurrent writes are not possible. To prevent this, you can select available Iceberg catalogs</a>. For a lightweight and vendor-agnostic catalog, you can use REST catalog with Postgres, and these would be only required stateful backend for Apache Iceberg setup. Did I mention that by default, Iceberg uses ztsd compression, which means smaller file size than the default snappy compression on parquet? Reduce operational costs with Terraform Unknown — Sat, 04 Nov 2023 00:00:00 +0000 Background</a> </h2> Think of websites you visit each day. Most likely they are hosted on a cloud provider such as AWS, GCP, Azure. The good news is it's very easy to create a simple deployment with a virtual machine, but for scalable and high-availability workloads, usual recommendations is to use a container-based runtime such as AWS ECS/EKS, GCP Cloud Run/GKE. These services also require more configurations than a simple VM deployment. The problem</a> </h2> Usually DevOps team would be provisioning required cloud resources through a cloud provider's web console (known as ClickOps), and if you're lucky, they write down the deployment details in a shared company documentation site, which has its own cost, especially if you require SSO. This also means that, for each deployment pattern, there needs to be a corresponding entry in the documentation. Even more, some deployments require setting up environment variables to access a database or services requiring authentication, in which a DevOps engineer would also have to specify in the documentation where the secrets/configs are stored, since environment variables are dynamic, and they are tied to a deployment environment per each service. This also poses another challenge, that a deployment steps outlined in the documentation can be out-of-date, since it requires an engineer to update the steps manually. If this keeps going on, no one would trust the documentation, which can result in a massive operation overhead (from tracking down the actual deployment steps, locate the secrets, etc) and knowledge loss if engineers leave the team. The operational costs</a> </h2> Assuming a team of DevOps consists of: 8 engineers</li> 3 deployments per engineer per day</li> $140K salary -> 1 man-minute costs 140,000 USD / 12 months / 20 days / 8 hours / 60 minutes</code> = 1.2 USD/minute</code></li> </ul> Provisioning cloud resources via a provider's web console has following costs: Setting up a shared documentation website; Confluence costs 6.05 USD * 8 engineers * 1 month</code> = 48.4 USD / month</code></li> Tracking down deployment steps, locating secrets, making sure that environment variables / secrets are correct: 3 services * 30 minutes * 8 engineers * 1.2 USD/minute * 20 days</code> = 17,280 USD / month</code></li> Updating deployment steps in the documentation: 3 services * 30 minutes * 8 engineers * 1.2 USD/minute * 20 days</code> = 17,280 USD / month</code></li> </ul> What about other costs?</a> </h2> Sometimes the finance department would inquire the cost of each service, in which a DevOps engineer would have to comb through the documentation, list cloud services used, then calculate the cost. This process can take around 15-30 minutes, and since this is done manually, some errors might occur due to human errors when obtaining resource types or looking up the costs. Assuming there are 100 services, costs are calculated each quarter, and it takes an engineer 15 minutes to calculate the cost of each service: 100 services * 15 minutes * 1 engineers * 1.2 USD/minute * 4 quarters</code> = 7,200 USD / year</code> The solution</a> </h2> Terraform</a> is an IaC (infrastructure as code) tool, which means you can declare cloud resources programmatically, and you can uses it to apply cloud configurations. It can also act as a living documentation, since terraform code directly translates to desired cloud resources. This would eliminate the need for a documentation instance, tracking down steps/configurations and updating the documentation manually. Plus, it takes almost no time to perform a cost breakdown by using Infracost</a> CLI against a terraform project. In total, you would save 48.4 USD + 17,280 USD + 17,280 USD</code> = 34,608.4 USD / month</code>. In a single year, this means 34,608.4 USD / month * 12 months</code> = 415,300.8 USD / year</code> can be saved! Very crazy! Spark on Kubernetes Unknown — Tue, 12 Sep 2023 00:00:00 +0000 Background</a> </h2> For data processing tasks, there are different ways you can go about it: using SQL to leverage a database engine to perform data transformation</li> dataframe-based frameworks such as pandas, ray, dask, polars</li> big data processing frameworks such as spark</li> </ul> Check out this article</a> for more info on polars vs spark benchmark. The problem</a> </h2> At larger data scale, other solutions (except spark) can work, but with a lot of vertical scaling, and this can get very expensive. For a comparison, our team had to scale a database to 4/16 GB and it still took the whole night, whereas spark on a single node can process the data in 2 minutes flat. Using spark would solve a lot of scaling problems, the only problem is that spark in cluster mode is very hard to set up. That's why most if not all cloud providers provide managed spark runtime (AWS EMR, GCP Dataproc, Azure Databricks). And since these are managed spark, this means you can't test it locally (and you'll have to wait at least 5 minutes for the cluster to start and finish bootstrapping.) The solution</a> </h2> As of Apache Spark 3.1 release in March 2021, spark on kubernetes is generally available. This is great in so many ways, including but not limited to: you have full control of your infra</li> takes less than 10 seconds to start a spark cluster</li> can store logs in a central location, to be viewed later via spark history server</li> can use minio as local storage backend (better throughput compared to calling S3 via home/work internet)</li> cheaper than all managed solutions, even serverless variants (more on this later)</li> </ul> Demo</a> </h2> Start a local kubernetes cluster</a> </h3> Minikube, Kind or K3d should work. k3d cluster create spark-cluster --api-port 0.0.0.0:6443 # verify that a cluster is created kubectl get nodes</code></pre>Create a namespace for spark jobs</a> </h3> kubectl create namespace spark</code></pre>Create a service account for spark</a> </h3> kubectl create serviceaccount spark --namespace spark kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=spark:spark --namespace=spark</code></pre>Install local spark (so you can run spark-submit</code>)</a> </h3> brew install temurin brew install apache-spark</code></pre>Run a demo spark job</a> </h3> If encounter To use support for EC Keys</code> error: https://stackoverflow.com/questions/75796747/spark-submit-error-to-use-support-for-ec-keys-you-must-explicitly-add-this-depe</a> # get kubernetes control plane url via `kubectl cluster-info` spark-submit \ --master k8s://https://0.0.0.0:6443 \ --deploy-mode cluster \ --name spark-pi \ --conf spark.kubernetes.namespace=spark \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ --conf spark.executor.instances=3 \ --conf spark.kubernetes.container.image=spark:3.4.1 \ local:///opt/spark/examples/src/main/python/pi.py</code></pre>Tearing down k3d</a> </h3> k3d cluster delete spark-cluster</code></pre>Full Demo</a> </h2> Regarding pricing</a> </h2> Since spark-submit on kubernetes is akin to serverless spark, we will be comparing costs with serverless spark runtime. Setup: CPU: 8</li> Memory: 16</li> Runtime: 15 Hours</li> Ephemeral storage: 20 GB</li> Region: Singapore</li> </ul> Remarks: spark on kubernetes in this article assumes you're using GKE Autopilot, which has a management fee of $60/month (unless it's your only cluster). Compute is spot. Serverless Runtime</th> Reference</th> Total Cost</th> Break even point if you run at least 33x workloads</th></tr></thead> AWS EMR Serverless</td> https://aws.amazon.com/emr/pricing/</a></td> 9.61 USD</td> 317.22 USD</td></tr> GCP Dataproc Serverless</td> https://cloud.google.com/products/calculator</a></td> 4.26 USD</td> 140.58 USD</td></tr> Azure Databricks Serverless</td> https://www.databricks.com/product/pricing/product-pricing/instance-types</a></td> 5.61 USD</td> 185.13 USD</td></tr> (Bonus) spark on kubernetes</td> https://cloud.google.com/kubernetes-engine/pricing</a></td> 2.417 USD</td> 139.76 USD (Including cluster management fee)</td></tr> </tbody></table> The script I used to find a break event point: aws = 9.61272 gcp = 4.26 databricks = 5.61 k8s = 2.417 found_break_even_point = False multiplier = 0 while not found_break_even_point: multiplier += 1 print(multiplier) aws_multiplied = aws * multiplier gcp_multiplied = gcp * multiplier databricks_multiplied = databricks * multiplier k8s_reference = (k8s * multiplier) + 60 # add gke autopilot management fee if k8s_reference < min([aws_multiplied, gcp_multiplied, databricks_multiplied]): found_break_even_point = True</code></pre>Why GKE Autopilot</a> </h3> Traditionally, when you're creating a kubernetes cluster, you'll have to attach nodes yourself. This means, if you're running spark jobs where the compute requirement exceeds the available capacity, your jobs won't be able to start. By using GKE Autopilot, GCP automatically provision nodes and attach to your GKE cluster automatically, which means you won't ever face "insufficient cpu / memory" errors. Closing</a> </h2> If you're already using kubernetes, and also use spark for data processing, migrating workloads to k8s can reduce a significant amount of cost. For our case, existing spark jobs are run on a 4vCPU/16GB VM (109.79 USD / Month</code>), and this cause a lot of dent in our bills. But if we migrate to spark on k8s, it would only cost us around16.41 USD / month</code> 😱. So that's a whopping 85% price reduction 🤯. For more advanced use cases, check out https://github.com/kahnwong/spark-on-k8s</a>. Data Engineering Resources Unknown — Sat, 09 Sep 2023 00:00:00 +0000 Note: if you've seen the list elsewhere, it was probably me. I first posted this list on Data Engineering Discord</a> and Data Engineer Cafe</a>. Books</a> </h2> Data fundamentals (good entrypoint)</a> </h3> Fundamentals of Data Engineering - Joe Reis & Matt Housley</li> Seven Databases in Seven Weeks - Luc Perkins & Eric Redmond & Jim Wilson</li> Designing Data-Intensive Applications - Martin Kleppmann</li> The Data Warehouse Toolkit - Ralph Kimball & Margy Ross</li> Data Science for Business - Foster Provost & Tom Fawcett</li> Practical Statistics for Data Scientists - Peter Gedeck & Peter Bruce & Andrew Bruce</li> </ul> Software engineering</a> </h3> Python Crash Course - Eric Matthes</li> The Pragmatic Programmer - Andrew Hunt & David Thomas</li> </ul> Platform</a> </h3> Terraform: Up & Running - Yevgeniy Brikman</li> </ul> Management</a> </h3> Team Topologies - Matthew Skelton & Manuel Pais</li> Radical Candor - Kim Scott</li> Data Teams - Jesse Anderson</li> Practical DataOps - Harvinder Atwal</li> </ul> Resources</a> </h2> https://brendanthompson.com/posts/2021/11/my-terraform-development-workflow</a></li> https://www.terraform-best-practices.com/</a></li> https://github.com/open-guides/og-aws</a></li> https://awesomedataengineering.com/</a></li> https://github.com/opendatadiscovery/awesome-data-catalogs</a></li> https://github.com/datastacktv/data-engineer-roadmap</a></li> https://www.moderndatastack.xyz/stacks</a></li> https://www.secoda.co/glossary</a></li> https://www.gentlydownthe.stream/</a></li> https://b-greve.gitbook.io/beginners-guide-to-clean-data/</a></li> </ul> A networking God tale: all I want is to run a speedtest behind a firewall Unknown — Sun, 27 Aug 2023 00:00:00 +0000 Imagine going to your client's site to deploy a software. During the deployment process, you notice that the speed is atrociously slow. You have a suspicion that your client's network bandwidth is the issue. To test this theory, you go to a speedtest website and run a test. Turns out you can't because it's blocked at the firewall level. Then you try another speedtest website, oops still got blocked. Then you try a few more, still no dice. At this point you're pretty sure that your client downloaded a hostfile blocklist and apply it to their firewall. But you need to know that the issue is at your client's network bandwidth, so you can tell them with confidence that the issue is them, not you. Stepping back a bit, a speedtest is just network performance on download and upload. So technically if you have a server that allows you to download and upload files, you can also measure the transfer speed. With this insight, you spin up a cheap VPS, then deploy a file sharing server. On your client's machine, you download a large binary from somewhere, maybe a linux ISO installer image. Then you upload it to your VPS. You write down the upload speed. Then you download the same file from the VPS, this should give you the download speed. Now you have what you need to tell your client that the issue is on their end! Maybe not really a happy ending, but at least you know your code works fine (for now). Spatial data to QGIS server playbook (yes, this is for prod) Unknown — Thu, 10 Aug 2023 00:00:00 +0000 Some of you might be familiar with geoserver for serving spatial data as consumable WMS/WFS layers. The issue is that as far as I know, you have to manually manage assets upload and specifying styles manually. Also the tool is a bit dated. One modern alternative is QGIS server, you can find pre-made docker image online, and it also syncs with the Desktop version. The good thing about QGIS server is that you can create a QGIS project via the desktop application, then upload it wholesale to Postgres instance as QGIS server backend. Assuming we're starting with raw spatial data as hardfiles, and you want to use Postgres as data store backend, and that you don't want to expose the database to public: Playbook</a> </h2> Import data to QGIS desktop.</li> Fix geometries</code> > save to Postgres.</li> Collect geometries</code> > save to Postgres. (This is for some cases where the raw data contain both Polygon</code> and MultiPolygon</code> in the same file, Postgres backend doesn't allow mixed geometry type).</li> DB Manager</code> > Import Layer/File</code>, and set desired CRS</code> + spatial index here.</li> Import uploaded layers into current QGIS project.</li> Import geoserver styles (in SLD format) in QGIS via python console, then attach to layers.</li> </ol> layer = iface.activeLayer() layer.loadSldStyle("style.sld")</code></pre> Properties</code> > set rendering</code> to 3</code>.</li> Enable WMS</code> and WFS</code> under project properties</code>.</li> Save current project as *.qgz</code>.</li> Unzip *.qgz</code>, manually replace database hostname</code> with private ip address</code>.</li> Re-zip the assets into *.qgz</code>. Important: have to use command-line on MacOS.</li> Turn on airplane mode, then open the modified QGIS project. (This is so that it won't keep reaching Postgres via private ip.)</li> Disable airplane mode, then write the project to Postgres. (Need to whitelist your home ip so that you can reach prod db behind vpc.)</li> Voila 🎉</li> </ol> Notes</a> </h2> To be sure, restart the server every time you update the project.</li> Need to use private ip for database hostname.</li> On M1, need to use QGIS installed via mamba. (Otherwise you'll have missing requirements for Fix geometries</code>).</li> </ul> Create Kubernetes service accounts with Terraform Unknown — Tue, 01 Aug 2023 00:00:00 +0000 Sometimes you'll have to grant other people (or entities) access to your Kubernetes cluster. Easiest is you can give them your admin credentials, but this is similar to giving your house key to a friend, when they only need access to your living room. You can give them different keys, depending on access level required. Those could be readonly</code> access to view services status, or deploy</code> service account that can create/update services. You can mix and match these terraform configs, but most importantly, you should read up on Kubernetes RBAC before diving in further. You've been warned! Deploy service account</a> </h2> # ------------------------ service account ------------------------ # # https://github.com/hashicorp/terraform-provider-kubernetes/issues/1943#issuecomment-1369546028 resource "kubernetes_secret" "sa_github" { metadata { annotations = { "kubernetes.io/service-account.name" = kubernetes_service_account.sa_github.metadata.0.name } namespace = "default" name = "${kubernetes_service_account.sa_github.metadata.0.name}-token" } type = "kubernetes.io/service-account-token" wait_for_service_account_token = true } resource "kubernetes_service_account" "sa_github" { metadata { name = "sa-github" namespace = "default" } } # ------------------------ cluster role ------------------------ # resource "kubernetes_cluster_role" "allow_deploy" { metadata { name = "cr-allow-deploy" } # https://stackoverflow.com/a/70669341 rule { api_groups = [""] resources = ["pods"] verbs = ["list", "get", "watch", "create", "delete"] } rule { api_groups = [""] resources = ["pods/exec"] verbs = ["create"] } rule { api_groups = [""] resources = ["pods/log"] verbs = ["get"] } rule { api_groups = [""] resources = ["pods/attach"] verbs = ["list", "get", "create", "delete", "update"] } rule { api_groups = [""] resources = ["secrets"] verbs = ["list", "get", "create", "delete", "update"] } rule { api_groups = [""] resources = ["configmaps"] verbs = ["list", "get", "create", "delete", "update"] } rule { api_groups = [""] resources = ["services"] verbs = ["list", "get", "create", "delete", "update", "patch"] } rule { api_groups = ["apps"] resources = ["deployments"] verbs = ["list", "get", "create", "delete", "update", "patch"] } } # ------------------------ cluster role binding ------------------------ # resource "kubernetes_cluster_role_binding" "allow_deploy" { # https://github.com/kubernetes/kubernetes/issues/30924#issuecomment-240887810 metadata { name = "crb-allow-deploy-default" } role_ref { api_group = "rbac.authorization.k8s.io" kind = "ClusterRole" name = kubernetes_cluster_role.allow_deploy.metadata.0.name } subject { kind = "ServiceAccount" name = kubernetes_service_account.sa_github.metadata.0.name namespace = "default" } }</code></pre>Readonly service account</a> </h2> # ------------------------ service account ------------------------ # # https://github.com/hashicorp/terraform-provider-kubernetes/issues/1943#issuecomment-1369546028 resource "kubernetes_secret" "foo" { metadata { annotations = { "kubernetes.io/service-account.name" = kubernetes_service_account.foo.metadata.0.name } namespace = "default" name = "${kubernetes_service_account.foo.metadata.0.name}-token" } type = "kubernetes.io/service-account-token" wait_for_service_account_token = true } resource "kubernetes_service_account" "foo" { metadata { name = "sa-foo" namespace = "default" } } # ------------------------ cluster role ------------------------ # resource "kubernetes_cluster_role" "readonly" { metadata { name = "cr-readonly" } rule { api_groups = [""] resources = ["namespaces", "pods", "pods/log"] verbs = ["get", "list", "watch"] } } # ------------------------ cluster role binding ------------------------ # resource "kubernetes_cluster_role_binding" "readonly" { metadata { name = "crb-readonly" } role_ref { api_group = "rbac.authorization.k8s.io" kind = "ClusterRole" name = kubernetes_cluster_role.readonly.metadata.0.name } subject { kind = "ServiceAccount" name = kubernetes_service_account.foo.metadata.0.name namespace = "default" } }</code></pre> pglogical setup Unknown — Thu, 20 Jul 2023 00:00:00 +0000 In certain cases, you can't do a full postgres replication to another instance, or you prefer a fine-grained control on what to replicate, pglogical is one way to achieve partial replication, albeit this requires more manual setup. Below are steps I used to do a pglogical replication from AWS RDS to on-premise database. Note: If a subscriber (from the above example, the on-premise database) is offline, postgres WAL would balloon up. You'll have to remove all traces of pglogical extension, including uninstalling pglogical extension, then reinitialize everything again to resolve the problem. 1. ON PROVIDER INSTANCE: Create a role for subscriber to fetch data from provider instance</a> </h2> CREATE USER $SUBSCRIBER_ROLE_NAME WITH ENCRYPTED PASSWORD $SUBSCRIBER_ROLE_PASSWORD; GRANT rds_superuser TO $SUBSCRIBER_ROLE_NAME; -- RDS specific GRANT CONNECT ON DATABASE $DATABASE_TO_BE_REPLICATED TO $SUBSCRIBER_ROLE_NAME; GRANT USAGE ON SCHEMA public TO $SUBSCRIBER_ROLE_NAME; GRANT SELECT ON ALL TABLES IN SCHEMA public TO $SUBSCRIBER_ROLE_NAME; GRANT rds_replication TO $SUBSCRIBER_ROLE_NAME; -- RDS specific</code></pre>2. ON PROVIDER INSTANCE: Init pglogical extension</a> </h2> CREATE EXTENSION pglogical; GRANT ALL ON SCHEMA pglogical TO $SUBSCRIBER_ROLE_NAME;</code></pre>3. ON PROVIDER INSTANCE: Define provider node</a> </h2> SELECT pglogical.create_node( node_name := 'provider', dsn := 'host=$PROVIDER_INSTANCE_HOSTNAME port=5432 user=$SUBSCRIBER_ROLE_NAME dbname=$DATABASE_TO_BE_REPLICATED password=$SUBSCRIBER_ROLE_PASSWORD' );</code></pre>4. ON PROVIDER INSTANCE: Define tables to be replicated</a> </h2> select pglogical.create_replication_set('replication_set'); select pglogical.replication_set_add_table( set_name := 'replication_set', relation := '$TABLE_NAME', synchronize_data := true );</code></pre>5. ON SUBSCRIBER INSTANCE: Init pglogical extension</a> </h2> CREATE EXTENSION pglogical;</code></pre>6. ON SUBSCRIBER INSTANCE: Define table schemas for replicated tables</a> </h2> pglogical does not transmit schema definition. Basically you need to do the equivalent of schema migration. create table public.$TABLE_NAME ( subject_id varchar not null, foo integer, bar varchar, primary key (id) ); -- also define indexes as well,</code></pre>7. ON SUBSCRIBER INSTANCE: Define subscriber node</a> </h2> $SUBSCRIBER_INSTANCE_HOSTNAME</code> can be localhost</code> SELECT pglogical.create_node( node_name := 'subscriber', dsn := 'host=$SUBSCRIBER_INSTANCE_HOSTNAME port=5432 user=$SUBSCRIBER_INSTANCE_ROLE_NAME dbname=$DATABASE_TO_BE_REPLICATED password=$SUBSCRIBER_INSTANCE_ROLE_PASSWORD' );</code></pre>8. ON SUBSCRIBER INSTANCE: Define subscription</a> </h2> SELECT pglogical.create_subscription( subscription_name := 'aws_sub', provider_dsn := 'host=$PROVIDER_INSTANCE_HOSTNAME port=5432 dbname=$DATABASE_TO_BE_REPLICATED user=$SUBSCRIBER_ROLE_NAME password=$SUBSCRIBER_ROLE_PASSWORD', replication_sets := ARRAY['replication_set'] );</code></pre>9. ON PROVIDER INSTANCE: Verify that pglogical is working</a> </h2> SELECT subscription_name, status FROM pglogical.show_subscription_status ();</code></pre> Book Highlights - Atlas of the Heart by Brene Brown Unknown — Thu, 06 Jul 2023 00:00:00 +0000 People will do almost anything to not feel pain, including causing pain and abusing power. Very few people can handle being held accountable without rationalizing, blaming, or shutting down. Without understanding how our feelings, thoughts, and behaviors work together, it’s almost impossible to find our way back to ourselves and each other. When we don’t understand how our emotions shape our thoughts and decisions, we become disembodied from our own experiences and disconnected from each other. Book Highlights - Build by Tony Fadell Unknown — Thu, 06 Jul 2023 00:00:00 +0000 Asshole assholes: They suck at work and everything else. These are the mean, jealous, insecure jerks who you’d avoid at a party, but who inevitably sit immediately next to you at the office. They cannot deliver, are deeply unproductive, so they do everything possible to deflect attention away from themselves. They will lie, craft gossip, and manipulate others to get people off their scent. The only good thing about these assholes is that they’re generally out the door pretty quickly—they can only deflect for so long before people start noticing that they bring zero value. And nobody likes working with them. Mission-driven "assholes": The people who are crazy passionate—and a little crazy. They speak most frankly, trampling the politics of the modern office, and steamroll right over the delicate social order of "how things are done around here." Much like true assholes, they are neither easygoing nor easy to work with. Unlike true assholes, they care. They give a damn. They listen. They work incredibly hard and push their team to be better—often against their will. They are unrelenting when they know they’re right, but are open to changing their minds and will praise other people’s efforts if they’re genuinely great. A good way to know if you’re working with a mission-driven "asshole" is to listen to the mythos around them—there are always a few choice stories floating around about some crazy thing they’ve done, and the people who’ve worked with them closely are always telling everyone that they’re not that bad, really. Most tellingly, the team ultimately trusts them, respects what they do, and looks back at the experience of working with them fondly, because they pushed the team to do the best work of their lives. It’s the assholes who are focused on people—on controlling people—who make work miserable. Real assholes always make it personal. Their motivation is their ego, not the work. As long as they’re winning, they don’t give a shit about what’s happening to the product or what the customer has to deal with. These are the assholes who make it progressively more difficult to create something you’re proud of. She rolled her eyes and completely ignored his order. But she had to deal with this guy to get her project shipped. So she did the only thing you can do when faced with a controlling asshole: Kill ’em with kindness. Ignore them. Try to get around them. Quit. Political assholes thrive in large organizations where they can pull the kind of Machiavellian BS that makes you sound crazy and paranoid when you’re describing it. They find people who aren’t exceptional at their jobs and protect them in exchange for their allegiance. They get dirt on their peers—who’s having an affair with his admin? Can we get HR to cover it up?—then those people are indebted for life. That’s when your team needs to have a counternarrative. The bullshit-asymmetry theory, Brandolini’s law, will be at play here: "The amount of energy needed to refute bullshit is an order of magnitude higher than to produce it." I’ve seen way too many people come out of the corporate world, decide to start a company, and be completely unprepared for what it takes. If they’ve never been on a small team starting from scratch, they’re often a fish out of water. They spend too much money too fast. Hire too many people. Don’t put in the time, don’t have the startup mentality, can’t make hard decisions, are buried by consensus thinking. They end up making mediocre products or nothing at all. Steve Jobs was clear about the lesson he’d learned and made sure we all learned it, too: any company that tries to do both B2B and B2C will fail. My dad was on commission but he would often sacrifice a sale in order to build a personal connection. The best salespeople are the ones who maintain relationships even if it means not making money that day. To be clear, I fully support massages. I love massages. I get them all the time. Everyone should get massages. But at no point should your company culture be formed around the idea that massages are your due. At no point should you promise employees that they’ll get massages forever. At no point should perks define your business or drag it down. Perks are frosting. High-fructose corn syrup. And nobody will begrudge you a little sugar—everyone likes sweets from time to time. But stuffing your face full of them from morning to night isn’t exactly a recipe for happiness. Just as dessert shouldn’t come before dinner, perks shouldn’t come before the mission you’re there to achieve. The mission should fill and fuel your company. The perks should be a sprinkle of sugar on top. Hassle-free Kubernetes monitoring with Coroot Unknown — Fri, 09 Jun 2023 00:00:00 +0000 Successfully deploying services are not the end, maintenance is coming to town! When you want to see how your system works, usually people rely on SaaS like Datadog or New Relic to do the heavy lifting. Also a lot of $$$ is required. Also with SaaS like these, usually you have to config your application to forward metrics/logs to your monitoring provider, this could mean a few months of engineering man-days. In contrast, what if you can install a single service that lives locally on your Kubernetes cluster, and it can provide almost everything Datadog can offer, without having to configure your applications to access the metrics/logs? Spoiler: everyone is happy and you save a lot more money! Utilizing eBPF, Coroot can access telemetry data at the Linux Kernel level. Installation is as easy as: helm repo add coroot https://coroot.github.io/helm-charts helm repo update helm install --namespace coroot --create-namespace coroot coroot/coroot</code></pre> And access via http://localhost:8080</a> kubectl port-forward -n coroot service/coroot 8080:8080</code></pre> Most of my services are on Kubernetes installed via k3s</code>. Since I am installing Coroot on my home server, only a single node is reported, and it's mostly stay idle because I'm the only user. When you access each service, it would display overview status for each observability metrics. Hello four golden signals! Under tracing</code>, you can see which routes are being requested and you can see that Coroot is tailor-made for SRE disciplines because they also provide a quick-access button to see error traces</code> and latency SLO violations</code>. You can tinker much more with Coroot, so head over to their official website</a> and let me know how it works with your setup! Book Highlights - Managing Transitions: Making the Most of Change by William Bridges Unknown — Thu, 27 Apr 2023 00:00:00 +0000 It isn’t the changes themselves that the people in these cases resist. It’s the losses and endings that they have experienced and the transition that they are resisting. A large university reassigned one of its vice presidents to a far less important area than the one he had previously headed, and although no one called it a demotion, it was hard to see it as anything else. Everyone knew that he had been ineffective in his previous job, and his new job actually fit his talents far better, but he was deeply hurt by the move. Discussing the situation, we discovered that the man was less troubled by the fact of the move than by how he thought it would be perceived by his colleagues. “I told the supervisors. It’s their job to tell their teams.” The supervisors are likely to be in transition themselves, and they may not even sufficiently understand the information to convey it accurately. Maybe they’re still in denial. Information is power, so they may not want to share it yet. Don’t assume that accurate information trickles down reliably. There may be times when information must be withheld temporarily. The Securities and Exchange Commission (SEC) may require it, for example, or you may not be able to talk about a strategic move because competitors will learn of it. But most of the time information is withheld because leaders or managers are uncomfortable giving it. That discomfort often arises not from the anticipated long-term effects but simply from the short-term impact—the setting off of the “grieving” emotions discussed earlier. Self-absorption. Anxious people become preoccupied with their own situations and lose their concern for fellow workers or customers. In a game of musical chairs, the only real questions are, “When is the music going to stop?” and “Will there be a chair left for me?” Larger issues of teamwork, good service, and high quality get lost. Inspirational pep talks on the values of teamwork, good service, and high quality don’t do much good when people are self-absorbed. Use SQL against CSV (or other hard files) without CLI Unknown — Tue, 25 Apr 2023 00:00:00 +0000 CSV as a file format is very versatile, almost any programs can parse it. The only issue is you can't use SQL against CSV files directly. This is a major pain point, since using SQL is so much faster than firing up a jupyter notebook and wrangle the data in python, or use Excel and apply transformations until you get desired results. But the question is how do we use SQL against CSV files in the first place. Many people want this to become a reality, so a few tools exist on github: csvq</a></li> q</a></li> trdsql</a></li> </ul> Only issue is these tools require you to use CLI interface. Your analysts would be screaming at the top of their lungs because they are afraid of a black box with a blinking cursor 😱. But you can also standing up a postgres database, and import data as a table and use SQL against it, but guess how many analysts want to learn how to spin up a database and config a database client to connect to a database instance? Interestingly enough, you can use in-memory DuckDB</a> against local files, and the best part about this is you can use GUI-based database IDE! I only tested against DBeaver</a> and DataGrip</a>, but any tools that use JDBC</code> driver to talk to DuckDB should also works. As for JDBC URI</code>, set it as jdbc:duckdb::memory:</code> DBeaver: DataGrip: The whole process takes less than 5 minutes. Give it a try and let me know how it goes! DevX starts at your local machine Unknown — Sat, 22 Apr 2023 00:00:00 +0000 Platform engineering is all the rage these days. Often, you'll often hear this term with the keyword DevX</code>. How are they related? Imagine you are working on a microservice backend. You are just starting out, so you don't have much features to work on yet. But as a PoC, you only need to [fetch data] and [return aggregated price]. You can do microservices on Kubernetes, but you are not familiar with DevOps so you turn to a cloud provider - AWS. AWS has a service known as Lambda, a serverless computing platform. This basically means "hey we can run your code, and you don't have to manage the infra at all." You go check it out, and you are very pleased to see that Lambda supports a lot of runtime. The honeymoon period</a> </h2> You go through the setup wizard, at the end you see a Lambda function goes live, and it also gives you a code editor! You really love this, because you don't have to go through the edit code > zip > upload to Lambda > test on Lambda</code> loop for eternity. You add more lines of code, getting closer to your end product. Then you hit a block because a calculation function you need to use is very hard to do manually, you need to use a library for that. You go looking for a library, and fortune is on your side because you found it. Then you froze, because you just realize that you can't install packages in Lambda, since it doesn't give you a terminal console 😱. You are a web developer, so you are very familiar with nodejs. Luckily, this is a language you chose for this api, and you also remember that you can just pack node_modules</code> folder in your deployment zip image and upload it to Lambda 🎉. It's almost the end of day, and you are almost done with an MVP. Then you got a call from your project manager, who informed you that you need to add a few features before they can show it to the client. You are not very happy about it, but you understand since you can't go forward with unfinished MVP. You don't have a lot of time on your hand, since the feature requests were at the last minute. You know this will be a tech debt, but given the time frame, it's the best course of action, so you decide to add more features to existing Lambda function. These features, it's quite interesting, because they do completely different things, requiring different libraries. You installed a few more packages locally, so you can init new environment on Lambda. You do your usual workflow: zip > upload > test</code>, but this time it wouldn't deploy! It says the package size is too large. But you can still upload the zip package to s3 and submit it as a deployment artifact on Lambda, so you're good for now. You can tell that this project is going to be a success! Reality hits</a> </h2> This product did indeed become a success, and you have a few people to help you out on this project. Suddenly you have a realization that "wait the existing zip > upload > test</code>" isn't going to work, because you can't collaborate 😥. DevX to the rescue</a> </h2> You remember you have a friend who's an ops wizard, she told you of a magical tool known as Docker - a ship shipping ship shipping ship 🐳. You wine and dined your friend, and she spilled this workflow setup: Develop your app like you would locally</li> Define Dockerfile</code> for deployment, where the base image is FROM public.ecr.aws/lambda/python:3.9</code>, or other runtime of your choice.</li> Setup CI/CD to build image to ECR and deploy to Lambda</li> </ol> She also told you that given this setup, you literally don't have to touch the AWS console, and you can also emulate Lambda via docker as a local endpoint and test it with Postman. Platform engineering enters the chat</a> </h2> Your friend's secret sauce works well, but over time you have more teams and services to cover. You also notice that a lot of steps are boilerplates. Your friend was also facing similar issues with her own team. During your weekend hangout, she told you that there's a way out of this: to provide a [base repository for serverless service], in which it contains boilerplate project init, and [base docker image] with common tools installed, so you don't have to waste CI minutes and rebuild it from scratch every time. Happy ending I think 🤠. The mythical ChatOps in action Unknown — Tue, 18 Apr 2023 00:00:00 +0000 Imagine having multiple services running, each has its own logs. Most people don't read them, and they shouldn't, because services emit a lot of logs! But we need them, because it's the only way to diagnose and troubleshoot system errors. But you might say "my service is not a system! It's only doing tiny stuff!" Gotta break it to you, your small part is a piece in a large system networks stitched together! So your seemingly-tiny service is also important! So how do people monitor logs? Actually they don't. To actually do that, you need to do a lot of mouse-scrolling, and reading a lot of redundant logs but with different identifiers - which doesn't sound glamorous. What usually happens is, people set up alerts on logs, so when alerts are triggered, someone would be going through the logs to investigate the issue. Now it's more like a murder mystery! What if it's not with applications logs but services status? Say, your service deployments on AWS ECS. Luckily you can still utilize the same concept as above: intercept the logs and sent out alerts based on a trigger. One instance where you might want to send alerts on services status is for services autoscaling, so you know when there's a spike in traffic. The issue isn't with the increased cost for your compute, rather to monitor related services to make sure they are operational. Hello databases, api endpoints, etc. This is important because, your service on autoscaling might be your backend. You can have multiple instances of backend services, but they might be pointing to the same database or api endpoints. Service health on auxiliary services are important, since this is your backend service's bottleneck. If you're on AWS, you can utilize Cloudwatch Events to send out messages based on a trigger via Lambda. Basically: ECS emits metrics / logs to Cloudwatch</li> Alerts (Cloudwatch Alarms) are set to trigger on specific events (e.g. service {scale up | scale down | stop})</li> Alert events are sent to SNS, so it can trigger actions we want after receiving alerts</li> Actions are defined via a Lambda function to send out messages</li> </ol> There's a lot of jumps there, so to recap: ECS -> Cloudwatch Alarms -> SNS -> Lambda. So what about ChatOps? Let's say the whole point of this setup is to send out alerts, where do you think people would have the most chance to see? Yup, in their work slack, or discord, or whatever people use. This way, service alerts are public, and everyone can see it. It's even better than email because it's not really made for real-time communication... Which means for critical alerts we might want to work with latest alerts, and collaborate to help mitigate the issue. And one of the biggest benefits is that you don't have to seek out the alerts, instead they come to you! (Now you can't use excuses like "I forgot to monitor the alerts!") For my organization's workspace, we have set up alerts for ECS services status, to see when scaling or service termination happens. This especially comes in handy during a new service deployment, when if we see service termination alerts for the same service in succession, it means there's issues with this deployment, this limits the blast radius so we can pinpoint the likely root cause. One time, someone merged a PR and suddenly we see a lot of service termination alerts. Turns out, this commit introduced a new feature that requires a new environment variable, but it wasn't added in the deployment specs yet. This also allows for fast feedback cycle, so you have an easier to time to trace your steps for likely event(s) that caused a service disruption. Ops is easy, said no one, I think 😅. DuckDB vs Polars vs Spark! Unknown — Fri, 07 Apr 2023 00:00:00 +0000 I think everyone who has worked with data, in any role or function, used pandas 🐼 at certain point. I first used pandas in 2017, so it's 6 years already. Things have come a long way, and so is data size I'm working with! Pandas has its own issues, namely no native support for nested schema. In addition, it's very heavy-handed regarding data types inference. It can be a blessing, but it's a bane for data engineering work, where you have to make sure that your data conforms to agreed-upon schema (hello data contracts!). But the worst issue? Pandas can't open data that doesn't fit into memory. So if you have a 16 GB RAM machine, you can't read 12GB data with pandas 😭. Rust is all the rage these days, so it's only natural that a dataframe framework would also exist in rust. Polars 🐻‍❄️ is a new kid in town, and it does pack a punch 🥊! Under the hood it uses apache arrow format, so there's no extra overhead for serialization/deserialization. Still, polars isn't a distributed framework. And why is this an issue? Because at certain point, your data would grow larger and larger, and it would no longer be economical to add more resources and hope</del> pray that your polars job won't fail. But what if your analysts are only comfortable with sql? Then there's DuckDB where it's like SQLite, but on steroids! But then this also pose the same issue as polars, that it doesn't work in a distributed manner. And we're back to spark, where it's not that easy to setup, but if you get it to work, you pretty much don't have to worry about scaling, since you can have a cluster of spark working in tandem to crunch your pipelines ✨. Benchmark result</a> </h2> Enough talking, here's the benchmark: If you see a small green bar chart for duckdb, the reason there's no values at > 30M rows is because it's been running for 14 minutes, and I would guess it would take much longer to finish the task. For comparison, a spark job against 270M rows only take 400 seconds / 60 seconds = 6.5 minutes</code>. Also, notice that the swap usage for spark is almost constant, whereas with polars, it spikes up exponentially post 120M rows 😱. Interestingly, it also parallels spark job runtime. But most pipelines are run in batches at night, so compute time might not be as important in relation to required compute. Experiment design</a> </h2> Contrast with most dataframe benchmarks, I intentionally abuse each framework by: Create a timestamp diff column trip_length_minute</code>, converted to minute</li> Create percentile on trip_length_minute</code> as trip_length_minute_percentile</code></li> Filter only trip_length_minute_percentile</code> between (0.2, 0.8)</li> Group by on VendorID</code>, payment_type</code></li> Aggregate min, max, avg on passenger_count</code>, trip_distance</code>, total_amount</code></li> </ol> Input data is equally partitioned into 8 chunks. Repo is here</a> if you want run it against different parameters. Please drop me a chart if you see something interesting! Update 2023-04-10</a> </h2> What if you Window on multi-key partitions? Spark wins! Makes sense, since spark can do parallel processing on each partition independently of each other. Also apparently a certain polar bear is very hungry! Cost optimization with Kubernetes Unknown — Sat, 01 Apr 2023 00:00:00 +0000 Correction 2023-07-02: fix homelab specs and corresponding AWS EC2 instance class (it's actually 32GB RAM, not 64GB) Congratulations, you managed to successfully deployed a few services on kubernetes! But this is not the end 👀. Unfortunately money doesn't grow on trees, and if you can't justify your infra expenses, finance department won't be happy. If you're using Terraform, you can use Infracost</a> to create a cost report. Pretty nifty. But what about kubernetes? Given cost reporting is a basic feature, kubernetes is no exception. Enter OpenCost</a>, a vendor-neutral open source project for measuring and allocating infrastructure and container costs in real time. This also means it is vendor-agnostic. Meaning as long as it's kubernetes, it would work regardless of which cloud it's on (and it works with on-prem setup as well 😉). Under the hood, it utilizes [prometheus metrics], in conjunction with [provisioned resources for containers - defined in deployment manifest], to calculate cost based on [cloud pricing per compute unit - you can adjust this later]. You can follow OpenCost install instructions here</a>. Wait for a few days for prometheus to collect usage metrics, and check out OpenCost dashboard. This is what mine looks like: Notice efficiency</code> column, this tells how well your resources are being utilized. If you see a low number here, consider using function-as-a-service or other cloud compute where pricing is calculated per usage, not active time (something like gcp's cloud run). Boring math part 😴</a> </h2> So that's $35.57 / 6 days</code>, which would be around $5.9 / day</code>. Per year this would cost $5.9 * 52 = $306.8 / year</code> 😱. My current home server setup is around $880</code>, so that's around 3 years</code> before it would break even 🤣. But what if we're talking about raw compute price? My server is 8 cores, 32GB RAM, 1TB SSD</code>, cheapest compute on AWS with similar specs per year is t3a.2xlarge</code>, which is $3307 / year</code>. Pricing is hard... Kubernetes with Grafana Cloud Unknown — Sat, 01 Apr 2023 00:00:00 +0000 Kubernetes is awesome, I think this is obvious if you have more than a handful of services to manage. If you use cloud, either VM or container-based runtime, it would provide you a dashboard to see the metrics. But what about kubernetes? Since you would have multiple services inside a single cluster, in which it's backed by VMs, at best you would only see into your VMs' metrics, but doesn't provide separate metrics per each service. In the end, what metrics dashboards are is just visualizations on collected metrics, which are stored somewhere. On open source front, usually people store metrics in prometheus, and visualize it via grafana. To try this out, you can use this</a> to install grafana+prometheus on your kubernetes cluster. Just port forward the grafana pod and you should be able to see the metrics. But there's a catch: if you were to tear this down, all the collected metrics are gone. This might be ideal if you're in the development phase, where you want to make sure things are actually reproducible. But for prod this is a liability. There is a way of out this though - use managed grafana (via grafana cloud</a>), and ship your cluster metrics to it via grafana agent</a>. If you follow the instructions successfully, head to your grafana cloud dashboard, and you should see your cluster metrics pop up: And the best part is it's the same dashboards you get from installing kube-prometheus-stack</code> above! And you can access the dashboard from anywhere as well, instead of doing a port forward every time you want to see the dashboard. Bare metal works, until it doesn't. Hello, cloud. Unknown — Fri, 24 Mar 2023 00:00:00 +0000 Background</a> </h2> Ever wonder how websites (and everything in between) work? Chances are you can create a project running on your local machine. It works as you expected, but to let other people access it, you have to "deploy" it. For many years, to support a lot of request volumes you need to run your applications in a data center. These days this setup is known as on-premise. Architecture</a> </h2> Let's take a look at a simple e-commerce website architecture. This would involve frontend, backend, database. But if you have a feature where you allow users to batch update catalogs, you have to add redis / pub-sub as not to overload your backend. This continues to work, except your website grew so popular that your cpu and memory usage is always at max capacity. You tried vertical scaling, but it's gotten to the point where it's no longer feasible for your budget. Oops. But wait, you can use a load balancer to spread requests throughout the nodes. This would work nicely, but it takes a lot of time to set it up properly. But you figured it out, so yay. And you just happened to secured a major deal for your website, and they want a few customized features. You crank out a lot of features, but because it takes quite a while to deploy (manually) you made a lot of mistakes along the way, and you wish you can go back to release-once-a-month cycle. Cloud enters the chat</a> </h2> All these challenges are well known among on-premise users, and there are ways to overcome these. But using cloud reduce ops overhead and allows for faster time-to-market. Let's break them down one-by-one to see how cloud can help: 1. Frontend, backend, database, etc</a> </h3> If you have to develop against local resources, you would need to spin up each component separately, but this means more context to management. Using cloud means you can develop against some resources that were already deployed on cloud. For example, you can iterate on a UI change against a development instance of backend and database, so you can focus on developing the UI instead of worrying about setting up other components properly. 2. Scaling</a> </h3> It is a well-known fact with on-prem setup that you can't scale beyond your physical capacity, this essentially means if one fateful day, your marketing campaign works really well that a lot of people use your website, your traffic would spike up, in turn increasing resources consumption. If you're on-prem, your options are limited to either waiting it out or disable your website temporarily. But this would be catastrophic for your website's reputation. If you use cloud, there are many ways to set up (auto) scaling, namely using container-based runtime and link running instances to a load balancer to spread out traffic. You can also temporarily increase your compute's provisioned resources during peak hours, then scale down later for cost optimization. On AWS, you can use ECS for compute and define a load balancer. On GCP, Cloud Run can achieve the same thing. 3. CI/CD</a> </h3> From my experience and a lot of people I've talked to, deploying with on-prem setup usually means "copy deploy artifacts over to the server (manually)." Obviously this process is prone to human errors. You can set up Jenkins to achieve CI/CD, but this requires more overhead than using a cloud-managed build solutions. With AWS, you can sync a repo with AWS Code Build - for building deploy artifacts, and deploy it via Code Pipeline. On GCP, similar services also available as well. Additionally, you can also do all this with your VCS providers (GitHub, Gitlab, etc) if you prefer it. Personally I would do CI/CD with via VCS providers. It requires more ops overhead, but yields better observability. Closing</a> </h2> A lot of people ask me which cloud to use. My answer would be: "use what other people around you use, so you can ask them when you have issues." Some clouds do certain things better than other clouds, that's up for you to evaluate. But if you're just starting out and don't have people who you can ask for support, AWS would be a better choice since it has significantly more users than other cloud providers. GCP is also good, but there's not as many users, which means unless you can understand official documentations, you're pretty much on your own. Also, I did not mention Kubernetes. This is because unless you have a lot of services to manage or an ops team to support, Kubernetes' ops overhead would be too much for your scale. And you can definitely achieve what you can do with cloud on on-premise system. But do you have enough people to setup and maintain it? Google Analytics v4 ingestion via BigQuery Unknown — Sun, 19 Mar 2023 00:00:00 +0000 Background</a> </h2> You want to track who access your site, Google Analytics can do that. To see the data, you can use Google Analytics dashboard, the default settings is good enough for most use cases. But what if you have a lot of tracking data, and you want to streamline a way to analyze it? You could use Data Studio for this, so it's cool for the moment. But what if you want to use Google Analytics data in conjunction with other data? Say, product SKUs? There are multiple ways to achieve that, but one approach that provides a lot of flexibility and allow for long-term maintenance is to write a custom pipeline yourself. So we're back to grabbing the tracking data and store it somewhere. Google Analytics provide APIs that you can call to obtain the data. This works, with a few caveats... One being that if you have a lot of data, you would run into API rate-limiting 🥲. But you probably are not the only one who amassed a lot of tracking data, and luckily there's a solution to this problem: dunk Google Analytics data into BigQuery, then you can use Spark to ingest it ✨. (You can set it to "streaming" mode as well, if you want near real-time data.) Setup</a></a> </h2> Create a new project in GCP console.</li> Enable BigQuery API</code>.</li> Go to Google Analytics console</a> > admin > product links > BigQuery links.</li> Click Link</code> > select your BigQuery project where you want to store Google Analytics data.</li> Fill in information as required.</li> Wait for a day. (Initial setup only.)</li> Head back to BigQuery console</a> and you should see the results.</li> </ol> Ingestion</a> </h2> You can use any tools you want to read data off BigQuery. Personally I would go with Spark</code>, because you can do fine-grained ingestion control that way, and it plays nicely with s3a</code> or gs</code> protocol for write destination. spark = ( SparkSession.builder.config("spark.executor.memory", "4g") .config("spark.driver.memory", "4g") .config( "spark.jars.packages", "com.google.cloud.spark:spark-bigquery-with-dependencies_2.12:0.29.0", ) .getOrCreate() ) df = ( spark.read.format("bigquery") .option("table", "ga4-to-bigquery-xxxxxx.analytics_xxxxxxxxx.events_20230318") .load() )</code></pre> Now you'll get data that looks like this: -RECORD 0--------------------------------------------- event_date | 20230318 event_timestamp | 1679110653461084 event_name | session_start event_params | [{engaged_session... event_previous_timestamp | null event_value_in_usd | null event_bundle_sequence_id | 1779024476 event_server_timestamp_offset | null user_id | null user_pseudo_id | 653547550.1673451049 privacy_info | {null, null, No} user_properties | [] user_first_touch_timestamp | 1673451049581992 user_ltv | {0.0, USD} device | {mobile, Samsung,... geo | {Asia, Thailand, ... app_info | null traffic_source | {(referral), refe... stream_id | 4111827120 platform | WEB event_dimensions | null ecommerce | null items | [] -RECORD 1--------------------------------------------- event_date | 20230318 event_timestamp | 1679110653461084 event_name | page_view event_params | [{medium, {referr... event_previous_timestamp | null event_value_in_usd | null event_bundle_sequence_id | 1779024476 event_server_timestamp_offset | null user_id | null user_pseudo_id | 653547550.1673451049 privacy_info | {null, null, No} user_properties | [] user_first_touch_timestamp | 1673451049581992 user_ltv | {0.0, USD} device | {mobile, Samsung,... geo | {Asia, Thailand, ... app_info | null traffic_source | {(referral), refe... stream_id | 4111827120 platform | WEB event_dimensions | null ecommerce | null items | [] -RECORD 2--------------------------------------------- event_date | 20230318 event_timestamp | 1679110659760543 event_name | scroll event_params | [{campaign, {(ref... event_previous_timestamp | null event_value_in_usd | null event_bundle_sequence_id | 1785323935 event_server_timestamp_offset | null user_id | null user_pseudo_id | 653547550.1673451049 privacy_info | {null, null, No} user_properties | [] user_first_touch_timestamp | 1673451049581992 user_ltv | {0.0, USD} device | {mobile, Samsung,... geo | {Asia, Thailand, ... app_info | null traffic_source | {(referral), refe... stream_id | 4111827120 platform | WEB event_dimensions | null ecommerce | null items | []</code></pre> And nice thing about spark is that it supports nested schema! root |-- event_date: string (nullable = true) |-- event_timestamp: long (nullable = true) |-- event_name: string (nullable = true) |-- event_params: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- key: string (nullable = true) | | |-- value: struct (nullable = true) | | | |-- string_value: string (nullable = true) | | | |-- int_value: long (nullable = true) | | | |-- float_value: double (nullable = true) | | | |-- double_value: double (nullable = true) |-- event_previous_timestamp: long (nullable = true) |-- event_value_in_usd: double (nullable = true) |-- event_bundle_sequence_id: long (nullable = true) |-- event_server_timestamp_offset: long (nullable = true) |-- user_id: string (nullable = true) |-- user_pseudo_id: string (nullable = true) |-- privacy_info: struct (nullable = true) | |-- analytics_storage: string (nullable = true) | |-- ads_storage: string (nullable = true) | |-- uses_transient_token: string (nullable = true) |-- user_properties: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- key: string (nullable = true) | | |-- value: struct (nullable = true) | | | |-- string_value: string (nullable = true) | | | |-- int_value: long (nullable = true) | | | |-- float_value: double (nullable = true) | | | |-- double_value: double (nullable = true) | | | |-- set_timestamp_micros: long (nullable = true) |-- user_first_touch_timestamp: long (nullable = true) |-- user_ltv: struct (nullable = true) | |-- revenue: double (nullable = true) | |-- currency: string (nullable = true) |-- device: struct (nullable = true) | |-- category: string (nullable = true) | |-- mobile_brand_name: string (nullable = true) | |-- mobile_model_name: string (nullable = true) | |-- mobile_marketing_name: string (nullable = true) | |-- mobile_os_hardware_model: string (nullable = true) | |-- operating_system: string (nullable = true) | |-- operating_system_version: string (nullable = true) | |-- vendor_id: string (nullable = true) | |-- advertising_id: string (nullable = true) | |-- language: string (nullable = true) | |-- is_limited_ad_tracking: string (nullable = true) | |-- time_zone_offset_seconds: long (nullable = true) | |-- browser: string (nullable = true) | |-- browser_version: string (nullable = true) | |-- web_info: struct (nullable = true) | | |-- browser: string (nullable = true) | | |-- browser_version: string (nullable = true) | | |-- hostname: string (nullable = true) |-- geo: struct (nullable = true) | |-- continent: string (nullable = true) | |-- country: string (nullable = true) | |-- region: string (nullable = true) | |-- city: string (nullable = true) | |-- sub_continent: string (nullable = true) | |-- metro: string (nullable = true) |-- app_info: struct (nullable = true) | |-- id: string (nullable = true) | |-- version: string (nullable = true) | |-- install_store: string (nullable = true) | |-- firebase_app_id: string (nullable = true) | |-- install_source: string (nullable = true) |-- traffic_source: struct (nullable = true) | |-- name: string (nullable = true) | |-- medium: string (nullable = true) | |-- source: string (nullable = true) |-- stream_id: string (nullable = true) |-- platform: string (nullable = true) |-- event_dimensions: struct (nullable = true) | |-- hostname: string (nullable = true) |-- ecommerce: struct (nullable = true) | |-- total_item_quantity: long (nullable = true) | |-- purchase_revenue_in_usd: double (nullable = true) | |-- purchase_revenue: double (nullable = true) | |-- refund_value_in_usd: double (nullable = true) | |-- refund_value: double (nullable = true) | |-- shipping_value_in_usd: double (nullable = true) | |-- shipping_value: double (nullable = true) | |-- tax_value_in_usd: double (nullable = true) | |-- tax_value: double (nullable = true) | |-- unique_items: long (nullable = true) | |-- transaction_id: string (nullable = true) |-- items: array (nullable = true) | |-- element: struct (containsNull = true) | | |-- item_id: string (nullable = true) | | |-- item_name: string (nullable = true) | | |-- item_brand: string (nullable = true) | | |-- item_variant: string (nullable = true) | | |-- item_category: string (nullable = true) | | |-- item_category2: string (nullable = true) | | |-- item_category3: string (nullable = true) | | |-- item_category4: string (nullable = true) | | |-- item_category5: string (nullable = true) | | |-- price_in_usd: double (nullable = true) | | |-- price: double (nullable = true) | | |-- quantity: long (nullable = true) | | |-- item_revenue_in_usd: double (nullable = true) | | |-- item_revenue: double (nullable = true) | | |-- item_refund_in_usd: double (nullable = true) | | |-- item_refund: double (nullable = true) | | |-- coupon: string (nullable = true) | | |-- affiliation: string (nullable = true) | | |-- location_id: string (nullable = true) | | |-- item_list_id: string (nullable = true) | | |-- item_list_name: string (nullable = true) | | |-- item_list_index: string (nullable = true) | | |-- promotion_id: string (nullable = true) | | |-- promotion_name: string (nullable = true) | | |-- creative_name: string (nullable = true) | | |-- creative_slot: string (nullable = true)</code></pre> Neato! SecretOps with Teller Unknown — Sun, 19 Mar 2023 00:00:00 +0000 Raise your hands if you normally have to send .env</code> files to your team members so they can start a project in dev environment. While there is nothing wrong with this approach, it could introduce a lot of security risks, namely sharing secrets via plaintext protocol. Sure, you can share them on Slack, Discord, etc. But unless it's encrypted with your own keys, it could be leaked if the said communication platform were to be breached. You could uses a one-time pastebin to share the secrets. This would be a step up, but you still have to manually send each team member the secrets. Some might say there are services like https://keybase.io</a> where you can send messages encrypted with PGP key. Yay for security, but it has the same issue as above. You use cloud, right? So most if not all cloud providers provide their own secrets management solution, so let's use that. But then developers have to somehow fetch the secrets, and set it as environment variable so they can start a project. This would work until you update the dev secrets, and some developers would eventually forget to pull a new set of secrets before starting a project. This could result in half a day of wack-a-mole hunt for why it's not working on their machine, but works on their friends'. Or you don't, if you fetch secrets during a project initialization. But the twelve-factor-app</a> would like a word. I stumbled upon teller</a> accidentally, but it seems to be able to do what I want: abstracting away secrets fetching and populate it as environment variable in one command. Setting up is as easy as: brew tap spectralops/tap && brew install teller</code>. To populate secrets as environment variables: eval "$(teller sh)"</code> I have a repo</a> here you can tinker with. Please let me know if you find some quirks, since I plan to use this with my team next year 🎊. Data transformation - Python vs SQL showdown Unknown — Sat, 18 Mar 2023 00:00:00 +0000 For most people, using SQL to transform data is a no-brainer, seeing it's a very versatile language, and doesn't have quite a steep learning curve compared to python. There are some cases where SQL is more suitable for a task, but the reverse can also happen as well. For instance, given a string conversion problem: if a string occurs only one time, replace it with #</code></li> if a string occurs multiple times, replace it with &</code></li> </ul> > one ### > three ###&& > Heartbreak hotel &&&&&#&&&##&#&&#</code></pre> A solution in python would be: from collections import Counter s = "three" s_counter = Counter(s) output_str = "" for char in s: if s_counter[char] > 1: output_str += "&" elif s_counter[char] == 1: output_str += "#"</code></pre> But a solution in SQL is...(thanks Emily @data-engineering-discord!): CREATE TABLE data (string_value TEXT); INSERT INTO data VALUES ('one'), ('three'), ('Heartbreak hotel'); select string_value, translate( lower(string_value), string_agg ( chr, '' order by chr ), string_agg ( subs, '' order by chr ) ) from ( select string_value, chr, case when count = 1 then '#' else '&' end as subs from ( select p.*, count(*) from ( select string_value, regexp_split_to_table (lower(string_value), '') as chr from data ) as p group by 1, 2 ) as q ) as r group by 1</code></pre> 😱😱😱😱😱😱😱😱😱 Terraform RDS module with DNS setup Unknown — Fri, 17 Mar 2023 00:00:00 +0000 I love not having to manage databases. Hosting it on your compute is guaranteed to be cheaper, but I don't want to be constantly worrying about backups and database upgrade / maintenance. AWS offers managed databases, known as RDS. Mostly I use postgres, which works well for most use cases. (It's also a given that - if you don't really need NoSQL database, then don't use it). If I spin up RDS postgres, I can reach it assuming I have set up the correct networking pathways. (And you shouldn't expose your database to public, unless it's for playground with no sensitive data.) RDS itself returns a hostname with following format: mydb.123456789012.us-east-1.rds.amazonaws.com</code>. This works well for all intent and purposes, but there are some cases where you might want it to be easy to remember, or that you want to switch database instance without incurring downtime. A simple solution is to setup a DNS record, but it would be nice if you can set it all up during database creation! Enough intro, below is the terraform code (and yes, this works for a replica as well): resource "aws_db_instance" "db" { engine = var.engine allocated_storage = var.allocated_storage engine_version = var.engine_version instance_class = var.instance_class identifier = var.identifier db_name = var.db_name username = var.username password = var.password replicate_source_db = var.source_db_id # for replica parameter_group_name = var.parameter_group_name performance_insights_enabled = true performance_insights_kms_key_id = var.performance_insights_kms_key_id deletion_protection = var.deletion_protection ### networking vpc_security_group_ids = var.vpc_security_group_ids db_subnet_group_name = var.db_subnet_group_name ### backup (in UTC) maintenance_window = var.maintenance_window backup_window = var.backup_window backup_retention_period = var.backup_window != null ? 3 : 0 # Backups are required in order to create a replica skip_final_snapshot = var.skip_final_snapshot apply_immediately = var.apply_immediately } resource "aws_route53_record" "db" { zone_id = var.zone_id name = var.dns_name type = "CNAME" ttl = 5 records = [aws_db_instance.db.address] }</code></pre> Book Highlights - Engineering Management for the Rest of Us by Sarah Drasner Unknown — Thu, 16 Mar 2023 00:00:00 +0000 Organizational health lies in being able to work together. As a manager, our job is to try as much as we can to drive balance and clarity. A happy, driven team sees the wider purpose of their work and also feels empowered to execute tasks individually. “You can’t call yourself a leader by coming into a situation that is by nature uncertain, ambiguous—and create confusion. You have to create clarity where none exists.” —Satya Nadella, CEO of Microsoft I have never seen employees more demoralized than when they’re unsure where their career is headed and whether their title or compensation is fair. It’s frustrating, exhausting, and can lead to burnout. It’s also incredibly distracting. Who can get their job done when they have no clue if what they’re doing is valued? Our collective aim as managers should be taking the careers of our employees as seriously as we do the team’s technical/work processes. If you’re strong and care about your team as people, it can feel unnatural to teach someone to advocate for themselves instead of moving things out of their way. The first part of building healthy teams is admitting you won’t know everything, and this may be a time where you need to listen more than you talk. The culture of your team is only as strong as the worst behavior it tolerates. It’s your job to speak up. It’s best if a manager learns an employee’s Ultimate Goals early on. Where do they see themselves in five years? What kind of work do they like to do most? What environments do they work in best and which ones are the most difficult? A manager can’t always facilitate the ideal situation, but having this information is still extremely valuable for cultivating a person’s career trajectory, for the work that needs to be done, and for a general understanding of what will keep people working well together. Career ladders alone will not create clarity—as managers, we need to put them into action. On career ladders: To get to “senior,” you’re the best “you” you can be — you perform your role exceedingly well, and you’ve reached a high potential for your own output.</li> To get to “staff,” your focus is really to expand beyond yourself. You start teaching people the great things you’ve learned and help serve their needs.</li> To get to “principal,” you’re creating systems that scale beyond yourself. You’re no longer helping folks operate like you — you’re helping them where they are. A lot of your activities are related to enabling the success of everyone around you.</li> </ul> I typically have an employee read aloud every list item in their current role to me, then self-assess the progress they’ve made on each item. I sound off a bit. We generally align; people tend to be fair and honest about their progress. I personally think it’s important that the employee read their list to me instead of the other way around. There’s a sense of ownership that way. It’s tempting to think that over-performers need less guidance, but I’ve found that they tend to need more clarity on tasks, not less, in order to define scope and help point them in a good direction. I’ve also seen underperformers turn around after the career laddering process. What one might see as a lazy quality in a person might actually be a symptom of misalignment with the purpose of the tasks. A career ladder helps them recognize what, when, why, and how the things they’re working on fit into the bigger picture. Ask your team members how they prefer to get feedback, and listen to the answer. The more they are in tune with their own state of mind, the better they think critically and communicate. If you’re inviting everyone to meetings out of fear of hurt feelings, it’s likely not a problem with your meetings, and more a sign that roles and responsibilities aren’t clear for everyone. It’s the job of a manager to disambiguate healthy conflict from attack, so that respectful discourse is encouraged. “You have to know your preferences well, because no matter what you do, someone will tell you you’re wrong.” Book Highlights - The Staff Engineer's Path by Tanya Reilly Unknown — Wed, 15 Mar 2023 00:00:00 +0000 Opportunity cost</a> </h2> Staff engineers’ skills are usually in high demand. If you’re assigned to a single team, you may not be top of mind for solving a problem elsewhere in the org, or your manager may be unwilling to let you go. Losing empathy</a> </h2> It’s easy to overfocus and forget that the rest of the world exists, or start thinking of other technology areas as trivial compared to your rich, nuanced domain. Escaping the echo chamber</a> </h2> When you find yourself in an echo chamber where everyone you meet holds the same set of opinions, it can be a shock when you connect with peers in other groups and discover that some of their views are just…different. A staff engineer navigating tricky terrain</a> </h2> Secret or open?</a> </h2> How much does everyone know? In secret organizations, information is currency and nobody gives it away easily. Everyone’s calendars are private. Slack channels are invite-only. Often you can get access to something if you ask for it, but you have to know it exists! When all information is need-to-know, it’s harder to come up with creative solutions or really understand why something’s not working. In open organizations, you’ll have access to everything (even messy first drafts!). You might get decision fatigue from choosing which information to consume. You might not know which documents are official and need action, and which are just early ideas. And open information can lead to more drama: it’s harder for bad ideas to be quietly shut down. Knowing the cultural expectations around sharing is crucial. In a culture that keeps knowledge locked down, you’ll lose your boss’s trust if you reshare something they told you in confidence. In a more open company, you’ll be considered political or untrustworthy if you withhold information or don’t make sure everyone knows what’s going on. Most companies will be somewhere in the middle on each of these attributes</a> </h2> Pathological</a> </h2> A low-cooperation culture where power and status are the goal and people hoard information; in Westrum’s words, “a preoccupation with personal power, needs, and glory” The chasm between an infrastructure and a product engineering organization</a> </h2> Disputed territory</a> </h2> It’s very hard to draw team boundaries in a way that lets each team work autonomously. No matter how opinionated your APIs, contracts, or team charters, there will inevitably be some pieces of work that multiple teams think they own, and navigating those disputes can feel risky. The new paved road is beautiful, but most of the places people actually want to go are deep in the marsh</a> </h2> Asking to join in</a> </h2> Once you discover a meeting where important decisions get made, it’s natural to want to be part of it. But you’ll need a compelling story for why that should happen. It seems obvious, but your reasons should be about impact to your organization, not to you personally. No matter how much your peer managers like you, framing your exclusion as being bad for your career advancement will be unlikely to change hearts and minds. Show how including you will make your organization better at achieving its goals. Show what you can bring that’s not already there. Have a clear narrative about why you need access, practice your talking points, and go ask to join. You will probably get some resistance. Adding someone to a group is rarely free for the people who are already there. Every extra person in any meeting slows it down, extends discussions, and reduces attendees’ willingness to be vulnerable or brutally honest. If the group is used to working together, every new person resets the dynamic; to some extent, attendees have to learn to work together again. Shadow org chart</a> </h2> Shadow org chart: the unwritten structures through which power and influence flow. The shadow org chart helps you understand who the influencers of the group are, and it’s probably not the same as the actual org chart. These influencers are the people you need to convince before a change can happen. Engineers stop growing</a> </h2> Focusing only on short-term goals limits the way you think about and frame your work, and how much ownership you take of the work that falls into the cracks between tasks. If the team is trying to achieve a big project, they’ll have to identify the gaps between the assigned tasks and figure out how to fill them, building skills in the process. A team that’s used to iterating on short, clearly specified goals won’t build muscle for bigger, more difficult projects and won’t be able to tell the story of why they did what they did. The story of where you were and how much progress you’ve made as well as where you’re going</a> </h2> Tragedy of the commons</a> </h2> Teams pursuing their own best action without coordinating with others leads to an outcome that is bad for everyone. There’s that local maximum again. Depending on the size of the problem, you might start with an engineering-wide vision, a team-scoped vision, or something in between. Don’t create a vision, strategy, etc., unless you need it</a> </h2> Embrace the Boring Ideas</a> </h2> I kind of think writing about engineering strategy is hard because good strategy is pretty boring, and it’s kind of boring to write about. Also I think when people hear “strategy” they think “innovation.” If you write something interesting, it’s probably wrong. This seems directionally correct, certainly most strategy is synthesis and that point isn’t often made! Step away</a> </h2> Tech companies’ promotion systems can incentivize engineers to feel like they need to “win” a technical direction or be the face of a project. This competition can lead to “ape games”: dominance plays and politicking, where each person tries to establish themselves as the leader in the space, considering other people’s ideas to be threatening. It’s toxic to collaboration and makes success much harder to achieve. Robert Konigsberg</a> </h2> “Don’t forget that just because something came from your brain doesn’t make it the best idea.” Set Scope</a> </h2> Be practical about what’s possible. If your vision of the future involves something entirely out of your control, like a change of CEO or an adjustment in your customers’ buying patterns, you’ve crossed into magical thinking. Work around your fixed constraints, rather than ignoring them or wishing they were different. Nemawashi</a> </h2> If you keep your stakeholders aligned as you go along, your document won’t ever have a point where you’re sharing a finished document with a group of people who are learning about it for the first time. When I spoke with Zach Millman, pillar tech lead at Asana, about creating a strategy there, he told me that he used the process of nemawashi, one of the pillars of the Toyota Production System. It means sharing information and laying the foundations so that by the time a decision is made, there’s already a consensus of opinion. Recruiting people</a> </h2> Try to avoid relying on one person’s very specific skill set. People move between companies often, and you don’t want a single point of failure. Blocked by a Single $%@$%</code> Button Click</a> </h2> Why did we insist on a week’s notice? Because these configurations weren’t our only work. Hundreds of teams used our banks of balancers, and load balancing was just one of the four critical services my team supported. We didn’t want to react constantly: we wanted to plan out our weeks, and to make these configuration changes in batches rather than continually restarting services every time. As a result, we had little sympathy for people who came in hot and angry about why we hadn’t done the thing they’d told us about only a few hours ago. Our team motto became “lack of planning on your part is not an emergency on mine.” What Are You Asking For?</a> </h2> I love the “Three Bullets and a Call to Action” method that Brian Fitzpatrick and Ben Collins-Sussman outline in their book, Debugging Teams (O’Reilly). As they write: “A good Three Bullets and a Call to Action email contains (at most) three bullet points detailing the issue at hand, and one—and only one—call to action. That’s it, nothing more—you need to write an email that can be easily forwarded along. If you ramble or put four completely different things in the email, you can be certain that they’ll pick only one thing to respond to, and it will be the item that you care least about. Or worse, the mental overhead is high enough that your mail will get dropped entirely.” The team declared victory and went home—but there was another, better treasure they never got to</a> </h2> It’s Built on a Shaky Foundation</a> </h2> I used to work in a data center, a long time ago, and one thing I learned there is that there’s no such thing as a temporary solution. If someone ran a cable from one rack to another without neatly cabling and labeling it, it would stay there until the server was decommissioned. The same is true for every temporary hack: if you don’t have it in good shape by the end of the project, it’s going to take extraordinary effort to clean it up later. Know Things</a> </h2> No matter how good your leadership, you can’t be a technical leader without the “technical” part. Your big-picture thinking, project execution, credibility, and influence are underpinned by knowledge and experience. A big part of the value proposition of hiring you is your knowledge: you have seen some things. Build experience</a> </h2> It may feel like a compliment when someone offers you the job of manager—hell, take the compliment —but they are not doing you any favors when it comes to your career or your ability to be effective. Admit what you don’t know</a> </h2> Admitting ignorance is one of the most important things we can do as tech leads, senior engineers, mentors, managers, and other influencers of team culture. Understand your own context</a> </h2> A huge part of self-awareness is understanding that you have a perspective, that your context is not the universal context, and that your opinions and knowledge are specific to you. You’ll need to escape your echo chamber every time you talk to teams in other areas or explain technical topics to nonengineers. You’ll know what information you have that they might not, so you can bridge that gap. Don’t delegate through neglect</a> </h2> If glue work is needed for your organization or your project, recognize it and understand who is doing it. Be aware that managers, promotion committees, and future employers might consider this work to be leadership when a staff engineer does it, but dismiss it when a more junior engineer does. So take ownership and do a lot of the work that’s not anybody’s job but that furthers your goals. Redirect your junior colleagues toward tasks that will develop their careers instead. Remember There’s a Team</a> </h2> The final thought in focusing on the mission: remember you’re not doing this work alone. While you may be the best coder on the team, the most experienced engineer, or the fastest problem solver, that doesn’t mean you should jump on all of the problems. You’re working as part of a team, not a collection of competing individuals. Don’t become a single point of failure where the team can’t get anything done when you’re not available. It’s not sustainable. It hides problems. Create Future Leaders</a> </h2> The degree to which other people want to work with you is a direct indication of how successful you’ll be in your career as an engineer. Be the engineer that everyone wants to work with. Peer reviews</a> </h2> It’s a common joke in tech women circles that you know you’re acting at senior level when you get your first peer review saying you’re “abrasive.” Process Preamble</a> </h2> If you write down every edge case, you end up with a three-ring binder of policy and legalese, and it probably still won’t cover every situation. And everyone still hates it! Lara Hogan</a> </h2> Your colleagues won’t learn as much if you only delegate the work after you’ve turned it into “beautifully packaged, cleanly wrapped gifts.” If you instead give them “a messy, unscoped project with a bit of a safety net,” they’ll get a chance to hone their problem-solving abilities, build their own support system, and stretch their skill set. A messy project is a learning opportunity that’s hard to get otherwise. Delegation</a> </h2> This pattern is common in recruiting mails: “Come do exactly the thing you’re currently doing, but at another company.” There are times when that will work, but the most successful recruitment I’ve seen is for roles that offer people a step up, something slightly scary. Change Teams</a> </h2> Moving between teams or organizations can be an excellent way of building bridges: you’ll still have contacts on your own team, and you’ll bring knowledge and culture with you to the new one. You’ll also bring perspective: you already have an outside view of the team and how they’re perceived by the rest of the organization. Take a Management Role</a> </h2> The best frontline eng managers in the world are the ones that are never more than 2-3 years removed from hands-on work, full time down in the trenches. The best individual contributors are the ones who have done time in management. And the best technical leaders in the world are often the ones who do both. Back and forth. Like a pendulum. Charity Majors</a> </h2> “If you’re a manager, your job is to get better at management. Don’t try to cling to your former glory.” Prepare to Reset</a> </h2> Not all newly hired senior leaders are entirely committed to or feel comfortable turning themselves into the leader the organization truly needs, rather than the leader they’ve grown to be over the past years. Many leaders take the opposite approach of trying to mold the organization in their image or the image of the past workplace. Engineering leaders brought into embattled organizations tasked with stabilizing the chaos are often heavily incentivized to do this. Many a time these folks, in my experience, tend to fail harder and more often than those who try to learn the organizational ropes and tailor their leadership style to fit the organizational culture. Load balancer 101, thou shalt not be alone Unknown — Sat, 10 Dec 2022 00:00:00 +0000 Scaling, the dreaded word among developers, because this means more complexity. But why do we need scaling? Imagine a super busy corner store. During early mornings, there might not be a lot of customers, so one cashier might be enough to handle all customers. But during afternoons or evenings, more customers would flock to the store, and our only cashier couldn't checkout fast enough, and this means losing potential customers. Is there a way to solve this? Good news is that's a "yes." However, there are a few implications. You could replace the only cashier with a cashier who can do a checkout faster. But, you could also hire more cashiers as temps during peak hours, and this means cheaper cost per customer compared to maxing out the only cashier. Maximizing a single resource is "vertical scaling", whereas adding more resources of the same caliber is called "horizontal scaling." Couldn't find a more apt name myself. I did mention about complexity. Vertical scaling is less complex than horizontal scaling, because you don't need to coordinate the resources. Because if you have three instances of a website running, but there's only one entrypoint (the website's domain name) then how are you going to distribute the requests to each instance? Load balancer to the rescue. Essentially it acts as a reception front, where it sends incoming requests to available instances. Think of a busy check-in queue in a hotel. You would be waiting in a single line, and when a counter frees up the next person would go there. So how do we actually do this with a website? Common wisdom says using cloud, add a task to container runtime engine, set scaling policy, and route them to a load balancer. There's a load balancer challenge from pulumi</a> you could try out, since their example is tested to be working, which would save a lot of time from stitching up stuff from online articles and official documentations. Note that in this challenge, task scaling isn't implemented, so get your itchy fingers working and implement it! Cross-platform package (+env) management with Nix Unknown — Sat, 03 Dec 2022 00:00:00 +0000 For many years, installing a package on linux means either: Compiling a binary from source, then install it. -> I think we know why this didn't catch on for the mass.</li> Downloading a compiled binary for your system's architecture and platform. -> This requires you to also move the executable to something like /usr/local/bin</code> otherwise it won't be discoverable throughout the system.</li> Using system's package manager: apt</code>, apk</code>, yum</code>, brew</code>, etc. -> Yay finally something that's easy to use. Phew!</li> </ul> Then the dot-com era happened, and the digital transformation, you name it. This was before cloud, so companies set up their own data centers and have to administer and maintain the servers themselves. And it's not fun if you have to perform the same machine configuration for the whole fleet. This problem was solved by tools like Ansible, Chef, Puppet, etc, to set up a machine's configuration en masse. So why is this still an issue? Because there are sizable amount of developers / engineers working on a Mac, and deploy to a Unix-based system. This means those machine setup configurations don't work with Mac, and vice versa, because they use different package managers. Technically you can create a separate set of configs for Mac, but this means maintaining two different setup scripts, which aim to do the same thing, just on different platforms. Sooner or later the configuration would get out of sync 😱. However there are still issues with using system's package manager in linux, namely for some bleeding edge packages, or new package versions that are not yet available in your current linux version, usually involve adding an explicit repository url, adding a keyring, or having to upgrade the system altogether so it can fetch the new repository isn't fun. I've yet to see why we can't install a new package version when its functionalities have almost nothing to do with system version. Otherwise system's package manager is fast and reliable. Meanwhile there's brew</code> for darwin. I only use it because it was one of the few options we have as close as an actual package manager. But it's very slow and performing repo update and installing new apps can take forever, especially if you have to check multiple packages. Not only that, once you have initialized a system, and over time you modified its configurations, how can you be sure that all the changes are populated back to your Ansible script? This would be a manual process prone to errors. But humans are awesome, so the best and brightest came up with nix</code>, as in *nix</code>, a cross-platform package manager that works on both Unix and darwin (a platform name of Mac OS). With nix, you can utilize home-manager</code> to populate the packages/configs on your system, in which it would symlink your configs to nix home, to be symlinked via home-manager switch</code> to actual destination, with read only</code> file permission. This means if you use nix to initialize ~/.ssh/config</code> and you want to change it by hand, it would throw "this file has read-only permission" error, this way the only way to update the configs is through nix. Also with the full system configurations, it takes 6 seconds flat to apply the delta diff 😎. Some interesting snippets from home.nix</code>: Initialize dotfiles</a> </h2> home.file.".ssh/config".source = ./dotfiles/.ssh/config;</code></pre>Git config</a> </h2> Notice the delta</code> block, this automatically populate required configs in .gitignore</code>. programs.git = { # git config --global --edit for raw config content enable = true; userName = ""; userEmail = ""; delta = { enable = true; options = { navigate = true; side-by-side = true; }; }; extraConfig = { diff.colorMoved = "default"; merge.conflictstyle = "diff3"; }; };</code></pre>Neovim config</a> </h2> Nix doesn't use vim-plug</code> by default, and I found some plugins failed installing via nix, hence the extraConfig</code> block for installing vim plugins. Don't forget to run nvim --headless +PlugInstall +qall</code>. programs.neovim = { enable = true; viAlias = true; vimAlias = true; vimdiffAlias = true; plugins = with pkgs.vimPlugins; [ vim-plug ]; extraConfig = '' runtime! plug.vim call plug#begin() "diff Plug 'mhinz/vim-signify' call plug#end() """" config set number ''; };</code></pre>Conditional package list for each platform</a> </h2> Maybe you only use linux for servers, so you might not need GUI apps, or some packages are not available on darwin, etc. home.packages = with pkgs; let # Packages to always install. common = [ fish starship ]; linux_only = [ iotop ntfs3g progress ]; mac_only = [ mpv ]; in common ++ (if stdenv.isLinux then linux_only else mac_only);</code></pre> Head over to the repo</a> for my full setup. And remember: great artists steal. Load credentials into your shell via Bitwarden CLI - Fish edition Unknown — Tue, 29 Nov 2022 00:00:00 +0000 Recently I work with GitHub CLI a lot, and having to constantly fire up Bitwarden app to retrieve GITHUB_TOKEN</code> gets old real fast... I was thinking of storing it in a gist in a password manager, luckily someone had the same idea and implemented it</a>. The only issue is that I use fish shell. But we live in a world where there are many ways to interact with the shell, so it follows that you can translate zsh syntax to fish syntax. For the original snippet in zsh, translated as fish: function unlock_bw_if_locked if test -z BW_SESSION echo 'bw locked - unlocking into a new session' export BW_SESSION="$(bw unlock --raw)" end end function load_github unlock_bw_if_locked set -l github_pat_id $BITWARDEN_GIST_ID set -l github_token set -l github_token "$(bw get notes $github_pat_id)" export GITHUB_OAUTH_TOKEN="$github_token" export GITHUB_TOKEN="$github_token" export GIT_TOKEN="$github_token" end</code></pre> Cloudflare DNS management with Terraform Unknown — Sun, 20 Nov 2022 00:00:00 +0000 I self hosted a lot of services, sometimes I try out a few apps that would get deleted within the same day. All this requires setting up CNAME for reverse-proxy (because I want to make sure there's no funny reverse-proxy shenanigans going on, for future reference). I can always log into Cloudflare console and manually add CNAME entries, but this is getting too tiresome since all I really need is another CNAME with the same config as the rest of the CNAMEs - pointing to the same DNS for my homelab. Cue lightbulb moment when I realize I can use Terraform to set it up. It's as simple as: locals { selfhosted_proxied = toset([ "service_a", "service_b", "service_c", ]) selfhosted_non_proxied = toset([]) } locals { proxied_dict = { for name in local.selfhosted_proxied : name => true } non_proxied_dict = { for name in local.selfhosted_non_proxied : name => false } selfhosted_dns = merge(local.proxied_dict, local.non_proxied_dict) } resource "cloudflare_record" "selfhosted_dns" { for_each = local.selfhosted_dns name = each.key proxied = each.value ttl = 1 type = "CNAME" value = var.ddns zone_id = var.zone_id }</code></pre> Told you it was really easy 😎. Deploy more efficiently with templating Unknown — Sat, 05 Nov 2022 00:00:00 +0000 You are building a website, it's a simple frontend that needs to call the database for [total lead drops this week]. Your website is still at an infancy stage, with only a few features. At this point, you contemplate whether you need a proper backend or not. But to deploy a backend properly, it would involve docker, backend database, persistence storage, DNS, load balancer, among other things. But it looks like you don't have enough time, so you decide to go with serverless, since it takes less time to implement and you don't have to worry about scaling. So you go about creating a serverless function to fetch the information from the database. But the frontend can't call the function directly, so you have to expose the function through an api gateway, so frontend can talk to it. You are happy with the current setup, since it relives you a lot of maintenance effort of creating a proper backend. But your business is still trying to find its footing, so you had an idea of tweaking the query every so often. Then you are getting more annoyed at the fact that, every time you have to update the query, you have to go to the cloud console and manually update the query in your serverless app. You feel like there should be an easier way of just updating a few lines of code. You came upon a black magic called CI/CD. You tinkered with it a bit, then you smiled because this means you don't ever have to log in the cloud console ever again as long as you hook up the CI/CD to your serverless code properly. A few weeks went on, you couldn't be happier. Then you had an idea for a few more features, but at this point you still are not ready for deploying a proper backend yet, so you have to create a few serverless functions for each feature. Suddenly you froze, having realized that you forgot how you setup the first serverless function, not to mention how to connect it to api gateway, and hook it up to CI/CD. Sweats tricking down your forehead, bracing for the eternal doom of doing everything by hand manually every time you update the code. Luckily it's 2022, and the devops tooling space improved significantly compared to the dark ages of VMware. Apparently smart peoples utilize a certain form of template to spin up serverless functions, since it's been dawned on them that most of the configuration differences are your code, not the infrastructure setup. So basically if you wish to create a serverless api to interface with via frontend, this magical template would: Create a serverless function</li> Inject your code into the function</li> Hook up the function with api gateway</li> Also sprinkle a magic CI/CD hook so the whole flow would be triggered again from start to finish after a git push</li> </ol> Most leading IaC tools have this feature, but what if you are not familiar with HashiCorp configuration language? Then you might find such tools with normal programming languagae interface more familiar, in which Pulumi is a contender in this space. HashiCorp also recently released CDK to accomplish the same thing if you want to give it a go as well. But for a start, you could check out Pulumi's Deployments Mini-Challenge</a> to try out deploying a serverless function with templating and CI/CD. For Terraform CDK, check out this repo</a>. IaC is cool, until someone specified the wrong machine size 💸 Unknown — Thu, 03 Nov 2022 00:00:00 +0000 Back in the day, there was no cloud. If you want a lot of computing power, you need to build your own data center, and this is very expensive. Then cloud happened, and suddenly you can work with a lot of flexibility like you couldn't before. Want to try out a small deployment? Sure! If your workload is heavier you can always increase the VM specs. It would almost be the end of the story, until someone realize "oh my God how did I set up VPC peering with another VPC, and which route table did I choose? I need to replicate it with another client and I forgot how". THe issue isn't that working with the cloud's web ui doesn't work, rather it doesn't leave a trail for you, so over time the knowledge of how things were set up are lost, gone forever. I'm sure you can always mitigate this by documenting things, but then it can happen that someone modified the infrastructure without updating the documentation. Can't blame them, it's only normal when you have to synchronize things manually. So people came up with IaC - infrastructure as code. Basically this means, you write declarative blocks of what a resource's configuration should be, and how it's connected to another resources. Then you "apply" the configuration on your cloud provider. If you are researching or want to get something up and running fast, I wouldn't suggest you use IaC. But if you have to modify the configuration, you might find it easier to maintain in the long run, plus you can always use git diff</code> to show the changes between each commit. And as how most things work, this certainly is not the end, because what if you have to create a new VM fleet, a small cluster for a research project. IaC can do this, but what if you were working late at night and you copied the wrong instance size into your IaC config - what could be 50 USD/month small cluster could turn out to be a 2000 USD / month large cluster, and it won't even be utilized that much because your research workload is very light 💸💸💸💸💸. Luckily humans excel at adaptation, so we came up with "IaC policy", where you can define a set of constraints that your resources should comply to. This could be anything from "only instance smaller than $x is allowed", or "total monthly cost most not exceed $y". It can also be "maximum storage allowed is 500GB", or even "daily backup is required for $z resource". The world is not that gloomy, a lot of people made sure of that 🚀. And I know you are eager to get your hands on at policy as code thing, so head to a pulumi challenge</a> and take a stab at it! Deploy static site with branch preview via Cloudflare Pages Unknown — Wed, 05 Oct 2022 00:00:00 +0000 Updated 2023-02-20: update terraform code For frontends, if no server-side rendering is required, we can deploy it as a static site. If you already use GitHub, you might be familiar with GitHub Pages. One common use case is to deploy your personal landing page / blog via GitHub Actions. Interestingly enough, this might cause problems if you are working in a team. For example, if you are working on a UI change, and you need to have someone else approve the changes, they would need to build the site locally to do so. Luckily, "branch preview" feature exists. Essentially it's a mechanism to generate a preview site for every pull request. We are going to use Cloudflare Pages for this (alternatives are Vercel, etc.) You should have a github repo with the source code for your site. Then, Terraform</a> </h2> resource "cloudflare_pages_project" "this" { account_id = var.account_id name = var.project_name production_branch = var.production_branch lifecycle { ignore_changes = [deployment_configs, source] } } resource "cloudflare_record" "this" { depends_on = [cloudflare_pages_project.this] name = var.subdomain proxied = true ttl = 1 type = "CNAME" value = cloudflare_pages_project.this.subdomain zone_id = var.zone_id } resource "cloudflare_pages_domain" "this" { depends_on = [cloudflare_record.this] account_id = var.account_id project_name = var.project_name domain = var.domain_name }</code></pre> Notice that cloudflare_pages_domain</code> is commented out, this is because it has a bug that would throw "error creating domain". Although behind the scenes it creates a subdomain successfully (and is actually working with cloudflare pages).</del> GitHub Actions</a> </h3> We are going to use github actions as a runner to build and deploy the site. name: Deploy on: push: workflow_dispatch: concurrency: group: environment-${{ github.ref }} cancel-in-progress: true env: PROJECT_NAME: PROJECT_NAME # CHANGE ME BUILD_DIR: BUILD_DIR # CHANGE ME jobs: publish: runs-on: ubuntu-latest permissions: contents: read deployments: write name: Publish to Cloudflare Pages steps: - name: Checkout uses: actions/checkout@v3 # ---------- build ---------- - name: Setup node uses: actions/setup-node@v3.5.0 with: node-version: "18" cache: "yarn" - name: Install requirements run: yarn install - name: Build run: yarn run build # ---------- publish ---------- - name: Deploy to Cloudflare Pages uses: cloudflare/wrangler-action@v3 with: apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }} accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }} command: pages deploy ${{ env.BUILD_DIR }} --project-name=${{ env.PROJECT_NAME }}</code></pre> Notice the "build" section, you can adjust this per your site's setup. This actions would work for both a push to master and a pull request Terraform with ECS task on EC2 backend Unknown — Tue, 04 Oct 2022 00:00:00 +0000 Previously</a> I wrote about setting up ECS task on fargate backend. But we can also use EC2 as backend too, in some cases where the workload is consistent, ie scaling is not required, since EC2 would be cheaper than fargate backend, even more so if you have reserved instance on top. There's a few modifications from the fargate version to make it work with EC2 backend, if you are curious you can try to hunt those down 😎. Repo here</a>. Task definition</a> </h2> #tfsec:ignore:aws-cloudwatch-log-group-customer-key resource "aws_cloudwatch_log_group" "this" { retention_in_days = 14 name = "/aws/ecs/${var.service_name}" } resource "aws_ecs_task_definition" "this" { family = var.service_name cpu = var.cpu memory = var.memory execution_role_arn = var.task_role task_role_arn = var.task_role container_definitions = jsonencode( [ { name = var.service_name image = var.image_uri essential = true environment = [] portMappings = [ { protocol = "tcp" containerPort = 80 hostPort = 80 } ] logConfiguration = { logDriver = "awslogs" options = { awslogs-group = aws_cloudwatch_log_group.this.name awslogs-region = var.aws_region awslogs-stream-prefix = "ecs" } } } ] ) } resource "aws_ecs_service" "this" { name = var.service_name cluster = var.ecs_cluster_id task_definition = aws_ecs_task_definition.this.arn desired_count = 1 # https://github.com/hashicorp/terraform/issues/26950 depends_on = [aws_lb.this, aws_alb_target_group.this] ordered_placement_strategy { type = "spread" field = "instanceId" } load_balancer { target_group_arn = aws_alb_target_group.this.arn container_name = var.service_name container_port = 80 } }</code></pre>SSL certificate</a> </h2> resource "aws_acm_certificate" "this" { domain_name = var.domain_name validation_method = "DNS" lifecycle { create_before_destroy = true } tags = { Name = var.domain_name } }</code></pre>Load balancer</a> </h2> #tfsec:ignore:aws-elb-alb-not-public #tfsec:ignore:aws-elb-drop-invalid-headers resource "aws_lb" "this" { name = var.service_name internal = false load_balancer_type = "application" security_groups = [var.alb_id] subnets = var.subnet_id } resource "aws_alb_target_group" "this" { name = var.service_name port = 80 protocol = "HTTP" vpc_id = var.vpc_id target_type = "instance" health_check { port = "traffic-port" path = var.health_check_path matcher = "200-499" } } #tfsec:ignore:aws-elb-http-not-used resource "aws_alb_listener" "http" { load_balancer_arn = aws_lb.this.id port = 80 protocol = "HTTP" default_action { type = "redirect" redirect { port = "443" protocol = "HTTPS" status_code = "HTTP_301" } } } #tfsec:ignore:aws-elb-use-secure-tls-policy resource "aws_alb_listener" "https" { load_balancer_arn = aws_lb.this.id port = 443 protocol = "HTTPS" ssl_policy = "ELBSecurityPolicy-2016-08" certificate_arn = aws_acm_certificate.this.arn default_action { type = "forward" target_group_arn = aws_alb_target_group.this.id } }</code></pre> Intro to Dagster Cloud Unknown — Tue, 27 Sep 2022 00:00:00 +0000 Imagine you have a few data pipelines to schedule. Simplest solution would be cronjob. Time goes by and next thing you know, you have around 50 pipelines to manage. The fun starts when you have to hunt down which pipeline doesn't run normally. And by then it would be super hard to do tracing if you haven't set up logging and monitoring. Luckily there are tools we can use to improve the situation. Task orchestrators are born exactly for this, to schedule and monitor pipelines. These days there are more bells and whistles, such as backfilling and sensor triggers. Some also integrate with data catalog tools and provide table specs and data lineage. Sounds familiar? If you have been looking into Airflow, this is exactly what it does. But there are other alternatives too, and in this post we're going to find out what Dagster Cloud can do. (Spoiler: yay CI/CD). Dagster architecture</a></a> </h2> This is how dagster work. But to set all this up yourself, you need to at least: Package each dagster repository into its own image</li> Setup dagit container for UI</li> Set up dagster daemon instance to trigger tasks</li> Set up postgres for dagster metastore</li> Set I/O manager to use blob storage</li> Set up default executors and concurrent run limits</li> CI/CD to update code and base dagster image</li> </ul> And if you have multiple users, you need to set up the auth yourself too. Enter Dagster Cloud</a> </h2> But folks at Dagster know people are finding it very tricky to set it up, so they provide a few cloud offerings, one of which is serverless. This essentially means you only need to supply the code and set up CI/CD, and Dagster would provide a VM to run your tasks. To test it out, you can sign up on Dagster Cloud</a> and follow instructions here</a>. Cool dagster features</a> </h2> Branch preview</a> </h2> Frontend usually set up CI/CD to return a preview of a PR, now you can do the same with dagster! This means your team can preview what the DAG would look like, or give it a spin before deploying to prod 🚀. Multiple teams you say?</a> </h2> Imagine working in an organization with many teams. Some teams might use dbt, another might use spark. Sometimes each team has different virtual environment. Even if you can use the same setup for multiple teams, over time the number of pipelines would grow significantly, which means it's harder to manage. But we can set up a dagster repo for each team, then link them together via Dagster Cloud for a single control plane. Notice each code location has attached git hash. Yay tracing! Table lineage</a> </h2> Running multiple pipelines are cool. But if you update this pipeline, do you also need to update downstream pipelines? I'm sure you can have a list of pipelines dependency somewhere, but it's so much more convenient to see it right from dagster. Notice the Materialize</code> button on top right. This means you can trigger run on an upstream table, and it would automatically update downstream tables. Dagster has more features, don't forget to check out their docs</a> (it's well written)! Intro to Pulumi Unknown — Mon, 26 Sep 2022 00:00:00 +0000 For IaC, no doubt that Terraform is the leader. But there are other alternatives too, one of them is Pulumi. Currently Pulumi provides fun challenges to get started with their services. Best of all, they give you swags too! We are going to create a simple Pulumi project for hosting a static site through Cloudfront CDN. Challenge url: https://www.pulumi.com/challenge/startup-in-a-box/</a> Pre-requisites</a> </h2> Pulumi</a> account</li> Checkly</a> account</li> AWS account</li> Install Pulumi cli: brew install pulumi/tap/pulumi</code></li> </ul> Steps</a> </h2> Init</a> </h3> Init pulumi project mkdir pulumi-challenge-startup-in-a-box pulumi new aws-typescript</code></pre>Create S3 bucket</a> </h3> Then we are going to create an s3 bucket to host static website, your index.ts</code> should look like this: import * as pulumi from "@pulumi/pulumi"; import * as aws from "@pulumi/aws"; import * as awsx from "@pulumi/awsx"; const bucket = new aws.s3.BucketV2("bucketV2", { tags: { Name: "pulumi-challenge-karn-wong", }, }); const bucketAcl = new aws.s3.BucketAclV2("bAcl", { bucket: bucket.id, acl: aws.s3.PublicReadAcl, });</code></pre>Set MIME type</a> </h3> To set MIME type, we could leverage Node API: npm install mime @types/mime</code></pre> Then append this to index.ts</code>: import * as fs from "fs"; import * as mime from "mime"; const staticWebsiteDirectory = "website"; fs.readdirSync(staticWebsiteDirectory).forEach((file) => { const filePath = `${staticWebsiteDirectory}/${file}`; const fileContent = fs.readFileSync(filePath).toString(); new aws.s3.BucketObject(file, { bucket: bucket.id, source: new pulumi.asset.FileAsset(filePath), contentType: mime.getType(filePath) || undefined, acl: aws.s3.PublicReadAcl, }); });</code></pre>Create static assets</a> </h3> But we are going to serve a static website, so we need to provide static assets: website/index.html</code></a> </h4> <!doctype html> <html> <head> <meta charset="utf-8" /> <title>Pulumi Challenge</title> <link rel="stylesheet" href="style.css" /> <link rel="stylesheet" href="normalize.css" /> </head> <body> <header>  <div class="logo"> <ul> <li><i class="fas fa-feather"></i></li> <li><p>Company Name</p></li> </ul> </div> <ul class="social">  <li> <a href="http://github.com/" target="_blank" ><i class="fab fa-github-alt"></i ></a> </li> <li> <a href="http://twitter.com/" target="_blank" ><i class="fab fa-twitter"></i ></a> </li> <li> <a href="http://linkedin.com/" target="_blank" ><i class="fab fa-linkedin-in"></i ></a> </li> </ul> </header> <div class="banner">  <h1>Your Startup Name Here</h1> <h3>Your Tagline</h3> <p> We're $CompanyName, and we're changing what it means to $Task. Our innovative use of $Technology makes life easier for $JobTitles, so they can focus on what they're really good at instead of wasting time and effort on $MenialOrDifficultTask. Streamline your $TaskProcess with $Product and take to the skies! </p> </div> </body> <script src="https://kit.fontawesome.com/b4747495ea.js" crossorigin="anonymous" ></script> </html></code></pre>website/style.css</code></a> </h4> @import url("https://fonts.googleapis.com/css?family=News+Cycle|Teko&display=swap"); body { background-color: #f7f7fa; opacity: 0.8; background-image: radial-gradient(#f79645 0.5px, #f7f7fa 0.5px); background-size: 10px 10px; } ul { list-style-type: none; } ul li { display: inline-block; } a { color: white; -webkit-transition: color 0.5s ease-out; transition: color 0.5s ease-out; text-decoration: none; } a:hover, a:active { color: rgb(55, 188, 250); } header { background-color: rgba(214, 73, 73, 0.6); height: 80px; position: absolute; top: 0; width: 100%; box-shadow: 0px 2px 7px -1px rgba(0, 0, 0, 0.75); -webkit-box-shadow: 0px 2px 7px -1px rgba(0, 0, 0, 0.75); -moz-box-shadow: 0px 2px 7px -1px rgba(0, 0, 0, 0.75); } header li { color: white; } .active a { color: rgb(255, 157, 112); } .social { position: absolute; right: 50px; top: -5px; font-size: 30px; } .social li { margin: 0 5px 0 5px; } .logo { font-family: Teko; position: absolute; left: 5px; top: -60px; font-size: 40px; } .banner { width: 60vw; font-family: Teko; font-size: 2vw; text-align: center; margin-top: 15vw; margin-left: 20vw; } .banner h1 { color: rgb(214, 73, 73); } .banner p, .about p { font-family: News Cycle; }</code></pre>website/normalize.css</code></a> </h4> Download from https://github.com/necolas/normalize.css/blob/master/normalize.css</a> Create Cloudfront CDN</a> </h3> We can always fetch the site via s3 directly, but your s3 bucket is in a single region. Users in far-away-regions might have a significant latency. To circumvent this, we could utilize CDN to reduce the site latency, since CDN would serve the content from a region nearest to users. To achieve this, append the following to index.ts</code>: const s3OriginId = "myS3Origin"; const cloudfrontDistribution = new aws.cloudfront.Distribution( "s3Distribution", { origins: [ { domainName: bucket.bucketRegionalDomainName, originId: s3OriginId, }, ], enabled: true, isIpv6Enabled: true, comment: "Some comment", defaultRootObject: "index.html", defaultCacheBehavior: { allowedMethods: [ "DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT", ], cachedMethods: ["GET", "HEAD"], targetOriginId: s3OriginId, forwardedValues: { queryString: false, cookies: { forward: "none", }, }, viewerProtocolPolicy: "allow-all", minTtl: 0, defaultTtl: 3600, maxTtl: 86400, }, priceClass: "PriceClass_200", restrictions: { geoRestriction: { restrictionType: "whitelist", locations: ["US", "CA", "GB", "DE"], }, }, viewerCertificate: { cloudfrontDefaultCertificate: true, }, }, );</code></pre>Reusable module</a> </h3> Think of it as instead of creating resources manually all the time, you can call a function instead. Refactor away we go! Remove cloudfront block in index.ts</code></li> Create cdn-website/index.ts</code> with following content:</li> </ol> import * as pulumi from "@pulumi/pulumi"; import * as aws from "@pulumi/aws"; import * as fs from "fs"; import * as mime from "mime"; // This is a simpler version of: // https://github.com/pulumi/pulumi-aws-static-website export class CdnWebsite extends pulumi.ComponentResource { private bucket: aws.s3.BucketV2; private bucketAcl: aws.s3.BucketAclV2; private cloudfrontDistribution: aws.cloudfront.Distribution; private s3OriginId: string = "myS3Origin"; private staticWebsiteDirectory: string = "./website"; constructor(name: string, args: any, opts?: pulumi.ComponentResourceOptions) { super("pulumi:challenge:CdnWebsite", name, args, opts); this.bucket = new aws.s3.BucketV2( "bucketV2", { tags: { Name: "My bucket", }, }, { parent: this, }, ); this.bucketAcl = new aws.s3.BucketAclV2( "bAcl", { bucket: this.bucket.id, acl: aws.s3.PublicReadAcl, }, { parent: this, }, ); this.cloudfrontDistribution = new aws.cloudfront.Distribution( "s3Distribution", { origins: [ { domainName: this.bucket.bucketRegionalDomainName, originId: this.s3OriginId, }, ], enabled: true, isIpv6Enabled: true, comment: "Some comment", defaultRootObject: "index.html", defaultCacheBehavior: { allowedMethods: [ "DELETE", "GET", "HEAD", "OPTIONS", "PATCH", "POST", "PUT", ], cachedMethods: ["GET", "HEAD"], targetOriginId: this.s3OriginId, forwardedValues: { queryString: false, cookies: { forward: "none", }, }, viewerProtocolPolicy: "allow-all", minTtl: 0, defaultTtl: 3600, maxTtl: 86400, }, priceClass: "PriceClass_200", restrictions: { geoRestriction: { restrictionType: "whitelist", locations: ["US", "CA", "GB", "DE"], }, }, viewerCertificate: { cloudfrontDefaultCertificate: true, }, }, { parent: this, }, ); fs.readdirSync(this.staticWebsiteDirectory).forEach((file) => { const filePath = `${this.staticWebsiteDirectory}/${file}`; const fileContent = fs.readFileSync(filePath).toString(); new aws.s3.BucketObject( file, { bucket: this.bucket.id, source: new pulumi.asset.FileAsset(filePath), contentType: mime.getType(filePath) || undefined, acl: aws.s3.PublicReadAcl, }, { parent: this.bucket, }, ); }); // We also need to register all the expected outputs for this // component resource that will get returned by default. this.registerOutputs({ bucketName: this.bucket.id, cdnUrl: this.cloudfrontDistribution.domainName, }); } get url(): pulumi.Output<string> { return this.cloudfrontDistribution.domainName; } }</code></pre> Inindex.ts</code> modify the file so that it looks like this:</li> </ol> // Deploy Website to S3 with CloudFront // Also shows the challenger how to build a ComponentResource import { CdnWebsite } from "./cdn-website"; const website = new CdnWebsite("your-startup", {});</code></pre>Sanity checks</a> </h3> If everything works correctly, we should be able to deploy the website. But it would be so much better if we can see whether the website is working normally or not. Status checks to the rescue! Here we set up checkly</code> provider: npm install @checkly/pulumi # API KEY: https://app.checklyhq.com/settings/account/api-keys pulumi config set checkly:apiKey --secret # AccountID: https://app.checklyhq.com/settings/account/general pulumi config set checkly:accountId</code></pre> Then append this to index.ts</code>: export const websiteUrl = website.url; import * as checkly from "@checkly/pulumi"; import * as fs from "fs"; new checkly.Check("index-page", { activated: true, frequency: 10, type: "BROWSER", locations: ["eu-west-2"], script: websiteUrl.apply((url) => fs .readFileSync("checkly-embed.js") .toString("utf8") .replace("{{websiteUrl}}", url), ), });</code></pre> But we need to supply logic for sanity checks, we can do this by creating checkly-embed.js</code> (which the above checkly resource fetches). const playwright = require("playwright"); const expect = require("expect"); const browser = await playwright.chromium.launch(); const page = await browser.newPage(); await page.goto("https://{{websiteUrl}}"); expect(await page.title()).toBe("Pulumi Challenge"); await browser.close();</code></pre>Cherry on top (who wants swags?)</a> </h3> Kindly provided by folks at Pulumi, we can get cool swags by calling an API endpoint as follows: npm install got@11.8.0</code></pre> Then paste the following in swag-provider/index.ts</code>: import * as pulumi from "@pulumi/pulumi"; const submittionUrl: string = "https://hooks.airtable.com/workflows/v1/genericWebhook/apptZjyaJx5J2BVri/wflmg3riOP6fPjCII/wtr3RoDcz3mTizw3C"; interface SwagInputs { name: string; email: string; address: string; size: "XS" | "S" | "M" | "L" | "XL" | "XXL" | "XXXL"; } interface SwagCreateResponse { success: boolean; } interface SwagOutputs extends SwagInputs { id: string; } class SwagProvider implements pulumi.dynamic.ResourceProvider { private name: string; constructor(name: string) { this.name = name; } async create(props: SwagInputs): Promise<pulumi.dynamic.CreateResult> { const got = (await import("got")).default; let data = await got .post(submittionUrl, { headers: { "Content-Type": "application/json", }, json: { ...props, }, }) .json<SwagCreateResponse>(); return { id: props.email, outs: props }; } } export class Swag extends pulumi.dynamic.Resource { constructor( name: string, props: SwagInputs, opts?: pulumi.CustomResourceOptions, ) { super(new SwagProvider(name), name, props, opts); } }</code></pre> Finally, paste this to index.ts</code>: import { Swag } from "./swag-provider"; const swag = new Swag("your-startup", { name: "YOUR NAME", email: "YOUR EMAIL", address: "YOUR ADDRESS", size: SIZE, });</code></pre>Open sesame</a> </h3> 'Nuff said pulumi up</code></pre> If everything works correctly, your terminal should display something like this: Previewing update (dev) View Live: https://app.pulumi.com/kahnwong/pulumi-challenge-startup-in-a-box/dev/previews/xxxxxxxx Type Name Plan + pulumi:pulumi:Stack pulumi-challenge-startup-in-a-box-dev create + ├─ pulumi:challenge:CdnWebsite your-startup create + │ ├─ aws:s3:BucketV2 bucketV2 create + │ │ ├─ aws:s3:BucketObject index.html create + │ │ ├─ aws:s3:BucketObject style.css create + │ │ └─ aws:s3:BucketObject normalize.css create + │ ├─ aws:s3:BucketAclV2 bAcl create + │ └─ aws:cloudfront:Distribution s3Distribution create + ├─ pulumi-nodejs:dynamic:Resource your-startup create + └─ checkly:index:Check index-page create Outputs: websiteUrl: output<string> Resources: + 10 to create Do you want to perform this update? yes Updating (dev) View Live: https://app.pulumi.com/kahnwong/pulumi-challenge-startup-in-a-box/dev/updates/1 Type Name Status + pulumi:pulumi:Stack pulumi-challenge-startup-in-a-box-dev created + ├─ pulumi:challenge:CdnWebsite your-startup created + │ ├─ aws:s3:BucketV2 bucketV2 created + │ │ ├─ aws:s3:BucketObject style.css created + │ │ ├─ aws:s3:BucketObject index.html created + │ │ └─ aws:s3:BucketObject normalize.css created + │ ├─ aws:cloudfront:Distribution s3Distribution created + │ └─ aws:s3:BucketAclV2 bAcl created + ├─ pulumi-nodejs:dynamic:Resource your-startup created + └─ checkly:index:Check index-page created Outputs: websiteUrl: "dyacxs5ksi1r9.cloudfront.net" Resources: + 10 created Duration: 3m25s</code></pre> Don't you love it when everything works as you planned 😎. To tear down all resources, run pulumi destroy</code>. If successful, your terminal should return: Previewing destroy (dev) View Live: https://app.pulumi.com/kahnwong/pulumi-challenge-startup-in-a-box/dev/previews/xxxxxxxx Type Name Plan - pulumi:pulumi:Stack pulumi-challenge-startup-in-a-box-dev delete - ├─ checkly:index:Check index-page delete - ├─ pulumi-nodejs:dynamic:Resource your-startup delete - └─ pulumi:challenge:CdnWebsite your-startup delete - ├─ aws:cloudfront:Distribution s3Distribution delete - ├─ aws:s3:BucketAclV2 bAcl delete - └─ aws:s3:BucketV2 bucketV2 delete - ├─ aws:s3:BucketObject normalize.css delete - ├─ aws:s3:BucketObject style.css delete - └─ aws:s3:BucketObject index.html delete Outputs: - websiteUrl: "dyacxs5ksi1r9.cloudfront.net" Resources: - 10 to delete Do you want to perform this destroy? yes Destroying (dev) View Live: https://app.pulumi.com/kahnwong/pulumi-challenge-startup-in-a-box/dev/updates/2 Type Name Status - pulumi:pulumi:Stack pulumi-challenge-startup-in-a-box-dev deleted - ├─ checkly:index:Check index-page deleted - ├─ pulumi-nodejs:dynamic:Resource your-startup deleted - └─ pulumi:challenge:CdnWebsite your-startup deleted - ├─ aws:cloudfront:Distribution s3Distribution deleted - ├─ aws:s3:BucketAclV2 bAcl deleted - └─ aws:s3:BucketV2 bucketV2 deleted - ├─ aws:s3:BucketObject index.html deleted - ├─ aws:s3:BucketObject normalize.css deleted - └─ aws:s3:BucketObject style.css deleted Outputs: - websiteUrl: "dyacxs5ksi1r9.cloudfront.net" Resources: - 10 deleted Duration: 3m22s The resources in the stack have been deleted, but the history and configuration associated with the stack are still maintained. If you want to remove the stack completely, run `pulumi stack rm dev`.</code></pre> Data engineer archetypes Unknown — Fri, 26 Aug 2022 00:00:00 +0000 I have been working in the data industry since almost half a decade ago. Over time I have noticed so-called archetypes within various data engineering roles. Below are main skills and combinations I have seen over the years. This is by no means an exhaustive list, rather what I often see. SQL + SSIS</a> </h2> Using SQL to manipulate data via SSIS, in which data engine is Microsoft SQL Server.</li> Commonly found in enterprise organizations that use Microsoft stack.</li> </ul> SQL + Hive</a> </h2> Using SQL to manipulate data via Hive, a filesystem that support columnar data format, usually accessed via Zeppelin.</li> Often found in enterprise organizations that work with big data before Spark was released.</li> </ul> SQL + DBT</a> </h2> Using SQL to manipulate data via DBT, an abstraction later for data pipelines scheduler that allows users to use SQL interface with various database engines. DBT is often mentioned in Modern Data Stack.</li> Often found in newly established organizations in the last few years.</li> </ul> Python + pandas</a> </h2> Using python with pandas to manipulate data, usually with data that can fit into memory (ie less than 5GB)</li> This is also common if you have data scientists manipulate data, since pandas is what they are familiar with. In addition, most people who write pandas are not known for writing well-optimized code, but it’s negligible for small data.</li> </ul> Python + pyspark</a> </h2> Using python with pyspark to manipulate data, can be either SQL or Spark SQL.</li> Usually organizations that use pyspark also does machine learning as well.</li> Often found in organizations that work with big data, and have established data lake platform.</li> </ul> Scala + spark</a> </h2> Using Scala to manipulate data via spark.</li> Often found on enterprise organizations where they have been using spark before pyspark was released. Has more limited data ecosystem.</li> </ul> Python + Task orchestrator (airflow, dagster, etc)</a> </h2> Using task orchestrators to run pipelines on a regular basis, the application logic is written in python. Inside can be anything from pure python to pyspark. Or you can use bash and use any unix tools.</li> People who fall under this category often have software engineering background.</li> </ul> Platform engineering (setting up data infrastructure, etc)</a> </h2> These are people that set up database, infrastructure, networking, and everything required to allow engineers/users to create data pipelines and consume data at downstream.</li> Usually they are DevOps who transitioned from working with app infra to data infra.</li> </ul> Updated 2022-09-02 GUI-based solutions</a> </h2> Using GUI-based tools to create data pipelines, such as Talend, AWS Glue, Azure Data Factory, etc. May or may not use in conjunction with SQL / python / pyspark.</li> </ul> Minimal ECS task with Fargate backend Unknown — Fri, 26 Aug 2022 00:00:00 +0000 To deploy a web application, there are many ways to go about it. I could spin up a bare VM and set up the environment manually. To make things easier, I could have package the app into docker image. But this still means I have to "update" the app manually if I add changes to it. Things would be super cool if: after I push the changes to master branch, the app would be deployed automatically. In order to achieve this, I could use AWS ECS task to deploy the app, and add CI/CD to it (because this is 2022 after all). And things would be even better if I don't have to set up the infra manually every time I want to deploy an app, enters terraform! Below are minimal ecs task with fargate backend setup 😎. Repo here</a>. Updated 2022-09-02 Notes: you might need to set up autoscaling on LB connections per target. Also this example contains two target tracking policies for the same service. Race conditions can result in undesirable scaling issues. Thanks John Mille! Task definition</a> </h2> This is equivalent to docker-compose.yaml resource "aws_cloudwatch_log_group" "this" { retention_in_days = 14 name = "/aws/ecs/${var.service_name}" } resource "aws_ecs_task_definition" "this" { family = var.service_name network_mode = "awsvpc" requires_compatibilities = ["FARGATE"] cpu = 256 memory = 512 execution_role_arn = var.task_role task_role_arn = var.task_role container_definitions = jsonencode( [ { name = var.service_name image = var.image_uri essential = true environment = [] portMappings = [ { protocol = "tcp" containerPort = 80 hostPort = 80 } ] logConfiguration = { logDriver = "awslogs" options = { awslogs-group = aws_cloudwatch_log_group.this.name awslogs-region = var.aws_region awslogs-stream-prefix = "ecs" } } } ] ) } resource "aws_ecs_service" "this" { name = var.service_name cluster = var.ecs_cluster_id task_definition = aws_ecs_task_definition.this.arn desired_count = 1 deployment_minimum_healthy_percent = 50 deployment_maximum_percent = 200 launch_type = "FARGATE" scheduling_strategy = "REPLICA" network_configuration { security_groups = [var.alb_id] subnets = var.subnet_id assign_public_ip = true } load_balancer { target_group_arn = aws_alb_target_group.this.arn container_name = var.service_name container_port = 80 } }</code></pre>SSL certificate</a> </h2> resource "aws_acm_certificate" "this" { domain_name = var.domain_name validation_method = "DNS" lifecycle { create_before_destroy = true } tags = { Name = var.domain_name } }</code></pre>Load balancer</a> </h2> resource "aws_lb" "this" { name = var.service_name internal = false load_balancer_type = "application" security_groups = [var.alb_id] subnets = var.subnet_id idle_timeout = 3600 enable_deletion_protection = true } resource "aws_alb_target_group" "this" { name = var.service_name port = 80 protocol = "HTTP" vpc_id = var.vpc_id target_type = "ip" health_check { healthy_threshold = "3" interval = "30" protocol = "HTTP" matcher = "200" timeout = "3" path = var.health_check_path unhealthy_threshold = "2" } } resource "aws_alb_listener" "http" { load_balancer_arn = aws_lb.this.id port = 80 protocol = "HTTP" default_action { type = "forward" target_group_arn = aws_alb_target_group.this.arn } } resource "aws_alb_listener" "https" { load_balancer_arn = aws_lb.this.id port = 443 protocol = "HTTPS" ssl_policy = "ELBSecurityPolicy-2016-08" certificate_arn = aws_acm_certificate.this.arn default_action { target_group_arn = aws_alb_target_group.this.id type = "forward" } }</code></pre>Autoscaling</a> </h2> Because we are using cloud, and I love taking advantage of dynamic resources allocation. resource "aws_appautoscaling_target" "this" { max_capacity = 2 min_capacity = 1 resource_id = "service/${var.ecs_cluster_name}/${aws_ecs_service.this.name}" scalable_dimension = "ecs:service:DesiredCount" service_namespace = "ecs" } resource "aws_appautoscaling_policy" "memory" { name = "memory-autoscaling" policy_type = "TargetTrackingScaling" resource_id = aws_appautoscaling_target.this.resource_id scalable_dimension = aws_appautoscaling_target.this.scalable_dimension service_namespace = aws_appautoscaling_target.this.service_namespace target_tracking_scaling_policy_configuration { predefined_metric_specification { predefined_metric_type = "ECSServiceAverageMemoryUtilization" } target_value = 40 } } resource "aws_appautoscaling_policy" "cpu" { name = "cpu-autoscaling" policy_type = "TargetTrackingScaling" resource_id = aws_appautoscaling_target.this.resource_id scalable_dimension = aws_appautoscaling_target.this.scalable_dimension service_namespace = aws_appautoscaling_target.this.service_namespace target_tracking_scaling_policy_configuration { predefined_metric_specification { predefined_metric_type = "ECSServiceAverageCPUUtilization" } target_value = 60 } }</code></pre> What SQL can't do for data engineering Unknown — Sun, 15 May 2022 00:00:00 +0000 I often hear people ask "if you can do data engineering with SQL, then what's the point of learning spark or python?" Data ingestion</a> </h2> Let's circle back at bit. I think we all can agree that: there's a point in time where there's no data in the data warehouse (which DE-who-use-SQL's use as base of operation). The source data could be anything from hard CSV/Excel or API endpoints. No data in data warehouse, DE can't use SQL to do stuff with the data. So who put the data into the data warehouse? Data engineers of course! But not the kind that use SQL. These data engineers are known as "data platform engineers," where their main focus is data ingestion, platform and scalability. The confusion lies in the fact that, there seems to be different tools and skills required for platform or analytics type for data engineers, but some places still refer to both roles as data engineer. And guess how data platform engineers write large amount of data to data warehouse. With help of spark and python/java of course! Data cleaning</a> </h2> What if you need to perform complex data cleaning processes for your data? You might have to create multiple temp columns, then coalesce them at the end to get the final result. You can do this in SQL, with the help of jinja template, to certain extent. But the same process expressed in python would be much more concise and readable. Especially if it in involves multiple steps spanning 700 lines of code. Not to mention you might need to perform debugging somewhere in between, and setting up SQL debugging is not something I wish on my worst enemy. Some people say "but you can also create functions in SQL." Yes, you can, but it's clunky and very fragile, and not very readable. Optimization</a> </h2> When you're writing SQL for data transformation, the actual execution logic is being translated by the database's engine. As with all forms of translations, some information is lost, and can result in non-optimized instructions. For instance, if you need to perform longest matching against a list of string, one way to optimize this is to set a break</code> condition after you found a match. In SQL, this most likely results in instructions to compare every string from the list, instead of stopping the matching process and move onto the next operation after a match is found. These are some but not all instances where SQL falls short for data engineering tasks. I hope this article sheds some light on the importance of using python/spark for data engineering tasks. Use SSH key during Docker Build without embedding the key via ssh-agent Unknown — Sun, 06 Feb 2022 00:00:00 +0000 Imagine working in a company, and they have a super cool internal module! The module works great, except that it is a private module, which means you need to install it by cloning the source repo and install it from source. That shouldn't be an issue if you work on your local machine. But for production usually this means you somehow need to bundle this awesome module into your docker image. You go create a Dockerfile and there's one little problem: it couldn't clone the module repo because it doesn't have the required SSH key that can access the repo. A very simple solution would just be bundling the SSH key into the docker image itself. This works great, until security comes knocking because: anyone who can access the image can also access the source repo! And we wouldn't want that. So what's the solution? Luckily there's a thing called ssh-agent forwarding. Think of it as passing an SSH key into docker during build, then poof! It's gone. Instructions</a> </h2> Set up SSH key on your host machine. This essentially means you should be able to perform git clone $REPO</code> on the module repo.</li> In your Dockerfile, add something like the following:</li> </ol> # Authorize SSH Host RUN mkdir -p -m 700 /root/.ssh && \ touch -m 600 /root/.ssh/known_hosts && \ ssh-keyscan github.com > /root/.ssh/known_hosts RUN --mount=type=ssh,id=github $SOME_COMMAND_THAT_NEEDS_SSH_KEY</code></pre> Build docker image with docker build --ssh github=$SSH_PRIVATE_KEY_PATH -t $IMAGE_NAME .</code></li> </ol> GitHub Actions</a> </h2> In GitHub actions, #1 is translated as: - name: Setup SSH Keys and known_hosts run: | mkdir -p ~/.ssh ssh-keyscan github.com >> ~/.ssh/known_hosts ssh-agent -a $SSH_AUTH_SOCK > /dev/null ssh-add - <<< "${{ secrets.SSH_PRIVATE_KEY }}" env: SSH_AUTH_SOCK: /tmp/ssh_agent.sock - name: Build, tag, and push image to Amazon ECR uses: docker/build-push-action@v2 env: SSH_AUTH_SOCK: /tmp/ssh_agent.sock with: context: . push: true tags: ${{ steps.meta.outputs.tags }} cache-from: type=gha cache-to: type=gha,mode=max ssh: | github=${{ env.SSH_AUTH_SOCK }}</code></pre> Voila 🎉 Use Pyspark locally with Docker Unknown — Tue, 21 Dec 2021 00:00:00 +0000 For data that doesn't fit into memory, spark is often a recommended solution, since it can utilize map-reduce to work with data in a distributed manner. However, setting up local spark development from scratch involves multiple steps, and definitely not for a faint of heart. Thankfully using docker means you can skip a lot of steps 😃 Instructions</a> </h2> Install Docker Desktop</a></li> Create docker-compose.yml</code> in a directory somewhere</li> </ol> version: "3.3" services: pyspark: container_name: pyspark image: jupyter/pyspark-notebook:latest ports: - "8888:8888" volumes: - ./:/home/jovyan/work</code></pre> Run docker-compose up</code> from the same folder where the above file is located.</li> You should see something like this. It's the same from running jupyter notebook</code> locally. Click the link at the end to access jupyter notebook.</li> </ol> Creating pyspark ... done Attaching to pyspark pyspark | WARNING: Jupyter Notebook deprecation notice https://github.com/jupyter/docker-stacks#jupyter-notebook-deprecation-notice. pyspark | Entered start.sh with args: jupyter notebook pyspark | /usr/local/bin/start.sh: running hooks in /usr/local/bin/before-notebook.d as uid / gid: 1000 / 100 pyspark | /usr/local/bin/start.sh: running script /usr/local/bin/before-notebook.d/spark-config.sh pyspark | /usr/local/bin/start.sh: done running hooks in /usr/local/bin/before-notebook.d pyspark | Executing the command: jupyter notebook pyspark | [I 12:36:04.395 NotebookApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret pyspark | [W 2021-12-21 12:36:05.487 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. pyspark | [W 2021-12-21 12:36:05.488 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. pyspark | [W 2021-12-21 12:36:05.488 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. pyspark | [W 2021-12-21 12:36:05.488 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release. pyspark | [I 2021-12-21 12:36:05.497 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.9/site-packages/jupyterlab pyspark | [I 2021-12-21 12:36:05.498 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab pyspark | [I 12:36:05.504 NotebookApp] Serving notebooks from local directory: /home/jovyan pyspark | [I 12:36:05.504 NotebookApp] Jupyter Notebook 6.4.6 is running at: pyspark | [I 12:36:05.504 NotebookApp] http://bd20652c22d3:8888/?token=408f2020435dfb489c8d2710736a83f7a3c7cd969b3a1629 pyspark | [I 12:36:05.504 NotebookApp] or http://127.0.0.1:8888/?token=408f2020435dfb489c8d2710736a83f7a3c7cd969b3a1629 pyspark | [I 12:36:05.504 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). pyspark | [C 12:36:05.509 NotebookApp] pyspark | pyspark | To access the notebook, open this file in a browser: pyspark | file:///home/jovyan/.local/share/jupyter/runtime/nbserver-7-open.html pyspark | Or copy and paste one of these URLs: pyspark | http://bd20652c22d3:8888/?token=408f2020435dfb489c8d2710736a83f7a3c7cd969b3a1629 pyspark | or http://127.0.0.1:8888/?token=408f2020435dfb489c8d2710736a83f7a3c7cd969b3a1629</code></pre> This snippet volumes: - ./:/home/jovyan/work</code></pre> means that anything you put in [the folder where docker-compose.yml</code> is] can be accessed by [jupyter notebook running inside docker], and what you do from inside jupyter notebook can be seen on the host system too. See? Easy as a 🥧. Reduce Docker image size with Alpine Unknown — Sun, 19 Dec 2021 00:00:00 +0000 Creating scripts are easy. But creating a small docker image is not 😅. Not all Linux flavors are created equal, some are bigger than others, etc. But this difference is very crucial when it comes to reducing docker image size. A simple bash script docker image</a> </h2> Given a Dockerfile (change apk</code> to apt</code> for ubuntu</code>): FROM alpine:3 WORKDIR /app RUN apk update && apk add jq curl COPY water-cut-notify.sh ./ ENTRYPOINT ["sh", "/app/water-cut-notify.sh"]</code></pre>Base image</th> Docker image size</th></tr></thead> alpine</td> 11.1MB</td></tr> ubuntu</td> 122MB</td></tr> </tbody></table> Ubuntu image size is 1099%</code> larger!!!!!! What about a light python image?</a> </h2> FROM python:3.9-alpine WORKDIR /app RUN pip install requests</code></pre>Base image</th> Docker image size</th></tr></thead> alpine</td> 53.6MB</td></tr> ubuntu</td> 920MB</td></tr> </tbody></table> Ubuntu image size is 1716%</code> larger!!!!!! Should you use alpine image for everything?</a> </h2> From above two experiments, we get: It's obvious that alpine has a very significant lighter footprint than ubuntu. But don't use alpine image for everything. For bash and go, using alpine results in lighter footprint. But for python apps, it's better to go with debian-based images. This article</a> explains in details why it's so. Secrets management with SOPS, AWS Secrets Manager and Terraform Unknown — Tue, 30 Nov 2021 00:00:00 +0000 Correction 2023-07-06: I only recently realized SSM and Secrets Manager are not the same. At my organization we use sops to check in encrypted secrets into git repos. This solves plaintext credentials in version control. However, say, you have 5 repos using the same database credentials, rotating secrets means you have to go into each repo and update the SOPS credentials manually. Also worth nothing that, for GitHub actions, authenticating AWS means you have to add repo secrets. This means for all the repos you have CI enabled, you have to populate the repo secrets with AWS credentials. When time comes for rotating the creds, you'll encounter the same situation as above. I did some research and consensus for AWS / Terraform setup is to: encrypt secrets via SOPS, and use Terraform to create AWS secret entries. That way, you have a trail for credentials. This setup means: You don't have to populate repos with AWS creds, instead supplying an ARN role instead.</li> You don't have to change credentials in projects, since they all get the secrets from AWS Secrets Manager.</li> </ol> Implementation</a> </h2> Repo here: https://github.com/kahnwong/terraform-sops-ssm</a> 1. Bootstrap Terraform</a> </h3> terraform { required_providers { sops = { source = "carlpett/sops" version = "0.6.3" } } } provider "aws" { region = "ap-southeast-1" profile = "playground" } provider "sops" {} terraform { required_version = ">= 1.0" }</code></pre>2. Create KMS key for SOPS</a> </h3> https://github.com/mozilla/sops/#kms-aws-profiles</a> resource "aws_kms_key" "sops" { description = "Keys to decrypt SOPS encrypted values" } resource "aws_kms_alias" "sops" { name = "alias/sops" target_key_id = aws_kms_key.sops.key_id }</code></pre>3. Create secrets</a> </h3> Create a folder named secrets</code>, inside it create JSON files and encrypt each with sops. locals { secrets = toset([ "db-foo", ]) } data "sops_file" "sops_secrets" { for_each = local.secrets source_file = "secrets/${each.key}.sops.json" } # aws keeps the secrets for 7 days before actual deletion. consider using random names during test resource "aws_secretsmanager_secret" "ssm_secrets" { for_each = local.secrets name = each.key } resource "aws_secretsmanager_secret_version" "ssm_secrets" { for_each = local.secrets secret_id = aws_secretsmanager_secret.ssm_secrets["${each.key}"].id secret_string = jsonencode(data.sops_file.sops_secrets["${each.key}"].data) }</code></pre>4. Create IAM policy for SSM access</a> </h3> data "aws_iam_policy_document" "secrets_ro" { statement { actions = [ "secretsmanager:GetResourcePolicy", "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret", "secretsmanager:ListSecretVersionIds" ] resources = [ "arn:aws:secretsmanager:ap-southeast-1:$AWS_ACCOUNT_ID:secret:*", ] } statement { actions = [ "secretsmanager:ListSecrets" ] resources = ["*"] } } resource "aws_iam_policy" "secrets_ro" { name = "secrets_ro" path = "/" policy = data.aws_iam_policy_document.secrets_ro.json }</code></pre>5. Create IAM user for local dev</a> </h3> You shouldn't supply AWS credentials for deployment, since you can grant access via IAM roles instead. resource "aws_iam_user" "playground-prod-dev" { name = "playground-prod-dev" path = "/users/" } resource "aws_iam_access_key" "playground-prod-dev" { user = aws_iam_user.playground-prod-dev.name }</code></pre>Grant IAM user access to Secrets</a> </h4> resource "aws_iam_user_policy_attachment" "playground-prod-dev" { user = aws_iam_user.playground-prod-dev.name for_each = toset([ aws_iam_policy.secrets_ro.arn, ]) policy_arn = each.value }</code></pre>6. Create IAM role for Lambda</a> </h3> resource "aws_iam_role" "lambda_role" { name = "lambda_role" path = "/sa/" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [ { "Effect" : "Allow", "Principal" : { "Service" : "lambda.amazonaws.com" }, "Action" : "sts:AssumeRole" }, ] }) managed_policy_arns = [ aws_iam_policy.secrets_ro.arn, ] inline_policy { name = "create_cloudwatch_logs" policy = jsonencode({ "Version" : "2012-10-17", "Statement" : [ { "Action" : [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents" ], "Effect" : "Allow", "Resource" : "*" }, ] }) } }</code></pre>7. Create IAM role for GitHub actions</a> </h3> Need to create OIDC so GitHub can assume AWS roles resource "aws_iam_openid_connect_provider" "github" { url = "https://token.actions.githubusercontent.com" client_id_list = ["sts.amazonaws.com"] thumbprint_list = ["a031c46782e6e6c662c2c87c76da9aa62ccabd8e"] }</code></pre> Assume role policy is on per-repo basis locals { repositories = [ "terraform-sops-ssm", ] } data "aws_iam_policy_document" "github_actions_assume_role" { statement { actions = ["sts:AssumeRoleWithWebIdentity"] principals { type = "Federated" identifiers = [aws_iam_openid_connect_provider.github.arn] } condition { test = "ForAnyValue:StringLike" variable = "token.actions.githubusercontent.com:sub" values = [for v in local.repositories : "repo:kahnwong/${v}:*"] } } }</code></pre> Finally attach the above policy to role resource "aws_iam_role" "playground-prod-github" { name = "playground-prod-github" path = "/sa/" assume_role_policy = data.aws_iam_policy_document.github_actions_assume_role.json managed_policy_arns = [ aws_iam_policy.secrets_ro.arn, ] }</code></pre> The end 🎉 Bonus</a> </h2> GitHub actions with AWS login</a></li> AWS Lambda with SSM template</a></li> </ul> Run GitHub Actions faster with cache for Pipenv and Docker build Unknown — Tue, 09 Nov 2021 00:00:00 +0000 Update 2021-11-29 Recently we create more PRs, notice that there are a lot of redundant steps (env setup before triggering checks, etc). Found out you can cache steps in GitHub Actions, so I did some research. Got it working and turns out I reduce at least 60% actions time for a large docker image build (since only the later RUN</code> directives are changed more frequently). For pipenv it shaved off 1 minute 18 seconds. Pretty neat! pipenv cache</a> </h3> # https://github.com/actions/setup-python#caching-packages-dependencies - uses: actions/setup-python@v2 with: python-version: "3.9" cache: "pipenv"</code></pre>docker build cache</a> </h3> - name: Set up Docker Buildx uses: docker/setup-buildx-action@v1 - name: Add IMAGE_TAG to env run: echo "IMAGE_TAG=`echo ${GITHUB_SHA::7}`" >> $GITHUB_ENV - name: Build, tag, and push image to Amazon ECR uses: docker/build-push-action@v2 env: IMAGE_NAME: service/app with: context: . push: true tags: ${{ env.IMAGE_NAME }}:latest,${{ env.IMAGE_NAME }}:${{ env.IMAGE_TAG }} cache-from: type=gha cache-to: type=gha,mode=max</code></pre> ecs-cli snippets Unknown — Fri, 08 Oct 2021 00:00:00 +0000 ecs-cli configure profile \ --access-key $KEY \ --secret-key $SECRET \ --profile-name $PROFILE ### launch mode: fargate ecs-cli configure \ --cluster $CLUSTER \ --default-launch-type FARGATE \ --config-name $NAME \ --region ap-southeast-1 ecs-cli up \ --cluster-config $NAME \ --vpc $VPCID\ --subnets $SUBNETID1, $SUBNETID2 ### launch mode: ec2 ecs-cli configure \ --cluster $CLUSTER \ --region ap-southeast-1 \ --default-launch-type EC2 \ --config-name $NAME ecs-cli up --keypair $KEYPAIR \ --extra-user-data userData.sh \ --capability-iam --size 1 \ --instance-type t2.large \ --cluster-config $NAME \ --verbose \ --force \ --aws-profile $PROFILE ecs-cli compose \ --cluster-config $NAME \ --file docker-compose.yml up \ --create-log-groups</code></pre> Self-hosting primer Unknown — Sun, 22 Aug 2021 00:00:00 +0000 Updated: 2025-01-26 Self-hosting is a practice for running and managing websites / services using your own server. Some people do this because they are concerned about their privacy, or some services are free if they host it themselves. Below are instructions for how to do self-hosting (also applies to hosting your own website too). Requirements</a> </h2> Domain name</li> Server (can be your own computer at home or VPS)</li> </ul> Instructions</a> </h2> Set up and secure the server. Don't forget to disable password login.</li> Deploy a website on your server (follow instructions for each service. I recommend deploying via Docker).</li> If you are using a server at home which has dynamic IP, setup DDNS (I recommend duckdns.org</del> Cloudflare DNS, since it has very fast TTL). See https://github.com/favonia/cloudflare-ddns</a>. Set it to update ddns</code> CNAME.</li> Go to your domain name registrar, under DNS, add a CNAME record for your desired subdomain, and set the value to your DDNS (example: ddns.yourdomain.com</code>)</li> On your server, install a webserver for reverse-proxy. I recommend nginx or Caddy.</li> Create a virtual host config for your website in your webserver of choice.</li> On your router configuration page, under port forwarding, create two entries for port 80 and 443.</li> Wait for a few minutes for the DNS to be updated, and you should be able to access your website from the specified domain.</li> </ol> As for actual implementation, I suggest you read a few articles for each step, so you can get the overall idea of what's to be done. Generally, the common steps should be the same across all articles, since that's the "baseline" for each process. Python venv management Unknown — Fri, 02 Jul 2021 00:00:00 +0000 Update 2023-09-09: I now revert back to plain requirements.txt</code>, since pipenv is very poor at resolving large dependencies list. Poetry still remains funky so that's off the table as well. Update 2024-01-20: I've been using poetry in production for a few months now and so far it works great! Looks like they ironed out a lot of rough edges. Although I would still recommend people to use pre-commit to generate plain requirements.txt</code>, this is so that in ci or dockerfile you don't have to install poetry and set in-project venv every time. When you create a project in python, you should create requirements.txt</code> to specify dependencies, so other people can have the same environment when using your project. However, if you don’t specify module versions in requirements.txt</code>, you could end up with people using the wrong module version, where some APIs can be deprecated or have different behaviors than older versions. Another issue is that maybe you’re working on a few python projects, each uses different python versions (eg. projectA uses python3.6, projectB uses python3.9, etc). Enter pyenv</code> and poetry</code> (pipenv</code> is considered outdated now), where you can easily switch python versions, and have different environment (with python version locking) for projects you’re working on. Setup pyenv</a> </h2> Follow instructions here</a>. For windows, use this</a>. Useful commands</a> </h3> # list available python versions pyenv install --list # install specific version pyenv install 3.8.0 # list installed versions pyenv versions # activate new env pyenv shell 3.8.0 # support multiple version # config venv pyenv virtualenv 3.8.0 my-data-project # set env per folder/project pyenv local my-data-project</code></pre>Setup pipenv</a></a> </h2> 2024-01-20: See below for poetry, this only stays for reference only Notes: make sure pyenv</code> is installed, and remove anaconda / miconda</code> & python3</code> installed via official installer from your system. Then run: $ pip install pipenv # run this command every time pip installs a .exe $ pyenv rehash</code></pre>pipenv workflow</a> </h3> pipenv --python 3.7 # install a specific module pipenv install jupyterlab==2.2.9 # install from existing requirements.txt or from Pipfile definition pipenv install # remove venv pipenv --rm # running inside venv pipenv run jupyter lab pipenv run python main.py # is equivalent to `pipenv shell && python3 main.py`</code></pre>Windows only</a> </h2> 2024-01-20: See below for poetry, this only stays for reference only pyenv install 3.7.7 # see Pipfile for required python version pyenv local 3.7.7 # IMPORTANT. global / shell doesn't work with pipenv pyenv rehash pip install pipenv # done once per pyenv python version pyenv rehash pipenv --python 3.7 pipenv install pipenv run python tokenization_sandbox.py</code></pre>Setup poetry</a></a> </h2> pipx install poetry</code></pre>Useful commands</a> </h3> # init project poetry init # add deps poetry add $package # add dev dependencies poetry add $package --group dev # activate venv poetry shell # specify python version poetry env use 3.11 # normally it picks up global python via pyenv</code></pre>Notes</a> </h2> On linux/mac, do not use system python. OS updates would mean python version upgrade, in turn making all your installed modules gone. Use python installed via pyenv instead.</li> On windows, start fresh with pyenv.</li> Do not use anaconda distribution. It does too much background magic that can make things harder to manage environment properly. In addition, venv definition from anaconda is often doesn’t work cross-platform (eg. venv def from windows wouldn’t work on mac due to different wheel binary versions).</li> Always create venv via pipenv per each project. Although you can have a playground venv via pyenv, so you can shell into it and do a quick analysis / scripting on an adhoc basis.</li> I heard good things about poetry</a> but it doesn’t integrate with pyenv</code> natively. It would work if you use it to publish python modules, since it simplifies a lot of processes.</del> poetry also picks up the wrong python version from pyenv. And if you sync python version via pyenv, it has to be the same python version across all OSes, including minor version. pipenv doesn’t have this restriction, and it also picks up the correct python version from pyenv by default (via pipenv --python 3.8</code>).</del></li> </ul> </li> </ul> Don't write large table to Postgres with Pandas Unknown — Sun, 27 Jun 2021 00:00:00 +0000 We have a few tables where the data size is > 3GB (in parquet, so around 10 GB uncompressed). Loading it into postgres takes an hour. (Most of our tables are pretty small, hence the reason why we don't use columnar database). I want to explore whether there's a faster way or not. The conclusion is writing to postgres with spark seems to be fastest, given we can't use COPY</code> since our data contain free text, which means it would make CSV parsing impossible. I also found out that the write performance from pandas to postgres is excruciatingly slow because: It first decompresses the data in-memory. For a 30MB parquet (around 100MB uncompressed) it used more than 20GB of RAM (I killed the task before it finishes, since by this time the RAM usage is climbing up) But even with reading plain JSON line in pandas with chunksize and use to_sql</code> with multi</code> option, it's still very slow. In contrast, writing the said 30MB parquet file to postgres takes only 1 minute. Big data is fun, said data scientists 🧪 (until they run out of RAM 😆) Data engineering toolset (that I use) glossary Unknown — Fri, 04 Jun 2021 00:00:00 +0000 Big data</a> </h2> Spark: Map-reduce framework for dealing with big data, especially for data that doesn't fit into memory. Utilizes parallelization.</li> </ul> Cloud</a> </h2> AWS: Cloud platform for many tools used in software engineering.</li> AWS Fargate: A task launch mode for ECS task</code>, where it automatically shuts down once a container exits. With EC2 launch mode, you'll have to turn off the machine yourself.</li> AWS Lambda: Serverless function, can be used with docker image too. Can also hook this with API gateway to make it act as API endpoint.</li> AWS RDS: Managed databases from AWS.</li> ECS Task: Launch a task in ECS cluster. For long-running services, launch via EC2. For small periodical tasks, trigger via Cloudwatch. For the latter, think of cron-like schedule for a task. Essentially at specified time, it runs a predefined docker image (you should configure your entrypoint.sh</code> accordingly).</li> </ul> Data</a> </h2> Parquet: Columnar data blob format, very efficient due to column-based compression with schema definition baked in.</li> </ul> Data engineering</a> </h2> Dagster: Task orchestration framework with built-in pipelines validation.</li> ETL: Stands for extract-transform-load. Essentially it means "moving data from A to B, with optional data wrangling in the middle."</li> </ul> Data science</a> </h2> NLP: Using machine (computer) to work on human languages. For instance, analyze whether a message is positive or negative.</li> </ul> Data wrangling</a> </h2> Pandas: Dataframe wrangler, think of programmable Excel.</li> </ul> Database</a> </h2> Postgres: RMDBS with good performance.</li> </ul> DataOps</a> </h2> Great expectations: A framework for data validation.</li> </ul> DevOps</a> </h2> Docker: Virtualization via containers.</li> Git: Version control.</li> Kubernetes: Container orchestration system.</li> Terraform: Infrastructure as code tool, essentially you use it to store a blueprint for your infra setup. If you were to move to another account, you can re-conjure existing infra with one command. This makes editing infra config easier too, since it automatically cleans up / update config automatically.</li> </ul> GIS</a> </h2> PostGIS: GIS extension for Postgres.</li> </ul> MLOps</a> </h2> MLflow: A framework to track model parameters and output. Can also store model artifact as well.</li> </ul> Notebook</a> </h2> Jupyter: Python notebook, used for exploring solutions before converting it to .py.</li> </ul> Automatic Scrapy deployment with GitHub actions Unknown — Wed, 02 Jun 2021 00:00:00 +0000 Repo here</a> Scrapy is a nice framework for web scraping. But like all local development processes, some settings / configs are disabled. This wouldn't pose an issue, but to deploy a scrapy project to zyte (a hosted scrapy platform) you need to run shub deploy</code>, and if you run it and forget to reset the config back to prod settings, a Titan may devour your home. You can set auto deployment from github via the UI in zyte, but it only works with github only. Plus if you want to run some extra tests during CI/CD you're out of luck. So here's how to set up CI/CD to deploy automatically: Note: I would assume that you have your scrapy project set up already. Create scrapinghub.yml + add repo secrets</a> </h2> project: ${PROJECT_ID} requirements: file: requirements.txt stack: scrapy:${YOUR_SCRAPY_VERSION_IN_PIPFILE} apikey: null</code></pre> Notice that apikey</code> is left blank. This is because it's considered a good practice to not check in sensitive information & credentials in version control. Instead apikey</code> will be added to github secrets, so it can be called as environment variable. Create github workflow file</a> </h2> name: Deploy on: push: branches: [master, main] jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Set up Python 3.9 uses: actions/setup-python@v2 with: python-version: 3.9 - name: Install dependencies run: | python -m pip install --upgrade pip pip install pyyaml shub - name: Deploy to zyte if: github.ref == 'refs/heads/master' run: python3 utils/edit_deploy_config.py && shub deploy env: APIKEY: ${{ secrets.APIKEY }}</code></pre> Translation: On push to this repo (this doesn't work for PRs)</li> Download this repo</li> Setup python3.9</li> Install some pip modules</li> Run a script to overwrite scrapinghub.yml's apikey value, in which the value is obtained from github secrets</li> Execute deploy command</li> </ul> Elasticsearch with custom dictionary Unknown — Mon, 03 May 2021 00:00:00 +0000 Elasticsearch is a search engine with built-in analyzers (combination of tokenizer and filters), which makes it easier to set it up and get it running, seeing you don’t have to implement NLP logic from scratch. However, for some languages such as Thai, the built-in Thai analyzer may not be working quite as expected. For instance, for region name search autocomplete, it doesn’t recommend anything when I type เชียง</code>, but it should be showing เชียงใหม่</code> or เชียงราย</code>. This is because these two regions are recognized as one token, which is why it doesn’t recommend anything when querying with เชียง</code>. But if I create a custom dictionary for tokenizers with เชียง</code> as one of the tokens, it manages to recommend the two regions when querying with the prefix. Below is an index_config</code> for using a custom dictionary for tokenizer: { "settings": { "analysis": { "analyzer": { "thai_dictionary": { "tokenizer": "standard", "filter": ["char_limit_dictionary"], } }, "filter": { "char_limit_dictionary": { "type": "dictionary_decompounder", "word_list": tokens, # <-- word list array here "max_subword_size": 22, } }, } }, "mappings": { "properties": { "title": { # <-- search key "type": "text", "analyzer": "thai_dictionary", } } }, }</code></pre> See elasticsearch documentation for more details: https://www.elastic.co/guide/en/elasticsearch/reference/7.12/analysis-dict-decomp-tokenfilter.html</a> Shapefile to data lake Unknown — Fri, 23 Apr 2021 00:00:00 +0000 Background: we use spark to read/write to data lake. For dealing with spatial data & analysis, we use sedona</a>. Shapefile is converted to TSV then read by spark for further processing & archival. Recently I had to archive shapefiles in our data lake. It wasn't rosy for the following reasons: Invalid geometries</a> </h2> Sedona (and geopandas too) whines if it encounters invalid geometry</code> during geometry casting. The invalid geometries could be from many reasons, one of them being unclean polygon clipping. Solution: use gdal</code> to filter out invalid geometries. Spatial projection</a> </h2> Geometric projections requires projection</code>, otherwise you could be on the wrong side of the globe. This matters because by default, the worldwide-coverage projection is EPSG:4326</code>, but the unit is in degrees</code>, so sometimes for analysis the data is converted to a local projection which covers a smaller geographical region, but uses meter</code> as the unit. This means that if the source projection is in A</code>, and you didn't cast it to EPSG:4326</code>, spark would mistakenly think it's on EPSG:4326</code> by default. Something like seeing the entirely of the UK in Africa. Solution: verify the source projection and cast to EPSG:4326</code> before writing to data lake. Extra new line character</a> </h2> Sometimes when editing shapefile data by hand using applications like ArcGIS or QGIS, you could copy a text which might contain "new line" character, and set it as a cell value. Spark doesn't play nice with "new line" characters in a middle of a record. Solution: strip new line characters by hand. Yes, I really did that 😶. Thankfully it was a very small shapefile that has the issue. Takeaways: count yourself lucky if you never have to deal with spatial data. Spark join OOM fix Unknown — Sun, 11 Apr 2021 00:00:00 +0000 I have a big pipelines where one step performs crossjoin on 130K x 7K</code>. It fails quite often, and I have to pray to the Rice God for it to pass. Today I found the solution: repartition</code> before crossjoin. The root cause is that the dataframe with 130K records has 6 partitions, so when I perform crossjoin (one-to-many) it's working against those 6 partitions. Total output in parquet is around 350MB, which means my computer (8 cores, 10GB RAM provisioned for spark) needs to be able to hold all uncompressed data in memory. It couldn't hence the frequent OOM. So by increasing the partition size from 6 to 24, the current working dataframe size is smaller, which means things could pass along faster while not filling up my machine's RAM. Add Ghost content to Hugo Unknown — Wed, 31 Mar 2021 00:00:00 +0000 Ghost CMS is very easy to use, but the deployment overhead (maintaining db, ghost version, updates and etc) might be too much for some. Luckily, there's a way to convert a Ghost site to static pages, which you can later host on Github pages or something similar. Setup</a> </h2> static site engine: Hugo</li> a Ghost instance</li> </ul> Usage</a> </h2> Install https://github.com/Fried-Chicken/ghost-static-site-generator</a></li> cd</code> to static</code> directory in your Hugo folder</li> run</li> </ol> gssg --domain ${YOUR_GHOST_INSTANCE_URL} --dest posts --url ${YOUR_STATIC_SITE_DOMAIN_WITHOUT_TRAILING_SLASH} --subDir posts</code></pre> Update your hugo config to link to the above folder:</li> </ol> [[menu.main]] identifier = "posts" name = "Posts" url = "/posts"</code></pre> All done! 🎉🎉🎉 Hello Caddy Unknown — Sun, 07 Mar 2021 00:00:00 +0000 Since starting self-hosting back in 2017, I've always used apache2 since it's the first webserver I came across. Over time adding more services and managing separate vhost config is a bit tiresome. Enter Caddy. It's very simple to set up and configure. Some services where I have trouble setting up in apache2 do not need extra config at all, even TLS is set up by default. Starting from Caddy2 it works with CNAME by default without extra setups. You can set it up using a Caddy docker container, but some containers I use also expose port 443, so I have to install Caddy natively instead. For multiple sites config setup: # /etc/caddy/Caddyfile SUBDOMAIN1.DOMAIN.com { reverse_proxy 127.0.0.1:${PORT} } SUBDOMAIN2.DOMAIN.com { reverse_proxy 127.0.0.1:${PORT} }</code></pre> For basic authentication, it's very, very simple (to the point I regret time researching it in apache2): # generate password hash caddy hash-password --algorithm bcrypt # add basicauth to Caddyfile SUBDOMAIN1.DOMAIN.com { basicauth * { ${USERNAME} ${CADDY_PASSWORD_HASH} } reverse_proxy 127.0.0.1:${PORT} }</code></pre> And run systemctl reload caddy</code>. You're all set! Password auth with apache2 reverse-proxy Unknown — Mon, 22 Feb 2021 00:00:00 +0000 EDIT: see here</a> for Caddy, also easier to set up too. Sometimes you found an interesting project to self-hosted, but it doesn't have password authentication built-in. Luckily, we need to reverse-proxy them anyway and apache2/ nginx / httpd happen to provide password auth with reverse-proxy by default. To set up password auth with apache2 via reverse-proxy: echo "${PASSWORD}" | htpasswd -c -i /etc/apache2/.htpasswd ${USER}</code> on your host machine which has apache2 installed.</li> create a vhost config:</li> </ol> <VirtualHost *:80> ProxyPreserveHost On ProxyPass / http://localhost:${EXPOSED_CONTAINER_PORT}/ ProxyPassReverse / http://localhost:${EXPOSED_CONTAINER_PORT}/ ServerName ${YOUR_DOMAIN} <Proxy *> Order deny,allow Allow from all Authtype Basic Authname "Password Required" AuthUserFile /etc/apache2/.htpasswd Require valid-user </Proxy> </virtualhost></code></pre> That's it! Buying tea when you have Celiac Unknown — Sun, 14 Feb 2021 00:00:00 +0000 Might come as a surprise to some of you, but tea can contain gluten from additives & cross-contamination, in which barley or malt is added for flavorings. Teavana is known for adding such additives in their tea (and I got glutented from it one time). Say, I'm interested in some tea. I first need to look up its country of origin, because that tells me about how good the food labeling laws is. If it's from Australia / New Zealand, this means the label will always state gluten. If it's from EU (not pan-Europe) it's also good, except they allow products with < 20 PPM to be labeled gluten-free. But this is still good. If it's from USA, a lot of scrutiny is called because the US FDA does not enforce gluten labeling, only wheat is required. But not all gluten-y stuff is wheat :/ In this case, I'm interested in Turkish tea. So I asked a Turkish friend to do some digging for me on the manufacturer's website, since I couldn't find the info in English. He reported that he didn't find any info about gluten on the website, but he send me a photo of the package where the ingredients are listed. I spotted German on the labels, this is a good sign because it means this product is also sold in Germany, and food labels have to comply with the country it's being sold in too*. That's pretty much end of story. Ordered the tea and brew it. Tastes very good! *Thank you a local Thai QA who told me about this 🙏 Workarounds for archiving large shapefile in data lake Unknown — Sun, 31 Jan 2021 00:00:00 +0000 If you work with spatial data, chances are you are familiar with shapefile</code>, a file format for viewing / editing spatial data. Essentially, shapefile is just a tabular data like csv</code>, but it does thingamajig with geometry</code> data type, where any gis tools like qgis</code> or arcgis</code> can understand right away. If you have a csv file with geometry column in wkt</code> format (something like POLYGON ((30 10, 40 40, 20 40, 10 20, 30 10))</code>), you'll have to specify which column is to be used for geometry. If you want to store shapefile in data lake, it's best to store it as parquet or any format you normally use, since it's faster to read and filter. For comparison, parsing a 5GB+ shapefile and filter takes longer than reading a gzipped json, filter, and export to shapefile. Normally I would use geopandas</code> to read spatial data and convert it to pandas dataframe, then send it to spark. But since the shapefile is very large, it takes forever to read in geopandas. This tells me that there is a parsing bottleneck going on. And geopandas can't read shapefile with multiple geometry types (this shouldn't happen, but sometimes during editing, clipping this here and there can cause invalid geometry). Qgis has a tool to fix invalid geometries, so I tried exporting shapefile to csv, but qgis went OOM. But both qgis and geopandas use gdal</code> for backend, and it has a CLI interface, so I look up how to export shapefile to tsv (tab</code> as a separator makes it faster to parse since it rarely occurs). Now things work perfectly. As a bonus, gdal also skip invalid geometries by default (unlike in geopandas where it will throw an error and there's no way to ignore it and tell the parser to keep going). At this point I have a nice tsv file, and reading & archiving via spark is now a breeze. Yay! Takeaway</a> </h2> If it takes too long to read, maybe it's a parsing bottleneck. Find a way to convert it to another format so it's easier to parse.</li> Sometimes your initial tools of choice might have some quirks. In most cases there will be similar tools out there that can workaround the issues. (In this case, use gdal to convert to csv in lieu of geopandas because gpd can't work with invalid geometries & takes longer to read compared to feeding spark a straight csv/tsv).</li> </ul> Mongodb export woes Unknown — Wed, 27 Jan 2021 00:00:00 +0000 There's a task where I need to export 4M+ records out of mongodb, total uncompressed size is 17GB+</del> 26GB export methods</a> </h2> mongoexport</a> </h3> The recommended way to export is using mongoexport</code> utility, but you have to specify the output attributes, which doesn't work for me because the schema from older set of records are less than the newer set DIY python script</a> </h3> the vanilla way</a> </h4> But you can interact with mongodb from python, and if you read from it it'll return a dict, which is perfect for this because you don't have to specify the required attributes beforehand. So what I do is: cursor = collection.find({}) total_records = collection.estimated_document_count() with open(filename, "w") as f: for i in tqdm(cursor, total=total_records): f.write(json.dumps(i, default=myconverter, ensure_ascii=False)) f.write("\n")</code></pre> The cons for this solution is it needs a lot of hdd space since it's uncompressed. But it works best if you need to export a collection with mismatched schema. the incremental export way</a> </h4> You can also incrementally export your collection from mongodb using .skip($START_INDEX).limit($INCREMENT_SIZE)</code> , but it performs worse than the vanilla way, since what mongodb does is just iterating through everything all over again to get to your specified start:end</code> index. Performance comparison</a> </h2> On my local machine (<10 MB/s transfer speed) I could export a collection with around 4.5M records within 1 hour, but on a VPS with incremental export it takes 9 hours and counting. Takeaway</a> </h2> Please do not store a large dataset in mongodb where you need to dump everything out, especially if you use it as a raw data source. It's fine if you store prepped output for API to be queried via _id</code> (primary key). CPU upgrade is a breeze, only if you know how Unknown — Sun, 20 Dec 2020 00:00:00 +0000 I want to update my pc config to use a more powerful cpu (from Ryzen 1300x --> Ryzen 3600). Watched a few videos and it didn't look complicated enough, so I order the new CPU and plan to install it myself. Things didn't go exactly to plan. To sum up relevant things I didn't know from watching "how to install AM4 CPU" videos: on some motherboard, you need to "support" the backplate from under the motherboard so the socket would protrude far up enough to "lock" the CPU fan.</li> even if ab350 says it supports Ryzen 3rd generation CPUs after flashing the supported BIOS version, it is not a given that it will work --> I end up upgrading my motherboard to b450</li> if you boot Windows after changing CPU / GPU, it won't boot --> have to reset CMOS</li> </ul> Of course, I didn't figure all this out myself. I took the pc to a local repair shop to install the CPU fan but it didn't POST. We thought it's the from the CPU pins I accidentally bent and maybe I screw it up when I straighten it back. But it's still within 7 days of purchase, so I took it to the retailer for a claim, only to find out that it works. The tech ask me which motherboard I have, only then it hit them that I need to upgrade the motherboard. Then I took it back to a repair shop to install the new motherboard + CPU, but it still doesn't POST. Figured it out later that you need to reset CMOS. The "how to install AM4 CPU" didn't lie or anything, it's just that all the above points are "basic knowledge" for everyone who works on pc hardware should know. It's the same as "try a newly installed CLI from a new terminal session" <– because your shell pre-index executables when it inits a session. Takeaway: don't underestimate domain knowledge basics. There are caveats behind every statement Unknown — Tue, 01 Dec 2020 00:00:00 +0000 You may have come across news headlines with something like "woman diagnoses herself with rare disease." If you take this statement at face value, you might think "hmmm so this woman, out of the blue, just diagnose herself with a rare disease." What most people don't often realize is that for a rare disease to be considered, she must have gone through many different doctors who diagnosed her with various diseases, but she feels that neither of them accurately describe her symptoms. She might do some research, and came across a rare disease that perfectly describe her symptoms. A more accurate (but more verbose) headline would be "doctors failed to accurately diagnose woman with rare disease". Takeaway? Think twice before taking anything at face value. Visualizing map region prefix/suffix Unknown — Thu, 03 Sep 2020 00:00:00 +0000 import geopandas as gpd import geoplot as gplt import matplotlib.pyplot as plt import numpy as np import pandas as pd from geoplot import polyplot from pythainlp.tokenize import syllable_tokenize, word_tokenize</code></pre>Data structure</a> </h2> name: target region name</li> geometry: spatial column</li> *: parent region name, e.g. in "district" dataset it would have a "province" column</li> </ul> Dissolving dataset in case you have multiple region level in the same file</a> </h2> ## assuming you have a district dataset and want to dissolve to province only district_filename = "FILE_PATH_HERE" gdf = gpd.read_file(district_filename) used_columns = [ "province", "district", ] gdf = gdf.rename( columns={ "prov_namt".upper(): "province", # change to dummy "amp_namt".upper(): "district", } ) gdf = gdf[used_columns + ["geometry"]] ## desired data 🛎🛎🛎 please do create a datasest with outermost region, so we can use it as boundary for visualization province = gdf.dissolve(by="province") province = ( province.reset_index().rename(columns={"province": "name"}).drop(columns="district") ) province</code></pre>

Karn Wong - Karn Wong's Blog

LLM tracing with MLflow AI Gateway

Garmin development setup on Linux

Development environment setup with flox and direnv

Observability with eBPF and OTel

Classical models vs neural networks for tabular data

Intro</a> </h2> I started working with machine But it stuck with me - given Experiment </h2> The dataset is NYC Taxi

Experiment </h2> The dataset is NYC Taxi

Experiment </h2> The dataset used are considered

Multi-account git setup

Book Highlights - Of Monsters and Mainframes by Barbara Truelove

Reduce workload attack surface with distroless image

ONNX multi-platform benchmark

Multi-machine work setup

Spatial join performance

Tools & Frameworks Used</a> </h2> DataFusion</li> DuckDB</li> Pandas (via GeoPandas)</li> Polars</li> PostGIS</li>

Static lib linking in Go program

AWS cost optimization tactics

Terraform project structure

The true cost of deploying on EC2

Dataframe processing benchmarks (ภาษาไทย)

Dataframe processing benchmarks

Simplify SSH (and HTTP & Postgres) access management with Warpgate

Using Nix for CI, but with a twist

Hello Garage, goodbye MinIO

Cloudflare's terraform v5 provider migration strategy

Model inference in ONNX

My code search setup throughout the years

WASM FFI performance benchmark

WASM deployment benchmark

Migrate from Docker Desktop to Podman on Darwin

Cross-platform keyring-like setup (also works without desktop environment)

My journey for fully switching from Windows over to Linux

Information gathering infrastructure

For feeds</a> </h2> Subscribe them via Miniflux, an RSS feed aggregator. It's hosted on my homelab.</li> For articles I want to read later, I send them to Wallabag via Miniflux integration. Walalbag is a Pocket-like app, self-hostable.</li>

Native implementation vs WASM for Go, Python and Rust benchmark

Various notes on career-related stuff

Skills</a> </h2> PM / PO / EM / Sales should understand technical stuff, as in they are aware of pros and cons of solutions being used. Bonus if they used to be an IC.</li>

Job change</a> </h2> If you don't learn something new at work for a while, maybe it's time to move.</li>

Calling C from Go, Python and Rust benchmark

Experiment </h2>
The dataset used are considered

For feeds </a> </h2>
For newsletters

Skills </a> </h2>
Career ladder </a> </h2>

Job change</a> </h2>
Career change