Archives

2024 ¹⁷

October ⁵

Various notes on career-related stuff

October 19, 2024 · 5 min · Karn Wong

Calling C from Go, Python and Rust benchmark

October 12, 2024 · 2 min · Karn Wong

LLM serving latency benchmark

October 9, 2024 · 2 min · Karn Wong

When (not) to use Terraform

October 5, 2024 · 3 min · Karn Wong

AWS IAM credentials best practices

October 5, 2024 · 2 min · Karn Wong

September ³

Hello World API Performance Benchmark (Go, Node, Python, Rust)

September 20, 2024 · 3 min · Karn Wong

Streamlit load test performance

September 7, 2024 · 2 min · Karn Wong

Simplify self-hosting backups to S3 with docker

September 7, 2024 · 4 min · Karn Wong

August ¹

Reasons why you shouldn’t use programming languages for IaC

August 5, 2024 · 2 min · Karn Wong

July ¹

GCP’s service account credentials can be a security risk. Here’s how to mitigate them.

July 14, 2024 · 2 min · Karn Wong

June ¹

Thoughts on summarization service system design

June 9, 2024 · 3 min · Karn Wong

April ²

Faster spark workloads with comet

April 7, 2024 · 2 min · Karn Wong

Slim down python docker image size with poetry and pip

April 7, 2024 · 2 min · Karn Wong

March ¹

Dataframe write performance to Postgres

March 17, 2024 · 2 min · Karn Wong

February ¹

How to connect to Cloud SQL from Cloud Run (no, you don’t need a VPC)

February 10, 2024 · 3 min · Karn Wong

January ²

What is platform engineering?

January 21, 2024 · 2 min · Karn Wong

Collaboration model for data science projects

January 20, 2024 · 2 min · Karn Wong

2023 ³⁰

December ³

Should data scientists deploy models to production?

December 30, 2023 · 2 min · Karn Wong

Setting up Postgres locally, what could go wrong?

December 23, 2023 · 3 min · Karn Wong

Things to watch out for GCP SSL with Cloudflare DNS

December 18, 2023 · 1 min · Karn Wong

November ⁴

Serverless real-time machine learning inference with AWS

November 28, 2023 · 3 min · Karn Wong

Some problems can be solved with workflows

November 24, 2023 · 3 min · Karn Wong

Using Apache Iceberg to reduce data lake operations overhead

November 15, 2023 · 4 min · Karn Wong

Reduce operational costs with terraform

November 4, 2023 · 3 min · Karn Wong

September ²

Spark on Kubernetes

September 12, 2023 · 4 min · Karn Wong

Data Engineering Resources

September 9, 2023 · 1 min · Karn Wong

August ³

A Networking God Tale: All I Want is to Run a Speedtest Behind a Firewall

August 27, 2023 · 2 min · Karn Wong

Spatial data to QGIS server playbook (yes, this is for prod)

August 10, 2023 · 2 min · Karn Wong

Create Kubernetes service accounts with Terraform

August 1, 2023 · 3 min · Karn Wong

July ³

pglogical setup

July 20, 2023 · 2 min · Karn Wong

Book Highlights - Build by Tony Fadell

July 6, 2023 · 4 min · Karn Wong

Book Highlights - Atlas of the Heart by Brene Brown

July 6, 2023 · 1 min · Karn Wong

June ¹

Hassle-free Kubernetes monitoring with Coroot

June 9, 2023 · 2 min · Karn Wong

April ⁷

Book Highlights - Managing Transitions: Making the Most of Change by William Bridges

April 27, 2023 · 2 min · Karn Wong

Use SQL against CSV (or other hard files) without CLI

April 25, 2023 · 2 min · Karn Wong

DevX starts at your local machine

April 22, 2023 · 4 min · Karn Wong

The mythical ChatOps in action

April 18, 2023 · 3 min · Karn Wong

DuckDB vs Polars vs Spark!

April 7, 2023 · 3 min · Karn Wong

Kubernetes with Grafana Cloud

April 1, 2023 · 2 min · Karn Wong

Cost optimization with kubernetes

April 1, 2023 · 2 min · Karn Wong

March ⁷

Bare metal works, until it doesn’t. Hello, cloud.

March 24, 2023 · 4 min · Karn Wong

Google Analytics v4 ingestion via BigQuery

March 19, 2023 · 7 min · Karn Wong

SecretOps with teller

March 19, 2023 · 2 min · Karn Wong

Data transformation - python vs sql showdown

March 18, 2023 · 1 min · Karn Wong

Terraform RDS module with DNS setup

March 17, 2023 · 2 min · Karn Wong

Book Highlights - Engineering Management for the Rest of Us by Sarah Drasner

March 16, 2023 · 4 min · Karn Wong

Book Highlights - The Staff Engineer’s Path by Tanya Reilly

March 15, 2023 · 12 min · Karn Wong

2022 ¹⁴

December ²

Load balancer 101, thou shalt not be alone

December 10, 2022 · 2 min · Karn Wong

Cross-platform package (+env) management with Nix

December 3, 2022 · 4 min · Karn Wong

November ⁴

Load credentials into your shell via Bitwarden CLI - Fish edition

November 29, 2022 · 1 min · Karn Wong

Cloudflare DNS management with Terraform

November 20, 2022 · 1 min · Karn Wong

Deploy more efficiently with templating

November 5, 2022 · 3 min · Karn Wong

IaC is cool, until someone specified the wrong machine size 💸

November 3, 2022 · 3 min · Karn Wong

October ²

Deploy static site with branch preview via Cloudflare Pages

October 5, 2022 · 2 min · Karn Wong

Terraform with ECS task on EC2 backend

October 4, 2022 · 2 min · Karn Wong

September ²

Intro to Dagster Cloud

September 27, 2022 · 3 min · Karn Wong

Intro to Pulumi

September 26, 2022 · 9 min · Karn Wong

August ²

Minimal ECS task with fargate backend

August 26, 2022 · 3 min · Karn Wong

Data engineer archtypes

August 26, 2022 · 2 min · Karn Wong

May ¹

What SQL can’t do for data engineering

May 15, 2022 · 3 min · Karn Wong

February ¹

Use SSH key during docker build without embedding the key via ssh-agent

February 6, 2022 · 2 min · Karn Wong

2021 ¹⁹

December ²

Use pyspark locally with docker

December 21, 2021 · 3 min · Karn Wong

Reduce docker image size with alpine

December 19, 2021 · 1 min · Karn Wong

November ²

Secrets management with SOPS, AWS Secrets Manager and Terraform

November 30, 2021 · 4 min · Karn Wong

Run GitHub Actions faster with cache for pipenv and docker build

November 9, 2021 · 1 min · Karn Wong

October ¹

ecs-cli snippets

October 8, 2021 · 1 min · Karn Wong

August ¹

Self-hosting primer

August 22, 2021 · 2 min · Karn Wong

July ¹

Python venv management

July 2, 2021 · 4 min · Karn Wong

June ³

Don’t write large table to postgres with pandas

June 27, 2021 · 1 min · Karn Wong

Data engineering toolset (that I use) glossary

June 4, 2021 · 2 min · Karn Wong

Automatic scrapy deployment with GitHub actions

June 2, 2021 · 2 min · Karn Wong

May ¹

Elasticsearch with custom dictionary

May 3, 2021 · 1 min · Karn Wong

April ²

Shapefile to data lake

April 23, 2021 · 2 min · Karn Wong

Spark join OOM fix

April 11, 2021 · 1 min · Karn Wong

March ²

Add Ghost content to Hugo

March 31, 2021 · 1 min · Karn Wong

Hello Caddy

March 7, 2021 · 1 min · Karn Wong

February ²

Password auth with apache2 reverse-proxy

February 22, 2021 · 1 min · Karn Wong

Buying tea when you have Celiac

February 14, 2021 · 2 min · Karn Wong

January ²

Workarounds for archiving large shapefile in data lake

January 31, 2021 · 2 min · Karn Wong

Mongodb export woes

January 27, 2021 · 2 min · Karn Wong

2020 ⁶

December ²

CPU upgrade is a breeze, only if you know how

December 20, 2020 · 2 min · Karn Wong

There are caveats behind every statement

December 1, 2020 · 1 min · Karn Wong

September ¹

Visualizing map region prefix/suffix

September 3, 2020 · 4 min · Karn Wong

May ¹

Impute pipelines

May 22, 2020 · 8 min · Karn Wong

April ¹

Word-based analysis with song lyrics

April 15, 2020 · 8 min · Karn Wong

2024 17

October 5

Various notes on career-related stuff

Calling C from Go, Python and Rust benchmark

LLM serving latency benchmark

When (not) to use Terraform

AWS IAM credentials best practices

September 3

Hello World API Performance Benchmark (Go, Node, Python, Rust)

Streamlit load test performance

Simplify self-hosting backups to S3 with docker

August 1

Reasons why you shouldn’t use programming languages for IaC

July 1

GCP’s service account credentials can be a security risk. Here’s how to mitigate them.

June 1

Thoughts on summarization service system design

April 2

Faster spark workloads with comet

Slim down python docker image size with poetry and pip

March 1

Dataframe write performance to Postgres

February 1

How to connect to Cloud SQL from Cloud Run (no, you don’t need a VPC)

January 2

What is platform engineering?

Collaboration model for data science projects

2023 30

December 3

Should data scientists deploy models to production?

Setting up Postgres locally, what could go wrong?

Things to watch out for GCP SSL with Cloudflare DNS

November 4

Serverless real-time machine learning inference with AWS

Some problems can be solved with workflows

Using Apache Iceberg to reduce data lake operations overhead

Reduce operational costs with terraform

September 2

Spark on Kubernetes

Data Engineering Resources

August 3

A Networking God Tale: All I Want is to Run a Speedtest Behind a Firewall

Spatial data to QGIS server playbook (yes, this is for prod)

Create Kubernetes service accounts with Terraform

July 3

pglogical setup

Book Highlights - Build by Tony Fadell

Book Highlights - Atlas of the Heart by Brene Brown

June 1

Hassle-free Kubernetes monitoring with Coroot

April 7

Book Highlights - Managing Transitions: Making the Most of Change by William Bridges

Use SQL against CSV (or other hard files) without CLI

DevX starts at your local machine

The mythical ChatOps in action

DuckDB vs Polars vs Spark!

Kubernetes with Grafana Cloud

Cost optimization with kubernetes

March 7

Bare metal works, until it doesn’t. Hello, cloud.

Google Analytics v4 ingestion via BigQuery

SecretOps with teller

Data transformation - python vs sql showdown

Terraform RDS module with DNS setup

Book Highlights - Engineering Management for the Rest of Us by Sarah Drasner

Book Highlights - The Staff Engineer’s Path by Tanya Reilly

2022 14

December 2

Load balancer 101, thou shalt not be alone

Cross-platform package (+env) management with Nix

November 4

Load credentials into your shell via Bitwarden CLI - Fish edition

Cloudflare DNS management with Terraform

Deploy more efficiently with templating

IaC is cool, until someone specified the wrong machine size 💸

October 2

Deploy static site with branch preview via Cloudflare Pages

Terraform with ECS task on EC2 backend

September 2

Intro to Dagster Cloud

2024 ¹⁷

October ⁵

September ³

August ¹

July ¹

June ¹

April ²

March ¹

February ¹

January ²

2023 ³⁰

December ³

November ⁴

September ²

August ³

July ³

June ¹

April ⁷

March ⁷

2022 ¹⁴

December ²

November ⁴

October ²

September ²

August ²

May ¹

February ¹

2021 ¹⁹

December ²

November ²

October ¹

August ¹

July ¹

June ³

May ¹

April ²

March ²

February ²

January ²

2020 ⁶

December ²

September ¹

May ¹

April ¹

January ¹

2019 ¹

November ¹

2018 ³

October ¹

July ²

2017 ²

November ¹

September ¹