Devops

Repo here Scrapy is a nice framework for web scraping. But like all local development processes, some settings / configs are disabled. This wouldn’t pose an issue, but to deploy a scrapy project to zyte (a hosted scrapy platform) you need to run shub deploy, and if you run it and forget to reset the config back to prod settings, a Titan may devour your home. You can set auto deployment from github via the UI in zyte, but it only works with github only. Plus if you want to run some extra tests during CI/CD you’re out of luck. So here’s how to set up CI/CD to deploy automatically: ...

Ghost CMS is very easy to use, but the deployment overhead (maintaining db, ghost version, updates and etc) might be too much for some. Luckily, there’s a way to convert a Ghost site to static pages, which you can later host on Github pages or something similar. Setup static site engine: Hugo a Ghost instance Usage Install https://github.com/Fried-Chicken/ghost-static-site-generator cd to static directory in your Hugo folder run gssg --domain ${YOUR_GHOST_INSTANCE_URL} --dest posts --url ${YOUR_STATIC_SITE_DOMAIN_WITHOUT_TRAILING_SLASH} --subDir posts Update your hugo config to link to the above folder: [[menu.main]] identifier = "posts" name = "Posts" url = "/posts" All done! 🎉🎉🎉 ...

Since starting self-hosting back in 2017, I’ve always used apache2 since it’s the first webserver I came across. Over time adding more services and managing separate vhost config is a bit tiresome. Enter Caddy. It’s very simple to set up and configure. Some services where I have trouble setting up in apache2 do not need extra config at all, even TLS is set up by default. Starting from Caddy2 it works with CNAME by default without extra setups. ...

EDIT: see here for Caddy, also easier to set up too. Sometimes you found an interesting project to self-hosted, but it doesn’t have password authentication built-in. Luckily, we need to reverse-proxy them anyway and apache2/ nginx / httpd happen to provide password auth with reverse-proxy by default. To set up password auth with apache2 via reverse-proxy: echo "${PASSWORD}" | htpasswd -c -i /etc/apache2/.htpasswd ${USER} on your host machine which has apache2 installed. create a vhost config: <VirtualHost *:80> ProxyPreserveHost On ProxyPass / http://localhost:${EXPOSED_CONTAINER_PORT}/ ProxyPassReverse / http://localhost:${EXPOSED_CONTAINER_PORT}/ ServerName ${YOUR_DOMAIN} <Proxy *> Order deny,allow Allow from all Authtype Basic Authname "Password Required" AuthUserFile /etc/apache2/.htpasswd Require valid-user </Proxy> </virtualhost> That’s it! ...