nidito
/
joao
1
0
Fork 0
Code Issues Pull Requests Packages Releases Wiki Activity

add letter to santa

This commit is contained in:
Roberto Hidalgo 2022-12-31 13:04:19 -06:00
parent d8ff27124a
commit 52f900eaa0
2 changed files with 76 additions and 19 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

44
README.md
View File

@ -1,25 +1,29 @@
# `joao`
a very wip configuration manager. keep config as YAML in the filesystem, backs it up to 1password. Makes it available to services via 1Password Connect + vault.
A very wip configuration manager. Keeps config entries encoded as YAML in the filesystem, backs it up to 1Password, and syncs scrubbed copies to git. robots consume entries via 1Password Connect + Vault.
## Why
So I wanted to operate on my configuration mess...
- With a workflow something like [SOPS](https://github.com/mozilla/sops)',
- but that talks UNIX like [go-config-yourself](https://github.com/unRob/go-config-yourself) (plus it's later `bash` + `jq` + `yq` [re-implementation](https://github.com/unRob/nidito/tree/0812e0caf6d81dd06b740701c3e95a2aeabd86de/.milpa/commands/nidito/config)'s multi-storage improvements),
- [git-crypt](https://github.com/AGWA/git-crypt)'s sweet git filters,
- compatibility with [1Password's neat ecosystem](https://developer.1password.com/), and finally
- a way to make it all available through Hashicorp's [Vault](https://vaultproject.io/) without touching git
- but that talks UNIX, like [go-config-yourself](https://github.com/unRob/go-config-yourself) (plus its later `bash` + `jq` + `yq` [re-implementation](https://github.com/unRob/nidito/tree/0812e0caf6d81dd06b740701c3e95a2aeabd86de/.milpa/commands/nidito/config)'s multi-storage improvements),
- That emulates [git-crypt](https://github.com/AGWA/git-crypt)'s sweet git filters,
- and plays nice with [1Password's neat ecosystem](https://developer.1password.com/),
- as well as Hashicorp's [Vault](https://vaultproject.io/),
- but is still just files, folders and git for all I care.
So I set to write me, yet again, some configuration toolchain that:
And thus, I set to write me, yet again, some configuration toolchain that:
- Allows the _structure_ of config trees to live happily in the filesystem: that is, I like to structure the configuration values I operate with as nested trees, and want a tool that understands these.
- Keeps secrets off remote repositories: I really dig `git-crypt`'s filters, not quite sure about how to safely operate them yet...
- Makes it easy to edit locally, as well as on web and native apps: I mean, it's YAML locally, and 1Password's tools are pretty great for quick edits.
- Operates on configuration trees, wether from a single file or a set of them, with ease: my home+cloud DC needs a lot of configuration that feels weird to keep in a single file; my one-off services don't really need the whole folder structure. I don't wanna use two tools.
- Allows the _structure_ of config trees to live happily **in the filesystem**: my home+cloud DC uses a lot of configuration spread over multiple files, one-off services don't really need the whole folder structure—I want a single tool to handle both.
- Prevents _secrets_ from ending up in **remote repositories**: I really dig `git-crypt`'s filters, not quite sure about how to safely operate them yet...
- Makes it **easy to edit** entries locally, as well as on the go: Easy for me to R/W, so YAML files, and 1Password's tools are pretty great for quick edits remotely.
- Is capable of bootstrapping other secret mangement processes: A single binary can talk to `op`'s CLI (hello, touch ID on macos!), to a 1password-connect server, and to vault as a plugin.
For a deeper dive on these points above, check out my [docs/letter-to-secret-santa.md](docs/letter-to-secret-santa.md).
---
## Configuration
Schema for configuration and non-secret values live along the code, and are pushed to remote origins. Secrets can optionally and temporally be flushed to disk for editing or other sorts of operations. Git filters are available to prevent secrets from being pushed to remotes. Secrets are grouped into files, and every file gets its own 1Password item.
@ -37,7 +41,10 @@ The ideal workflow is:
---
`joao` operates on two modes, **repo** and **single-file**. Repo mode is useful when keeping all configurations in a single folder and expecting their filenames to map to their item names. Single-file mode is useful when a single file contains all of the desired configuration, and its 1Password details are better kept in that same file.
`joao` operates on two modes, **repo** and **single-file**.
- **Repo** mode is useful to have multiple configuration files in a folder structure while configuring their 1Password mappings (vault and item names) in a single file.
- **Single-file** mode is useful when a single file contains all of the desired configuration, and its 1Password mapping is defined in that same file.
### Repo mode
@ -47,10 +54,9 @@ Basically, configs are kept in a directory and their relative path maps to their
# config/.joao.yaml
# the 1password vault to use as storage
vault: infra
# the optional prefix to prepend to all configs from this directory
# without it, config/host/juazeiro.yaml turns into host:juazeiro
# with `bahianos` specified, name would be bahianos:host:juazeiro
# prefix: bahianos
# the optional nameTemplate is a go-template specifying the desired items' names
# turns config/host/juazeiro.yaml to host:juazeiro
nameTemplate: '{{ DirName }}:{{ FileName}}'
```
```yaml
@ -73,11 +79,11 @@ token:
### Single file mode
In single file mode, `joao` expects every file to have a `_joao: !!config` key with a vault name, and a name for the 1Password item.
In single file mode, `joao` expects every file to have a `_joao: !!config` key with a vault name, and a name for the 1Password item.
```yaml
# src/git/config.yaml
_config: !!joao
_config: !!joao
vault: bahianos
name: service:git
smtp:
@ -94,7 +100,7 @@ smtp:
# NAME can be either a filesystem path or a colon delimited item name
# for example: config/host/juazeiro.yaml or [op-vault-name/]host:juazeiro
# DOT_DELIMITED_PATH is
# DOT_DELIMITED_PATH is
# for example: tls.cert, roles.0, dc
# get a single value/tree from a single item/file
@ -113,7 +119,7 @@ joao repo init [PATH]
joao repo list [PREFIX]
# print the repo config root
joao repo root
#
#
joao repo status
joao repo filter clean FILE
joao repo filter diff PATH OLD_FILE OLD_SHA OLD_MODE NEW_FILE NEW_SHA NEW_MODE

51
docs/letter-to-secret-santa.md Normal file
View File

@ -0,0 +1,51 @@
# No, really, why the fuck you'd write another tool?
Here's a list of my grievances and ideas, since there's already much nicer tools in this space.
## Configuration means addressable values
- files and folder structures are great for organizing stuff, and there's already a bunch of great tools to operate on them. Configuration values can be arranged into a multitude of different _config trees_. files and folder names are less than ideal when the interface is not a filesystem, for example during IPC/RPC, but URIs are!
- configuration trees are collections of configuration _entries_:
1. a **key**, or more likely, a path (for example: `smtp.password`).
2. a **type**, or kind of value it holds, such as secrets, scalar values or collections.
3. the **value**, that is, the actual data representing the configuration value, and,
4. **comments**, let's keep happy the human in the loop!
- the _key_ of a configuration value is actually formed of two keys: a **global** and a **local** one. The global one depends on who's asking, that is, a local user may use _file names_ (`api/config.prod.yaml`), while an automated process should use an _URI_ (`op://prod/api`). The local one is always the same, regardless of who's asking, i.e. `smtp.password`. Adding these together produces a _fully qualified key_, or a pointer to the corresponding entry's value.
- So being _addressable_ means configuration _values_ can be extracted from _config trees_ by their _fully qualified key_.
For example, let's say we've got a file named `api/config.prod.yaml` with the following contents:
```yaml
_config: !!joao
vault: prod
name: api
smtp:
username: alguem
password: !!secret "hô-bá-lá-lá"
```
There's many entries in that tree, and the SMTP password value would be addressed with:
- `joao get api/config.prod.yaml smtp.password`
- `op item get op://prod/api/smtp.password`
- `vault kv read config/kv/prod/api/smtp.password`
## Source of truth is hard
- Running a single source of truth (SSoT) for our config values also means worrying about disaster recovery, high availability, opsec and many more wonderful things that I don't feel the least inclined to worry about. 1Password already worries about it, and that's good enough for me personally.
- I don't have a need for a SSoT, really. Since it's only me, there's no need for a change-management, auditing and reporting process. Buuuuut, there is in fact two of them: git and 1password! That doesn't mean they can prevent me from doing something stupid, or evil, but allow me to experiment without adding process I personally don't need on my personal projects.
- Automated processes running on their own need something like a SSoT, or at least a source of truth. Can't rely on git since that means automated processes (think a scheduler listening on changes) would need to have credentials (for git transport), a process to refresh, probably some keys to decrypt data, and all that jazz. That's what 1Password connect is for.
- So okay, at least a few of sources of truth:
1. my **local filesystem**, where I experiment and provision. At some points, this can be considered _the_ source of truth, since it contains all components of a config value: the key, type, data and comments.
2. another up at **git remotes** keeping everything but secret values, this is better than `.template.yaml` files since they're also useful to share knowledge and keep track of progress,
3. last one in **1Password**, one item per file, sans comments
- But now you've got secrets in plaintext lying around in the local fileystem 😱! Preventing these from landing in the wrong remote filesystems is important, so scrubbing them after editing/before uploading is necessary.
## Useful for humans and robots alike
- I'd like a tool that doesn't make a tradeoff between my robots' convenience and mine. I can let Vault figure out permissions and access for robots. I can unlock 1Password locally with my fingerprint. I can use 1Password Connect tokens for provisioning new hardware or vms. Tooling needs to catch up to my personal needs.
- YAML is fine because comments and "custom" types (i.e. `!!secret`), but that's about all I like about it. I'm sure this is going to become a pain later on, but until that day arrives, I'm sticking to YAML.
- YAML sucks because it's not JSON and `jq` only does JSON. Robots love JSON, but JSON doesn't have comments and it writing it by hand makes me sad (but also, why would you when there's `jq`!). So there's `yq` you say, and it's truly great! but also not `jq` and now you need `yq` and `jq` (plus bash, because reasons) to make local scripting a thing.
- Now try editing YAML in a phone screen's default textbox editor. Yeah, it's going to suck. I can workflow my way around it but I don't wanna build and maintain UIs or sync processes. I mostly just wanna do quick edits or saves on particular fields, so losing comments is fine, really trusting dvcs to not drop the ball there. By translating entries into 1Password fields, I rely on them to do the heavy lifting.
- Robots love JSON, 1Password talks JSON (unless `op item edit` 😢), Vault talks JSON, turning YAML into JSON (and back) is pretty easy and there's golang libraries for all of this. Executable has to be good enough for robot and roborto 🤖, so it needs to deal with JSON and YAML.
- git is hard, but `git-crypt` seems to work great, need to make sure this tool makes doing something stupid harder.