I have been playing around with / working productively with AI in vim for a
while now. For religious and practical reasons with local llms, but also with AI agents,
namely kiro-cli,
opencode and gemini
cli, integrated into my workflow and environment with
sidekick.
Through sidekick, the different AI agents were integrated in a separate
buffer but with common and uniform keybindings, context integration (file
content, diagnostics, ..) etc.
Still, every AI agent has its own interaction model, UI details, look and feel
etc, and environment integration by sidekick, while impressive and mostly
surprisingly stable, was rather hacky.
For my personal environment, I’ve been using gemini. My employer only
provides kiro-cli and (limited) opencode, and I want to have an as-uniform
experience as possible. So, for a while I would have wanted to have an
abstraction over the cli agents in my environment.
Theoretically, that’s what
ACP is for.
However, ACP support in kiro-cli has not been present from the start, and I
became rather familiar with the setup I had in the meantime.
Until I had a bit of time at my hands to touch the setup again last week.
nvimkiro-cli, opencode, gemini)nvim environment, no matter which AI agent powers my AI workflowThe Agent Client Protocol (ACP) standardizes communication between code editors/IDEs and coding agents.
If all used agents support the ACP, a single ACP client configuration could provide the solution I want.
Sadly, when I started the fun with “my” main agent kiro-cli, it has not
supported ACP yet.
A while ago, kiro-cli added support for ACP 🥳!
No more reason not to try it out.
After a bit of research, I decided to use
codecompanion to replace sidekick
for the AI integration.
The full configuration can be found here, but I will write about a few details of the configuration.
I had the following sidekick keybindings:
A-a to open/toggle the agent window, including the selection as context, and
A-i to toggle with the predefined prompt catalogue.
keys = {
{
"<A-a>",
function()
require("sidekick.cli").toggle({ filter = { installed = true } })
end,
desc = "Sidekick Toggle",
mode = { "n", "t", "i" },
},
{
"<A-a>",
function()
require("sidekick.cli").send({
filter = { installed = true },
msg = "The current file is {file}, this is the current selection: \n\n```\n{selection}\n```",
})
end,
desc = "Sidekick Toggle with context",
mode = { "x" },
},
{
"<A-i>",
function()
require("sidekick.cli").prompt({ filter = { installed = true } })
end,
desc = "Sidekick Select Prompt",
mode = { "n", "t", "i", "x" },
},
},
For codecompanion, the same can be achieved with these keybindings:
vim.keymap.set({ "n", "t", "i" }, "<A-a>", "<cmd>CodeCompanionChat Toggle<cr>", { desc = "CodeCompanion Toggle" })
vim.keymap.set({ "x" }, "<A-a>", "<cmd>CodeCompanionChat Add<cr>", { desc = "CodeCompanion add selection" })
vim.keymap.set(
{ "n", "t", "i", "x" },
"<A-i>",
"<cmd>CodeCompanionActions<cr>",
{ desc = "CodeCompanion Actions" }
)
The codecompanion buffer behaves like a normal buffer. Which means that if I
jump in the jumplist while being in it, it would be replaced. Or if a custom /
weird LSP action is triggered. Or…
So, I wanted to protect the codecompanion buffer, as identified through its
virtual filetype.
init = function()
-- If new buffers are opened or jumps being triggered while focus is on codecompanion, the action should be triggered outside of the codecompanion buffer
vim.api.nvim_create_autocmd("FileType", {
pattern = "codecompanion",
callback = function(args)
vim.bo[args.buf].buflisted = false
local win = vim.fn.bufwinid(args.buf)
if win == -1 then
return
end
-- Guard this window: redirect any non-cc buffer that lands here
vim.api.nvim_create_autocmd("BufWinEnter", {
callback = function(ev)
if not vim.api.nvim_win_is_valid(win) then
return true
end
if vim.api.nvim_get_current_win() ~= win then
return
end
if ev.buf == args.buf then
return
end
-- Restore cc buffer in this window, move new buffer elsewhere
vim.api.nvim_win_set_buf(win, args.buf)
for _, w in ipairs(vim.api.nvim_tabpage_list_wins(0)) do
if w ~= win and vim.bo[vim.api.nvim_win_get_buf(w)].filetype ~= "codecompanion" then
vim.api.nvim_win_set_buf(w, ev.buf)
vim.api.nvim_set_current_win(w)
return
end
end
vim.cmd("split | buffer " .. ev.buf)
end,
})
end,
})
end
Note: Recently,
pyreflyhas gained the capabilities that were missing for me to be able to fully switch to it as sole lsp. My current setup doesn’t make use of cherry-picking LSP capabilities anymore.
As I have written about here, for most of my file edititing / programming / note-taking / … I use vim (neovim).
Not a bare-bones neovim, though. I use many plugins, and I have even more of them set up. Too many? Maybe. Probably. Especially considering the ever growing number of software supply chain attacks..
Anyways, that is a different topic for a different day.
Besides convenient plugins, the main building block for developer experience enhancement and efficiency are language servers, which provide stuff like diagnostics, code completion, navigation enhancements and more.
Besides playing around with its config, I try to actually use vim for
productive work, including programming in python.
My go-to language server for python has been
basedpyright, a fork of
microsoft’s pyright, which basically
adds the functionality of the proprietary pylance extension to pyright and
adds some more bells and whistles.
While basedpyright is stable, helpful, convenient and generally excellent,
there are a few new kids on the block:
There is a new generation of python language servers, written in Rust,
targeting high performance and productivity in large and complex code bases:
There is Astral’s ty, Meta’s
pyrefly and David Halter’s (Jedi’s original author)
zuban, and probably some more.
I have been intrigued by them, mainly hoping for a smoother and more performant
workflow in large and growing codebases. So, I have played around with them for
a while now, eventually ending up using ty only for type checking while still
having to rely on basedpyright for completion, auto-import, symbol renaming,
navigation etc.
Why?
In single capabilities, none of the new lsp servers match basedpyright in my
workflow. Yet.
E.g.:
pyrefly has great completion, navigation and even auto-import, but it’s
really bad in renaming symbols. It can rename variables, but no functions.ty has limited support of pydantic, which I use a lot, completion is not
as powerful as pyrefly’s, but renaming works great.Why not use both?
If you enable several language servers with the (nominally) same capabilities for the same file type, you get duplicate results/actions for every capability. Go to definition? Select where you want to jump to from these two identical alternatives!
My goal was to mix-and-match, e.g. use ty for renaming and pyrefly for
everything else. However, while especially pyrefly’s configuration is very
flexible, the lsps don’t support tailored enabling and disabling of select
capabilities.
I tried for an unsuccessful while to configure the lsp server by restricting the handed-over client capabilities, but that didn’t go anywhere.
The only almost compromise that worked for me with the configuration through
the lsp servers themselves was disabling basedpyright’s type checking and
disabling ty’s language server capabilities, besides type checking. Like
this, at least a little bit of the heavy lifting could be offloaded from
basedpyright to ty, but I was not happy.
Now, I sort of am happy! I found out how to disable specific lsp capabilities
for specific lsp servers through nvim’s lsp
configuration. Instead of trying to get
the lsp server to not provide the functionality, the client ignores it!
I like working with Python, also with larger and growing code bases. There are
LSP servers that offer excellent performance. However, none match the full
functionality of basedpyright when used independently. Unfortunately,
basedpyright lacks in performance. Combining multiple LSP servers introduces
duplicate functionality and a bit of a mess. Such combination could be feasible
if we could selectively disable or enable conflicting capabilities. However, it
is not possible to instruct LSP servers to withhold certain functionalities
unless they provide an interface for that, e.g. with dedicated configuration
options.
Enable conflicting lsp servers, but tell the lsp client (nvim) that it should
not use specific lsp server’s capabilities through an on_init hook 🥳!
The configuration is probably not fully final and will change, but this is the general thing.
local lsp_capabilities = require("blink.cmp").get_lsp_capabilities(nil, true)
---@param client vim.lsp.Client
local ty_on_init = function(client)
client.server_capabilities.callHierarchyProvider = false
client.server_capabilities.completionProvider = false
client.server_capabilities.inlineCompletionProvider = false
client.server_capabilities.inlineValueProvider = false
client.server_capabilities.declarationProvider = false
client.server_capabilities.definitionProvider = false
client.server_capabilities.implementationProvider = false
client.server_capabilities.inlayHintProvider = false
client.server_capabilities.signatureHelpProvider = false
client.server_capabilities.hoverProvider = false
client.server_capabilities.referencesProvider = false
client.server_capabilities.codeActionProvider = false
end
vim.lsp.config("ty", {
capabilities = lsp_capabilities,
on_init = { ty_on_init },
settings = {
ty = {
disableLanguageServices = false,
diagnosticMode = "workspace",
experimental = {
rename = true,
},
},
},
})
---@param client vim.lsp.Client
local pyrefly_on_init = function(client)
client.server_capabilities.renameProvider = false
end
vim.lsp.config("pyrefly", {
capabilities = lsp_capabilities,
on_init = { pyrefly_on_init },
settings = {
python = {
pyrefly = {
disableLanguageServices = false,
displayTypeErrors = "force-on",
},
analysis = {
diagnosticMode = "workspace",
inlayHints = {
callArgumentNames = "all",
variableTypes = true,
functionReturnTypes = true,
pytestParameters = true,
},
},
},
},
})
on_init and
on_attach etc.As I have written about here, I sort of have accepted a compromise in security for added convenience, auto-unlocking my LUKS encrypted root partition making use of my system’s built in TPM in order to only auto-unlock the system under specific (secure) conditions.
I’ve used clevis so far to manage the setup, using the trusty
initramfs-tools to configure and bake the initramfs with the required
modules and configuration, relying on cryptsetup to manage and reset the
specifics of the configuration, e.g. whenever a bios update invalidated a PCR
measurement.
Not that elegant. And there are new, more elegant and capable mechanisms available!
systemd-cryptenroll to enroll physical tokens to LUKS2 encrypted
volumes, simplifying the setup and management of this part of the processdracut for the configuration and management of initramfs images,
the designated successor of initramfs-toolsWhile my previous setup worked, I wanted to try setting it up with the (designated) successor systems.
Well. To be honest, first I just wanted to try out systemd-cryptenroll
instead of clevis, realized that the intended way of configuring
auto-unlocking didn’t work with initramfs-tools, realized that (due to that,
amongst other things) there have been discussions about replacing
initramfs-tools with dracut as default and then decided to redo the whole
process.
Well, it’s not really a new problem, it’s an itch and a mental problem, I guess 😅. The original challenge, shifting the compromise of convenience vs. security a little by auto-unlocking the encrypted system with the help of the TPM has been working for a while.
But:
systemd-cryptenroll exists, potential successor to clevisdracut exists, potential successor to initramfs-toolsInstalling systemd-cryptenroll on Debian is easy enough. It comes with
systemd-cryptsetup, which might be on your system already?
If not:
sudo apt-get install systemd-cryptsetup
Then, enrolling the TPM with the encrypted partition is pretty straightforward:
sudo systemd-cryptenroll --tpm2-device=auto /dev/nvme0n1p3
Depending on your needs/level-of-paranoia, you might want to add other PCRs, as
systemd-cryptenroll per default only uses PCR 7, maybe you even want to use
TPM + PIN:
sudo systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=7+11 --tpm2-with-pin=yes /dev/nvme0n1p3
You follow the guided process, and then can verify success. Hopefully.
sudo systemd-cryptenroll /dev/nvme0n1p3
SLOT TYPE
0 password
1 tpm2
Configuration should (™️) work by configuring the tpm in /etc/crypttab and
rebuilding the initramfs with it:
cat /etc/crypttab
nvme0n1p3_crypt UUID=e010d6c1-6016-4f2d-bd05-1be8f383ada9 none tpm2-device=auto,luks,discard
But, alas, rebuilding the initial ram disk fails on Debian and should also fail on Ubuntu:
sudo update-initramfs -u -k all
...
cryptsetup: WARNING: nvme0n1p3_crypt: ignoring unknown option 'tpm2-device'
...
After a bit of research: initramfs-tools doesn’t support tpm2 devices. There
is a bug report
and even an open merge request with a
fix,
but the situation appears to be stale.
However, there is the designated successor of initramfs-tools:
dracut, which has
replaced initramfs-tools in a bunch of linux distributions already.
On debian and Ubuntu, it can be installed from the package sources like this,
replacing initramfs-tools in the process:
sudo apt-get install dracut
Configuration of dracut happens in /etc/dracut.conf.d/*.conf.
I configured dracut to run the host-only initramfs creation with tpm2 support
and compress with lz4, all in separate files:
cat /etc/dracut.conf.d/compression.conf
compress="lz4"
cat /etc/dracut.conf.d/hostonly.conf
hostonly="yes"
hostonly_cmdline="no"
cat /etc/dracut.conf.d/tpm.conf
add_dracutmodules+=" tpm2-tss crypt "
add_drivers+=" tpm tpm_tis_core tpm_tis tpm_crb "
With that configuration in place, the initramfs can be (re)built with
dracut -f
After that, a reboot should end up with an automatically unlocked encrypted partition.
If you trust your system firmware and tpm enough, you can set your system up to
decrypt automatically without providing a passphrase using
systemd-cryptenroll and dracut.
There are also other methods, but this one appears to be the most future proof and easy to set-up and maintain.
]]>My employer is a typical German large scale corporation. One of the attributes that come with that is that we are, of course, interested in generative AI, but also very sceptical and cautious.
Slowly, however, more and more AI services become available to us, but basically only web-interface chat solutions without APIs and IDE/development related integrations.
I was moderately happy when recently it was announced that we can use Amazon Q Developer! Awesome! Finally a development focused solution!
There are extensions for JetBrains IDEs, VS Code, Visual Studio and a preview for Eclipse, and there is a cli.
That should be enough, right?
…
Well. Not for all of us neovim users.
I want to have integration to the full potential of generative AI, adhering to applicable compliance restrictions, and still remain in the tool environment that makes me as productive and makes my work as enjoyable as possible.
There are many pretty excellent neovim AI plugins out there, e.g. Avante or Copilot Chat.
While those are pretty nice and powerful, most of them are taylored around a number of the big AI API providers, and are pretty big to work with.
Amazon Q does not provide a nice API, especially not an API that is compatible with existing clients and integrations. I spent a few minutes looking through Avante and did not see a quick path to extend the existing (web API centric) model adaption with Q.
I have been tinkering with a fork of neoai.nvim, extending it a little for my usecases, focussing on low-overhead integration of local language models with llama.cpp. I’ve been very happy with it, especially because the plugin is still so small that I don’t have a hard time understanding what it does and how to extend it without having to spend too much time. I’m lazy..
So, I decided to simply extend the plugin I’m most familiar with, my fork.
As written above, Amazon Q does not provide a nice API, but has a cli, called
q. I decided to keep it simple and simply dump my prompt into q in
non-interactive mode, parsing its stdout output. I hoped that it would
allow me to use the existing way of providing persistent chat sessions and
context, as well as having a simple way to apply existing functionality, e.g.
code recognition for easy pasting with :put c etc.
It was surprisingly easy, the core of it all is fifty-ish lines of lua:
local utils = require("neoai.utils")
--- This model definition supports Amazon Q CLI integration
---@type ModelModule
local M = {}
M.name = "Amazon Q"
M._chunks = {}
M.get_current_output = function()
return table.concat(M._chunks, "")
end
---@param chunk string
---@param on_stdout_chunk fun(chunk: string) Function to call whenever a stdout chunk occurs
M._recieve_chunk = function(chunk, on_stdout_chunk)
-- Split the input chunk by newlines
on_stdout_chunk(chunk)
table.insert(M._chunks, chunk)
end
---@param chat_history ChatHistory
---@param on_stdout_chunk fun(chunk: string) Function to call whenever a stdout chunk occurs
---@param on_complete fun(err?: string, output?: string) Function to call when model has finished
M.send_to_model = function(chat_history, on_stdout_chunk, on_complete)
-- Format messages for Amazon Q CLI
local messages_json = vim.json.encode(chat_history.messages)
chunks = {}
local command = {}
command = {
"chat",
"--no-interactive",
}
for _, v in ipairs(chat_history.params) do
table.insert(command, v)
end
table.insert(command, messages_json)
-- Execute the Amazon Q CLI command
utils.exec("q", command, function(chunk)
M._recieve_chunk(chunk, on_stdout_chunk)
end, function(err, _)
-- Clean up the temporary file
on_complete(err, M.get_current_output())
end)
end
return M
Q behaves a bit weird, it only marks source-code it produces with colors, using ansi codes, instead of using the de-facto standard backtics. Luckily it appears to be consistent, so code snippet extraction works like this:
extract_code_snippets = function(text)
local matches = {}
-- Amazon Q marks code by color-coding it green with ansi codes.
-- Everything between "start green" and "reset color" is assumed to be code.
for match in string.gmatch(text, "\27%[38;5;%d+m(.-)\27%[0m") do
table.insert(matches, match)
end
return table.concat(matches, "\n\n")
end
I run environments that do not have q available, so I made my config
sensitive to its availability.
config = function()
local q_available = vim.fn.executable("q") == 1
local models = {}
local extract_code_snippets
if q_available then
extract_code_snippets = function(text)
local matches = {}
-- Amazon Q marks code by color-coding it green with ansi codes.
-- Everything between "start green" and "reset color" is assumed to be code.
for match in string.gmatch(text, "\27%[38;5;%d+m(.-)\27%[0m") do
table.insert(matches, match)
end
return table.concat(matches, "\n\n")
end
table.insert(models, {
name = "q",
model = "q",
params = { "--trust-all-tools" },
})
else
startLlama()
extract_code_snippets = function(text)
local matches = {}
for match in string.gmatch(text, "```%w*\n(.-)```") do
table.insert(matches, match)
end
-- Next part matches any code snippets that are incomplete
local count = select(2, string.gsub(text, "```", "```"))
if count % 2 == 1 then
local pattern = "```%w*\n([^`]-)$"
local match = string.match(text, pattern)
table.insert(matches, match)
end
return table.concat(matches, "\n\n")
end
table.insert(models, {
name = "openai",
model = "llama 3",
params = nil,
})
end
require("neoai").setup({ ---@diagnostic disable-line: missing-fields
ui = {
output_popup_text = "AI",
input_popup_text = "Prompt",
width = 45, -- As percentage eg. 45%
output_popup_height = 80, -- As percentage eg. 80%
submit = "<Enter>", -- Key binding to submit the prompt
},
models = models,
register_output = {
["a"] = function(output)
return output
end,
["c"] = extract_code_snippets,
},
inject = {
cutoff_width = 75,
},
prompts = {
context_prompt = function(context)
return "Please only follow instructions or answer to questions. Be concise. "
.. (vim.api.nvim_buf_get_name(0) ~= "" and "This is my currently opened file: " .. vim.api.nvim_buf_get_name(
0
) or "")
.. "I'd like to provide some context for future "
.. "messages. Here is the code/text that I want to refer "
.. "to in our upcoming conversations:\n\n"
.. context
end,
default_prompt = function()
return "Please only follow instructions or answer to questions. Be concise. "
.. (
vim.api.nvim_buf_get_name(0) ~= ""
and "This is my currently opened file: " .. vim.api.nvim_buf_get_name(0)
or ""
)
end,
},
mappings = {
["select_up"] = "<C-k>",
["select_down"] = "<C-j>",
},
open_ai = {
url = "http://127.0.0.1:9741/v1/chat/completions",
display_name = "llama.cpp",
api_key = { ---@diagnostic disable-line: missing-fields
value = nil,
get = function()
return ""
end,
},
},
})
end,
Reading through the config you might see a dangerous option: params = {
"--trust-all-tools" },.
This allows q to execute all commands that are available to it in the current
context.
Yes, it’s dangerous, probably also a bit stupid. But really powerful, as well. And a lot of fun, to be honest.
First I was annoyed by Amazon Q not providing a web API, but using the cli
as API makes the tool integration agentic, basically for free!
I give the currently open file as context in the system prompt.
.. (
vim.api.nvim_buf_get_name(0) ~= ""
and "This is my currently opened file: " .. vim.api.nvim_buf_get_name(0)
or ""
)
With this, I can refer to the current file and let q interact with it.
Or even the entire project!
I was pretty amazed by pressing <Alt-a> in my editor and watching the
integration perform a full Please run the tests in this project and tell me if
they were a success. and things like Create a feature branch hinting that I
created a new UI, commit the latest changes and push them.
clicli can be integrated in neovimWhen working on several projects in different environments, interacting with distributed and relatively loosely coupled systems and platforms, such as different Jira, GitLab, … instances and other API providers, e.g. OpenAI or an Artifactory or whatever.
I want to be able to consume these systems from my development environment,
which is basically my shell with dedicated command line interfaces and
integrations in vim.
How can I now configure all the endpoints and other environmental aspects,
including API tokens and other secrets, independently for different project
contexts, without having to copy-paste secrets or hard-code secrets somewhere,
e.g. putting clear-text tokens into .env files?
The solution should handle
In pretty much every Linux environment that is not KDE, you should have the
gnome-keyring available. The gnome-keyring is basically a local database
that is encrypted with the user’s login password (-> automatically decrypted
on login through pam), which provides a convenient centralised interface for
secrets/keys. gnome-keyring supports gpg and ssh keys, which it can unlock
and act as a key agent for, and generic passwords.
All passwords can then be retrieved by libsecret and the according tooling. There is a built-in git integration (built into git, but you still have to configure it!), and many applications use it already, e.g. the network manager, Chrome, Chromium, Edge, …
This opens a solution path, if it would be possible to automatically retrieve
secrets from the gnome-keyring and create environment variables etc.
There is a cli, secret-tool, which allows you to store and retrieve generic
secrets from/into the gnome-keyring. In combination with your shell setup
through direnv, this allows for pretty nice and still secure automation,
fulfilling all the goals.
Well. You need the gnome-keyring to be set up, ideally unlocked automatically
on login. If you’re on gnome, that should come as a given. If you are using a
different environment, it might be a given.
Install the secret-tool. On Debian based systems, which includes all the
Ubuntu flavours, that’s done with
sudo apt-get install libsecret-tools
Install direnv, e.g., if you are on Debian based systems, with
sudo apt-get install direnv
And make sure to set it up and integrate it into your shell.
I am using zsh, so you will find the configuration in the .zshrc
...
if command -v direnv 1>/dev/null 2>&1; then
_evalcache direnv hook zsh
fi
...
For example, I want to have a gitlab API token in all my repositories for
https://private-gitlab.com, so that I can quickly create new projects or open
/ interact with merge requests with the gitlab cli, glab. So, I have a file
~/workspace/private-gitlab.com/.envrc with the following content:
export GITLAB_TOKEN=$(secret-tool lookup protocol https server private-gitlab.com user <bob@secret.com>)
export GITLAB_URI="https://private-gitlab.com"
export GITHUB_TOKEN=$(secret-tool lookup protocol https server github.com user <dominik.gierlach@gmail.com>)
Together with direnv set up and integrated in to the shell, as can be seen
im my dotfiles., this retrieves a token
from the keyring whenever I enter a repository in
~/workspace/private-gitlab.com/... and sets it up as local environment
variable in this context.
The secrets themselves can be set up like this:
secret-tool store --label='Git: <https://github.com/>' schema org.gnome.keyring.NetworkPassword protocol https user <dominik.gierlach@gmail.com> server github.com
Or by using seahorse, a graphical manager for the gnome-keyring.
With direnv, it’s possible to nest environments in order to extend them.
Assume a directory structure like this
~/workspace/github.com/..
- .envrc
+ - project_group_1
+ - project_a
+ - project_b
+ - project_group_2
- .envrc
+ - project_c
+ - project_d
I want to be able to use the environment configuration that is part of
~/workspace/github.com/.envrc in every project, but I want to change/extend it
for all projects in ~/workspace/github.com/project_group_2.
That is possible by referring to the overarching .envrc from
~/workspace/github.com/project_group_1/.envrc:
source ~/workspace/github.com/.envrc
export AWS_PROFILE=PROJECT_GROUP_2_AWS
gnome-keyringAs written above, gnome-keyring can also store git credentials, when using
https authentication. While this is somewhat orthogonal to setting up shell
environments with secrets, it’s connected and important.
git can use credential helpers
to store and automagically use credentials for different repositories. The
trivial and built-in credential helper is store, which simply stores the
credentials on disk in clear text, so let’s not go down that path.
The credential helper to integrate with gnome-keyring on linux is
git-credential-libsecret.
git-credential-libsecretOn Debian based systems, including Ubuntu, the credential helper is not
available as compiled binary, but the sourcecode is part of the git package.
Install it e.g. like this:
sudo apt-get install libsecret-1-dev
mkdir -p ~/workspace/other/libsecret
cd ~/workspace/other/libsecret
cp -r /usr/share/doc/git/contrib/credential/libsecret/* ./
make
mkdir -p ~/.local/bin
cp git-credential-libsecret ~/.local/bin
git-credential-libsecretAfter it has been installed, let’s make git use it.
git config --global credential.helper libsecret
Yes, that was annoyingly complex, especially for a thing that should not be forgotten whenever a new machine is set up.
That’s why I integrated the setup into my dotfiles. I don’t want to have to remember (= look up and re-understand) how to set it up whenever I set up a new machine, and I don’t want to forget to set it up at all.
The git configuration is part of the global
.gitconfig,
installation is automated through
tuning.
gnome-keyringsecret-tooldirenvI am a big vim (neovim) fan and do basically all my programming and writing in it, the exception being work things that need MS Office. Luckily, most of those things are possible with the browser-based version of office, the exception being macros and obscure things like Powerpoint footer changes. Anyways, that is a completely different story.
Naked, vanilla vim is nice and present in basically every environment. So, it’s good to be able to use it, especially for remote configuration/documentation/troubleshooting work. The real power is unleashed with more advanced capabilities and extensions, which extend the barebone editor with modern creature comforts, such as linting, automatic fixing, code completion, auto formatting, syntax highlighting, test execution, debugging and intelligent navigation.
My code completion and intelligent navigation setup started with
YouCompleteMe in 2017. I needed it
mainly for C, and it did a good job.
A bit later, Microsoft’s Language Server Protocol gained much more traction, with more and more really usable language servers being really usable and taking off, mainly n the context of visual studio code. Most of the Language servers where installed and integrated into vscode through dedicated plugins.
In comes the neovim plugin coc (conquer of completion). This plugin acts as a language server client, meaning that it can use any language server that implements the lsp standard, and it can integrate slightly adapted vscode pluins as coc plugins. This meant that the managed installation of language servers and the integration thereof became really simple and straightforward, with a very active ecosystem and community around it. I migrated to coc in 2019, adding support and better integration for many more languages into my setup.
coc with a set of plugins and a handful of “manually” integrated language servers as well as with a set of linters and formatters, integrated with diagnostic-languageserver, was my default setup for 5 years. I changed the supported languages all the time, based on what I needed, but besides that and rewriting the config to lua, my setup stayed like this for 5 years.
In the meantime, neovim gained native language server client support, removing the need for a dedicated plugin for lsp support. However, coc still provided the easy integration and awesome out of the box functionality.
The awesome and very capable community embraced the feature, though. There is mason, an nvim native package/lsp manager, and many integration layer plugins between neovim and mason, making the integration of language servers slimmer (no extra layer), extremely flexible and manageable, with concise and powerful configuration.
Powered by all of that, projects like lazyvim started, providing much more out of the box functionality and ease of use than coc had. Lazyvim is basically a full-featured one size fits all preconfiguration of nvim, with very sane defaults and a smart plugin and configuration system that makes it feature rich without bloat. I know a few people who recently went down the rabbithole of vim, and they started with lazyvim right away. I respect it a lot, even though I want to have a bit more control over my setup, so not for me. The ecosystem grew and grew and is now more active and powerful than coc’s ecosystem.
I was very interested in the principle of the powerful and manageable modularization, but still, the normal out of the box experience was a lot worse than with coc, as it was basically not a box, but a collection of elements you had to build your box with yourself. The capabilities of coc are split into separate and independent components for completion (with lsp as source amongst others), dedicated components for formatters, linters, dedicated configuration of all components and system integration.
Interesting, but it’s a lot. Especially because my current setup just worked and did mostly what I wanted it to do, and I had no real functioning reason to migrate besides the itch. So, I put a migration on the back of my to do list, to do some other time.
That is until for the first time ever I had to do things in C#. There is a
coc plugin for C#,
coc-omnisharp, which in the
README greets you with the message:
⛔ This project is lacking proper maintanence. I would recommend csharp-ls at this moment.
This leaves manual integration of the csharp-language-server. Awesome, I did it right away.
But there was a problem.
With the coc integration, I was unable to find a way to replace the default sync handler for system libraries only available as binaries, used e.g. for navigation, such as go_to_definition, with a buffer showing the decompiled sourcecode. This is explicitly (and easily) possible with the flexible configuration of native language servers in nvim, as documented.
So, last weekend, I did it.
I migrated my configuration.
I use lazy as plugin manager, which makes it really simple to split up plugin configuration. I decided to split my language server config into a dedicated file for the abstract definition of the behaviour of lsps, e.g. with key mappings, sources of the completion system and so on, a dedicated file for the specific definition of supported languages, and a dedicated file for the separate configuration of linters and fixers, while also revamping the test execution and debugging setup.
/home/gierdo/.dotfiles/.config/nvim/lua/plugins
❯ tree
.
├── debugging.lua
├── filetree.lua
├── fuzzyness.lua
├── git.lua
├── init.lua
├── linters.lua
├── lsp-behaviour.lua
├── lsp-languages.lua
├── neoai.lua
├── testing.lua
├── theme.lua
└── treesitter.lua
Copy-pasting the few hundred lines of lua here probably makes this post even less readable, but I’ll point out some things here that are not just obvious configuration.
If you want to see the full configuration, check out my dotfiles.
local cs_config = {
capabilities = lsp_capabilities,
handlers = {
["textDocument/definition"] = require("csharpls_extended").handler,
["textDocument/typeDefinition"] = require("csharpls_extended").handler,
},
}
lspconfig.csharp_ls.setup(cs_config)
vim.keymap.set(
"n",
"gd",
require('telescope.builtin').lsp_definitions,
{ buffer = event.buf, desc = "Go to definition." }
)
vim.keymap.set(
"n",
"gi",
require('telescope.builtin').lsp_implementations,
{ buffer = event.buf, desc = "Go to implementation." }
)
vim.keymap.set(
"n",
"go",
require('telescope.builtin').lsp_type_definitions,
{ buffer = event.buf, desc = "Go to type definition." }
)
vim.keymap.set(
"n",
"gr",
require('telescope.builtin').lsp_references,
{ buffer = event.buf, desc = "Show references." }
)
lspconfig.jsonls.setup({
capabilities = lsp_capabilities,
settings = {
json = {
schemas = require("schemastore").json.schemas(),
validate = { enable = true },
schemaDownload = { enable = false },
},
},
})
lspconfig.yamlls.setup({
capabilities = lsp_capabilities,
settings = {
yaml = {
schemaStore = {
enable = false,
url = "",
},
schemas = require("schemastore").yaml.schemas({
extra = {
{
name = "Cloudformation",
description = "Cloudformation Template",
fileMatch = { "*.template.y*ml", "*-template.y*ml" },
url = "https://raw.githubusercontent.com/awslabs/goformation/master/schema/cloudformation.schema.json",
},
},
}),
customTags = {
-- Cloudformation tags
"!And scalar",
"!If scalar",
"!Not",
"!Equals scalar",
"!Or scalar",
"!FindInMap scalar",
"!Base64",
"!Cidr",
"!Ref",
"!Sub",
"!GetAtt sequence",
"!GetAZs",
"!ImportValue sequence",
"!Select sequence",
"!Split sequence",
"!Join sequence",
},
},
},
})
local wk = require("which-key")
wk.add({
{ "<leader>t", group = " Test" },
{ "<leader>tr", neotest.run.run, desc = "Run nearest test." },
{
"<leader>tf",
function()
neotest.run.run(vim.fn.expand("%"))
end,
desc = "Run current file.",
},
{
"<leader>td",
function()
neotest.run.run({ strategy = "dap" })
end,
desc = "Debug nearest test.",
},
{
"<leader>ts",
neotest.summary.toggle,
desc = "Show/hide summary.",
},
})
wk.add({
{ "<leader>d", group = " Debug" },
{
"<leader>dd",
telescope.extensions.dap.commands,
desc = "Show debug commands.",
},
{
"<leader>dt",
dap.toggle_breakpoint,
desc = "Toggle line breakpoint on the current line.",
},
{
"<leader>dbs",
dap.set_breakpoint,
desc = "Set breakpoint.",
},
{
"<leader>dbl",
telescope.extensions.dap.list_breakpoints,
desc = "List breakpoints.",
},
{
"<leader>dv",
telescope.extensions.dap.variables,
desc = "Show variables.",
},
{
"<leader>dbc",
dap.clear_breakpoints,
desc = "Clear breakpoints.",
},
{
"<leader>dbe",
dap.set_exception_breakpoints,
desc = "Set Exception breakpoints.",
},
{
"<leader>dc",
dap.continue,
desc = "When debugging, continue. Otherwise start debugging.",
},
{
"<leader>dfd",
dap.down,
desc = "Move down a frame in the current call stack.",
},
{
"<leader>dfu",
dap.up,
desc = "Move up a frame in the current call stack.",
},
{ "<leader>dp", dap.pause, desc = "Pause debuggee." },
{
"<leader>dr",
dap.run_to_cursor,
desc = "Continues execution to the current cursor.",
},
{
"<leader>dre",
dap.restart,
desc = "Restart debugging with the same configuration.",
},
})
I’m a big fan of atuin, a shell history manager, which replaces the flat command history file with a local database, fully searcheable and with builtin synchronization support.
My setup of atuin, installation and configuration, can be seen in my dotfiles. But the documentation is excellent.
In case you are wondering why the sync configuration is not in my config: I try
to keep all specific configuration of stuff that is not publicly available and
might not be needed in all my environments outside of the dotfiles. I set
$ATUIN_SYNC_ADDRESS in my ~/.profile.local.
export ATUIN_SYNC_ADDRESS=http://raspberrypi.local:8888
Atuin offers a free synchronization server at https://api.atuin.sh. Yes, it’s not a good idea to show and open up your command history to everybody. Which is why the synchronization is completely end-to-end encrypted by atuin. The synchronization server only sees gibberish, and all your clients have to be set up with a passkey to enable sync.
If you, like me, still don’t want to trust that data to a sync service, you can also run your own synchronization server, which is pretty straightforward, assuming you know a bit about services and postgresql.
Of course that means that you have to run a server somewhere…
I happen to have a rasperry pi 3 in my home network, which mainly acts as an
always-on member of syncthing-synchronized
directories, and as a git repository server for private repositories with
gitolite. I don’t expose the Pi at
all to the public internet, it is only available internally, resolveable
through mDNS. I wanted to set it up as atuin synchronization server. It’s
already there, it is available for all devices in my local home network. This,
of course, means that I can only synchronize shell history with my machines at
home. But I don’t have a problem with this limitation at all.
Atuin is not installable through the raspbian package sources. This is also
true for Debian at the time of writing. On my machine, I install it with cargo,
orchestrated with tuning.
So, how do you install atuin on my raspberry pi, which uses an ARM CPU? Especially if you don’t want to compile it on the pi itself, wait for ages and fill your pi with stuff you don’t really need?
Atuin is a program written in Rust. This means that
cargoAs usual, the devil is in the details, however.
With rustup as rust toolchain manager, it’s easy to install the rust specific
target and toolchain for the raspberry pi.
$ rustup target add armv7-unknown-linux-gnueabihf
This is not enough, however. For successful cross compilation, rust also needs the target specific gcc toolchain to link against C libraries and sysroot.
You need gcc for cross-compilation with the armhf target, as well as the
armhf specific libraries for dynamic linking.
On a Debian system, this can be installed like so.
$ sudo apt-get install gcc-arm-linux-gnueabihf libgcc-13-dev-armhf-cross
The installation of gcc and the sysroot is on a system level, independent from the local rust setup. This means that cargo has to be set up correctly so that it uses the correct linker.
Target specific attributes can be configured in ~/.cargo/config.toml
[target.armv7-unknown-linux-gnueabihf]
linker = "arm-linux-gnueabihf-gcc"
After everything has been set up, cargo can be built in the local directory with this command:
$ cargo install --target=armv7-unknown-linux-gnueabihf --target-dir=$(pwd) --root=$(pwd) atuin
This will download the sources of atuin and all dependencies, cross-compile
it and copy it to a bin subdirectory.
$ readelf -h bin/atuin
ELF Header:
Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00
Class: ELF32
Data: 2's complement, little endian
Version: 1 (current)
OS/ABI: UNIX - System V
ABI Version: 0
Type: DYN (Position-Independent Executable file)
Machine: ARM
Version: 0x1
Entry point address: 0xd363d
Start of program headers: 52 (bytes into file)
Start of section headers: 32358908 (bytes into file)
Flags: 0x5000400, Version5 EABI, hard-float ABI
Size of this header: 52 (bytes)
Size of program headers: 32 (bytes)
Number of program headers: 10
Size of section headers: 40 (bytes)
Number of section headers: 44
Section header string table index: 43
Copy the binary onto your raspberry pi, e.g. to /usr/local/bin/atuin, and
you’re good to go!
You might have read my previous posts about using a local codellama for a vim coding assistant, setting it up to use my laptop’s GPU for inference and the performance evaluation and so on.
Also, my previous evaluation of different models, which formed the base of my selection of which model to run locally.
Since quite a bit of time, there has been a new kid on the block: Meta’s llama 3 is here! I’ve been running it since April, and it basically replaced my previous local model.
I will not repeat the evaluation of the previous models, but apply the same methods to llama 3. For the others, please take a look at the previous article. Or, even better, try them out yourself.
The general setup is described already in previous articles, mainly the performance evaluation of GPU assisted inference
Here is the short summary:
This results an boundary conditions for the models to chose from: The models have to be usable with llama.cpp and cope with 8GB of VRAM.
llama 3 fits that requirement nicely in the quanticised 8B instruct variant:
Meta-Llama-3-8B-Instruct.Q4_K_M.gguf
I gave each model a specific task:
Please create a c function for an embedded platform (no stl or time.h, no
POSIX), which returns the unix timestamp for a given time.
With the created code in the context, I want to test each model’s ability to work with given code, e.g. for comments and improvement.
What is happening here?
What are potential problems with the given code?
Please give me a well-documented version of this code. Please follow best
practices and do not comment overly verbosely.
Llama 3 happy starts right away, ignoring significant parts of the instruction, however: The given value is not unix timestamp compliant.
--------
Here is a simple C function that calculates the Unix timestamp (seconds since
January 1, 1970, 00:00:00 UTC) from a given year, month, day, hour, minute and
second:
```c
#include <stdint.h>
uint32_t unix_timestamp(int year, int month, int day, int hour, int min, int sec) {
uint32_t timestamp = 0;
// Calculate the seconds since January 1, 1970, 00:00:00 UTC
timestamp += (year - 1970) * 31536000; // 31536000 is the number of seconds in a year
// Adjust for leap years
if ((year % 4 == 0 && year % 100 != 0) || year % 400 == 0)
timestamp++; // Add one second for February 29th
int month_days[] = {31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31};
timestamp += (month - 1) * 2628000 + day * 86400; // Add days and hours
timestamp += (hour * 3600); // Add hours
timestamp += (min * 60); // Add minutes
timestamp += sec;
return timestamp;
}
```
Note: This function assumes that the input values are valid. It does not
perform any error checking or handling.
--------
The good thing is that llama is aware of it when asked:
--------
Are you sure this function returns a unix compliant timestamp?
--------
I made a mistake. The function I provided earlier does not correctly calculate
the Unix timestamp.
A Unix timestamp is the number of seconds that have elapsed since January 1,
1970, at midnight in Coordinated Universal Time (UTC). To correctly calculate
this, we need to consider the following:
* The epoch date (January 1, 1970)
* The leap years
* The daylight saving time
Here's a revised version of the function that calculates the Unix timestamp
more accurately:
```c
#include <stdint.h>
uint32_t unix_timestamp(int y, int m, int d, int h, int mi, int s) {
uint32_t timestamp = 0;
// Calculate the seconds since January 1, 1970, 00:00:00 UTC
timestamp += (y - 1970) * 31536000; // 31536000 is the number of seconds in a year
// Adjust for leap years
if ((y % 4 == 0 && y % 100 != 0) || y % 400 == 0)
timestamp++; // Add one second for February 29th
int month_days[] = {31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31};
timestamp += (m - 1) * 2628000 + d * 86400; // Add days and hours
timestamp += h * 3600; // Add hours
timestamp += mi * 60; // Add minutes
timestamp += s;
return timestamp;
}
```
This function takes into account the leap years, but it still does not consider
daylight saving time. If you need to account for daylight saving time, you
would need a more complex algorithm.
Keep in mind that this function is still not fully compliant with the Unix
timestamp standard, as it does not take into account the start and end times of
daylight saving time.
--------
Now, this is pretty nice. And we are all just humans, after all.
Now, let’s see if it understands what it has created.
With the (revised) generated function in the context, let’s ask it what is happening, and what potential issues could be:
What is happening here?
--------
This is a C function named `unix_timestamp` that takes six integer arguments:
year (`y`), month (`m`), day (`d`), hour (`h`), minute (`mi`), and second
(`s`). It calculates the Unix timestamp (number of seconds since January 1,
1970, 00:00:00 UTC) based on these input values.
--------
What are potential problems with the given code?
--------
1. The function does not handle invalid inputs such as dates that do not exist
(e.g., February 30, 29 or April 31), negative months, or times outside of
the range 0-23 for hours and 0-59 for minutes.
2. It does not consider daylight saving time (DST) changes, which can affect
the accuracy of the timestamp calculation.
3. The function assumes that the system's clock is set to UTC, but if the
system is in a different timezone, the calculated timestamp will be off by
the difference between the local timezone and UTC.
4. It does not handle overflow when calculating the timestamp. If the sum of
seconds exceeds 2^32-1 (the maximum value for a uint32_t), the calculation
will wrap around, resulting in an incorrect timestamp.
5. The function can be improved to use more descriptive variable names instead
of single-letter variable names like 'y', 'm', 'd', 'h', 'mi', 's'.
6. It does not consider the fact that month days are different for February and
it's not correct to subtract 1 from month, because January is also a month.
7. The function should return a time_t type instead of uint32_t.
8. The code can be simplified by removing the conditional statements in the
calculation of leap year.
9. The variable `month_days` is not necessary and can be removed.
10. The comment for the line `timestamp += (y - 1970) * 31536000;` should be
updated to accurately describe what it does.
---------
Can you give me an improved version?
--------
Here is a revised version of your code:
```c
#include <stdint.h>
time_t unix_timestamp(int year, int month, int day, int hour, int minute, int second) {
if (month < 1 || month > 12)
return -1;
time_t timestamp = 0;
// Calculate the seconds since January 1, 1970
int total_seconds_in_year = 31536000; // 31536000 is the number of seconds in a year
if ((year % 4 == 0 && year % 100 != 0) || year % 400 == 0)
++total_seconds_in_year;
int[] month_days = {31,28,31,30,31,30,31,31,30,31,30,31};
int day_of_year = 0;
for(int i = 1; i < month; i++) {
day_of_year += month_days[i-1];
}
day_of_year += day - 1;
timestamp = (year - 1970) * total_seconds_in_year + day_of_year * 86400 + hour * 3600 + minute * 60 + second;
return timestamp;
}
```
This version includes error checking for the month, adds a comment to describe what each section of code does, and simplifies some calculations.
--------
Please give me a well-documented version of this code. Please follow best
practices and do not comment overly verbosely.
--------
Here is the code with clear comments:
```c
#include <stdint.h>
// Convert a date in the format (year, month, day, hour, minute, second) to a Unix timestamp.
time_t unix_timestamp(int year, int month, int day, int hour, int minute, int second) {
// Check if the month is within valid range (1-12)
if (month < 1 || month > 12) {
return -1; // Return an error
}
time_t timestamp = 0;
// Calculate the number of seconds in a year, taking into account leap years
int total_seconds_in_year = 31536000;
if ((year % 4 == 0 && year % 100 != 0) || year % 400 == 0) {
++total_seconds_in_year; // Leap year adjustment
}
// Array of days in each month (excluding February)
int month_days[] = {31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31};
int day_of_year = 0;
// Calculate the total number of days before the given month
for (int i = 1; i < month; i++) {
day_of_year += month_days[i - 1];
}
day_of_year += day - 1; // Adjust for 0-based indexing
// Calculate the Unix timestamp
timestamp = (year - 1970) * total_seconds_in_year + day_of_year * 86400 + hour * 3600 + minute * 60 + second;
return timestamp;
}
```
--------
Yes! nice. It also tries to improve it’s code.
Llama 3 understands the given task but is lacking a bit of attention to detail. However, it can improve upon itsolf on reflection.
Ignore the content of the given task, it is merely a repetition of the task from the previous evaluation.
I put the following bullet points in the context of each model, and asked the modell to write the given points in full sentences for a short tech article.
Please write the given points in full sentences for a short tech article.
--------
Here is the rewritten text:
Mistral is capable and competent when it comes to tackling coding tasks.
Additionally, Mistral demonstrates proficiency in general language-related
tasks. Codellama, on the other hand, excels at coding tasks, showcasing its
strong understanding of various libraries. However, Codellama's abilities
dwindle when it comes to general language tasks, indicating a specific area
where it requires improvement.
--------
Please elaborate a bit more.
--------
Here is an elaboration on the text:
Mistral's strengths lie in its ability to tackle coding tasks with ease and
proficiency. Whether it's working on a complex algorithm or debugging a piece
of code, Mistral is well-equipped to handle a wide range of programming
challenges.
In addition to its coding prowess, Mistral also demonstrates a strong
foundation in general language skills. It can understand and communicate
effectively through written text, making it an asset in any team that values
clear communication.
Codellama, on the other hand, stands out for its exceptional abilities in
coding tasks. Its deep understanding of various programming libraries and
frameworks allows it to create efficient, effective, and scalable solutions.
However, Codellama's strengths are largely limited to coding-related areas.
When it comes to general language tasks, such as writing coherent sentences or
conversing naturally, Codellama struggles. This highlights an area where it
requires improvement, potentially through exposure to more diverse linguistic
contexts and training exercises.
Overall, Mistral's versatility in both coding and language skills makes it a
valuable asset, while Codellama's specialized expertise in coding is noteworthy
but may benefit from further development in other areas.
--------
The expanded text, especially the elaborative version, is pretty much spot-on with the given information.
I switched to llama. It yielded very nice results, while (this is not covered in this particular article, maybe I’ll write another one?) being very performant and being able to handle a relatively large context window locally in my environment, which is a large contributing factor to the decision.
]]>This boils down to
There are many different approaches to achieve this, e.g.
I have played around/worked with a few of the given options and changed my setup a few times already. Maybe the observations might be useful to you, or my later self.
The boundary conditions of my setup are as follows
On my personal setup, I don’t have any usecases requiring windows or other non-Linux operating systems, and my security requirements are not that high. Consequently, I have not created and worked with a virtualization setup enough to form an opinion strong enough to write about.
Nix, and with it NixOS, revolve around a functional domain specific language
that can fully declare environments, including installed packages/tools, making
use of the Nixpkgs packages collection, as well as their configuration.
These environments are fully self-contained, reproducible and very flexible. With Nix, it is possible to declare the desired system state, but also independent environments in e.g. the context of a specific development project, or a temporary test environment that can be created and discarded afterwards.
This example defines a nix-shell environment with a specific ruby version:
{ pkgs ? import <nixpkgs> {} }:
pkgs.mkShell {
# nativeBuildInputs is usually what you want -- tools you need to run
nativeBuildInputs = with pkgs.buildPackages; [ ruby_3_2 ];
}
What’s especially powerful with Nix is the fully self-contained and
reproducible build definition, which includes the full definition of the build
environment, as shown in this example, defining the build of chord:
{
pkgs ? import (fetchTarball {
url = "https://github.com/NixOS/nixpkgs/archive/4fe8d07066f6ea82cda2b0c9ae7aee59b2d241b3.tar.gz";
sha256 = "sha256:06jzngg5jm1f81sc4xfskvvgjy5bblz51xpl788mnps1wrkykfhp";
}) {}
}:
pkgs.stdenv.mkDerivation rec {
pname = "chord";
version = "0.1.0";
src = pkgs.fetchgit {
url = "https://gitlab.inria.fr/nix-tutorial/chord-tuto-nix-2022";
rev = "069d2a5bfa4c4024063c25551d5201aeaf921cb3";
sha256 = "sha256-MlqJOoMSRuYeG+jl8DFgcNnpEyeRgDCK2JlN9pOqBWA=";
};
buildInputs = [
pkgs.simgrid
pkgs.boost
pkgs.cmake
];
configurePhase = ''
cmake .
'';
buildPhase = ''
make
'';
installPhase = ''
mkdir -p $out/bin
mv chord $out/bin
'';
Nix’ development is driven by a very active expert community and happens at an impressively quick pace. Most activity, understandably, targets NixOS, not Nix as third-party package manager.
Nix works by having a package store, which contains every package in every installed version in a unique directory, distinguished with a unique hash. In any requested environment, specific versions of everything are “activated” specifically.
The result resembles Containers a lot, but with full system integration as upside and less flexibility as downside. It’s even possible to build an OCI compatible container image out of a Nix environment.
I only tested Nix as package manager on Debian, which, I have to admit, is
somewhat limited. The main real issue I’ve had was with regards to OpenGL,
which is not supported in non-NixOS environments. This prevents any programs to
make proper use of graphics cards, and when I tested it, I tried to define my
sway setup with Nix. Which did not work without a lot of hassle.
There are solutions, but it’s all pretty meh. Besides that, the package management with Nix was pretty slow, taking a lot of additional space and (for my usecases and experience) less powerful than using “normal” OCI containers with Docker/podman.
Guix resembles Nix quite a bit. The principle is the same, you use a functional programming language to fully declare the desired state of your system/package/…, different versions of stuff are put into a package store thing and are activated specifically, it’s a dedicated Distribution (Guix System) and a package manager (Guix), etc.
The main difference is that the theoretical foundation is very nice, clean and impressive. There is an extremely strong focus on software freedom, which is nice, and it doesn’t use a domain specific (and hence a bit limited) description language, but a full fledged functional programming language, GNU Guile, which is a Scheme implementation. It’s pretty nice. And also, pretty weird.
This is an example of a package definition for postgresql-autodoc, which I
created and put into my guix-channel, which is pretty easy to do. Which is,
again, pretty nice.
(define-module (postgresql-autodoc)
#:use-module (guix utils)
#:use-module ((guix licenses) #:prefix license:)
#:use-module (guix packages)
#:use-module (guix git-download)
#:use-module (guix build-system gnu)
#:use-module (gnu packages base)
#:use-module (gnu packages perl)
#:use-module (gnu packages web)
#:use-module (gnu packages databases))
(define-public postgresql-autodoc
(package
(name "postgresql-autodoc")
(version "80a6b150febb5c0c91f2daa433cc089ff1278841")
(source
(origin
(method git-fetch)
(uri (git-reference
(url "https://github.com/cbbrowne/autodoc")
(commit version)))
(file-name (git-file-name name version))
(sha256
(base32
"0ip1xy5qmm5hj392vxfs5fkgxgdv8d9kj7pp6p4x79npxczkxyzd"))))
(build-system gnu-build-system)
(arguments
'(#:phases
(modify-phases %standard-phases
(delete 'configure)
(delete 'check)
(add-before 'install 'patch-prefix
(lambda _
(substitute* (cons* "Makefile")
(("/usr/local") (assoc-ref %outputs "out")))
#t))
(add-after 'install 'wrap-program
(lambda* (#:key outputs #:allow-other-keys)
(let* ((out (assoc-ref outputs "out"))
(path (getenv "PERL5LIB")))
(wrap-program (string-append out "/bin/postgresql_autodoc")
`("PERL5LIB" ":" prefix
(,(string-append out "/lib/perl5/site_perl"
":"
path)))))
#t))
)))
(native-inputs
`(("which", which)))
(inputs
`(("perl" ,perl)
("perl-html-template", perl-html-template)
("perl-term-readkey", perl-term-readkey)
("perl-dbd-pg", perl-dbd-pg)))
(home-page "https://github.com/cbbrowne/autodoc")
(synopsis "PostgreSQL Autodoc")
(description
"This is a utility which will run through PostgreSQL system tables and returns HTML, DOT, and several styles of XML which describe the database.")
(license license:bsd-3)))
Guix doesn’t have Nix’ problems with OpenGL, and it immediately felt much
nicer than Nix to me. I have used it for around a year to manage most of my
environment. The main beef I had with it was details and weird sluggishness in
details. Like with Nix, package management is/can be quite slow. Due to the
nature of things, depending on the current state of the system, it can happen
that you won’t install binary packages but build everything from scratch. Which
happens reliably, reproducibly and nicely, as expected. That’s the point, after
all. But it still takes SO MUCH TIME!! and the fan noise, and … Additionally,
for the graphical environment, I had weird issues I couldn’t quite resolve.
E.g. starting my desktop (sway) would take several seconds longer than with
pure Debian. Similarly, the initial start of my switcher application, rofi,
would take something between 3 and 10 seconds, compared to practically instant
on Debian. I spent a few evenings trying to solve the issue, eventually giving
up.
After a while, I accepted, again, that “normal” containers fit my usecase better.
OCI containers work with namespaces to isolate system resources (processes,
networking, users, mountpoints, ipc, time) and virtual filesystems. The desired
state of a container is freezed in a container image, which basically defines
the virtual base filesystem in layers and certain system interfaces, from
which an actual container can be started. These images are usually defined
in a Dockerfile, which is the build definition of a container image, and can
be stored, distributed and shared. This way, containers can solve problems in
system dependency management, software distribution and orchestration.
This is an example of a Dockerfile, defining an environment based on
ubuntu:jammy:
FROM ubuntu:jammy
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
wget \
gpg \
pkg-config \
&& rm -rf /var/lib/{apt,dpkg,cache,log}/
Containers can be built FROM scratch or based on existing environments, e.g.
benefiting on a strong community and ecosystem. There are base images from
basically every major Linux distribution and distroless images, limiting the
attack surface of application environments. Base images and final images are
cryptographically identified and can be referenced reproducibly and explicitly.
I have been using podman as highlevel container
management solution and alternative to Docker, mainly due to it’s
(root-)daemonless nature.
I use it to run and expose permanent network based local services (see also
this or development environments.
One advantage with podman is that even without additional magic, due to the
user ID mapping, the root UID within a container maps to the UID of the user
who started the container outside of the container. Rootless containers for the
win! This allows you to simply mount a workspace or other resources into a
container and work with it inside of the container without having to worry
about file permissions and ownership.
E.g. like this.
$ podman run --annotation run.oci.keep_original_groups=1 -v $(pwd)/.local/lib/llama/models:/models --device /dev/kfd --device /dev/dri --rm -it localhost/llama-cpp-python-server-rocm:0.2.20 bash
I like the flexibility, manageability and reusability given by containers a lot and use them heavily.
For the creation of Dockerfiles, I make use of
dockerfile-language-server
through coc-docker and a fancy shell
setup with zsh
podman,
and lazydocker, which makes
writing Dockerfiles and working with containers pretty convenient.
However, system integration is not ideal. Some (most?) of that is by design, a container is supposed to be separated from the host system, after all. What if I just want to isolate a specific tool and specify it in the context of a project? Especially, if the tool does not have any specific system dependencies, but is rather self-contained?
As the README of asdf states:
Once upon a time there was a programming language There were many versions of it So people wrote a version manager for it To switch between versions for projects Different, old, new.
Then there came more programming languages So there came more version managers And many commands for them
I installed a lot of them I learnt a lot of commands
Then I said, just one more version manager Which I will write instead
So, there came another version manager asdf version manager - https://github.com/asdf-vm/asdf
A version manager so extendable for which anyone can create a plugin To support their favourite language No more installing more version managers Or learning more commands
In come shim managers! These programs manage the installation of other programs, capable of updating them, installing and activating different versions of the desired program and (ideally) integrating seamlessly with the rest of the environment and ecosystem. This only works, of course, if the managed program does not have any specific system dependencies, e.g. specific shared libraries etc.
There are language specific managers, e.g. for python there is
pyenv, which integrates very nicely with
e.g. poetry and general python virtual environments. Rust has it’s own
installer and toolchain manager, rustup, which is also
capable of managing different versions of rust.
Even more powerful are general purpose shim managers like
asdf and, written in rust, more modern and
compatible with asdf, mise-en-place.
mise has a rich ecosystem of plugins, which allows it to manage versions of a
plethora of programs. These can be activated on a system level, temporary shell
level or in the context of (project) directories.
This could be a system level configuration at ~/.config/mise/config.toml, or
a specific project configuration in ~/workspace/foo/.mise.toml
[tools]
go = "1.22.0"
node = "21.6.1"
java = "adoptopenjdk-21.0.2+13.0.LTS"
ruby = "3.3.0"
awscli = "2.15.19"
direnv = "2.34.0"
mise makes sure that, in the respective context, the specified version of the
specified tools are configured and take precedence in the $PATH.
I have been using pyenv, asdf and rustup for a long time, recently
replacing asdf with mise. Obviously, pyenv and rustup are only used to
manage python and rust.
I have replaced asdf with mise, as it’s much faster, more actively
maintained and maintainable at all. While asdf is very impressive, it’s
written in bash(!), and I’m almost sure that the authors regularly curse
their choice. mise is compatible with asdf, but extends its functionality.
And it works great!
$ java --version
openjdk 21.0.2 2024-01-16 LTS
OpenJDK Runtime Environment Temurin-21.0.2+13 (build 21.0.2+13-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.2+13 (build 21.0.2+13-LTS, mixed mode, sharing)
$ mise shell java@corretto-21.0.2.14.1
mise java@corretto-21.0.2.14.1 downloading amazon-corretto-21.0.2.1 72.68 MiB/199.70 MiB (14s) ███████░░░░░░░░░░░░░ 7s
.
.
.
$ java --version
openjdk 21.0.2 2024-01-16 LTS
OpenJDK Runtime Environment Corretto-21.0.2.14.1 (build 21.0.2+14-LTS)
OpenJDK 64-Bit Server VM Corretto-21.0.2.14.1 (build 21.0.2+14-LTS, mixed mode, sharing)
$ mise use java@corretto-21.0.2.14.1 -p ./
mise ./.mise.toml tools: java@corretto-21.0.2.14.1
$ cat .mise.toml
[tools]
java = "corretto-21.0.2.14.1"
Integration is absolutely seamless, once it’s set up correctly.
If you are interested in the setup, check the (linked) documentation, or take a look at my dotfiles :)
My previous containerization refinement of llama for the local AI assistant stopped after fulfilling the following requirements:
With these requirements, I created a single stage Dockerfile, resulting in a
quite large container image, and stopped right there. It worked, after all.
podman inspect -f "{{ .Size }}" ghcr.io/gierdo/dotfiles/llama-cpp-python-server-rocm:0.2.44-6.0.2-11.0.2 | numfmt --to=si
23G
FROM ubuntu:jammy
RUN apt-get update && apt-get install -y --no-install-recommends \
ninja-build \
libopenblas-dev \
python3-pip \
wget \
gpg \
git \
cmake \
pkg-config \
build-essential
ARG ROCM_VERSION=6.0.2
RUN mkdir --parents --mode=0755 /etc/apt/keyrings \
&& wget https://repo.radeon.com/rocm/rocm.gpg.key -O - | \
gpg --dearmor | tee /etc/apt/keyrings/rocm.gpg > /dev/null \
&& echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/${ROCM_VERSION} jammy main" > /etc/apt/sources.list.d/rocm.list \
&& echo 'Package: *\nPin: release o=repo.radeon.com\nPin-Priority: 600' > /etc/apt/preferences.d/rocm-pin-600 \
&& apt-get update \
&& apt-get install -y --no-install-recommends rocm-hip-sdk
ARG OVERRIDE_GFX_VERSION=11.0.2
ENV HSA_OVERRIDE_GFX_VERSION=${OVERRIDE_GFX_VERSION}
ARG LLAMA_CPP_VERSION=0.2.44
RUN CC=hipcc CXX=hipcc CMAKE_ARGS="-DLLAMA_HIPBLAS=on" pip install llama-cpp-python[server]==${LLAMA_CPP_VERSION}
ENV HOST=0.0.0.0
ENV PORT=8000
EXPOSE 8000
CMD ["bash", "-c", "python3 -m llama_cpp.server --n_gpu_layers 15 --host ${HOST} --port ${PORT}"]
As you can see, I simply dumped the entire rocm-hip-sdk into the image and
that’s that. That means, of course, that the build time dependencies, so
everything needed to compile software to use ROCm is part of the image, next
to everything that is required to execute compiled software. The sole purpose
of the final container is to execute the compiled llama-cpp-python server, so
all the build time dependencies are just wasted space.
Container builds can be split into stages, with only the last stage being baked into the final image. Those stages can depend upon previous stages or share content, copied from one stage into the other.
With that approach, a separate stage can be set up with all the build time dependencies, used to compile the desired artifact, which is then copied into the final image which only contains the run time dependencies.
As docker/podman caches and reuses identical layers, you can build your
Dockerfile in a way that allows the build process to reuse as much as
possible, while still keeping the final image small, in order to speed up your
local build process. You don’t want to wait for apt to download 14Gb of
libraries just because you changed the version of a python package to be
installed, after all.
podman inspect -f "{{ .Size }}" ghcr.io/gierdo/dotfiles/llama-cpp-python-server-rocm:0.2.54-6.0.2-11.0.2 | numfmt --to=si
15G
FROM ubuntu:jammy AS rocm_base
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
wget \
gpg \
pkg-config \
&& rm -rf /var/lib/{apt,dpkg,cache,log}/
ARG ROCM_VERSION=6.0.2
RUN mkdir --parents --mode=0755 /etc/apt/keyrings \
&& wget https://repo.radeon.com/rocm/rocm.gpg.key -O - | \
gpg --dearmor | tee /etc/apt/keyrings/rocm.gpg > /dev/null \
&& echo "deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/${ROCM_VERSION} jammy main" > /etc/apt/sources.list.d/rocm.list \
&& echo 'Package: *\nPin: release o=repo.radeon.com\nPin-Priority: 600' > /etc/apt/preferences.d/rocm-pin-600
# Allow caching the runtime base even if llama_cpp is rebuilt
FROM rocm_base AS runtime_base
RUN apt-get update && apt-get install -y --no-install-recommends \
python3-pip \
rocm-hip-libraries \
&& rm -rf /var/lib/{apt,dpkg,cache,log}/
FROM runtime_base AS builder
RUN apt-get update && apt-get install -y --no-install-recommends \
ninja-build \
libopenblas-dev \
git \
cmake \
pkg-config \
build-essential \
rocm-hip-sdk \
&& rm -rf /var/lib/{apt,dpkg,cache,log}/
ARG OVERRIDE_GFX_VERSION=11.0.2
ENV HSA_OVERRIDE_GFX_VERSION=${OVERRIDE_GFX_VERSION}
ARG LLAMA_CPP_VERSION=0.2.54
RUN CC=hipcc CXX=hipcc CMAKE_ARGS="-DLLAMA_HIPBLAS=on" pip install --user llama-cpp-python[server]==${LLAMA_CPP_VERSION}
FROM runtime_base AS final
LABEL maintainer="Dominikus Gierlach <dominik.gierlach@gmail.com>"
ARG OVERRIDE_GFX_VERSION=11.0.2
ENV HSA_OVERRIDE_GFX_VERSION=${OVERRIDE_GFX_VERSION}
COPY --from=builder /root/.local /root/.local
ENV HOST=0.0.0.0
ENV PORT=8000
EXPOSE 8000
CMD ["bash", "-c", "python3 -m llama_cpp.server --n_gpu_layers 15 --host ${HOST} --port ${PORT}"]
The resulting image is still large and could be slimmed down. It now contains the full collection of ROCm libraries and runtime dependencies. I could probably walk through them and just identify the stuff that is strictly required. But losing 8Gb of image size, basically for free, is a better use of my time, I think.