Local LLMs are great and you should be using them
One of the benefits of being laid off is getting an extra laptop to play around with. One of the things everyone should be playing around with right now is local llm models.
Did you know you could just run an LLM?
There are a couple goals with this project:
- Get a local llm running on my network
- Build an agent that can
Recently, I have been working with a couple folks to help build out an "AI Executive Assistant" to summarize recent emails, chats, documents, and other tools into a single daily report. As I was putting it together I realized that while some of the services like Google Workspace and Notion and others have these types of tools built in, there isn't a great way to connect them all together.
You can use tools that are out there like n8n. Or ideally, you can use something like Claude Cowork to be the center piece that brings it all together - but this might not be an option for someone in a corporate setting where there are privacy concerns. (Lets assume that Google Workspace and all are secure enough, which likely also is why their AI tools are isolated today.)
Either way - this has been working well enough, but I wanted to see what I could do for myself to get an always on local llm that wasn't exposed to the internet itself, but allowed me to use it.
Local Setup
Locally, I have this MacBook Pro M4 with 48gb of RAM (thanks to my most recent employer). This is running ollama with a couple different models available. There are some other setup details for keeping the laptop awake all the time. It also is running Claude and Claude Code CLI - but those are for other developer tooling I might do in the future.
In general, I use Obsidian for all my note taking - business, personal projects, writing, non-profit work, todo lists, and really anything else. Almost everything gets tracked there. The nice part about Obsidian is that it is all filesystem based and the notes are plain text markdown.
I've already been using Obsidian as my "second brain" for organizing thoughts - but it is also a perfect place for an agent's second brain. Claude already stores alot of plain text memory for itself. I know people are all excited about using HTML with LLMs - but for memory and context, the only thing better than plain text is some kind of vector database.
So, the M4 Pro is also running Obsidian to keep a set of notes all up to date.
The Agent
I already have a Claude Cowork scheduled task to pull my Google Workspace AI summaries and some other sources. This is mainly because it has the connections built in, but it is an effective workflow builder.
- Google Workspace Gem builds summary and creates a daily Google Doc dossier
- Claude pulls the summary from Google and other email and calendar sources
- Claude creates a markdown summary of all of it in the Obsidian vault
The missing piece is Obsidian notes themselves. Obviously, Claude could handle this - but Claude is big and I want to save my tokens for dev work not just writing summaries and consolidating todo lists.
Dev work like building an agent in Rust.
One other problem that I have is that sometimes I like to take quick notes - someone mentions a book to check out or I come across an article I want to read later.
This is solvable, the local llm is already running and accessible - lets put it to work.
Here is the spec for the agent:
- An "Inbox" feature that would organize a note added to the Inbox folder, adding tags, moving to the appropriate folder, summarizing URL links, and creating action items
- An immediate feature, where I could add a specific tag to summarize or do research on a note
- A daily process to consolidate and report on notes touched the previous day
This little agent runs in the terminal and watches my Obsidian vault. Anything added to the Inbox folder that is idle for 30 minutes gets processed. Any note tagged with tags it recognizes will be queued up to be processed. All of it going through the local LLM.
This works because I don't need ground breaking insights - I just need fuzzy organization and summaries. Local LLMs are more than capable at this task. In fact, I'm certain that it could do even more complicated work - but, you have to crawl before you can run, you know?
Why Rust? Because, why not? Really, we can build whatever we want now.
I am on about the third iteration to add some more functionality and clean up the daily note. The next step is to add some better configuration options and editing, then do a general release for Obsidian users.
Stay tuned!