pro tip use Linux distro from any of the lists posted here, use docker for your local llm, which are often Ubuntu based. consider getting a more modern graphics card like 30 series with 16gb vram
I had a contract come up and had to shelve this for a bit, and your comment immediately annoyed me, because it really isn’t what I wanted to hear
But it also stuck with me because it sounded like the advice I throw at new devs starting a project, knowing it’s a PITA up front, but pays dividends pretty quick.
So I looked it up, and despite my bad experiences with docker and kubernetes (I was tasked with doing weird, off label things with them and it sucked), I’ve decided to take your advice and stop looking for docker workarounds
And since it seems like it comes from a place of experience, I figured I’d share a bit more about what I want to do and see if you had any more advice
Basically, I want to link together basic models trained to do different things, with the end goal being something between a conversation partner and an assistant. The idea being I build very specific prompts to bypass the limitations of smaller models - the first goal is to take one LLM and a conventional management program and summarize key information, then use very specific structured prompts to generate a response to be vocalized and metadata that changes the state of the management system.
My thought is to take something like alpaca or falcon 7B to track and summarize relevant information, feed it into another such model trained as a conversation partner with this input and output format, then throw together a web interface and do text<->speech on my phone or dev computer.
When it comes to neural networks and LLMs, I have a good understanding of the theory of them and a great one of how brains work, but I’m mostly looking to use these systems as a black box initially. My initial goals are to generate dialogue trees for games and maybe practice my Spanish with a chatbot - accuracy and capabilities don’t matter too much, I’ve played with projects that could do this by just sending prompts to an endpoint
Down the road, the goal is to have something extremely modular. This tech is moving fast and I envision linking a bunch of modules together to perform different tasks, and as better modules come out or I add/upgrade hardware, I want to be able to write something to act like autopilot in my ide or pilot a model in a game engine
The main objective is to learn and to run agents on my own hardware. I’m looking for a side project that will be useful enough to keep up my interest, but also give me a starting point to modify from so I’m not sitting at a python terminal forcing myself through a tensor flow course before I get to the good stuff
Any thoughts, advice, or projects you think I should know about when starting this journey?