Fine-tuning Psico-IA: From 337 Reports to Our Own Model

After my first "Hello World!" with agents using n8n and comparing LLMs through their APIs, one question kept nagging me: what if, instead of paying per token on every request, I trained my own specialized model?

With Psico-IA (the natural successor to PsicoEnLine@, my psychometric reporting system built on if-else statements) I had the perfect use case: 337 real reports accumulated since 2015, all with the same structure, all written by human analysts following a consistent pattern. If any dataset was begging for fine-tuning, it was this one.

The dilemma: RAG, context, or fine-tuning

My first instinct was to ask whether I needed RAG. I quickly ruled it out: RAG is for retrieving existing information, and I didn't need to look up old reports, I needed to generate new reports in the same style. Stuffing all 337 examples into every prompt didn't make sense either: they don't fit in the context window, and they don't "teach" the pattern robustly, they just show it once in passing.

The right answer was SFT (Supervised Fine-Tuning): instruction/response pairs, where the instruction is the raw psychometric test results and the response is the professional write-up an analyst produces by hand today. It's exactly the kind of repetitive, structured task where fine-tuning shines.

Hardware: the constraint that defined everything

Unlike my previous n8n experiment (where I simply paid for API calls), this time I set myself a different goal: do it with as little hardware as possible, without buying anything. That led me straight to QLoRA, the combination of LoRA with 4-bit quantization, which trains only a small adapter on top of a frozen, compressed base model instead of retraining billions of parameters.

With QLoRA, a 7-billion-parameter model (I chose Alibaba's Qwen2.5-7B-Instruct) fits comfortably on the 16GB T4 GPU that Google Colab offers for free. I also considered larger models such as DeepSeek V4 Flash (284 billion total parameters) or Kimi K2.6 (1 trillion parameters), and ruled both out immediately: even though their MoE architecture activates few parameters per token, they still need all their experts loaded in memory, which requires H100 GPU clusters and hundreds of dollars just for fine-tuning. Nothing a laptop or free Colab can handle.

Preparing the dataset: the ugly part of the job

My 337 reports lived in a MySQL table, with psychometric test results mixed into a single free-text field alongside the job description and the analyst's conclusions. The first technical challenge was to correctly separate the "input" (numeric results + required competencies) from the "output" (the narrative analysis the model had to learn to write).

I built a query that wraps everything in JSON_OBJECT to avoid escaping problems with the free text:

SELECT JSON_OBJECT(
    'instruction', CONCAT('Resultados de pruebas:\n', ...),
    'output', s.observaciones
) as jsonl_row
FROM grupo_sujetos_tabla s
INNER JOIN grupo g ON g.codigo = s.codigo_grupo
WHERE s.observaciones IS NOT NULL AND s.observaciones <> ""

Then came a Python script with regex to split the job description (input) from the competency-by-competency analysis (output), using the phrase "Tomando en cuenta esta información..." ("Taking this information into account...") as the marker. Surprise along the way: not every report followed the same format. Those based on projective tests (such as Wartegg) went straight into the narrative analysis, with no job description. I had to adjust the script to recognize both formats without discarding any as an "error".

The end result: a .jsonl file where each line is a complete JSON object with instruction and output, the format Unsloth (the framework I ended up using) expects for training data.

Training: theory meets free infrastructure

The training script itself was fairly simple: load Qwen 7B quantized to 4-bit, apply LoRA adapters to the attention and feed-forward layers, and train with HuggingFace's SFTTrainer for 3 epochs. What wasn't simple was dealing with the quirks (to put it politely) of free Colab:

• Colab wipes everything that isn't in Google Drive when the session restarts. I lost a full training run of over an hour because I didn't mount Drive from the start.
• Writing large files directly to a mounted Drive can silently corrupt them. A multi-gigabyte file ended up at 0 bytes, with no visible error until I tried to use it.
• Unsloth appends its own _gguf suffix to the folder name you give it, creating a folder with a duplicated name that almost made me think the final file was never generated.
• Colab doesn't publish a monthly limit on free hours, only a 12-hour cap per session, with GPU availability that fluctuates with demand and no guarantees at all.

After three attempts (two of them completely lost), I managed to export the merged, quantized model as a 4-bit GGUF file: 4.68 GB, ready for Ollama.

Running the model at home

With Ollama installed on my Ubuntu machine (the same Ryzen 7 with 16GB of RAM from the previous experiment, this time with an AMD GPU that turned out to have just 2GB of VRAM, not enough to accelerate anything), the model runs on CPU with responses taking 10 to 30 seconds: slow, but good enough to validate quality.

The first test revealed unexpected behavior: when asked to evaluate only 4 specific competencies for a candidate, the model produced an analysis of 12. The reason: my training dataset never explicitly taught it "evaluate only these competencies and nothing else"; it simply learned the general pattern of "generate a competency analysis" from test results. I had to reinforce the system prompt with explicit rules and repeat the constraint in the user prompt so it would respect the limit.

I also noticed that when I didn't give it the candidate's name, the model made up names it had seen during training ("Mr. Rodríguez", "Torrens"...), an uncomfortable reminder that I never anonymized the original dataset before training.

Preliminary Results

Comparing our own fine-tuned model against what I had already tested in my previous n8n experiment:

Unlike the "stock" deepseek-r1:7b, which was sluggish and didn't follow instructions in my previous experiment, the Qwen 7B fine-tuned on my own 337 reports does produce analyses consistent with my analysts' real style. The difference between a generic model and one specialized in my specific domain is remarkable, although careful prompt tuning is still needed for it to respect specific constraints.

Conclusion

The whole project, end to end, cost me less than $5 in actual compute: almost the entire investment was time, not money. I also compared it against using the DeepSeek API directly for the same estimated volume (around 10,000 reports a year): in pure dollars, the API is still cheaper than hosting my own model. But here a variable comes in that didn't exist in my n8n experiment: this is HR psychometric data, sensitive information about people being evaluated for jobs. Running the model on my own infrastructure, even if it costs a few more dollars a year, isn't a cost decision, it's a decision about control over where that data lives.