TOK is a special trigger word that is used in the training to associate the style you have trained with something you can include in the prompt. If you run the model without the trigger word in the prompt, you won't really activate the creative style you trained into the model's knowledgebase. To make it more confusing, TOK is really just a random choice, with the unique benefit of being, well, unique. The model doesn't have much of anything associated with TOK, so it is a good candidate for a trigger word. When you train the model, you can choose any trigger word your heart desires. There's also a lot more detail I could go into with captioning data for training, but that will have to be its own article.

I recently started exploring Replicate to host my fine tuned models, and along the way discovered that Replicate has some handy and efficient ways to train models using their machines in the cloud. To run the trainings for these models, I used the Replicate Training API which I accessed via a Google CoLab notebook that someone on the replicate team made (thank you!). This proved to be an extremely efficient way to fine tune models, but with a few important trade offs:

1. You have to send your data to a cloud server. I believe that Replicate provides a secure and private environment, but realistically this will be a non-starter for certain clients. I’m my case, this is not client work, so no biggie for me.

2. The training process is dramatically simplified by the behind-the-scenes training parameters being preset by Replicate, but you lose a lot of control in the choices you can make about how you train the model such as what optimizer you want use.

Replicate also includes a handy guide to help you get started when you want to Fine tune SDXL with your own images, as well as a great interface for using your model after training. I’ve been extremely impressed by the Replicate team and platform, they are definitely ones to watch in this space.

In the past I have used the open source Kohya_SS framework to run my model tunings. This has the benefit of running locally on your machine, so when working with sensitive client materials there are advantages to this approach which allows you to maintain tight control on where data gets sent. But it is also much more complicated, as you must take full control over every training parameter, which is both awesome but daunting. Model finetuning is not an exact science, and for every unique dataset, there are likely parameter difference that would improve the performance of the model. You also want to be comfortable using command line tools (CLI) for this. Or just have ChatGPT handy to explain stuff to you.

To assess the performance of finetuned models, we employ a multi-faceted approach:

1. Visual Evaluation: We generate a diverse set of images and assess them for style consistency, quality, and adherence to the intended aesthetic.

2. Prompt Sensitivity: We test the model's responsiveness to minor prompt variations to gauge its stability and flexibility.

3. Comparative Analysis: We compare outputs with those of the base model to measure the impact of finetuning.

4. User Feedback: We collect opinions from artists and designers to evaluate the model's practical utility.

5. Computational Metrics: We analyze parameters like loss curves during training and inference times to ensure efficiency.

Synthetic data refers to data created by AI that is used in the training of new AI models. The LoRA models I created used images generated with an AI system (Midjourney) to create the dataset used for training.