Local LLM as a private writing assistant

(This post is written as of August 2024.)

I just leave some notes for my experiment of setting up a local LLM that helps to improve my writing by giving internal peer-review. Though the outcome was not satisfying in my case, it showed some potentials to be usable in the not-far future.

Conditions and goals

• Full text is already written in English. It should be understandable, but contains grammatical errors and could be largely improved.
• Sufficient scientific discussions are included, but formulations of them are sometimes not clear.
• The author tends to write lengthy and indirect sentences and paragraphs. Squeezing those blocks contributes to better readability.
• Available GPU is for home-use or office-use with its memory up to 16 GB.

1. A LLM gives review comments on the entire manuscript like a proper peer-review, but only about language and style improvements.
2. Suggest better formulation of an entire section or subsection, including a change in the strusture and the order of paragraphs.
3. Suggest grammatical corrections and reformulations without violating the content.

Conclusions

• Small LLMs that are able to run on an office-use GPU (with 16GB memory) do not provide nice review, but are available for corecting grammatical errors paragraph by paragraph.
• Their context window is too small to even store one subsection. A single paragraph would be a largest block as input.
• Feeding PDF was not working nicely as expected. Encoding or emmbedding PDF seems not perfect yet.

Some results

• The context is not recognised at all. The models print an original abstract of a document that is missing its abstract. It seemed like a copy from a paper using data from NOAA, from which I has never used any data in my study.
• Their output tends to be hyperbolic and to use bullet points for everything.