在幽兰上体验大语言模型（LLM）

大语言模型是今年的一个热门技术。它到底怎么样呢？其实只要通过下面的三步操作，大家就可以在幽兰上亲自体验一下LLM。

1. 准备工作，安装docker

详见本wiki的高级技巧部分：https://www.nanocode.cn/wiki/docs/youlan/docker

2. 下载docker镜像

在大模型如火如荼的今天，互联网上公开了很多训练好的大模型，比如下面的github项目列出了一些已经封装为docker镜像的大模型推理环境，方便大家体验。
https://github.com/theone10zero/llm-docker-images/tree/main/mlc-llm-rk3588

以下是格蠹团队曾测试过的几个docker镜像，供大家参考：

sudo docker run –rm -it –privileged vicalloy/mlc-llm-rk3588:FlagAlpha-Llama2-Chinese-7b-Chat-q4f16_1

docker run –rm -it –privileged docker.io/milas/mlc-llm:redpajama-3b

3. 启动mlc-llm，与大模型对话

sudo docker run –rm -it –privileged docker.io/milas/mlc-llm:redpajama-3b

当你看到[human]提示符，就可以向大模型提问了。

Use MLC config: “/mlc-llm/dist/prebuilt/RedPajama-INCITE-Chat-3B-v1-q4f16_1/mlc-chat-config.json”
Use model weights: “/mlc-llm/dist/prebuilt/RedPajama-INCITE-Chat-3B-v1-q4f16_1/ndarray-cache.json”
Use model library: “/mlc-llm/dist/prebuilt/lib/RedPajama-INCITE-Chat-3B-v1-q4f16_1-mali.so”
You can use the following special commands:
/help print the special commands
/exit quit the cli
/stats print out the latest stats (token/sec)
/reset restart a fresh chat
/reload [local_id] reload model local_id from disk, or reload the current model if local_id is not specified

Loading model…
arm_release_ver of this libmali is ‘g6p0-01eac0’, rk_so_ver is ‘7’.
Loading finished
Running system prompts…
System prompts finished
: