From 66927d05d54798475c133d2f1a145eac04442e6c Mon Sep 17 00:00:00 2001 From: Prabhu Subramanian Date: Sat, 8 Feb 2025 15:55:47 +0000 Subject: [PATCH] document temperature and system prompt sensitivity Signed-off-by: Prabhu Subramanian --- contrib/fine-tuning/Modelfile | 6 +- contrib/fine-tuning/README.md | 10 +- contrib/xBOMEval/README.md | 2 + contrib/xBOMEval/cdx1-NOTES.md | 236 +++++++++++++++++++++++++++++++++ 4 files changed, 245 insertions(+), 9 deletions(-) create mode 100644 contrib/xBOMEval/README.md create mode 100644 contrib/xBOMEval/cdx1-NOTES.md diff --git a/contrib/fine-tuning/Modelfile b/contrib/fine-tuning/Modelfile index 287e6c8b6..e523d0d78 100644 --- a/contrib/fine-tuning/Modelfile +++ b/contrib/fine-tuning/Modelfile @@ -1,11 +1,9 @@ FROM ./cdx1-gguf-q8_0.gguf -PARAMETER num_ctx 16384 +PARAMETER num_ctx 16000 PARAMETER temperature 0.05 -PARAMETER top_k 10 -PARAMETER top_p 0.5 -SYSTEM """You are cdxgen, an expert in CycloneDX and xBOM.""" +SYSTEM """You are a helpful assistant.""" LICENSE """ apache-2.0 diff --git a/contrib/fine-tuning/README.md b/contrib/fine-tuning/README.md index 7be409be0..5f0aa7954 100644 --- a/contrib/fine-tuning/README.md +++ b/contrib/fine-tuning/README.md @@ -30,13 +30,13 @@ bash fine-tune-mlx.sh cp -rf prabhuat ~/.lmstudio/models/ lms ls lms server status -lms load CycloneDX/cdx1-mlx --exact --gpu max --identifier cdx1-test --context-length 8192 +lms load CycloneDX/cdx1-mlx-8bit --exact --gpu max --identifier cdx1-test --context-length 16000 ``` System prompt: ```text -You are cdxgen, an expert in CycloneDX and xBOM. +You are a helpful assistant. ``` ### gguf testing with ollama @@ -53,18 +53,18 @@ ollama show cdx1-gguf Model architecture llama parameters 14.7B - context length 16384 + context length 16000 embedding length 5120 quantization F16 Parameters - num_ctx 16384 + num_ctx 16000 temperature 0.05 top_k 10 top_p 0.5 System - You are cdxgen, an expert in CycloneDX and xBOM. + You are a helpful assistant. License apache-2.0 diff --git a/contrib/xBOMEval/README.md b/contrib/xBOMEval/README.md new file mode 100644 index 000000000..3d07efe55 --- /dev/null +++ b/contrib/xBOMEval/README.md @@ -0,0 +1,2 @@ +# Introduction + diff --git a/contrib/xBOMEval/cdx1-NOTES.md b/contrib/xBOMEval/cdx1-NOTES.md new file mode 100644 index 000000000..7254aa234 --- /dev/null +++ b/contrib/xBOMEval/cdx1-NOTES.md @@ -0,0 +1,236 @@ +## System Prompt sensitivity + +Using the system prompt `You are a helpful assistant.` improves the quality of the generated text. Below we document the responses for the same question with alternative system prompts. + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.05 --system-prompt "You are cdxgen, an expert in CycloneDX and xBOM." +========== +The bomFormat property specifies the format of the BOM. It can be used to indicate the version of the BOM format, such as "bom-1.4" for CycloneDX 1.4. +========== +``` + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.05 --system-prompt "You are cdxgen, an helpful assistant." +========== +The bomFormat property specifies the format of the BOM. It is a required property and should be set to "application/vnd.cyclonedx.bom+json" for a CycloneDX BOM. +========== +``` + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.05 --system-prompt "You are cdxgen, an expert in CycloneDX specification and cdxgen tool." +========== +The bomFormat property specifies the format of the BOM. It can be set to "application/vnd.cyclonedx.bom+json" for JSON format or "application/vnd.cyclonedx.bom+xml" for XML format. +========== +``` + +Good results with this system prompt. + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.05 --system-prompt "You are a helpful assistant." +========== +The `bomFormat` property specifies the format of the BOM. It is a required property and should be set to "CycloneDX". +========== +``` + +Repeat the test with multiple temperature. + +Temperature: 0.1 + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.1 --system-prompt "You are a helpful assistant." +========== +The `bomFormat` property specifies the format of the BOM. It is a required property and should be set to "CycloneDX". +========== +``` + +Above 0.1, the models gets **lazy**. + +Temperature: 0.2 + +``` +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.2 --system-prompt "You are a helpful assistant." +========== +The `bomFormat` property specifies the format of the BOM. It is a string value that identifies the format of the BOM. +========== +``` + +Temperature: 0.3 + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.3 --system-prompt "You are a helpful assistant." +========== +The `bomFormat` property specifies the format of the BOM. It is a string value and is required. +========== +``` + +Temperature: 0.4 + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.4 --system-prompt "You are a helpful assistant." +========== +The `bomFormat` property specifies the format of the BOM. It is a string value and can be used to indicate the format of the BOM, such as "cyclonedx", "spdx", or "sbom". +========== +``` + +Temperature: 0.5 + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.5 --system-prompt "You are a helpful assistant." +========== +The `bomFormat` property specifies the format of the BOM. +========== +``` + +Temperature: 0.6 - 1.0 (Identical responses) + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.6 --system-prompt "You are a helpful assistant." +========== +The `bomFormat` property specifies the format of the BOM. +========== +``` + +## Temperature sensitivity + +We tested temperature sensitivity without a system prompt. Below are the results: + +0.05 - 0.1 is the optimal temperature for `cdx1` family of models. Anything above 0.1 reduces the quality of the generated response. + +### Test + +Question: What is the purpose of the bomFormat property in a CycloneDX BOM? +Answer: The `bomFormat` property specifies the format of the Bill of Materials (BOM). It is used to identify the file as a CycloneDX BOM because BOMs do not inherently have a filename convention, and JSON schema does not support namespaces for this purpose. The value of this property must always be \"CycloneDX\" to clearly indicate that the document is indeed a CycloneDX BOM. + +### Responses + +For the fused mlx model `bf16`, `0.1` appears to yield a decent response. + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.05 +========== +The bomFormat property specifies the format of the BOM. It is used to indicate the format of the BOM, such as JSON, XML, or YAML. +========== +``` + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.1 +========== +The bomFormat property specifies the format of the BOM. It can be used to indicate the version of the CycloneDX specification or other relevant information about the BOM format. +========== +``` + +For the quantized versions, 0.05 and 0.1 yield acceptable response. + +Temperature: 0 + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0 +========== +The bomFormat property specifies the format of the BOM. It is used to indicate the version of the CycloneDX specification that the BOM conforms to. +========== +``` + +Temperature: 0.01 + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.01 +========== +The bomFormat property specifies the format of the BOM. It can be used to indicate the version of the CycloneDX specification or other formats. +========== +``` + +Temperature: 0.05 + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.05 +========== +The bomFormat property specifies the format of the BOM. It can be used to indicate the version of the CycloneDX specification or other formats. +========== +``` + +Temperature: 0.1 + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.1 +========== +The bomFormat property specifies the format of the BOM. It can be used to indicate the version of the CycloneDX specification or other formats. +========== +``` + +Temperature: 0.2 + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.2 +========== +The bomFormat property specifies the format of the BOM. It can be used to indicate the version of the CycloneDX specification or other format identifiers. +========== +``` + +Temperature: 0.3 + +Creative answers begin from 0.3 onwards. + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.3 +========== +The bomFormat property specifies the format of the BOM. It's always set to "CBOR" for CycloneDX BOMs. +========== +``` + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.4 +========== +The bomFormat property specifies the format of the BOM. It's always set to "CBOR" for CycloneDX BOMs. +========== +``` + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.5 +========== +The bomFormat property specifies the format of the BOM, such as JSON or XML. +========== +``` + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.6 +========== +The bomFormat property specifies the format of the BOM, such as JSON or XML. +========== +``` + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.7 +========== +The bomFormat property represents the version of the BOM specification that the document adheres to. It's used to ensure that the BOM is compliant with the specified version. +========== +``` + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.8 +========== +The bomFormat property represents the version of the BOM specification that the document adheres to. It's used to ensure that the schema used to validate the document is correct. +========== +``` + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 0.9 +========== +The bomFormat property represents the version of the BOM specification that the document adheres to, allowing for interoperability and understanding of schema format. +========== +``` + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 1.0 +========== +The bomFormat property represents the version of the BOM specification that the document adheres to, allowing for context and interpretation of the schema format. +========== +``` + +```text +mlx_lm.generate --model ./CycloneDX/cdx1-mlx-8bit --prompt "What is the purpose of the bomFormat property in a CycloneDX BOM?" --temp 1.1 +========== +The bomFormat property represents the version of the BOM specification that the document adheres to, supported formats include bom-1, bom-10, bom-11, bom-12, and nonconformant for custom implementations. +========== +``` + +Beyond 1.1, the responses degrade significantly.