Compare commits

...

12 commits

Author SHA1 Message Date
e3e091974d doc: use the new config format
All checks were successful
continuous-integration/drone/push Build is passing
2024-01-24 14:18:00 +01:00
74a52a43e0 fix: ensure files are sorted by name
All checks were successful
continuous-integration/drone/push Build is passing
2024-01-23 16:23:32 +01:00
2d97187e4f doc: add example prompts 2024-01-23 16:23:11 +01:00
b60f030824 fix: make unit tests work with CODE_PRELOADER_DETECT=no
All checks were successful
continuous-integration/drone/push Build is passing
2024-01-05 11:48:12 +01:00
bde15cfd25 feat: env CODE_PRELOADER_DETECT=no disables config auto-detect 2024-01-05 11:47:52 +01:00
d39d597128 feat: use embedded filesystem for default files
Some checks failed
continuous-integration/drone/push Build is failing
2024-01-04 23:14:14 +01:00
97a05896a3 feat: add defaults (template & config) as files 2024-01-04 23:03:21 +01:00
a53ae57f51 Merge branch 'develop' of code.apps.glenux.net:glenux/code-preloader into develop
Some checks failed
continuous-integration/drone/push Build is failing
2024-01-04 22:55:33 +01:00
46ef698224 fix: add missing patch for jinja 2024-01-04 22:54:32 +01:00
3c28d5eace refactor: move docs/ to misc/logo/ 2024-01-04 22:53:38 +01:00
3f75ccca34 refactor: move prompts/ to misc/ 2024-01-04 22:53:21 +01:00
7ff161f4ee Merge pull request 'refactor: store processed content for crinja (later)' (#15) from feature/issue/9-add-support-for-jinja-template into develop
Some checks failed
continuous-integration/drone/push Build is failing
Reviewed-on: #15
2024-01-04 21:51:46 +00:00
23 changed files with 133 additions and 71 deletions

View file

@ -17,5 +17,5 @@ output_path: null
prompt: prompt:
header_path: null header_path: null
footer_path: null footer_path: null
template_path: misc/templates/default.j2 template_path: null
# #

View file

@ -158,9 +158,12 @@ ignore_list:
- bin - bin
- prompts - prompts
prompt:
header_path: prompts/context.txt
footer_path: prompts/request-readme.txt
template_path: null
output_file_path: null output_file_path: null
header_prompt_file_path: prompts/context.txt
footer_prompt_file_path: prompts/request-readme.txt
``` ```
Then you can type a shorter command like: Then you can type a shorter command like:

View file

@ -0,0 +1,7 @@
Please focus on «@@ FILE ...» and analyze the content of «@@ FILE "{{FIXME: name of the file you want ot analyze}}"».
Provide your analysis as an appreciative feedback, stating:
- what you appreciate in current version
- what you would amplify / augment
- what you would adjust / change / remplace

View file

@ -0,0 +1,3 @@
Focus on the existing «@@ FILE "README.md"». and concentrate on your analysis and appreciative feedback too.
Please take in account each change you propose and write a fixed/improved version of the «@@ FILE "README.md"»

View file

@ -0,0 +1,10 @@
I would like to replace the hardcoded structure of the output "@@ ... " with a jinja template that would be handler with the crinja.cr shard.
My strategy is to preload the content during process_file and store it in a hash.
Then the hash will be used in the end to render a final template (from crinja).
Can you please tell me :
* where the changes should occur (which file? which class? which method?)
* what kind of changes must be made there?
Please do not write code yet, simply explain.

View file

@ -0,0 +1,7 @@
We want to improve the README.md file as for an open-source project following the best practices.
Please focus on the code provided in the « @@ File ... » sections and analyze the existing « @@ File "README.md" », then take in account the content of class Config which analyzes the command line.
In this analysis of the existing « @@ File "README.md" », please provide me with a list of things that you appreciate, a list of things to amplify, a list of things to adjust.

View file

@ -0,0 +1,7 @@
We want to improve the README.md file as for an open-source project following the best practices.
Please focus on the code provided in the « @@ File ... » sections and analyze the existing « @@ File "README.md" », then take in account the content of class Config which analyzes the command line.
Please concentrate on the detail of your analysis in your previous message and write a new improved/fixed version of « @@ File "README.md"

View file

@ -1,5 +1,3 @@
@@REQUEST
I would like to change CLI parameters structure add add two subcommands: I would like to change CLI parameters structure add add two subcommands:
* init : which will create an example .code_preloader.yml file (with comments) * init : which will create an example .code_preloader.yml file (with comments)
* pack : which will create the packed version of the current directory for LLM prompting * pack : which will create the packed version of the current directory for LLM prompting

View file

@ -1,16 +0,0 @@
@@ REQUEST (ANALYZE)
We want to improve the README.md file as for an open-source project following the best practices.
Please focus on the code provided in the « @@ File ... » sections and analyze the existing « @@ File "README.md" », then take in account the content of class Config which analyzes the command line.
In this analysis of the existing « @@ File "README.md" », please provide me with a list of things that you appreciate, a list of things to amplify, a list of things to adjust.
@@ REQUEST (WRITE)
We want to improve the README.md file as for an open-source project following the best practices.
Please focus on the code provided in the « @@ File ... » sections and analyze the existing « @@ File "README.md" », then take in account the content of class Config which analyzes the command line.
Please concentrate on the detail of your analysis in your previous message and write a new improved/fixed version of « @@ File "README.md"

View file

@ -1,5 +1,9 @@
version: 2.0 version: 2.0
shards: shards:
baked_file_system:
git: https://github.com/schovi/baked_file_system.git
version: 0.10.0
crinja: crinja:
git: https://github.com/straight-shoota/crinja.git git: https://github.com/straight-shoota/crinja.git
version: 0.8.1 version: 0.8.1

View file

@ -18,6 +18,9 @@ dependencies:
github: alexherbo2/walk.cr github: alexherbo2/walk.cr
version_from_shard: version_from_shard:
github: hugopl/version_from_shard github: hugopl/version_from_shard
baked_file_system:
github: schovi/baked_file_system
version: 0.10.0
# completion: # completion:
# github: f/completion # github: f/completion

View file

@ -12,10 +12,11 @@ ignore_list:
- complex/ignore2 - complex/ignore2
# Path to the output file (if null, output to STDOUT) # Path to the output file (if null, output to STDOUT)
output_file_path: complex_output.txt output_path: complex_output.txt
# Optional: Path to a file containing the header prompt prompt:
header_prompt_file_path: null # Optional: Path to a file containing the header prompt
header_path: null
# Optional: Path to a file containing the footer prompt # Optional: Path to a file containing the footer prompt
footer_prompt_file_path: null footer_path: null

View file

@ -11,10 +11,11 @@ ignore_list:
- simple/ignore - simple/ignore
# Path to the output file (if null, output to STDOUT) # Path to the output file (if null, output to STDOUT)
output_file_path: simple_output.txt output_path: simple_output.txt
# Optional: Path to a file containing the header prompt prompt:
header_prompt_file_path: null # Optional: Path to a file containing the header prompt
header_path: null
# Optional: Path to a file containing the footer prompt # Optional: Path to a file containing the footer prompt
footer_prompt_file_path: null footer_path: null

View file

@ -50,7 +50,7 @@ describe CodePreloader::Config do
config.subcommand.should eq(Config::Subcommand::Pack) config.subcommand.should eq(Config::Subcommand::Pack)
config.pack_options.should be_truthy config.pack_options.should be_truthy
config.pack_options.try do |opts| config.pack_options.try do |opts|
opts.repository_path_list.should eq ["path/to/repo1", "path/to/repo2"] opts.source_list.should eq ["path/to/repo1", "path/to/repo2"]
end end
end end
@ -72,7 +72,7 @@ describe CodePreloader::Config do
config.subcommand.should eq(Config::Subcommand::Pack) config.subcommand.should eq(Config::Subcommand::Pack)
config.pack_options.should be_truthy config.pack_options.should be_truthy
config.pack_options.try do |opts| config.pack_options.try do |opts|
opts.output_file_path.should eq "output.txt" opts.output_path.should eq "output.txt"
end end
end end
@ -88,9 +88,9 @@ describe CodePreloader::Config do
# Assuming the simple_config.yml has specific settings # Assuming the simple_config.yml has specific settings
config.pack_options.should be_truthy config.pack_options.should be_truthy
config.pack_options.try do |opts| config.pack_options.try do |opts|
opts.repository_path_list.should eq ["path/to/repo"] opts.source_list.should eq ["path/to/repo"]
opts.ignore_list.should eq ["simple/ignore"] opts.ignore_list.should eq ["simple/ignore"]
opts.output_file_path.should eq "simple_output.txt" opts.output_path.should eq "simple_output.txt"
end end
end end
@ -103,9 +103,9 @@ describe CodePreloader::Config do
# Assuming the complex_config.yml has specific settings # Assuming the complex_config.yml has specific settings
config.pack_options.should be_truthy config.pack_options.should be_truthy
config.pack_options.try do |opts| config.pack_options.try do |opts|
opts.repository_path_list.should eq [repo_path] opts.source_list.should eq [repo_path]
opts.ignore_list.should eq ["complex/ignore1", "complex/ignore2"] opts.ignore_list.should eq ["complex/ignore1", "complex/ignore2"]
opts.output_file_path.should eq "complex_output.txt" opts.output_path.should eq "complex_output.txt"
end end
end end

View file

@ -1,4 +1,5 @@
require "spec" require "spec"
# require "../src/" ENV["CODE_PRELOADER_DETECT"] = "no"

View file

@ -6,6 +6,7 @@ require "crinja"
require "./config" require "./config"
require "./filelist" require "./filelist"
require "./file_storage"
# The CodePreloader module organizes classes and methods related to preloading code files. # The CodePreloader module organizes classes and methods related to preloading code files.
module CodePreloader module CodePreloader
@ -40,35 +41,14 @@ module CodePreloader
abort("Unexpected nil value for init_options!") if init_options.nil? abort("Unexpected nil value for init_options!") if init_options.nil?
# Default path for the .code_preloader.yml file # Default path for the .code_preloader.yml file
default_config_path = "example.code_preloader.yml" default_config_path = ".code_preloader.yml"
# Use the specified path if provided, otherwise use the default # Use the specified path if provided, otherwise use the default
config_path = init_options.config_path || default_config_path config_path = init_options.config_path || default_config_path
abort("ERROR: configuration file already exist: #{config_path}") if File.exists? config_path
# Content of the .code_preloader.yml file # Content of the default .code_preloader.yml file
config_content = [ config_content = FileStorage.get("default_config.yml").gets_to_end
"---",
"# Example configuration for Code-Preloader",
"",
"# List of repository paths to preload",
"# source_list:",
"# - \"path/to/repo1\"",
"# - \"path/to/repo2\"",
"",
"# List of patterns to ignore during preloading",
"ignore_list:",
" - ^\\.git/.*",
"",
"# Path to the output file (if null, output to STDOUT)",
"output_path: null",
"",
"# Optional: Path to a file containing the header prompt",
"header_path: null",
"",
"# Optional: Path to a file containing the footer prompt",
"footer_path: null",
""
].join("\n")
# Writing the configuration content to the file # Writing the configuration content to the file
File.write(config_path, config_content) File.write(config_path, config_content)
@ -116,8 +96,12 @@ module CodePreloader
filelist.reject { |path| !!(path =~ Regex.new(ignore_pattern)) } filelist.reject { |path| !!(path =~ Regex.new(ignore_pattern)) }
end end
abort("No prompt file defined!") if prompt_template_path.nil? STDERR.puts "Loading template file from: #{prompt_template_path ? prompt_template_path : "<internal>" }".colorize(:yellow)
prompt_template_content = File.read(prompt_template_path) if prompt_template_path
prompt_template_content = File.read(prompt_template_path)
else
prompt_template_content = FileStorage.get("default_template.j2").gets_to_end
end
if !prompt_header_path.nil? if !prompt_header_path.nil?
@ -143,7 +127,7 @@ module CodePreloader
STDERR.puts "Processing source directories: #{source_list}".colorize(:yellow) STDERR.puts "Processing source directories: #{source_list}".colorize(:yellow)
processed_files = [] of ProcessedFile processed_files = [] of ProcessedFile
filelist.each do |file_path| filelist.to_a.sort.each do |file_path|
STDERR.puts "Processing file: #{file_path}".colorize(:yellow) STDERR.puts "Processing file: #{file_path}".colorize(:yellow)
file_result = process_file(file_path, output_file) file_result = process_file(file_path, output_file)
processed_files << file_result processed_files << file_result

View file

@ -86,8 +86,10 @@ module CodePreloader
def parse_pack_options(parser) def parse_pack_options(parser)
@pack_options = PackOptions.new @pack_options = PackOptions.new
config_file = detect_config_file unless ENV["CODE_PRELOADER_DETECT"]? =~ /(no|false|0)/i
config_file.try { |path| load_pack_config(path) } config_file = detect_config_file
config_file.try { |path| load_pack_config(path) }
end
parser.banner = [ parser.banner = [
"Usage: code-preloader pack [options] DIR ...\n", "Usage: code-preloader pack [options] DIR ...\n",
@ -99,7 +101,7 @@ module CodePreloader
parser.on( parser.on(
"-c FILE", "-c FILE",
"--config=FILE", "--config=FILE",
"Load parameters from FILE\n(default: \".code_preload.yml\", if present)" "Load parameters from FILE\n(default: autodetect)"
) do |config_file| ) do |config_file|
@pack_options.try { |opt| load_pack_config(config_file) } @pack_options.try { |opt| load_pack_config(config_file) }
end end

7
src/file_storage.cr Normal file
View file

@ -0,0 +1,7 @@
require "baked_file_system"
class FileStorage
extend BakedFileSystem
bake_folder "../static"
end

View file

@ -46,7 +46,7 @@ module CodePreloader
seen = Set(String).new seen = Set(String).new
# walk each source # walk each source
@sources.each do |dir| @sources.sort.each do |dir|
walker = Walk::Down.new(dir) walker = Walk::Down.new(dir)
walker = walker.filter do |path| walker = walker.filter do |path|
@ -88,7 +88,7 @@ module CodePreloader
self.each do |path| self.each do |path|
files << path.to_s files << path.to_s
end end
files files.sort
end end
end end
end end

24
static/default_config.yml Normal file
View file

@ -0,0 +1,24 @@
---
# Example configuration for Code-Preloader
# List of repository paths to preload
# source_list:
# - "path/to/repo1"
# - "path/to/repo2"
# List of patterns to ignore during preloading
ignore_list:
- ^\.git/.*
# Path to the output file (if null, output to STDOUT)
output_path: null
prompt:
# Optional: Path to a file containing the prompt header
header_path: null
# Optional: Path to a file containing the prompt footer
footer_path: null
# Optional: Path to a file container a jinja template to structure the prompt
template_path: null

View file

@ -0,0 +1,16 @@
{%- if prompt_header -%}
@@ CONTEXT
{{ prompt_header }}
{%- endif -%}
{%- for file in prompt_files -%}
@@ FILE "{{ file.path }}" WITH MIME-TYPE "{{ file.mime_type }}"
{{- file.content -}}
{%- endfor -%}
{%- if prompt_footer -%}
@@ REQUEST
{{ prompt_footer }}
{%- endif -%}