Repository navigation
Conversation
tobocop2
force-pushed
the
feat/analyze-core
branch
from
September 27, 2026 15:33
8663544 to
f35e8db
Compare
tobocop2
changed the title
tobocop2
force-pushed
the
feat/analyze-core
branch
from
September 27, 2026 23:51
dd527c6 to
8565c4a
Compare
tobocop2
force-pushed
the
feat/analyze-core
branch
from
September 28, 2026 00:42
8565c4a to
9bb920a
Compare
Corpus analysis reads what a folder or the indexed corpus holds and recommends a profile. File types and the code share cover every file; a sample of at most analyze_max_files files (500 by default, evenly strided over the sorted keys) goes through xberg with OCR and layout off and language detection on, in batches of batch_extraction_size. Code, image and archive files are counted, not extracted. A failed read or extraction is kept with its reason. The signals are the ones xberg reports directly: scanned pages from the PDF metadata (each image file counts as one scanned page), tables, PDF page counts or character counts, and the first detected language of each file. The first built-in whose share reaches its threshold is picked; the thresholds are unmeasured constants. The most common language sets fts_language when a stemmer exists. With scans, the common text languages that Tesseract has become ocr_language. Each adjustment is compared against the built-in's own value, so a language the current profile set is restated in the new one. The recommendation is saved as "<built-in> (<project>)" with a description that marks it as analyze output. A later run replaces its own file and refuses a hand-made file with the same name. Applying it goes through the normal apply path, so your values and LILBEE_* variables stay. state.toml next to config.toml records when analyze last completed and whether the tip is hidden. It is never a Config field. Every profile apply hides the tip, and a failed state write only warns.
lilbee analyze [DIR] prints the report: files read, file types, code share, scans, tables, lengths, languages with their stemmer and Tesseract support, and the recommendation with what applying it changes. --apply saves the recommendation and switches to it, --save NAME saves it without switching, --target picks the folder, and --off only hides the tip. --json prints the report as one object, the shape the HTTP route, MCP tool, TUI and plugin will share. Ctrl+C sets a cancel token through the existing sigint helper; a cancelled run saves and records nothing. lilbee init and lilbee add print a one-line tip in text mode, before ingest starts, while the project is on Default, never analyzed, and has not hidden the tip.
analyze sampled every classified file and then dropped code, image and archive files from extraction, so files_read counted files that were never read and the cap was spent on them. The sample is now drawn from the documents only. The report carries documents_total and files_counted beside files_read, which counts the documents extracted without error. Image files are never read, so they weigh into the scanned share at the rate the documents were sampled. A target folder without apply or save is refused before any file is read, instead of being accepted and ignored. Detected language codes that Tesseract names differently (nob, cmn, pes) map to its names (nor, chi_sim, fas) for the support check and for ocr_language. The Config guard again refuses any analyze or profile field except analyze_max_files, and a test ties every built-in analyze can pick to the shipped built-in profiles.
The option help and the request docstring now state the rule that analyze enforces: a target folder takes effect only with apply or save, and without one the profile goes to the project folder when there is one, else the global one.
Corpus analysis sent xberg a disabled OCR block. xberg 1.2.7 OCRs every page image whenever an OCR block is present, enabled or not, so analyzing a scanned PDF paid a Tesseract pass per page and threw the text away. Analysis now sends no OCR block and keeps disable_ocr set, the same shape the page-count probe uses. On an 8-page scanned PDF, lilbee analyze wrote 8 OCR cache files before and writes none now, with the same scanned-page count and recommendation.
tobocop2
force-pushed
the
feat/analyze-core
branch
from
September 28, 2026 06:27
9bb920a to
771f8b1
Compare
Owner
Author
|
This change is now part of #1003, which collects every Profiles pull request on one branch. Review and merge happen there. The branch stays in place. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Picking a profile means knowing your documents: how many are scans, how much is code, which languages they are in. Nothing in lilbee looked, so people guessed or kept the default.
Solution
lilbee analyze [DIR]reads the files lilbee would index, the same setsyncuses with.lilbeeignoreand linked folders, and reports what it finds without running OCR or any model. It counts file types and code, and reads a sample through the normal extraction path with OCR off for scanned pages, tables, lengths and languages. It then recommends a built-in profile with any language changes.--applysaves the recommendation as " ()" and switches the project to it. Your own values andLILBEE_*values stay.--save NAMEsaves it without switching.--offstops the tip.--jsonprints the report.lilbee initandlilbee addprint a one-line tip before ingest starts on a project that has not been analyzed and still uses Default.Notes
analyze_max_files(default 500) caps how many documents are read; above it, an evenly spaced sample is used. Code, image and archive files are counted, not read. The JSON report givesfiles_total,documents_total,files_read,files_countedandcap. The cap and the thresholds that pick a built-in are not measured yet.This stacks on #958.