Define a corpus by purpose and audience
Company documents may include manuals, policies, proposals, research notes, product specifications, presentations, and drafts. A useful corpus has a recognizable boundary. Start with a class of material created for a defined purpose, such as approved troubleshooting manuals, rather than exporting an entire shared drive.
Describe document counts, formats, languages, creation dates, and expected uses. Retrieval evaluation, question answering, and writing-style training require different structures. Ask whether the buyer needs full documents, passages with source links, or paired questions and reviewed answers.
Review authorship and embedded material
Identify employee authors, contractors, agencies, acquired businesses, and third-party contributions. Check assignments and licenses where needed. A document can contain stock images, research excerpts, licensed charts, customer quotes, and screenshots whose permissions differ from the surrounding prose.
Classify confidential material before sampling. Legal advice, board documents, employee records, private contracts, and customer-specific proposals deserve separate scrutiny. Do not assume a file is shareable because it was stored in a broadly accessible internal folder or because its title lacks a confidentiality label.
Keep versions and extraction quality visible
Repeated exports, tracked changes, comments, and draft variants can multiply counts without adding much distinct content. Group versions under a stable document family and mark which version is authoritative. Preserve superseded versions only when the historical change is relevant and authorized.
Inspect extracted text against the original layout. Tables, footnotes, diagrams, and scanned pages can lose meaning during conversion. Record OCR quality and missing visual content. If redaction removes a required definition, flag the resulting passage as incomplete instead of leaving it to appear self-contained.
- Separate final, draft, and superseded documents.
- Check hidden comments and speaker notes.
- Report scanned pages and extraction failures.
Offer a documented pilot corpus
Provide a small cleared set with a manifest, taxonomy, version rules, and transformation notes. Ask the buyer to specify whether it needs citations, section hierarchy, table structures, or permissions to reproduce source passages. These requirements affect both delivery work and license scope.
Define acceptance using actual document families and agreed extraction quality. Address future updates, removal requests, and the handling of derivative question-answer pairs. A document archive with clear boundaries and reliable provenance is easier to assess than a larger folder whose ownership and version history cannot be explained.
Common questions
Does converting PDFs to text clear rights issues?
No. Format conversion changes representation, not the underlying permissions or confidentiality obligations.
Should comments and revisions be included?
Only if needed for the intended task and separately reviewed. They can reveal sensitive reasoning, identities, or material removed from the final version.