Most enterprise libraries do not fail because someone deleted the wrong field. They fail because nobody ever asked which fields were doing work. A SharePoint document library accumulates columns the way a shared drive accumulates folders: one request at a time, each reasonable in isolation, none reviewed as a set. Ten years later the form has sixty fields, the search index has forty of them, and the retention schedule references three.
This is a diagnostic problem before it is a design problem. Before you remove a column, you need evidence about which columns are populated, which are queried, and which are actually used to make a retention or access decision. That evidence is exportable from the platforms you already run. The audit is not a governance project. It is a coverage report.
What a column is actually for
A metadata field earns its place when it does at least one of four jobs: it helps a person find a record, it helps a system route or retain a record, it satisfies an external reporting obligation, or it disambiguates two otherwise identical records. A field that does none of these is decoration, and decoration has a cost — it slows data entry, it drifts out of date, and it gives the illusion of control.
The distinction matters because the four jobs have different evidence trails. Findability is evidenced by query logs and crawl logs. Retention is evidenced by label coverage and hold inventories. Reporting is evidenced by the standard or regulation that requires the field. Disambiguation is evidenced by duplicate rates in the records themselves.
Start with the standards, not the form
Most sixty-column libraries contain a small core of fields that map to a published standard and a large tail that maps to nothing. Knowing which is which tells you where the burden of proof lies.
Dublin Core, maintained by the Dublin Core Metadata Initiative, defines a fifteen-term element set — title, creator, date, subject, description, and so on — plus a larger set of extension terms in the /terms/ namespace. The DCMI specification notes that the fifteen core terms were mirrored into /terms/ in 2008 and that the most useful properties and classes have been published as ISO 15836-2:2019. The practical point: if a field on your form corresponds to a Dublin Core term, you are not inventing a requirement, you are inheriting one. If it does not, you should be able to say who asked for it.
SKOS, the W3C Simple Knowledge Organization System, is the standard behind controlled vocabularies — the picklists and term stores that populate fields like Subject or Document Type. The SKOS Reference defines concepts, labels (skos:prefLabel, skos:altLabel, skos:hiddenLabel), notations, and semantic relations. When a taxonomy drifts — when the same concept appears under two labels, or when a deprecated term is still selectable — the drift is visible in the SKOS data if the vocabulary is maintained as data rather than as a hard-coded list in a form designer.
DCAT, the W3C Data Catalog Vocabulary, is the standard for describing datasets and data services in a catalog. DCAT 3, published as a W3C Recommendation in August 2024, defines classes and properties for catalogs, datasets, distributions, and data services, and explicitly allows profiles to add cardinality constraints, including a minimum set of required metadata fields. That is the mechanism by which a CKAN or Socrata portal decides which fields are mandatory. If your portal has sixty fields and DCAT requires a dozen, the other forty-eight are local policy, and local policy can be revised.
NARA’s General Records Schedules provide disposition authority for common federal records in the United States. NARA states that use of the GRS is mandatory for covered records and that agencies must use the GRS unless they can justify an agency-specific schedule. The GRS is organized by record type, not by column. That is the key insight for a field audit: retention is a property of the record series, not of an arbitrary metadata field. A column called Retention Code that nobody populates is not a retention control. It is a hope.
What the platforms will tell you
The audit artifacts differ by platform, but the logic is the same: find the report that shows field-level population, and find the log that shows field-level use.
SharePoint and Microsoft 365. SharePoint exposes column usage through site and library reports, and the search schema shows which managed properties are mapped to crawled columns. A managed property that is mapped but never appears in a query is a candidate for demotion. The practical export is a list of columns with their types, their managed-property mappings, and their population counts across the library. Columns with near-zero population and no managed-property mapping are the first candidates for deprecation.
Purview. Microsoft Purview records management uses retention labels to mark items as records, and the documentation describes a file plan for migrating or building retention requirements, retention labels with retention periods and actions, event-based retention, disposition review, and an export option for information about disposed items. The audit artifact here is a label coverage report: what percentage of items in a given library carry a retention label, and which labels are actually applied. A library with sixty columns and 4% label coverage is not managing records. It is managing a form.
Confluence. Confluence pages carry labels and space-level metadata. The export is a label coverage report by space, plus a list of page properties macros and their usage. A page properties macro that appears on three pages out of ten thousand is a field that did not earn its place.
Elasticsearch and Solr. Both expose field usage through query logs and index statistics. Elasticsearch’s field usage stats API and Solr’s field statistics in the admin UI show which fields are actually queried. A field that is indexed but never appears in a query is a candidate for removal from the index, even if it remains in the source system. This is the cleanest evidence available, because it is behavioral rather than declarative.
OpenText and other ECM platforms. The audit artifact is a metadata usage report from the administration console, cross-referenced with the retention policy applied to the folder or record series. Where the platform supports it, export the list of fields with their population counts and the list of retention policies with their scope.
CKAN and Socrata. Both are open data portals, and both are typically DCAT-aligned. CKAN exposes the dataset schema through its API, and Socrata exposes metadata through the SODA API. The audit artifact is a field coverage report across all published datasets: for each field in the portal schema, what fraction of datasets populate it. DCAT 3 explicitly permits profiles to define a minimum set of required fields, so a portal with sixty optional fields and no required fields is a portal that has not made a decision.
The audit procedure
The procedure is deliberately small. It should take days, not quarters, and it should produce an artifact you can attach to a change request.
- Export the field inventory. From the platform’s administration interface, export every column or field with its type, its source (standard, local policy, or inherited from a template), and its population count. In SharePoint this is a library settings export plus a content query. In CKAN or Socrata it is an API call across the catalog.
- Export the query log sample. From Elasticsearch, Solr, or the platform’s search analytics, export a sample of queries — a few thousand is enough — and extract the field names that appear in filters and facets. This tells you which fields people actually use to find things.
- Export the label coverage report. From Purview, Confluence, or the ECM platform, export the count of items carrying each retention label or content label, by library or space. This tells you which fields drive retention decisions.
- Export the permission export. From SharePoint, Purview, or the ECM platform, export the permission assignments by library or site. This tells you whether a field is being used to drive access — for example, a Department column that feeds a permission group. A field that drives permissions is load-bearing even if it is sparsely populated.
- Cross-reference against the standard. For each field, note whether it maps to a Dublin Core term, a SKOS concept scheme, a DCAT property, or a NARA GRS item. Fields that map to a standard have an external justification. Fields that do not need a local one.
- Classify. Sort fields into four buckets: load-bearing (high population, high query use, or drives retention or permissions), standard-mandated (required by a published standard or regulation), sparse but justified (low population but required for a specific reporting obligation), and decorative (low population, no query use, no standard, no permission or retention role).
- Propose the reversible change. For decorative fields, the proposal is not deletion. It is demotion: hide the field from the default form, remove it from the search index, and keep the data in place. This is reversible. If someone objects, you restore the field. If nobody objects in a quarter, you have evidence for the next step.
Why demotion beats deletion
Deletion is irreversible and it invites argument. Demotion is reversible and it invites evidence. Hiding a field from the default form reduces data-entry burden immediately. Removing it from the search index reduces index size and query noise. Keeping the data in place means no migration is required and no historical record is lost.
The trade-off is explicit: demotion leaves the column in the schema, so the schema remains larger than it needs to be. That is acceptable. The goal of the first pass is not a clean schema. It is a defensible one. A schema with sixty columns, forty of which are hidden and unindexed, is a schema with twenty working fields and forty documented decisions.
Governance by convention and why it fails
Most metadata governance in practice is governance by convention: a style guide says to fill in the Subject field, a wiki page says to use the controlled vocabulary, and a training deck says to apply the retention label. None of these are enforced, and none of them produce an artifact you can inspect.
The alternative is not heavy enforcement. It is lightweight evidence. A quarterly field coverage report, generated from the platform’s own exports, tells you whether the convention is holding. If label coverage drops from 60% to 40%, you know before the auditor does. If a field’s query use drops to zero, you know before the next schema review.
The artifact is the governance. A report that can be regenerated from an export is a control. A convention that lives in a wiki page is a wish.
What to do with the sixty-column library
Run the audit. Produce the coverage report. Classify the fields. Demote the decorative ones. Keep the export as the record of the decision.
The library will still have sixty columns. But twenty of them will be doing work, ten will be mandated by a standard you can name, and thirty will be hidden, unindexed, and documented as candidates for the next review. That is a library you can defend, and it took a week, not a program.
Frequently asked questions
How many fields should a document library have? There is no universal number. The useful question is how many fields are load-bearing. A library with sixty fields and twenty load-bearing fields is healthier than a library with fifteen fields and three load-bearing fields, because the first has documented its decisions and the second has not.
What if a field is required by a regulation but rarely populated? That is the sparse but justified bucket. Keep it, document the regulation, and report its population separately. Do not demote a field that satisfies an external obligation.
Can I remove a field from the search index without removing it from the form? Yes, in most platforms. In SharePoint this is a managed-property mapping decision. In Elasticsearch and Solr it is a mapping change. The field remains in the source record and can be re-indexed if needed.
What if the query log shows a field is used but the population count is near zero? That is a contradiction worth investigating. It usually means the field is populated in one library and not others, or that the query log includes queries from a different system. Resolve the contradiction before acting.
How often should the audit run? Quarterly is enough for most organizations. The exports are cheap to generate and the comparison across quarters is where the signal is.
Sources
- DCMI Metadata Terms, Dublin Core Metadata Initiative — https://www.dublincore.org/specifications/dublin-core/dcmi-terms/
- SKOS Simple Knowledge Organization System Reference, W3C Recommendation, 18 August 2009 — https://www.w3.org/TR/skos-reference/
- Data Catalog Vocabulary (DCAT) — Version 3, W3C Recommendation, 22 August 2024 — https://www.w3.org/TR/vocab-dcat-3/
- What are the General Records Schedules (GRS), National Archives and Records Administration — https://www.archives.gov/records-mgmt/grs
- Learn about records management, Microsoft Purview — https://learn.microsoft.com/en-us/purview/records-management
