Three practical methods turn the SEC's "millions of filings annually" into usable datasets, according to the SEC (2024). Choose raw EDGAR HTML when you need full-fidelity narrative text and exhibits, the SEC XBRL endpoints when you want machine-readable line items, or a third-party normalized API when you need ready-made canonical fields. Developers, analysts, journalists and investment teams use each route for different tasks: audited annual figures come from 10-Ks and 10-Qs, material events from 8-Ks, and shareholder positions from 13F filings. The right choice depends on balancing setup time, flexibility and how much upstream filing semantics you must preserve.

The thesis is simple: pick the access method that matches what you need to extract, not the one that sounds fastest on a slide. EDGAR is the legal, primary repository for US public company disclosures and has been in operation since 1996. That archive handles "millions of filings annually," the SEC reports, and it exposes several concrete access points, from index files to XBRL feeds and a per-company submissions JSON.

Raw HTML: fidelity at the cost of plumbing

Raw EDGAR HTML gives you everything a filer submitted, including narrative sections such as Item 7 Management Discussion and Analysis and Item 1A Risk Factors, plus exhibit PDFs and any bespoke formatting. The common workflow is to query the per‑filer submissions JSON at data.sec.gov/submissions/CIK{CIK}.json to locate recent filings, extract the accession number and the primary document, then download and parse the filing HTML. Developer guides emphasise a few practical rules: always send a User-Agent header, and be prepared for accession folder names that drop hyphens.

That route is the only reliable path when your target is narrative text, exhibits or anything not tagged in XBRL. It also gives the strongest audit trail: what you analyse is precisely what the company filed. The trade-off is obvious. HTML scraping forces you to maintain parsers for table layouts, cope with filing-to-filing variation, and build logic to convert tables into model-ready series. For teams that place a high premium on traceability and the verbatim filing content, that engineering burden is worth it.

XBRL and third-party APIs: structured facts versus convenience

The SEC exposes machine-readable financial facts through its XBRL endpoints. XBRL is the right tool when you want numeric line items such as revenue, net income, balance sheet figures and cash flow entries without scraping tables. It removes the need to interpret layout, but it doesn't eliminate work. Tag inconsistency across filers is real, requiring mapping multiple tags to canonical concepts. There's also a year-to-date versus period-level conversion problem: you must decide whether to normalise YTD XBRL facts into period values for comparability. The SEC’s XBRL feeds and submissions history endpoints are authoritative, but they demand engineering effort to normalise tags and aggregate YTD values correctly.

Mastheads

For many teams that burden is the exact point of decision. Build XBRL parsing in-house and you retain control of semantics and the audit trail. Buy normalized outputs from a vendor and you trade control for speed. Third-party normalized financial APIs ingest filings, resolve tag inconsistencies, perform YTD conversions, and return canonical fields and item-level JSON ready for modelling.

They commonly offer extractor endpoints that classify filings into parts and items, and converters that turn XBRL into JSON-formatted financial statements.

The obvious counter-argument is that outsourcing normalisation can obscure upstream filing semantics. That's a valid concern. Relying on a vendor shifts work from your engineers to an external provider and can hide how a particular line was mapped, or how a YTD figure was transformed. But the practical defence of vendors is equally straightforward: if your immediate need is clean numeric series to feed models, and you lack the engineering bandwidth to normalise at scale, the time saved and the reduction in integration risk often justify the dependency.

Timing and workflow shape the choice too. EDGAR index files are published on a schedule, typically late evening Eastern Time, so a bulk index approach can miss filings submitted after the daily cutoff.

For historical retrieval that may be acceptable, and developer-focused projects emphasise three clear workflows: index-based historical retrieval, company-submissions lookup by CIK for faster per-filer access, and a current-filings stream for real-time detection. For event-driven use cases such as earnings announcements or Form 8-K disclosures, consumers often call the SEC’s latest-filings endpoints, use EDGAR full-text search, or subscribe to real-time feeds rather than relying on daily index files.

Decisions about speed, control and scope are also shaped by the forms you care about. Researchers seeking audited annual figures typically target 10-Ks and 10-Qs. Teams tracking material events focus on 8-Ks. Analysts mapping institutional holdings consult 13F filings. Each form carries different structure and cadence, and that affects whether scraping HTML, parsing XBRL, or calling a vendor API is the sensible default.

Related Articles

If you want per‑filer lookups, start with data.sec.gov/submissions/CIK{CIK}.json. That URL is the practical pivot between raw HTML, the SEC XBRL feeds and any vendor that sits on top of them.

This article was created with AI assistance.