Cloud Cost API: Pulling Reconciled AWS, Azure and GCP Spend into Your Own Warehouse (CSV, JSON, Parquet)

Cloud Cost API: Pulling Reconciled AWS, Azure and GCP Spend into Your Own Warehouse (CSV, JSON, Parquet)

Tópico: Cloud Cost API: Pulling Reconciled AWS, Azure and GCP Spend into Your Own Warehouse (CSV, JSON, Parquet)
Categoria: Tutoriais | Programação & Tecnologia
Idioma Principal: Português (Conteúdo de Tecnologia)

Descrição do Conteúdo / Informações:
-------------------------------------------------------------------------
Sooner or later, cost data has to live where the rest of the company's data lives: in the warehouse, next to revenue and tenants, where analysts can join it instead of screenshotting dashboards. And the first thing every engineer discovers is that "the cloud cost API" barely exists. What exists is two different mechanisms per cloud, three incompatible schemas, three format menus, and one silent data-corruption trap that gets almost everyone once.

Here's the whole map: what to call, what to schedule, which formats each cloud actually ships, and the ingestion pattern that keeps your warehouse agreeing with the invoice.

The rule that organizes everything: APIs for questions, exports for warehouses

Every cloud offers both a query API (ask for aggregated numbers, get JSON back) and a bulk export (billing data lands on a schedule as files or tables). They are not interchangeable:

• Query APIs are rate-limited, aggregated, and sometimes metered. AWS's Cost Explorer API bills about $0.01 per request (as of early 2026). Azure's Cost Management query API throttles aggressively on large tenants. These are built for "what did service X cost last week", not for history at line-item grain.

• Exports are complete, line-item level, effectively free to produce (you pay only the storage), and designed for exactly this job.

If the destination is a warehouse, the answer is always the export. The query APIs are for spot checks, alert bots, and validating the pipeline you build on the export.

AWS: Data Exports (CUR 2.0) to S3, Parquet or CSV

The mechanism is Data Exports, the successor to the Cost and Usage Report. You define an export (there's a SQL-like column selection, so you can drop the 300 columns you'll never read), point it at an S3 bucket, and AWS delivers on a daily cadence at hourly, daily, or monthly granularity, with resource-level detail as an opt-in.

•
Formats: Parquet, or gzip-compressed CSV. Take Parquet for a warehouse: typed columns, a fraction of the size, and every engine reads it (Athena, Snowflake, BigQuery, Databricks, DuckDB).

•
The two cost bases live in different column families (unblended and amortized among them); decide which your downstream reports use and carry both if finance and engineering answer different questions.

•
The critical behavior: AWS rewrites the current month's files repeatedly as usage data arrives and rates settle, and can restate a month until the invoice finalizes a few days after close. The delivery is a fresh full snapshot of the billing period, not a diff.

Azure: Cost Management exports to a storage account

Azure's mechanism is Cost Management exports: scheduled delivery into a storage account container, with a choice of datasets: actual cost or amortized cost (the same distinction that decides whether reservation purchases show as spikes or as spread-out effective cost). The modern export experience also offers Parquet and the FOCUS dataset alongside the classic CSV; if you're starting now, start there.

•
The gotcha that corrupts pipelines: the daily file for the month-to-date export is a full month-to-date snapshot, not a daily increment. Day 14's file contains days 1-14. Load these append-only and every day of the month is counted up to thirty times. Each new file replaces everything prior for that billing period.

• Invoice-grade reconciliation for EA and MCA accounts comes from the same export family; the query API is not the reconciliation surface.

GCP: the export is a BigQuery table

GCP skips files entirely: the billing export lands directly in BigQuery (standard export, or the detailed export with resource-level records; enable it early because it is not retroactive). If BigQuery is your warehouse, you're done at this step.

If your warehouse is elsewhere, extract on a schedule: an EXPORT DATA statement writes the billing table out to Cloud Storage as Parquet, CSV, or Avro (JSON via a query if something downstream insists), and your loader picks it up like any other feed. Costs and credits are separate records in the export; forgetting to apply credit rows is the classic reason a GCP warehouse number runs high.

FOCUS: the schema that makes the three clouds one table

The three native schemas agree on almost nothing: column names, cost bases, even what a "service" is. The FinOps Foundation's FOCUS specification (FinOps Open Cost and Usage Specification) is the common schema designed to fix that, with normalized columns like BilledCost and EffectiveCost in place of each cloud's private vocabulary. AWS and Azure both ship native FOCUS-format exports; GCP publishes a FOCUS mapping for its BigQuery export.

If you're building a multi-cloud cost table in 2026, ingesting FOCUS where it's native and mapping GCP into it is the least-regret schema decision: one table, one set of column semantics, and every downstream report stops caring which cloud a row came from.

The restatement trap, and the ingestion pattern that survives it

All three clouds restate recent data. Usage records arrive up to a day or more late, commitment math settles, credits post, and the "final" month exists only once the invoice is issued, typically several days after month close. This has one hard implication:

Cost data must be ingested as replaceable snapshots, never appended.

The pattern that works everywhere:

• Land every delivery with two metadata columns: billing_period and snapshot_id (the delivery timestamp or manifest id).

• Load into a raw table without deleting anything (history of snapshots is free forensics).

• Serve every downstream query from a view that keeps only the newest snapshot per billing period:

SELECT *
FROM raw_billing
WHERE (billing_period, snapshot_id) IN (
SELECT billing_period, MAX(snapshot_id)
FROM raw_billing
GROUP BY billing_period
)

• Treat the trailing few days as provisional in reports (a data_through stamp on every dashboard saves a monthly argument).

• After each invoice arrives, run one reconciliation check: the warehouse's month total for that account should equal the invoice within your known delta (credits, taxes, and support handled per your reconciliation policy). Alert on drift; drift means a broken load, not a broken cloud.

Skip step 3 and the warehouse double-counts Azure month-to-date files and stale AWS snapshots. Skip step 5 and nobody notices until finance does.

Format guidance in one paragraph

Parquet for anything a warehouse or engine will read: typed, columnar, splittable, and routinely 10-20x smaller than the equivalent CSV. CSV for handoffs to humans and finance tools that want a file they can open. JSON only as query-API output for bots and spot checks; nobody should be warehousing billing JSON at line-item scale. And whatever the format, compress in flight and partition storage by billing period, because five years of hourly line items arrives faster than you think.

FAQ

Does AWS have a cost API?

Two things wear that name: the Cost Explorer API (aggregated queries, JSON, about $0.01 per request, rate-limited) and Data Exports/CUR (bulk line-item delivery to S3 in Parquet or CSV). For dashboards and bots, the API; for a warehouse, always the export.

How do I get Azure cost data programmatically?

The Cost Management query API works for spot queries but throttles hard at enterprise scale. For anything recurring, schedule a Cost Management export (actual or amortized dataset, CSV or Parquet, FOCUS available) to a storage account, and ingest the files, remembering that month-to-date files replace, never append.

How do I export GCP billing data as files?

Enable the BigQuery billing export first (it's the only complete source and isn't retroactive), then extract with EXPORT DATA to Cloud Storage in Parquet, CSV, or Avro on a schedule if your warehouse isn't BigQuery.

What is FOCUS and should I use it?

An open, cross-cloud billing schema from the FinOps Foundation that normalizes the three providers' incompatible column vocabularies into one (BilledCost, EffectiveCost, and friends). If you're starting a multi-cloud cost table now, ingest FOCUS-format exports where they're native; it removes the worst part of the job, which is schema reconciliation.

Why did numbers in my cost warehouse change after I loaded them?

Because the clouds restate: late usage, settling commitment math, posting credits. Deliveries are snapshots of a billing period, not immutable facts. Ingest with snapshot semantics (keep only the latest per period), treat trailing days as provisional, and reconcile each month against the issued invoice.

What's the best file format for cloud billing data?

Parquet, in almost every case: typed, columnar, and dramatically smaller, which matters at hourly line-item grain. Keep CSV for human handoffs. If a pipeline is moving billing data as JSON files, that's the sign it's built on the query API and should be rebuilt on the export.