docs: reorder README for data users, remove unused files (xdg-open, gera_schemas.py, open_gui.sh, docs/)

docs: expand /query curl usage, remove outdated UI references
replace duckdb-ui with ttyd shell: add /query HTTP endpoint, fix utf-8/locale, region config
2026-03-26 12:01:46 +01:00 · 2026-03-26 11:58:14 +01:00 · 2026-03-26 11:54:46 +01:00 · 2026-03-25 13:39:15 +01:00 · 2026-03-25 13:30:25 +01:00 · 2026-03-25 13:27:51 +01:00
15 changed files with 405 additions and 148832 deletions
--- a/.env.sample
+++ b/.env.sample
@@ -12,3 +12,6 @@ S3_CONCURRENCY=""
 PARALLEL_UPLOADS=""
 AWS_ACCESS_KEY_ID=""
 AWS_SECRET_ACCESS_KEY=""
 # Caddy serving duckdb
 BASIC_AUTH_PASSWORD=""
--- a/.gitignore
+++ b/.gitignore
@@ -1,6 +1,7 @@
 .env
 .DS_Store
 logs/
 *.duckdb
 *.duckdb.wal
 done_tables.txt
 done_transfers.txt
 # CocoIndex Code (ccc)
 /.cocoindex_code/
--- a/24
+++ b/24
@@ -0,0 +1,24 @@
 :8080 {
    handle /health {
        respond 200
    }
    handle /login {
        reverse_proxy 127.0.0.1:8081
    }
    handle /query {
        reverse_proxy 127.0.0.1:8081
    }
    handle {
        forward_auth 127.0.0.1:8081 {
            uri /auth
            copy_headers Cookie
        }
        reverse_proxy localhost:7681 {
            flush_interval -1
        }
    }
 }
--- a/33
+++ b/33
@@ -0,0 +1,33 @@
 FROM debian:12-slim
 RUN apt-get update -qq && \
    apt-get install -y --no-install-recommends \
        curl ca-certificates unzip bsdmainutils python3 \
        less ncurses-bin && \
    curl -fsSL \
        "https://github.com/caddyserver/caddy/releases/download/v2.9.1/caddy_2.9.1_linux_amd64.tar.gz" \
        | tar -xz -C /usr/local/bin caddy && \
    chmod +x /usr/local/bin/caddy && \
    curl -fsSL \
        "https://github.com/duckdb/duckdb/releases/download/v1.5.1/duckdb_cli-linux-amd64.zip" \
        -o /tmp/duckdb.zip && \
    unzip /tmp/duckdb.zip -d /usr/local/bin && \
    chmod +x /usr/local/bin/duckdb && \
    rm /tmp/duckdb.zip && \
    apt-get clean && rm -rf /var/lib/apt/lists/* && \
    duckdb :memory: "INSTALL httpfs;" && \
    curl -fsSL "https://github.com/tsl0922/ttyd/releases/latest/download/ttyd.x86_64" \
        -o /usr/local/bin/ttyd && \
    chmod +x /usr/local/bin/ttyd
 ENV LANG=C.UTF-8 \
    LC_ALL=C.UTF-8
 WORKDIR /app
 COPY basedosdados.duckdb Caddyfile start.sh auth.py ./
 RUN chmod +x start.sh
 EXPOSE 8080
 ENTRYPOINT ["./start.sh"]
--- a/README.md
+++ b/README.md
@@ -1,80 +1,126 @@
 # baseldosdados
-Mirror completo das tabelas públicas do projeto [Base dos Dados](https://basedosdados.org/) — 533 tabelas, ~675 GB em Parquet+zstd — hospedado no Hetzner Object Storage e acessível via DuckDB.
+Mirror completo das tabelas públicas do projeto [Base dos Dados](https://basedosdados.org/) — 533 tabelas, ~675 GB em Parquet+zstd.
-## O que tem aqui
+Os dados foram exportados do BigQuery para o Hetzner Object Storage (Helsinki) no formato Parquet com compressão zstd, organizados por dataset e tabela. O acesso é feito diretamente sobre os arquivos via DuckDB, sem necessidade de importar nada localmente — as queries leem os parquets do S3 sob demanda.
-| Script | Função |
+---
 |---|---|
 | `roda.sh` | Exporta BigQuery → GCS → Hetzner S3 (pipeline principal) |
 | `prepara_gui.py` | Cria views DuckDB sobre os parquets do S3 para exploração local |
 | `gera_schemas.py` | Gera `schemas.json` e `file_tree.md` com metadados de todos os parquets |
-**Arquivos gerados:**
+## Consultando os dados
-| Arquivo | Descrição |
+Acesso via browser ou curl, protegido por senha. Peça a senha para o administrador.
 |---|---|
 | `schemas.json` | Schema completo de todas as 533 tabelas (colunas, tipos, tamanhos) |
 | `file_tree.md` | Árvore do bucket S3 com tamanhos e contagem de arquivos |
 | `basedosdados.duckdb` | Banco DuckDB com views para todas as tabelas (gerado por `prepara_gui.py`) |
 | `all_tables.txt` | Lista completa de tabelas descobertas |
 | `done_tables.txt` | Tabelas exportadas com sucesso para o GCS |
 | `done_transfers.txt` | Datasets transferidos com sucesso para o S3 |
 | `failed_tables.txt` | Tabelas que falharam após 3 tentativas |
-## Fluxo de exportação
+### Shell no browser
 Acesse **https://db.xn--2dk.xyz** → autentique → shell DuckDB interativo direto no browser.
 ### SQL via curl
 Endpoint `POST /query` — SQL no body, resultado como texto plano:
 ```bash
 # Query inline
 curl -s -X POST https://db.xn--2dk.xyz/query \
  -H "X-Password: <senha>" \
  --data-binary "SELECT count(*) FROM br_anatel_banda_larga_fixa.densidade_brasil"
 # A partir de um arquivo .sql
 curl -s -X POST https://db.xn--2dk.xyz/query \
  -H "X-Password: <senha>" \
  --data-binary @minha_query.sql
 # Heredoc (útil em scripts)
 curl -s -X POST https://db.xn--2dk.xyz/query \
  -H "X-Password: <senha>" \
  --data-binary @- << 'SQL'
 SELECT sigla_uf, sum(densidade) AS total
 FROM br_anatel_banda_larga_fixa.densidade_uf
 WHERE ano = 2023
 GROUP BY 1
 ORDER BY 2 DESC
 SQL
 # Salvar resultado em arquivo
 curl -s -X POST https://db.xn--2dk.xyz/query \
  -H "X-Password: <senha>" \
  --data-binary @query.sql > resultado.csv
 ```
 ### Descobrindo tabelas
 ```sql
 -- listar todos os datasets (schemas)
 SHOW SCHEMAS;
 -- listar tabelas de um dataset
 SHOW TABLES IN br_anatel_banda_larga_fixa;
 -- ver colunas de uma tabela
 DESCRIBE br_anatel_banda_larga_fixa.densidade_brasil;
 ```
 No shell do browser, `.tables` lista tudo de uma vez.
 ### Exportar em CSV ou JSON
 O DuckDB permite formatar a saída diretamente na query:
 ```sql
 -- CSV com header (pipe para arquivo via curl)
 COPY (SELECT * FROM br_ibge_censo2022.municipios LIMIT 1000)
 TO '/dev/stdout' (FORMAT csv, HEADER true);
 -- JSON
 SELECT * FROM br_ibge_censo2022.municipios LIMIT 10
 FORMAT JSON;
 ```
 ---
 ## Exploração local
 Para rodar as queries na sua própria máquina com DuckDB instalado:
 ```bash
 python prepara_db.py   # gera basedosdados.duckdb com views apontando para o S3
 duckdb basedosdados.duckdb
 ```
 Requer as credenciais do S3 no `.env` (veja seção de configuração abaixo).
 ---
 ## Pipeline de exportação
 > Seção para mantenedores — não necessário para consulta dos dados.
 ### Fluxo
 ```
 BigQuery (basedosdados) → GCS (Parquet + zstd) → Hetzner Object Storage (rclone)
 ```
 1. Descobre automaticamente todos os datasets e tabelas via API do BigQuery
-2. Exporta todas as tabelas em paralelo no formato Parquet com compressão zstd
+2. Exporta em paralelo no formato Parquet com compressão zstd
 3. Transfere GCS → Hetzner Object Storage via rclone (streaming direto, sem disco local)
-4. Verifica a contagem de arquivos entre GCS e S3
+4. Verifica contagem de arquivos entre GCS e S3
 5. Oferece opção de deletar o bucket GCS ao final
-O script suporta **resume automático**: se interrompido, basta rodar novamente — tabelas e transfers já concluídos são pulados.
+Resume automático: se interrompido, basta rodar novamente.
-## Estrutura dos dados no S3
+### Scripts
-```
+| Script | Função |
-s3://<HETZNER_S3_BUCKET>/
+|---|---|
-└── <dataset>/
+| `roda.sh` | Pipeline principal de exportação |
-    └── <tabela>/
+| `prepara_db.py` | Gera `basedosdados.duckdb` com views para todas as tabelas |
        └── *.parquet
 ```
-## Pré-requisitos
+### Configuração (`.env`)
 **Exportação (`roda.sh`) — execução local:**
 - `google-cloud-sdk` (`bq`, `gcloud`, `gsutil`)
 - `parallel` (GNU parallel)
 - `rclone`
 - `flock`
 **Execução via VM (`--gcloud-run`):** apenas `gcloud` localmente — dependências instaladas automaticamente na VM.
 **Scripts Python** (`prepara_gui.py`, `gera_schemas.py`):
 - `duckdb`, `pyarrow`, `boto3`, `s3fs`, `python-dotenv`
 Autenticação GCP (uma vez antes da exportação):
 ```bash
 gcloud auth login
 gcloud auth application-default login
 gcloud config set project SEU_PROJECT_ID
 ```
 ## Configuração
 Crie um arquivo `.env` na raiz:
 | Variável | Descrição |
 |---|---|
-| `YOUR_PROJECT` | ID do seu projeto GCP (para faturamento) |
+| `YOUR_PROJECT` | ID do projeto GCP (para faturamento) |
 | `BUCKET_NAME` | Nome do bucket GCS intermediário |
-| `BUCKET_REGION` | Região do bucket — deve ser `US` |
+| `BUCKET_REGION` | Região do bucket S3 (ex: `eu-central`) |
 | `SOURCE_PROJECT` | Projeto fonte (`basedosdados`) |
 | `PARALLEL_EXPORTS` | Jobs paralelos de exportação BigQuery (padrão: 8) |
 | `HETZNER_S3_BUCKET` | Nome do bucket no Hetzner Object Storage |
@@ -83,29 +129,36 @@ Crie um arquivo `.env` na raiz:
 | `PARALLEL_UPLOADS` | Datasets enviados em paralelo (padrão: 4) |
 | `AWS_ACCESS_KEY_ID` | Access key do Hetzner Object Storage |
 | `AWS_SECRET_ACCESS_KEY` | Secret key do Hetzner Object Storage |
 | `BASIC_AUTH_PASSWORD` | Senha do shell web e endpoint `/query` |
-## Uso
+### Executando
 ```bash
 # Exportação
 chmod +x roda.sh
-./roda.sh --dry-run    # estima tamanho e custo antes de rodar
+./roda.sh --dry-run    # estima tamanho e custo
 ./roda.sh              # execução local
-./roda.sh --gcloud-run # cria VM no GCP, roda lá e deleta a VM ao final
+./roda.sh --gcloud-run # cria VM no GCP, roda lá e deleta ao final
 # Exploração via DuckDB
 python prepara_gui.py  # cria basedosdados.duckdb com views para todas as tabelas
 duckdb --ui basedosdados.duckdb
 # Dump de schemas
 python gera_schemas.py  # gera schemas.json e file_tree.md (~21 MB de egress)
 ```
-### `--gcloud-run`
+Autenticação GCP necessária antes da primeira exportação:
-Cria uma VM `e2-standard-4` Debian 12 em `us-central1-a`, copia o script e o `.env`, instala as dependências e executa via SSH. Variáveis opcionais:
+```bash
 gcloud auth login
 gcloud auth application-default login
 gcloud config set project SEU_PROJECT_ID
 ```
 #### `--gcloud-run`
 Cria uma VM `e2-standard-4` Debian 12 em `us-central1-a`, copia o script e o `.env`, instala dependências e executa via SSH.
 | Variável | Padrão | Descrição |
 |---|---|---|
 | `GCP_VM_NAME` | `bd-export-vm` | Nome da instância |
 | `GCP_VM_ZONE` | `us-central1-a` | Zona do Compute Engine |
 ### Deploy do servidor
 ```bash
 haloy deploy
 ```
--- a/auth.py
+++ b/auth.py
@@ -0,0 +1,96 @@
 #!/usr/bin/env python3
 """Minimal cookie-session auth gate for DuckDB shell."""
 import hmac, hashlib, os, secrets, subprocess, time
 from http.server import HTTPServer, BaseHTTPRequestHandler
 from urllib.parse import parse_qs
 PASSWORD = os.environ.get('BASIC_AUTH_PASSWORD', '').encode()
 _SECRET  = secrets.token_bytes(32)
 def _make_token():
    day = str(int(time.time()) // 86400)
    return hmac.new(_SECRET, day.encode(), hashlib.sha256).hexdigest()
 def _valid(token):
    if not token:
        return False
    for delta in (0, 1):
        day = str(int(time.time()) // 86400 - delta)
        expected = hmac.new(_SECRET, day.encode(), hashlib.sha256).hexdigest()
        if hmac.compare_digest(token, expected):
            return True
    return False
 LOGIN_HTML = """<!DOCTYPE html>
 <html><head><title>DB Shell</title><style>
 body{display:flex;justify-content:center;align-items:center;min-height:100vh;margin:0;background:#0f1117;font-family:sans-serif}
 form{background:#1a1d27;padding:2rem;border-radius:8px;display:flex;flex-direction:column;gap:1rem;min-width:280px}
 h2{color:#fff;margin:0}
 input{padding:.6rem;border-radius:4px;border:1px solid #333;background:#0f1117;color:#fff;font-size:1rem}
 button{padding:.6rem;border-radius:4px;border:none;background:#f4c543;color:#000;font-size:1rem;cursor:pointer;font-weight:600}
 </style></head>
 <body><form method="POST" action="/login">
  <h2>DB Shell</h2>
  <input type="password" name="password" placeholder="Password" autofocus>
  <button type="submit">Enter</button>
 </form></body></html>""".encode()
 class H(BaseHTTPRequestHandler):
    def _cookie(self):
        for part in self.headers.get('Cookie', '').split(';'):
            part = part.strip()
            if part.startswith('ddb_auth='):
                return part[9:]
        return ''
    def do_GET(self):
        if self.path == '/auth':
            if _valid(self._cookie()):
                self._resp(200)
            else:
                self.send_response(302)
                self.send_header('Location', '/login')
                self.end_headers()
        else:
            self._resp(200, LOGIN_HTML, 'text/html; charset=utf-8')
    def do_POST(self):
        if self.path == '/query':
            pwd = self.headers.get('X-Password', '').encode()
            if not hmac.compare_digest(pwd, PASSWORD):
                self._resp(401, b'Unauthorized\n')
                return
            sql = self.rfile.read(int(self.headers.get('Content-Length', 0))).decode(errors='replace')
            try:
                r = subprocess.run(
                    ['duckdb', '-readonly', '--init', '/app/ssh_init.sql', '/app/basedosdados.duckdb'],
                    input=sql, capture_output=True, text=True, timeout=120
                )
                out = (r.stdout + r.stderr).encode()
            except subprocess.TimeoutExpired:
                out = b'timeout\n'
            self._resp(200, out, 'text/plain; charset=utf-8')
            return
        body = self.rfile.read(int(self.headers.get('Content-Length', 0))).decode(errors='replace')
        pwd  = parse_qs(body).get('password', [''])[0].encode()
        if hmac.compare_digest(pwd, PASSWORD):
            self.send_response(302)
            self.send_header('Set-Cookie', f'ddb_auth={_make_token()}; Path=/; HttpOnly; SameSite=Strict')
            self.send_header('Location', '/')
            self.end_headers()
        else:
            self._resp(200, LOGIN_HTML, 'text/html; charset=utf-8')
    def _resp(self, code, body=b'', ct='text/plain'):
        self.send_response(code)
        if body:
            self.send_header('Content-Type', ct)
            self.send_header('Content-Length', str(len(body)))
        self.end_headers()
        if body:
            self.wfile.write(body)
    def log_message(self, *_):
        pass
 HTTPServer(('127.0.0.1', 8081), H).serve_forever()
--- a/basedosdados.duckdb
+++ b/basedosdados.duckdb
--- a/file_tree.md
+++ b/file_tree.md
@@ -1,870 +0,0 @@
 # S3 File Tree: baseldosdados
 ## br_anatel_banda_larga_fixa/  (4 tables, 206.1 MB, 119 files)
  - **densidade_brasil/**  (1 files, 2.9 KB, 3 cols)
  - **densidade_municipio/**  (2 files, 10.3 MB, 5 cols)
  - **densidade_uf/**  (1 files, 50.8 KB, 4 cols)
  - **microdados/**  (115 files, 195.8 MB, 12 cols)
 ## br_anatel_indice_brasileiro_conectividade/  (1 tables, 443.4 KB, 1 files)
  - **municipio/**  (1 files, 443.4 KB, 11 cols)
 ## br_anp_precos_combustiveis/  (1 tables, 79.0 MB, 69 files)
  - **microdados/**  (69 files, 79.0 MB, 14 cols)
 ## br_ans_beneficiario/  (1 tables, 8.3 GB, 3573 files)
  - **informacao_consolidada/**  (3573 files, 8.3 GB, 22 cols)
 ## br_bcb_estban/  (3 tables, 2.3 GB, 2207 files)
  - **agencia/**  (1524 files, 1.4 GB, 9 cols)
  - **dicionario/**  (1 files, 2.9 KB, 5 cols)
  - **municipio/**  (682 files, 894.3 MB, 10 cols)
 ## br_bcb_sicor/  (11 tables, 19.6 GB, 2056 files)
  - **dicionario/**  (1 files, 9.3 KB, 5 cols)
  - **empreendimento/**  (1 files, 51.6 KB, 15 cols)
  - **liberacao/**  (65 files, 155.7 MB, 8 cols)
  - **operacao/**  (127 files, 521.6 MB, 53 cols)
  - **operacoes_desclassificadas/**  (1 files, 160.8 KB, 8 cols)
  - **recurso_publico_complemento_operacao/**  (44 files, 244.7 MB, 8 cols)
  - **recurso_publico_cooperado/**  (1 files, 1.2 MB, 9 cols)
  - **recurso_publico_gleba/**  (61 files, 3.8 GB, 7 cols)
  - **recurso_publico_mutuario/**  (29 files, 297.1 MB, 10 cols)
  - **recurso_publico_propriedade/**  (15 files, 491.1 MB, 9 cols)
  - **saldo/**  (1711 files, 14.2 GB, 10 cols)
 ## br_bd_diretorios_brasil/  (23 tables, 141.2 MB, 357 files)
  - **area_conhecimento/**  (1 files, 21.1 KB, 8 cols)
  - **cbo_1994/**  (1 files, 41.3 KB, 2 cols)
  - **cbo_2002/**  (1 files, 74.3 KB, 11 cols)
  - **cep/**  (1 files, 119.0 MB, 8 cols)
  - **cid_10/**  (1 files, 260.7 KB, 9 cols)
  - **cid_9/**  (1 files, 16.8 KB, 2 cols)
  - **cnae_1/**  (1 files, 23.4 KB, 8 cols)
  - **cnae_2/**  (1 files, 57.8 KB, 14 cols)
  - **curso_superior/**  (1 files, 6.2 KB, 5 cols)
  - **distrito_1991/**  (1 files, 126.6 KB, 4 cols)
  - **distrito_2000/**  (1 files, 145.4 KB, 4 cols)
  - **distrito_2010/**  (1 files, 150.1 KB, 4 cols)
  - **empresa/**  (335 files, 1.5 MB, 32 cols)
  - **escola/**  (1 files, 12.8 MB, 19 cols)
  - **etnia_indigena/**  (1 files, 5.4 KB, 2 cols)
  - **instituicao_ensino_superior/**  (1 files, 118.3 KB, 7 cols)
  - **municipio/**  (1 files, 328.0 KB, 27 cols)
  - **natureza_juridica/**  (1 files, 16.5 KB, 3 cols)
  - **regiao/**  (1 files, 637.0 B, 2 cols)
  - **setor_censitario_2010/**  (1 files, 1.6 MB, 14 cols)
  - **setor_censitario_2022/**  (1 files, 4.9 MB, 22 cols)
  - **subatividade_ibge/**  (1 files, 7.1 KB, 2 cols)
  - **uf/**  (1 files, 1.6 KB, 4 cols)
 ## br_bd_diretorios_mundo/  (4 tables, 1.4 MB, 4 files)
  - **continente/**  (1 files, 1.0 KB, 3 cols)
  - **nomenclatura_comum_mercosul/**  (1 files, 790.3 KB, 14 cols)
  - **pais/**  (1 files, 19.1 KB, 13 cols)
  - **sistema_harmonizado/**  (1 files, 655.9 KB, 16 cols)
 ## br_bd_metadados/  (2 tables, 14.7 MB, 2 files)
  - **bigquery_tables/**  (1 files, 68.6 KB, 10 cols)
  - **prefect_flow_runs/**  (1 files, 14.6 MB, 16 cols)
 ## br_camara_dados_abertos/  (28 tables, 267.7 MB, 222 files)
  - **deputado/**  (1 files, 277.8 KB, 12 cols)
  - **deputado_ocupacao/**  (1 files, 587.2 KB, 6 cols)
  - **deputado_profissao/**  (1 files, 58.0 KB, 5 cols)
  - **despesa/**  (22 files, 125.4 MB, 25 cols)
  - **evento/**  (1 files, 7.2 MB, 11 cols)
  - **evento_orgao/**  (1 files, 322.0 KB, 3 cols)
  - **evento_presenca_deputado/**  (5 files, 6.0 MB, 4 cols)
  - **evento_requerimento/**  (1 files, 287.1 KB, 3 cols)
  - **frente/**  (1 files, 183.9 KB, 10 cols)
  - **frente_deputado/**  (1 files, 767.1 KB, 5 cols)
  - **funcionario/**  (1 files, 316.7 KB, 10 cols)
  - **legislatura/**  (1 files, 2.8 KB, 5 cols)
  - **legislatura_mesa/**  (1 files, 7.9 KB, 13 cols)
  - **licitacao/**  (1 files, 327.5 KB, 18 cols)
  - **licitacao_contrato/**  (1 files, 198.6 KB, 19 cols)
  - **licitacao_item/**  (1 files, 6.9 MB, 21 cols)
  - **licitacao_pedido/**  (1 files, 597.0 KB, 11 cols)
  - **licitacao_proposta/**  (1 files, 738.5 KB, 13 cols)
  - **orgao/**  (1 files, 176.6 KB, 11 cols)
  - **orgao_deputado/**  (1 files, 511.2 KB, 9 cols)
  - **proposicao_autor/**  (5 files, 8.9 MB, 8 cols)
  - **proposicao_microdados/**  (81 files, 72.5 MB, 25 cols)
  - **proposicao_tema/**  (80 files, 2.1 MB, 6 cols)
  - **votacao/**  (1 files, 10.4 MB, 17 cols)
  - **votacao_objeto/**  (3 files, 8.7 MB, 10 cols)
  - **votacao_orientacao_bancada/**  (1 files, 335.7 KB, 5 cols)
  - **votacao_parlamentar/**  (5 files, 10.8 MB, 9 cols)
  - **votacao_proposicao/**  (1 files, 3.1 MB, 10 cols)
 ## br_ce_fortaleza_sefin_iptu/  (1 tables, 1.5 MB, 1 files)
  - **face_quadra/**  (1 files, 1.5 MB, 13 cols)
 ## br_cgu_beneficios_cidadao/  (6 tables, 61.4 GB, 6751 files)
  - **auxilio_brasil/**  (643 files, 3.0 GB, 10 cols)
  - **auxilio_emergencial/**  (1426 files, 5.9 GB, 14 cols)
  - **bolsa_familia_pagamento/**  (1479 files, 25.8 GB, 10 cols)
  - **bpc/**  (1667 files, 15.9 GB, 15 cols)
  - **garantia_safra/**  (92 files, 443.9 MB, 7 cols)
  - **novo_bolsa_familia/**  (1444 files, 10.4 GB, 10 cols)
 ## br_cgu_cartao_pagamento/  (4 tables, 40.2 MB, 30 files)
  - **dicionario/**  (1 files, 2.9 KB, 5 cols)
  - **microdados_compras_centralizadas/**  (12 files, 7.5 MB, 14 cols)
  - **microdados_defesa_civil/**  (1 files, 1.3 MB, 20 cols)
  - **microdados_governo_federal/**  (16 files, 31.5 MB, 15 cols)
 ## br_cgu_dados_abertos/  (3 tables, 6.9 MB, 3 files)
  - **conjunto/**  (1 files, 1.4 MB, 13 cols)
  - **organizacao/**  (1 files, 119.7 KB, 9 cols)
  - **recurso/**  (1 files, 5.3 MB, 11 cols)
 ## br_cgu_emendas_parlamentares/  (1 tables, 2.2 MB, 1 files)
  - **microdados/**  (1 files, 2.2 MB, 25 cols)
 ## br_cgu_licitacao_contrato/  (8 tables, 2.6 GB, 553 files)
  - **contrato_apostilamento/**  (1 files, 532.2 KB, 14 cols)
  - **contrato_compra/**  (1 files, 35.5 MB, 26 cols)
  - **contrato_item/**  (15 files, 65.9 MB, 12 cols)
  - **contrato_termo_aditivo/**  (1 files, 17.2 MB, 12 cols)
  - **licitacao/**  (22 files, 104.1 MB, 19 cols)
  - **licitacao_empenho/**  (39 files, 418.5 MB, 12 cols)
  - **licitacao_item/**  (66 files, 392.4 MB, 16 cols)
  - **licitacao_participante/**  (408 files, 1.6 GB, 15 cols)
 ## br_cgu_orcamento_publico/  (1 tables, 8.7 MB, 1 files)
  - **orcamento/**  (1 files, 8.7 MB, 26 cols)
 ## br_cgu_receitas_publicas/  (1 tables, 15.0 MB, 16 files)
  - **receitas/**  (16 files, 15.0 MB, 16 cols)
 ## br_cgu_servidores_executivo_federal/  (7 tables, 14.8 GB, 2993 files)
  - **afastamentos/**  (27 files, 72.7 MB, 8 cols)
  - **cadastro_aposentados/**  (133 files, 781.5 MB, 30 cols)
  - **cadastro_pensionistas/**  (167 files, 1.1 GB, 34 cols)
  - **cadastro_reserva_reforma_militares/**  (51 files, 223.1 MB, 29 cols)
  - **cadastro_servidores/**  (1413 files, 6.1 GB, 46 cols)
  - **observacoes/**  (73 files, 421.9 MB, 7 cols)
  - **remuneracao/**  (1129 files, 6.0 GB, 40 cols)
 ## br_cnj_improbidade_administrativa/  (1 tables, 2.6 MB, 1 files)
  - **condenacao/**  (1 files, 2.6 MB, 63 cols)
 ## br_cnpq_bolsas/  (2 tables, 86.6 MB, 21 files)
  - **dicionario/**  (1 files, 3.6 KB, 5 cols)
  - **microdados/**  (20 files, 86.6 MB, 31 cols)
 ## br_cvm_administradores_carteira/  (3 tables, 416.5 KB, 7 files)
  - **pessoa_fisica/**  (3 files, 120.3 KB, 7 cols)
  - **pessoa_juridica/**  (3 files, 238.6 KB, 24 cols)
  - **responsavel/**  (1 files, 57.6 KB, 3 cols)
 ## br_cvm_oferta_publica_distribuicao/  (1 tables, 1.2 MB, 1 files)
  - **dia/**  (1 files, 1.2 MB, 44 cols)
 ## br_datahackers_state_data/  (1 tables, 445.7 KB, 1 files)
  - **microdados/**  (1 files, 445.7 KB, 353 cols)
 ## br_fbsp_absp/  (2 tables, 41.3 KB, 2 files)
  - **uf/**  (1 files, 32.6 KB, 29 cols)
  - **violencia_escola/**  (1 files, 8.8 KB, 5 cols)
 ## br_fgv_igp/  (7 tables, 111.1 KB, 7 files)
  - **igp_10_mes/**  (1 files, 17.4 KB, 7 cols)
  - **igp_di_ano/**  (1 files, 4.2 KB, 5 cols)
  - **igp_di_mes/**  (1 files, 38.7 KB, 7 cols)
  - **igp_m_ano/**  (1 files, 2.9 KB, 5 cols)
  - **igp_m_mes/**  (1 files, 22.8 KB, 9 cols)
  - **igp_og_ano/**  (1 files, 3.3 KB, 5 cols)
  - **igp_og_mes/**  (1 files, 21.8 KB, 7 cols)
 ## br_geobr_mapas/  (25 tables, 245.7 MB, 26 files)
  - **amazonia_legal/**  (1 files, 212.8 KB, 1 cols)
  - **area_minima_comparavel_2010/**  (1 files, 12.5 MB, 3 cols)
  - **area_risco_desastre/**  (1 files, 1.7 MB, 8 cols)
  - **arranjo_populacional/**  (1 files, 2.1 MB, 8 cols)
  - **bioma/**  (2 files, 15.4 MB, 4 cols)
  - **concentracao_urbana/**  (1 files, 1.5 MB, 8 cols)
  - **escola/**  (1 files, 3.1 MB, 3 cols)
  - **estabelecimentos_saude/**  (1 files, 4.2 MB, 5 cols)
  - **limite_vizinhanca/**  (1 files, 5.1 MB, 12 cols)
  - **mesorregiao/**  (1 files, 3.4 MB, 4 cols)
  - **microrregiao/**  (1 files, 6.7 MB, 4 cols)
  - **municipio/**  (1 files, 17.2 MB, 3 cols)
  - **pais/**  (1 files, 455.7 KB, 1 cols)
  - **pegada_urbana/**  (1 files, 6.3 MB, 6 cols)
  - **regiao/**  (1 files, 884.8 KB, 3 cols)
  - **regiao_imediata/**  (1 files, 6.1 MB, 4 cols)
  - **regiao_intermediaria/**  (1 files, 3.9 MB, 4 cols)
  - **regiao_metropolitana_2017/**  (1 files, 2.8 MB, 8 cols)
  - **saude/**  (1 files, 2.1 MB, 4 cols)
  - **sede_municipal/**  (1 files, 384.0 KB, 8 cols)
  - **semiarido/**  (1 files, 2.0 MB, 3 cols)
  - **setor_censitario_2010/**  (1 files, 139.3 MB, 13 cols)
  - **terra_indigena/**  (1 files, 3.1 MB, 15 cols)
  - **uf/**  (1 files, 1.4 MB, 3 cols)
  - **unidade_conservacao/**  (1 files, 3.8 MB, 14 cols)
 ## br_ibge_censo_2022/  (16 tables, 4.1 GB, 512 files)
  - **alfabetizacao_grupo_idade_sexo_raca/**  (1 files, 1.7 MB, 6 cols)
  - **cadastro_enderecos/**  (425 files, 2.8 GB, 35 cols)
  - **caracteristica_domicilio_grupo_idade_raca_destino_lixo/**  (11 files, 9.9 MB, 6 cols)
  - **caracteristica_domicilio_grupo_idade_raca_esgotamento_sanitario/**  (14 files, 9.4 MB, 6 cols)
  - **caracteristica_domicilio_grupo_idade_raca_ligacao_abastecimento_agua/**  (20 files, 22.7 MB, 6 cols)
  - **caracteristica_domicilio_grupo_idade_raca_tipo_domicilio/**  (9 files, 5.4 MB, 6 cols)
  - **dicionario/**  (1 files, 2.2 KB, 5 cols)
  - **domicilio_recenseado/**  (1 files, 218.4 KB, 3 cols)
  - **indice_envelhecimento_raca/**  (1 files, 342.8 KB, 6 cols)
  - **municipio/**  (1 files, 177.5 KB, 13 cols)
  - **populacao_grupo_idade_sexo_raca/**  (5 files, 5.4 MB, 6 cols)
  - **populacao_grupo_idade_uf/**  (1 files, 2.9 KB, 3 cols)
  - **populacao_idade_sexo/**  (7 files, 5.7 MB, 7 cols)
  - **setor_censitario/**  (13 files, 1.3 GB, 1423 cols)
  - **terra_indigena/**  (1 files, 14.7 KB, 6 cols)
  - **territorio_quilombola/**  (1 files, 11.8 KB, 6 cols)
 ## br_ibge_censo_demografico/  (33 tables, 4.2 GB, 1058 files)
  - **dicionario/**  (1 files, 31.0 KB, 5 cols)
  - **microdados_domicilio_1970/**  (17 files, 37.7 MB, 26 cols)
  - **microdados_domicilio_1980/**  (17 files, 27.9 MB, 26 cols)
  - **microdados_domicilio_1991/**  (17 files, 97.5 MB, 43 cols)
  - **microdados_domicilio_2000/**  (25 files, 130.2 MB, 56 cols)
  - **microdados_domicilio_2010/**  (50 files, 176.0 MB, 76 cols)
  - **microdados_pessoa_1970/**  (100 files, 308.6 MB, 41 cols)
  - **microdados_pessoa_1980/**  (150 files, 380.6 MB, 64 cols)
  - **microdados_pessoa_1991/**  (150 files, 636.0 MB, 100 cols)
  - **microdados_pessoa_2000/**  (150 files, 838.2 MB, 110 cols)
  - **microdados_pessoa_2010/**  (350 files, 1.0 GB, 244 cols)
  - **setor_censitario_alfabetizacao_homens_mulheres_2010/**  (1 files, 26.9 MB, 172 cols)
  - **setor_censitario_alfabetizacao_total_2010/**  (1 files, 16.6 MB, 87 cols)
  - **setor_censitario_basico_2010/**  (1 files, 16.7 MB, 14 cols)
  - **setor_censitario_domicilio_caracteristicas_gerais_2010/**  (2 files, 32.4 MB, 243 cols)
  - **setor_censitario_domicilio_moradores_2010/**  (1 files, 32.8 MB, 134 cols)
  - **setor_censitario_domicilio_renda_2010/**  (1 files, 5.9 MB, 16 cols)
  - **setor_censitario_entorno_2010/**  (5 files, 175.9 MB, 1064 cols)
  - **setor_censitario_idade_homens_2010/**  (1 files, 17.4 MB, 136 cols)
  - **setor_censitario_idade_mulheres_2010/**  (1 files, 17.8 MB, 136 cols)
  - **setor_censitario_idade_total_2010/**  (1 files, 21.7 MB, 136 cols)
  - **setor_censitario_pessoa_renda_2010/**  (1 files, 55.6 MB, 134 cols)
  - **setor_censitario_raca_alfabetizacao_idade_genero_2010/**  (1 files, 18.8 MB, 157 cols)
  - **setor_censitario_raca_idade_0_4_genero_2010/**  (1 files, 2.5 MB, 12 cols)
  - **setor_censitario_raca_idade_genero_2010/**  (2 files, 33.1 MB, 253 cols)
  - **setor_censitario_registro_civil_2010/**  (1 files, 1.4 MB, 5 cols)
  - **setor_censitario_relacao_parentesco_conjuges_2010/**  (1 files, 18.5 MB, 215 cols)
  - **setor_censitario_relacao_parentesco_filhos_2010/**  (1 files, 18.7 MB, 206 cols)
  - **setor_censitario_relacao_parentesco_filhos_enteados_2010/**  (2 files, 15.9 MB, 256 cols)
  - **setor_censitario_relacao_parentesco_outros_2010/**  (2 files, 12.4 MB, 242 cols)
  - **setor_censitario_responsavel_domicilios_homens_total_2010/**  (2 files, 24.7 MB, 218 cols)
  - **setor_censitario_responsavel_domicilios_mulheres_2010/**  (1 files, 9.9 MB, 110 cols)
  - **setor_censitario_responsavel_renda_2010/**  (1 files, 48.2 MB, 134 cols)
 ## br_ibge_estadic/  (1 tables, 4.5 KB, 1 files)
  - **dicionario/**  (1 files, 4.5 KB, 5 cols)
 ## br_ibge_inpc/  (4 tables, 3.6 MB, 4 files)
  - **mes_brasil/**  (1 files, 17.8 KB, 8 cols)
  - **mes_categoria_brasil/**  (1 files, 283.8 KB, 8 cols)
  - **mes_categoria_municipio/**  (1 files, 1.5 MB, 10 cols)
  - **mes_categoria_rm/**  (1 files, 1.8 MB, 10 cols)
 ## br_ibge_ipca/  (4 tables, 3.6 MB, 4 files)
  - **mes_brasil/**  (1 files, 17.3 KB, 8 cols)
  - **mes_categoria_brasil/**  (1 files, 286.3 KB, 8 cols)
  - **mes_categoria_municipio/**  (1 files, 1.5 MB, 10 cols)
  - **mes_categoria_rm/**  (1 files, 1.8 MB, 10 cols)
 ## br_ibge_ipca15/  (4 tables, 2.3 MB, 4 files)
  - **mes_brasil/**  (1 files, 9.4 KB, 8 cols)
  - **mes_categoria_brasil/**  (1 files, 279.2 KB, 8 cols)
  - **mes_categoria_municipio/**  (1 files, 411.5 KB, 10 cols)
  - **mes_categoria_rm/**  (1 files, 1.6 MB, 10 cols)
 ## br_ibge_pam/  (2 tables, 67.3 MB, 148 files)
  - **lavoura_permanente/**  (75 files, 32.9 MB, 9 cols)
  - **lavoura_temporaria/**  (73 files, 34.4 MB, 9 cols)
 ## br_ibge_pevs/  (2 tables, 3.3 MB, 74 files)
  - **producao_extracao_vegetal/**  (37 files, 2.5 MB, 7 cols)
  - **producao_silvicultura/**  (37 files, 805.8 KB, 9 cols)
 ## br_ibge_pib/  (2 tables, 3.6 MB, 2 files)
  - **gini/**  (1 files, 24.8 KB, 7 cols)
  - **municipio/**  (1 files, 3.6 MB, 9 cols)
 ## br_ibge_pnad/  (3 tables, 116.5 MB, 57 files)
  - **dicionario/**  (1 files, 3.3 KB, 5 cols)
  - **microdados_compatibilizados_domicilio/**  (17 files, 19.0 MB, 39 cols)
  - **microdados_compatibilizados_pessoa/**  (39 files, 97.5 MB, 70 cols)
 ## br_ibge_pnad_covid/  (1 tables, 7.4 KB, 1 files)
  - **dicionario/**  (1 files, 7.4 KB, 5 cols)
 ## br_ibge_pnadc/  (4 tables, 27.4 GB, 1377 files)
  - **dicionario/**  (1 files, 22.4 KB, 5 cols)
  - **educacao/**  (39 files, 1.1 GB, 279 cols)
  - **microdados/**  (1287 files, 25.6 GB, 424 cols)
  - **rendimentos_outras_fontes/**  (50 files, 707.1 MB, 293 cols)
 ## br_ibge_pof/  (1 tables, 38.7 KB, 1 files)
  - **dicionario/**  (1 files, 38.7 KB, 5 cols)
 ## br_ibge_populacao/  (3 tables, 646.5 KB, 3 files)
  - **brasil/**  (1 files, 1.1 KB, 2 cols)
  - **municipio/**  (1 files, 639.4 KB, 4 cols)
  - **uf/**  (1 files, 6.1 KB, 3 cols)
 ## br_ibge_ppm/  (4 tables, 12.7 MB, 160 files)
  - **efetivo_rebanhos/**  (52 files, 5.8 MB, 5 cols)
  - **producao_aquicultura/**  (10 files, 578.7 KB, 6 cols)
  - **producao_origem_animal/**  (49 files, 4.5 MB, 7 cols)
  - **producao_pecuaria/**  (49 files, 1.9 MB, 5 cols)
 ## br_inep_ana/  (1 tables, 4.1 KB, 1 files)
  - **dicionario/**  (1 files, 4.1 KB, 5 cols)
 ## br_inep_avaliacao_alfabetizacao/  (7 tables, 57.2 MB, 17 files)
  - **alunos/**  (11 files, 56.6 MB, 12 cols)
  - **dicionario/**  (1 files, 1.8 KB, 5 cols)
  - **meta_alfabetizacao_brasil/**  (1 files, 4.0 KB, 11 cols)
  - **meta_alfabetizacao_municipio/**  (1 files, 159.7 KB, 13 cols)
  - **meta_alfabetizacao_uf/**  (1 files, 6.4 KB, 12 cols)
  - **municipio/**  (1 files, 411.7 KB, 15 cols)
  - **uf/**  (1 files, 10.6 KB, 15 cols)
 ## br_inep_censo_educacao_superior/  (3 tables, 190.4 MB, 129 files)
  - **curso/**  (111 files, 186.0 MB, 193 cols)
  - **dicionario/**  (1 files, 2.1 KB, 5 cols)
  - **ies/**  (17 files, 4.4 MB, 71 cols)
 ## br_inep_censo_escolar/  (3 tables, 684.8 MB, 340 files)
  - **dicionario/**  (1 files, 5.1 KB, 5 cols)
  - **escola/**  (112 files, 243.3 MB, 455 cols)
  - **turma/**  (227 files, 441.5 MB, 76 cols)
 ## br_inep_educacao_especial/  (15 tables, 30.7 MB, 145 files)
  - **brasil_distorcao_idade_serie/**  (1 files, 1.4 KB, 3 cols)
  - **brasil_taxa_rendimento/**  (1 files, 2.5 KB, 5 cols)
  - **distorcao_idade_serie/**  (1 files, 5.6 KB, 4 cols)
  - **docente_aee/**  (1 files, 105.5 KB, 7 cols)
  - **docente_formacao/**  (1 files, 247.6 KB, 5 cols)
  - **etapa_ensino/**  (24 files, 6.6 MB, 6 cols)
  - **faixa_etaria/**  (21 files, 3.5 MB, 6 cols)
  - **localizacao/**  (22 files, 4.1 MB, 7 cols)
  - **matricula_aee/**  (1 files, 4.8 KB, 5 cols)
  - **sexo_raca_cor/**  (23 files, 6.1 MB, 7 cols)
  - **taxa_rendimento/**  (1 files, 11.0 KB, 6 cols)
  - **tempo_ensino/**  (22 files, 4.1 MB, 7 cols)
  - **tipo_deficiencia/**  (24 files, 6.0 MB, 6 cols)
  - **uf_distorcao_idade_serie/**  (1 files, 4.8 KB, 4 cols)
  - **uf_taxa_rendimento/**  (1 files, 8.9 KB, 6 cols)
 ## br_inep_enem/  (28 tables, 7.6 GB, 1631 files)
  - **dicionario/**  (1 files, 50.9 KB, 5 cols)
  - **microdados/**  (845 files, 5.8 GB, 63 cols)
  - **questionario_socioeconomico_1998/**  (1 files, 3.8 MB, 138 cols)
  - **questionario_socioeconomico_1999/**  (1 files, 7.5 MB, 130 cols)
  - **questionario_socioeconomico_2000/**  (1 files, 8.3 MB, 128 cols)
  - **questionario_socioeconomico_2001/**  (50 files, 74.3 MB, 243 cols)
  - **questionario_socioeconomico_2002/**  (22 files, 68.8 MB, 220 cols)
  - **questionario_socioeconomico_2003/**  (19 files, 55.7 MB, 189 cols)
  - **questionario_socioeconomico_2004/**  (17 files, 45.0 MB, 206 cols)
  - **questionario_socioeconomico_2005/**  (50 files, 95.8 MB, 224 cols)
  - **questionario_socioeconomico_2006/**  (50 files, 122.5 MB, 224 cols)
  - **questionario_socioeconomico_2007/**  (50 files, 135.1 MB, 224 cols)
  - **questionario_socioeconomico_2008/**  (50 files, 125.6 MB, 224 cols)
  - **questionario_socioeconomico_2009/**  (100 files, 145.6 MB, 294 cols)
  - **questionario_socioeconomico_2010/**  (17 files, 44.6 MB, 58 cols)
  - **questionario_socioeconomico_2011/**  (21 files, 65.8 MB, 76 cols)
  - **questionario_socioeconomico_2012/**  (20 files, 66.1 MB, 63 cols)
  - **questionario_socioeconomico_2013/**  (50 files, 98.8 MB, 77 cols)
  - **questionario_socioeconomico_2014/**  (50 files, 115.2 MB, 77 cols)
  - **questionario_socioeconomico_2015/**  (50 files, 103.8 MB, 51 cols)
  - **questionario_socioeconomico_2016/**  (50 files, 114.9 MB, 51 cols)
  - **questionario_socioeconomico_2017/**  (17 files, 58.3 MB, 28 cols)
  - **questionario_socioeconomico_2018/**  (17 files, 49.8 MB, 28 cols)
  - **questionario_socioeconomico_2019/**  (17 files, 43.3 MB, 26 cols)
  - **questionario_socioeconomico_2020/**  (17 files, 47.8 MB, 26 cols)
  - **questionario_socioeconomico_2021/**  (15 files, 29.2 MB, 26 cols)
  - **questionario_socioeconomico_2022/**  (16 files, 30.2 MB, 26 cols)
  - **questionario_socioeconomico_2023/**  (17 files, 34.6 MB, 26 cols)
 ## br_inep_formacao_docente/  (1 tables, 1.9 KB, 1 files)
  - **dicionario/**  (1 files, 1.9 KB, 5 cols)
 ## br_inep_ideb/  (5 tables, 27.0 MB, 9 files)
  - **brasil/**  (1 files, 8.6 KB, 11 cols)
  - **escola/**  (5 files, 21.3 MB, 14 cols)
  - **municipio/**  (1 files, 5.5 MB, 13 cols)
  - **regiao/**  (1 files, 21.0 KB, 12 cols)
  - **uf/**  (1 files, 87.7 KB, 12 cols)
 ## br_inep_indicador_nivel_socioeconomico/  (2 tables, 5.8 MB, 2 files)
  - **dicionario/**  (1 files, 3.5 KB, 5 cols)
  - **escola/**  (1 files, 5.8 MB, 18 cols)
 ## br_inep_indicadores_educacionais/  (11 tables, 395.6 MB, 107 files)
  - **brasil/**  (19 files, 1.5 MB, 214 cols)
  - **brasil_remuneracao_docentes/**  (1 files, 11.7 KB, 12 cols)
  - **brasil_taxa_transicao/**  (1 files, 42.5 KB, 67 cols)
  - **escola/**  (34 files, 218.8 MB, 208 cols)
  - **municipio/**  (32 files, 152.6 MB, 215 cols)
  - **municipio_taxa_transicao/**  (15 files, 19.9 MB, 68 cols)
  - **regiao/**  (1 files, 526.3 KB, 215 cols)
  - **regiao_taxa_transicao/**  (1 files, 80.2 KB, 68 cols)
  - **uf/**  (1 files, 1.8 MB, 215 cols)
  - **uf_remuneracao_docentes/**  (1 files, 106.9 KB, 13 cols)
  - **uf_taxa_transicao/**  (1 files, 172.7 KB, 68 cols)
 ## br_inep_saeb/  (11 tables, 7.9 GB, 1025 files)
  - **aluno_ef_2ano/**  (3 files, 8.7 MB, 38 cols)
  - **aluno_ef_5ano/**  (321 files, 2.4 GB, 243 cols)
  - **aluno_ef_9ano/**  (386 files, 2.6 GB, 267 cols)
  - **aluno_em_34ano/**  (50 files, 241.0 MB, 105 cols)
  - **brasil/**  (1 files, 59.0 KB, 17 cols)
  - **brasil_taxa_alfabetizacao/**  (1 files, 2.1 KB, 5 cols)
  - **dicionario/**  (1 files, 19.2 KB, 5 cols)
  - **municipio/**  (11 files, 32.1 MB, 19 cols)
  - **proficiencia/**  (249 files, 2.6 GB, 21 cols)
  - **uf/**  (1 files, 1.1 MB, 18 cols)
  - **uf_taxa_alfabetizacao/**  (1 files, 7.4 KB, 6 cols)
 ## br_inep_sinopse_estatistica_educacao_basica/  (18 tables, 256.2 MB, 559 files)
  - **dicionario/**  (1 files, 2.3 KB, 5 cols)
  - **docente_deficiencia/**  (16 files, 2.8 MB, 6 cols)
  - **docente_escolaridade/**  (36 files, 19.6 MB, 6 cols)
  - **docente_etapa_ensino/**  (52 files, 34.4 MB, 7 cols)
  - **docente_faixa_etaria_sexo/**  (54 files, 36.6 MB, 7 cols)
  - **docente_localizacao/**  (61 files, 43.0 MB, 7 cols)
  - **docente_regime_contrato/**  (38 files, 21.5 MB, 7 cols)
  - **educacao_especial_etapa_ensino/**  (23 files, 2.8 MB, 6 cols)
  - **educacao_especial_faixa_etaria/**  (19 files, 1.7 MB, 6 cols)
  - **educacao_especial_localizacao/**  (20 files, 3.8 MB, 7 cols)
  - **educacao_especial_sexo_raca_cor/**  (22 files, 5.5 MB, 7 cols)
  - **educacao_especial_tempo_ensino/**  (20 files, 3.8 MB, 7 cols)
  - **educacao_especial_tipo_deficiencia/**  (22 files, 3.0 MB, 6 cols)
  - **etapa_ensino_serie/**  (38 files, 23.1 MB, 7 cols)
  - **faixa_etaria/**  (27 files, 7.0 MB, 6 cols)
  - **localizacao/**  (35 files, 17.9 MB, 7 cols)
  - **sexo_raca_cor/**  (43 files, 15.8 MB, 7 cols)
  - **tempo_ensino/**  (32 files, 14.0 MB, 7 cols)
 ## br_inmet_bdmep/  (1 tables, 1.3 GB, 210 files)
  - **microdados/**  (210 files, 1.3 GB, 22 cols)
 ## br_inpe_prodes/  (1 tables, 862.4 KB, 1 files)
  - **municipio_bioma/**  (1 files, 862.4 KB, 8 cols)
 ## br_inpe_queimadas/  (1 tables, 268.1 MB, 65 files)
  - **microdados/**  (65 files, 268.1 MB, 13 cols)
 ## br_inpe_sisam/  (1 tables, 1.5 GB, 417 files)
  - **microdados/**  (417 files, 1.5 GB, 14 cols)
 ## br_ipea_avs/  (1 tables, 35.5 MB, 1 files)
  - **municipio/**  (1 files, 35.5 MB, 92 cols)
 ## br_mdr_snis/  (2 tables, 63.0 MB, 2 files)
  - **municipio_agua_esgoto/**  (1 files, 31.3 MB, 133 cols)
  - **prestador_agua_esgoto/**  (1 files, 31.7 MB, 144 cols)
 ## br_me_caged/  (4 tables, 1.6 GB, 705 files)
  - **dicionario/**  (1 files, 38.0 KB, 5 cols)
  - **microdados_movimentacao/**  (689 files, 1.5 GB, 25 cols)
  - **microdados_movimentacao_excluida/**  (1 files, 5.2 MB, 30 cols)
  - **microdados_movimentacao_fora_prazo/**  (14 files, 71.1 MB, 27 cols)
 ## br_me_cno/  (1 tables, 1.9 KB, 1 files)
  - **dicionario/**  (1 files, 1.9 KB, 5 cols)
 ## br_me_cnpj/  (5 tables, 194.1 GB, 8473 files)
  - **dicionario/**  (1 files, 8.2 KB, 5 cols)
  - **empresas/**  (3070 files, 45.0 GB, 10 cols)
  - **estabelecimentos/**  (3353 files, 128.6 GB, 35 cols)
  - **simples/**  (100 files, 283.9 MB, 7 cols)
  - **socios/**  (1949 files, 20.2 GB, 14 cols)
 ## br_me_comex_stat/  (5 tables, 1.1 GB, 445 files)
  - **dicionario/**  (1 files, 11.4 KB, 5 cols)
  - **municipio_exportacao/**  (83 files, 154.6 MB, 9 cols)
  - **municipio_importacao/**  (113 files, 217.7 MB, 9 cols)
  - **ncm_exportacao/**  (106 files, 264.4 MB, 12 cols)
  - **ncm_importacao/**  (142 files, 501.1 MB, 14 cols)
 ## br_me_rais/  (3 tables, 51.9 GB, 3541 files)
  - **dicionario/**  (1 files, 54.5 KB, 5 cols)
  - **microdados_estabelecimentos/**  (566 files, 803.2 MB, 26 cols)
  - **microdados_vinculos/**  (2974 files, 51.1 GB, 66 cols)
 ## br_me_sic/  (2 tables, 200.4 KB, 2 files)
  - **dicionario/**  (1 files, 11.7 KB, 5 cols)
  - **transferencia/**  (1 files, 188.7 KB, 17 cols)
 ## br_me_siconfi/  (7 tables, 441.6 MB, 281 files)
  - **municipio_balanco_patrimonial/**  (27 files, 51.4 MB, 8 cols)
  - **municipio_despesas_funcao/**  (60 files, 127.7 MB, 10 cols)
  - **municipio_despesas_orcamentarias/**  (89 files, 143.9 MB, 10 cols)
  - **municipio_receitas_orcamentarias/**  (72 files, 115.4 MB, 10 cols)
  - **uf_despesas_funcao/**  (11 files, 1.4 MB, 10 cols)
  - **uf_despesas_orcamentarias/**  (11 files, 1.0 MB, 10 cols)
  - **uf_receitas_orcamentarias/**  (11 files, 792.3 KB, 10 cols)
 ## br_mec_prouni/  (1 tables, 1.7 KB, 1 files)
  - **dicionario/**  (1 files, 1.7 KB, 5 cols)
 ## br_mec_sisu/  (1 tables, 1.4 GB, 314 files)
  - **microdados/**  (314 files, 1.4 GB, 52 cols)
 ## br_mg_belohorizonte_smfa_iptu/  (2 tables, 2.4 GB, 221 files)
  - **dicionario/**  (1 files, 1.7 KB, 5 cols)
  - **iptu/**  (220 files, 2.4 GB, 26 cols)
 ## br_mme_consumo_energia_eletrica/  (1 tables, 370.2 KB, 1 files)
  - **uf/**  (1 files, 370.2 KB, 6 cols)
 ## br_mp_pep/  (1 tables, 7.4 MB, 7 files)
  - **cargos_funcoes/**  (7 files, 7.4 MB, 16 cols)
 ## br_ms_cnes/  (14 tables, 24.4 GB, 1424 files)
  - **dados_complementares/**  (52 files, 18.5 MB, 94 cols)
  - **dicionario/**  (1 files, 25.1 KB, 5 cols)
  - **equipamento/**  (52 files, 863.1 MB, 11 cols)
  - **equipe/**  (41 files, 105.0 MB, 24 cols)
  - **estabelecimento/**  (214 files, 1.7 GB, 204 cols)
  - **estabelecimento_ensino/**  (13 files, 135.6 KB, 14 cols)
  - **estabelecimento_filantropico/**  (19 files, 320.4 KB, 14 cols)
  - **gestao_metas/**  (19 files, 746.3 KB, 15 cols)
  - **habilitacao/**  (41 files, 15.6 MB, 16 cols)
  - **incentivos/**  (24 files, 3.0 MB, 15 cols)
  - **leito/**  (41 files, 15.7 MB, 10 cols)
  - **profissional/**  (820 files, 21.0 GB, 23 cols)
  - **regra_contratual/**  (17 files, 4.3 MB, 15 cols)
  - **servico_especializado/**  (70 files, 788.0 MB, 15 cols)
 ## br_ms_pns/  (3 tables, 51.9 MB, 8 files)
  - **dicionario/**  (1 files, 36.3 KB, 5 cols)
  - **microdados_2013/**  (3 files, 20.6 MB, 1000 cols)
  - **microdados_2019/**  (4 files, 31.2 MB, 1087 cols)
 ## br_ms_populacao/  (1 tables, 16.3 MB, 9 files)
  - **municipio/**  (9 files, 16.3 MB, 5 cols)
 ## br_ms_sia/  (3 tables, 46.2 GB, 7629 files)
  - **dicionario/**  (1 files, 129.3 KB, 5 cols)
  - **producao_ambulatorial/**  (7159 files, 45.3 GB, 59 cols)
  - **psicossocial/**  (469 files, 962.4 MB, 41 cols)
 ## br_ms_sih/  (3 tables, 31.6 GB, 5824 files)
  - **aihs_reduzidas/**  (1794 files, 7.6 GB, 109 cols)
  - **dicionario/**  (1 files, 206.7 KB, 5 cols)
  - **servicos_profissionais/**  (4029 files, 23.9 GB, 37 cols)
 ## br_ms_sim/  (2 tables, 872.2 MB, 138 files)
  - **dicionario/**  (1 files, 6.6 KB, 5 cols)
  - **microdados/**  (137 files, 872.2 MB, 92 cols)
 ## br_ms_sinan/  (3 tables, 616.2 MB, 215 files)
  - **dicionario/**  (1 files, 7.2 KB, 5 cols)
  - **microdados_dengue/**  (179 files, 503.7 MB, 151 cols)
  - **microdados_influenza_srag/**  (35 files, 112.5 MB, 205 cols)
 ## br_ms_sinasc/  (2 tables, 1.4 GB, 352 files)
  - **dicionario/**  (1 files, 6.4 KB, 5 cols)
  - **microdados/**  (351 files, 1.4 GB, 66 cols)
 ## br_ms_sisvan/  (2 tables, 19.2 GB, 1540 files)
  - **dicionario/**  (1 files, 2.3 KB, 5 cols)
  - **microdados/**  (1539 files, 19.2 GB, 28 cols)
 ## br_ms_vacinacao_covid19/  (1 tables, 3.8 KB, 1 files)
  - **dicionario/**  (1 files, 3.8 KB, 5 cols)
 ## br_poder360_pesquisas/  (1 tables, 1.3 MB, 1 files)
  - **microdados/**  (1 files, 1.3 MB, 24 cols)
 ## br_rf_arrecadacao/  (5 tables, 5.9 MB, 57 files)
  - **cnae/**  (9 files, 352.9 KB, 20 cols)
  - **ir_ipi/**  (6 files, 41.0 KB, 10 cols)
  - **itr/**  (8 files, 3.3 MB, 5 cols)
  - **natureza_juridica/**  (9 files, 577.9 KB, 20 cols)
  - **uf/**  (25 files, 1.7 MB, 45 cols)
 ## br_rf_cafir/  (2 tables, 3.6 GB, 450 files)
  - **dicionario/**  (1 files, 2.1 KB, 5 cols)
  - **imoveis_rurais/**  (449 files, 3.6 GB, 14 cols)
 ## br_rf_cno/  (5 tables, 32.5 GB, 2110 files)
  - **areas/**  (1236 files, 4.6 GB, 8 cols)
  - **cnaes/**  (489 files, 2.3 GB, 4 cols)
  - **dicionario/**  (1 files, 2.1 KB, 5 cols)
  - **microdados/**  (193 files, 25.1 GB, 25 cols)
  - **vinculos/**  (191 files, 497.9 MB, 7 cols)
 ## br_rj_isp_estatisticas_seguranca/  (14 tables, 2.8 MB, 14 files)
  - **armas_apreendidas_mensal/**  (1 files, 229.4 KB, 42 cols)
  - **armas_fogo_apreendidas_mensal/**  (1 files, 11.1 KB, 7 cols)
  - **evolucao_mensal_cisp/**  (1 files, 1005.0 KB, 61 cols)
  - **evolucao_mensal_municipio/**  (1 files, 361.0 KB, 58 cols)
  - **evolucao_mensal_uf/**  (1 files, 60.2 KB, 56 cols)
  - **evolucao_mensal_upp/**  (1 files, 89.2 KB, 38 cols)
  - **evolucao_policial_morto_servico_mensal/**  (1 files, 50.4 KB, 5 cols)
  - **feminicidio_mensal_cisp/**  (1 files, 17.4 KB, 9 cols)
  - **relacao_cisp_aisp_risp/**  (1 files, 7.2 KB, 6 cols)
  - **taxa_evolucao_anual_municipio/**  (1 files, 75.7 KB, 56 cols)
  - **taxa_evolucao_anual_uf/**  (1 files, 29.5 KB, 55 cols)
  - **taxa_evolucao_mensal_municipio/**  (1 files, 852.5 KB, 58 cols)
  - **taxa_evolucao_mensal_uf/**  (1 files, 59.9 KB, 56 cols)
  - **taxa_letalidade/**  (1 files, 6.6 KB, 6 cols)
 ## br_seeg_emissoes/  (3 tables, 2.5 GB, 557 files)
  - **dicionario/**  (1 files, 13.7 KB, 5 cols)
  - **municipio/**  (457 files, 2.5 GB, 17 cols)
  - **uf/**  (99 files, 87.7 MB, 13 cols)
 ## br_sfb_sicar/  (2 tables, 28.3 GB, 947 files)
  - **area_imovel/**  (946 files, 28.3 GB, 11 cols)
  - **dicionario/**  (1 files, 1.7 KB, 5 cols)
 ## br_simet_educacao_conectada/  (1 tables, 10.0 MB, 1 files)
  - **escola/**  (1 files, 10.0 MB, 54 cols)
 ## br_sp_saopaulo_geosampa_iptu/  (1 tables, 2.3 GB, 447 files)
  - **iptu/**  (447 files, 2.3 GB, 27 cols)
 ## br_stf_corte_aberta/  (2 tables, 87.3 MB, 33 files)
  - **decisoes/**  (32 files, 87.3 MB, 17 cols)
  - **dicionario/**  (1 files, 2.3 KB, 5 cols)
 ## br_trase_supply_chain/  (6 tables, 59.9 MB, 24 files)
  - **beef/**  (3 files, 44.2 MB, 22 cols)
  - **beef_slaughterhouses/**  (1 files, 718.9 KB, 19 cols)
  - **soy_beans/**  (17 files, 14.8 MB, 25 cols)
  - **soy_beans_crushing_facilities/**  (1 files, 19.6 KB, 13 cols)
  - **soy_beans_refining_facilities/**  (1 files, 9.4 KB, 9 cols)
  - **soy_beans_storage_facilities/**  (1 files, 225.1 KB, 12 cols)
 ## br_tse_eleicoes/  (22 tables, 8.2 GB, 4324 files)
  - **bens_candidato/**  (18 files, 134.1 MB, 10 cols)
  - **candidatos/**  (29 files, 149.3 MB, 28 cols)
  - **despesas_candidato/**  (255 files, 1.5 GB, 45 cols)
  - **detalhes_votacao_municipio/**  (16 files, 17.4 MB, 25 cols)
  - **detalhes_votacao_municipio_zona/**  (16 files, 19.7 MB, 26 cols)
  - **detalhes_votacao_secao/**  (90 files, 401.2 MB, 24 cols)
  - **dicionario/**  (1 files, 2.3 KB, 5 cols)
  - **partidos/**  (21 files, 7.9 MB, 21 cols)
  - **perfil_eleitorado_local_votacao/**  (15 files, 106.1 MB, 23 cols)
  - **perfil_eleitorado_municipio_zona/**  (98 files, 128.2 MB, 13 cols)
  - **perfil_eleitorado_secao/**  (1425 files, 1.9 GB, 15 cols)
  - **receitas_candidato/**  (108 files, 806.4 MB, 55 cols)
  - **receitas_comite/**  (7 files, 7.9 MB, 36 cols)
  - **receitas_orgao_partidario/**  (7 files, 9.0 MB, 50 cols)
  - **resultados_candidato/**  (37 files, 43.7 MB, 16 cols)
  - **resultados_candidato_municipio/**  (99 files, 126.6 MB, 16 cols)
  - **resultados_candidato_municipio_zona/**  (125 files, 187.0 MB, 17 cols)
  - **resultados_candidato_secao/**  (1367 files, 2.0 GB, 17 cols)
  - **resultados_partido_municipio/**  (25 files, 17.4 MB, 13 cols)
  - **resultados_partido_municipio_zona/**  (26 files, 19.6 MB, 14 cols)
  - **resultados_partido_secao/**  (523 files, 608.5 MB, 15 cols)
  - **vagas/**  (16 files, 594.7 KB, 9 cols)
 ## br_tse_filiacao_partidaria/  (2 tables, 941.9 MB, 106 files)
  - **microdados/**  (44 files, 483.5 MB, 22 cols)
  - **microdados_antigos/**  (62 files, 458.4 MB, 16 cols)
 ## dataset_new_arch/  (1 tables, 1.6 KB, 1 files)
  - **tabela_new_arch/**  (1 files, 1.6 KB, 5 cols)
 ## logs/  (2 tables, 5.5 GB, 4675 files)
  - **cloudaudit_googleapis_com_activity/**  (1826 files, 570.1 MB, 18 cols)
  - **cloudaudit_googleapis_com_data_access/**  (2849 files, 4.9 GB, 18 cols)
 ## mundo_transfermarkt_competicoes/  (2 tables, 529.9 KB, 25 files)
  - **brasileirao_serie_a/**  (22 files, 458.6 KB, 35 cols)
  - **copa_brasil/**  (3 files, 71.3 KB, 38 cols)
 ## mundo_transfermarkt_competicoes_internacionais/  (1 tables, 114.2 KB, 1 files)
  - **champions_league/**  (1 files, 114.2 KB, 55 cols)
 ## test_dataset/  (1 tables, 1.3 KB, 1 files)
  - **test_table/**  (1 files, 1.3 KB, 4 cols)
 ## us_harvard_ned/  (2 tables, 26.8 MB, 423 files)
  - **parliamentary_elections/**  (233 files, 13.4 MB, 238 cols)
  - **presidential_elections/**  (190 files, 13.4 MB, 325 cols)
 ## world_ampas_oscar/  (1 tables, 16.7 KB, 1 files)
  - **winner_demographics/**  (1 files, 16.7 KB, 10 cols)
 ## world_iea_pirls/  (8 tables, 310.5 MB, 8 files)
  - **dictionary/**  (1 files, 27.0 KB, 5 cols)
  - **home_context/**  (1 files, 9.7 MB, 120 cols)
  - **school_context/**  (1 files, 680.6 KB, 103 cols)
  - **student_achievement/**  (1 files, 106.5 MB, 864 cols)
  - **student_context/**  (1 files, 104.2 MB, 157 cols)
  - **student_teacher_link/**  (1 files, 81.1 MB, 51 cols)
  - **teacher_context/**  (1 files, 1003.3 KB, 186 cols)
  - **within_country_scoring_reliability/**  (1 files, 7.2 MB, 1057 cols)
 ## world_iea_timss/  (11 tables, 484.4 MB, 11 files)
  - **dictionary/**  (1 files, 15.8 KB, 5 cols)
  - **home_context_grade_4/**  (1 files, 8.9 MB, 114 cols)
  - **school_context_grade_4/**  (1 files, 672.5 KB, 111 cols)
  - **school_context_grade_8/**  (1 files, 478.4 KB, 103 cols)
  - **student_achievement_grade_4/**  (1 files, 236.3 MB, 110 cols)
  - **student_achievement_grade_8/**  (1 files, 213.0 MB, 135 cols)
  - **student_context_grade_4/**  (1 files, 13.0 MB, 127 cols)
  - **student_context_grade_8/**  (1 files, 9.4 MB, 115 cols)
  - **teacher_context_grade_4/**  (1 files, 680.6 KB, 87 cols)
  - **teacher_mathematics_grade_8/**  (1 files, 627.1 KB, 168 cols)
  - **teacher_science_grade_8/**  (1 files, 1.3 MB, 217 cols)
 ## world_imdb_movies/  (1 tables, 3.7 MB, 1 files)
  - **top_movies_per_year/**  (1 files, 3.7 MB, 23 cols)
 ## world_oecd_pisa/  (1 tables, 742.4 MB, 28 files)
  - **student/**  (28 files, 742.4 MB, 250 cols)
 ## world_oecd_public_finance/  (1 tables, 709.9 KB, 1 files)
  - **country/**  (1 files, 709.9 KB, 163 cols)
 ## world_olympedia_olympics/  (6 tables, 23.5 MB, 6 files)
  - **athlete_bio/**  (1 files, 16.5 MB, 11 cols)
  - **athlete_event_result/**  (1 files, 3.9 MB, 11 cols)
  - **country/**  (1 files, 3.2 KB, 2 cols)
  - **game/**  (1 files, 5.7 KB, 10 cols)
  - **game_medal_tally/**  (1 files, 14.6 KB, 9 cols)
  - **result/**  (1 files, 3.0 MB, 11 cols)
 ## world_sofascore_competicoes_futebol/  (2 tables, 1.4 MB, 27 files)
  - **brasileirao_serie_a/**  (9 files, 603.4 KB, 85 cols)
  - **uefa_champions_league/**  (18 files, 811.5 KB, 85 cols)
 ## world_wb_mides/  (9 tables, 37.4 GB, 4645 files)
  - **dicionario/**  (1 files, 16.5 KB, 5 cols)
  - **empenho/**  (1488 files, 13.1 GB, 25 cols)
  - **licitacao/**  (22 files, 160.9 MB, 32 cols)
  - **licitacao_item/**  (210 files, 2.3 GB, 24 cols)
  - **licitacao_participante/**  (22 files, 100.5 MB, 17 cols)
  - **liquidacao/**  (1346 files, 7.2 GB, 20 cols)
  - **orgao_unidade_gestora/**  (1 files, 2.1 MB, 8 cols)
  - **pagamento/**  (1522 files, 14.3 GB, 25 cols)
  - **relacionamentos/**  (33 files, 132.7 MB, 5 cols)
 ## world_wwf_hydrosheds/  (3 tables, 9.1 GB, 735 files)
  - **basins_atlas/**  (279 files, 5.6 GB, 296 cols)
  - **lakes_atlas/**  (132 files, 1.5 GB, 307 cols)
  - **rivers_atlas/**  (324 files, 2.0 GB, 297 cols)
 ---
 **Total: 533 tables · 675.4 GB · 77885 parquet files**
--- a/gera_schemas.py
+++ b/gera_schemas.py
@@ -1,268 +0,0 @@
 import os
 import json
 import sys
 import pyarrow.parquet as pq
 import s3fs
 import boto3
 import duckdb
 from dotenv import load_dotenv
 load_dotenv()
 S3_ENDPOINT  = os.environ["HETZNER_S3_ENDPOINT"]
 S3_BUCKET    = os.environ["HETZNER_S3_BUCKET"]
 ACCESS_KEY   = os.environ["AWS_ACCESS_KEY_ID"]
 SECRET_KEY   = os.environ["AWS_SECRET_ACCESS_KEY"]
 s3_host = S3_ENDPOINT.removeprefix("https://").removeprefix("http://")
 # --- boto3 client (listing only, zero egress) ---
 boto = boto3.client(
    "s3",
    endpoint_url=S3_ENDPOINT,
    aws_access_key_id=ACCESS_KEY,
    aws_secret_access_key=SECRET_KEY,
 )
 # --- s3fs filesystem (footer-only reads via pyarrow) ---
 fs = s3fs.S3FileSystem(
    client_kwargs={"endpoint_url": S3_ENDPOINT},
    key=ACCESS_KEY,
    secret=SECRET_KEY,
 )
 # ------------------------------------------------------------------ #
 # Phase 1: File inventory via S3 List API (zero data egress)
 # ------------------------------------------------------------------ #
 print("Phase 1: listing S3 objects...")
 paginator = boto.get_paginator("list_objects_v2")
 inventory = {}  # "dataset/table" -> {files: [...], total_size: int}
 for page in paginator.paginate(Bucket=S3_BUCKET):
    for obj in page.get("Contents", []):
        key = obj["Key"]
        if not key.endswith(".parquet"):
            continue
        parts = key.split("/")
        if len(parts) < 3:
            continue
        dataset, table = parts[0], parts[1]
        dt = f"{dataset}/{table}"
        if dt not in inventory:
            inventory[dt] = {"files": [], "total_size_bytes": 0}
        inventory[dt]["files"].append(key)
        inventory[dt]["total_size_bytes"] += obj["Size"]
 print(f"  Found {len(inventory)} tables across {S3_BUCKET}")
 # ------------------------------------------------------------------ #
 # Phase 2: Schema reads — footer only (~30 KB per table)
 # ------------------------------------------------------------------ #
 print("Phase 2: reading parquet footers...")
 def fmt_size(b):
    for unit in ("B", "KB", "MB", "GB", "TB"):
        if b < 1024 or unit == "TB":
            return f"{b:.1f} {unit}"
        b /= 1024
 def extract_col_descriptions(schema):
    """Try to pull per-column descriptions from Arrow metadata."""
    descriptions = {}
    meta = schema.metadata or {}
    # BigQuery exports embed a JSON blob under b'pandas' with column_info
    pandas_meta_raw = meta.get(b"pandas") or meta.get(b"pandas_metadata")
    if pandas_meta_raw:
        try:
            pm = json.loads(pandas_meta_raw)
            for col in pm.get("columns", []):
                name = col.get("name")
                desc = col.get("metadata", {}) or {}
                if isinstance(desc, dict) and "description" in desc:
                    descriptions[name] = desc["description"]
        except Exception:
            pass
    # Also try top-level b'description' or b'schema'
    for key in (b"description", b"schema", b"BigQuery:description"):
        val = meta.get(key)
        if val:
            try:
                descriptions["__table__"] = val.decode("utf-8", errors="replace")
            except Exception:
                pass
    return descriptions
 schemas = {}
 errors = []
 for i, (dt, info) in enumerate(sorted(inventory.items())):
    dataset, table = dt.split("/", 1)
    first_file = info["files"][0]
    s3_path = f"{S3_BUCKET}/{first_file}"
    try:
        schema = pq.read_schema(fs.open(s3_path))
        col_descs = extract_col_descriptions(schema)
        # Build raw metadata dict (decode bytes keys/values)
        raw_meta = {}
        if schema.metadata:
            for k, v in schema.metadata.items():
                try:
                    dk = k.decode("utf-8", errors="replace")
                    dv = v.decode("utf-8", errors="replace")
                    # Try to parse JSON values
                    try:
                        dv = json.loads(dv)
                    except Exception:
                        pass
                    raw_meta[dk] = dv
                except Exception:
                    pass
        columns = []
        for field in schema:
            col = {
                "name": field.name,
                "type": str(field.type),
                "nullable": field.nullable,
            }
            if field.name in col_descs:
                col["description"] = col_descs[field.name]
            # Check field-level metadata
            if field.metadata:
                for k, v in field.metadata.items():
                    try:
                        dk = k.decode("utf-8", errors="replace")
                        dv = v.decode("utf-8", errors="replace")
                        if dk in ("description", "DESCRIPTION", "comment"):
                            col["description"] = dv
                    except Exception:
                        pass
            columns.append(col)
        schemas[f"{dataset}.{table}"] = {
            "path": f"s3://{S3_BUCKET}/{dataset}/{table}/",
            "file_count": len(info["files"]),
            "total_size_bytes": info["total_size_bytes"],
            "total_size_human": fmt_size(info["total_size_bytes"]),
            "columns": columns,
            "metadata": raw_meta,
        }
        print(f"  [{i+1}/{len(inventory)}] ✓ {dataset}.{table} ({len(columns)} cols, {fmt_size(info['total_size_bytes'])})")
    except Exception as e:
        errors.append({"table": f"{dataset}.{table}", "error": str(e)})
        print(f"  [{i+1}/{len(inventory)}] ✗ {dataset}.{table}: {e}", file=sys.stderr)
 # ------------------------------------------------------------------ #
 # Phase 3: Enrich from br_bd_metadados.bigquery_tables (small table)
 # ------------------------------------------------------------------ #
 META_TABLE = "br_bd_metadados.bigquery_tables"
 meta_dt = "br_bd_metadados/bigquery_tables"
 if meta_dt in inventory:
    print(f"Phase 3: enriching from {META_TABLE}...")
    try:
        con = duckdb.connect()
        con.execute("INSTALL httpfs; LOAD httpfs;")
        con.execute(f"""
            SET s3_endpoint='{s3_host}';
            SET s3_access_key_id='{ACCESS_KEY}';
            SET s3_secret_access_key='{SECRET_KEY}';
            SET s3_url_style='path';
        """)
        meta_path = f"s3://{S3_BUCKET}/br_bd_metadados/bigquery_tables/*.parquet"
        # Peek at available columns
        available = [r[0] for r in con.execute(f"DESCRIBE SELECT * FROM '{meta_path}' LIMIT 1").fetchall()]
        print(f"  Metadata columns: {available}")
        # Try to find dataset/table description columns
        desc_col = next((c for c in available if "description" in c.lower()), None)
        ds_col   = next((c for c in available if c.lower() in ("dataset_id", "dataset", "schema_name")), None)
        tbl_col  = next((c for c in available if c.lower() in ("table_id", "table_name", "table")), None)
        if desc_col and ds_col and tbl_col:
            rows = con.execute(f"""
                SELECT {ds_col}, {tbl_col}, {desc_col}
                FROM '{meta_path}'
            """).fetchall()
            for ds, tbl, desc in rows:
                key = f"{ds}.{tbl}"
                if key in schemas and desc:
                    schemas[key]["table_description"] = desc
            print(f"  Enriched {len(rows)} table descriptions")
        else:
            print(f"  Could not find expected columns (dataset_id, table_id, description) — skipping enrichment")
        con.close()
    except Exception as e:
        print(f"  Enrichment failed: {e}", file=sys.stderr)
 else:
    print("Phase 3: br_bd_metadados.bigquery_tables not in S3 — skipping enrichment")
 # ------------------------------------------------------------------ #
 # Phase 4a: Write schemas.json
 # ------------------------------------------------------------------ #
 print("Phase 4: writing outputs...")
 output = {
    "_meta": {
        "bucket": S3_BUCKET,
        "total_tables": len(schemas),
        "total_size_bytes": sum(v["total_size_bytes"] for v in schemas.values()),
        "total_size_human": fmt_size(sum(v["total_size_bytes"] for v in schemas.values())),
        "errors": errors,
    },
    "tables": dict(sorted(schemas.items())),
 }
 with open("schemas.json", "w", encoding="utf-8") as f:
    json.dump(output, f, ensure_ascii=False, indent=2)
 print(f"  ✓ schemas.json ({len(schemas)} tables)")
 # ------------------------------------------------------------------ #
 # Phase 4b: Write file_tree.md
 # ------------------------------------------------------------------ #
 lines = [
    f"# S3 File Tree: {S3_BUCKET}",
    "",
 ]
 # Group by dataset
 datasets_map = {}
 for dt_key, info in sorted(inventory.items()):
    dataset, table = dt_key.split("/", 1)
    datasets_map.setdefault(dataset, []).append((table, info))
 total_files  = sum(len(v["files"]) for v in inventory.values())
 total_bytes  = sum(v["total_size_bytes"] for v in inventory.values())
 for dataset, tables in sorted(datasets_map.items()):
    ds_bytes = sum(i["total_size_bytes"] for _, i in tables)
    ds_files = sum(len(i["files"]) for _, i in tables)
    lines.append(f"## {dataset}/  ({len(tables)} tables, {fmt_size(ds_bytes)}, {ds_files} files)")
    lines.append("")
    for table, info in sorted(tables):
        schema_entry = schemas.get(f"{dataset}.{table}", {})
        ncols = len(schema_entry.get("columns", []))
        col_str = f", {ncols} cols" if ncols else ""
        table_desc = schema_entry.get("table_description", "")
        desc_str = f" — {table_desc}" if table_desc else ""
        lines.append(f"  - **{table}/**  ({len(info['files'])} files, {fmt_size(info['total_size_bytes'])}{col_str}){desc_str}")
    lines.append("")
 lines += [
    "---",
    f"**Total: {len(inventory)} tables · {fmt_size(total_bytes)} · {total_files} parquet files**",
 ]
 with open("file_tree.md", "w", encoding="utf-8") as f:
    f.write("\n".join(lines) + "\n")
 print(f"  ✓ file_tree.md ({len(inventory)} tables)")
 print()
 print("Done!")
 print(f"  schemas.json  — full column-level schema dump")
 print(f"  file_tree.md  — bucket tree with sizes")
 if errors:
    print(f"  {len(errors)} tables failed (see schemas.json _meta.errors)")
--- a/haloy.yml
+++ b/haloy.yml
@@ -0,0 +1,25 @@
 name: basedosdados
 server: haloy.xn--2dk.xyz
 api_token:
  from:
    env: HALOY_TOKEN
 domains:
  - domain: db.xn--2dk.xyz
 port: 8080
 health_check_path: /health
 env:
  - name: HETZNER_S3_ENDPOINT
    from:
      env: HETZNER_S3_ENDPOINT
  - name: AWS_ACCESS_KEY_ID
    from:
      env: AWS_ACCESS_KEY_ID
  - name: AWS_SECRET_ACCESS_KEY
    from:
      env: AWS_SECRET_ACCESS_KEY
  - name: BASIC_AUTH_PASSWORD
    from:
      env: BASIC_AUTH_PASSWORD
  - name: BUCKET_REGION
    from:
      env: BUCKET_REGION
--- a/prepara_db.py
+++ b/prepara_db.py
@@ -0,0 +1,69 @@
 import os
 import duckdb
 import boto3
 from collections import defaultdict
 from dotenv import load_dotenv
 load_dotenv()
 BUCKET       = os.environ['HETZNER_S3_BUCKET']
 ENDPOINT_URL = os.environ['HETZNER_S3_ENDPOINT']
 ACCESS_KEY   = os.environ['AWS_ACCESS_KEY_ID']
 SECRET_KEY   = os.environ['AWS_SECRET_ACCESS_KEY']
 # DuckDB expects the endpoint without scheme
 s3_endpoint = ENDPOINT_URL.removeprefix('https://').removeprefix('http://')
 # Lista todos os objetos do bucket de uma vez, agrupando por dataset/tabela
 s3 = boto3.client('s3',
                  endpoint_url=ENDPOINT_URL,
                  aws_access_key_id=ACCESS_KEY,
                  aws_secret_access_key=SECRET_KEY)
 paginator = s3.get_paginator('list_objects_v2')
 table_files = defaultdict(lambda: defaultdict(list))
 for page in paginator.paginate(Bucket=BUCKET):
    for obj in page.get('Contents', []):
        key = obj['Key']
        if not key.endswith('.parquet'):
            continue
        parts = key.split('/')
        if len(parts) >= 3:
            dataset, table = parts[0], parts[1]
            table_files[dataset][table].append(f"s3://{BUCKET}/{key}")
 # Cria conexão DuckDB e configura S3
 con = duckdb.connect('basedosdados.duckdb')
 con.execute("INSTALL httpfs; LOAD httpfs;")
 con.execute(f"""
    SET s3_endpoint='{s3_endpoint}';
    SET s3_access_key_id='{ACCESS_KEY}';
    SET s3_secret_access_key='{SECRET_KEY}';
    SET s3_url_style='path';
    SET enable_object_cache=true;
    SET threads=4;
    SET memory_limit='6GB';
    SET preserve_insertion_order=false;
    SET http_keep_alive=true;
    SET http_retries=3;
 """)
 # Cria schemas e views com lista explícita de arquivos
 for dataset, tables in table_files.items():
    con.execute(f"CREATE SCHEMA IF NOT EXISTS {dataset}")
    for table, files in tables.items():
        file_list = ", ".join(f"'{f}'" for f in sorted(files))
        try:
            con.execute(f"""
                CREATE OR REPLACE VIEW {dataset}.{table} AS
                SELECT * FROM read_parquet([{file_list}], hive_partitioning=true, union_by_name=true)
            """)
            print(f"✓ {dataset}.{table} ({len(files)} files)")
        except Exception as e:
            if 'Geoparquet' in str(e) or 'geometria' in str(e) or 'geometry' in str(e).lower():
                print(f"  skip (geoparquet) {dataset}.{table}")
            else:
                raise
 con.close()
 print("Done! Open with: duckdb --ui basedosdados.duckdb")
--- a/prepara_gui.py
+++ b/prepara_gui.py
@@ -1,62 +0,0 @@
 import os
 import duckdb
 import boto3
 from dotenv import load_dotenv
 load_dotenv()
 S3_ENDPOINT = os.environ["HETZNER_S3_ENDPOINT"]          # https://hel1.your-objectstorage.com
 S3_BUCKET   = os.environ["HETZNER_S3_BUCKET"]            # baseldosdados
 ACCESS_KEY  = os.environ["AWS_ACCESS_KEY_ID"]
 SECRET_KEY  = os.environ["AWS_SECRET_ACCESS_KEY"]
 # Strip protocol for DuckDB httpfs (expects bare hostname)
 s3_host = S3_ENDPOINT.removeprefix("https://").removeprefix("http://")
 con = duckdb.connect('basedosdados.duckdb')
 con.execute("INSTALL httpfs; LOAD httpfs;")
 con.execute(f"""
    CREATE OR REPLACE PERSISTENT SECRET hetzner (
        TYPE S3,
        KEY_ID '{ACCESS_KEY}',
        SECRET '{SECRET_KEY}',
        ENDPOINT '{s3_host}',
        URL_STYLE 'path'
    );
 """)
 # List all dataset/table prefixes in the bucket
 s3 = boto3.client(
    's3',
    endpoint_url=S3_ENDPOINT,
    aws_access_key_id=ACCESS_KEY,
    aws_secret_access_key=SECRET_KEY,
 )
 paginator = s3.get_paginator('list_objects_v2')
 datasets = {}
 for page in paginator.paginate(Bucket=S3_BUCKET, Delimiter='/'):
    for prefix in page.get('CommonPrefixes', []):
        dataset = prefix['Prefix'].rstrip('/')
        datasets[dataset] = []
        for page2 in paginator.paginate(Bucket=S3_BUCKET,
                                        Prefix=dataset + '/',
                                        Delimiter='/'):
            for p in page2.get('CommonPrefixes', []):
                table = p['Prefix'].rstrip('/').split('/')[-1]
                datasets[dataset].append(table)
 # Create schemas and views
 for dataset, tables in datasets.items():
    con.execute(f"CREATE SCHEMA IF NOT EXISTS {dataset}")
    for table in tables:
        path = f"s3://{S3_BUCKET}/{dataset}/{table}/*.parquet"
        con.execute(f"""
            CREATE OR REPLACE VIEW {dataset}.{table} AS
            SELECT * FROM '{path}'
        """)
        print(f"✓ {dataset}.{table}")
 con.close()
 print("Done! Open with: duckdb --ui basedosdados.duckdb")
--- a/requirements.txt
+++ b/requirements.txt
@@ -0,0 +1,3 @@
 duckdb
 boto3
 python-dotenv
--- a/schemas.json
+++ b/schemas.json
--- a/start.sh
+++ b/start.sh
@@ -0,0 +1,28 @@
 #!/bin/bash
 set -euo pipefail
 S3_ENDPOINT="${HETZNER_S3_ENDPOINT#https://}"
 S3_ENDPOINT="${S3_ENDPOINT#http://}"
 # Init SQL para o terminal web (credenciais não ficam expostas como env vars)
 cat > /app/ssh_init.sql <<SQL
 LOAD httpfs;
 SET s3_endpoint='${S3_ENDPOINT}';
 SET s3_access_key_id='${AWS_ACCESS_KEY_ID}';
 SET s3_secret_access_key='${AWS_SECRET_ACCESS_KEY}';
 SET s3_region='${BUCKET_REGION}';
 SET s3_url_style='path';
 SET enable_object_cache=true;
 SET threads=4;
 SET memory_limit='4GB';
 SQL
 chmod 600 /app/ssh_init.sql
 echo "[start] Starting ttyd terminal..."
 ttyd --port 7681 --writable duckdb -readonly --init /app/ssh_init.sql /app/basedosdados.duckdb &
 echo "[start] Starting auth service..."
 python3 /app/auth.py &
 echo "[start] Starting Caddy..."
 exec caddy run --config /app/Caddyfile --adapter caddyfile
Author	SHA1	Message	Date
rafapolo	ed81e52254	docs: reorder README for data users, remove unused files (xdg-open, gera_schemas.py, open_gui.sh, docs/)	2026-03-26 12:01:46 +01:00
rafapolo	5239a03ea8	docs: expand /query curl usage, remove outdated UI references	2026-03-26 11:58:14 +01:00
rafapolo	41e7f7a972	replace duckdb-ui with ttyd shell: add /query HTTP endpoint, fix utf-8/locale, region config - swap DuckDB UI for ttyd web terminal (--writable, -readonly db) - add POST /query endpoint with X-Password auth for curl-based SQL execution - fix UTF-8 rendering: set LANG/LC_ALL=C.UTF-8 in container - pass BUCKET_REGION env var for correct S3 signing region - simplify start.sh: drop Xvfb, views.duckdb generation, blocking duckdb -ui - add less, ncurses-bin to Dockerfile for proper pager/terminal support - update Caddyfile: single route to ttyd with flush_interval -1 for websocket - update README to reflect current architecture and document /query usage - remove duckdb-ui.service, schemas.json, file_tree.md (generated artifacts)	2026-03-26 11:54:46 +01:00
rafapolo	cd94603fac	update haloy config: use hostname for server, fix env var format	2026-03-25 13:39:15 +01:00
rafapolo	8c22944bbb	fix duckdb filename, add db file and gitignore cleanup - Dockerfile + start.sh: use basedosdados.duckdb (not basedosdados3.duckdb) - add basedosdados.duckdb (3.5 MB, needed for Docker build) - add requirements.txt (local dev use) - .gitignore: remove *.duckdb exclusion, add .DS_Store	2026-03-25 13:30:25 +01:00
rafapolo	0d77f83045	simplify container: skip db prep, password via env var, fixed server IP - start.sh: remove prepara_db.py step; load S3 creds via DuckDB init file - Caddyfile: switch to basic_auth with {env.BASIC_AUTH_HASH} — no rebuild to rotate password - Dockerfile: drop Python/pip layers (no longer needed at runtime) - haloy.yml: set server to 89.167.95.136, add BASIC_AUTH_HASH to env - remove requirements.txt (only needed for local prepara_db.py, not the container)	2026-03-25 13:27:51 +01:00
rafapolo	9eb2dee013	containerize with Haloy: Dockerfile, Caddy basicauth, haloy.yml for db.xn--2dk.xyz - Dockerfile: debian slim, installs DuckDB CLI, Python deps, Caddy - start.sh: runs prepara_db.py → starts Caddy (basicauth) → starts DuckDB UI - Caddyfile: updated for container (no TLS, port 8080, Haloy handles HTTPS) - haloy.yml: deploys to db.xn--2dk.xyz on port 8080 - requirements.txt: duckdb, boto3, python-dotenv - prepara_db.py, open_gui.sh, duckdb-ui.service: add previously untracked files - remove prepara_gui.py (replaced by prepara_db.py)	2026-03-25 13:23:59 +01:00