Reading Google Play and App Store reviews straight from their JSON, no browser

Saudações, malta do **webmastersmz.com**!

Como especialista em tecnologia, analisei com muito interesse o tópico **"Reading Google Play and App Store reviews straight from their JSON, no browser"**. Esta é uma abordagem extremamente relevante para desenvolvedores, webmasters e analistas de dados que precisam de monitorizar o feedback dos utilizadores sem o *overhead* (consumo excessivo de recursos) que as ferramentas tradicionais de automação trazem.

Abaixo, apresento a minha análise técnica sobre os pontos principais desta técnica e como ela se aplica ao nosso contexto de desenvolvimento.

---

### Análise Técnica: JSON Direto vs. Scraping com Browser (Headless)

Tradicionalmente, para extrair dados da Google Play Store ou da Apple App Store, muitos desenvolvedores recorrem a browsers *headless* (como Puppeteer, Selenium ou Playwright). Embora eficazes, estas ferramentas simulam um navegador completo, o que consome muita memória RAM e CPU — algo crítico quando rodamos scripts em VPSs com recursos limitados aqui em Moçambique.

A abordagem de ler diretamente dos endpoints de JSON/API (sem browser) traz vantagens brutais:

#### 1. Performance Extrema e Baixo Consumo de Recursos
Ao ignorar a renderização de HTML, CSS e a execução de scripts pesados das lojas, fazemos apenas requisições HTTP diretas (usando `fetch`, `axios` ou `cURL`). O payload retornado é puro JSON. A velocidade de resposta aumenta drasticamente e o consumo de memória cai para quase zero.

#### 2. Facilidade no Parseamento de Dados
Trabalhar com JSON é o "pão nosso de cada dia". Em vez de estarmos a lutar com seletores CSS complexos e instáveis (`div > span > class="xyz"`), que as lojas mudam constantemente para evitar *scraping*, o JSON entrega-nos uma estrutura de dados limpa, tipada e direta (ex: `review.author`, `review.rating`, `review.text`).

#### 3. Como funciona nos bastidores?
*   **Apple App Store:** A Apple facilita bastante este processo. Ela disponibiliza feeds RSS/JSON públicos para as avaliações de cada aplicação (basta formatar o URL correto com o ID da app e o código do país, por exemplo, `mz` para Moçambique).
*   **Google Play Store:** O ecossistema da Google é mais fechado. Não há um feed público simples. Para obter o JSON direto sem browser, é necessário intercetar as chamadas internas de API (`POST` requests para os endpoints de RPC da Google Play) ou utilizar bibliotecas que já fazem engenharia reversa destas chamadas (como o pacote `google-play-scraper` em Node.js).

#### Os Desafios (O reverso da medalha)
Nem tudo são flores. Ao usar endpoints de JSON não documentados (especialmente na Google Play), corremos o risco de:
*   **Bloqueios de IP (Rate Limiting):** Requisições em massa sem um comportamento "humano" são facilmente detetadas. O uso de proxies ou rotação de IPs torna-se obrigatório para grandes volumes de dados.
*   **Mudanças repentinas nas APIs internas:** Como não são APIs públicas oficiais, a Google ou a Apple podem alterar a estrutura do JSON ou os parâmetros de autenticação sem aviso prévio, partindo o nosso código.

---

### Vamos debater no Fórum webmastersmz.com!

Esta técnica abre portas para criarmos ferramentas locais incríveis, como dashboards de monitoria de apps moçambicanas (bancos, operadoras, serviços públicos) ou alertas automáticos no Telegram/Slack sempre que uma nova avaliação negativa for publicada.

Gostaria de lançar o debate para a nossa comunidade:
1. **Já tentaram extrair dados de reviews das lojas? Que abordagem usaram?**
2. **Para projetos em Moçambique, acham que o uso de APIs não oficiais compensa o risco de manutenção constante, ou preferem pagar por APIs de terceiros (como a SerpApi)?**
3. **Quem aqui já desenvolveu um script em Node.js ou Python para este propósito? Partilhem os vossos pedaços de código (*snippets*) connosco!**

Participem no debate e vamos elevar o nível do desenvolvimento web no nosso país!

---

Para garantir que os vossos projetos e fóruns rodam sem falhas, convido-vos a conhecer as soluções de alojamento de alta performance da AplicHost em https://aplichost.com.

Reading Google Play and App Store reviews straight from their JSON, no browser



Tópico: Reading Google Play and App Store reviews straight from their JSON, no browser
Categoria: Tutoriais | Programação & Tecnologia
Idioma Principal: Português (Conteúdo de Tecnologia)

Descrição do Conteúdo / Informações:
-------------------------------------------------------------------------
I spent an afternoon last year fighting a headless Chrome that scraped Play Store reviews. CAPTCHA, then a proxy bill, then a week later the DOM shifted and the whole thing returned empty arrays. Threw it out.

Turns out I never needed the browser. Both Google Play and the Apple App Store serve reviews as plain JSON you can hit with an HTTP request. No login, no proxies, no Playwright. This is the request shapes, the pagination caps that aren't in any docs, and the one place Google's format bit me.

All Node.js, all on got-scraping — a drop-in got replacement that copies a real browser's TLS and header fingerprint. That fingerprint earns its keep. The identical request from stock axios or fetch would sometimes come back 403 while got-scraping walked right through, because Play is fingerprinting the TLS handshake, not reading your User-Agent.



Apple first, because Apple made it easy


Apple publishes reviews as an RSS feed in JSON. One endpoint:

https://itunes.apple.com/{country}/rss/customerreviews/page={1-10}/id={appId}/sortby={mostrecent|mosthelpful}/json

•
country — a storefront code (us, gb, de...). Every storefront keeps its own reviews.

•
appId — the numeric id from the store URL: apps.apple.com/us/app/whatsapp-messenger/id310633997.

•
page — 1 to 10, and 10 is the wall. Fifty reviews a page, so ~500 per storefront. Coming back to that.

import { gotScraping } from 'got-scraping';

async function fetchAppleReviews(appId, { country = 'us', maxReviews = 200 } = {}) {
const out = [];
const pages = Math.min(10, Math.ceil(maxReviews / 50));
for (let page = 1; page <= pages; page++) {
const url = `https://itunes.apple.com/${country}/rss/customerreviews/page=${page}/id=${appId}/sortby=mostrecent/json`;
const res = await gotScraping({ url, responseType: 'json' });
const entries = res.body?.feed?.entry ?? [];
// The first entry is sometimes app metadata, not a review — guard on im:rating.
for (const e of entries) {
if (!e['im:rating']) continue;
out.push({
id: e.id.label,
author: e.author.name.label,
rating: Number(e['im:rating'].label),
title: e.title.label,
text: e.content.label,
version: e['im:version'].label,
date: e.updated.label,
});
}
if (!entries.length) break;
}
return out.slice(0, maxReviews);
}

That 500-review cap is hard. No continuation token gets you past it — I looked. What does work: reviews are scoped per storefront, so pull us, then gb, au, ca, de, and the rest of the English-language storefronts, deduping on review id. There are about ten of them; at ~500 each that's roughly 5,000 recent reviews, which is usually plenty.

App metadata — title, average rating, total count, current version — lives at a second, simpler endpoint:

https://itunes.apple.com/lookup?id={appId}&country=us



Google Play, and the batchexecute rabbit hole


Play has no clean REST endpoint. The store front-end talks to an internal RPC called batchexecute, and the payload is ugly and documented nowhere. The payoff for climbing through it: Play paginates with no ceiling. Apple caps you at 500 a storefront; Play just keeps going.

The endpoint:

POST https://play.google.com/_/PlayStoreUi/data/batchexecute?hl=en&gl=us
Content-Type: application/x-www-form-urlencoded;charset=UTF-8

The body is a URL-encoded f.req parameter wrapping the RPC id UsvDTd and its arguments:

function buildBody(pkg, { count = 100, token = null, sort = 2 } = {}) {
const tok = token ? `\\"${token}\\"` : 'null';
const inner = `[null,null,[2,${sort},[${count},null,${tok}],null,[]],[\\"${pkg}\\",7]]`;
const freq = `[[["UsvDTd","${inner}",null,"generic"]]]`;
return 'f.req=' + encodeURIComponent(freq);
}

sort is 2 for newest, 1 for relevance, 3 for rating. pkg is the package name (com.whatsapp). token is the continuation cursor the previous response handed back.

The response is where it gets weird. It opens with an anti-JSON-hijacking guard, the literal )]}', and then a nested envelope where the real data sits as a JSON string inside the outer JSON. You parse twice:

function parse(raw) {
const envelope = JSON.parse(raw.slice(raw.indexOf('[')));   // strip )]}'
const inner = envelope?.[0]?.[2];                            // a JSON *string*
if (!inner) return { reviews: [], nextToken: null };
const data = JSON.parse(inner);
const reviews = (data[0] ?? []).map((r) => ({
id: r[0],
author: r[1][0],
rating: r[2],
text: r[4],
date: new Date(r[5][0] * 1000).toISOString(),
thumbsUp: r[6],
reply: r[7]?.[1] ?? null,     // developer reply text
appVersion: r[10],
}));
const nextToken = data[1]?.[1] ?? null;
return { reviews, nextToken };
}

Then loop, feeding nextToken back until you've got enough or it comes back null:

async function fetchPlayReviews(pkg, { maxReviews = 200 } = {}) {
const out = [];
let token = null;
while (out.length < maxReviews) {
const res = await gotScraping({
url: 'https://play.google.com/_/PlayStoreUi/data/batchexecute?hl=en&gl=us',
method: 'POST',
body: buildBody(pkg, { count: 150, token }),
headers: { 'content-type': 'application/x-www-form-urlencoded;charset=UTF-8' },
});
const { reviews, nextToken } = parse(res.body);
if (!reviews.length) break;
out.push(...reviews);
if (!nextToken) break;
token = nextToken;
}
return out.slice(0, maxReviews);
}

A few things I only learned by running this against real apps:

• Play reviews carry no title. Just a rating and a body. Apple gives you both. If you're merging the two stores into one schema, title has to be nullable or you'll drop half your Play data on a strict validator.

• Developer replies hide in slot [7] on Play — text at [7][1], timestamp at [7][2][0]. Apple's public feed doesn't surface replies at all.

• The loop fires requests back to back with no delay. Proxyless, from my laptop and from a datacenter, I haven't been rate-limited doing this — but it's the assumption most likely to break at tens of thousands of reviews, and I'd put a throttle in front of it before trusting it at that scale.

And here's where it got me. I first pulled appVersion from slot [8], eyeballed one response, saw a version string, shipped it. Some apps came back with a country code there instead. The version is [10]. The index parsing is brittle by design — Google can reshuffle slots whenever they like, and nothing tells you. So I pinned a test against a known app with a review I can eyeball and assert the fields on it. When Google moves something, that test screams before my users notice.



Is this actually worth skipping the browser?


For me, yes — and the number that convinced me: 250 Play reviews land in about 0.6 seconds this way. You're reading the exact API the store's own frontend reads, so a store redesign doesn't touch you. My Playwright version was 10 to 20 times slower, wanted a proxy budget the moment I scaled it, and died on the next UI refresh. I don't miss it.



If you'd rather not babysit the slot indices


I bundled both stores into one actor on Apify — one schema, handles the pagination and the batchexecute envelope, runs proxyless: App Reviews Scraper. It's mostly there so I stop re-fixing the [10]-versus-[8] kind of thing every quarter. But the code above is the whole trick, and rolling your own is very doable.

The batchexecute envelope eats afternoons if you go in blind — if you get stuck on it, leave a comment and I'll dig in.