Millions of products. Two gigabytes of memory.
Product Information Management
- Company
- Tyroo Technologies
- My project role
- Collaborator
- Project lead & creator
- Siddharth Juyal
- Employment period
- Feb 2023 — Present
Built by
Siddharth Juyal.
Shout-out to Siddharth Juyal for leading and building the PIM ingestion rebuild. I collaborated on this project at Tyroo, and the credit for the rebuild belongs to him.
- Project lead & creator
- Siddharth Juyal
- My role
- CollaboratorVaibhav Lehra
From 16 GB to 2 GB: How We Rebuilt Our Product Ingestion Pipeline and Cut Infra Cost by ~85%
The story of the rebuild, told by the engineer who built it.
Read Siddharth’s articleSiddharth Juyal built this PIM ingestion rebuild at Tyroo; I was a collaborator. The two-pass Polars pipeline processes multi-million-product catalogs within 2 GB of RAM, with field-level parity checks. This study explains his rebuild and the related team work.
- Python
- Polars
- SQS
- MySQL
- Redis
- S3
7.2M products processed on 2 vCPU / 2 GB in the replacement ingestion pilot.
Both 4.1 GB / 2.8M-product and 6.2 GB / 7.2M-product feeds processed within the supplied resource limit.
Parity corpus tested alongside real feeds and corrupted encodings; field-level results were byte-identical.
Separate worker improvement using long polling and cached clients / queue URLs; not an ingestion speedup.
These are reported project and team results. The ingestion rebuild was led and built by Siddharth Juyal; my role was collaborator. The figures describe the individual comparisons below.
Follow the system.
Explore the constraint, the decision, how it was checked, and the result. The full narrative follows below.
The catalog outgrew the memory model.
The previous ingestion engine held whole feeds in memory at roughly 15× the file size. It exhausted 6.5 GB in seconds, leaving an S3 chunk-splitting workaround between the feed and processing.
- Whole-feed allocation at approximately 15× file size
- Target envelope: 2 vCPU / 2 GB
- Keep field-level output compatible
- Large product feed
- Whole-feed allocation
- Memory exhaustion
Data systems
Tyroo TechnologiesSiddharth’s project. A collaboration at Tyroo.
Siddharth Juyal led and built the PIM ingestion rebuild. I was a collaborator on the project, and the credit for the rebuild belongs to him.
The engineering story below describes the project’s constraints, approach, and reported results. Siddharth’s own article is linked above for his account of the work.
The problem was allocation, not just feed size.
PIM ingests multi-million-product catalog feeds in Tyroo’s product-data platform. This study follows the ingestion rebuild built by Siddharth Juyal, which I participated in as a collaborator.
The previous engine’s whole-feed approach used roughly 15× the input file size in memory and exhausted 6.5 GB in seconds. An S3 chunk-splitting workaround supported that constraint. The replacement needed to change the resource profile while preserving the data produced by the existing engine.
- 4.1 GB feed containing 2.8M products
- 6.2 GB feed containing 7.2M products
- Replacement pilot resource envelope: 2 vCPU / 2 GB
Streaming with a separate correctness check.
Siddharth’s replacement uses a two-pass Polars streaming DataFrame pipeline at flat memory. It processed the documented feeds within 2 GB of RAM and removed the need for the S3 chunk-splitting workaround.
A field-level parity harness compared the existing and replacement engines before the proposed cutover. The project’s checks covered real feeds, corrupted encodings, and a 19-case hostile edge corpus, producing byte-identical results.
- Two-pass streaming ingestion
- Field-level comparisons between both engines
- Real-data and hostile-input validation
The worker path needed attention too.
Related team work improved event-driven SQS workers. Long polling and cached clients / queue URLs roughly doubled per-worker throughput.
The related service work also included stronger MySQL and Redis reconnect handling and a Python 3.9 to 3.11 migration. These project changes address worker efficiency, connection recovery, and the runtime independently of the ingestion memory result.
- SQS long polling
- Cached clients and queue URLs
- MySQL / Redis reconnect handling
- Python 3.9 → 3.11
A correctness-led pilot.
Siddharth’s ingestion rebuild pairs a smaller resource envelope with explicit output-compatibility checks. Worker throughput, reconnect handling, and the runtime migration form a separate set of team improvements to the event-driven processing path.
Have a similar systems challenge?
Let’s talk