Building high-perf image processing pipeline to create vernacular catalogs

Preview: Catalog for Hindi and English created on run-time.

Why an async pipeline for processing images?

A good rule of thumb is to avoid api requests which run longer than 300ms. ~ Experience

Setting high throughput message broker

Redis Push and Pop operation on Google’s n1-standard-2.

Workers pool

  1. Having a lock to avoid executing the same job in multiple workers.
  2. Able to handle and store failed jobs to be able to triage.
  3. Able to prioritize jobs based on the message.

Monitoring and Dashboard

Dashboard showing the state of workers and job

Implementation with Python and Redis

Architecture diagram in GCP.
{
"id": 1,
"style": {
"fill_color": "white",
"stroke_color": "black",
"x": 512,
"y": 900
},
"title": {
"hindi": "जोकर",
"english": "Joker"
},
"fonts": {
"hindi": "NotoSans-Bold.ttf",
"english": "NotoSans-Bold.ttf"
}
}
Image’s title showing Hindi and English script based on the above mentioned config.

Sample Project on Docker

Other use-cases for Image processing pipeline

  1. Adaptive resolution: Adjusting image size and resolution for a better experience across screen size.
  2. Annotations: Adding badges, icons over the image to grab attention.
  3. Creative Filter: Treating image to blend and make the image visually appealing
Example of image processing in Native Ads and OTT thumbnail

Further Reading

  1. Redis’s Push/Pop operation: https://redis.io/commands/rpush
  2. RQ: https://github.com/Parallels/rq-dashboard
  3. RQ-Dashboard: https://github.com/Parallels/rq-dashboard
  4. Wand.py, ctypes-based simple ImageMagick binding for Python.

--

--

--

Code, arts, process and aspirations. co-Founded GreedyGame | IIT Ropar. Found at www.arinkverma.in

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Open Source Low-Code Platforms: Changing The Landscape Of Coding

Building a Docker image

AWS RDS Aurora, the Good, the Bad and the Ugly

How to use git blame effectively

How to Build a Development Team for an IT Project

How To Upgrade Elasticsearch, Kibana, Logstash and Filebeat On Kubernetes With ECK

How I Got Started As A Software Developer

A cubicle farm of customer service agents.

An elegant pattern to craft cache-efficient functions in Swift

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Arink Verma

Arink Verma

Code, arts, process and aspirations. co-Founded GreedyGame | IIT Ropar. Found at www.arinkverma.in

More from Medium

IoT Tech Stack and its Uses in IoT Solutions 3STechLabs

Bot framework composer error — ONNX Multilingual model failed to load, Bot is failing Failed to…

How to Train a Machine Learning Model as a REST API and Build a Spam Classifier (Part 1)

Banner image of laptop threatened by a spammer.

Kafka Instances, Made Simple: Introducing KafMira