Responsibilities
- Consume processing tasks provided by the queue
- Currently supports document processing tasks
Key Points
- Shares the same code base as playground API
- Does not provide an API
- It has separate entry point and docker image
- Can scale horizontally and vertically to process more tasks
- The number of processes per instance depends on CPU cores
- Any changes to playground processing logic will affect worker logic
- Worker creates a separate build folder without redundant code from the src directory
- Worker checks queue on fixed interval
- Worker is a NON-API version of the playground API
Required Configuration
- Same as playground backend
Management
- See playground make file for worker build & push
- CICD pipelines trigger on merging to development branch
- Package JSON scripts:
- Worker:start
- Worker:build
- Worker:dev
Future Improvements
- Support audio processing tasks
- Auto scaling – allows API to auto spawn worker instances according to queue load and shut them down once completed
- Auto scaling – OPTIONAL - allow workers to auto shutdown when no tasks are pending in queue