.png){kind=small}
Create Your First Project
Start adding your projects to your portfolio. Click on "Manage Projects" to get started
Big Data Processing for AgriTech Application
Project Type
Image feature extraction and classification
Tools
Python, PySpark, TensorFlow/Keras, PCA, t-SNE, AWS (EMR, EC2, S3)
Skills
Big Data processing, PySpark development, TensorFlow model broadcasting, Dimensionality reduction techniques, Cloud computing (AWS), Data visualization
Developed scalable data processing solutions for a mobile app aimed at identifying fruits using photos.
Designed and implemented PySpark scripts to efficiently process and prepare data for large-scale deployment in a cloud environment. Integrated a Convolutional Neural Network (CNN) using TensorFlow Keras for feature extraction and implemented model weight broadcasting to optimize distributed inference across multiple nodes.
Incorporated PCA for dimensionality reduction and t-SNE for visualization, assessing the effectiveness of feature extraction and its impact on classification.
The entire system was set up on AWS, utilizing EMR for big data processing, EC2 for computation, and S3 for data storage, ensuring compliance with GDPR.
