AWS DevOps Engineer (Data Cataloguing)

About AI TalentFlow:

AI TalentFlow is not just about filling roles; we're about building the future of data-driven businesses. We accelerate startup growth, scale enterprise solutions, and nurture the careers of the brilliant minds who make it all happen. Our success is your success, and we are committed to empowering our clients with exceptional talent in AI, Data, Analytics, Product, and Agile.

The Opportunity:

We are seeking an AWS DevOps Engineer specializing in data cataloging to join our Data Engineering team. In this role, you will play a pivotal role in building and maintaining our AWS cloud infrastructure, with a strong focus on designing and implementing data cataloging solutions. You will collaborate closely with data engineers, data scientists, and other stakeholders to ensure efficient data discovery, governance, and accessibility.

Responsibilities:

• Data Cataloging: Design, develop, and maintain a comprehensive data cataloging system on AWS, utilizing tools such as AWS Glue Data Catalog, AWS Lake Formation, or similar technologies.
• Metadata Management: Establish and enforce metadata management standards, ensuring data quality, accuracy, and consistency across the organization.
• Data Discovery: Implement intuitive data discovery mechanisms to empower users to find and understand relevant data assets easily.
• Data Governance: Collaborate with data governance teams to enforce data access controls, security policies, and compliance requirements.
• AWS Infrastructure: Design and manage scalable AWS cloud infrastructure to support data cataloging operations, including EC2, S3, Lambda, and other relevant services.
• Automation: Automate data cataloging processes using tools like AWS Step Functions, AWS Lambda, or custom scripts to enhance efficiency and reduce manual effort.
• Monitoring and Troubleshooting: Monitor data cataloging system health, performance, and security, promptly addressing any issues or anomalies.
• Documentation: Create and maintain comprehensive documentation for data cataloging processes, infrastructure, and best practices.

Qualifications:

• AWS Expertise: Proven experience with AWS cloud services, particularly those related to data cataloging, data lakes, and data warehouses (e.g., Glue, Lake Formation, Redshift).
• DevOps Experience: Strong background in DevOps practices, including CI/CD pipelines, infrastructure as code (IaC), and automation using tools like CloudFormation or Terraform.
• Data Cataloging Knowledge: Familiarity with data cataloging concepts, metadata management, and data governance principles.
• Programming Skills: Proficiency in Python or other scripting languages for automation and data manipulation tasks.
• Collaboration: Excellent communication and interpersonal skills to work effectively with cross-functional teams.