Big data engineer (10+ Years)
Company: Object Technology Solutions, Inc.
Posted on: November 10, 2018
Role: Big data engineerLocation: Englewood, CO 80112Duration: 12+ MonthsAs a Big Data Engineer, you will provide engineering knowledge to create and enhance data solutions enabling seamless delivery of data across our enterprise. You will be on the cutting edge of finding and integrating new technology and tools for data centric projects. Additionally, you will provide technical consulting to peer data engineers during design and development of highly complex and critical data projects. Some of these projects will include designing and developing data ingestion and processing/transformation frameworks leveraging open source tools such as NiFi, EMR, Java, Scala, Spark APIs, AWS Glue, etc. Additionally, you will work on real time processing solutions using tools such as Spark Streaming, MQ, Kafka, and AWS Kinesis. You will deploy application code using CI/CD tools and techniques.Responsibilities
- Develop data driven solutions utilizing current and next generation technologies to meet evolving business needs.
- Ability to quickly identify an opportunity and recommend possible technical solutions.
- Utilize multiple development languages/tools such as Python, SPARK, Hive to build prototypes and evaluate results for effectiveness and feasibility.
- Operationalize data ingestion and data-analytic tools for enterprise use.
- Utilize tools available to you across AWS Services
- Develop real-time data ingestion and stream-analytic solutions leveraging technologies such as Kafka, Apache Spark, NIFI, Python, HBase and Hadoop.
- Custom Data pipeline development (Cloud and locally hosted)
- Work heavily within AWS and Hadoop ecosystems
- Provide support for deployed data applications and analytical models by being a trusted advisor to Data Scientists and other data consumers by identifying data problems and guiding issue resolution with partner Data Engineers and source data providers.
- Provide subject matter expertise in the analysis, preparation of specifications and plans for the development of data processes.
- Ensure proper data governance policies are followed by implementing or validating Data Lineage, Quality checks, classification, etcRequirements
- Bachelor's in computer science or related field is required (masters preferred)
- You have a minimum of 7 years of experience in the design, development, and deployment of large-scale, distributed, and cloud-deployed software services.
- Expected to be an expert in SQL and RDBMS. Good at modelling data for relational, analytical and big data workloads
- You have a minimum of 3-4 years of experience in Big Data software development technologies (e.g., Hadoop, Hive, Spark, Kafka) and exposure to resource/cluster management technologies.
- Minimum of 2-3 years of experience with AWS (e.g., EC2, S3, EMR, SNS, SQS, Aurora, Redshift).
- Experience with various software technologies/solutions and understand where to use them. 2 years of experience with Data Virtualization like Denodo - provided by Dice You have a minimum of 7 years of experience in the design, development, and deployment of large-scale, distributed, and cloud-deployed software services.
Keywords: Object Technology Solutions, Inc., Denver , Big data engineer (10+ Years), Engineering , Englewood, Colorado
Didn't find what you're looking for? Search again!