In the documentation for the current release of Oracle GoldenGate for Big Data (12.2.0.1.1.011) there’s a helpful sample configuration, which isn’t so helpful … [...] gg.handler.kafkahandler.ProducerRecordClass = com.company.kafka.CustomProducerRecord [...] This value for gg.handler.kafkahandler.ProducerRecordClass will cause a failure when you start the replicat: [...] Class not found: "com.company.kafka.CustomProducerRecord" [...] If you comment this configuration item out, it’ll use the default (oracle.goldengate.handler.kafka.DefaultProducerRecord) and work swimingly! (Image credit: https://unsplash.com/@vanschneider)

There are various reasons for this error, but the one I hit was that the table name is case sensitive, and returned from Oracle by the JDBC driver in uppercase. If you specify the tablename in your connecter config in lowercase, it won’t be matched, and this error is thrown. You can validate this by setting debug logging (edit etc/kafka/connect-log4j.properties to set log4j.rootLogger=DEBUG, stdout), and observe: (I’ve truncated some of the output for legibility)

I’ve been doing some noodling around with Confluent’s Kafka Connect recently, as part of gaining a wider understanding into Kafka. If you’re not familiar with Kafka Connect this page gives a good idea of the thinking behind it. One issue that I hit defeated my Google-fu so I’m recording it here to hopefully help out fellow n00bs. The pipeline that I’d set up looked like this: Eneco’s Twitter Source streaming tweets to a Kafka topic Confluent’s HDFS Sink to stream tweets to HDFS and define Hive table automagically over them It worked great, but only if I didn’t enable the Hive integration part.

With my new server I bought a UPS, partly just as a Good Thing, but also because I suspect a powercut fried the motherboard on a previous machine that I had, and this baby is too precious to lose ;) The idea is that the UPS will smooth out the power supply to my server, protecting it from surges or temporarily blips in power loss. If there’s a proper power cut, the UPS is connected to my server and can initiate a graceful shutdown instead of system crash.

Trying to use SparkSQL to read a JSON file, from either pyspark or spark-shell, I got this error: java.io.IOException: No input paths specified in job scala> sqlContext.read.json("/u02/custom/twitter/twitter.json") java.io.IOException: No input paths specified in job at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:202) Despite the reference articles that I found using this local path syntax (/u02/custom/twitter/twitter.json), it turned out that I needed to prefix it with file://: scala> sqlContext.read.json("file:///u02/custom/twitter/twitter.json") res3: org.apache.spark.sql.DataFrame = [@timestamp: string, @version: string, contributors: string, coordinates: string, created_at: string, entities: struct<hashtags:array<struct<indices:array<bigint>,text:string>>,media:array<struct<display_url:string,expanded_url:string,id:bigint,id_str:string,indices:array<bigint>,media_url:string,media_url_https:string,sizes:struct<large:struct<h:bigint,resize:string,w:bigint>,medium:struct<h:bigint,resize:string,w:bigint>,small:struct<h:bigint,resize:string,w:bigint>,thumb:struct<h:bigint,resize:string,w:bigint>>,source_status_id:bigint,source_status_id_str:string,source_user_id:bigint,source_user_id_str:string,type:string,url:string>>,symbols:array<struct<indices:array<bigint>,text:string>>,urls:array<struct<display_url:string,expanded_url:string.

TL;DR When defining networking on Proxmox 4 LXC containers, use an appropriate CIDR suffix (e.g. 24) - don’t use 32! On my Proxmox 4 server I’m running a whole load of lovely LXC containers. Unfortunately, I had trouble connecting to them. From a client machine, I got the error ssh_exchange_identification: read: Connection reset by peer On the server I was connecting to (which I could get a console for through the Proxmox GUI, or a session on using pct enter from the Proxmox host) I ran a SSHD process with debug to see what was happening:

(Ref) The bit that caught me out was this kept failing with Error: Password not present and a Python stack trace that ended with subprocess.CalledProcessError: Command '/var/run/cloudera-scm-agent/process/78-hue-HUE_SERVER/altscript.sh sec-1-secret_key' returned non-zero exit status 1 The answer (it seems) is to ensure that HUE_SECRET_KEY is set (to any value!) Launch shell: export HUE_SECRET_KEY=foobar /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/bin/hue shell Reset password for hue, activate account and make it superuser from django.contrib.auth.models import User user = User.objects.get(username='hue') user.

Vanilla download of Apache Drill 1.6, attempting to follow the Followed the Drill in 10 Minutes tutorial - but kept just getting the error No current connection. Here’s an example: [oracle@bigdatalite apache-drill-1.6.0]$ ./bin/drill-embedded Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 com.fasterxml.jackson.databind.JavaType.isReferenceType()Z apache drill 1.6.0 "the only truly happy people are children, the creative minority and drill users" 0: jdbc:drill:zk=local> SELECT version FROM sys.version; No current connection 0: jdbc:drill:zk=local> Whether SELECT version FROM sys.

I wasted literally two hours on this one, so putting down a note to hopefully help future Googlers. Symptom Here’s all the various errors that I got in the hive-server2.log during my attempts to get a CREATE EXTERNABLE TABLE to work against a MongoDB table in Hive: Caused by: java.lang.ClassNotFoundException: com.mongodb.hadoop.io.BSONWritable Caused by: java.lang.ClassNotFoundException: com.mongodb.util.JSON Caused by: java.lang.ClassNotFoundException: org.bson.conversions.Bson Caused by: java.lang.ClassNotFoundException: org.bson.io.OutputBuffer Whilst Hive would throw errors along the lines of:

Installing CDH 5.7 on Linux Containers (LXC) hosted on Proxmox 4. Everything was going well until Cluster Setup, and which point it failed on Start YARN (MR2 included) Completed only 0/1 steps. First failure: Failed to execute command Start on service YARN (MR2 Included) Log /var/log/hadoop-yarn/hadoop-cmf-yarn-NODEMANAGER-cdh57-01-node-02.moffatt.me.log.out showed: org.apache.hadoop.service.AbstractService: Service containers-monitor failed in state INITED; cause: java.lang.NumberFormatException: For input string: "18446744073709550364" java.lang.NumberFormatException: For input string: "18446744073709550364" Looking down the stack trace, this came from org.