I use BigDataLite for a lot of my sandboxing work. This is a OVA provided by Oracle which can be run on VirtualBox, VMWare, etc and has the Cloudera Hadoop platform (CDH) along with all of Oracle’s Big Data goodies including Big Data Discovery and Big Data Spatial and Graph (BDSG). Something that kept tripping me up during my work with BDSG was that HBase would become unavailable. Not being an HBase expert and simply using it as a data store for my property graph data, I wrote it off as mistakes on my part.

Continue Reading

Using the holt function in Timelion to do anomaly detection on Metricbeat data in Kibana: Expression: $thres=0.02, .es(index='metricbeat*',metric='max:system.cpu.user.pct').lines(1).if(eq, 0, null).holt(0.9, 0.1, 0.9, 0.5h).color(#eee).lines(10).label('Prediction'), .es(index='metricbeat*',metric='max:system.cpu.user.pct').color(#666).lines(1).label(Actual), .es(index='metricbeat*',metric='max:system.cpu.user.pct').lines(1).if(eq, 0, null).holt(0.9, 0.1, 0.9, 0.5h).subtract(.es(index='metricbeat*',metric='max:system.cpu.user.pct')).abs().if(lt, $thres, null, .es(index='metricbeat*',metric='max:system.cpu.user.pct')).points(10,3,0).color(#c66).label('Anomaly').title('max:system.cpu.user.pct / @rmoff') References: https://twitter.com/rashidkpc/status/762754396111327232 https://github.com/elastic/timelion/issues/87 https://github.com/elastic/timelion/blob/master/FUNCTIONS.md

Continue Reading

The world beyond batch: Streaming 101 The world beyond batch: Streaming 102 Data architectures for streaming applications SE-Radio Episode 272: Frances Perry on Apache Beam (img credit)

Continue Reading

By default, the kafka-avro-console-producer will assume that the schema registry is on port 8081, and happily connect to it. Unfortunately, this can lead to some weird errors if another process happens to be listening on port 8081 already! [oracle@bigdatalite tmp]$ kafka-avro-console-producer \ > --broker-list localhost:9092 --topic kudu_test \ > --property value.schema='{"type":"record","name":"myrecord","fields":[{"name":"id","type":"int"},{"name":"random_field", "type": "string"}]}' {"id": 999, "random_field": "foo"} org.apache.kafka.common.errors.SerializationException: Error registering Avro schema: {"type":"record","name":"myrecord","fields":[{"name":"id","type":"int"},{"name":"random_field","type":"string"}]} Caused by: io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null') at [Source: sun.

Continue Reading

tl;dr Make sure that key.converter.schema.registry.url and value.converter.schema.registry.url are specified, and that there are no trailing whitespaces. I’ve been building on previous work I’ve done with Oracle GoldenGate and Kafka Connect, looking at how to have the change records from the Oracle database come through to Kafka in Avro format rather than the default JSON that the sample configuration gives. Simply changing the Kafka Connect OGG configuration file (confluent.properties) from value.converter=org.apache.kafka.connect.json.JsonConverter key.

Continue Reading

I hit this error running Kafka Connect HDFS connector from Confluent Platform v3.1.1 on BigDataLite 4.6: [oracle@bigdatalite ~]$ connect-standalone /etc/schema-registry/connect-avro-standalone.properties /etc/kafka-connect-hdfs/quickstart-hdfs.properties [...] Exception in thread "main" java.lang.IncompatibleClassChangeError: Implementing class at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) The fix was to unset the CLASSPATH first: unset CLASSPATH

Continue Reading

Presented without comment, warranty, or context - other than these might help a wandering code hacker. When using SigV4, you must specify a ‘host’ parameter boto.s3.connection.HostRequiredError: BotoClientError: When using SigV4, you must specify a 'host' parameter. To fix, switch conn_s3 = boto.connect_s3() for conn_s3 = boto.connect_s3(host='s3.amazonaws.com') You can see a list of endpoints here. boto.exception.S3ResponseError: S3ResponseError: 400 Bad Request Make sure you’re specifying the correct hostname (see above) for the bucket’s region.

Continue Reading

Similar to the previous issue, the sample config in the docs causes another snafu: OGG-15051 Java or JNI exception: oracle.goldengate.util.GGException: Class not found: "kafkahandler". kafkahandler Class not found: "kafkahandler". kafkahandler This time it’s in the kafka.props file: gg.handler.kafkahandler.Type = kafka Should be gg.handler.kafkahandler.type = kafka No capital T in Type! (Image credit: https://unsplash.com/@vanschneider)

Continue Reading

In the documentation for the current release of Oracle GoldenGate for Big Data (12.2.0.1.1.011) there’s a helpful sample configuration, which isn’t so helpful … [...] gg.handler.kafkahandler.ProducerRecordClass = com.company.kafka.CustomProducerRecord [...] This value for gg.handler.kafkahandler.ProducerRecordClass will cause a failure when you start the replicat: [...] Class not found: "com.company.kafka.CustomProducerRecord" [...] If you comment this configuration item out, it’ll use the default (oracle.goldengate.handler.kafka.DefaultProducerRecord) and work swimingly! (Image credit: https://unsplash.com/@vanschneider)

Continue Reading

There are various reasons for this error, but the one I hit was that the table name is case sensitive, and returned from Oracle by the JDBC driver in uppercase. If you specify the tablename in your connecter config in lowercase, it won’t be matched, and this error is thrown. You can validate this by setting debug logging (edit etc/kafka/connect-log4j.properties to set log4j.rootLogger=DEBUG, stdout), and observe: (I’ve truncated some of the output for legibility)

Continue Reading

I’ve been doing some noodling around with Confluent’s Kafka Connect recently, as part of gaining a wider understanding into Kafka. If you’re not familiar with Kafka Connect this page gives a good idea of the thinking behind it. One issue that I hit defeated my Google-fu so I’m recording it here to hopefully help out fellow n00bs. The pipeline that I’d set up looked like this: Eneco’s Twitter Source streaming tweets to a Kafka topic Confluent’s HDFS Sink to stream tweets to HDFS and define Hive table automagically over them It worked great, but only if I didn’t enable the Hive integration part.

Continue Reading

With my new server I bought a UPS, partly just as a Good Thing, but also because I suspect a powercut fried the motherboard on a previous machine that I had, and this baby is too precious to lose ;) The idea is that the UPS will smooth out the power supply to my server, protecting it from surges or temporarily blips in power loss. If there’s a proper power cut, the UPS is connected to my server and can initiate a graceful shutdown instead of system crash.

Continue Reading

Trying to use SparkSQL to read a JSON file, from either pyspark or spark-shell, I got this error: java.io.IOException: No input paths specified in job scala> sqlContext.read.json("/u02/custom/twitter/twitter.json") java.io.IOException: No input paths specified in job at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:202) Despite the reference articles that I found using this local path syntax (/u02/custom/twitter/twitter.json), it turned out that I needed to prefix it with file://: scala> sqlContext.read.json("file:///u02/custom/twitter/twitter.json") res3: org.apache.spark.sql.DataFrame = [@timestamp: string, @version: string, contributors: string, coordinates: string, created_at: string, entities: struct<hashtags:array<struct<indices:array<bigint>,text:string>>,media:array<struct<display_url:string,expanded_url:string,id:bigint,id_str:string,indices:array<bigint>,media_url:string,media_url_https:string,sizes:struct<large:struct<h:bigint,resize:string,w:bigint>,medium:struct<h:bigint,resize:string,w:bigint>,small:struct<h:bigint,resize:string,w:bigint>,thumb:struct<h:bigint,resize:string,w:bigint>>,source_status_id:bigint,source_status_id_str:string,source_user_id:bigint,source_user_id_str:string,type:string,url:string>>,symbols:array<struct<indices:array<bigint>,text:string>>,urls:array<struct<display_url:string,expanded_url:string.

Continue Reading

TL;DR When defining networking on Proxmox 4 LXC containers, use an appropriate CIDR suffix (e.g. 24) - don’t use 32! On my Proxmox 4 server I’m running a whole load of lovely LXC containers. Unfortunately, I had trouble connecting to them. From a client machine, I got the error ssh_exchange_identification: read: Connection reset by peer On the server I was connecting to (which I could get a console for through the Proxmox GUI, or a session on using pct enter from the Proxmox host) I ran a SSHD process with debug to see what was happening:

Continue Reading

(Ref) The bit that caught me out was this kept failing with Error: Password not present and a Python stack trace that ended with subprocess.CalledProcessError: Command '/var/run/cloudera-scm-agent/process/78-hue-HUE_SERVER/altscript.sh sec-1-secret_key' returned non-zero exit status 1 The answer (it seems) is to ensure that HUE_SECRET_KEY is set (to any value!) Launch shell: export HUE_SECRET_KEY=foobar /opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hue/build/env/bin/hue shell Reset password for hue, activate account and make it superuser from django.contrib.auth.models import User user = User.objects.get(username='hue') user.

Continue Reading

Vanilla download of Apache Drill 1.6, attempting to follow the Followed the Drill in 10 Minutes tutorial - but kept just getting the error No current connection. Here’s an example: [oracle@bigdatalite apache-drill-1.6.0]$ ./bin/drill-embedded Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M; support was removed in 8.0 com.fasterxml.jackson.databind.JavaType.isReferenceType()Z apache drill 1.6.0 "the only truly happy people are children, the creative minority and drill users" 0: jdbc:drill:zk=local> SELECT version FROM sys.version; No current connection 0: jdbc:drill:zk=local> Whether SELECT version FROM sys.

Continue Reading

I wasted literally two hours on this one, so putting down a note to hopefully help future Googlers. Symptom Here’s all the various errors that I got in the hive-server2.log during my attempts to get a CREATE EXTERNABLE TABLE to work against a MongoDB table in Hive: Caused by: java.lang.ClassNotFoundException: com.mongodb.hadoop.io.BSONWritable Caused by: java.lang.ClassNotFoundException: com.mongodb.util.JSON Caused by: java.lang.ClassNotFoundException: org.bson.conversions.Bson Caused by: java.lang.ClassNotFoundException: org.bson.io.OutputBuffer Whilst Hive would throw errors along the lines of:

Continue Reading

Installing CDH 5.7 on Linux Containers (LXC) hosted on Proxmox 4. Everything was going well until Cluster Setup, and which point it failed on Start YARN (MR2 included) Completed only 0/1 steps. First failure: Failed to execute command Start on service YARN (MR2 Included) Log /var/log/hadoop-yarn/hadoop-cmf-yarn-NODEMANAGER-cdh57-01-node-02.moffatt.me.log.out showed: org.apache.hadoop.service.AbstractService: Service containers-monitor failed in state INITED; cause: java.lang.NumberFormatException: For input string: "18446744073709550364" java.lang.NumberFormatException: For input string: "18446744073709550364" Looking down the stack trace, this came from org.

Continue Reading

I’ve got an EdgeRouter LITE (ERL) which I used as my home router until a powercut fried it a while ago (looks like I’m not the only one to have this issue). The symptoms were it powering on but not giving any DHCP addresses, or after a factory reset responding on the default IP of 192.168.1.1. It was a real shame, because it had been a great bit of kit up until then.

Continue Reading

(Previously, previously, previously) Since Proxmox 4 has a recent Linux kernel and mainline one at that, it means that Docker can be run on it. I’ve yet to really dig into Docker and work out when it makes sense in place of Linux Containers (LXC), so this is going to be a learning experience for me. To install Docker, add Backports repo to apt: root@proxmox01:~# cat /etc/apt/sources.list.d/backports.list deb http://ftp.debian.org/debian jessie-backports main And then install:

Continue Reading