StokeBloke.com

Archive for the ‘Programming’ Category

The number of messages in a Kafka Topic

Friday, November 11th, 2016

This is how you find out how many messages exist for a kafka topic.


/opt/kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list kafka:9092 --topic test --time -2

This gives output like this

test:1:28179212
test:0:28180921


/opt/kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list kafka:9092 --topic test --time -1

This gives output like this

test:1:28228989
test:0:28230734

Simply subtract these values and you get the sizes.
0:28230734 – 28180921 = 49813
1:28228989 – 28179212 = 49777

Netbeans log code template

Tuesday, May 3rd, 2011

For some years I have been adding this code template to Netbeans.

private static final Logger log = Logger.getLogger(${classVar editable="false" currClassName default="getClass()"}.class.getName());

You can add this by going to Netbeans->Tools Menu->Options->Editor->Code Templates dialog.

Netbeans Code Templates

Edit @ 29/08/2011
In newer Netbeans you must ensure that the “On Template Expansion:” is set to “Reformat text” else it gets extra new lines added in the code template.

${no-format}private static final Logger log = Logger.getLogger(${classVar editable="false" currClassName default="getClass()"}.class.getName());

JVMTI IterateOverReachableObjects and very poor documentation.

Wednesday, March 25th, 2009

As part of my daily work I’ve been working with Steve to write a heap parser so that our tests can find any memory leaks without the need to attach a profiler and hunt though the results.

We created a jvmti agent which could tag an object on the heap then find out if the object was reachable via any GC roots.

This worked well and we had no problems for over a year but recently we started to get some strange array index out of bounds.

download interview

download joe the king online

The cause was the very poor documentation of the jvmtiObjectReferenceCallback call where the field information was being calculated.

The documentation was wrong in 1.5 but even in 1.6 it simply does not provide enough information to write a heap walker which can print the field names.

I.e for the JVMTI_REFERENCE_FIELD reference kind it says the following:

hamlet 2 dvd

Reference from an object to the value of one of its instance fields. For references of this kind the referrer_index parameter to the jvmtiObjectReferenceCallback is the index of the the instance field. The index is based on the order of all the object’s fields. This includes all fields of the directly declared static and instance fields in the class, and includes all fields (both public and private) fields declared in superclasses and superinterfaces. The index is thus calculated by summing the index of field in the directly declared class (see GetClassFields), with the total number of fields (both public and private) declared in all superclasses and superinterfaces. The index starts at zero.

This may seem ok but how do you parse the super classes and super interfaces?  The documentation states nothing about the order of the fields so do you scan all interfaces first or last, do you walk the super classes once for classes and once for interfaces?

Being very vague (even google didnt know much) it was difficult to get the output, when a leak was detected, to show the correct field.

It would randomly output fields which were the wrong type and many other issues.

After days of trial and error I was about to give up, then I found hprof_reference.c via google, and then found the rest of the hprof source as part of the JDK directory.

back to gaya download

After looking though this code it was possible to figure out exactly how the class and interface heirachy should be processed.

How are we expected to use the JVMTI when its so poorly documented?

Hang in Java FileSystemView getFileSystemView() getSystemIcon()

Wednesday, March 18th, 2009

Recently I was working on a deadlock in our startup code.

redline online It took some time to track it down as it never appeared as a deadlock (via CTRL-Break) it just stopped running with 0 CPU usage.

nim s island online The cause was

boarding gate download

FileSystemView.getFileSystemView().getSystemIcon(file)

It appears that this call fails when called from multiple threads asking for the same system icon.

Below is an example on how to show the issue. Its was tested with Java 1.6.0_07

    public void testGetSystemIcon() throws IOException,
                                           InterruptedException {
        final FileImageThread t1 = new FileImageThread(".jpg");
        final FileImageThread t2 = new FileImageThread(".jpg");
        t1.start();
        t2.start();
        t1.join();
        t2.join();
    }
 
    public class FileImageThread extends Thread {
        final File file;
 
        public FileImageThread(final String suffix) throws IOException {
            file = File.createTempFile("test", suffix);
        }
 
        @Override
        public void run() {
            FileSystemView.getFileSystemView().getSystemIcon(file);
        }
    }
 <em style="display:none"></em>

It does not hang every time, but it happens enough to be an issue.

The hang call stack is

download sleeping murder movie

Thread  id=22 name=Thread-4 inState=WAITING deadlocked=false isNative=false
  sun.misc.Unsafe.park(Native Method)
  java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
  java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
  java.util.concurrent.FutureTask.get(FutureTask.java:83)
  sun.awt.shell.Win32ShellFolderManager2$ComInvoker.invoke(Win32ShellFolderManager2.java:495)
  sun.awt.shell.Win32ShellFolder2.(Win32ShellFolder2.java:225)
  sun.awt.shell.Win32ShellFolderManager2.getDrives(Win32ShellFolderManager2.java:101)
  sun.awt.shell.Win32ShellFolderManager2.isFileSystemRoot(Win32ShellFolderManager2.java:323)
  sun.awt.shell.ShellFolder.isFileSystemRoot(ShellFolder.java:234)
  javax.swing.filechooser.FileSystemView.isFileSystemRoot(FileSystemView.java:310)
  com.osm.ui.FileImageUtilsTest$FileImageThread.run(FileImageUtilsTest.java:62)

Thread  id=21 name=Thread-3 inState=WAITING deadlocked=false isNative=false
  sun.misc.Unsafe.park(Native Method)
  java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
  java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
  java.util.concurrent.FutureTask.get(FutureTask.java:83)
  sun.awt.shell.Win32ShellFolderManager2$ComInvoker.invoke(Win32ShellFolderManager2.java:495)
  sun.awt.shell.Win32ShellFolder2.(Win32ShellFolder2.java:225)
  sun.awt.shell.Win32ShellFolderManager2.getDrives(Win32ShellFolderManager2.java:101)
  sun.awt.shell.Win32ShellFolderManager2.isFileSystemRoot(Win32ShellFolderManager2.java:323)
  sun.awt.shell.ShellFolder.isFileSystemRoot(ShellFolder.java:234)
  javax.swing.filechooser.FileSystemView.isFileSystemRoot(FileSystemView.java:310)
  com.osm.ui.FileImageUtilsTest$FileImageThread.run(FileImageUtilsTest.java:62)

No one else seemed to see this issue, but I saw it lots of times on my PC.

emacs p4 edit integration

Thursday, March 12th, 2009

I have been using perforce for some time and I wrote this little function to perform a p4 edit when I press the F9 key.

lost boys the tribe online

(defun p4-edit ()
  "does a p4 edit on the current file"
  (interactive)
  (let* (
         (filename (buffer-file-name))
         (cmd)
         (result)
         )
    (setf cmd (concat "p4 " "edit " filename))
    (setf result (shell-command-to-string cmd))
    (message (substring result 0 (- (length result) 1)))
    ;; reload the file
    (revert-buffer t t t)
    )
  ) 
 
(global-set-key [f9] 'p4-edit)

I know there are lots of other perforce integrations to emacs, but I wrote this and tried to keep it small and simple.

download ricochet dvd

neverwas movie

the astronaut farmer online