StokeBloke.com

This is how you find out how many messages exist for a kafka topic.


/opt/kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list kafka:9092 --topic test --time -2


This gives output like this

test:1:28179212
test:0:28180921



/opt/kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list kafka:9092 --topic test --time -1


This gives output like this

test:1:28228989
test:0:28230734


Simply subtract these values and you get the sizes.
0:28230734 – 28180921 = 49813
1:28228989 – 28179212 = 49777

For some years I have been adding this code template to Netbeans.

private static final Logger log = Logger.getLogger(${classVar editable="false" currClassName default="getClass()"}.class.getName());

You can add this by going to Netbeans->Tools Menu->Options->Editor->Code Templates dialog.

Edit @ 29/08/2011
In newer Netbeans you must ensure that the “On Template Expansion:” is set to “Reformat text” else it gets extra new lines added in the code template.

${no-format}private static final Logger log = Logger.getLogger(${classVar editable="false" currClassName default="getClass()"}.class.getName());

As part of my daily work I’ve been working with Steve to write a heap parser so that our tests can find any memory leaks without the need to attach a profiler and hunt though the results.

We created a jvmti agent which could tag an object on the heap then find out if the object was reachable via any GC roots.

This worked well and we had no problems for over a year but recently we started to get some strange array index out of bounds.

The cause was the very poor documentation of the jvmtiObjectReferenceCallback call where the field information was being calculated.

The documentation was wrong in 1.5 but even in 1.6 it simply does not provide enough information to write a heap walker which can print the field names.

I.e for the JVMTI_REFERENCE_FIELD reference kind it says the following:

Reference from an object to the value of one of its instance fields. For references of this kind the referrer_index parameter to the jvmtiObjectReferenceCallback is the index of the the instance field. The index is based on the order of all the object’s fields. This includes all fields of the directly declared static and instance fields in the class, and includes all fields (both public and private) fields declared in superclasses and superinterfaces. The index is thus calculated by summing the index of field in the directly declared class (see GetClassFields), with the total number of fields (both public and private) declared in all superclasses and superinterfaces. The index starts at zero.

This may seem ok but how do you parse the super classes and super interfaces?  The documentation states nothing about the order of the fields so do you scan all interfaces first or last, do you walk the super classes once for classes and once for interfaces?

Being very vague (even google didnt know much) it was difficult to get the output, when a leak was detected, to show the correct field.

It would randomly output fields which were the wrong type and many other issues.

After days of trial and error I was about to give up, then I found hprof_reference.c via google, and then found the rest of the hprof source as part of the JDK directory.

After looking though this code it was possible to figure out exactly how the class and interface heirachy should be processed.

How are we expected to use the JVMTI when its so poorly documented?

Recently I was working on a deadlock in our startup code.

It took some time to track it down as it never appeared as a deadlock (via CTRL-Break) it just stopped running with 0 CPU usage.

The cause was

 FileSystemView.getFileSystemView().getSystemIcon(file)

It appears that this call fails when called from multiple threads asking for the same system icon.

Below is an example on how to show the issue. Its was tested with Java 1.6.0_07

    public void testGetSystemIcon() throws IOException,
                                           InterruptedException {
        final FileImageThread t1 = new FileImageThread(".jpg");
        final FileImageThread t2 = new FileImageThread(".jpg");
        t1.start();
        t2.start();
        t1.join();
        t2.join();
    }

    public class FileImageThread extends Thread {
        final File file;

        public FileImageThread(final String suffix) throws IOException {
            file = File.createTempFile("test", suffix);
        }

        @Override
        public void run() {
            FileSystemView.getFileSystemView().getSystemIcon(file);
        }
    }

It does not hang every time, but it happens enough to be an issue.

The hang call stack is

Thread  id=22 name=Thread-4 inState=WAITING deadlocked=false isNative=false
  sun.misc.Unsafe.park(Native Method)
  java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
  java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
  java.util.concurrent.FutureTask.get(FutureTask.java:83)
  sun.awt.shell.Win32ShellFolderManager2$ComInvoker.invoke(Win32ShellFolderManager2.java:495)
  sun.awt.shell.Win32ShellFolder2.(Win32ShellFolder2.java:225)
  sun.awt.shell.Win32ShellFolderManager2.getDrives(Win32ShellFolderManager2.java:101)
  sun.awt.shell.Win32ShellFolderManager2.isFileSystemRoot(Win32ShellFolderManager2.java:323)
  sun.awt.shell.ShellFolder.isFileSystemRoot(ShellFolder.java:234)
  javax.swing.filechooser.FileSystemView.isFileSystemRoot(FileSystemView.java:310)
  com.osm.ui.FileImageUtilsTest$FileImageThread.run(FileImageUtilsTest.java:62)

Thread  id=21 name=Thread-3 inState=WAITING deadlocked=false isNative=false
  sun.misc.Unsafe.park(Native Method)
  java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
  java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
  java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:905)
  java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1217)
  java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
  java.util.concurrent.FutureTask.get(FutureTask.java:83)
  sun.awt.shell.Win32ShellFolderManager2$ComInvoker.invoke(Win32ShellFolderManager2.java:495)
  sun.awt.shell.Win32ShellFolder2.(Win32ShellFolder2.java:225)
  sun.awt.shell.Win32ShellFolderManager2.getDrives(Win32ShellFolderManager2.java:101)
  sun.awt.shell.Win32ShellFolderManager2.isFileSystemRoot(Win32ShellFolderManager2.java:323)
  sun.awt.shell.ShellFolder.isFileSystemRoot(ShellFolder.java:234)
  javax.swing.filechooser.FileSystemView.isFileSystemRoot(FileSystemView.java:310)
  com.osm.ui.FileImageUtilsTest$FileImageThread.run(FileImageUtilsTest.java:62)

No one else seemed to see this issue, but I saw it lots of times on my PC.

I have been using perforce for some time and I wrote this little function to perform a p4 edit when I press the F9 key.

(defun p4-edit ()
  "does a p4 edit on the current file"
  (interactive)
  (let* (
         (filename (buffer-file-name))
         (cmd)
         (result)
         )
    (setf cmd (concat "p4 " "edit " filename))
    (setf result (shell-command-to-string cmd))
    (message (substring result 0 (- (length result) 1)))
    ;; reload the file
    (revert-buffer t t t)
    )
  ) 

(global-set-key [f9] 'p4-edit)

I know there are lots of other perforce integrations to emacs, but I wrote this and tried to keep it small and simple.