Select for update skip locked

In SQL, there is this possibility to lock the rows you select, warning this way the rest of the world that you may very well modify them.
Let's say you want to reserve the 10 first rows of the EMPLOYEES table of your HR schema.
In Oracle's SQL*Plus, this would look like this:

 SQL> select employee_id from employees where rownum <= 10 for update skip locked;

This would return a report like this:

The problem is that, when we issue the same command from another process or thread, no row is returned, even if unlocked rows are available in the table:

The reason for that is that the 10 first records are already locked, and identified by their rownum.

The idea is to get the first unlocked records, and then to lock them. Let's try the following PLSQL block:

set serveroutput on

 declare
   counter integer := 0;
   cursor emp_curs is
     select employee_id from employees for update skip locked;
   emp_row emp_curs%ROWTYPE;
   emp_id employees.employee_id%TYPE;
 begin
   open emp_curs;
   loop
     fetch emp_curs into emp_row;
     exit when emp_curs%NOTFOUND;
     counter := counter + 1;
     select employee_id into emp_id from employees where employee_id = emp_row.employee_id for update;
     dbms_output.put_line('-> Locked: ' || to_char(emp_id));
     if counter >= 10 then
       exit;
     end if;
   end loop;
 end;
 /

We just display the rows we want to lock.
We run it:

Records with EMPLOYEE_ID 100 to 109 are selected, and locked.
Now, before releasing the lock, let's run it from another process or thread:

That works, records with EMPLOYEE_ID 110 to 119 are selected.
Etc, etc.

In our case, we used EMPLOYEE_ID as the key for the records. We could have as well used the ROWID. It also uniquely identifies the records to lock, and this in any case. In that case, the script becomes:

declare
   counter integer := 0;
   cursor emp_curs is
     select e.*, rowid from employees e for update skip locked;
   emp_row emp_curs%ROWTYPE;
   emp_id employees.employee_id%TYPE;
 begin
   open emp_curs;
   loop
     fetch emp_curs into emp_row;
     exit when emp_curs%NOTFOUND;
     counter := counter + 1;
     select employee_id into emp_id from employees where rowid = emp_row.rowid for update;
     dbms_output.put_line('-> Locked: ' || to_char(emp_id));
     if counter >= 10 then
       exit;
     end if;
   end loop;
 end;
 /

It is even possible to use dynamic SQL to do this job (this is on another schema, as you can tell):

set serveroutput on size 3000
 set ver off
 set timing on
 accept nbrows2lock prompt 'Max Nb of Rows to Lock > '
 -- +---------------------------------------------------
 declare
   maxReturnedRows integer   := &nbrows2lock;
   table_name varchar2(128)  := 'QUEUE_2100';
   vch_tsolabel varchar2(64) := 'MANAGE_USER'; 

   counter integer             := 0;
   sql_stmt_One varchar2(4096) := '';
   vch_hint varchar2(128)      := '/*+ Rule */';
  
   tsoidvar varchar2(60);
   orderidvar varchar2(60);
   msisdnvar varchar2(60);
   imsivar varchar2(60);
   priorityvar number(4,0);
   instmsdt date;
   row_id rowid;

   type QueueCursorType is ref cursor;
   queue_curs_var QueueCursorType;     
  
   type recordType is record
   (
     row_id      rowid,
     tsoidvar    varchar2(60),
     orderidvar  varchar2(60),
     msisdnvar   varchar2(60),
     imsivar     varchar2(60),
     priorityvar number(4,0),
     instmsdt    date
   );
   recordHolder recordType;
   type tableOfRecord is table of recordType index by binary_integer;
   tableHolder tableOfRecord;
   i integer := 1; -- Index for tableOfRecord
 begin
   sql_stmt_One := 'SELECT ' || vch_hint || ' rowid, '                         ||
                                            ' t.tsoid, '                       ||
                                            ' nvl(t.orderid, '''') orderid, '  ||
                                            ' nvl(t.msisdn, '''')  msisdn, '   ||
                                            ' nvl(t.imsi, '''')    imsi, '     ||
                                            ' nvl(t.priority, 1)   priority, ' ||
                                            ' t.ins_tms  '                     ||
                   'FROM ' || table_name || ' t ' ||
                   'WHERE t.bp_flag = 0 and '     ||
                         't.tsolabel = :1 '       ||
                   'for update skip locked';
                  
   open queue_curs_var for sql_stmt_One using vch_tsolabel;
   loop
     -- --------------------------------------------------------------
     -- The trick is to have an 'exit when' before and after the fetch
     -- --------------------------------------------------------------
     exit when counter = maxReturnedRows;
     fetch queue_curs_var 
     into row_id,
          tsoidvar, 
          orderidvar,
          msisdnvar,
          imsivar,
          priorityvar, 
          instmsdt;
     exit when queue_curs_var%NOTFOUND;
     counter := counter + 1;
     dbms_output.put_line('Locking row #' || to_char(counter));
     --
     -- for tests. Concurrent locks...
     --
     if false then -- set to true to wait
       dbms_lock.sleep(1); -- Needs grant execute from SYS
     end if;
     --
     recordHolder.row_id      := row_id;
     recordHolder.tsoidvar    := tsoidvar;
     recordHolder.orderidvar  := orderidvar;
     recordHolder.msisdnvar   := msisdnvar;
     recordHolder.imsivar     := imsivar;
     recordHolder.priorityvar := priorityvar;
     recordHolder.instmsdt    := instmsdt;
     tableHolder(i) := recordHolder;
     i := i + 1;
   end loop;
   close queue_curs_var;
   dbms_output.put_line('Done with the array, ' || to_char(counter) || ' entry(ies)');
   if counter > 0 then
     -- Loop on the selected values to process them
     for i in tableHolder.FIRST..tableHolder.LAST loop
       dbms_output.put_line ('Processing [' || tableHolder(i).tsoidvar || ']');
       -- Processing takes place here
       execute immediate 
       'update ' || table_name || ' t ' ||
       'set t.bp_flag = 1 ' ||
       'where rowid = :1'
       using tableHolder(i).row_id;
     end loop;
   else
     dbms_output.put_line('No record to manage');
   end if;
 exception
   when others then
     dbms_output.put_line('ERR:' || sqlerrm);
 end;
 /
 show errors

We are actually using here the same kind of technique as above, having the SQL generated dynamically does not make much difference. We are using the ROWID approach. Notice the following details:

• The exit when statement, before and after the fetch of the first cursor
• The way we populate a table of records, so we can handle it after putting the lock
• The dbms_lock.sleep, for tests, that allows to simulate what happens if two or more such blocks are executed at the same time. We actually see that it makes no problem, no row can be locked more than once. It is interesting to understand that, for the first cursor (that has a for update clause), the lock is actually set when the record is fetched. This explains the exit when mechanism mentionned above.

This being said, this block behaves like the one described at the top of this post, it selects a given number of unlocked rows from a table, this selection having the possibility to be done simultaneously from several processes or threads that do not have to wait for any lock to be released before returning the expected result.

Hanoi Tower

The backend algorithm is just a few lines big..., it's all about rendering.
This browser does not support Applets. It could be a matter of authorization. Or install Java or use another browser...
The core code is pretty cool:

 public class BackendAlgorithm
 {
   public static void move(int n, String from, String to, String using)
   {
     if (n == 0)
       return;
     move(n-1, from, using, to);
     HanoiContext.getInstance().fireMoveRequired(from, to); // Broadcast the move
     move(n-1, using, to, from);
   }
 }

Again, it's mostly about rendering... 99% of the code of the applet above is for the graphical display!
The simplest user interface would look like this:

 package hanoitower;
 
 public class BackendAlgorithm
 {
   public static void move(int n, String from, String to, String using)
   {
     if (n == 0)
       return;
     move(n-1, from, using, to);
     System.out.println("Moving from " + from + " to " + to);
     move(n-1, using, to, from); 
   }
  
   public static void main(String[] args) throws Exception
   {
     move(Integer.parseInt(args[0]), "A", "C", "B"); // Moving A to C using B
   }
 }

Run it this way:

 Prompt> java hanoitower.BackendAlgorithm 4
 Moving from A to B
 Moving from A to C
 Moving from B to C
 Moving from A to B
 Moving from C to A
 Moving from C to B
 Moving from A to B
 Moving from A to C
 Moving from B to C
 Moving from B to A
 Moving from C to A
 Moving from B to C
 Moving from A to B
 Moving from A to C
 Moving from B to C

Boom!

Smooth moves with Swing

It's about data rendering in Swing. How to make a smooth rendering of real time data...
The secret seems to be in SwingUtilities.invokeAndWait

This browser does not support Applets.

Use the button to start or stop the applet

Above are two JPanels, displaying, on the left a speed, on the right a heading.
Both are fed by real time data (simulated in this case).
Obviously - and we will not show this code, unless anyone insists - the position of the display (namely the position of the hand) is done when the repaint() is invoked, and its implementation is done by overriding the paintComponent(Graphics) method.
The smoothing is done in a setValue(double) method.
When this method is invoked, the difference between the previous and the next values is splitted in small intervals, between which a repaint() is invoked.
All the difficulty - the problem - is that when a repaint() is invoked, maybe the previous one is not completed yet... This would result in a jerky display, far from being smooth, at the opposite of the expected result.
A solution is to tell Swing to wait, which happens to be shamefully simple... Here is the code for the speed display:

  public void setSpeed(final double d)
  {
    this.speed = d;
    double from = this.prevSpeed; 
    double to = d;
    
    int sign = (from>to)?-1:1;
    prevSpeed = d;
    // Smooth rotation
    for (double s=from; (sign==1 && s<=to) || (sign==-1 && s>=to); s+=(0.1*sign)) // 0.1 is the damping factor
    {
      final double _s = s;
      try
      {
        SwingUtilities.invokeAndWait(new Runnable()
          {
            public void run()
            {
              speed = _s;
              repaint();
            }
          });
      }
      catch (Exception ex)
      {
        ex.printStackTrace();
      }
    }
  }

The paintComponent(Graphics) method uses the speed member to position the hand on the display.
Notice the SwingUtilities.invokeAndWait method that takes a Runnable as a parameter. This Runnable is the one invoking the repaint() method. This way, Swing will do nothing until the repaint is completed, and it is not even necessary to worry about synchronizing anything...

See also SwingUtilities.invokeLater. It allows Swing to perform demanding tasks asynchronously.

Who just called me?

In Java, when you are in a specific method and you want to know who called you, who invoked the method, this is so simple!
A few lines will do the job:

  private void callee()
  {
    // Who called me
    Throwable t = new Throwable(); 
    StackTraceElement[] elements = t.getStackTrace(); 
    System.out.println("----------------------------------");
    for (StackTraceElement ste : elements)
      System.out.println(ste.toString());
    System.out.println("----------------------------------");
   
    // More stuff goes here... 
  }

Whenever the code above is invoked, like from a button click for example

  JButton bing = new JButton("Hit me!");
  bing.addActionListener(new ActionListener()
        {
          public void actionPerformed(ActionEvent e)
          {
            callee();
          }
        });

An output like that one can be expected:

----------------------------------
tideengineimplementation.gui.TideInternalFrame.resetData(TideInternalFrame.java:3035)
tideengineimplementation.gui.TideInternalFrame.refreshButton_actionPerformed(TideInternalFrame.java:3829)
tideengineimplementation.gui.TideInternalFrame.access$8200(TideInternalFrame.java:140)
tideengineimplementation.gui.TideInternalFrame$35.actionPerformed(TideInternalFrame.java:3002)
javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1995)
javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2318)
javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:387)
javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:242)
javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:272)
java.awt.Component.processMouseEvent(Component.java:6289)
javax.swing.JComponent.processMouseEvent(JComponent.java:3267)
java.awt.Component.processEvent(Component.java:6054)
java.awt.Container.processEvent(Container.java:2041)
java.awt.Component.dispatchEventImpl(Component.java:4652)
java.awt.Container.dispatchEventImpl(Container.java:2099)
java.awt.Component.dispatchEvent(Component.java:4482)
java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4577)
java.awt.LightweightDispatcher.processMouseEvent(Container.java:4238)
java.awt.LightweightDispatcher.dispatchEvent(Container.java:4168)
java.awt.Container.dispatchEventImpl(Container.java:2085)
java.awt.Window.dispatchEventImpl(Window.java:2478)
java.awt.Component.dispatchEvent(Component.java:4482)
java.awt.EventQueue.dispatchEventImpl(EventQueue.java:644)
java.awt.EventQueue.access$000(EventQueue.java:85)
java.awt.EventQueue$1.run(EventQueue.java:603)
java.awt.EventQueue$1.run(EventQueue.java:601)
java.security.AccessController.doPrivileged(Native Method)
java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:98)
java.awt.EventQueue$2.run(EventQueue.java:617)
java.awt.EventQueue$2.run(EventQueue.java:615)
java.security.AccessController.doPrivileged(Native Method)
java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
java.awt.EventQueue.dispatchEvent(EventQueue.java:614)
java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
----------------------------------

Easy!

serialVersionUID

Serialization is a powerful mechanism available in Java. It is - among many others - used by RMI (Remote Method Invocation), that can establish a communication between different JVMs.
A problem can appear when the class definition available on JVM#1 is not the same as the one available on JVM#2.
When defining a class implementing the java.io.Serializable interface, modern compilers will often reproach you with not correctly defining a final static long serialVersionUID.
We are briefly going to touch base on that, trying to explain what that means and what this is all about.

There is nothing like a good example to illustrate this kind of things. I hope you will like the following one.
Let us define a Serializable class:

public class Description
  implements Serializable
{
  @SuppressWarnings("compatibility:5602934367223767090")
  private static final long serialVersionUID = 1L;

  private String stringDataOne;
  private String stringDataTwo;
//private double numericData;
  private float numericData;

  public void setStringDataOne(String s)
  {
    this.stringDataOne = s;
  }

  public void setStringDataTwo(String s)
  {
    this.stringDataTwo = s;
  }

  public String getStringDataOne()
  {
    return this.stringDataOne;
  }

  public String getStringDataTwo()
  {
    return this.stringDataTwo;
  }

  public void setNumericData(float dummyNumber)
//public void setNumericData(double dummyNumber)
  {
    this.numericData = dummyNumber;
  }

  public float getNumericData()
//public double getNumericData()
  {
    return numericData;
  }
  
  @Override
  public String toString()
  {
    return new StringBuffer("One  : ").append(this.stringDataOne)
                  .append("\nTwo  : ").append(this.stringDataTwo)
                  .append("\nNum  : " + NumberFormat.getInstance().format(numericData)).toString();
  }
}

Notice the final static long serialVersionUID = 1;. We'll talk about it shortly.
This class being defined, we are going to:

1. Serialize an instance of it into a file
2. De-serialize this instance from the file

Serializer

public class WriteObject
{
  public static void main(String[] args)
  {
    Description dataObject = new Description();
    dataObject.setStringDataOne("First line of data");
    dataObject.setStringDataTwo("Second line, as expected");
    dataObject.setNumericData(12345);

    try
    {
      FileOutputStream fout = new FileOutputStream("description.ser");
      ObjectOutputStream oos = new ObjectOutputStream(fout);
      oos.writeObject(dataObject);
      oos.close();
      System.out.println("Done");
    }
    catch (Exception ex)
    {
      ex.printStackTrace();
    }
  }
}

Serializer at work:

 Done

De-serializer

public class ReadObject
{
  public static void main(String[] args)
  {
    Description restoredObject;

    try
    {
      FileInputStream fin = new FileInputStream("description.ser");
      ObjectInputStream ois = new ObjectInputStream(fin);
      restoredObject = (Description) ois.readObject();
      ois.close();

      System.out.println(restoredObject);
    }
    catch (Exception ex)
    {
      ex.printStackTrace();
    }
  }
}

De-serializer at work:

 One  : First line of data
 Two  : Second line, as expected
 Num  : 12,345

This is what happens when everything goes well.
Now, let's modify the Serializable class, let's change the float numeric data to a double (member, getter, setter), and without re-serializing it, let's try to de-serialize it:

java.io.InvalidClassException: serialversionui.Description; incompatible types for field numericData
 at java.io.ObjectStreamClass.matchFields(ObjectStreamClass.java:2210)
 at java.io.ObjectStreamClass.getReflector(ObjectStreamClass.java:2105)
 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:602)
 at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582)
 at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
 at serialversionui.ReadObject.main(ReadObject.java:17)

Basically, it says it cannot cast a Description to a Description...
If the serialVersionUID had been modified when changing the float to a double - like from 1 to 2 for example

  private static final long serialVersionUID = 2L;

you would have seen the following message:

java.io.InvalidClassException: serialversionui.Description; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = 2
 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:562)
 at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582)
 at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
 at serialversionui.ReadObject.main(ReadObject.java:17)

It does not work any better, but it returns useful information to communicate with the development teams.

Java recommends a way to come up with the appropriate value for the serialVersionUID. This is the reason for the @SuppressWarnings annotation in the code snippets available above. Find it here.

Serializing data

Let's say you are developing Java code, and you have large objects - Maps and Lists - to read through in order to compute some result.
The problem, illustrating the point, was to load in memory all the data and coefficients for a Tidal Computation. Before you start doing any work, you have to read those data from somewhere, load them into your working memory, so they are available when you need them.
Several technologies are available to achieve this goal, the question here is to know which one to go for. Many different data storage techniques could be used. We will compare here XML, Java Serialization, and json (Java Script Object Notation). Notice that we are not here talking about Databases, which is certainly a possible approach. The rationale was to keep the solution as light and self-contained as possible, the data are to be read only (and not modified by any mean). We felt that in that case, the Database approach was overkilling.
XML itself can be approached in different ways, DOM, SAX, or StAX. As we don't wish here to modify any of the data to load, we will use SAX (Simple API for XML).
All the required data are stored into some files, which have to be loaded into objects at runtime.
So, to recapitulate before we get started, we will restrict our comparison to those three approaches:

• XML, using a SAX parser to load the data
• Java Serialization/Deserialization
• Json Serialization/Deserialization

In order to compare what is comparable (apples to apples), we stored the data files into an archive, a zip in this case.
Those data were then deserialized into the exact same objects.
The archive containing the XML data is 1,546,860 bytes big.
The archive containing the Java Serialized (.ser) data is 1,772,159 bytes big.
The archive containing the json data is 2,543,174 bytes big.
We iterated 5 times on the same deserializations, and then did an average.

Here is the code:

  public static void main(String[] args) throws Exception
  {
    long[] elapsed = { 0L, 0L, 0L };
    long before = 0L, after = 0L;
    BackEndTideComputer.setVerbose(false);
    for (int i=0; i<5; i++)
    {
      before = System.currentTimeMillis();
      BackEndTideComputer.connect(BackEndTideComputer.XML_OPTION);
      after = System.currentTimeMillis();
      elapsed[0] += (after - before);
      BackEndTideComputer.disconnect();
      
      before = System.currentTimeMillis();
      BackEndTideComputer.connect(BackEndTideComputer.JAVA_SERIALIZED_OPTION);
      after = System.currentTimeMillis();
      elapsed[1] += (after - before);
      BackEndTideComputer.disconnect();
  
      before = System.currentTimeMillis();
      BackEndTideComputer.connect(BackEndTideComputer.JSON_SERIALIZED_OPTION);
      after = System.currentTimeMillis();
      elapsed[2] += (after - before);
      BackEndTideComputer.disconnect();
    }
    System.out.println("XML:" + Long.toString(elapsed[0] / 5) + " ms" +
                     "  JavaSer:" + Long.toString(elapsed[1] / 5) + " ms" +
                     "  json:" + Long.toString(elapsed[2] / 5) + " ms");
  }

... and here are the results:

 XML:3540 ms  JavaSer:5455 ms  json:4225 ms

If we tune those results to take the data file size in account, by rendering the milliseconds per megabyte, we come up with the following table:

	XML	ser	json
archive size in bytes	1,546,860	1,772,159	2,543,174
elapsed time in ms	3,540	5,455	4,225
average ms/Mb	2340	3228	1742

The first to cross the line in the SAX XML Parser (I used the one from Oracle).
The best average award (ms/Mb) goes to json (I used Gson, the json parser from Google), but notice that that data we started from are the bigest.
Surprisingly, the red lantern remains the Java Serialization...

A note about JAX-B

We've not talked about JAX-B here, which stands for Java API for XML Binding. It sounds like it could deserve its place in this article... But it actually does not. The JAX-B process starts from an XML Schema, which we don't have in this case (OK, we could have built one. But we did not). This article talks about it.

So, we've been able to compare three approaches, and the "native" one seems to be the worse, which is a bit of a surprise.
This is probably enough to have a better idea of what technology to use to address this kind of problem, but as a bonus, let us take a look at the amount of code required to put them at work.

Java Deserializer

  public static <T> T loadObject(InputStream resource, Class<T> cl) throws Exception 
  {
    T tideObject = null;
    try
    {
      ObjectInputStream ois = new ObjectInputStream(resource);
      tideObject = (T)ois.readObject();
      ois.close();
    }
    catch (Exception ex)
    {
      throw ex;
    }
    return tideObject;
  }

JSON deserializer

  public static <T> T loadObject(InputStream resource, Class<T> cl) throws Exception 
  {
    T tideObject = null;
    try
    {
      BufferedReader br = new BufferedReader(new InputStreamReader(resource));
      StringBuffer sb = new StringBuffer();
      String line = "";
      boolean go = true;
      while (go)
      {
        line = br.readLine();
        if (line == null)
          go = false;
        else
          sb.append(line);
      }
      br.close();
      if (gson == null)
        gson = new GsonBuilder().setPrettyPrinting().create();
      tideObject = gson.fromJson(sb.toString(), cl);        
    }
    catch (Exception ex)
    {
      throw ex;
    }
    return tideObject;
  }

XML SAX Handler

Here is the code starting the SAX process

  public static Map getStationData() throws Exception
  {
    Map stationData = new HashMap();
    StationFinder sf = new StationFinder(stationData);
    try
    {
      SAXParserFactory factory = SAXParserFactory.newInstance();
      SAXParser saxParser = factory.newSAXParser();      
      InputSource is = BackEndTideComputer.getZipInputSource(ARCHIVE_STREAM, STATIONS_ENTRY);
      saxParser.parse(is, sf);       
    }
    catch (Exception ex)
    {
      ex.printStackTrace();
    }
    
    return stationData;
  }

Notice that this code invokes a custom handler, named StationFinder:

  public static class StationFinder extends DefaultHandler
  {
    private String stationName = "";
    private TideStation ts = null;
    private Map stationMap = null;
    
    public void setStationName(String sn)
    {
      this.stationName = sn;
    }
    
    public StationFinder()
    {
    }

    public StationFinder(Map map)
    {
      this.stationMap = map;
    }
    
    public TideStation getTideStation()
    {
      return ts;
    }

    private boolean foundStation        = false;
    private boolean foundNameCollection = false;
    private boolean foundStationData    = false;
    
    @Override
    public void startElement(String uri, String localName, String qName, Attributes attributes)
      throws SAXException
    {
//    super.startElement(uri, localName, qName, attributes);
      if (!foundStation && "station".equals(qName))
      {
        String name = attributes.getValue("name");
        if (name.contains(this.stationName))
        {
          foundStation = true;
          ts = new TideStation();
          ts.setFullName(name);
        }
      }
      else if (foundStation)
      {
        if ("name-collection".equals(qName))
        {
          foundNameCollection = true;
        }
        else if ("name-part".equals(qName) && foundNameCollection)
        {
          ts.getNameParts().add(attributes.getValue("name"));
        }
        else if ("position".equals(qName))
        {
          ts.setLatitude(Double.parseDouble(attributes.getValue("latitude")));
          ts.setLongitude(Double.parseDouble(attributes.getValue("longitude")));
        }
        else if ("time-zone".equals(qName))
        {
          ts.setTimeZone(attributes.getValue("name"));
          ts.setTimeOffset(attributes.getValue("offset"));
        }
        else if ("base-height".equals(qName))
        {
          ts.setBaseHeight(Double.parseDouble(attributes.getValue("value")));
          ts.setUnit(attributes.getValue("unit"));
        }
        else if ("station-data".equals(qName))
        {
          foundStationData = true;
        }
        else if (foundStationData && "harmonic-coeff".equals(qName))
        {
          String name = attributes.getValue("name");
          double amplitude = Double.parseDouble(attributes.getValue("amplitude"));
          double epoch     = Double.parseDouble(attributes.getValue("epoch")) * TideUtilities.COEFF_FOR_EPOCH;
          Harmonic h = new Harmonic(name, amplitude, epoch);
          ts.getHarmonics().add(h);
        }
      }
    }

The code snippets speak for themselves... SAX is the most demanding, by far. Also notice that it requires a full knowledge of the data to parse. Java and Json de-serialization do not. As long as the data have been correctly serialized, they will be correctly de-serialized (which brings up the discussion about the famous serialVersionUID in Java..., it's another story).

All the code, including the archives containing the data, is available from Google Code. The specific class displayed above is right here.

Functional Programming in Java

Functional Programming brings new and interesting paradigms to Java code development.
I am not saying that functional programming is a new concept, OK?
Several of those languages (Scala, Clojure) run on top of a Java VM, but Java itself is NOT as functional programming language..., how is that possible, there must be a way!
One of the aspects of the functional programming languages is this possibility to use a pointer on a function as a parameter. Here is a possible approach, even easier to access since the diamond notation has been implemented in the recent versions of the Java language.
We are going to use a simple example as an illustration. We will choose the function to execute, based on the value of a parameter. What we want to do here is to greet someone in the language passed as a parameter.

Note:

Only for the clarity of the code, we are not using here the usual Java convention for the diamond notation (T for Type, E for Element, K for Key, V for Value, N for Number, S, U, V, etc, for 2nd, 3rd, 4th,... types).

Let us consider a first Java interface defined as follow:

 package functions;

 public interface FunctionOneArgReturnVoid<SomeType>
 {
   public void execute(SomeType a);
 }

And let us write the following implementation:

 package functions;

 public class ImplementationSampleOne
 {
   FunctionOneArgReturnVoid<String> funcEnglish = 
                      new FunctionOneArgReturnVoid<String>()
    {
      public void execute(String s)
      {
        System.out.println("Hello " + s + "!");
      }
    };
   FunctionOneArgReturnVoid<String> funcFrench  = 
                      new FunctionOneArgReturnVoid<String>()
    {
      public void execute(String s)
      {
        System.out.println("Bonjour " + s + " !");
      }
    };

   public FunctionOneArgReturnVoid<String> getFunction(String lng)
   {   
     if ("FR".equalsIgnoreCase(lng))
       return funcFrench;
     else
       return funcEnglish;
   } 

   public void execute(FunctionOneArgReturnVoid<String> func, String arg)
   {
     func.execute(arg); 
   }

   public static void main(String[] args)
   {
     String lng = "EN";
     if (args.length > 0)
       lng = args[0]; 

     ImplementationSampleOne impl = new ImplementationSampleOne();
     impl.execute(impl.getFunction(lng), "Joe Shmow");
   }
 }

See the two implementations of the FunctionOneArgReturnVoid interface, and the way they are invoked in the main method.
The typography rules are not the same in English and French. In French you need a space before a '!'.
The execution would look like this:

 Prompt> java functions.ImplementationSampleOne "FR"
 Bonjour Joe Shmow !

 Prompt> java functions.ImplementationSampleOne "EN"
 Hello Joe Shmow!

The return type can be also described by an interface:

 package functions;

 public interface FunctionTwoArgsReturnType<SomeTypeOne, 
                                            SomeTypeTwo, 
                                            ReturnType>
 {
   public ReturnType execute(SomeTypeOne a, SomeTypeTwo b);
 }

The interface implementation would look like this:

  FunctionTwoArgsReturnType<String, String, String> funcEnglish = 
                 new FunctionTwoArgsReturnType<String, String, String>()
   {
     public String execute(String a, String b)
     {
       return "Hello " + a + ", from " + b + "!";
     }
   };
  FunctionTwoArgsReturnType<String, String, String> funcFrench  = 
                 new FunctionTwoArgsReturnType<String, String, String>()
   {
     public String execute(String a, String b)
     {
       String s = "Bonjour " + a + ", de la part ";
       if (b.toUpperCase().toCharArray()[0] == 'A' ||
           b.toUpperCase().toCharArray()[0] == 'E' ||
           b.toUpperCase().toCharArray()[0] == 'I' ||
           b.toUpperCase().toCharArray()[0] == 'O' ||
           b.toUpperCase().toCharArray()[0] == 'U' ||
           b.toUpperCase().toCharArray()[0] == 'Y')
         s += ("d'" + b);
       else
         s += ("de " + b);
       s += " !";
       return s;
     }
   };

The main like that:

  public static void main(String[] args)
  {
    String lng = "EN";
    if (args.length > 0)
      lng = args[0];

    ImplementationSampleTwo impl = new ImplementationSampleTwo();
    String greeting = impl.execute(impl.getFunction(lng), "Joe Shmow", "Olivier");
    System.out.println(greeting);
    greeting = impl.execute(impl.getFunction(lng), "Joe Shmow", "Mr X");
    System.out.println(greeting);
  }

The output would become

 Prompt> java functions.ImplementationSampleTwo "EN"
 Hello Joe Shmow, from Olivier!
 Hello Joe Shmow, from Mr X!

 Prompt> java functions.ImplementationSampleTwo "FR"
 Bonjour Joe Shmow, de la part d'Olivier !
 Bonjour Joe Shmow, de la part de Mr X !

The complexity is isolated in the implementation of the function. The code using it does not need to know anything about it.

This possibility of using pointers on functions does not make Java a functional programming language, but this is a first step towards it.

"Rich" console output from Java

A while back - before the graphical terminals were available, in the old VT time - we used to build forms from scratch, using ANSI escape sequences. That was quite limited, but it was not that ugly, some times...
Linux still supports this kind of thing, and from Java, you can definitely add any effect to the text you spit out using System.out or System.err.
Surprisingly, it's another story on Windows, you have to mention the ansi.sys file in the config.nt one, which magically disappeared in Windows 7.
There is a solution to this problem, at jansi.fusesource.org.
It's small library available for free, which allows you to write the usual ANSI codes, whatever the system you run on is.
You can define your own codes:

  public final static char ESC = '\u001b'; // (char) 27;

  public final static String ANSI_BLACK   = "0";
  public final static String ANSI_RED     = "1";
  public final static String ANSI_GREEN   = "2";
  public final static String ANSI_YELLOW  = "3";
  public final static String ANSI_BLUE    = "4";
  public final static String ANSI_MAGENTA = "5";
  public final static String ANSI_CYAN    = "6";
  public final static String ANSI_WHITE   = "7";

  public static final String ANSI_CLS         = ESC + "[2J";
  public static final String ANSI_HOME        = ESC + "[H"; // 0,0
  public static final String ANSI_HEAD        = ESC + "[1G"; // Start of current line, position 1

  public static final String ANSI_NORMAL      = ESC + "[0m";
  public static final String ANSI_BOLD        = ESC + "[1m";
  public static final String ANSI_FAINT       = ESC + "[2m";
  public static final String ANSI_ITALIC      = ESC + "[3m";
  public static final String ANSI_UNDERLINE   = ESC + "[4m";
  public static final String ANSI_BLINK       = ESC + "[5m";
  public static final String ANSI_BLINK_FAST  = ESC + "[6m";
...

and run like this:

    AnsiConsole.systemInstall();
    AnsiConsole.out.println(ANSI_CLS);
    AnsiConsole.out.println(ANSI_NORMAL + "Normal text and " +
                            ANSI_WHITEONBLUE + "bold" +
                            ANSI_NORMAL + " text.");
    AnsiConsole.out.println(ANSI_NORMAL + "Normal " +
                            ansiSetTextColor(ANSI_YELLOW) + "yellow" +
                            ANSI_NORMAL + " text and " +
                            ansiSetTextAndBackgroundColor(ANSI_WHITE, ANSI_BLACK) +
                            "bold" + ANSI_NORMAL + " text.");
    
    System.out.println(ANSI_NORMAL + "Normal text and " +
                       ANSI_WHITEONBLUE + "bold" +
                       ANSI_NORMAL + " text.");
    System.out.println(ANSI_NORMAL + "Normal " +
                       ansiSetTextColor(ANSI_YELLOW) + "yellow" +
                       ANSI_NORMAL + " text and " +
                       ansiSetTextAndBackgroundColor(ANSI_WHITE, ANSI_BLACK) +
                       "bold" + ANSI_NORMAL + " text.");

or even easier, use some static classes and methods provided by Jansi:

    AnsiConsole.systemInstall();
    System.out.println( ansi().eraseScreen().fg(RED).a("Hello").fg(GREEN).a(" World").reset() );
    System.out.println( ansi().bg(CYAN).fg(RED).bold().a("Hello").fg(GREEN).a(" World").reset() );
    System.out.println();
    System.out.println();
    System.out.print( ansi().fg(WHITE).a("Hello World") );
    System.out.flush();
    try { Thread.sleep(1000); } catch (Exception ex) {}
    System.out.println( ansi().eraseLine(Erase.ALL).cursorLeft("Hello World".length()).fg(YELLOW).a("Hello World, again").reset() );
    AnsiConsole.systemUninstall();

And the output is the same on Windows, on Linux,...

By the way, this is what Gradle is using for their output.

Start Windows Normally

I've been experiencing several crashes recently, apparently the video card on my Dell is going nut. The only way out is to hard-reset the machine.
And after that, when I restart, I always have the same screen:

Ah that's good one! I wish I could do that!

NMEA Station Calibration

We intend here to show the importance and the impact of an accurate calibration and tuning of your NMEA station, in order to correctly evaluate the parameters to calculate, namely the current direction and speed. The impact of an incorrect tuning leads to completely useless computation, and wrong evaluation of the expected parameters.
For a better understanding of the mutual impact of all those parameters, we've illustrated this document with several interactive applets, which you can play with or interact with the different parameters for yourself.
Values in fields can be changed (hit the [Enter] key for that, after modifying the value in the field), sliders are draggable with the mouse, as well as the compasses that figure in the applets, to change the various headings.

Steps

In order to log and visualize data efficiently, you need to complete several steps, in a specific order:

1. Instruments Calibration
   You need to make sure the values read by your instruments are accurate
2. Polars Elaboration
   You need to know what your speed should be in the condition you sail in
3. Data Logging
   You need to log the right data
4. Data Visualization
   You need to render the logged data appropriately, rate your tacks and gybes, visualize laylines and frames, etc.

Instruments Calibration

Most of the in-board NMEA stations provide access to calibration parameters that will lead to an accurate data logging and display. The goal of this process is to evaluate the parameters we will need to feed the NMEA station with.
You need to simulate the values of the parameters to enter in the NMEA station. This process follows several steps:

1. Log data on the boat
2. Display the data in some graphical dashboard
3. Elaborate the calibration parameters until the display looks good
4. Set the calibration parameters in the NMEA station
5. Log again, to make sure your evaluation of the tuning parameters was right

Polars Elaboration

Polars are a mathematic and theoric model that will tell you what BSP you should theorically reach for a given TWA and TWS.
Polars can be used to predict your potential track (routing), or to rank your boat handling.
There are two main ways to get the polars of your boat:

• VPP (Velocity Prediction Programs)
  Those data are (or should be) provided by the designer of your boat.
• Logging
  In case your designer did not provide them, or if the VPP is wrong...

Data Logging

For an appropriate rendering of your performances, you need to log the right set of data. The format of the data logged by your navigation station is NMEA.
Depending of the data your station is able to produce, you will have to choose the right ones.

Data Visualization

The main question to address is "What do you want to see?". And there is no ready-made answer to that question.

To summarize

1. Appropriate Calibration leads to the right evaluation of the True Wind (Angle, Speed and Direction) and of the Current (Speed and Direction)
2. With the right evaluation of the True Wind, you can refer to your polars, so you know what your speed should be
3. Having logged the right data, comparing them to what the polars tell you, you can evaluate your performances
4. Based on the evaluation of your performances, you can render your data appropriately, see where you did good, and where there is room for improvement

Instruments Calibration

Calibration impacts the data read on the boat through the devices attached to the NMEA station, like the boat speed transducer, the compass, the wind vane, and the anemometer.
We want here to

1. show how the different data interact with each other
2. see how to apply calibration factors (coefficients and offsets) to the values read from the station

The data are logged on the boat, that carries the devices (speedometer, compass, wind vane, anemometer).
It is far from unusual to have to tweak or tune those devices.
The anemometer is usually the most reliable one, as it is running away from most of the interferences that make the other devices in need for calibration.
For example, the compass and the wind vane could have been installed not exactly in line with the boat. This means that an offset is to be applied to the data they read.
Depending on the location of the boat speed transducer, the speed of the molecules of water does not necessary directly reflect the speed of the boat. A difference can even be seen from one tack to the other. A coefficient might then be required.
Many data can be recorded, we're only - in this document - going to talk about the data we need to evaluate the performances of the boat.
For example, DBT (Depth Below Transducer) is useless for us, in the framework on this document.
The main data recorded by the instruments are:

• Boat Speed (BSP), sometimes aka STW (Speed Through Water)
• Heading (HDG, HDM, HDT)
• Apparent Wind Speed (AWS)
• Apparent Wind Angle (AWA)
• Boat Position (from the GPS)
• Speed Over Ground (SOG)
• Course Over Ground (COG)

One data is to be estimated:

• Leeway (see about that the section about Damping)
  Mostly depends on AWA, AWS, and sea state (that involves waves, swell direction... pretty tricky).

The main data computed, based on the data above, are:

• True Wind Speed (TWS)
  Depends on COG & SOG, AWA, AWS.
  Not on BSP & HDG, as usually admitted.
• True Wind Angle (TWA)
  Depends on COG & SOG, AWA, AWS.
  Not on BSP & HDG, as usually admitted.
• True Wind Direction (TWD)
  Depends on TWA, HDG
• Current Speed
  Depends on BSP, HDG, SOG, COG, Leeway.
• Current Direction
  Depends on BSP, HDG, SOG, COG, Leeway.

Note: The tuple (SOG, COG) is semantically (in this case) equivalent to the Boat Position. Both (SOG, COG) and Boat Position come from the GPS.

A note about BSP calibration:

This one can be very tricky, specially if the transducer is not exactly set in the axis of the boat, as it is usually the case.
When you're sailing upwind, depending on the tack, the reading may substancially vary, as for a given speed, the pressure of the water molecules on the transducer is definitely not the same on both tacks, the difference of reading between tacks being more or less proportional to the heel.
To address that issue, most of the NMEA Stations provide a way to set the boat speed calibration for each tacks.
Unfortunatelly, this is not good enough, the Apparent Wind Angle should be taken in account as well. You certainly do not want the same coefficient to be applied on the boat speed when you are heading close hauled or running downwind.

Calibrating the Boat Speed and Heading

Here is an interactive applet:

---

This browser does not support Applets.

This applet shows how to evaluate/compute the current. It provides the possibility to interact with the calibration parameters and see how they impact the final resiult. Use the sliders in the applet on the left to modify the values of Boat Speed (BSP) and Leeway. To change the Heading (HDG) value, drag the mouse over the compass. Course Over Ground is fixed, set to 80°. Speed Over Ground is fixed, set to 5.5 knots. Both COG and SOG come from the GPS. Current Speed & Direction are computed based on the five values above. In green, data read from the NMEA Station In red, parameters used for computation In blue, computed data At the bottom, are two calibration parameters. The BSP Coefficient is applied to the Boat Speed. The HDG Offset is added to the Heading. To modify a parameter, change its value, and hit the [Enter] key. The display on the right side render the value read by the station. The pane on the left displays the corrected values, used for the computing of the current.

Something to say about leeway:

Leeway - mentionned above, and used below - is a dimension proportional (among others) to the Apparent Wind Angle (AWA), and not the True Wind Angle (TWA).
The Apparent Wind is the wind moving the boat.
Sailing on a very fast boat (like a racing multihull, an ice-boat, etc), it is not unusual to fell the wind on the nose, and see the wind-generated waves (not on an ice-boat, I know) coming from the quarter. The sails are trimmed according to the Apparent Wind, and the force they generate is expressed accordingly.
If your sails are trimmed as if you're going upwind (i.e. up the Apparent Wind), then you make leeway accordingly.
Picture this:
Sailing in the same external conditions (TWD, TWS), we have two boats, well trimmed, sailing at about 135° of the True Wind (TWA=135°):

• A cruising monohull, sailing between 6 and 8 knots
• A racing multihull, sailing above 25 knots

The first one will be sailing with the spinnaker up.
The second will have - in the best case scenario - a closed hauled gennaker.

Obiously, their leeway will not depend on the same parameters.
The fast boat will make more leeway than the cruising one, because of his close hauled sails.

For now, in the Dashboard Preferences, you have a value named "Max Leeway". This value will be used in conjunction with the Apparent Wind Angle, when it is between 0°and 90°, as follow:

Leeway = MaxLeeway × cos(awa)

This browser does not support Applets.
The True Heading, corrected with the Leeway is called Course Made Good (CMG).

Calibration of the Compass

Usually under-estimated! You think that because you sail on a plastic boat, your compass does not suffer any deviation... Think again!
Unless you have a gyroscope, you will need a compass to know what your heading is. Your heading is the base of many calculations, see above.
The first parameter to take in account id the magnetic declination, noted D. This Declination depends on your location, and the time. It varies with time, because the magnetic pole navigates around the geographical one. Any nautical chart will tell you what the declination is in the region and at the time your sail. You can get its value from several web-sites, including the NOAA's one. The RMC sentence also returns this data. Some computer algorythms are also available to calculate this important datum. Declination is the same for everyone sailing the same waters at the same time.
This is not the case of the deviation. The deviation is a parameter depending on the boat you sail. It is generated by the magnetic masses around the compas. As such, it depends:

• On the boat (holding those magnetic masses)
• The heading, as those masses "move" around the compass.

Steel boats have a big one, ferro-cement ones have a huge one (50° is not absurd for them...). Plastic boats, you never know! A loudspeaker close to a compass will have quite an impact on its reading... An engine (or a generator), same. When the engine is running, it might even be different... There is a lot of litterature about this topic, go check for yourself if you are interested.

---

This browser does not support Applets.

The generic shape of a deviation curve is d = A + (B.sin(Z)) + (C.cos(Z)) + (D.sin(2.Z)) + (E.cos(2.Z)) where Z is the magnetic heading. The deviation curve represented on the left is following this equation. This interactive applet shows the 3 different headings: True, Magnetic, and Compass. The Compass is the displayed (and recorded) one. You can modify the value of the Magnetic Declination. Plus (+) is East, minus (-) is West, for Declination and deviation. The algebric sum of the Declination and of the deviation is called Variation, noted W. The true heading is the one to be plotted on the chart, as the chart too refers to the true North (a.k.a. Geographical North).

The compass calibration is crucial. The data it provides is a parameter of all the computations we are interested in. If the deviation is not taken in account, its value will be put on the current's account, and that would be a big mistake.

Calibration of the Apparent Wind Speed and Direction

Here is another interactive applet. Like before, you can use the sliders, modify the values in the fields (hit the [Enter] key in the field to apply your changes), and drag the compass' rose left or right.

---

This browser does not support Applets.

This one shows how the calibration of the station interacts with the computation of the True Wind, which is the key to access the polars. The displays (on the right) are the one you would see on the boat when sailing, the data displayed on the main pane, on the left, are the (ideally) "real" data, taking the calibration parameters in account. Warning: This vector combination is the usually admitted one, but this is a simplified one! The real one - closer to the truth - will be exposed next.

Let's now go closer to the truth...
Unlike what's commonly admitted, the True Wind should be calculated with the Speed Over Ground and the Course Over Ground, instead of respectively the Boat Speed and the Heading.
Imagine that: you are "sailing" in absolutely NO wind, but you have a strong current, about 6 knots. Well, in that case, you feel an apparent wind of about 6 knots. If you do the calculation of the True Wind with the Boat Speed, the calculation will return a True Wind of 6 knots, in the direction opposite to the current.
But if you do it with the Course Over Ground and Speed Over Ground, then it will return a True Wind Speed of zero knot, which is right.
If the GPS data are available, they should be preferred for this computation.

This browser does not support Applets.

This one contains all the possible parameters (for now...), and computes the True Wind with the GPS Data (SOG & COG). For the clarity of the figure, Compass calibration parameters (deviation curve) are not represented. All the values are modifiable like previously. Heading and Course Over Ground have their own compass' rose, which is dragable on both cases. As you can tell, the figure is quite complex. And it becomes even more complex when you modify the data and parameters. Again, it is very difficult to isolate the parameters from each other, modifying one value impacts mostly all of the others... As a matter of fact this reflects the reality of an NMEA Station calibration. It is a long and precise process, which should not be under-estimated. Ultimately, what we look for here is the direction and the speed of the current, represented in green on the applet on the left, as this is what we will compute the route to give to the driver with, decide when to tack, etc. All the parameters we mentionned so far have a direct impact on this evaluation. If one of them is wrong, then the evaluation of the current is impacted.

Full validation: Stability of True Wind Direction and Current Direction

Now, let's put all the parameters together, as in the real world. The main difficulty is precisely due to this fact, there is no clear way to isolate them...
Below is the display of some data logging, the logging is done with a non-well calibrated NMEA station.
It displays two legs upwind, and two legs downwind.
The True Wind Direction is displayed in blue, the Current Direction is displayed in red,
The vertical panes on the left display the True Wind Direction (TWD), and the Current Direction (CDR).
The horizontal panes display the Boat Speed (BSP), True Wind Speed (TWS) and Current Speed (CSP).
TWD, CDR, TWS, TWA and CSP are calculated, based on the data logged on board. Any error in the data read from the devices (BSP, HDG, AWA, AWS) will deeply impact this calculation.
You can impact the parameters to apply on the data read from the station by modifying their values on the right pane of the applet.
See how the current direction varies everytime the boat heading is changing... The TWD is less impacted, but it is not steady either.

This browser does not support Applets.

This interactive applet represents data logged with a non properly calibrated NMEA Station. Play with the parameters, to see how they impact the rendering of the track. To modify a value, change the number in the field and press [Enter]. There is a slider you can use, just above the BSP Panel at the bottom. To display the right data, you must enter the following parameters: Multiply BSP by 1.1 HDG Offset 5° Multiply AWS by 1.01 AWA Offset 2° Max Leeway 10° Those paratemers should be used later on to calibrate your NMEA Station. Ideally, when the parameters are set correctly, the graphs rendering TWD, CDR, TWS & CSP should display a straight line. (Assuming of course that the wind and current conditions were stable during this logging session)

For now, the leeway is calculated as a function of the Cosinus of the AWA, from 0 to 90 degrees only (we assume no leeway beyond 90° of Apparent Wind Angle).
This is likely to be improved.

Data Logging

Logging everything the station produces is a possibility, but it can be a lot, and saturate the recording device.
For the best rendering, you need to log at least:

• The Boat Speed (BSP)
• The Boat Heading (HDG, HDM, or HDT)
• The Apparent Wind Speed (AWS)
• The Apparent Wind Angle (AWA)
• The Boat Position
• The Course Over Ground (COG)
• The Speed Over Ground (SOG)

Those data can be found - for example - in the following sentences:

Required Data	Available in NMEA Sentences
Boat Speed	VHW
Boat Heading	VHW, HDG, HDM, HDT
Apparent Wind Speed	MWV, VWR
Apparent Wind Angle	MWV, VWR
Boat Position	RMC, GLL
Speed Over Ground	RMC, GLL, VTG
Course Over Ground	RMC, GLL, VTG

The NMEA sentences mentionned in the table above have the following meaning:

VHW	Water Speed and Heading (Velocity Heading Water)
HDG	Heading, Deviation & Variation
HDM	Heading, Magnetic
HDT	Heading, True
MWV	Wind Speed and Angle
VWR	Relative wind direction and speed (Velocity Wind Relative)
RMC	Recommended Minimum Specific GPS/TRANSIT Data
GLL	Geographic Position, Latitude/Longitude
VTG	Track Made Good and Ground Speed

Note: RMC gives the boat position just like GLL, but also returns the Magnetic Declination.

Data Visualization

Damping

The damping is the kind of smoothing that will eliminate the aberrations. This is specially usefull when the data are logged in tough conditions. For example, when you are sailing upwind in choppy seas, the wind vane being set at the top of the mast will be shaked substancially. This will generate some data that will not reflect the reality.
The damping is taking for any given logged point X points before,and X points after, and calculates the average.

This browser does not support Applets.
Use the slider to change the damping factor.

Several panels of the Dashboard provide access to a damping factor.

A note about current evaluation

The way we've calculated the current so far is based on the values read and calculated at a given moment in time (BSP, TWS, TWD, COG, SOG).
Some of those values are themsleves based on others (TWS & TWA are based on BSP, CMG - based on HDG, leeway, Declination and Deviation -, SOG, COG.
The current calculation we've shown so far is a triangulation based on instant values.
Basing the calculation on damped values does not really help, but there is another way to smooth those values. We have seen that the current is the vector that sits between the tuple (CMG, BSP) and the tuple (COG, SOG).
The idea here is to evaluate over a period of time this difference between the position where the boat is, and the position where it should be with no current.
We can even simultaneously evaluate the current with different period of time, like 1 minute and 10 minutes, for example. This would be usefull when the boat heading is changing, for example when tacking or rounding a mark.

After one tack. See the different values in the three panels

After two tacks. See the different values in the three panels

After two tacks, motoring head to wind. See the different values in the three panels

Notice in each case the differences between the current speeds, and current directions.

Evaluate and rate your tacks and gybes

We'll be talking here of the method well-known under the name of "Tack Loss".
Its principle is simple, and it is the same for tacks and gybes; we'll expose it for a tack, you'll be after that able to transpose it to a gybe.

• You're sailing upwind at a constant speed
• You evaulate you VMG (Velocity Made Good)
• The tack begins
• After the tack, and after a given amount of time, you're sailing on the other tack, and you've recovered your best speed
• This is when the tack is being rated:
  • You calculate, based on the VMG before the tack, how far upwind you would have been if you had not tacked
  • You see how far upwind you actually are
  • You compare the two (the difference is evaluated in meters)

The tack loss is not - as one could expect - always negative. If you appropriately tack in a wind shift, you might have very good ratings!
The opposite can be true as well...
The tack loss is usually evaluated as a distance, measured - even on American and English boats - in meters.
It can also be rendered as a percentage, as we're going to explain.
A tack can be decomposed is three main steps.

1. You're heading full speed, and you begin the tack. Your VMG increases, due to the momentum of the boat
2. The boat not being anymore powered by the wind, and having the wind on the nose, the boat (and the VMG) slows down as the wind is crossed
3. The boat bears away, about to re-catch the wind, the sails will soon be trimmed on the other tack, the boat is recovering its speed

The steps described above correspond to this figure:

Rating the tack as a percentage corresponds to the comparison of those two areas (in grey):

this area in grey...

... with this one.

Next time, we'll talk about the data rendering.
It's also a pretty vast topic.

Smoothing a color chart in Java

The goal here is to smooth a colored drawing, to avoid the tile effect.

Before

After

Although they look similar, the path for the two images above is pretty different.
The regular path to draw on a javax.swing.JPanel is to override the method

  public void paintComponent(Graphics gr)

We have defined an array of colors, for the example

  private final static int[][] COLOR_VALUES =
  {
   {255, 255, 255 },
   {208, 244, 250 },
   {161, 233, 246 },
   {115, 222, 241 },
   {68, 211, 237 },
   {21, 200, 232 },
   {21, 207, 223 },
   {20, 214, 214 },
  ...
   {59, 2, 64 },
   {30, 1, 32 },
   {0, 0, 0 }
  };

The tile panel

The method used to the tile (left) approach would look like this:

  private static JPanel colorPane = new JPanel()
    {
      public void paintComponent(Graphics gr)
      {
        ((Graphics2D)gr).setRenderingHint(RenderingHints.KEY_TEXT_ANTIALIASING,
                                          RenderingHints.VALUE_TEXT_ANTIALIAS_ON);      
        ((Graphics2D)gr).setRenderingHint(RenderingHints.KEY_ANTIALIASING,
                                          RenderingHints.VALUE_ANTIALIAS_ON);      
        int w = this.getWidth();
        int h = this.getHeight();
        
        paintColors(gr, w, h);

        gr.setColor(Color.black);
        gr.setFont(gr.getFont().deriveFont(40f));
        
        String str = "Some text on top";
        int strWidth  = gr.getFontMetrics(gr.getFont()).stringWidth(str);        
        gr.drawString(str, (w / 2) - (strWidth / 2), (h / 2) + (40 / 2));
      }
      
      private void paintColors(Graphics gr, int w, int h)
      {
        // Draw 7x7
        int wStep = (int)((float)w / 7f);
        int hStep = (int)((float)h / 7f);
        for (int hor=0; hor<7; hor++)
        {
          for (int vert=0; vert<7; vert++)
          {
            Color c = new Color(COLOR_VALUES[(hor * 7) + vert][0],
                                COLOR_VALUES[(hor * 7) + vert][1],
                                COLOR_VALUES[(hor * 7) + vert][2]);
            gr.setColor(c);
            gr.fillRect(hor * wStep, vert * hStep, wStep, hStep);
          }
        }        
      }
    };

The smoothed panel

The other approach is a bit less straightforward.
The way to proceed is to

• Generate an Image
• Apply a blur filter on it
• Display the image

The blur filter could look like this:

  public static BufferedImage blur(BufferedImage bimg, int matrixDim)
  {
    float blurMatrix[] = new float[matrixDim * matrixDim];
    for (int i=0; i< blurMatrix.length; i++)
      blurMatrix[i] = 1f / (float)(matrixDim * matrixDim);

    Kernel kernel = new Kernel(matrixDim, matrixDim, blurMatrix);
    BufferedImageOp blurFilter = new ConvolveOp(kernel, 
                                                ConvolveOp.EDGE_NO_OP,
                                                null);
    bimg = blurFilter.filter(bimg, null);
    return bimg;
  }

And it can be used as follow in the paintComponent method:

      public void paintComponent(Graphics gr)
      {
        ((Graphics2D)gr).setRenderingHint(RenderingHints.KEY_TEXT_ANTIALIASING,
                                          RenderingHints.VALUE_TEXT_ANTIALIAS_ON);      
        ((Graphics2D)gr).setRenderingHint(RenderingHints.KEY_ANTIALIASING,
                                          RenderingHints.VALUE_ANTIALIAS_ON);      
        int w = this.getWidth();
        int h = this.getHeight();
        
        // Create the image
        BufferedImage bufferedImage = new BufferedImage(w, h, BufferedImage.TYPE_INT_ARGB);       
        // Create a graphics contents on the buffered image
        Graphics2D g2d = bufferedImage.createGraphics();

        this.paintColors((Graphics)g2d, w, h);
        // Graphics context no longer needed so dispose it
        g2d.dispose();
        // Blur it
        bufferedImage = blur(bufferedImage, w / 7);
        // Display it
        gr.drawImage(bufferedImage, 0, 0, null);

        gr.setColor(Color.black);
        gr.setFont(gr.getFont().deriveFont(40f));
        
        String str = "Some text on top";
        int strWidth  = gr.getFontMetrics(gr.getFont()).stringWidth(str);        
        gr.drawString(str, (w / 2) - (strWidth / 2), (h / 2) + (40 / 2));
      }

And that's it!
It's fully compatible with transparency,

  g2d.setComposite(AlphaComposite.getInstance(AlphaComposite.SRC_OVER, 0.5f));

But beware, depending on the size of the Kernel matrix, the blur operation can be pretty demanding.

Time and Date management in Java

Java is very well equipped to manage dates and times. Some utility classes will make your live much easier than if you had to do it from scratch. There are a couple of things to understand first, though.

This problem has several different aspects:

Astronomical aspects

• For a given date and time in location A, what time is it in location B?
• What is the UTC time, now?

Legal aspects

• For a given UTC time, what legal time is it in location X?
• What is the first week of the year in country A?
• What is the first week of the year in country B?

Misc aspects

• What is the difference between Time Zone and Time Offset?

The following Objects are used to manage time in Java:

• java.util.Date
• java.util.Calendar
• java.text.SimpleDateFormat
• java.util.TimeZone

The TimeZone can be used with

• the format
• the calendar

This is probably the most important thing to remember here:
When you set the TimeZone at the Calendar level, you are setting the date at a given TimeZone. For example, March 1^st 2012 at 12:00 Pacific Time.
When you set the TimeZone at the Format level, you do so to display the date & time set previously as viewed from another time zone. For example you set the date as above (US West Coast), and you want to view it from the US East Coast. Notice that in that case, the date remains unchanged. Only the way to display it is modified.

Note:

We are explicitly setting the TimeZones here, for clarity. If not explicitly set, the default TimeZone is used.

  private final static SimpleDateFormat SDF = new SimpleDateFormat("EEE dd MMMM yyyy HH:mm:ss.SSS Z (z)");    
    ...
    Calendar cal = Calendar.getInstance();
    cal.setTimeZone(TimeZone.getTimeZone("America/Los_Angeles"));
    cal.set(Calendar.YEAR, 2012);
    cal.set(Calendar.MONTH, Calendar.MARCH);
    cal.set(Calendar.DAY_OF_MONTH, 1);
    cal.set(Calendar.HOUR_OF_DAY, 12);
    cal.set(Calendar.MINUTE, 0);
    cal.set(Calendar.SECOND, 0);
    cal.set(Calendar.MILLISECOND, 0);
    SDF.setTimeZone(TimeZone.getTimeZone("America/New_York"));
    System.out.println("New York:     \t Date is " + SDF.format(cal.getTime()));    
    SDF.setTimeZone(TimeZone.getTimeZone("America/Los_Angeles"));
    System.out.println("San Francisco:\t Date is " + SDF.format(cal.getTime()));    

Output:

 New York:       Date is Thu 01 March 2012 15:00:00.000 -0500 (EST)
 San Francisco:  Date is Thu 01 March 2012 12:00:00.000 -0800 (PST)

More examples will be given below.

The default time zone is found from the TimeZone Object:

    TimeZone tz = TimeZone.getDefault();
    System.out.println("Default Time Zone:" + tz.getID() + ", " + tz.getDisplayName());

Output:

 Default Time Zone:America/Los_Angeles, Pacific Standard Time

The TimeZone object will also provide the list of all the valid time zones.

    String[] ids = TimeZone.getAvailableIDs();
    for (String s : ids)
      System.out.println(s);

Output:

 Etc/GMT+12
 Etc/GMT+11
 MIT
 Pacific/Apia
 Pacific/Midway
 Pacific/Niue
 Pacific/Pago_Pago
 Pacific/Samoa
 US/Samoa
 America/Adak
 America/Atka
 Etc/GMT+10
 HST
 Pacific/Fakaofo
 Pacific/Honolulu
 Pacific/Johnston
 Pacific/Rarotonga
 Pacific/Tahiti
 SystemV/HST10
 US/Aleutian
 ...

Notice in the list above the different kind of time zones:

• Etc/GMT+12
• PDT
• America/Los_Angeles

• Etc/GMT+12 refers to a fixed time offset, of 12 hours in that case
• PDT refers to the time in use during the winter on the US West Coast
• America/Los_Angeles refers to a location where the time in use can be PDT and PST. The time offset is different for those two times.

Let us illustrate this last point, we will display two dates, March first and June first:

  private final static SimpleDateFormat SDF = new SimpleDateFormat("EEE dd MMMM yyyy HH:mm:ss.SSS Z (z)");

  ...
    // March 1st 2012
    Calendar cal = GregorianCalendar.getInstance();
    SDF.setTimeZone(TimeZone.getTimeZone("America/Los_Angeles"));
    cal.set(Calendar.YEAR, 2012);
    cal.set(Calendar.MONTH, Calendar.MARCH);
    cal.set(Calendar.DAY_OF_MONTH, 1);
    System.out.println("Pacific Coast:\t Date is " + SDF.format(cal.getTime()));
    
    // June 1st 2012
    cal.set(Calendar.YEAR, 2012);
    cal.set(Calendar.MONTH, Calendar.JUNE);
    cal.set(Calendar.DAY_OF_MONTH, 1);
    System.out.println("Pacific Coast:\t Date is " + SDF.format(cal.getTime()));    

Output:

 Pacific Coast:  Date is Thu 01 March 2012 00:00:00.000 -0800 (PST)
 Pacific Coast:  Date is Fri 01 June 2012 00:00:00.000 -0700 (PDT)

Notice the Time in use, and the Time Offset. The Time Zone remains the same.
Some Time Zones use Daylight saving, other do not:

    cal.set(Calendar.YEAR, 2012);
    cal.set(Calendar.MONTH, Calendar.MARCH);
    cal.set(Calendar.DAY_OF_MONTH, 1);
    SDF.setTimeZone(TimeZone.getTimeZone("America/Los_Angeles"));
    System.out.println("Pacific Coast:\t Date is " + SDF.format(cal.getTime()));
    SDF.setTimeZone(TimeZone.getTimeZone("Pacific/Honolulu"));
    System.out.println("Hawaii:\t Date is " + SDF.format(cal.getTime()));

    cal.set(Calendar.YEAR, 2012);
    cal.set(Calendar.MONTH, Calendar.JUNE);
    cal.set(Calendar.DAY_OF_MONTH, 1);
    SDF.setTimeZone(TimeZone.getTimeZone("America/Los_Angeles"));
    System.out.println("Pacific Coast:\t Date is " + SDF.format(cal.getTime()));    
    SDF.setTimeZone(TimeZone.getTimeZone("Pacific/Honolulu"));
    System.out.println("Hawaii:\t Date is " + SDF.format(cal.getTime()));

Output:

 Pacific Coast:  Date is Thu 01 March 2012 00:00:00.000 -0800 (PST)
 Hawaii:         Date is Wed 29 February 2012 22:00:00.000 -1000 (HST)
 Pacific Coast:  Date is Fri 01 June 2012 00:00:00.000 -0700 (PDT)
 Hawaii:         Date is Thu 31 May 2012 21:00:00.000 -1000 (HST)

Hawaiian Standard Time remains the same, Pacific Time goes form Daylight to Standard.
The TimeZone can be set at the Format level, or at the Calendar level:

    Calendar cal = GregorianCalendar.getInstance();
    System.out.println("Cal:" + cal.getTime());
    System.out.println("Default Time zone:\t Date is " + SDF.format(cal.getTime()));
    // Set the time zone on the FORMAT to display the UTC time
    SDF.setTimeZone(TimeZone.getTimeZone("Etc/UTC"));
    System.out.println("Mask set to UTC:\t Date is " + SDF.format(cal.getTime()));
    
    // Set the TimeZone at the DATE level
    System.out.println("HOUR is     " + cal.get(Calendar.HOUR_OF_DAY) + ", before changing the Date's TZ");
    cal.setTimeZone(TimeZone.getTimeZone("Etc/UTC"));
    System.out.println("HOUR is now " + cal.get(Calendar.HOUR_OF_DAY));
    System.out.println("Date changed to UTC:\t Date is " + SDF.format(cal.getTime()));
    // The date remains the same.
    System.out.println("Cal:" + cal.getTime());

Output:

 Cal:Tue Jun 26 10:26:35 PDT 2012
 Default Time zone:    Date is Tue 26 June 2012 10:26:35.606 -0700 (PDT)
 Mask set to UTC:      Date is Tue 26 June 2012 17:26:35.606 +0000 (UTC)
 HOUR is     10, before changing the Date's TZ
 HOUR is now 17
 Date changed to UTC:  Date is Tue 26 June 2012 17:26:35.606 +0000 (UTC)
 Cal:Tue Jun 26 10:26:35 PDT 2012

Notice above that changing the Date's TimeZone automatically adjusts the offsets (the HOUR_OF_DAY in that case). Only the TimeZone is changed, not the actual original date, it still describes the same instant, the exact same point in time.

For a given UTC Time, what time is it in San Francisco, and in Hawai'i:

    Calendar cal = GregorianCalendar.getInstance();
    cal.setTimeZone(TimeZone.getTimeZone("Etc/UTC")); 
    cal.set(Calendar.YEAR, 2012);
    cal.set(Calendar.MONTH, Calendar.MARCH);
    cal.set(Calendar.DAY_OF_MONTH, 1);
    cal.set(Calendar.HOUR_OF_DAY, 12);
    cal.set(Calendar.MINUTE, 0);
    cal.set(Calendar.SECOND, 0);
    cal.set(Calendar.MILLISECOND, 0);
    SDF.setTimeZone(TimeZone.getTimeZone("Etc/UTC"));
    System.out.println("UTC:\t Date is " + SDF.format(cal.getTime()));    
    SDF.setTimeZone(TimeZone.getTimeZone("America/Los_Angeles"));
    System.out.println("San Francisco:\t Date is " + SDF.format(cal.getTime()));    
    SDF.setTimeZone(TimeZone.getTimeZone("Pacific/Honolulu"));
    System.out.println("Hawaii:\t Date is " + SDF.format(cal.getTime()));    

In the code, we set the TimeZone at the Calendar level, meaning that we set the date at March 1^st, 2012, at 12:00 UTC. Then we display it, viewed from "Etc/UTC", "America/Los_Angeles", and "Pacific/Honolulu".

 UTC:            Date is Thu 01 March 2012 12:00:00.000 +0000 (UTC)
 San Francisco:  Date is Thu 01 March 2012 04:00:00.000 -0800 (PST)
 Hawaii:         Date is Thu 01 March 2012 02:00:00.000 -1000 (HST)

Also, notice something important: some TimeZones do not have the same TimeOffsets all over the year. The Time Offset is automatically adjusted when you use a TimeZone subject to daylight saving.

You can manipulate the Calendar directly, without having to use a format. Let's assume you've set - like above - a Calendar at 12:00 UTC, on March 1^st 2012. And you want to know the hour of the day on the US West Coast:

    cal.setTimeZone(TimeZone.getTimeZone("America/Los_Angeles"));
    System.out.println(cal.get(Calendar.YEAR) + "-" + 
                      (cal.get(Calendar.MONTH) + 1) + "-" +
                       cal.get(Calendar.DAY_OF_MONTH) + ", " + 
                       cal.get(Calendar.HOUR_OF_DAY) + " hour(s).);

Output:

 2012-3-1, 4 hour(s).

Notice that the months index begins with 0 (we had to add 1).

An aspect of the Country, the legal time and time zone

In some countries, the week begins on Sundays. In some others, it begins on Mondays.
Let us run the following code different ways:

    Calendar cal = new GregorianCalendar(2012, Calendar.JANUARY, 1);
    SimpleDateFormat sdf = new SimpleDateFormat("EEEE dd-MMMM-yyyy '\tis in week' w");    
    System.out.println(sdf.format(cal.getTime()));

The US way:

 Prompt> java -Duser.language=en -Duser.country=US sometests.FirstWeekTest
 Sunday 01-January-2012  is in week 1

The British way:

 Prompt> java -Duser.language=en -Duser.country=GB sometests.FirstWeekTest
 Sunday 01-January-2012  is in week 52

The French way:

 Prompt> java -Duser.language=fr -Duser.country=FR sometests.FirstWeekTest
 dimanche 01-janvier-2012  is in week 52

The Swedish way:

 Prompt> java -Duser.language=sv -Duser.country=SE sometests.FirstWeekTest 
 söndag 01-januari-2012  is in week 52

Sunday is the first day of the week in the US, but not in Europe. As a result, the number of the week is affected! Week #1 is the first one to begin in the year.
In Europe, January 1^st 2012 is part of the last week of 2011. This last week's number is 53 (yes 53, this is not a typo) in the US, 52 in Europe.
This can be something to consider when making cross-continent appointments, like in "Let us meet on week 17".

Never underestimate date and time management, it can be a quite complex problem, to say the least.
At the end of the XIX^th century, a clerk was affected to the synchronization of the clocks on the railroad network of Switzerland.
He was later awarded the Nobel Price of Physics in 1921.
His name was Albert Einstein...