Elapsed time in a Windows Shell script

That is easy to do in bash (just add time in front of the command you want to time), and it is also possible to do it in DOS. It just took me about 70 lines of script ;0) Here is an example:

 @setlocal
 @echo off
 set begintime=%time%
 @echo -----------------------------------------
 @echo Starting: %begintime% ...
 set /p var=Tell me something and hit [return] ^> 
 @echo -----------------------------------------
 set finishtime=%time%
 call :timediff %begintime% %finishtime%
 ::@echo.
 @echo Done at %finishtime%, it took you %timetaken% sec to tell me [%var%].
 @echo -----------------------------------------
 goto eos
 ::
 :timediff
 :: echo %1, %2
 set starttime=%1
 set endtime=%2
 if [%starttime:~1,1%] == [:] set starttime=0%starttime%
 if [%endtime:~1,1%] == [:] set endtime=0%endtime%
 set startcsec=%starttime:~9,2%
 set startsecs=%starttime:~6,2%
 set startmins=%starttime:~3,2%
 set starthour=%starttime:~0,2%
 set /a starttime=(%starthour%*60*60*100)+(%startmins%*60*100)+(%startsecs%*100)+(%startcsec%) 
 ::
 set endcsec=%endtime:~9,2%
 set endsecs=%endtime:~6,2%
 set endmins=%endtime:~3,2%
 set endhour=%endtime:~0,2%
 if %endhour% LSS %starthour% set /a endhour+=24
 set /a endtime=(%endhour%*60*60*100)+(%endmins%*60*100)+(%endsecs%*100)+(%endcsec%)
 set /a timetaken= ( %endtime% - %starttime% )
 set /a timetakens= %timetaken% / 100
 set timetaken=%timetakens%.%timetaken:~-2%
 goto eos
 :: End Of Script
 :eos
 @endlocal

Notice the code under the timediff label, this is where the skill is. It takes the time before and the time after as parameters, and computes the value of a timetaken variable that you can display.
It is all based on the value of the %time% variable:

 Prompt> echo %time%
 10:44:29.25

and also on the substring manipulation, performed with the ":~" characters.
It will generate an output like this one:

 -----------------------------------------
 Starting: 10:40:02.63 ...
 Tell me something and hit [return] > Whatever
 -----------------------------------------
 Done at 10:40:11.17, it took you 8.54 sec to tell me [Whatever].
 -----------------------------------------

There is a gotcha, though. A problem remains, when you write something like this:

 set /a num=(010 + 010)

Windows comes up with

 16

It is because it considers that 010 is in octal, as such, understood as 8 (decimal). Whatever begins with a '0' is an octal digit, so '08', '09', etc, are invalid numbers.
In the calculation above, you need to remove the leading '0' for the computation to be done in decimal...
Being aware of this, the script becomes (its timediff part, and on):

 :timediff
 :: echo from [%1] to [%2]
 set starttime=%1
 set endtime=%2
 if [%starttime:~1,1%] == [:] set starttime=0%starttime%
 if [%endtime:~1,1%] == [:] set endtime=0%endtime%
 set startcsec=%starttime:~9,2%
 set startsecs=%starttime:~6,2%
 set startmins=%starttime:~3,2%
 set starthour=%starttime:~0,2%
 :: Remove leading 0 (considered as octal numbers)
 call :removeLeadingZero %startcsec%
 set startcsec=%truncated%
 call :removeLeadingZero %startsecs%
 set startsecs=%truncated%
 call :removeLeadingZero %startmins%
 set startmins=%truncated%
 call :removeLeadingZero %starthour%
 set starthour=%truncated%
 ::
 set /a starttime=(%starthour%*60*60*100)+(%startmins%*60*100)+(%startsecs%*100)+(%startcsec%) 
 ::
 set endcsec=%endtime:~9,2%
 set endsecs=%endtime:~6,2%
 set endmins=%endtime:~3,2%
 set endhour=%endtime:~0,2%
 :: Remove leading 0 (considered as octal numbers)
 call :removeLeadingZero %endcsec% 
 set endcsec=%truncated%
 call :removeLeadingZero %endsecs%
 set endsecs=%truncated%
 call :removeLeadingZero %endmins%
 set endmins=%truncated%
 call :removeLeadingZero %endhour%
 set endhour=%truncated%
 ::
 if %endhour% LSS %starthour% set /a endhour+=24
 set /a endtime=(%endhour%*60*60*100)+(%endmins%*60*100)+(%endsecs%*100)+(%endcsec%)
 set /a timetaken= ( %endtime% - %starttime% )
 set /a timetakens= %timetaken% / 100
 set timetaken=%timetakens%.%timetaken:~-2%
 goto eos
 ::
 :removeLeadingZero
 set truncated=%1
 :top
 if not [%truncated:~1%] == [] (
   if [%truncated:~0,1%] == [0] (
     set truncated=%truncated:~1%
     goto top
   )
 )
 :: if not [%1] == [%truncated%] @echo [%1] becomes [%truncated%]
 goto eos
 :: End Of Script
 :eos
 @endlocal

Notice the new sub-routine removeLeadingZero. It leaves at least one character in the string to truncate ('0' is the same in octal and decimal). The statement if not [%truncated:~1%] == [] (... means "if the length of %truncated% is greater than 1". The statement

 if [%starttime:~1,1%] == [:] set starttime=0%starttime%

is here to manage the times before 10:00. It's returned like

  9:55:10.34

That makes the first ':' in position 2, instead of 3, as expected. This last line in transformed so it's saying

  09:55:10.34

This way, all is good!
Now, if you don't want to copy-paste this code every time you want to time a command, start the script this way:

 @setlocal
 @echo off
 set begintime=%time%
 @echo -----------------------------------------
 @echo Starting: %begintime% ...
 @echo -----------------------------------------
 :: Execution to time goes here
 %*
 ::
 @echo -----------------------------------------
 set finishtime=%time%
 call :timediff %begintime% %finishtime%
 ::@echo.
 @echo Done at %finishtime%, execution of [%*] took %timetaken% sec.
 @echo -----------------------------------------
 goto eos
 ::
 :timediff
 :: echo from [%1] to [%2]
   ...

Name it something like "elapsed.cmd", and run it this way:

 Prompt> elapsed java -version
 -----------------------------------------
 Starting:  9:24:08.48 ...
 -----------------------------------------
 java version "1.6.0_20"
 Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
 Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)
 -----------------------------------------
 Done at  9:24:08.63, execution of [java -version] took 0.15 sec.
 -----------------------------------------

And that's it. The full script is 74 lines long, including comments. Here is the full script:

@setlocal
@echo off
@echo Usage:
@echo   %0 ^<command ^<prm1 ^<prm2 ^<...^>^>^>
set begintime=%time%
@echo -----------------------------------------
@echo Starting: %begintime% ...
@echo -----------------------------------------
@echo %CD%^> %*
:: Execution to time goes here
%*
::
@echo -----------------------------------------
set finishtime=%time%
call :timediff %begintime% %finishtime%
::@echo.
@echo Done at %finishtime%, execution of [%*] took %timetaken% sec.
@echo -----------------------------------------
goto eos
::
:timediff
:: echo from [%1] to [%2]
set starttime=%1
set endtime=%2
if [%starttime:~1,1%] == [:] set starttime=0%starttime%
if [%endtime:~1,1%] == [:] set endtime=0%endtime%
set startcsec=%starttime:~9,2%
set startsecs=%starttime:~6,2%
set startmins=%starttime:~3,2%
set starthour=%starttime:~0,2%
:: Remove leading 0 (considered as octal numbers)
call :removeLeadingZero %startcsec%
set startcsec=%truncated%
call :removeLeadingZero %startsecs%
set startsecs=%truncated%
call :removeLeadingZero %startmins%
set startmins=%truncated%
call :removeLeadingZero %starthour%
set starthour=%truncated%
::
set /a starttime=(%starthour%*60*60*100)+(%startmins%*60*100)+(%startsecs%*100)+(%startcsec%)
::
set endcsec=%endtime:~9,2%
set endsecs=%endtime:~6,2%
set endmins=%endtime:~3,2%
set endhour=%endtime:~0,2%
:: Remove leading 0 (considered as octal numbers)
call :removeLeadingZero %endcsec%
set endcsec=%truncated%
call :removeLeadingZero %endsecs%
set endsecs=%truncated%
call :removeLeadingZero %endmins%
set endmins=%truncated%
call :removeLeadingZero %endhour%
set endhour=%truncated%
::
if %endhour% LSS %starthour% set /a endhour+=24
set /a endtime=(%endhour%*60*60*100)+(%endmins%*60*100)+(%endsecs%*100)+(%endcsec%)
set /a timetaken= ( %endtime% - %starttime% )
set /a timetakens= %timetaken% / 100
set timetaken=%timetakens%.%timetaken:~-2%
goto eos
::
:removeLeadingZero
set truncated=%1
:top
if not [%truncated:~1%] == [] (
  if [%truncated:~0,1%] == [0] (
    set truncated=%truncated:~1%
    goto top
  )
)
:: if not [%1] == [%truncated%] @echo [%1] becomes [%truncated%]
goto eos
:: End Of Script
:eos
@endlocal
    

Technology vs Skills & Intelligence

Maleskine vs Harrison.
The full story can be read in Dava Sobel's book, "Longitude", about this contest set by the British Admiralty, after a substantial number of war ships were lost on their way back from a naval battle (1707), where they had defeated the French navy.
Those ships were lost, mainly because of the lack of mean to evaluate the longitude, they wrecked on the Scilly Islands, thinking they were a couple of miles more east. Everyone already knew that the knowledge of the longitude relied on the exact knowledge of the time. Four seconds of time mean one second of longitude, and one second of arc on a meridian is - by definition - a nautical mile.

Another side effect of this lack of means to correctly estimate the longitude was the following one:
On a trip from east to west, for example from Europe to the West Indies, in order not to miss their destination, the ships used to reach as soon as possible the latitude of the port they were targeting. Once the right latitude was reached, they kept sailing due west. This way, even if they could not tell exactly what their longitude was, whenever they would make landfall, it would be close to their destination. And here is the trick. Knowing more or less where those ships were going, pirates and privateers just had to wait for them somewhere along the way, those big clumsy ships were for them like low-hanging fruits, just waiting to be picked-up... The safe route was risky because of the pirates, another one was risky because it was unknown and unsafe...
(About pirates and others villains, do check out the remarkable work of Marcus Rediker).

In her book, Dava Sobel seems to prefer John Harrison to Neville Maleskine - and I will not argue with that, but beside the political aspect of that situation, something interesting shows up, like a watermark..., this is not the first thing you see, but it becomes obvious after a while, as a first class part of the whole picture.

John Harrison came up with several very elaborated clocks and watches, real pieces of art, still visible in the Greenwich Royal Observatory. His idea was to build the most accurate time keeping piece.

The approach Neville Maleskine took was drastically different. To him, the accuracy of such a device was out of reach. He came up with a method to know by how much your time keeper is off. It relies on the moon position in the sky. As a matter of fact, the moon has a very special behavior. All celestial bodies have an apparent movement going from east to west. The sun, the planets, the stars, and the moon roam the sky from east to west. The moon though, behaves in a unique manner, the day after, it's at a location it would have had earlier the day before. From one day to the next, the moon moves backwards, west to east. This is what Maleskine used. He considered the sky as a clock-face, where the numbers are the stars, and the hand is the moon. He relied on a celestial clock, not a man-made one. Then by measuring the distance between a celestial body (as close as possible to the ecliptic) and the moon, you could calculate what "Celestial Time" it is.
On top of that, the observation of this distance (such a distance is an angle, measured with a sextant) does not depend on the position of the observer on the Earth.

Both Harrison's and Maleskine's methods do work.
Harrison's approach is much faster. You read the clock, then you know what time it is, period. The calculations involved in Maleskine's method are huge, and as a result, time consuming, and an error might very well sneak in.

But let's take a step backward.
Maleskine's method requires the observer to have solid mathematical background. You need to be fluent in trigonometry to at least apprehend the rationale. You also need to rely on some almanac, which you take at sea with you. If you are OK with that, you're all set, and like riding a bike, this is the kind of skills you'll never loose.

Harrison's path, by far the easiest for the user, happened to have been the most successful up to this day. The time keepers kept improving, we now have GPSs, atomic clocks, all kinds of instruments, all available at sea. Maleskine's tables stopped to be published in 1911, when radio beeps begun to be emitted for any ship equipped with a radio to have the accurate time.
Celestial Navigation is no longer taught in most naval academies. Now, to know where you are (not even to know what time it is), you push a button, or read a digital screen.

In short, Maleskine's method requires the user to be aware of the elaboration of the rationale, in its every smallest details.
Harrison's one just asks the user to be able to read a display; the machine behind the scene takes care of everything else.

An interesting point of view to mention: What if it fails?
If Maleskine's method fails, you can try to recalculate, this is probably where the problem comes from.
If Harrison's method fails, you must be able to repair the clock...

I'm not 100% comfortable with this idea of relying on a machine I did not build myself (and same for the ones I built). I very well know that those machines can shut down in no time, with no warning. This is a systemic feature of whatever we call "machine".

Remember the first Gulf War? The US Army - who owned the GPS satellites - decided to shut down the public access to their data. Just like that, boom. Some times, even the army can be snappy...

Whoever was at sea, using a GPS did indeed notice.
Whoever was at sea, using another method, kept sailing without noticing...

Navigation aids should bear a label saying something like "Use responsibly". Always have a backup plan.

Self sufficiency, freedom, all this takes time, energy, effort; it is not always easy, nor relaxing, it is always demanding. Freedom is no vacation. But that is so rewarding when it works! So much more rewarding than owning the last cutting edge piece of technology, iStuff or what not.

I'm not spitting in the soup. I'm just trying to identify what I really need, to reach my goals.
And what I don't.

Select for update skip locked

In SQL, there is this possibility to lock the rows you select, warning this way the rest of the world that you may very well modify them.
Let's say you want to reserve the 10 first rows of the EMPLOYEES table of your HR schema.
In Oracle's SQL*Plus, this would look like this:

 SQL> select employee_id from employees where rownum <= 10 for update skip locked;

This would return a report like this:

The problem is that, when we issue the same command from another process or thread, no row is returned, even if unlocked rows are available in the table:

The reason for that is that the 10 first records are already locked, and identified by their rownum.

The idea is to get the first unlocked records, and then to lock them. Let's try the following PLSQL block:

set serveroutput on

 declare
   counter integer := 0;
   cursor emp_curs is
     select employee_id from employees for update skip locked;
   emp_row emp_curs%ROWTYPE;
   emp_id employees.employee_id%TYPE;
 begin
   open emp_curs;
   loop
     fetch emp_curs into emp_row;
     exit when emp_curs%NOTFOUND;
     counter := counter + 1;
     select employee_id into emp_id from employees where employee_id = emp_row.employee_id for update;
     dbms_output.put_line('-> Locked: ' || to_char(emp_id));
     if counter >= 10 then
       exit;
     end if;
   end loop;
 end;
 /

We just display the rows we want to lock.
We run it:

Records with EMPLOYEE_ID 100 to 109 are selected, and locked.
Now, before releasing the lock, let's run it from another process or thread:

That works, records with EMPLOYEE_ID 110 to 119 are selected.
Etc, etc.

In our case, we used EMPLOYEE_ID as the key for the records. We could have as well used the ROWID. It also uniquely identifies the records to lock, and this in any case. In that case, the script becomes:

declare
   counter integer := 0;
   cursor emp_curs is
     select e.*, rowid from employees e for update skip locked;
   emp_row emp_curs%ROWTYPE;
   emp_id employees.employee_id%TYPE;
 begin
   open emp_curs;
   loop
     fetch emp_curs into emp_row;
     exit when emp_curs%NOTFOUND;
     counter := counter + 1;
     select employee_id into emp_id from employees where rowid = emp_row.rowid for update;
     dbms_output.put_line('-> Locked: ' || to_char(emp_id));
     if counter >= 10 then
       exit;
     end if;
   end loop;
 end;
 /

It is even possible to use dynamic SQL to do this job (this is on another schema, as you can tell):

set serveroutput on size 3000
 set ver off
 set timing on
 accept nbrows2lock prompt 'Max Nb of Rows to Lock > '
 -- +---------------------------------------------------
 declare
   maxReturnedRows integer   := &nbrows2lock;
   table_name varchar2(128)  := 'QUEUE_2100';
   vch_tsolabel varchar2(64) := 'MANAGE_USER'; 

   counter integer             := 0;
   sql_stmt_One varchar2(4096) := '';
   vch_hint varchar2(128)      := '/*+ Rule */';
  
   tsoidvar varchar2(60);
   orderidvar varchar2(60);
   msisdnvar varchar2(60);
   imsivar varchar2(60);
   priorityvar number(4,0);
   instmsdt date;
   row_id rowid;

   type QueueCursorType is ref cursor;
   queue_curs_var QueueCursorType;     
  
   type recordType is record
   (
     row_id      rowid,
     tsoidvar    varchar2(60),
     orderidvar  varchar2(60),
     msisdnvar   varchar2(60),
     imsivar     varchar2(60),
     priorityvar number(4,0),
     instmsdt    date
   );
   recordHolder recordType;
   type tableOfRecord is table of recordType index by binary_integer;
   tableHolder tableOfRecord;
   i integer := 1; -- Index for tableOfRecord
 begin
   sql_stmt_One := 'SELECT ' || vch_hint || ' rowid, '                         ||
                                            ' t.tsoid, '                       ||
                                            ' nvl(t.orderid, '''') orderid, '  ||
                                            ' nvl(t.msisdn, '''')  msisdn, '   ||
                                            ' nvl(t.imsi, '''')    imsi, '     ||
                                            ' nvl(t.priority, 1)   priority, ' ||
                                            ' t.ins_tms  '                     ||
                   'FROM ' || table_name || ' t ' ||
                   'WHERE t.bp_flag = 0 and '     ||
                         't.tsolabel = :1 '       ||
                   'for update skip locked';
                  
   open queue_curs_var for sql_stmt_One using vch_tsolabel;
   loop
     -- --------------------------------------------------------------
     -- The trick is to have an 'exit when' before and after the fetch
     -- --------------------------------------------------------------
     exit when counter = maxReturnedRows;
     fetch queue_curs_var 
     into row_id,
          tsoidvar, 
          orderidvar,
          msisdnvar,
          imsivar,
          priorityvar, 
          instmsdt;
     exit when queue_curs_var%NOTFOUND;
     counter := counter + 1;
     dbms_output.put_line('Locking row #' || to_char(counter));
     --
     -- for tests. Concurrent locks...
     --
     if false then -- set to true to wait
       dbms_lock.sleep(1); -- Needs grant execute from SYS
     end if;
     --
     recordHolder.row_id      := row_id;
     recordHolder.tsoidvar    := tsoidvar;
     recordHolder.orderidvar  := orderidvar;
     recordHolder.msisdnvar   := msisdnvar;
     recordHolder.imsivar     := imsivar;
     recordHolder.priorityvar := priorityvar;
     recordHolder.instmsdt    := instmsdt;
     tableHolder(i) := recordHolder;
     i := i + 1;
   end loop;
   close queue_curs_var;
   dbms_output.put_line('Done with the array, ' || to_char(counter) || ' entry(ies)');
   if counter > 0 then
     -- Loop on the selected values to process them
     for i in tableHolder.FIRST..tableHolder.LAST loop
       dbms_output.put_line ('Processing [' || tableHolder(i).tsoidvar || ']');
       -- Processing takes place here
       execute immediate 
       'update ' || table_name || ' t ' ||
       'set t.bp_flag = 1 ' ||
       'where rowid = :1'
       using tableHolder(i).row_id;
     end loop;
   else
     dbms_output.put_line('No record to manage');
   end if;
 exception
   when others then
     dbms_output.put_line('ERR:' || sqlerrm);
 end;
 /
 show errors

We are actually using here the same kind of technique as above, having the SQL generated dynamically does not make much difference. We are using the ROWID approach. Notice the following details:

• The exit when statement, before and after the fetch of the first cursor
• The way we populate a table of records, so we can handle it after putting the lock
• The dbms_lock.sleep, for tests, that allows to simulate what happens if two or more such blocks are executed at the same time. We actually see that it makes no problem, no row can be locked more than once. It is interesting to understand that, for the first cursor (that has a for update clause), the lock is actually set when the record is fetched. This explains the exit when mechanism mentionned above.

This being said, this block behaves like the one described at the top of this post, it selects a given number of unlocked rows from a table, this selection having the possibility to be done simultaneously from several processes or threads that do not have to wait for any lock to be released before returning the expected result.

Hanoi Tower

The backend algorithm is just a few lines big..., it's all about rendering.
This browser does not support Applets. It could be a matter of authorization. Or install Java or use another browser...
The core code is pretty cool:

 public class BackendAlgorithm
 {
   public static void move(int n, String from, String to, String using)
   {
     if (n == 0)
       return;
     move(n-1, from, using, to);
     HanoiContext.getInstance().fireMoveRequired(from, to); // Broadcast the move
     move(n-1, using, to, from);
   }
 }

Again, it's mostly about rendering... 99% of the code of the applet above is for the graphical display!
The simplest user interface would look like this:

 package hanoitower;
 
 public class BackendAlgorithm
 {
   public static void move(int n, String from, String to, String using)
   {
     if (n == 0)
       return;
     move(n-1, from, using, to);
     System.out.println("Moving from " + from + " to " + to);
     move(n-1, using, to, from); 
   }
  
   public static void main(String[] args) throws Exception
   {
     move(Integer.parseInt(args[0]), "A", "C", "B"); // Moving A to C using B
   }
 }

Run it this way:

 Prompt> java hanoitower.BackendAlgorithm 4
 Moving from A to B
 Moving from A to C
 Moving from B to C
 Moving from A to B
 Moving from C to A
 Moving from C to B
 Moving from A to B
 Moving from A to C
 Moving from B to C
 Moving from B to A
 Moving from C to A
 Moving from B to C
 Moving from A to B
 Moving from A to C
 Moving from B to C

Boom!

Smooth moves with Swing

It's about data rendering in Swing. How to make a smooth rendering of real time data...
The secret seems to be in SwingUtilities.invokeAndWait

This browser does not support Applets.

Use the button to start or stop the applet

Above are two JPanels, displaying, on the left a speed, on the right a heading.
Both are fed by real time data (simulated in this case).
Obviously - and we will not show this code, unless anyone insists - the position of the display (namely the position of the hand) is done when the repaint() is invoked, and its implementation is done by overriding the paintComponent(Graphics) method.
The smoothing is done in a setValue(double) method.
When this method is invoked, the difference between the previous and the next values is splitted in small intervals, between which a repaint() is invoked.
All the difficulty - the problem - is that when a repaint() is invoked, maybe the previous one is not completed yet... This would result in a jerky display, far from being smooth, at the opposite of the expected result.
A solution is to tell Swing to wait, which happens to be shamefully simple... Here is the code for the speed display:

  public void setSpeed(final double d)
  {
    this.speed = d;
    double from = this.prevSpeed; 
    double to = d;
    
    int sign = (from>to)?-1:1;
    prevSpeed = d;
    // Smooth rotation
    for (double s=from; (sign==1 && s<=to) || (sign==-1 && s>=to); s+=(0.1*sign)) // 0.1 is the damping factor
    {
      final double _s = s;
      try
      {
        SwingUtilities.invokeAndWait(new Runnable()
          {
            public void run()
            {
              speed = _s;
              repaint();
            }
          });
      }
      catch (Exception ex)
      {
        ex.printStackTrace();
      }
    }
  }

The paintComponent(Graphics) method uses the speed member to position the hand on the display.
Notice the SwingUtilities.invokeAndWait method that takes a Runnable as a parameter. This Runnable is the one invoking the repaint() method. This way, Swing will do nothing until the repaint is completed, and it is not even necessary to worry about synchronizing anything...

See also SwingUtilities.invokeLater. It allows Swing to perform demanding tasks asynchronously.

Who just called me?

In Java, when you are in a specific method and you want to know who called you, who invoked the method, this is so simple!
A few lines will do the job:

  private void callee()
  {
    // Who called me
    Throwable t = new Throwable(); 
    StackTraceElement[] elements = t.getStackTrace(); 
    System.out.println("----------------------------------");
    for (StackTraceElement ste : elements)
      System.out.println(ste.toString());
    System.out.println("----------------------------------");
   
    // More stuff goes here... 
  }

Whenever the code above is invoked, like from a button click for example

  JButton bing = new JButton("Hit me!");
  bing.addActionListener(new ActionListener()
        {
          public void actionPerformed(ActionEvent e)
          {
            callee();
          }
        });

An output like that one can be expected:

----------------------------------
tideengineimplementation.gui.TideInternalFrame.resetData(TideInternalFrame.java:3035)
tideengineimplementation.gui.TideInternalFrame.refreshButton_actionPerformed(TideInternalFrame.java:3829)
tideengineimplementation.gui.TideInternalFrame.access$8200(TideInternalFrame.java:140)
tideengineimplementation.gui.TideInternalFrame$35.actionPerformed(TideInternalFrame.java:3002)
javax.swing.AbstractButton.fireActionPerformed(AbstractButton.java:1995)
javax.swing.AbstractButton$Handler.actionPerformed(AbstractButton.java:2318)
javax.swing.DefaultButtonModel.fireActionPerformed(DefaultButtonModel.java:387)
javax.swing.DefaultButtonModel.setPressed(DefaultButtonModel.java:242)
javax.swing.plaf.basic.BasicButtonListener.mouseReleased(BasicButtonListener.java:236)
java.awt.AWTEventMulticaster.mouseReleased(AWTEventMulticaster.java:272)
java.awt.Component.processMouseEvent(Component.java:6289)
javax.swing.JComponent.processMouseEvent(JComponent.java:3267)
java.awt.Component.processEvent(Component.java:6054)
java.awt.Container.processEvent(Container.java:2041)
java.awt.Component.dispatchEventImpl(Component.java:4652)
java.awt.Container.dispatchEventImpl(Container.java:2099)
java.awt.Component.dispatchEvent(Component.java:4482)
java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4577)
java.awt.LightweightDispatcher.processMouseEvent(Container.java:4238)
java.awt.LightweightDispatcher.dispatchEvent(Container.java:4168)
java.awt.Container.dispatchEventImpl(Container.java:2085)
java.awt.Window.dispatchEventImpl(Window.java:2478)
java.awt.Component.dispatchEvent(Component.java:4482)
java.awt.EventQueue.dispatchEventImpl(EventQueue.java:644)
java.awt.EventQueue.access$000(EventQueue.java:85)
java.awt.EventQueue$1.run(EventQueue.java:603)
java.awt.EventQueue$1.run(EventQueue.java:601)
java.security.AccessController.doPrivileged(Native Method)
java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:98)
java.awt.EventQueue$2.run(EventQueue.java:617)
java.awt.EventQueue$2.run(EventQueue.java:615)
java.security.AccessController.doPrivileged(Native Method)
java.security.AccessControlContext$1.doIntersectionPrivilege(AccessControlContext.java:87)
java.awt.EventQueue.dispatchEvent(EventQueue.java:614)
java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
----------------------------------

Easy!

serialVersionUID

Serialization is a powerful mechanism available in Java. It is - among many others - used by RMI (Remote Method Invocation), that can establish a communication between different JVMs.
A problem can appear when the class definition available on JVM#1 is not the same as the one available on JVM#2.
When defining a class implementing the java.io.Serializable interface, modern compilers will often reproach you with not correctly defining a final static long serialVersionUID.
We are briefly going to touch base on that, trying to explain what that means and what this is all about.

There is nothing like a good example to illustrate this kind of things. I hope you will like the following one.
Let us define a Serializable class:

public class Description
  implements Serializable
{
  @SuppressWarnings("compatibility:5602934367223767090")
  private static final long serialVersionUID = 1L;

  private String stringDataOne;
  private String stringDataTwo;
//private double numericData;
  private float numericData;

  public void setStringDataOne(String s)
  {
    this.stringDataOne = s;
  }

  public void setStringDataTwo(String s)
  {
    this.stringDataTwo = s;
  }

  public String getStringDataOne()
  {
    return this.stringDataOne;
  }

  public String getStringDataTwo()
  {
    return this.stringDataTwo;
  }

  public void setNumericData(float dummyNumber)
//public void setNumericData(double dummyNumber)
  {
    this.numericData = dummyNumber;
  }

  public float getNumericData()
//public double getNumericData()
  {
    return numericData;
  }
  
  @Override
  public String toString()
  {
    return new StringBuffer("One  : ").append(this.stringDataOne)
                  .append("\nTwo  : ").append(this.stringDataTwo)
                  .append("\nNum  : " + NumberFormat.getInstance().format(numericData)).toString();
  }
}

Notice the final static long serialVersionUID = 1;. We'll talk about it shortly.
This class being defined, we are going to:

1. Serialize an instance of it into a file
2. De-serialize this instance from the file

Serializer

public class WriteObject
{
  public static void main(String[] args)
  {
    Description dataObject = new Description();
    dataObject.setStringDataOne("First line of data");
    dataObject.setStringDataTwo("Second line, as expected");
    dataObject.setNumericData(12345);

    try
    {
      FileOutputStream fout = new FileOutputStream("description.ser");
      ObjectOutputStream oos = new ObjectOutputStream(fout);
      oos.writeObject(dataObject);
      oos.close();
      System.out.println("Done");
    }
    catch (Exception ex)
    {
      ex.printStackTrace();
    }
  }
}

Serializer at work:

 Done

De-serializer

public class ReadObject
{
  public static void main(String[] args)
  {
    Description restoredObject;

    try
    {
      FileInputStream fin = new FileInputStream("description.ser");
      ObjectInputStream ois = new ObjectInputStream(fin);
      restoredObject = (Description) ois.readObject();
      ois.close();

      System.out.println(restoredObject);
    }
    catch (Exception ex)
    {
      ex.printStackTrace();
    }
  }
}

De-serializer at work:

 One  : First line of data
 Two  : Second line, as expected
 Num  : 12,345

This is what happens when everything goes well.
Now, let's modify the Serializable class, let's change the float numeric data to a double (member, getter, setter), and without re-serializing it, let's try to de-serialize it:

java.io.InvalidClassException: serialversionui.Description; incompatible types for field numericData
 at java.io.ObjectStreamClass.matchFields(ObjectStreamClass.java:2210)
 at java.io.ObjectStreamClass.getReflector(ObjectStreamClass.java:2105)
 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:602)
 at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582)
 at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
 at serialversionui.ReadObject.main(ReadObject.java:17)

Basically, it says it cannot cast a Description to a Description...
If the serialVersionUID had been modified when changing the float to a double - like from 1 to 2 for example

  private static final long serialVersionUID = 2L;

you would have seen the following message:

java.io.InvalidClassException: serialversionui.Description; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = 2
 at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:562)
 at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582)
 at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
 at serialversionui.ReadObject.main(ReadObject.java:17)

It does not work any better, but it returns useful information to communicate with the development teams.

Java recommends a way to come up with the appropriate value for the serialVersionUID. This is the reason for the @SuppressWarnings annotation in the code snippets available above. Find it here.

Serializing data

Let's say you are developing Java code, and you have large objects - Maps and Lists - to read through in order to compute some result.
The problem, illustrating the point, was to load in memory all the data and coefficients for a Tidal Computation. Before you start doing any work, you have to read those data from somewhere, load them into your working memory, so they are available when you need them.
Several technologies are available to achieve this goal, the question here is to know which one to go for. Many different data storage techniques could be used. We will compare here XML, Java Serialization, and json (Java Script Object Notation). Notice that we are not here talking about Databases, which is certainly a possible approach. The rationale was to keep the solution as light and self-contained as possible, the data are to be read only (and not modified by any mean). We felt that in that case, the Database approach was overkilling.
XML itself can be approached in different ways, DOM, SAX, or StAX. As we don't wish here to modify any of the data to load, we will use SAX (Simple API for XML).
All the required data are stored into some files, which have to be loaded into objects at runtime.
So, to recapitulate before we get started, we will restrict our comparison to those three approaches:

• XML, using a SAX parser to load the data
• Java Serialization/Deserialization
• Json Serialization/Deserialization

In order to compare what is comparable (apples to apples), we stored the data files into an archive, a zip in this case.
Those data were then deserialized into the exact same objects.
The archive containing the XML data is 1,546,860 bytes big.
The archive containing the Java Serialized (.ser) data is 1,772,159 bytes big.
The archive containing the json data is 2,543,174 bytes big.
We iterated 5 times on the same deserializations, and then did an average.

Here is the code:

  public static void main(String[] args) throws Exception
  {
    long[] elapsed = { 0L, 0L, 0L };
    long before = 0L, after = 0L;
    BackEndTideComputer.setVerbose(false);
    for (int i=0; i<5; i++)
    {
      before = System.currentTimeMillis();
      BackEndTideComputer.connect(BackEndTideComputer.XML_OPTION);
      after = System.currentTimeMillis();
      elapsed[0] += (after - before);
      BackEndTideComputer.disconnect();
      
      before = System.currentTimeMillis();
      BackEndTideComputer.connect(BackEndTideComputer.JAVA_SERIALIZED_OPTION);
      after = System.currentTimeMillis();
      elapsed[1] += (after - before);
      BackEndTideComputer.disconnect();
  
      before = System.currentTimeMillis();
      BackEndTideComputer.connect(BackEndTideComputer.JSON_SERIALIZED_OPTION);
      after = System.currentTimeMillis();
      elapsed[2] += (after - before);
      BackEndTideComputer.disconnect();
    }
    System.out.println("XML:" + Long.toString(elapsed[0] / 5) + " ms" +
                     "  JavaSer:" + Long.toString(elapsed[1] / 5) + " ms" +
                     "  json:" + Long.toString(elapsed[2] / 5) + " ms");
  }

... and here are the results:

 XML:3540 ms  JavaSer:5455 ms  json:4225 ms

If we tune those results to take the data file size in account, by rendering the milliseconds per megabyte, we come up with the following table:

	XML	ser	json
archive size in bytes	1,546,860	1,772,159	2,543,174
elapsed time in ms	3,540	5,455	4,225
average ms/Mb	2340	3228	1742

The first to cross the line in the SAX XML Parser (I used the one from Oracle).
The best average award (ms/Mb) goes to json (I used Gson, the json parser from Google), but notice that that data we started from are the bigest.
Surprisingly, the red lantern remains the Java Serialization...

A note about JAX-B

We've not talked about JAX-B here, which stands for Java API for XML Binding. It sounds like it could deserve its place in this article... But it actually does not. The JAX-B process starts from an XML Schema, which we don't have in this case (OK, we could have built one. But we did not). This article talks about it.

So, we've been able to compare three approaches, and the "native" one seems to be the worse, which is a bit of a surprise.
This is probably enough to have a better idea of what technology to use to address this kind of problem, but as a bonus, let us take a look at the amount of code required to put them at work.

Java Deserializer

  public static <T> T loadObject(InputStream resource, Class<T> cl) throws Exception 
  {
    T tideObject = null;
    try
    {
      ObjectInputStream ois = new ObjectInputStream(resource);
      tideObject = (T)ois.readObject();
      ois.close();
    }
    catch (Exception ex)
    {
      throw ex;
    }
    return tideObject;
  }

JSON deserializer

  public static <T> T loadObject(InputStream resource, Class<T> cl) throws Exception 
  {
    T tideObject = null;
    try
    {
      BufferedReader br = new BufferedReader(new InputStreamReader(resource));
      StringBuffer sb = new StringBuffer();
      String line = "";
      boolean go = true;
      while (go)
      {
        line = br.readLine();
        if (line == null)
          go = false;
        else
          sb.append(line);
      }
      br.close();
      if (gson == null)
        gson = new GsonBuilder().setPrettyPrinting().create();
      tideObject = gson.fromJson(sb.toString(), cl);        
    }
    catch (Exception ex)
    {
      throw ex;
    }
    return tideObject;
  }

XML SAX Handler

Here is the code starting the SAX process

  public static Map getStationData() throws Exception
  {
    Map stationData = new HashMap();
    StationFinder sf = new StationFinder(stationData);
    try
    {
      SAXParserFactory factory = SAXParserFactory.newInstance();
      SAXParser saxParser = factory.newSAXParser();      
      InputSource is = BackEndTideComputer.getZipInputSource(ARCHIVE_STREAM, STATIONS_ENTRY);
      saxParser.parse(is, sf);       
    }
    catch (Exception ex)
    {
      ex.printStackTrace();
    }
    
    return stationData;
  }

Notice that this code invokes a custom handler, named StationFinder:

  public static class StationFinder extends DefaultHandler
  {
    private String stationName = "";
    private TideStation ts = null;
    private Map stationMap = null;
    
    public void setStationName(String sn)
    {
      this.stationName = sn;
    }
    
    public StationFinder()
    {
    }

    public StationFinder(Map map)
    {
      this.stationMap = map;
    }
    
    public TideStation getTideStation()
    {
      return ts;
    }

    private boolean foundStation        = false;
    private boolean foundNameCollection = false;
    private boolean foundStationData    = false;
    
    @Override
    public void startElement(String uri, String localName, String qName, Attributes attributes)
      throws SAXException
    {
//    super.startElement(uri, localName, qName, attributes);
      if (!foundStation && "station".equals(qName))
      {
        String name = attributes.getValue("name");
        if (name.contains(this.stationName))
        {
          foundStation = true;
          ts = new TideStation();
          ts.setFullName(name);
        }
      }
      else if (foundStation)
      {
        if ("name-collection".equals(qName))
        {
          foundNameCollection = true;
        }
        else if ("name-part".equals(qName) && foundNameCollection)
        {
          ts.getNameParts().add(attributes.getValue("name"));
        }
        else if ("position".equals(qName))
        {
          ts.setLatitude(Double.parseDouble(attributes.getValue("latitude")));
          ts.setLongitude(Double.parseDouble(attributes.getValue("longitude")));
        }
        else if ("time-zone".equals(qName))
        {
          ts.setTimeZone(attributes.getValue("name"));
          ts.setTimeOffset(attributes.getValue("offset"));
        }
        else if ("base-height".equals(qName))
        {
          ts.setBaseHeight(Double.parseDouble(attributes.getValue("value")));
          ts.setUnit(attributes.getValue("unit"));
        }
        else if ("station-data".equals(qName))
        {
          foundStationData = true;
        }
        else if (foundStationData && "harmonic-coeff".equals(qName))
        {
          String name = attributes.getValue("name");
          double amplitude = Double.parseDouble(attributes.getValue("amplitude"));
          double epoch     = Double.parseDouble(attributes.getValue("epoch")) * TideUtilities.COEFF_FOR_EPOCH;
          Harmonic h = new Harmonic(name, amplitude, epoch);
          ts.getHarmonics().add(h);
        }
      }
    }

The code snippets speak for themselves... SAX is the most demanding, by far. Also notice that it requires a full knowledge of the data to parse. Java and Json de-serialization do not. As long as the data have been correctly serialized, they will be correctly de-serialized (which brings up the discussion about the famous serialVersionUID in Java..., it's another story).

All the code, including the archives containing the data, is available from Google Code. The specific class displayed above is right here.

Functional Programming in Java

Functional Programming brings new and interesting paradigms to Java code development.
I am not saying that functional programming is a new concept, OK?
Several of those languages (Scala, Clojure) run on top of a Java VM, but Java itself is NOT as functional programming language..., how is that possible, there must be a way!
One of the aspects of the functional programming languages is this possibility to use a pointer on a function as a parameter. Here is a possible approach, even easier to access since the diamond notation has been implemented in the recent versions of the Java language.
We are going to use a simple example as an illustration. We will choose the function to execute, based on the value of a parameter. What we want to do here is to greet someone in the language passed as a parameter.

Note:

Only for the clarity of the code, we are not using here the usual Java convention for the diamond notation (T for Type, E for Element, K for Key, V for Value, N for Number, S, U, V, etc, for 2nd, 3rd, 4th,... types).

Let us consider a first Java interface defined as follow:

 package functions;

 public interface FunctionOneArgReturnVoid<SomeType>
 {
   public void execute(SomeType a);
 }

And let us write the following implementation:

 package functions;

 public class ImplementationSampleOne
 {
   FunctionOneArgReturnVoid<String> funcEnglish = 
                      new FunctionOneArgReturnVoid<String>()
    {
      public void execute(String s)
      {
        System.out.println("Hello " + s + "!");
      }
    };
   FunctionOneArgReturnVoid<String> funcFrench  = 
                      new FunctionOneArgReturnVoid<String>()
    {
      public void execute(String s)
      {
        System.out.println("Bonjour " + s + " !");
      }
    };

   public FunctionOneArgReturnVoid<String> getFunction(String lng)
   {   
     if ("FR".equalsIgnoreCase(lng))
       return funcFrench;
     else
       return funcEnglish;
   } 

   public void execute(FunctionOneArgReturnVoid<String> func, String arg)
   {
     func.execute(arg); 
   }

   public static void main(String[] args)
   {
     String lng = "EN";
     if (args.length > 0)
       lng = args[0]; 

     ImplementationSampleOne impl = new ImplementationSampleOne();
     impl.execute(impl.getFunction(lng), "Joe Shmow");
   }
 }

See the two implementations of the FunctionOneArgReturnVoid interface, and the way they are invoked in the main method.
The typography rules are not the same in English and French. In French you need a space before a '!'.
The execution would look like this:

 Prompt> java functions.ImplementationSampleOne "FR"
 Bonjour Joe Shmow !

 Prompt> java functions.ImplementationSampleOne "EN"
 Hello Joe Shmow!

The return type can be also described by an interface:

 package functions;

 public interface FunctionTwoArgsReturnType<SomeTypeOne, 
                                            SomeTypeTwo, 
                                            ReturnType>
 {
   public ReturnType execute(SomeTypeOne a, SomeTypeTwo b);
 }

The interface implementation would look like this:

  FunctionTwoArgsReturnType<String, String, String> funcEnglish = 
                 new FunctionTwoArgsReturnType<String, String, String>()
   {
     public String execute(String a, String b)
     {
       return "Hello " + a + ", from " + b + "!";
     }
   };
  FunctionTwoArgsReturnType<String, String, String> funcFrench  = 
                 new FunctionTwoArgsReturnType<String, String, String>()
   {
     public String execute(String a, String b)
     {
       String s = "Bonjour " + a + ", de la part ";
       if (b.toUpperCase().toCharArray()[0] == 'A' ||
           b.toUpperCase().toCharArray()[0] == 'E' ||
           b.toUpperCase().toCharArray()[0] == 'I' ||
           b.toUpperCase().toCharArray()[0] == 'O' ||
           b.toUpperCase().toCharArray()[0] == 'U' ||
           b.toUpperCase().toCharArray()[0] == 'Y')
         s += ("d'" + b);
       else
         s += ("de " + b);
       s += " !";
       return s;
     }
   };

The main like that:

  public static void main(String[] args)
  {
    String lng = "EN";
    if (args.length > 0)
      lng = args[0];

    ImplementationSampleTwo impl = new ImplementationSampleTwo();
    String greeting = impl.execute(impl.getFunction(lng), "Joe Shmow", "Olivier");
    System.out.println(greeting);
    greeting = impl.execute(impl.getFunction(lng), "Joe Shmow", "Mr X");
    System.out.println(greeting);
  }

The output would become

 Prompt> java functions.ImplementationSampleTwo "EN"
 Hello Joe Shmow, from Olivier!
 Hello Joe Shmow, from Mr X!

 Prompt> java functions.ImplementationSampleTwo "FR"
 Bonjour Joe Shmow, de la part d'Olivier !
 Bonjour Joe Shmow, de la part de Mr X !

The complexity is isolated in the implementation of the function. The code using it does not need to know anything about it.

This possibility of using pointers on functions does not make Java a functional programming language, but this is a first step towards it.

"Rich" console output from Java

A while back - before the graphical terminals were available, in the old VT time - we used to build forms from scratch, using ANSI escape sequences. That was quite limited, but it was not that ugly, some times...
Linux still supports this kind of thing, and from Java, you can definitely add any effect to the text you spit out using System.out or System.err.
Surprisingly, it's another story on Windows, you have to mention the ansi.sys file in the config.nt one, which magically disappeared in Windows 7.
There is a solution to this problem, at jansi.fusesource.org.
It's small library available for free, which allows you to write the usual ANSI codes, whatever the system you run on is.
You can define your own codes:

  public final static char ESC = '\u001b'; // (char) 27;

  public final static String ANSI_BLACK   = "0";
  public final static String ANSI_RED     = "1";
  public final static String ANSI_GREEN   = "2";
  public final static String ANSI_YELLOW  = "3";
  public final static String ANSI_BLUE    = "4";
  public final static String ANSI_MAGENTA = "5";
  public final static String ANSI_CYAN    = "6";
  public final static String ANSI_WHITE   = "7";

  public static final String ANSI_CLS         = ESC + "[2J";
  public static final String ANSI_HOME        = ESC + "[H"; // 0,0
  public static final String ANSI_HEAD        = ESC + "[1G"; // Start of current line, position 1

  public static final String ANSI_NORMAL      = ESC + "[0m";
  public static final String ANSI_BOLD        = ESC + "[1m";
  public static final String ANSI_FAINT       = ESC + "[2m";
  public static final String ANSI_ITALIC      = ESC + "[3m";
  public static final String ANSI_UNDERLINE   = ESC + "[4m";
  public static final String ANSI_BLINK       = ESC + "[5m";
  public static final String ANSI_BLINK_FAST  = ESC + "[6m";
...

and run like this:

    AnsiConsole.systemInstall();
    AnsiConsole.out.println(ANSI_CLS);
    AnsiConsole.out.println(ANSI_NORMAL + "Normal text and " +
                            ANSI_WHITEONBLUE + "bold" +
                            ANSI_NORMAL + " text.");
    AnsiConsole.out.println(ANSI_NORMAL + "Normal " +
                            ansiSetTextColor(ANSI_YELLOW) + "yellow" +
                            ANSI_NORMAL + " text and " +
                            ansiSetTextAndBackgroundColor(ANSI_WHITE, ANSI_BLACK) +
                            "bold" + ANSI_NORMAL + " text.");
    
    System.out.println(ANSI_NORMAL + "Normal text and " +
                       ANSI_WHITEONBLUE + "bold" +
                       ANSI_NORMAL + " text.");
    System.out.println(ANSI_NORMAL + "Normal " +
                       ansiSetTextColor(ANSI_YELLOW) + "yellow" +
                       ANSI_NORMAL + " text and " +
                       ansiSetTextAndBackgroundColor(ANSI_WHITE, ANSI_BLACK) +
                       "bold" + ANSI_NORMAL + " text.");

or even easier, use some static classes and methods provided by Jansi:

    AnsiConsole.systemInstall();
    System.out.println( ansi().eraseScreen().fg(RED).a("Hello").fg(GREEN).a(" World").reset() );
    System.out.println( ansi().bg(CYAN).fg(RED).bold().a("Hello").fg(GREEN).a(" World").reset() );
    System.out.println();
    System.out.println();
    System.out.print( ansi().fg(WHITE).a("Hello World") );
    System.out.flush();
    try { Thread.sleep(1000); } catch (Exception ex) {}
    System.out.println( ansi().eraseLine(Erase.ALL).cursorLeft("Hello World".length()).fg(YELLOW).a("Hello World, again").reset() );
    AnsiConsole.systemUninstall();

And the output is the same on Windows, on Linux,...

By the way, this is what Gradle is using for their output.

Start Windows Normally

I've been experiencing several crashes recently, apparently the video card on my Dell is going nut. The only way out is to hard-reset the machine.
And after that, when I restart, I always have the same screen:

Ah that's good one! I wish I could do that!

NMEA Station Calibration

We intend here to show the importance and the impact of an accurate calibration and tuning of your NMEA station, in order to correctly evaluate the parameters to calculate, namely the current direction and speed. The impact of an incorrect tuning leads to completely useless computation, and wrong evaluation of the expected parameters.
For a better understanding of the mutual impact of all those parameters, we've illustrated this document with several interactive applets, which you can play with or interact with the different parameters for yourself.
Values in fields can be changed (hit the [Enter] key for that, after modifying the value in the field), sliders are draggable with the mouse, as well as the compasses that figure in the applets, to change the various headings.

Steps

In order to log and visualize data efficiently, you need to complete several steps, in a specific order:

1. Instruments Calibration
   You need to make sure the values read by your instruments are accurate
2. Polars Elaboration
   You need to know what your speed should be in the condition you sail in
3. Data Logging
   You need to log the right data
4. Data Visualization
   You need to render the logged data appropriately, rate your tacks and gybes, visualize laylines and frames, etc.

Instruments Calibration

Most of the in-board NMEA stations provide access to calibration parameters that will lead to an accurate data logging and display. The goal of this process is to evaluate the parameters we will need to feed the NMEA station with.
You need to simulate the values of the parameters to enter in the NMEA station. This process follows several steps:

1. Log data on the boat
2. Display the data in some graphical dashboard
3. Elaborate the calibration parameters until the display looks good
4. Set the calibration parameters in the NMEA station
5. Log again, to make sure your evaluation of the tuning parameters was right

Polars Elaboration

Polars are a mathematic and theoric model that will tell you what BSP you should theorically reach for a given TWA and TWS.
Polars can be used to predict your potential track (routing), or to rank your boat handling.
There are two main ways to get the polars of your boat:

• VPP (Velocity Prediction Programs)
  Those data are (or should be) provided by the designer of your boat.
• Logging
  In case your designer did not provide them, or if the VPP is wrong...

Data Logging

For an appropriate rendering of your performances, you need to log the right set of data. The format of the data logged by your navigation station is NMEA.
Depending of the data your station is able to produce, you will have to choose the right ones.

Data Visualization

The main question to address is "What do you want to see?". And there is no ready-made answer to that question.

To summarize

1. Appropriate Calibration leads to the right evaluation of the True Wind (Angle, Speed and Direction) and of the Current (Speed and Direction)
2. With the right evaluation of the True Wind, you can refer to your polars, so you know what your speed should be
3. Having logged the right data, comparing them to what the polars tell you, you can evaluate your performances
4. Based on the evaluation of your performances, you can render your data appropriately, see where you did good, and where there is room for improvement

Instruments Calibration

Calibration impacts the data read on the boat through the devices attached to the NMEA station, like the boat speed transducer, the compass, the wind vane, and the anemometer.
We want here to

1. show how the different data interact with each other
2. see how to apply calibration factors (coefficients and offsets) to the values read from the station

The data are logged on the boat, that carries the devices (speedometer, compass, wind vane, anemometer).
It is far from unusual to have to tweak or tune those devices.
The anemometer is usually the most reliable one, as it is running away from most of the interferences that make the other devices in need for calibration.
For example, the compass and the wind vane could have been installed not exactly in line with the boat. This means that an offset is to be applied to the data they read.
Depending on the location of the boat speed transducer, the speed of the molecules of water does not necessary directly reflect the speed of the boat. A difference can even be seen from one tack to the other. A coefficient might then be required.
Many data can be recorded, we're only - in this document - going to talk about the data we need to evaluate the performances of the boat.
For example, DBT (Depth Below Transducer) is useless for us, in the framework on this document.
The main data recorded by the instruments are:

• Boat Speed (BSP), sometimes aka STW (Speed Through Water)
• Heading (HDG, HDM, HDT)
• Apparent Wind Speed (AWS)
• Apparent Wind Angle (AWA)
• Boat Position (from the GPS)
• Speed Over Ground (SOG)
• Course Over Ground (COG)

One data is to be estimated:

• Leeway (see about that the section about Damping)
  Mostly depends on AWA, AWS, and sea state (that involves waves, swell direction... pretty tricky).

The main data computed, based on the data above, are:

• True Wind Speed (TWS)
  Depends on COG & SOG, AWA, AWS.
  Not on BSP & HDG, as usually admitted.
• True Wind Angle (TWA)
  Depends on COG & SOG, AWA, AWS.
  Not on BSP & HDG, as usually admitted.
• True Wind Direction (TWD)
  Depends on TWA, HDG
• Current Speed
  Depends on BSP, HDG, SOG, COG, Leeway.
• Current Direction
  Depends on BSP, HDG, SOG, COG, Leeway.

Note: The tuple (SOG, COG) is semantically (in this case) equivalent to the Boat Position. Both (SOG, COG) and Boat Position come from the GPS.

A note about BSP calibration:

This one can be very tricky, specially if the transducer is not exactly set in the axis of the boat, as it is usually the case.
When you're sailing upwind, depending on the tack, the reading may substancially vary, as for a given speed, the pressure of the water molecules on the transducer is definitely not the same on both tacks, the difference of reading between tacks being more or less proportional to the heel.
To address that issue, most of the NMEA Stations provide a way to set the boat speed calibration for each tacks.
Unfortunatelly, this is not good enough, the Apparent Wind Angle should be taken in account as well. You certainly do not want the same coefficient to be applied on the boat speed when you are heading close hauled or running downwind.

Calibrating the Boat Speed and Heading

Here is an interactive applet:

---

This browser does not support Applets.

This applet shows how to evaluate/compute the current. It provides the possibility to interact with the calibration parameters and see how they impact the final resiult. Use the sliders in the applet on the left to modify the values of Boat Speed (BSP) and Leeway. To change the Heading (HDG) value, drag the mouse over the compass. Course Over Ground is fixed, set to 80°. Speed Over Ground is fixed, set to 5.5 knots. Both COG and SOG come from the GPS. Current Speed & Direction are computed based on the five values above. In green, data read from the NMEA Station In red, parameters used for computation In blue, computed data At the bottom, are two calibration parameters. The BSP Coefficient is applied to the Boat Speed. The HDG Offset is added to the Heading. To modify a parameter, change its value, and hit the [Enter] key. The display on the right side render the value read by the station. The pane on the left displays the corrected values, used for the computing of the current.

Something to say about leeway:

Leeway - mentionned above, and used below - is a dimension proportional (among others) to the Apparent Wind Angle (AWA), and not the True Wind Angle (TWA).
The Apparent Wind is the wind moving the boat.
Sailing on a very fast boat (like a racing multihull, an ice-boat, etc), it is not unusual to fell the wind on the nose, and see the wind-generated waves (not on an ice-boat, I know) coming from the quarter. The sails are trimmed according to the Apparent Wind, and the force they generate is expressed accordingly.
If your sails are trimmed as if you're going upwind (i.e. up the Apparent Wind), then you make leeway accordingly.
Picture this:
Sailing in the same external conditions (TWD, TWS), we have two boats, well trimmed, sailing at about 135° of the True Wind (TWA=135°):

• A cruising monohull, sailing between 6 and 8 knots
• A racing multihull, sailing above 25 knots

The first one will be sailing with the spinnaker up.
The second will have - in the best case scenario - a closed hauled gennaker.

Obiously, their leeway will not depend on the same parameters.
The fast boat will make more leeway than the cruising one, because of his close hauled sails.

For now, in the Dashboard Preferences, you have a value named "Max Leeway". This value will be used in conjunction with the Apparent Wind Angle, when it is between 0°and 90°, as follow:

Leeway = MaxLeeway × cos(awa)

This browser does not support Applets.
The True Heading, corrected with the Leeway is called Course Made Good (CMG).

Calibration of the Compass

Usually under-estimated! You think that because you sail on a plastic boat, your compass does not suffer any deviation... Think again!
Unless you have a gyroscope, you will need a compass to know what your heading is. Your heading is the base of many calculations, see above.
The first parameter to take in account id the magnetic declination, noted D. This Declination depends on your location, and the time. It varies with time, because the magnetic pole navigates around the geographical one. Any nautical chart will tell you what the declination is in the region and at the time your sail. You can get its value from several web-sites, including the NOAA's one. The RMC sentence also returns this data. Some computer algorythms are also available to calculate this important datum. Declination is the same for everyone sailing the same waters at the same time.
This is not the case of the deviation. The deviation is a parameter depending on the boat you sail. It is generated by the magnetic masses around the compas. As such, it depends:

• On the boat (holding those magnetic masses)
• The heading, as those masses "move" around the compass.

Steel boats have a big one, ferro-cement ones have a huge one (50° is not absurd for them...). Plastic boats, you never know! A loudspeaker close to a compass will have quite an impact on its reading... An engine (or a generator), same. When the engine is running, it might even be different... There is a lot of litterature about this topic, go check for yourself if you are interested.

---

This browser does not support Applets.

The generic shape of a deviation curve is d = A + (B.sin(Z)) + (C.cos(Z)) + (D.sin(2.Z)) + (E.cos(2.Z)) where Z is the magnetic heading. The deviation curve represented on the left is following this equation. This interactive applet shows the 3 different headings: True, Magnetic, and Compass. The Compass is the displayed (and recorded) one. You can modify the value of the Magnetic Declination. Plus (+) is East, minus (-) is West, for Declination and deviation. The algebric sum of the Declination and of the deviation is called Variation, noted W. The true heading is the one to be plotted on the chart, as the chart too refers to the true North (a.k.a. Geographical North).

The compass calibration is crucial. The data it provides is a parameter of all the computations we are interested in. If the deviation is not taken in account, its value will be put on the current's account, and that would be a big mistake.

Calibration of the Apparent Wind Speed and Direction

Here is another interactive applet. Like before, you can use the sliders, modify the values in the fields (hit the [Enter] key in the field to apply your changes), and drag the compass' rose left or right.

---

This browser does not support Applets.

This one shows how the calibration of the station interacts with the computation of the True Wind, which is the key to access the polars. The displays (on the right) are the one you would see on the boat when sailing, the data displayed on the main pane, on the left, are the (ideally) "real" data, taking the calibration parameters in account. Warning: This vector combination is the usually admitted one, but this is a simplified one! The real one - closer to the truth - will be exposed next.

Let's now go closer to the truth...
Unlike what's commonly admitted, the True Wind should be calculated with the Speed Over Ground and the Course Over Ground, instead of respectively the Boat Speed and the Heading.
Imagine that: you are "sailing" in absolutely NO wind, but you have a strong current, about 6 knots. Well, in that case, you feel an apparent wind of about 6 knots. If you do the calculation of the True Wind with the Boat Speed, the calculation will return a True Wind of 6 knots, in the direction opposite to the current.
But if you do it with the Course Over Ground and Speed Over Ground, then it will return a True Wind Speed of zero knot, which is right.
If the GPS data are available, they should be preferred for this computation.

This browser does not support Applets.

This one contains all the possible parameters (for now...), and computes the True Wind with the GPS Data (SOG & COG). For the clarity of the figure, Compass calibration parameters (deviation curve) are not represented. All the values are modifiable like previously. Heading and Course Over Ground have their own compass' rose, which is dragable on both cases. As you can tell, the figure is quite complex. And it becomes even more complex when you modify the data and parameters. Again, it is very difficult to isolate the parameters from each other, modifying one value impacts mostly all of the others... As a matter of fact this reflects the reality of an NMEA Station calibration. It is a long and precise process, which should not be under-estimated. Ultimately, what we look for here is the direction and the speed of the current, represented in green on the applet on the left, as this is what we will compute the route to give to the driver with, decide when to tack, etc. All the parameters we mentionned so far have a direct impact on this evaluation. If one of them is wrong, then the evaluation of the current is impacted.

Full validation: Stability of True Wind Direction and Current Direction

Now, let's put all the parameters together, as in the real world. The main difficulty is precisely due to this fact, there is no clear way to isolate them...
Below is the display of some data logging, the logging is done with a non-well calibrated NMEA station.
It displays two legs upwind, and two legs downwind.
The True Wind Direction is displayed in blue, the Current Direction is displayed in red,
The vertical panes on the left display the True Wind Direction (TWD), and the Current Direction (CDR).
The horizontal panes display the Boat Speed (BSP), True Wind Speed (TWS) and Current Speed (CSP).
TWD, CDR, TWS, TWA and CSP are calculated, based on the data logged on board. Any error in the data read from the devices (BSP, HDG, AWA, AWS) will deeply impact this calculation.
You can impact the parameters to apply on the data read from the station by modifying their values on the right pane of the applet.
See how the current direction varies everytime the boat heading is changing... The TWD is less impacted, but it is not steady either.

This browser does not support Applets.

This interactive applet represents data logged with a non properly calibrated NMEA Station. Play with the parameters, to see how they impact the rendering of the track. To modify a value, change the number in the field and press [Enter]. There is a slider you can use, just above the BSP Panel at the bottom. To display the right data, you must enter the following parameters: Multiply BSP by 1.1 HDG Offset 5° Multiply AWS by 1.01 AWA Offset 2° Max Leeway 10° Those paratemers should be used later on to calibrate your NMEA Station. Ideally, when the parameters are set correctly, the graphs rendering TWD, CDR, TWS & CSP should display a straight line. (Assuming of course that the wind and current conditions were stable during this logging session)

For now, the leeway is calculated as a function of the Cosinus of the AWA, from 0 to 90 degrees only (we assume no leeway beyond 90° of Apparent Wind Angle).
This is likely to be improved.

Data Logging

Logging everything the station produces is a possibility, but it can be a lot, and saturate the recording device.
For the best rendering, you need to log at least:

• The Boat Speed (BSP)
• The Boat Heading (HDG, HDM, or HDT)
• The Apparent Wind Speed (AWS)
• The Apparent Wind Angle (AWA)
• The Boat Position
• The Course Over Ground (COG)
• The Speed Over Ground (SOG)

Those data can be found - for example - in the following sentences:

Required Data	Available in NMEA Sentences
Boat Speed	VHW
Boat Heading	VHW, HDG, HDM, HDT
Apparent Wind Speed	MWV, VWR
Apparent Wind Angle	MWV, VWR
Boat Position	RMC, GLL
Speed Over Ground	RMC, GLL, VTG
Course Over Ground	RMC, GLL, VTG

The NMEA sentences mentionned in the table above have the following meaning:

VHW	Water Speed and Heading (Velocity Heading Water)
HDG	Heading, Deviation & Variation
HDM	Heading, Magnetic
HDT	Heading, True
MWV	Wind Speed and Angle
VWR	Relative wind direction and speed (Velocity Wind Relative)
RMC	Recommended Minimum Specific GPS/TRANSIT Data
GLL	Geographic Position, Latitude/Longitude
VTG	Track Made Good and Ground Speed

Note: RMC gives the boat position just like GLL, but also returns the Magnetic Declination.

Data Visualization

Damping

The damping is the kind of smoothing that will eliminate the aberrations. This is specially usefull when the data are logged in tough conditions. For example, when you are sailing upwind in choppy seas, the wind vane being set at the top of the mast will be shaked substancially. This will generate some data that will not reflect the reality.
The damping is taking for any given logged point X points before,and X points after, and calculates the average.

This browser does not support Applets.
Use the slider to change the damping factor.

Several panels of the Dashboard provide access to a damping factor.

A note about current evaluation

The way we've calculated the current so far is based on the values read and calculated at a given moment in time (BSP, TWS, TWD, COG, SOG).
Some of those values are themsleves based on others (TWS & TWA are based on BSP, CMG - based on HDG, leeway, Declination and Deviation -, SOG, COG.
The current calculation we've shown so far is a triangulation based on instant values.
Basing the calculation on damped values does not really help, but there is another way to smooth those values. We have seen that the current is the vector that sits between the tuple (CMG, BSP) and the tuple (COG, SOG).
The idea here is to evaluate over a period of time this difference between the position where the boat is, and the position where it should be with no current.
We can even simultaneously evaluate the current with different period of time, like 1 minute and 10 minutes, for example. This would be usefull when the boat heading is changing, for example when tacking or rounding a mark.

After one tack. See the different values in the three panels

After two tacks. See the different values in the three panels

After two tacks, motoring head to wind. See the different values in the three panels

Notice in each case the differences between the current speeds, and current directions.

Evaluate and rate your tacks and gybes

We'll be talking here of the method well-known under the name of "Tack Loss".
Its principle is simple, and it is the same for tacks and gybes; we'll expose it for a tack, you'll be after that able to transpose it to a gybe.

• You're sailing upwind at a constant speed
• You evaulate you VMG (Velocity Made Good)
• The tack begins
• After the tack, and after a given amount of time, you're sailing on the other tack, and you've recovered your best speed
• This is when the tack is being rated:
  • You calculate, based on the VMG before the tack, how far upwind you would have been if you had not tacked
  • You see how far upwind you actually are
  • You compare the two (the difference is evaluated in meters)

The tack loss is not - as one could expect - always negative. If you appropriately tack in a wind shift, you might have very good ratings!
The opposite can be true as well...
The tack loss is usually evaluated as a distance, measured - even on American and English boats - in meters.
It can also be rendered as a percentage, as we're going to explain.
A tack can be decomposed is three main steps.

1. You're heading full speed, and you begin the tack. Your VMG increases, due to the momentum of the boat
2. The boat not being anymore powered by the wind, and having the wind on the nose, the boat (and the VMG) slows down as the wind is crossed
3. The boat bears away, about to re-catch the wind, the sails will soon be trimmed on the other tack, the boat is recovering its speed

The steps described above correspond to this figure:

Rating the tack as a percentage corresponds to the comparison of those two areas (in grey):

this area in grey...

... with this one.

Next time, we'll talk about the data rendering.
It's also a pretty vast topic.